Big data, human intellect – a winning combination

By: Heidi Dietzsch

There is one sure thing in this world we live in and that is change. Various developments and innovations, especially technological innovations, are reshaping the world faster than ever before. This is certainly the case for market research.

Change is, however, never without challenges. The future of market research is full of challenges – how to incorporate and use new technologies, how to redefine the role of researchers and how to build and strengthen partnerships with clients,[1] among others.

For decades, market research was in a stable, comfortable position, but is now being forced to evolve and revolutionise to keep up with a society that is changing rapidly. These could be scary times indeed for market researchers who are unwilling to adapt. Data quality, respondent recruitment, traditional methods and even the purpose of market research have all been subjected to transformation and will undoubtedly lead to uncertainty.[2]

One of the main changes relate to the survey as we know it. For the longest time surveys have been at the forefront of data collection. However, the growth of smartphone penetration globally has brought a great deal of change to traditional surveys.

A big challenge of orthodox surveys is that respondents need to rely totally on their memories – which is not always to be trusted. One of the solutions to this are short intercept surveys (SIS) where respondents document their views and perceptions on their smartphones while busy with a particular action – for instance, buying a new product at the store. SIS are short and unlikely to cause respondent fatigue and are done while requested information is still fresh in respondents’ memories.[3] Michalis Michael, the CEO of UK market research firm DigitalMR, believes that these types of studies will eventually replace long monthly customer tracking studies.

Michael also believes that social listening analytics will become an absolute must-have for market researchers.[4] Social listening entails the monitoring of a brand’s presence on social media, followed by an analysis of the perceptions social media users have of the brand. This tool allows brands to realise how big and influential their presence is in social media, how many mentions they receive and on which channels, and whether these mentions are positive or negative. Also, brands can see how they compare to their competitors and whether their image and reputation are favourable.

Social listening analytics can also enable brands to take concrete action. They can create the kind of social media content that attracts clients, come up with new ideas – and even products and services – based on industry trends. Brands will be able to improve the customer experience by interacting directly with clients and continuously shift their customer strategy to fit the current need.[5]

Another rising trend is the use of passive data that is collected without specifically asking people for it. This is also called implicit data. Conversely, use of active data or traditional data relies on the participation of people and is known as explicit data.[6]

Kristin Luck, founder of US management consultancy ScaleHouse, believes that market researchers will increasingly make use of passive data.[7] Passive data collection has signalled the era of big data – the volume of data which you can gather in a short space of time can be staggering. Imagine the scenario: people are purchasing on an app. It might only take them a few moments to complete the purchase but in that short space of time researchers can glean their location at point of purchase, their interests, their spending behaviour and preferences on payment methods. If they’re using a loyalty card, you then have all that information to mine as well.

The passive data collection process requires minimal involvement from researchers themselves. However, the researcher’s role as an analyst who mines a vast quantity of data to pinpoint key information will become more important.[8]

As in most areas of our everyday lives, market research is progressively becoming automated. This is good news for clients – who need results yesterday – as well as for researchers who can now free themselves of time-consuming tasks and focus on generating insight.

One of the most difficult tasks in market research is to recruit respondents to participate in studies – that is, enough respondents to reach clients’ samples and also fit their sometimes very specific respondent criteria. This process has become more and more automated due to the increase in aggregation services. These services refer to sample vendors that use multiple sources to create larger, more accessible respondent panels. While the premise of aggregation is simple, it is the filtering and selection technology behind it that thrusts vendors into automated territory. By simply selecting a few basic parameters, these panel providers send invitations to relevant respondents, making it easy to fulfil nearly any client requests.[9]

Then there is artificial intelligence (AI). People tend to confuse AI with automation, but these two concepts are not the same. Even in its most complex forms, when a task is automated, software follows the instructions it has been given. The software does not make any decisions or learn something new each time the process runs. Learning is what distinguishes AI from automation.

AI can empower researchers with insights and analysis that would not have been possible previously. For instance, it has the ability to process open-ended data or large, unstructured datasets using statistical analysis techniques.[10]

All these new developments and technological advancements can certainly create a sense of unease – especially in an industry that is renowned for its tried-and-tested methods. Nevertheless, they do have strong potential to enhance market research and help ensure the survival of the industry. Stalwarts desperately clinging to the “good old days” will, unfortunately, not be so lucky. New technologies should be embraced and researchers must be willing to learn and adapt. Most importantly, however, deep insights require the creativity and imaginability that no machine will be able to replace.