Data-driven market research
Times are changing fast. As CEO of Black Swan Data, I work with some of the biggest CPG brands in the world, and they all keep talking about it.
Why? Because these changes are not always in their favour.
Consumer trends are changing faster than ever, CPG companies are losing sales to small, agile competitors. These small competitors are fast and close to the ground: they can spot trends before the big brands and bring new products to the market faster than anyone. In fact, many of these companies are dedicated to just following these trends.
So why are blue chip companies getting pipped by these young guns? Surely these giants of the industry can muster the resources to respond?
Unfortunately, a large part of the problem for them is their market research. CPGs lost an estimated $2.3 billion on failed innovation in 2019 alone. Traditional research methods can’t give these companies what they need to respond in today’s dynamic marketplace. Companies either bet on the wrong trend or cannot act fast enough to take advantage of what they know.
Traditional market research is just not getting the job done in today’s world.
Market research is experiencing a paradigm change. Born into the post-war economy of Mad Men and Madison Avenue, market research was an appeal to science to bring rigour to understanding what customers wanted and what they would pay for.
Marketeers continued use of methods such as questionnaires, interviews and focus groups for the last 80 years pays testament to traditional research’s usefulness. However, instead of helping them win, the same methods that worked for large companies, even five years ago, are now starting to cost these companies sales and market share.
1. Overstatement bias (the ‘say/do’ gap)
2. Limited number of questions you can ask
3. Respondent engagement/quality
6. Ability to look forward and predict
Market research has always struggled with these problems, but until recently, companies had no other options to get the answers they needed.
The market research industry has not ignored the advent of the digital age. ESOMAR’s latest Global Research Report shows that spend on digital market research surpassed traditional methods back in 2018. However, not all digital market research has fixed the six big problems of the industry previously highlighted. This is worth a closer look.
Simply put, digital market research companies can be divided into two groups:
Can you guess where the real success story lies? Hint: the key doesn’t lie in just making old methods digital.
What is tech-enabled market research exactly? This is research performed by firms that apply more cost efficient, digital methods to gather the same data as before that used to require an army of researchers.
Tech-enabled market research allows companies to speed up their traditional market research and to do it at a much lower cost (two of the six problems). However, tech-enabled market research doesn’t crack the biggest research problems in today’s fast-moving world.
It doesn’t solve the claimed vs actual behaviour issue (i.e. the say/do gap). It still relies on prompted questions to an unmotivated sample of respondents which means it will rarely uncover something you didn’t already know about (the unknowns). And it won’t allow you to predict future behaviour with any degree of statistical accuracy.
Instead of tech-enabled firms that rely on old methods, let’s talk about tech-driven market research that’s new and creating something transformative.
With the advent of the internet, smart phones, and advancement in data collection, there is more data available on customers, their habits, and their preferences than ever before. Tech-driven market research utilises and unlocks the value of these behavioural data sources.
For example, UK market research company Streetbees is a tech-driven market research company that captures verbatim from 3.5M ‘bees’ across the globe who record product choices and consumption behaviour on their mobile phones. Streetbees’ AI and Natural Language Processing technology analyses and structures this vast collection of text and images into powerful consumer intelligence. Brands use it to gain a deeper understanding of natural, in-the-moment behaviour to identify how they can better serve their customers’ needs.
Another example is analysing social media conversation trends to anticipate changes in consumer behaviour and attitude. Every day, millions of us share our ideas, needs, wants, frustrations, and desires on social media platforms. For almost any consumer related topic that you can think of, somebody somewhere will be discussing and posting about it.
By analysing this vast and rich data source we can identify known and unknown consumer topics and understand how and why they are trending. Furthermore, by training data science models on historical social media data, we can scientifically predict which of today’s emerging trends will sustain growth and most impact consumer consumption and purchase behaviour in the future. This is incredibly useful information if you’re trying to create new product innovation ideas.
What people say now is only important if it can tell you what they will buy next.
Tech-driven market research’s combination of new data sources and data science techniques, mean that for the first time companies can overcome the original critical flaws in market research. And critically, progress from looking only at past behaviour in the rear-view mirror, to looking forward with statistical confidence about what’s on the horizon.
I know that my proposition sounds excellent, and you might be sceptical. How do we know this works? The proof is in the pudding.
PepsiCo is at the forefront of the market research paradigm change. Spearheaded by Chief Insights and Analytics Officer, Stephan Gans, PepsiCo has developed a tech-driven analytics capability called ‘Ada’.
Ada, named after mathematician Ada Lovelace, integrates and applies an ecosystem of progressive, cutting-edge market research tools into PepsiCo’s core business processes, bringing new capabilities to their 600-strong global team of Insight professionals.
In a recent webinar hosted by the Yale School of Management, Stephan talked about creating a “faster, stronger, better” Consumer Insights function. One that is “more predictive, more iterative, less sequential. More precise; end-to-end. More human, more empathetic, better connected with the everyday lives of our consumers.”
For example, they use an AI-driven tool called Tastewise to analyse millions of menu data points and understand what food and cuisines people are eating. They combine this with Black Swan’s trend prediction technology: Trendscope, to prioritise which emerging Beverage and Snacking trends are predicted to tip into the mainstream. Ideas created through this process are then rapidly tested and optimised through Zappi’s advanced new product concept testing platform.
Used alone, each of these tools offers a powerful, tech-driven solution that creates better answers to familiar questions. But integrated together through Ada, they combine to create an advantaged capability that sets PepsiCo apart from competitors in how they conceive new ideas and bring new products to market.
Industry research supports Gans’ vision. Looking at the CPG industry, BCG and Google saw that companies could improve their earnings by over 10% using AI and advanced analytics.
There are demonstratable gains to be made when companies start focusing on tech-driven methods in their research. Despite this, only 6% of CPGs in the study were using AI and advanced analytics to empower functions and drive key decisions at scale. Clearly this is a potential area of strategic competitive advantage. I suspect that Stephan Gans has the same notion.
Tech-driven market research can help you make more money, save more money and waste less money by helping you:
When forward-thinking companies like PepsiCo, Mondelēz International, Proctor & Gamble, Johnson & Johnson, and others are using tech-driven market research, it’s clear that the ship is sailing. Times are moving on.
If you are not on board yet, you are missing out.
Tech-driven market research already seems like an extreme paradigm shift, but this is just the first drop in the bucket. The application of Big Data and AI powered tools is not only going to mature, it is going to accelerate and spin-off into multiple use-cases and applications.
Can tech-driven research firms develop AI sophisticated enough to offer accurate, long term predictions of market size that can automate your entire new product innovation process? Can they offer a successful ‘Supply Chain as a Service’ outsourcing platform with accuracy in long term supply chain, and output with zero waste? I think so.
The opportunities are many, and the future is full of promise.
Do you have a good idea about the future of AI and market research? Do you think you are missing out? Get in touch, I would love to hear from you.