How to apply AI effectively and safely

Introduction

AI has been and will be the biggest disruptor of consumer research and product innovation since the birth of the Internet. At an increasing pace AI is automating workflows and providing access to new types of consumer insights and predictions fueling product innovation. It is no surprise that ambitious business leaders and teams are actively seeking AI-solutions that they can safely use.

AI raises a lot of excitement but also fair questions and concerns:

• How can AI be applied to its full potential in the context of product innovation and consumer research? Generative AI (one application being Chat GPT) is only the tip of the iceberg.
• How can we make a difference between a good and bad AI solution?
• How do we know that we can trust the outputs of an AI solution?
• How can we safely start using and scaling AI?

In this paper we address the above questions. Our aim is to educate the reader and encourage them to take an active and decisive role in applying AI.

The AI landscape and use case examples

Not all AI is equal and it comes in various forms and levels of complexity, so before diving into specific applications, let's start by looking at the definitions of these different types of AI available: 

Artificial Intelligence (AI) encompasses various subfields and techniques aimed at simulating intelligence in machines. Among the key subcategories of AI are Machine Learning (ML), Natural Language Processing (NLP), Robotics, and Expert Systems.

Machine Learning, a subset of AI, focuses on equipping computers with the ability to learn from data and make predictions or decisions without explicit programming. Its methodologies include Supervised Learning, Unsupervised Learning, Semi-supervised Learning, Reinforcement Learning, and Deep Learning. Through these techniques, machines can discern patterns, derive insights, and improve their performance on various tasks.

Generative AI (GenAI), an evolving branch of AI and Machine Learning, concentrates on creating models capable of generating new data samples akin to those in the training dataset. Generative AI facilitates tasks such as image generation, text synthesis, and data augmentation.

The use cases of AI vary widely, depending on the business context. To help understand how AI can be used, we have classified the use cases into more general categories in the table below: 

How AI is applied in consumer research within product innovation

In the area of consumer research and product innovation, AI is applied in versatile ways to solve small problems but also bigger more complex ones. In this next section, we introduce a few common and emerging use cases.  

Automated labelling of language - NLP. NLP (Natural Language Processing) is an effective way to capture key learnings from consumer feedback without human involvement. The most common use cases are content analysis and sentiment analysis. The field is very difficult because human language is nuanced and ambiguous, and meanings depend on context. Product innovation domain of consumer research requires its bespoke NLP models so that the output is relevant to the product innovator and so that it can be trusted.

Summarizing of content - NLP. Generative AI is well suited for summarizing research content from individual studies to meta-analyses. While GenAI is a great storyteller, it easily hallucinates and lacks the ability to generate nuanced and concrete insights. According to our experience, GenAI performs best in a tightly specified context and when combined with other AI solutions.

Generating new ideas - GenAI. As the training datasets of GenAI models are vast, GenAI is the world’s fastest idea generator inspiring the creative minds of us humans. One can think of GenAI as a great new team member who is always eager to experiment with new concept directions. The more context the GenAI gets, the more relevant ideas it can generate.

Predictive eye tracking - Prediction. Researcher (designer) inserts a stimulus, and an AI provides an instant prediction on what in the stimulus is likely to capture human attention. The model has been built by using specialized technology and devices that track eye movement and gaze. Respondents have been exposed to several types of stimuli in laboratory environment or via webcam during which their eye movement and gaze has been tracked. Based on patterns in this training dataset, the machine learning (ML) model predicts what captures attention. Based on our experiments, it provides most value when teams can still iterate the creative output, such as packaging design or web page.

AI interviewers - GenAI. The AI interviewer is built on GenAI which is applied across the process from prompting the research context and questions to generating AI summaries. Obvious advantages are time and cost savings since AI interviewers never get tired. Interviewing 500 respondents and analyzing data takes less than a week. Based on our experiences, the insights from the AI interviewers currently fall in between quantitative and qualitative research. They provide the biggest value in pre-studying a new field when resources are scarce. They are not yet best suited for contextual and in-depth exploration or feeding product launch decisions.

Survey data quality auditing – Anomaly detection. These AI solutions track the answering behavior of respondents and predict if the respondent is a fraud or a high-quality respondent. This is a difficult field as AI and especially GenAI develop so fast. The solutions that we have been piloting apply either machine learning or Generative AI. Until now we have not identified a solution that would beat our own automated rule-based solution. This field develops fast; as soon as we get proof that an external solution works better, we will switch to it.

Synthetic data - GenAI. By synthetic data we mean a case where AI predictions replace partly or fully human respondents (panels). One can also think of synthetic data as a panel where human respondents are replaced by predicted AI respondents. This is a highly complex and interesting field of AI. The complexity of this endeavor stems not only from the technical challenges associated with accurate modeling of human responses but also from the ethical and methodological considerations of ensuring that these synthetic responses are representative, unbiased, and reflective of the diversity inherent in human populations, essentially across different markets and demographic groups. We anticipate that synthetic panels will co-exist with human panels complementing them: allowing product innovators to test and iterate more frequently. At this point, many solutions in this space suffer from the so-called WEIRD-syndrome, i.e. the simulated responses are likely to work only for Western, Educated, Industrialized, Rich, Democratic populations.

Predicting in-market product innovation outcomes - Prediction. A group of AI solutions that focus on predicting how well the tested product concept is likely to succeed in the marketplace. In addition to synthetic data, we regard this as the most complex field within consumer research. It is a highly complex data science and marketing science problem to solve. ‘The license to live’ for these AI models is simply that there is clear proof that they can provide accurate and generalizable predictions: will the tested product (concept) be a market success or not. After all, businesses are making business-critical and expensive decisions based on these predictions. Prerequisite for success for these kind of AI models is high quality and versatile pre-launch and in-market data (post-launch data). Additionally, model choices need to be such that the models themselves as well as their outcomes are explainable to users. Accuracy, generalizability, transparency and proof of performance (value) are the key success criteria.

Characteristics of “Good AI solution”

As in any business, the field of AI is a playground of good and bad solutions. Of course, what is regarded good or bad is often subjective and a contextual conclusion, based on some evaluation criteria. Outlined below is what we regard as a good AI solution in the context of product innovation and consumer research.

Clear purpose and business application. The AI solution addresses and solves a clear pain and aim. In other words, in one way or another, it reduces waste from this planet while generating new value to the business users or consumers. In order to succeed in this aim, our experience is that the team needs to include both data science, domain experts and end users who co-create and iterate the solution that solves a problem. AI solutions need to be developed with the use case in mind, and choosing the tool based on that. Choosing the tool or technology first will result in bad quality.

Explainability. The multi-disciplinary team who has been working on the model can explain how the model has been built and why, and how it works to a business user who has no prior experience in AI or data science. The team and the business users need to be able to carry out a fruitful discussion and debate on the underpinnings of the model. When the model is easy to understand, it also makes the adaptation easier within the organization. Using mathematical tools that can explain why AI made a certain choice or prediction are likely to increase trust in the system.

Appropriate data. What you put in is what you get out also from the AI solution. What is an ‘appropriate’ dataset depends fully on the business problem that the model is solving. For example, when the AI model aims at predicting launch success, it needs both in-market performance data and survey data to do its job accurately. If the data is not appropriate, no AI model can serve the business purpose.

Model performance metrics in use. The team responsible for the AI models needs to have relevant metrics in use and share the model performance to the users either proactively or when asked. The metrics and their scores are the proof to the users that the AI model delivers what it promises. The metrics need to be evaluated on a validation dataset not used for model training, to provide insight on the model's generalizability outside of its training data. The metrics used depend on the type of the model. In the context of consumer research, users want valid and reliable data and predictions for their decision making. For example, in the context of NLP (automated text analytics) validity and reliability are measured using Precision (Hit Rate = how well the model ignores false content), Recall (Capture Rate = how well the model identifies all the correct content), and F1 Score (combination of these two). By logging metrics, it is also possible to improve the models gradually, with confidence that the new version is truly an improvement.

High model performance standards. We don’t believe in “fake it until you can make it”. The model needs to fulfill its purpose and help business users achieve their goals when it is launched. The team needs to set clear targets and thresholds that they follow. Working with AI solutions follows the modern product innovation best practice, meaning iterative approach: starting with experiments, working with a Beta version, launching the full solution and finally extending and improving it.

No data privacy or IPR violations or data leaks. The AI models need to follow the same business standards and rules as any other models or processes.

Conclusion 

We hope you have found this paper on AI useful and informative. Our aim is to share the knowledge and learnings we have gained over the last 5 years experimenting and building with AI to ultimately harness it to power our insights and innovation management platform.  We believe AI will be a positive development for everyone: business, teams, as well as consumers. Things that will need careful consideration along the way are quality of data and predictions,  data privacy and IP. 

If you are interested in reading more on this subject, you may want to check out the 22 questions Esomar have published which acts as a check list for buyers of AI services for market research and insights. You can find it here:

Contributors:

Dr. Heli Holttinen, Founder, CPO

Dr. Tommi Pajala, Data Science Team Lead

Dr. Hongyu Su, Data Science Team Technical Lead

Eetu Pursiainen, Senior Data Scientist