Podcast Summary: "AI Meets Human Insight: UX Research at Consumer Reports"

Podcast Information:

• Title: Insights Unlocked
• Host/Author: UserTesting
• Episode Title: AI meets human insight: UX research at Consumer Reports
• Release Date: February 10, 2025

Overview: In this episode of Insights Unlocked, hosted by Brett Leary, Melissa Garber, the Senior User Experience Researcher at Consumer Reports, delves into the synergistic relationship between Artificial Intelligence (AI) and human insight in the realm of UX research. Recorded at the Human Insight Summit in October, the conversation explores how Consumer Reports leverages nearly a century's worth of research data to enhance AI-driven tools, ensuring accuracy, trust, and enriched consumer experiences.

1. Introduction and Background

The episode kicks off with Brett Leary introducing Melissa Garber and setting the stage for an in-depth discussion on AI's impact on UX research at Consumer Reports.

• Melissa Garber shares her professional journey, highlighting her background in learning theory and cognitive science, which has been instrumental in her role in user research. She mentions, “I just started running research with folks doing card sorting, getting their mental models, sitting down with people and doing ethnographic interviews” (01:33).

2. Integrating AI with Human Insights

Melissa elaborates on how Consumer Reports integrates AI with human research to build a conversational AI agent. This integration is powered by a vast repository of data spanning nearly 90 years.

• Utilizing Historical Data: Melissa explains, “We're using a RAG framework to pull data from all of the data sources that we have from almost 90 years of research” (04:53). This extensive dataset enables the AI to predict user intent and retrieve relevant information effectively.

• Human-in-the-Loop Approach: To maintain trust and accuracy, Consumer Reports employs a human-in-the-loop system. Melissa states, “We have folks that are still looking at what the AI is creating and making sure that it is not just spouting out nonsense to people or that we are, like, maintaining our brand integrity and the trust that we've built over almost 90 years” (21:00).

3. Challenges and Evolution with AI

The discussion shifts to the challenges faced while integrating AI into UX research and how the approach has evolved over the past two years with the advent of Generative AI (GenAI).

• Data Privacy and Security: Melissa underscores the importance of data privacy, noting, “We work with trust and safety and security folks, like experts in the field to make sure like data privacy is a huge thing” (06:28).

• Adapting to AI Tools: Melissa reflects on the evolution of AI tools, emphasizing the need to critically assess the use cases for AI. She mentions, “Is it AI that we need to use for this solution, or are there other ways? Could we use machine learning?” (18:40).

• Overcoming Resistance: Introducing AI required navigating skepticism from seasoned researchers. Melissa shares her approach: “I was like, well, let me just prove this out. Let me just do it and maybe like, do it and ask forgiveness rather than permission” (11:23).

4. Building the Conversational AI Agent

Melissa provides a detailed account of creating Consumer Reports’ conversational AI agent.

• Collaboration with Experts: The Innovation Lab collaborated with subject matter experts to incorporate their deep knowledge into the AI system. “We talked to our subject matter experts and they brought in all of that information, all of that expertise to help define what this system was going to look like” (07:23).

• Training with Real Data: By utilizing transcripts from SCR 1.0, where real humans answered consumer questions, the team could model and predict common inquiries, enhancing the AI’s responsiveness. “We had something called SCR 1.0 that was real humans answering real questions” (07:23).

5. Audience Engagement and Q&A Highlights

During the summit, Melissa engages with the audience, addressing various questions that shed light on her methodologies and perspectives.

• Building the Research Function: Irene asked about expanding the research function. Melissa responded by emphasizing democratization and incremental value: “Let me just prove this out. Let me just do it and maybe like, do it and ask forgiveness rather than permission” (11:23). She further explained the division of research tasks between design and product management teams to optimize efficiency.

• Future of AI in UX Research: When queried about the future, Melissa emphasized AI’s role in empowering consumers rather than replacing human roles. “The way that we're leveraging AI is to help make our consumers smarter, be the superheroes of their own lives” (18:40).

6. Future Directions and Takeaways

Melissa concludes by reflecting on her key takeaways from the Human Insight Summit and her plans moving forward.

• Journey Mapping Innovations: Inspired by a session on journey mapping, Melissa plans to incorporate chaos maps that highlight critical moments, stating, “I felt like that was so powerful that I intend on bringing that back to Consumer Reports” (14:04).

• Evolving Measurement Metrics: With a new Chief Experience Officer, Melissa anticipates integrating customer experience more deeply into success metrics, aiming to transform organizational focus towards enhanced consumer satisfaction (13:53).

• Embracing Imperfection: Melissa realizes the importance of flexibility in presenting consumer journeys, overcoming previous barriers of perfection. “That’s always been a blocker for me… I feel like a little bit more freed to be able to just kind of pull out, like visually represent what is important and where we can impact things” (23:11).

7. Conclusion

The episode wraps up with Brett Leary thanking Melissa Garber for her insightful contributions. Listeners are encouraged to subscribe, share, and engage with Insights Unlocked through various platforms.

Notable Quotes:

• "It depends... sometimes I joke that I'm asked to be a research vending machine." — Melissa Garber (02:37)

• "People don't know what they don't know. So even if you have a conversational chatbot, you might not know what to ask it." — Melissa Garber (09:51)

• "AI doesn't reason. It can pull sentiment, but it doesn't understand emotion." — Melissa Garber (20:10)

Key Takeaways:

1. Integration of AI and Human Insight: Combining extensive historical data with human expertise leads to more accurate and trustworthy AI-driven tools in UX research.
2. Human-in-the-Loop: Maintaining human oversight ensures the integrity and reliability of AI outputs, safeguarding brand trust.
3. Gradual Adoption and Democratization: Introducing AI gradually and demonstrating its value fosters acceptance and collaboration among research teams.
4. Empowering Consumers: AI should aim to enhance consumer decision-making, making them more informed and empowered.
5. Continuous Learning and Adaptation: Staying abreast of innovations and integrating new methodologies, such as chaos mapping in journey mapping, can significantly improve customer experience strategies.

Final Thoughts: Melissa Garber's insights highlight the delicate balance between leveraging advanced AI technologies and retaining the essential human elements that drive meaningful consumer experiences. Her approach at Consumer Reports serves as a model for integrating AI in a manner that complements and enhances human research capabilities, ensuring that technological advancements lead to better, more informed, and trustworthy consumer interactions.