AI meets human insight: UX research at Consumer Reports

Summary

Podcast Summary: "AI Meets Human Insight: UX Research at Consumer Reports"

Podcast Information:

• Title: Insights Unlocked
• Host/Author: UserTesting
• Episode Title: AI meets human insight: UX research at Consumer Reports
• Release Date: February 10, 2025

Overview: In this episode of Insights Unlocked, hosted by Brett Leary, Melissa Garber, the Senior User Experience Researcher at Consumer Reports, delves into the synergistic relationship between Artificial Intelligence (AI) and human insight in the realm of UX research. Recorded at the Human Insight Summit in October, the conversation explores how Consumer Reports leverages nearly a century's worth of research data to enhance AI-driven tools, ensuring accuracy, trust, and enriched consumer experiences.

---

1. Introduction and Background

The episode kicks off with Brett Leary introducing Melissa Garber and setting the stage for an in-depth discussion on AI's impact on UX research at Consumer Reports.

• Melissa Garber shares her professional journey, highlighting her background in learning theory and cognitive science, which has been instrumental in her role in user research. She mentions, “I just started running research with folks doing card sorting, getting their mental models, sitting down with people and doing ethnographic interviews” (01:33).

2. Integrating AI with Human Insights

Melissa elaborates on how Consumer Reports integrates AI with human research to build a conversational AI agent. This integration is powered by a vast repository of data spanning nearly 90 years.

• Utilizing Historical Data: Melissa explains, “We're using a RAG framework to pull data from all of the data sources that we have from almost 90 years of research” (04:53). This extensive dataset enables the AI to predict user intent and retrieve relevant information effectively.

• Human-in-the-Loop Approach: To maintain trust and accuracy, Consumer Reports employs a human-in-the-loop system. Melissa states, “We have folks that are still looking at what the AI is creating and making sure that it is not just spouting out nonsense to people or that we are, like, maintaining our brand integrity and the trust that we've built over almost 90 years” (21:00).

3. Challenges and Evolution with AI

The discussion shifts to the challenges faced while integrating AI into UX research and how the approach has evolved over the past two years with the advent of Generative AI (GenAI).

• Data Privacy and Security: Melissa underscores the importance of data privacy, noting, “We work with trust and safety and security folks, like experts in the field to make sure like data privacy is a huge thing” (06:28).

• Adapting to AI Tools: Melissa reflects on the evolution of AI tools, emphasizing the need to critically assess the use cases for AI. She mentions, “Is it AI that we need to use for this solution, or are there other ways? Could we use machine learning?” (18:40).

• Overcoming Resistance: Introducing AI required navigating skepticism from seasoned researchers. Melissa shares her approach: “I was like, well, let me just prove this out. Let me just do it and maybe like, do it and ask forgiveness rather than permission” (11:23).

4. Building the Conversational AI Agent

Melissa provides a detailed account of creating Consumer Reports’ conversational AI agent.

• Collaboration with Experts: The Innovation Lab collaborated with subject matter experts to incorporate their deep knowledge into the AI system. “We talked to our subject matter experts and they brought in all of that information, all of that expertise to help define what this system was going to look like” (07:23).

• Training with Real Data: By utilizing transcripts from SCR 1.0, where real humans answered consumer questions, the team could model and predict common inquiries, enhancing the AI’s responsiveness. “We had something called SCR 1.0 that was real humans answering real questions” (07:23).

5. Audience Engagement and Q&A Highlights

During the summit, Melissa engages with the audience, addressing various questions that shed light on her methodologies and perspectives.

• Building the Research Function: Irene asked about expanding the research function. Melissa responded by emphasizing democratization and incremental value: “Let me just prove this out. Let me just do it and maybe like, do it and ask forgiveness rather than permission” (11:23). She further explained the division of research tasks between design and product management teams to optimize efficiency.

• Future of AI in UX Research: When queried about the future, Melissa emphasized AI’s role in empowering consumers rather than replacing human roles. “The way that we're leveraging AI is to help make our consumers smarter, be the superheroes of their own lives” (18:40).

6. Future Directions and Takeaways

Melissa concludes by reflecting on her key takeaways from the Human Insight Summit and her plans moving forward.

• Journey Mapping Innovations: Inspired by a session on journey mapping, Melissa plans to incorporate chaos maps that highlight critical moments, stating, “I felt like that was so powerful that I intend on bringing that back to Consumer Reports” (14:04).

• Evolving Measurement Metrics: With a new Chief Experience Officer, Melissa anticipates integrating customer experience more deeply into success metrics, aiming to transform organizational focus towards enhanced consumer satisfaction (13:53).

• Embracing Imperfection: Melissa realizes the importance of flexibility in presenting consumer journeys, overcoming previous barriers of perfection. “That’s always been a blocker for me… I feel like a little bit more freed to be able to just kind of pull out, like visually represent what is important and where we can impact things” (23:11).

7. Conclusion

The episode wraps up with Brett Leary thanking Melissa Garber for her insightful contributions. Listeners are encouraged to subscribe, share, and engage with Insights Unlocked through various platforms.

Notable Quotes:

• "It depends... sometimes I joke that I'm asked to be a research vending machine." — Melissa Garber (02:37)

• "People don't know what they don't know. So even if you have a conversational chatbot, you might not know what to ask it." — Melissa Garber (09:51)

• "AI doesn't reason. It can pull sentiment, but it doesn't understand emotion." — Melissa Garber (20:10)

Key Takeaways:

1. Integration of AI and Human Insight: Combining extensive historical data with human expertise leads to more accurate and trustworthy AI-driven tools in UX research.
2. Human-in-the-Loop: Maintaining human oversight ensures the integrity and reliability of AI outputs, safeguarding brand trust.
3. Gradual Adoption and Democratization: Introducing AI gradually and demonstrating its value fosters acceptance and collaboration among research teams.
4. Empowering Consumers: AI should aim to enhance consumer decision-making, making them more informed and empowered.
5. Continuous Learning and Adaptation: Staying abreast of innovations and integrating new methodologies, such as chaos mapping in journey mapping, can significantly improve customer experience strategies.

Final Thoughts: Melissa Garber's insights highlight the delicate balance between leveraging advanced AI technologies and retaining the essential human elements that drive meaningful consumer experiences. Her approach at Consumer Reports serves as a model for integrating AI in a manner that complements and enhances human research capabilities, ensuring that technological advancements lead to better, more informed, and trustworthy consumer interactions.

Transcript

[Host]

Welcome back to the Insights Unlocked podcast. In this episode, we're exploring how AI and human insight work together in UX research with Melissa Garber from Consumer Reports. She shares how her team is using 90 years of research data to power AI, all while keeping humans in the loop to ensure trust, accuracy, and better consumer experiences. This interview was previously recorded at the Human Insight Summit back in October. Enjoy the show.

[Brett Leary]

Welcome to Insights Unlocked, an original podcast from User Testing, where we bring you candid conversations and stories with the thinkers, viewers and builders behind some of the most successful digital products and experiences in the world, from concept to execution.

[Melissa Garber]

My name is Brett Leary and I get a chance to spend a few good minutes with Melissa Garber. I want to make sure I get her title right. Senior User Experience Researcher for Consumer Reports. Right. I get it right.

Yes.

All right, well, thank you for joining me. This is great.

Awesome. Thank you for having me.

So you didn't know we're going to be talking about sports and all that?

Absolutely not.

Okay. No, we're not going to be talking about sports. Maybe a little bit. Maybe we're going to just spend a few minutes getting to know you, getting to pick your brain a little bit about what you do and how you do it and maybe how things are going to be changed by this two letter word that everybody keeps talking about. It's not even a word actually. It's just AI. But before we get to that part, maybe you could tell us a little bit about yourself.

Sure. So I am a researcher at Consumer Reports. I've been in the research and user experience space for a while. Hopped around from startups to I worked for indeed for a while. And my background is actually in learning theory and cognitive science. So it's been like a really good combination for me to be in user research and to also have that background of learning theory and how do people interpret information.

You left out the best parts. Like you grew up a Philadelphia Phillies and Eagles sports fan and all that kind of stuff, Right?

Yeah, my family had season tickets growing up. I think it was more of like a push to be a fan. Like, I'm not sure I had a choice.

Okay, well, that's okay. All right. So they gave me a script of some questions. They called it the Fast five.

Sure.

I'm gonna completely ignore that. No, I'm just kidding. I'm just gonna use some. We'll mix in some of those and maybe some other things that come out. But I will state to the script and ask you what your favorite word is.

Sure, depends.

That's your favorite word.

That's my favorite word. Yep.

You wanna expound on why that's your favorite word?

Why do you think?

It depends.

It depends. So I think that especially being in research, everything has a context, right. And so you can't just have one right answer. Sometimes I joke that I'm asked to be a research vending machine. Right. But it depends based off of like, who are we talking about, what experience, where are they coming from? And also in my personal life, it depends. I have two small children and the answer is not usually no or yes. It's let's figure it out. It depends. Yeah.

All right. I kind of see the theme we got going here. Talk a little bit more about the experience of being a researcher. How did you actually get into research?

I think I kind of just fell into it. So I studied digital media design for learning and I was tasked with bringing this new learning management system to a company that I was working at. And my first thought was I need to go talk to the people, the administrators who have to use this system throughout the company. I need to go talk to the actual end users. And I just started running research with folks doing card sorting, getting their mental models, sitting down with people and doing ethnographic interviews. And I actually brought all of that to a different sort of case study where I was like, hey, this is how I used user research to develop an in house vendor. Basically it was like a third open source learning platform and that we were able to sort of iterate on continuously. So being able to bring user insights from both the admins and the end users on like a regular cadence is really powerful.

So you said you've been doing this for a while? Quite a while. I'm guessing that means before two years.

Yes. Right.

Okay, good. So I almost sort of look at the world and maybe the business world and like pre chatgpt and post chatgpt. So how has the way that you work and the way that you do things, how has that changed over the last two years since the growth of AI and Genai? And everybody is looking at how AI is disrupting everything, but how has it changed or impacted how you do your job?

I kind of want to reframe that. So I think research can impact how we build AI. Right. I don't think it's. It's not just a reactive thing that we're using in our day to day, which I do use it in the day to day, but I've also had the opportunity at Consumer Reports, we're working on a conversational AI agent that we built the orchestration ourself in house. So we're using a RAG framework to pull data from all of the data sources that we have from almost 90 years of research. Right. And so with research, we're able to look at that and say, okay, well this is what our users are actually looking for. And we can start sort of predicting like what are they looking for, what's their intent? And we use that to find out. Okay, so where, like what data are we going to start retrieving from all of these different sources? Like what kinds of data need to come in? And that is the context that informs the prompt that we create to the LLM to get that answer back to our users. And then we're also able to use research to say, okay, our users want a different kind of like way of getting that information. Maybe they want like a, in a card format. Maybe they want it in like a specific kind of format. What kind of data? How's the content design, what's the tone, what's the voice?

So was it that simple to actually jump into leveraging AI or was what were some of the things, maybe. Were there any apprehensions? Were there any, wow, is this going to change things too much? Kind of take us through how you kind of evolved, if that was. Is that the right term to really put it to use and actually start seeing how not only could it impact you, but impact the way that your consumers or your customers.

Yeah. So on the consumer side of it, we work with like trust and safety and security folks, like experts in the field to make sure like data privacy is a huge thing, a consumer Reports and protecting consumers data. So that is handled by the experts on that side. And then on like my personal side, I use, we have an enterprise OpenAI account. So when I'm using ChatGPT, I'm not using a system that's actually reporting it back to the broader world. Right. It is only it's not training data sets outside of Consumer Reports. So just making sure that, you know, really particular about like where you're putting your information.

You said you had 90 years worth of data almost. That's pretty, I mean, a lot of folks are so looking at what's current. How do you kind of mix that with all that's going on today to bring the kind of information or answers that customers are looking for?

Well, what's really cool is that in the, in the past, you know, you, you could look at the magazine, right? You could go to the website, you could try to find like the best refrigerator. But we have these testers at Consumer Reports that have been doing it for 30 plus years. They're experts in their field. They know so much about their area. And we realized like, okay, if you talk to like a tester in the fridge department and you were to say, like, oh, I think there's like a whirring sound with my fridge. They're like, I actually, I know what that issue is. Right. Like that's probably your dust condenser coil that you need to clean. And so being able to actually go around the company, and this is actually what our Innovation Lab team did, is they went around the company when they were creating this conversational agent and they talked to our subject matter experts and they brought in all of that information, all of that expertise to help define what this system was going to look like then. Additionally, we also had a original, it's called ASCR, this agent. But we had something called SCR 1.0 that was real humans answering real questions that people had. And we were able to take all of those transcripts and model those transcripts with the data science team and the Innovation Lab and figure out what were people asking about so we could start predicting the types of questions, questions people might have.

Nice. Then you said some of your researchers have been around like doing this for 30 years.

Yeah, over 30 years. Really great employee retention, I think.

Has it been. Were there any kind of challenges with getting those folks to kind of buy into the way things are moving with AI and leveraging that to do maybe do things differently than they did in the past?

Yeah, I mean, I'm sure, I think that there's probably like a back and forth, right. Showing what AI could do and how it can help their work, as well as having their work inform AI and make sure that it represents Consumer Reports and our expertise and our tone and our voice and including folks throughout that process. So our product team did a really amazing job, our Innovation Lab, bringing in people throughout the development process and making sure that we were dogfooding our product.

Has there been any real surprises or things that you didn't really think of as you started digging into AI that all of a sudden it's like, wow, I didn't know we could do that, or wow, I didn't know we should do this.

I think a big thing. And maybe it's not AI specific, it's just that this comes up in my research a lot. People don't know what they don't know. So even if you have a conversational chatbot, you Might not know what to ask it. It's really hard to have your search query. It's kind of why you go onto Google. And a lot of times it's that bird's eye view SE where you're just trying to get a sense of the landscape first and then refining that. So I think that a lot of times with a conversational AI, we might say, oh, a chatbot. That's a great end goal. Right. But for a user that requires a lot of work, a lot of cognitive load to be able to have the ability to know what to ask. So I think that working on how can we predict what people are actually going to ask and, and what they might be looking for, and how can we suggest prompts and how can we help kind of guide them through this experience?

So we got this really nice audience of folks out here, but I'm the only one with the mic. What I'm going to do. I'm going off script here.

Oh, boy.

Anybody have a question? And if you tell it to me, I'll repeat it in the mic so we can all hear it. So does anybody have a question for Melissa? I mean, I've got several, but we've got such a great audience here. They all look like they're really interested. So anybody have a question? Going once, going twice. All right, what's your name? Irene. Irene. Irene has a question. Ari, what's your question? Okay, so she was to want you to kind of help her build out her research function. What did you do to build your research function out?

Okay, awesome. I can sort of address that from, like, the democratization standpoint and a little bit of the buy in. So a lot of times I look at it as, like, just taking it in, like, very small steps and, like, proving value. So when I originally brought user research to this, like, training function, there was a lot of pushback. It's going to take a lot of time. Just like a lot of times product managers say, like, it's going to take a lot of time. Can we do that? So what I did was I basically was like, well, let me just prove this out. Let me just do it and maybe like, do it and ask forgiveness rather than permission. And then with democratization, we're actually doing that at Consumer Reports right now, where we're helping support our design teams to be able to run their own evaluative research. And it's really getting specific about the types of research that it makes sense for design to do versus the types of research that it might make more sense to have an expert in the field doing. Sure. So I think for designers, it's evaluative. Right. Like iterative development, being able to put out a design in front of users and then go back and redesign and redesign. Right. It's like a faster form of development there versus the more generative work that a lot of times, and the discovery work that we do in research. And then for PMs, I always like to say it's more of that. What is it like the summative work. Right. The evaluation at the end of, like, being able to pull those benchmarks. They're already looking at, like, a lot of that data on the site, that behavioral data. So being able to, like, run analysis on that. Yeah.

Talk a little bit about how the. The actual relationships of people are changing as you start to integrate a lot of this modern technology, AI, machine learning, LLMs. Like, how has the actual physical relationships between the people, researchers and consumers, and everybody else that's kind of working together. How have those changed?

Honestly, I would say right now not a lot has changed.

Okay.

I think we're all just still people working in companies, doing our jobs, and looking at our KPIs. And now we just have some tools that make things maybe a little easier or better or faster or worse sometimes.

How do you measure? You have your traditional KPIs. Are there any other new ways to measure for success or figure out if we're on the right track or we're not on the right track? Or have some KPIs been even more emphasized while others de emphasize?

I think after this conference, I think I have a lot to go back to Consumer Reports with around, like, how we can start bringing the customer experience more into how we're measuring our success.

Okay.

So I think that might be more on the roadmap. We have a really, like, a transforming organization right now. Last year, we got a new Chief Experience Officer, and we're really focused on improving the customer experience. So I think that as we evolve, we're probably going to be able to talk a little bit more about that. But right now we're still in the sort of transformation.

Like, what. What have you heard at this event so far that has got you thinking, hey, maybe there are some things that we can discuss and talk about.

So for me, I really loved the CMOS talk yesterday morning. And specifically, there was, like, an illustration of, like, this really beautiful journey map. And I was like, that. That looks familiar. Like we' thinking about the journey. In all of my research, I'm always asking, where have you Been and where are you going? And I always encourage the designers I work with to also think about the full experience. And then he showed the journey map, and I was like, okay, this looks familiar. And then the second journey map he showed, which was just like a chaos map of lines everywhere, and with the little moments that matter popping up, like, bolded. And I felt like that was so powerful that I intend on bringing that back to Consumer Reports.

Did that kind of look familiar, at least in your mind, or.

Very much so, in a way that I hadn't realized that I could illustrate that. I always felt like I would have to present a journey map that was like this. Like a journey of a person doing a thing. But for me, I'm like, that's never real. When you're actually talking to consumers and analyzing that data, you can see that some things are just random.

You know what?

It depends exactly.

Talk a little bit about, like, your group, your structure, how many people you actually work with in your team. Let us know a little bit about that.

Okay, sure. So I, on my team, I actually report into consumer insights. So my direct manager and one of my co workers are really working on the market research side of things. They're doing the brand tracker and MPS and our csat, and then it's myself and one other user researcher. And we work with. Not only are sort of. We call them our core product squads who are working on our digital experience, but we also have an innovation lab that we work with. We also work with our editorial team. We also work with our advocacy team. We just have been supporting a new ideas incubator within the company. So we kind of are able to get our hands in a lot of different pots, even though the company's not huge. We're doing so many amazing things as a nonprofit that we're able to sort of. We're not just focused on product. We actually have, like, a broader reach.

I love the fact that there's still a magazine, like a print magazine. How do you get the insights, you know, in all these different formats, but also remain true to kind of the traditional aspects of Consumer Reports?

I think our magazine is a pretty wonderful case of print that's still doing quite well. And my co worker, who works on the sort of market research side of things, is the one that keeps track of our satisfaction with our magazine. Yeah.

Nice. All right, let me go back to the script. Sure. What profession other than your own, would you like to try?

I had seen this question, and I really wanted to give, like, a very motivated answer, but I Would, like, love to be retired. I.

That's a great. I see you took a lot of thought into that one, too.

You know, I had thought about saying something that sounded like more driven, and I was just like, if I was retired, I could go back to school. I could, like, take courses and things that interest me. I could garden more and more, get new hobbies. Right.

So you really thought this one out? Apparently. Very good. All right, any other questions? All right, I'm gonna. I'm gonna go back then. All right, so talk a little bit more about, you know, the plans going forward with AI Because, I mean, every conference you go to, you can't escape it. I've noticed. I go to a lot of events in the year. At the beginning of the year, the AI discussion is one thing. End of the year is completely different. Talk a little bit about maybe what you see, the future of how you guys leverage AI and what you do and how that might kind of impact the overall company itself.

Yeah. I think that the way that we're leveraging AI is to help make our consumers smarter, be the superheroes of their own lives. Right. Being able to get this information that they're looking for and to be able to make better decisions. I think something about AI, and maybe this is on a personal level that I look at, is that, like, where. What is the use case? Right. Like, there's a lot of AI solutions. And I was talking about, like, chatbots. Like, how do you even know what to ask a chatbot? And I think it's really important to have a critical eye on just because it has AI on it. Like, is it doing something that's important? And knowing that the tokens that it takes to use generative AI and the water that it's using, I think that those are all things to think about, too. Is it AI that we need to use for this solution, or are there other ways? Could we use machine learning? Are there other ways that we could be getting this information or be delivering results? So even at Consumer Reports, we're thinking about that, about ways to reduce the energy we're using, ways that we can make our calls to the LLM when it makes sense and make our culture, our data when it makes sense.

Interesting. You hear a lot of people, a lot of apprehension around, is AI going to take my job, or is it going to be the one in charge? You're hearing now about, they call it agentic AI with lots of autonomy, like, the bot can actually think and do things for itself. What's the apprehension level in Research with AI at this point?

Oh, I'm not sure. I mean, I know that AI doesn't reason. Right. Like, there has been research that has just come out that said AI doesn't reason. It can pull sentiment, but it doesn't understand, like, emotion.

It's not sentient yet.

It is not sentient. It cannot pull insights for you. It can pull patterns. And it's wonderful at pulling patterns. Right. And it's wonderful looking across data sources and pulling patterns, but it's. It can't rationalize for you. So I think that, like, while it can, like, augment your work and it can make you smarter and make you do better, like, it's. We're not quite there yet. I can't remember what it's called where it's able to come in and actually has, like, a mind of its own. But we're not quite there yet, so. And it still hallucinates. Even when you're using your own data, it still hallucinates. So you actually. We have something at CR that we call human in the loop.

Yes.

Where we have folks that are still looking at what the AI is creating and making sure that it is not just spouting out nonsense to people or that we are, like, maintaining our brand integrity and the trust that we've built over almost 90 years.

Yeah, that's a really comforting thought. That they're actually still humans there.

Absolutely.

For those hallucinations, because. Yeah, they ain't going away. What question should I ask a researcher in 2024 for going into 2025?

Oh, that's a. That's a really good question.

Don't give me the. It depends either. I don't want to.

I feel like researchers are so used to asking the questions. It can be really difficult to be. Like, what questions you would ask of me? I would say what's most interesting about research is that we get to learn about people. So, like, what is most surprising about what you're learning about the people that you're working with and for, like, what is. Like, like what's surprising or shocking or weird? What are you, like, what's the last decision you made based off of something that you learned? Yeah.

What have you learned about people in the last couple of years that because of all that's going on, that you weren't able to learn about them before?

Oh, that's. That's interesting. I'm actually, I'm not sure that what I've learned is based off of, like, what I couldn't learn before. Right. I mean, people are using technology in different ways, but we're still human and we still have our goals and our needs and they're solution agnostic. Right. So we're not like tied to like whatever delivery system we have right now. I remember chatbots were popular in 2017 and it's 2024 and they're popular again. So it's all very like, people are still human. People's goals are very much tied to like what their needs are and who they are as humans and not necessarily who they are in relation to a product or a service solution.

Cool. One last question and then I'll ask you my, my sports question. Oh, no, I might have already asked you that. I don't know. So when you go home from this conference and you get a chance to think about and do stuff, what are you going to take away from this conference and try to do when you get back to where you were?

Yeah, so I think I spoke to this a little bit, but I, I have been sort of tasked with my co worker to start looking into the journeys consumers on Consumer Reports as well as like the new audiences that we're hoping to attract. And I think a big thing for me, the takeaway is that my, the way that we present that information or the way that we sort of start aggregating that information doesn't need to be perfect. That's always been a blocker for me. You know, like, I've been like, oh, but how can it possibly fit into this map? Right. And now feel, I feel like a little bit more freed to be able to just kind of pull out, like visually represent what is important and where we can impact things. Very cool.

All right.

Very tactical.

All right, Well, I like you. All right, Melissa, thank you so much for doing this. This will be great.

Thank you.

[Brett Leary]

Want to keep the conversation going? You can find the show [e-mail] podcast if you haven't already, don't forget to follow us on Apple Podcasts, Spotify, Overcast or Google Play, so you never miss an episode. And if you enjoyed today's show, please share it with a friend or leave us a rating and review on Apple Podcasts. And until next time, this is Insights Unlocked, an original podcast from User Testing.