← Insights Unlocked
Insights Unlocked
All the questions you had about synthetic users but were afraid to ask
00:42:51
Discover how AI is transforming UX research. Dr. John Whalen shares insights on synthetic users, AI tools, and the future of human-centered design.
Loading summary
Transcript43 lines
- [00:02]
Nathan Isaacs
Welcome back to the Insights Unlocked podcast. In this episode, we're diving into the future of customer research with Dr. John Whalen, cognitive scientist, author and founder of Brilliant Experience. He shares how AI is transforming the way we gather insights from synthetic or simulated users to AI moderated interviews, and why the human element still matters. More than ever, it's a smart, grounded look at what's next. Enjoy the show.
- [00:32]
Podcast Host / UserTesting Announcer
Welcome to Insights Unlocked, an original podcast from User Testing, where we bring you candid conversations and stories with the thinkers, doers and builders behind some of the most successful digital products and experiences in the world, from concept to execution.
- [00:51]
Nathan Isaacs
Welcome to the Insights Unlocked podcast. I'm Nathan Isaacs, principal content marketing manager at UserTesting, and joining us today's host is UserTesting's Leah Hogan, principal for Experience Research Strategy. Welcome to the show, Leah.
- [01:06]
Leah Hogan
Thank you, Nathan.
- [01:09]
Nathan Isaacs
And our guest today is John Whalen. He leads Insights and Innovation at Brilliant Experience, where he and his team have reimagined their research approach by deeply integrating AI into their processes. He teaches the Maven Top 100 course AI for Customer Research and hosts the podcast AI for UX, where he interviews founders of AI powered research tools. John Whalen is a cognitive scientist and author of Design for How People Think. Welcome to the show, John.
- [01:41]
Dr. John Whalen
Wow. Thank you so much. I appreciate it.
- [01:43]
Leah Hogan
I am thrilled to be joining you for this conversation today, John, because I think given your long history in customer research, user experience research, and now weaving in artificial intelligence, you have a lot of wisdom to share and I think it'll be helpful for our listeners to hear more about that perspective. But as we get started, could you just spend a little bit of time talking about the beginnings of your career in this space and what attracted you to the discipline and what about about it has kept you so passionate about it?
- [02:26]
Dr. John Whalen
Yeah. So I. I won't belabor it too much because it was in the land when dinosaurs still roamed the earth, but I did a postdoc at UCLA when some of the dot com boom was going. I saw things like Netscape Navigator 3.1 and saw how clunky it was, and I was like, gosh, why can't we put more psychology into this to make it more human friendly? And I sort of had a love for technology and I found that I could do as a research professor, you've got to be an expert in one thing and study it, study it, study it. And I found that in consulting, I could have a new thing I do and make a contribution, like every two months or every three months. And that was awesome. And so being able to merge my fun of learning new things and technology and trying to relate that back to our humane existence, really, I find it, like, endlessly curious to see how people are thinking and how can we align what we're doing with how people think. That's why I literally wrote a book called Design for How People Think.
- [03:28]
Leah Hogan
I love it. I love it. I think it's really that history really informs where it is that we are today. And I think the work that you're doing specifically around artificial intelligence and across the space in that new area is really fascinating because obviously there's a lot of hot takes on it, as we said earlier as we were preparing for this call, across the spectrum. And I think I'd like to start with the question around where, if at all, and I'm assuming, yes, you think at this time, AI could help to support the quality or accelerate what it is that we do as researchers? And is there anything that should be off limits?
- [04:22]
Dr. John Whalen
Yeah, well, first of all, it's true that I've sort of deeply integrated AI into our processes at brilliant experience. So. But we did that carefully. So just to take a quick step back, I saw a podcast about two Halloween. So now, almost two years ago, exactly. With a group that said, yes, we're doing synthetic ideators that are talking to synthetic users and cycling through that. And then we're interviewing 300 people over weekend, over six languages. And I was just like, what is this stuff? Is it any good? Like, should we trust it? Like, I must be behind the times if it's any good. So. So what I did is actually, we did two things, actually. So I. I'm one of the, you know, experimental psychologists here. So we put our money where our mouth is. And first I started interviewing people who are founders of some of these tools. We met them, we tried the tools. We're just literally, you know, experimenting and learning, learning, prompting, things like that. We also did what we called our AI versus human researcher sort of head to head contest. And really what we did is had. So I did the good old 2022 version of a human interviewing humans and do all the analysis and report it. We had a Second person, another PhD psychologist, do the same sort of thing, human to human interviews and analyze it. And the third that really matters here and is have an AI moderator and otherwise one of those people did the analysis and so on. Okay, so the moral of the story is we white labeled those, sent them to fellow senior researchers. And the key point is that really we were all set to hate AI moderation. And we were sort of dumbfounded that it hit about 80 or 85% of what we found as like very seasoned researchers and think we're okay at this. And so we were like, wow, we at least have to take this seriously. And so we started to look at all the different tools and, and really did a head to head comparison of timing and so on. Um, so let, let me say it this way that for sure, I think lots of people would love to agree with me that right now any of these tools are not as good as humans who have the time and can really dig in. But I also think that there are opportunities for, for example, take AI moderation for I just can't do 100 interviews in three days. And with this kind of tool I can. And so remember if I go back to my sort of 80, 85% idea, well, I'm getting something rather than nothing. If a stakeholder has a major decision in a few days, maybe having some insights rather than just the stakeholder's best guess is better than zero. And in other cases my Finnish and Thai and Brazilian Portuguese aren't what they should be. And this allows me to have more inclusion and bring in more things where that would have been time and cost prohibitive otherwise. So I think there are opportunities to expand what we're doing and provide folks more than we did before. I'm not ready to replace humans, but I think in some cases there are ways to really empower us to do more than we would have otherwise.
- [07:46]
Leah Hogan
Yeah, I think, you know what I, what I take away from that, it's really augmentation, acceleration, additional speed doesn't necessarily have to be precision, but the other side of that is when you've got something that's high risk, where you've got the resources and time.
- [08:04]
Dr. John Whalen
Absolutely.
- [08:07]
Leah Hogan
There should be more investment there.
- [08:09]
Dr. John Whalen
Right, so. Exactly right. So when it's the big $300 million decision, I don't want you to pull out your favorite AI moderator and just get its default results and do something. Absolutely not. I might use some of those tools for making sure I'm doing like a global study in that case to augment what I'm doing. Again. We also, when we started to use these things, we did get folks who were native to the geography we were talking about to look at transcriptions, to look at the like play the videos, see what sort of nuance they're missing. So I'm Canadian, so I might start talking about Chesterfields and Toques and you're in people like, what are you talking about? And so, you know, there could be these sort of. Or idioms like pot calling the kettle black and wanted to see how it did. And we just keep being. Just remember even now these things. I don't mean to be a fanboy, but logically you can say to yourself, and these are the worst they'll ever be. And they seem somewhat useful now. And so think about ChatGPT2, maybe two years ago and what ChatGPT is now. So think about what these tools could be two years from now. So we're looking at these seriously, not only to use them in our practice today, but to be prepared for tomorrow.
- [09:27]
Leah Hogan
Yeah, that's, I think, a really good point to shift gears a little bit and talk about a specific use case that I think a lot of people have strong feelings about, which is synthetic users.
- [09:41]
Dr. John Whalen
Sure.
- [09:41]
Leah Hogan
And I know that you've done a lot of work in this space too, and would love to know more about what made you decide that synthetic users were worth taking seriously.
- [09:54]
Dr. John Whalen
Yeah. So the first thing to say is, I had the exact same reaction many researchers have when I first heard about synthetic users. You know, guffaw like, that's ridiculous. And then later I was like, okay, we're researchers. We should actually take a more measured approach and test these things, contrast it with real human data, understand what its strengths and weaknesses are, just to make an informed decision. And that doesn't mean ever use them at all, but simply to make an informed decision, like we should. We should be guiding our stakeholders with the knowledge and power of all the resources we have and know when to use them or when to never use them. So I think a simple thing we've done. So I'm teaching a course called AI for Customer Research. In every cohort since February, we've actually been reaching out to real humans, doing interviews with them, then analyzing that data. And so we've got that analysis. We know what the major findings were in this case. We tend to do something very easy to recruit friends and family. So our example was how are you using AI in everyday life? And kind of, what. What are your pain points? What could be an ideal version of that? Okay, so then what we do is we take three different methods. So either just prompting and creating a synthetic user that's representative of that population, or using their syntheticusers.com. there's also we've been using a tool by a group called Vervey, like video survey, vervey.com and so for each of those other two, we gave it the same questions that we gave our real humans. And specifically, let's take the example of what would compose your ideal AI assistant? What would its features or capabilities be? The moral of the story is in cohort after cohort after cohort, I think we've got nine now. It's where there's maybe seven major points. It's those things are getting six or seven right, and not having three extra ones no one heard of. So. So I guess my point is that it's close enough that I think it's worth taking seriously and for a couple reasons. One is that might not be our major decision making process, but maybe these things not for that 300 million dollar thing we're talking about. But you know, there's a designer that wants to make a quick decision and just needs a little insight into his or her target audience. Or a product owner is like, oh, I wonder if they'd like this kind of model of like interacting with us, you know, something maybe smaller. And these things would just be that person's best guess. And maybe this is a way to inform that. So the way I think of synthetic user data is not that it's actually data or facts. I think of it as inspiration or a way to broaden my thinking and a way to prepare for being with the real users, real humans, I should say. I'll just be really clear.
- [13:03]
Leah Hogan
Yeah, yeah. And you know, that's a really kind of profound thing to think about because I think it's one of those moments where you really come to grips with the idea that human time is very valuable, like that direct contact and how do we enrich that preparation so that when we get to that moment, we're not wasting it. You know, like it's, it's really a great moment.
- [13:34]
Dr. John Whalen
So, so the synthetic user group actually has a very measured. So just know that they, they sometimes have bold statements, but they also are very measured actually. So they're saying, hey, we can do this initial study with synthetic users to really prepare or to be. So think of actually a normal research study back in 2022. I would do two or three interviews and then we'd like, you know, talk to our stakeholders, say, what should we change about our interview guide? What have we learned about the basics of these people? And we kind of feel like we're getting there already using synthetic users and just know that they don't have to just answer your questions. Right. So this is a new tool. It's just a tool. Right. We kind of gave it this humane property, but if we just act like it's a hammer or a printer and say it's just another tool, maybe that, you know, there are lots of things this could do, right? It could help us. So when we think of a set of questions we might have for an interview, we run through those with a synthetic user. And that can do a couple things. One is it could give us an answer we never expected. And maybe that's because our question actually didn't really hit the mark. And maybe it's because there's some answers that we aren't really expecting and we can be more prepared for them if we get them from real humans. And I think another thing we can do is we can ask what I like to call sort of impossible questions, questions I could never ask of a real human. So things like, which of my questions offended you the most? Or which one put you off guard? What's the question? I never asked you that I should have. What would I want to know? But you would never tell me in an interview, Right? So I'm not sure these are exactly what the real humans would say, but it again, fills my mental model with possibilities to be ready for. So I think it's actually. So I guess I'm saying this is a more utility player of a tool than just asking it the normal questions. The other thing is you can ask it, you know, what do you think of this concept? Or how would you normally do this? And, by the way, respond in jobs to be done format. And my normal humans that I interview aren't very good at that. And so it's just a way to think of things very differently. Lastly, and probably most frightening to our viewers is here's an irony. These things are synthetic users. And yet many of our clients have asked for a report of what research we've done, but also a synthetic user or set of them that embody the research we found, so that it sort of stays and sort of comes to life. And so what are they going to say? I'm not sure exactly, but I know the framework here. And so we prepare stakeholders for that. But interestingly, it excites the stakeholders about doing research, says, oh, I found this really interesting thing with this synthetic user. And because we've told them, of course, don't trust that for your $300 million question, they're coming back to us to do so. I guess I'm getting them excited about getting primary. So ironically, I have to use a synthetic user to do it. But. But it's a tool to, you know, draw everyone into the humane nature that we really want in the end.
- [16:57]
Leah Hogan
Yeah, it's kind of, it's, it's almost ironic sometimes that you have to think about like, oh yeah, thinking, getting people interested by not using necessarily a human to say like this really valuable moment, however, is something that can brought to you by the same people who created an artificial moment.
- [17:20]
Dr. John Whalen
Right? Yeah.
- [17:22]
Leah Hogan
This is interesting. At user testing, we're very focused on getting insights to drive better business outcomes and also outcomes for customers. That's the core of what it is that we do as UX and CX researchers. You know, you talked a little bit about some specific examples around how you might use synthetic users in certain contexts. But when you're looking out and across the entire process, as we think about it, what are some specific places where you see specifically synthetic users contributing to again, that acceleration, augmentation quality?
- [18:10]
Dr. John Whalen
Yeah, you know, let me give you an example. And this is another way to think of this as a tool and not, you know, something that's going to get you out of a job. So when we start with stakeholders and have sort of a stakeholder workshop to get us started on what are our research priorities and try and coalesce on those many times some stakeholders say, hey, you know, we really want to answer this question with this set of people. And, and so literally in the workshop, what we can do is we can spin up a synthetic user that's somewhat representative, that fast and dirty, and then ask a question that corresponds to maybe that's a strategic question. We change it a little bit to be a reasonable question to someone and get a response. And generally what often happens in that sort of circumstance is that they say, oh, oh, that. No, no, that's not exactly what I was after. I was after this instead. And you know, maybe that's not the right kind of person for these reasons. And so just then we've already captured a little more nuance into what really is the question that that stakeholder has and know a little bit more about how better to target our audience. So it's simply a tool. In that case, it almost doesn't matter what came out of the synthetic user. So I guess I'm saying that from earliest days to final reporting, there can be an interesting use for these tools. So we also test our. So we mentioned just kickoff, essentially preparing interview questions. When we're doing recruiting, we know the target audience we want, so we get our synthetic users to answer the questions to the screener to see would we have accidentally, you know, knocked out our target audience because they would think of that question in a different way. Than I did. And did I just make a mistake in the way I was trying to recruit? So there are ways that there are sort of checks and balances of just another way of thinking, a fresh set of eyes kind of thing. And note that we've been talking about synthetic users and it could be a synthetic expert, right? So we can have, for example, my specialty is psychology. I really love understanding how people think, but I wouldn't call myself a brand strategist out of the box. So having, for example, a set of interviews and getting the synthetic expert to look at those interviews from a different perspective, you know, I might be able to do things like accessibility or usability or a strategic question, but there might be these other, say, branding questions or marketing nuance that I might not see that maybe another vantage point might. And so especially when you're like a user research team of one, it's actually kind of nice to have some sort of partner, even if you're like, no, I don't think that's right. That helps you to solidify what you're thinking yourself. So it's not replacing my brain, but it's, it's giving me a new avenue to challenge or to think about for myself.
- [21:08]
Leah Hogan
That's interesting to think about because, you know, I've been seeing other folks who are experimenting with artificial intelligence as UX researchers thinking about what are specific agents that they could build to support their process at different stages specifically. Right. So you know, someone who's all about helping to scope and figure out and get over that hump of the white page syndrome, right? Versus the how do you help with that data collection piece and that the quality and acceleration of that process and then the synthesis process separately. And so I think your framing of synthetic user versus an expert versus, you know, whoever it is. And actually, I think you're just talking about those two things, but like how those have different roles at different stages, actually.
- [22:05]
Dr. John Whalen
Let me give you one more. So when we're doing a big presentation for a big senior executive will often spin up a synthetic user, you know, even we know, with our LinkedIn profile and stuff, right? So we could build that synthetic user, say we're going to present in this way, what are the first questions this person's going to ask? Or from their perspective, what have we failed to adequately account for? Or, you know, what, what nuance are we missing from their perspective? And there's been a couple times that the, the anticipated questions are, you know, one of those comes out of their mouth. And so again, even if it's just helping us to coach us, almost. That's not changing the results. That's still us. But I think it's a way to. I don't know, it's a new angle we can take to make sure we're thinking from every perspective.
- [22:58]
Leah Hogan
Wow, that's really fascinating. And I think kind of plays well to the next question I have, which is really around, you know, there's an implied sense of trust and confidence that you have in what you're getting from these agents across the process. And a lot of critics, and I'll say this, like, sometimes, myself included, would say, you know, are we giving people or giving these synthetic agents too much credence here? Like, what are some of the things that we can do to help check that level of confidence? Right. Or perhaps like, maybe safeguards, guardrails that can help us to avoid that trap of overly trusting some of these agents.
- [23:53]
Dr. John Whalen
Yeah. So let's take AI that provide, like, specialized tools or even a prompting that does analysis. Okay. So of, say, interviews. Let's just give a concrete example. We really, like. There are a couple tools. One of them is called Coloop, and the other one is called Quali. They're both like ko Loop AI, quali, AI. And the reason why is basically what it does, is it tries to frame. Here are your major questions, and here's each individual. Here's what we believe is sort of a terse summary of what we got for that question for that person. And then it sort of summarizes for everyone, here's what we think we're hearing. And so let me tell you why that's really good and why and what the weaknesses are inherent in that. So we're thinking from both perspectives. And by the way, I really like your perspective. I tell all my students, be curious, like, be open to at least testing these things, but skeptical. So it's totally. That's the way we're researchers. We should do that. So in this case, things like. And there are other tools like this, but these ones sort of embody this idea. Every one of those points it makes is a hyperlink. And you can click on that and go straight to exactly where in the transcripts. You know, there might be multiple places it believes give evidence to that bullet point or whatever it might be. And so this is a trust but verify moment. Right. So maybe this is a good way to get a quick sense of what happened. But, you know, obviously there's more nuance to it, and it sometimes merges three things together that aren't really a conceptual Point that makes sense to me. The other, there are a couple weaknesses inherent with these things too, just to be clear. So one of them is it just does not have yet. And some people are really working on this, the strategic background that you do at your company. So if you've got these specific strategic goals, you know that you tried something, you know, it didn't really work. You know, for the first thing that people ask for very often, there's another thing that people always ask for is a need. And that's, it's. That's not a blue ocean, it's a red ocean. There's so many people on that plate. We don't want to play in that space. We want to know what comes after that. Well, this tool typically will tell you, hey, here's the first thing we found. Here's the second thing we found. And to your stakeholders, like, get those out of the way. We need to know the new stuff. And so I guess my point is that it doesn't have a strategic mindset. Sometimes sort of the conceptual layers there aren't quite right. In most cases it is. I think it's quite good. But we really like that trust but verify. So we're like clicking on every one of those links. So we don't want to take anything for granted. The other thing I'll tell you that it's missing is you're averaging together, say, 24 people. And. But if we looked into it, maybe there's, let's take the AI for personal use example. Maybe there are some people who are very nervous about privacy and that's really important to them. There might be another audience that really feel there are environmental concerns about using AI and maybe want to limit their usage. Maybe there's someone else who just loves using it and can't wait to dive in. So those three sets of people, even if there's, you know, eight and eight and eight, we've then successfully averaged all those different thinking patterns into one summary. Right. So. So another thing we have to do is really understand what segments are there there. We have to do the homework as humans, then pull them out, then reanalyze with those segments. Right. So. So there's just a couple things. One is, is bringing things together can often be good, but isn't always. The other is it doesn't have that strategic mindset. And the third is it's sort of bringing people together that maybe it's not appropriate to average because of the way they're thinking.
- [27:42]
Leah Hogan
Yeah. And, you know, that's a really, I think Important point, that disaggregation piece, especially because it leads me back to the point that fundamentally you have to be able to structure really good studies, you know, from the ground up, so that you're creating a set of data that's really robust and that is truly representative because that quality has to be there for there to be an LLM to actually have something to work with. Right, right.
- [28:17]
Dr. John Whalen
Think of that representative idea for a moment. I had a stakeholder once, I guess an engineer who will remain nameless, who was like, oh, John, it's so sweet that you interviewed 24 people and found that we have 700 million active users in this. I don't care what 24 people said in good old human land. In 2022, we couldn't really do that much more with budget and time and everything for that group. We were able to do several hundred interviews globally in different languages, come back to that person and say, here's what we've got now. There was more willingness to back the data because of that perspective. I do want to say that when we think about representativeness, we want to be great at asking questions. Absolutely. But there is this notion of what are we missing because we're doing a scale maybe for these major global corporations, that we might not be doing it entirely justice without using some of the scale that we have. So the things I guess I'm thinking of, yeah, we can go much faster. I'm on the east coast of the U.S. it's not that much fun to interview people in New Zealand from here. And so having people be able to do it whenever they, they like is another benefit. Multilingual kind of things and just scaling up in ways we haven't before. So there are new ways to approach research. Right. The other is, logically, it doesn't have to be the study. I remember working for a government client and they're like, yes, we get one study this quarter, we better do every question we can think of. Well, with this kind of tool, it allows us to do, say, 15 minute studies, you know, with maybe three in a week, and, and we're doing sort of incremental learning as we go along. So I'm just saying you've got new, new quiver arrows in your quiver and new tools in your toolkit. We can take the analogy you want that we can use as researchers. So it sort of empowers us to think a little differently and go back to first principles. What do people really need as opposed to what are we used to doing?
- [30:27]
Leah Hogan
Wow, that's, that's A very thoughtful response. And I think, you know, getting people to, to really get back to, you know, there are some basic things that we may need to evolve a bit. You know, it's revisit, evolve with this new tool and new possibilities. That's, that's fascinating. So I guess, you know, to me that means you absolutely have to continue to do human to human interviews. Right. Because they help to expand your, I guess your thinking in a variety of different ways. But can you just talk a little bit more about why we need to still continue to do that human to human interview? Like what's the additional benefit that we get from that? I'm assuming I'm all in. Right. I'm always all in on human to human.
- [31:34]
Dr. John Whalen
Right. So yeah, sure. So, and, and so you think of, of okay, what we. I'm going to go back way, all the way back to 2022. Again, we think of mixed methods as, you know, maybe I'll do some qualitative interviews and then I'm going to scale up a survey and then we're going to find some interesting things in the survey we want to dig into. And then we want to do qualitative again. Well now I'm going to give you a twist on that. So we might do a handful of, I know, I'll just say human to human interviews just to be sparkly clear and then be able to use say AI moderation to scale up our work. And remember that we can now this notion of qual versus quant, like if I'm doing 200 interviews where we're getting qualitative data in addition to quantitative data, I mean there's this horrible neologism quality. But you know what I mean, we're sort of, maybe those distinctions aren't as black and white as they used to be. But then after doing that scaled up work, probably we're finding something interesting that the AI interviews maybe didn't do justice in following up on. We might do some human to human ones. Again, there's that method that's something we do actually fairly often. Yeah, so I think that was, that was one piece that I did want to mention. Sorry, there's one other thing around my brain, but I'll get there.
- [32:58]
Leah Hogan
I'm sure there's an idea that's really fascinating because usually when we think mixed methods research, it includes only human to human interactions. And then by expanding the scale that we can get to by using it. Looks like you just had that moment.
- [33:16]
Dr. John Whalen
Yeah, sorry, if you don't mind, I'll leap in before it goes away again. So let me give you a radical. Okay? So we're going to think a few years from now. So I guess it's. My theory is it's 2025 right now, so maybe like 2028, maybe this is the way the world will be. So remember, synthetic users, I do want people to take them seriously for one other reason. They are super inexpensive, right? So everyone has access to them. Even today you can do it through prompting. It's really low risk. If we tell someone, hey, we're thinking about doing this product to a synthetic user and it's like, that's the stupidest idea ever heard. It's okay, right? So it's not, not, we're not hurting anyone. We're just testing something, ideas. So it's fast, it's risk free. We can do these interesting things that are sort of out of the box, like, you know, what's the better idea? Or what does this really make you think of? You know, all these impossible questions. Okay, so my argument would be that the average engineer, product person, marketer, designer will have at their disposal these synthetic users to do a quick one off test on. Right? So the highest number of hits on a research tool in 2028, I'm going to anticipate, you heard it here first. Will be synthetic users. The second thing is because we've got these really interesting data repository tools that have AI powers. We can do so much better at asking questions of our past data to see what's on the cutting room floor where maybe we might not have to do as much one off research. Okay? So logically, so first they're trying things first with synthetic users because fast, cheap, you know, inspirational. We'll just leave it at that. We would next turn to, you know, asking questions of our. Because we'd ask questions now not just like search through videos of our 5001000 interviews that we've got in the, in the can in all of our surveys. And then the third thing we'll do is new one off research. So we might argue that there will be fewer one off research things. But there might be, to your point, regular research that's like what's new in. So for example, let me give you what the synthetic user groups do now. So Verve is one example that comes to mind. So they do about, I think they say on their website, so about 3 million interviews a year around the world with all sorts of different people, B2B and B2C and so on and old and young and. But my point is that you Might wind up having sort of baseline collection that you're doing constantly. So it'd be a little bit more of a continuous thing than doing our sort of. We need to do a new study on this because we don't know the answer. So I, I just think there will be this synthetic user try, you know, corpus of what we already know, try and then go to that. So logically, there may just be less one off and a little more continuous knowledge.
- [36:22]
Leah Hogan
That's really fascinating and I see it's super fascinating to me because it implies that there's. Just because the world is evolving, there's always going to be a need for basic research. And instead of saying like, hey, we need to dig into this one specific thing, the fishing expedition is now always in scope because you're designing for novelty in that, in that research strategy.
- [36:51]
Dr. John Whalen
Yeah, for sure. The innovation groups, the, you know, brand new, you know, you know, new field kind of folks are going to be doing research. So you don't have that. But if you're always got your SaaS product users, you know, then, then you might really be regularly polling them. The other thing to let you know is there are new tools. So, and I'm sorry, I might give you a bit of toolitis because I've interviewed so many, about 40 people now for the. The what My podcast. There's another one called Next, which is representative of a class of these sort of ones that are really focused on the whole customer experience. And it's designed so this is when we're going to talk about agentic AI. So it brings in, for example, everything from your app store reviews to like customer service calls, to social listening to the interviews you do to survey data and the idea. So imagine a group like Bosch. So they've got like dishwashers and microwaves, and I don't know what else they have. And so logically, the group that is doing marketing wants a certain kind of information. The phone group want a certain kind of information. Maybe the toaster people want to hear from the customer service calls and the social listening and the things you do. And so they essentially, the tool allows you to create sort of little researchers. Now I picture these little researchers running everywhere in the data. But anyway, the key point is that they're, they're logging things and categorizing them constantly to like, hey, this is a, you know, pain point. This is a new request. This is something else. And this has to do with toasters, this has to do with microwaves, and this has to do with branding. And so you can see that you can start to create a dashboard from all this stuff so that us researchers won't be limited to. Here's the set of interviews we've done this year and last year. And instead you've got a much broader corpus of stuff to work with and logically the product team would have access to that too. So just. It's a new. My heart sort of beats a little faster thinking about this too. It's a new world and a little scary. Just lastly, I'll say that I show an emotion wheel as the first thing I do in my AI for customer retail research class and say, how are you feeling about this? And almost everyone is saying I'm like excited and terrified. You know, like there, there's this like mixed emotion feeling, you know, overwhelmed and uncertain, but maybe a little bit excited, you know, so, so I. It's a, it's fair and I've gone through all the stages of grief. Should I still have a consulting company? Everything else. And it's a real transition for us, not just in the way we work, but just how we represent who we are as people, as researchers.
- [39:43]
Leah Hogan
That I think is a great note to end the conversation on because it's hopeful really. You still have a consulting company and I think that you've highlighted some ideas for people to continue to bring meaning and connection to other people through research work. So that's wonderful.
- [40:05]
Dr. John Whalen
Well, and just a last note, I really encourage folks to keep learning prompt engineering. It really is valuable. They call it context engineering too. But. But that whole side of the world is really valuable for us researchers. Learn and try these AI moderated interview tools, learn these AI powered analysis tools and learn synthetic users. So I'm not saying you have to bring them all in. I'm saying you have to have an educated opinion on these things and understand what works and, and what doesn't and then think to the future of suppose these got really good. What would I do then? And so be prepared for the future and evolve the way we need to.
- [40:41]
Leah Hogan
Yeah, that's such a great point. Like all of those things are, I think, relatively low risk strategies that you can use today to kind of prepare for the world that's coming. Awesome. So, you know, before we close out, I'd love for you to share how do people learn more about what it is that you're doing, where it is that you're going, what it is you're thinking?
- [41:06]
Dr. John Whalen
Yeah, I think you can just Google John Whalen and you'll get a bunch of stuff for better or for worse. So I have a podcast called AI for UX that's on Brilliant Experience is my company. So if you just go to YouTube and type in Brilliant Experience, you'll get there. So those, those founders of tools say who is this tool for? They give a demo of the tool so you might not be able to get access to it typically. And, and then say where do they think the future of research is going? Because they are in the know, they have to kind of be where the puck is going to go, not where it is now. So there's that. And I do a class on maven called AI for customer Research. And so there's a couple places to start and my. That should be enough. And yeah, please do reach out LinkedIn. I'd be delighted to keep in touch.
- [41:50]
Leah Hogan
Lovely. Thank you so much for this conversation today. I have to say, even if you are, as a listener skeptical about artificial intelligence, hopefully this gives you a little bit more insight into why you might want to challenge that notion, perception, position, whatever that is. Awesome. So thank you for joining me today and we'll close here.
- [42:19]
Dr. John Whalen
My pleasure. Thank you so much for having me.
- [42:22]
Podcast Host / UserTesting Announcer
Want to keep the conversation going? You can find the show notes@usertesting.com podcast if you haven't already, don't forget to follow us on Apple Podcasts, Spotify, Overcast or Google Play, so you never miss an episode. And if you enjoyed today's show, please share it with a friend or leave us a rating and review on Apple Podcasts. And until next time, this time, this is Insights Unlocked, an original podcast from User Testing.