← The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
#835: Qualtrics' Jordan Harper on using synthetic panels to get real insight
00:25:34
Loading summary
Transcript43 lines
- [00:01]
A
The agile brand.
- [00:07]
B
Welcome to season eight of the Agile Brand podcast. This season we're going all in on Expert Mode, MarTech, AI and Customer Experience, talking with the people and platforms behind the brands you know and love. I'm Greg Kilstrom, your host and I help Fortune 1000 companies make sense of martech, AI and marketing ops. Hit subscribe or Follow to make sure you always get the latest episodes. And leave us a rating so others can find us as well. And make sure you check out our sponsor, Tech Systems, an industry leader in full stack technology services, talent services and real world applications. For more information, go to teksystems.com now let's dive in.
- [00:49]
C
What if the most honest and insightful feedback you could get about your customers didn't come from an actual customer at all?
- [00:56]
B
We're here in seattle at Qualtrics X4
- [00:58]
C
summit, and today we're going to talk about a fundamental shift in how we gather customer insights. We're going to explore the diminishing returns of traditional research and dive into the potential of synthetic panels, AI models trained to represent audiences without the fatigue bias or social desirability that can skew human responses. It's a move from merely confirming what we think we know to discovering what's truly possible. To help me discuss this topic, I'd like to welcome Jordan Harper, Principal AI Thought Leader, EDGE center of Excellence at Qualtrics. Jordan, welcome to the show.
- [01:29]
A
Thanks for having me. Pleasure to be here.
- [01:31]
C
Yeah, looking forward to talking about this with you. Before we dive in though, why don't you give a little background on yourself and your role at Qualtrics?
- [01:36]
A
Sure. So my deep background is actually in science. So I studied physics at university and did a master's in astrophysics, worked for a couple of years in nuclear engineering and then made a pretty big pivot into marketing and technology agencies. So I work for a new media agency as a developer back in the early 2000s, building websites and apps and things like that. Ended up working through to becoming tech Director cto, spending most of my time advising senior clients C suite leaders on technology strategy and making smart decisions about new technologies, emerging technologies. And obviously the big thing now is obviously AI. And yeah, my role at Qualtrics is working inside the center of Excellence for our Edge team, which is basically our AI synthetic research product, and working between engineering and sales and customers to understand, to translate kind of the technology hype into something that means something real to customers and consumers of the product.
- [02:36]
C
Got it. All right. And I guess one more thing. A lot of our Listeners know Qualtrics as a leader in experience management and survey platforms. Maybe I know you touched on a little bit, but maybe set the stage on how Qualtrics sees its role evolving in this new era of AI driven insights, as you mentioned, and how you're helping to lead those capabilities.
- [02:58]
A
Yeah, I think like every business, Qualtrics recognizes that AI is going to transform not just the way we do business, but the way all of our customers do business and the kind of questions they ask and the kind of things that they need and expect from us and what their customers expect from them as well. Like, one of the things that really excited me about joining Qualtrics last year was the forward thinking approach to AI to really leaning into this technology rather than some companies do kind of lean away from it and become a little bit scared and threatened by it. What was really clear to me from speaking to everyone at Qualtrics was this was being fully embraced and lent into, but in the right way. So we're integrating AI technology and tooling into pretty much all of our platforms one way or another to help support customers, make their lives easier, to help make their experiences using our tools and software better, and to help them get better insights and action from the data that they're creating. Insights. The synthetic research work that we're doing specifically it's about creating like looking at the question that I think a lot of people have and a lot of people are talking about, which is the idea of synthetic research using AIs to simulate humans in answering questions or providing opinions on what humans might think about a particular scenario. And there's lots of different approaches that people are taking to that. Most of them, I think to their deficit are essentially wrapping things around general LLM models and creating very sort of stereotype driven Personas that revert to the mean. What was really interesting to me about kind of Qualtrics approach to that is starting at a really fundamental level, creating its own LLM from real survey data that understands humans are a little messy when they answer questions and don't always answer the obvious way when it comes to responding. And that's what we see in our model, that it's like a pretty good reflection of how customers respond to surveys and answer questions. And now we're working on making that kind of practically useful and available to our customers for them to test. Nice.
- [05:02]
C
So yeah, let's dive in here and you're going to talk about a few things here. But you've used a great metaphor describing human feedback as a mirror that reflects what People say, and synthetic panels as a lens that reveals deeper systems. Maybe unpack that for us a little bit and explain what do you mean by that?
- [05:21]
A
Yeah, it came from talking to the team about and actually thinking about the bigger picture of research, like why we do it and what we're doing and how it's evolved. Back when I was in the presentation I gave, back where I talk about when I was a newborn and when my brother was born as well, my mother took time off work to look after us and took some part time work as a market researcher. So I remember her stood on the high street kind of interviewing people, collecting all these piles of paper and sending them off to research companies. And she did it for sort of a number of years. And it always used to baffle me why this much effort was being put in to just collecting what people thought about things. And then obviously in my professional career later down the line, like realizing how valuable research and insight is, but thinking about how it had evolved from 1980 through to a couple of years ago, all we've essentially done is use technology to make it easier to access people. Whether that's, you know, email forms, smartphone apps, telephones, like everything that's kind of come in to make research easier has just kind of made people more accessible. And I think we all know that what that's really led to for people is just fatigue and frustration with the amount of questions you get asked in the interrogation. And so a lot of the evolution of the research discipline has been how can we alleviate that fatigue and frustration and make it easier for people to answer questions and reduce the number of questions and make it simpler and compact the information into matrices. So you're ticking boxes on big grids of things and what you're doing there is essentially reducing the cognitive load on the customer who's answering those surveys. But really the cognitive load is kind of the point of asking people questions. And when I started to think about it, you know, building all of these pictures of customers inside the company from asking more and more questions, it felt to me like building this big array of like mirrors of customers inside organizations. And what's interesting, I think about AI is it's not just a new tech for making it easier to access people. It's actually a new way of leveraging the data and the insight and the intelligence that we've gathered over years and years and years with humans and saying, can we leverage that in a way that can maybe take the load off humans and we only ask them the important questions or the more detailed Questions or the things that only they're going to know and things where we can infer useful insight from previous data. We with a lens that looks into that data and that's what kind of AI and LLMs and Edge audiences product in particular really represents to me. It's like that lens to peer into that big data set and extract the insight and useful things from it.
- [08:05]
C
Yeah, well, and I also want to get back to another thing you touched on in the intro is just this idea that synthetic models can be quote unquote, more honest than humans. And so I know there's, with LLMs, there's what is it? Stochasticity. They never answer the same exact way twice. But we talk a lot about the things about AI, but we don't talk quite as much about how humans may be, you know, there may be bias with humans, there may be, you know, depending on the recency or things like that. But also they just don't answer consistently like an LLM, you know, despite that other, that other component might answer more consistent or too consistent. Right. So like how do you look at synthetic as being that more honest?
- [08:53]
A
Whenever I say honest, I always put some air quotes around it because it's like, how can an LLM or an AI be honest or dishonest? Of course, it's just saying what it's saying. Right. But yeah, it's a useful framing for it, I think, because we know as researchers that humans do not always tell the truth. They think about the way they're going to be perceived by the people reading the survey. I'm sure we've all done it. I know I have. I know I've exaggerated or played down my thoughts on something or you know, when you think I really should tick that one. But I'm going to tick two boxes down from it because I don't really want to think of myself as someone who would tick that box there. And so we all control for that. And you know, researchers have decades worth of tooling and algorithms and processes for coping with that and dealing with it and mitigating it. What we've seen with synthetic a lot of the time is that it doesn't really fall for those same kinds of self reflection type tics. Like it's not interested with being perceived as a good person or not a good person. It's not. One of the other interesting things that's kind of connected to it is like priming, you know, that asking certain questions earlier in a survey can influence the answer to a question later in the survey because you planted some seeds in people's mind. Like, we did an experiment where the ultimate question at the end of the survey was, do you think smartphones are good or bad for the world? You know, more harm than good. More good than harm on a kind of like, at scale one to five. And for some of the cohorts. So we didn't ask any questions in the rest of the survey that might have influenced it. For another cohort, we asked some questions earlier on that touched on, you know, are smartphones really good for keeping in touch with your family, for helping you out with, like, daily chores or providing maps to allow you to navigate the world? And we had another cohort where we said a bunch of questions that were like, do you find that social media is distracting? Or are you worried about the impact it has on children? And that kind of thing? And so what we found with humans is exactly what you would expect, which is humans who'd answered questions about the negative parts of smartphones were more likely to answer negatively to that final question than the unprimed baseline. The ones who'd seen the positive ones skewed positive. What we saw with synthetic was almost no. I mean, there was some variance, but nothing that couldn't be explained by natural. And there certainly wasn't a clear pattern moving one way or the other in the variance that we sort out of that. And that's really interesting that we know it remembers the answers to previous questions as it goes through a survey. That's part of the fundamental design of the system we've built. Because obviously you need to remember if you answered you live in a rural area at the start of the survey, that later in the survey you answer like someone who lived in a rural area. So it has to remember it and use it as it goes through, but it doesn't let it kind of control its emotions.
- [11:34]
C
Yeah.
- [11:34]
A
So if it's been reminded of something earlier, that doesn't over overwhelm the baseline sentiment that it has about smartphones being good or bad.
- [11:43]
C
So, like, the bias is at the training level in that case versus with humans, it's in the moment.
- [11:48]
A
Exactly. And the training data has been like one of the questions that we get a lot is, you know, so how come if it's been trained on human survey data, it doesn't just exhibit all the same biases that humans exhibit? And it's because the data that's been trained on has been processed to mitigate all of that stuff has gone through that process of scrubbing and stripping and validating and Making sure that it reflects what people actually think rather than necessarily what they say.
- [12:15]
D
You know, that moment when marketing wants a landing page, design mocks it up and engineering says, yeah, we'll get to it. Thousands of businesses, from early stage startups to Fortune 500s are choosing to build their websites in Framer, where changes take minutes. Instead of Framer is a website builder that works like your team's favorite design tool. With real time collaboration, a robust CMS with everything you need for great SEO and advanced analytics that include integrated AB testing, your designers and marketers are empowered to build and maximize your dot com. From day one, changes to your Framer site go live to the web in seconds with one click without help from engineering. Framer is also an enterprise solution, giving brands like Perplexity, Miro and Mixpanel the confidence they need to build their websites in Framer. Like learn how you can get more out of your.com from a framer specialist or get started building for free today@framer.com Agile for 30% off a Framer Pro annual plan. That's framer.com Agile for 30 percent off framer.com Agile rules and restrictions may apply.
- [13:21]
C
So how does all of this change the role of traditional insights? And I talked with Ellie Enriquez a little bit earlier too about this. And what is the traditional market research? What do they get to do more of? What do they do less of in this kind of scenario?
- [13:38]
A
Yeah, I think it's really exciting because I think it's easy to see it as a threat, but if you see it as unlocking a skill that all market researchers have, I think, but rarely get to flex, which is that sort of experimental scientific mindset. Like, actually, I would like to ask this question in 15 different ways and see what the difference in our customer responses to that are. But you can't do that in the real world because you've got your question weary customers that you're asking questions. You've got to really value every question that you ask them. Or you have a department in your organization who wants to add some questions to your survey and you have to say, well, I'm sorry, we can't ask any more than the 15 questions we've got in here now, so maybe next time we'll include your questions then. You never do that research for that part of your business. I think it expands the ability to do more research, to become more experimental, to test a survey design before you put it out to customers. So if you're like, I don't know whether asking this Question in this way or that way is going to get a better response from customers. You can throw both ad synthetic and see if it shows you a pattern that's like, oh, that's interesting. A good example of that is actually one we did with a travel brand where they have a trend survey that they run every year. And there was a question in there that touched on solo travel. So they were looking for solo travelers or like, are you a solo. Have you traveled solo in the last year? And when we ran the same survey through synthetic, there was a really like, it was very similar to human for most of the answers to that question. But the solo travel one was split in a really strange way. And without getting too much into the basic steps of what they essentially found out was there was humans were interpreting that question in two different ways. Did I fly solo to X4 to meet up with thousands of people, or am I flying on a solo backpacking trip to the Andes where I'm going to walk through the mountains on my own and meditate? What the travel company was really interested in was the latter type of traveler. But a lot of people ticking that box as human was interpreting solo travelers literally traveling solo. The synthetic was very one or the other about it and essentially made it really easy to spot that, oh, there's probably a bit of a King Kim way we're framing this question for humans. So now we're going to rephrase that question going forward. So we actually capture the data that we're looking for.
- [15:57]
C
Well, yeah. And to find that out after you run a real, you know, a traditional survey is helpful but expensive and, you know, are you going to actually be able to rerun.
- [16:08]
A
They've been asking that question for years.
- [16:09]
C
Right, right.
- [16:10]
A
And it's never been considered something that might have been misinterpreted before.
- [16:15]
C
Yeah, yeah.
- [16:16]
A
And it was the synthetic experiment that highlighted didn't necessarily tell you what was wrong. And that's where the kind of role of the research is really important because it took the researcher to look at it, interpret it, understand where the problem might actually be, what might need to change in order to fix it. But it, what I, what I find, like when you experiment with synthetic, it does a lot is it throws that signal up, says, look here, something interesting over here that you might want to take a look at. I'm not going to tell you any more than that, but here's an area for you to focus your attention on. And that's where the skills of a researcher really come in to then interpreting that. And Understanding where the problem.
- [16:52]
C
Yeah, yeah. So certainly I talk about AI a lot on the show in a lot of different contexts, but there is often skepticism, at least in some corners, about. I'll ask about, you know, whether it's from a branding perspective or other areas as well. How do you build organizational trust in, you know, findings that aren't, you know, they're not coming from real humans? You've given a lot of reasons why, you know, it's important and valuable, but how do you make that case internally in an organization to, you know, kind of get the ball rolling with this?
- [17:25]
A
Yeah, I think it's a matter of, again, going back to, like, experimentation. It's validation. It's. It's taking the surveys that you've asked before, running them through synthetic and going, look, 99% of the time, synthetic is giving us exactly the same result as human. This 1% is where the interesting bit lies. That might be because the model is just not very good at answering that question, which definitely happens sometimes. And again, it takes the skill of the researcher to spot it. But other times it might be like the solo travel thing, where you're like, I think there's something more going on here. But the other 90% that is in line with humans and the distributions are similar. It's not just about, does it agree top two box. It's like, is it showing the same kind of distribution as humans? If you ask a general LLM, if you ask survey questions to ChatGPT or Gemini, quite often might get the top answer correct. But it's like on ChatGPT, it's 100% of respondents answered the top answer. And we know that humans are a little messier than that. Even if the answer is obvious and correct, objectively correct, there can be a spread of things. We did an experiment asking a question about climate change. You know, is it a. An important thing that humans should be worried about and concerned about? And, you know, humans did not answer 100%. Yes. Right. Because it's an interesting question that stirs up lots of different kinds of perspectives and thoughts. So there was a distribution with humans. You ask that question to ChatGPT or to Gemini a thousand times, and it was literally a thousand times yes. No, literally 100% of respondents said that. You run it through our model and we, we had a very similar distribution to humans where it was able to grasp the nuance of human messiness.
- [19:06]
C
Yeah, yeah. There's a lot that goes, questions like
- [19:09]
A
that when you show that it's not coincidence. Right. So when you're trying to prove that to you. When you're trying to prove to your internal stakeholders that this could be valuable if you're just showing them. Well, 100% of people said yes. Well, that's obviously nothing like what we actually see in the real world. But if you show actually that the distribution is very similar, we're getting a similar result for most of these questions. But we think there's some signal over here where there is a difference. It's the 90% aligned, 10% different is where you get that buy. If it was 90% different and 10% aligned, you're not going to get that. So you've got to do the testing, you've got to do the work to match your human results to synthetic results and keep doing the human surveys as well. You need to keep that baseline up to date and make sure that it's still matching the way people are changing in the real world. Yeah.
- [19:57]
C
So as we think a little bit more into the Future here, as AI models and all the LLMs included become more sophisticated, do you see the future of market research being less about asking questions after the fact and more, you know, where does predictive simulations really come into this mix? Because right now, you know, we're probably a lot more heavily on the reactive and, you know, kind of taking action actions post survey and stuff like that. You know, where are we moving into a more predictive realm?
- [20:31]
A
Yeah, I think, I think we probably should be doing more predictive than we are like, you know, being able to survey customers in advance before we develop a new product feature or before we advance something. But it's that fear of fatigue and frustration and customers that I think stop a lot of organizations doing that. And you go, well, actually the signal we can't let go of is the NPS after the fact or the like, what, you know, rate your experience after the fact. That's the signal we can't let go of. And maybe we'll have to sacrifice asking them quite so many questions about new features as we're developing them. And another thing with that is, you know, I have fear of asking people about new things. Like if Apple had sent out a survey about, you know, would you buy a flip, Would you buy a foldable phone from Apple a year ago? Like that's leaked instantly. And yeah, yeah, everybody's like, well, look, Apple are asking questions about foldable phones. I wonder what they're working on. But you could, you could use synthetic to ask those kind of questions. You can do more predictive research that would enable you to make decisions that were driven by data in advance. And I think you still should be asking humans those questions as well. But you can tailor the questions you ask to humans and make sure you're making the use of their valuable time. Get them to think about the questions and actually be a bit more detailed. Ask fewer people, more detailed questions rather than lots of people. Very superficial questions that you end up getting uninteresting answers from.
- [21:54]
C
Yeah. Well, Jordan, thanks so much for joining. I got two questions for you as we wrap up here. First one, what's been a highlight of X4 for you so far?
- [22:02]
A
So a few years ago I saw Priya Parker talk at south by Southwest. It was just after the pandemic and she talked about the art of gathering there. And it was really fun to see her speak again yesterday and again reflect on like how amazing it is to just have so many people in one place. So catching up, seeing Priya and catching up with all of my colleagues from around the country and around the world has been amazing.
- [22:24]
C
Nice.
- [22:24]
A
Nice.
- [22:25]
B
Love it.
- [22:25]
C
And last question for you. What do you do to stay agile in your role and how do you find a way to do it consistently?
- [22:31]
A
Yeah, I think like my, my job history probably talks to this a little bit, but like, like always staying curious, always trying to keep an eye on what's next and what's moving and not be like walk towards it rather than away from it. Ask what it can do to, to make things better rather than what can we do to avoid it being a problem. I think it's really important. Love it.
- [22:52]
C
Well again I'd like to thank Jordan Harper, Principal AI Thought Leader, Edge center of Excellence at Qualtrics for joining the show. You can learn more about Jordan and Qualtrics by following the links in the show notes.
- [23:06]
B
This episode is brought to you by Tech Systems. They're leaders in full stack tech services, talent solutions and helping companies put it all in action. You can learn more@teksystems.com and thanks again for listening to the Agile Brand podcast. If you like the episode, hit subscribe and drop a rating so others can find the show too. And if you're interested in consulting advisory work or if you need a speaker for your next event, feel free to reach out. Just visit GregKilstrom.com that's G R E G K-I H L S T R O M.com the Agile brand is produced by Missing Link, a Latina owned, strategy driven, cross creatively fueled production co op. From ideation to creation, they craft human connections through intelligent, engaging and informative content. Until next time, stay curious and stay agile.
- [24:01]
A
The Agile brand.