← The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
#653: Accelerating speed to insights using synthetic feedback with Ali Henriques, Qualtrics
00:27:24
Loading summary
Transcript58 lines
- [00:01]
Strayer University Announcer
The Jack Welch Management Institute at Strayer University helps you go from I know the way to I've arrived with our top 10 ranked online MBA. Gain skills you can learn today and apply tomorrow. Get ready to go from make it happen to made it happen and keep striving. Visit strayer.edu Jack Welchmba to learn more. Strayer University is certified to operate in Virginia by Chev and its many campuses, including at 2121 15th Street north in Arlington, Virginia.
- [00:31]
Carvana Customer
I sold my car in Carvana last night.
- [00:33]
Ali Enriquez
Well, that's cool.
- [00:33]
Carvana Customer
No, you don't understand. It went perfectly. Real offer down to the penny. They're picking it up tomorrow. Nothing went wrong.
- [00:39]
Ali Enriquez
So what's the problem?
- [00:41]
Carvana Customer
That is the problem. Nothing in my life goes as smoothly. I'm waiting for the catch.
- [00:44]
Ali Enriquez
Maybe there's no catch.
- [00:46]
Carvana Customer
That's exactly what a catch would want me to think.
- [00:48]
Ali Enriquez
Wow. You need to relax.
- [00:50]
Carvana Customer
I need to knock on wood. Do we have wood? Is this table wood?
- [00:52]
Ali Enriquez
I think it's laminate.
- [00:53]
Carvana Customer
Okay. Yeah, that's good. That's close enough.
- [00:55]
Ali Enriquez
Car Selling Without a Catch Sell your car today on Carvana. Pick up fees may apply the Agile Brand.
- [01:13]
Greg Kilstrom
Welcome to Season seven of the Agile Brand where we discuss the trends and topics marketing leaders need to know. Stay curious, stay agile and join the top enterprise brands and Martech platforms as we explore marketing technology, AI, e commerce and whatever's next for the Omnichannel customer experience. Together we'll discover what it takes to create an agile brand built for today and tomorrow and built for customers, employees and continued business growth. I'm your host Greg Kilstrom, advising Fortune 1000 brands on martech, AI and marketing operations. The Agile Brand Podcast is brought to you by Tech Systems, an industry leader in full stack technology services, talent services and real world application. For more information go to teksystems.com to make sure you always get the latest episodes, please hit subscribe on the app you listen to podcasts on and leave us a rating so others can find us as well. Now onto the show.
- [02:10]
We are recording live at Qualtrics X4 in Salt Lake City and seeing and hearing all about how to create and enable amazing customer and employee experiences. How long would you like to wait to get customer insights? Weeks, days or minutes? Today we're going to talk about how increasing the speed to insights can be a game changer for brands and how synthetic feed feedback allows rapid testing and on demand marketing intelligence. To help me discuss this topic, I'd like to welcome Ali Enriquez, Global Director
- [02:38]
of Edge at Qualtrics.
- [02:39]
Ali, welcome back to the show.
- [02:41]
Ali Enriquez
It is so great to be here and actually meet in person.
- [02:44]
Greg Kilstrom
I know, I know, yeah, you were on the show last fall. But always great to be able to meet in person and talk about this stuff. And as we'll quickly get into it, things are moving quick. So this is a good topic to talk about.
- [02:58]
Ali Enriquez
They sure are. I'm happy to.
- [03:00]
Greg Kilstrom
Yeah. So for those that didn't catch the show you were on before, why don't you just give a little background on yourself and your role at Qualtrics.
- [03:07]
Ali Enriquez
Absolutely. So again, Alain Rickes. I have the pleasure of running strategy and services for Qualtrics Edge. Our kind of history, if you will, and I'll keep it brief, has always been connecting clients to audiences that they don't have access to. So our CX clients have access to their customer databases. EX clients have of course, access to their employee rosters and such. Strategy and research clients don't often have access to their competitors, customers, their prospects, things like that. So that was kind of our history and upbringing. And so what we're doing now as Qualtrics Edge is as you just mentioned, putting this power back into the researchers hands to get to answers faster with a couple of new solutions. So it's been kind of fun. We are still doing the research projects. I call ourselves kind of an in house market research agency. So a full service strategy research firm running projects end to end for our clients, writing questionnaires and fielding them and analyzing them. We're just talking about different ways to cut out steps. That process with AI.
- [04:08]
Greg Kilstrom
Yeah, yeah, love it. So yeah, let's dive in here. And so the last time we spoke, we talked about synthetic Personas and research and some of the things that you just referred to. But I feel like a lot has happened since last fall. So you know, at X4 here, where we are today, Qualtrics announced Edge Instant Insights and Edge audiences, allowing teams to transform weeks of research into minutes. Leveraging synthetic feedback for rapid testing and gain on demand marketing intelligence. That was a lot there. So can you talk, Provide a little maybe. Let's just take a step back because those that aren't as familiar with that term, synthetic research, maybe start there and then talk about the products that use it.
- [04:55]
Ali Enriquez
Absolutely. Happy to. So I think our new favorite way to describe synthetic is AI modeled responses because that's really a theme of everything we're talking about here. But that's true. That's the nature of how we're getting these, what they're modeled off of. Is what matters. So I'll just bear with me for a second as they take you through kind of what's happened. Honestly, since last fall, we've started to categorize the types of synthetic offerings that are in the market. First, you have what I call wrapper models. So it's a really sleek interface on top of something like ChatGPT, where what it's doing is effectively building the prompts for you to emulate certain populations. It's great, it has great application, but it's only referencing publicly available information. Another approach is the RAG method, which is really machine learning on steroids, where we've collected say 200 human responses, but I want to turn those into 400. And so I'm going to consider everything I know of the patterns and behaviors within these 200 responses. Make it 400. The third approach is truly custom foundational model. That's what we're building and it will include the other two kind of elements of synthetic. But the idea and the differentiation here is that when you're building a truly foundational model, you have to have access to robust data sources. And so we have the benefit and luxury of decades of data collection across all of these different pillars. So that goes through a rigorous aggregation and anonymization process across our experience data, plus the research data that we've conducted. So by design, anonymous, we don't have any way to recontact these humans, but we have millions and millions and millions of responses, even just from the last couple of years, right? I'm talking like 30 million that have taken all sorts of different studies. So those behaviors, preferences, those are things like, how do you like this new product? Right? What do you think of this ad? If you weren't going to dine with us today, where else would you have dined? All of these types of, again, consumer behaviors, that's what's feeding our model in addition to what's publicly available. So we've got the weather, we've got the news, we've got just what's on the web. And then we're also feeding it with training data. So keeping a pulse, daily pulse on the markets to make sure that the data is constantly refreshed. Not just from what I'll call standardized training data, but also a continuous influx of all of the experience data and research that we're collecting every day. So that's kind of a little bit about synthetics. So that gives us an idea of what repository we're considering when we say I need 500 prospective cruisers, because that's What I needed back in the day when I was an analyst. Right. Need people who have never been on a cruise but might be open to it. Synthetic would be a perfect application for saying, I know generally what these people look like, what age range they are, what markets they come from, what other types of travel they're interested in, what do we want to ask them? So that's synthetic in a nutshell. And I've also kind of briefly described edge audiences. So I'm explaining why Qualtrics is in a unique position to deliver on this synthetic demand. And where we're at from the last time we spoke. Greg, it's here. You can go to the XM park and demo it today. So it's super cool and. Yeah, so I'll pause on audiences or shall I move on to Instant Insight?
- [08:29]
Greg Kilstrom
Well, yeah, I mean, just one thing on that and to just kind of underscore what you're saying. I mean, I think someone might be tempted to go to ChatGPT and like type in like, you know, do the thing like you are a blah, blah, blah, and try to emulate. And to your point, that can be somewhat effective. But I think to me the power of this is in the combination of all of those things. And also Qualtrics, obviously you have a software platform, but as a research organization too, I think the marriage of those two things brings a little more rigor to it, would you say?
- [09:03]
Ali Enriquez
Absolutely, yeah. And there are other agencies out there developing their own synthetic. We're all taking slightly different approaches and we're starting with more of an audience angle, pun intended or not. And that meaning, like we're targeting us general population consumers versus going use case. Right. So we could say, look, we're going to become the experts in replicating responses for concept tests or ad tests or something like that. And so we're taking a slightly different approach.
- [09:33]
Greg Kilstrom
Yeah, yeah, makes sense. So, yeah, why don't you talk about Instant Insights?
- [09:38]
Ali Enriquez
Yes, of course. Instant Insights is an industry specific kind of market intelligence platform. So what it does is it combines survey research. So a syndicated study, effectively I'll pick on restaurants because that's the one that's been running the longest. So we've got a lot of what you'd expect, brand perceptions, you know, where did you dine last? Where else would you consider some kind of competitive benchmarking type of components in there? But what we're doing is we're augmenting that with about five or six other data sources. So we're pulling in search trends. That gives us a Sense of rising and trailing kind of trends and topics. That way we can see promotional offers. What else is happening in the market that might be influencing the consumer behavior? We also get that from news and trends. So topics, anything that might even be popular or viral on Instagram will make its way here. So we're pulling in what's happening in the moment in addition to transaction data. And then what really makes this cool is behavioral data. And what that means is we've got in person kind of foot traffic. I heard this morning and absolutely loved the line. People vote with their feet, right? And so that's part of what we're tracking here. We've got physical in person location tracking as well as web and digital. And so part of the story you can tell here is if I'm not choosing Chipotle, where am I going and why and how do I spend my free time? Who is this person? Outside of the context of the transactions and the experiences I have with them, that's often really hard for brands to stitch together. They know their customers very well. They know when Greg came in the restaurant, they know what he ordered, they know when he left. No idea really who you are outside of that. So Instant Insights is here to kind of bring that to life, but also give you more of that kind of competitive context. So we start with these concentric circles of my category, my brand, my competitive set would kind of be that next ring. And then outside of that, how do you move about this world? Right? And where else are you spending your time and money?
- [11:44]
Greg Kilstrom
Well, I think the other thing there is things move very quickly. I mean, even my behavior could change rather quickly as well. So even taking a traditional predictive analytics or taking survey results from two months ago, because how many companies can run surveys constantly and stuff like that? So what I'm hearing there is, yes, there's that legacy stuff as well, but it's also augmented by things that are up to the minute. Because for better or worse things, these are the times we live in that anything could change sort of on a dime, right? So it's kind of, it's mixing the known and the now, right?
- [12:22]
Ali Enriquez
That is exactly right. Try as we may, my past behavior does not necessarily tell you what I'm going to do next.
- [12:32]
Greg Kilstrom
I want to talk a little bit about a use case with synthetic data. One of your sessions at X4 is a presentation with booking.com and how they're using synthetic data. So I'd love to hear just a little bit those that were here. I'm sure we'll check it out. But can you talk maybe at a high level about the partnership first?
- [12:53]
Ali Enriquez
Yeah, we actually just did that this morning. It was so fun. What I love about this, we're in this kind of pilot phase, and we're all going into it eyes wide open. And so the way that Elina, the client, represents the work is this is not an AB test. We did not run a human set of data and want to see how well the synthetic did against it. Instead, we want to understand the journey and the process. And so while she did have human responses to be able to compare, we certainly showed that it wasn't the spirit of the experiment. And so I think my favorite. And this was all her work, and it was done so beautifully. And I just kind of interjected with research on research and some of what we're seeing from the market. Her biggest kind of takeaway was that the researcher is still very, very important. Right. The way that we construct questions and the way that we think about presenting what we're curious about to human or AI responses is really important. That matters. So, for example, she took a question of which of the following types of trips did you take in 2024, Greg? And what she found is that there was actually recency bias in the humans. So the number one answer, because the study is conducted in January, what did you do in December? Oh, you visited friends and family. Right. Well, AI said no. I actually think most people went to the beach. Right. Beach trips are the most popular type of travel. And so she found herself questioning, have we been lying to ourselves for 10 years thinking that friends and family are the top travel? So it's just. It's so funny how this has just been a process of really scrutinizing our own work as humans and how we ask questions and how. I take one simple question, and I think each of us. Us would answer it differently. And so, you know, progressing that same kind of thought. Her next big finding was that synthetic responses are really, really good for attitudinal psychographic type of work. That makes sense because a lot of who we are doesn't change. Our behaviors will. Right. And there's absolutely an element of prediction there. The other finding around, oh, this one was fun. It was such a mouthful for her to get out. Humans. We did not trust humans. Reflection of how they've used AI in travel planning. AI actually told us how humans use AI for travel planning better than the humans did. And in here, we have evidence, we have operational data to tell us that we're using AI to help with itinerary planning and the sightseeing and what's not to miss in a certain place. But the humans reflected maybe a bit more, again based on recency, restaurant recommendations and things like that. So she said, here I have external evidence to suggest that the AI responses were more accurate than the human responses. And so just fascinating how we're really forcing ourselves as researchers to think a lot more critically about the questions that we write and the implications of who's on the other end. At the end of the day, we're lazy, right? We're going to. To skim the question, think we get the spirit of it, and choose the couple of things that are top of mind for us. AI is not doing that. It's thinking very, very critically and reflecting on that. And so you see that come through in a couple of other examples like solo travel or traveling alone. Right. And it's fascinating. So it was really, really great session.
- [16:33]
Greg Kilstrom
Yeah. I mean, and it's an interesting thing of, you know, certainly there's a lot of talk about bias in AI, and that's something companies and people should take very seriously, but there's not as much talk about the bias in humans. I find it, what you're saying, very interesting in that if AI has the opportunity, I mean, sure, there's ethical things that need to be taken care of, but it also can help us solve some of those biases that nefarious or not, there's just. There's bias, whether it's recency anchor, there's 100 cognitive biases. Right. So it's like, how can AI actually help us get past those? Because my assumption is at the end of the day, that actually helps the customer. I mean, if that's really what it's about, it's kind of helping us think past the holiday family visit that we did and the thing that really matters the rest of the year or something.
- [17:27]
Ali Enriquez
Exactly.
- [17:29]
Greg Kilstrom
For those. And booking.com, i mean, what I know about them is they've been very forward as far as testing and all of that stuff as well. So definitely sounds like a great partnership and a great opportunity for those that want to do something like that. What's the mindset? Or how can an organization kind of get ready to do some of this work?
- [17:53]
Ali Enriquez
It's such a great question. And a couple of folks approached us afterwards and said, how do I guide my stakeholders? Right. And I've heard both extremes of this. I actually had them do a quick poll before we got started. I wanted to know who the haters were in the room.
- [18:08]
Greg Kilstrom
Right.
- [18:09]
Ali Enriquez
Who do I need to convince that this is real and it will work for you? And then I had the opposite challenge. A woman approached me and said, I need to rein my people in because we've all just gotten enterprise access to ChatGPT and they all think they're researchers now. Right. And so it's a crazy spectrum here, but I think we get the question a lot right. How do I get started? What types of studies should I pilot? We talk a lot about early innovation testing, so idea screening, feature optimization, promotional offers, things like this. Those are great project types that are really purpose built for synthetic. But even back to our last topic, Greg, think about survey design, semantic and cognitive testing. That is no brainer, that is available to all of us today to just have these tools help us improve the questions that we're intending to ask of humans. Humans and make our response options more exhaustive. So there are just really low risk, tangible things that we could try tomorrow. But in terms of more fully synthetic responses and considering a blend or full replacement, I cannot answer that for most clients because it really depends on the weight of their decision. And I said in the room, if you're making a $5 million marketing decision based off of the output of this work, I would probably get some humans. Right?
- [19:36]
Greg Kilstrom
Yeah, yeah.
- [19:37]
Ali Enriquez
So let's be careful here because it is so new. But what I find most of, and I'll speak for the researchers out there, they want a look under the hood and they want to compare side by side as real time as possible. How did your synthetic compare to these humans? Accepting that the humans are flawed as they are. Right. And so that's what we're finding most often. I actually am coming to you just now from a session with Google Labs and they piloted our synthetic and we ran a parallel with humans so that we had something to compare it to. Right. And to be able to assess what our comfort levels were with different question types and topics. And so I'd recommend experiments like that to just get started to convince yourself that this will work for either the use case or the audience or the project that you're thinking about.
- [20:32]
Greg Kilstrom
Yeah, yeah. So let's maybe look out a couple years. I know things move very even a couple months, but like a couple years out, you know, we're at the beginning stages. As. As you're saying, things move quickly. I mean, if we would have guessed three years ago what ChatGPT would have been doing, you know, it would have been hard to predict. Where does synthetic kind of go from here? You know, what's, what's kind of next on the horizon there?
- [20:57]
Ali Enriquez
You want the Ally answer or the Qualtrics answer? No, I'm kidding. It's not very different. I think what's different is the kind of timescale. Right. And it's not our ability to develop the technology. I actually believe what will, I'll call it hold us back most is the client receptivity, their openness, their willingness and then their own organization's level of kind of on that spectrum of innovation and progression. So where I see this going, and I was just telling Google just before this, I imagine a world where yes And I realize I'm a researcher and I'm quite arguably putting myself out of a job. Right. I imagine a world where we just ask a question and get an answer. These models have gotten so smart, so real time that it's taking in the weather, the traffic, the every signal we can possibly feed it about what's happening around us right now in this moment. And it's guiding us in whatever we can think to ask. So what I mean by that is today the researcher needs a thousand rows of data to feel confident in the answer that they're going to give their stakeholders. We're holding onto that a lot. And I understand within the next few years we'll move away from the these thousand records and we'll accept one question answer. Right. And that's where we'll get talk about rapid insight testing. That's in everybody's hands. That power has to be in the marketer's hands, the operator's hands, the CX leaders hands, not just the researchers. And there will still be projects that only the researcher can execute. Right. And that will always be true. And so I joke but I do maintain that job security is still there because even with synthetics, like think about how much of this time we've spent talking about question design. Right.
- [23:00]
Greg Kilstrom
I mean that's what I'm thinking there is that, you know, doesn't that then elevate the role of the person asking the question, you know, some of the busy work and all that gets taken care of or even some of the details. But it would seem to elevate the researcher's ability to ask questions. Right?
- [23:20]
Ali Enriquez
That's the dream. Yeah, exactly right. And then we're, we are freed up to focus on the things that we actually really, really are genuinely passionate about and not coding the open ended responses and tabulating the data.
- [23:35]
Greg Kilstrom
Yeah, yeah, totally. Well, as we wrap up here, two questions for you. First, I know we're about halfway through the conference here. So maybe too soon to ask, but what's been a highlight for you so far?
- [23:49]
Ali Enriquez
What a good question. I love Main Stage. There is just so much energy and buzz and excitement. So I loved today's and was particularly, I found myself literally laughing out loud at Ken Hughes. Like, who's this guy? Where did he come from? And what got me the most was now I'm dating myself. But when you click save in a document, his child asking if that was a microwave over what for a fridge or I just, I'm dead. The kids holding up the cassette tapes. So I just, I love this kind of perspective on our world. Right. That we just eat, sleep and breathe every day. And then, you know, the breakouts are wonderful. I've had the pleasure of running a couple of them earlier today. The one with booking.com, of course. And so it was at the same time as the Google Lab session, so I missed that. So I'm turning my highlights into lowlights. We can't be in multiple places at once. I feel I'm missing out a little. But yeah, so no, it's great. I think then to sum it up, the energy and the buzz that you just get from being around folks who are all tackling the same kind of business challenges.
- [25:09]
Greg Kilstrom
Yeah. Love it.
- [25:10]
Well, last question.
- [25:11]
I know since you've been on the show, I've asked this before, but we'll see how close the answers are. That's a research project, your own validation. I know, I know. Right. So what do you do to stay agile in your role and how do you find a way to do it consistently?
- [25:26]
Ali Enriquez
You know, I'm trying to think about what I might have said last time. Hmm. You've stumped me, Greg. I think this might actually be different. Right. I've taken to podcasts a lot more, so I have a long commute. I love to first make sure I get news and just kind of really timely stuff out of the way. And then I spend most of the rest of my time just listening to very like business related type of content in particular AI. I may not understand the importance or significance of all of the chips and the storage and the who's who and the who's what in this category. But the more I listen, I think it's more important to just understand a lot of where some of these visionaries see this going because it helps me more confidently answer for you how quickly we're going to go from question to answer. So lots of just content consumption. But I think this one I'm confident I didn't tell you this last time. I have three kiddos and I absolutely love seeing the world from their perspectives and vantage points and so I think a lot lately about just the way that they take information and they take the world in and I try to kind of harness that and bring it to my day to day too. And in some cases that might be just approaching things totally. Eyes wide open, right? Genuinely excited to learn about something even if it may seem super simple and non consequential to my day to day. So yeah, I think a couple of those things.
- [27:12]
Greg Kilstrom
Yeah.
- [27:12]
Ali Enriquez
Yeah.
- [27:12]
Greg Kilstrom
Love it. Well again I'd like to thank Ali
- [27:15]
Enriquez, global director of edge at Qualtrics,
- [27:17]
for joining the show here at Qualtrics X4 in Salt Lake City. You can learn more about Ali and Qualtrics by following the links in the show notes.
- [27:27]
Thanks again for listening to the Agile Brand brought to you by Tech Systems. If you enjoyed the show, please take a minute to subscribe and leave us a rating so that others can find the show as well. You can access more episodes of the show@theagilebrand.com that's theagile brand.com and contact me. If you're interested in consulting or advisory services or are looking for a speaker for your next event, go to www.gregkilstrom.com that's G R E G K I H L S t r o m.com the Agile brand is produced by Missing Link, a Latina owned, strategy driven, creatively fueled production co op. From ideation to creation, they craft human connections through intelligent, engaging and informative content. Until next time, stay curious and stay agile.
- [28:22]
Ali Enriquez
The agile brand.
- [28:30]
Strayer University Announcer
The Jack Welch Management Institute at Strayer University helps you go from I know the way to I've arrived with our top 10 ranked online MBA. Gain skills you can learn today and apply tomorrow. Get ready to go from make it happen to Made it happen and keep striving. Visit strayer.edu Jack WelchMBA to learn more. Strayer University is certified to operate in Virginia by Chev and has many campuses including at 2121 15th Street north in Arlington, Virginia.