← The Marketing Architects
The Marketing Architects
Synthetic research vs human insights in marketing
00:24:14
A recent study comparing AI-driven research to traditional focus groups found synthetic research is faster, more efficient, and better at spotting trends in massive datasets than human-to-human methods. Elena, Angela, and Rob explore how synthetic...
Loading summary
Transcript72 lines
- [00:00]
Alena Jasper
There's the argument out there that synthetic isn't real or isn't human. And I think that just misunderstands how AI models are trained. They're built on the aggregation and analysis of real human behavior.
- [00:14]
Angela Voss
Marketing Architects hello and welcome to the Marketing Architects, a research first podcast dedicated to answering your toughest marketing questions. I'm Alena Jasper. I run the marketing team here at Marketing Architects, and I'm joined by my co hosts Angela Voss, the CEO of Marketing Architects, and Rob DeMars, the chief product architect of misfits and machines.
- [00:34]
Rob DeMars
Hello.
- [00:35]
Alena Jasper
Do we have a real Snoop Rob, or is it a synthetic Snoop Rob with us today.
- [00:40]
Rob DeMars
Domo edicato, Mr. Roboto.
- [00:44]
Angela Voss
So it's the real thing. Okay, perfect. We're back with our thoughts on some recent marketing news. Always trying to root our opinions in data research and what drives business results. Today we're talking about synthetic audiences or an AI generated group of virtual consumers that simulate real customer behavior, which helps marketers test strategies, refine messaging and predict outcomes before launching campaigns. We did cover this topic, but it was about a year ago and as you can imagine, a lot has changed and we have some firsthand experience using synthetic audiences to pre test marketing creative that we wanted to talk about. So should be interesting and an important topic for really any marketer. So let's dive in. I chose an article for MORC to feature today and it's titled Comparing the Quality of Synthetic Research to Human Insights. This article summarizes a study on synthetic research by Digital design and engineering studio Siberia. In partnership with synthetic Users and the James Beard foundation, they set up a head to head test synthetic research powered by AI versus human to human research like focus groups and interviews. And this all used the same psychographic and research briefs. Synthetic research relies on AI large data sets and natural language processing to generate insights. This is all without direct human interaction. They wanted to know if AI could truly replace the deep nuanced understanding that human research provides. What they found was AI driven research was faster. It was more efficient than human research when it came to gathering raw insights. It was great at sifting through massive data sets and spotting trends that a human might miss. They did find that AI couldn't replicate body language or emotional cues. So for research that required deep understanding or research that was highly competitive, they recommended human to human methods. So that is what just one study found. Rob, I wanted to ask you how much do these findings align or not align with our own experience with synthetic research? And do you share sort of those same concerns about more like the human areas of research not being ideal for.
- [02:41]
Rob DeMars
For AI, yeah, AI is definitely. I mean, I know I'm going to probably state the obvious here, but it's a quantum leap forward in terms of what you're able to do with research. I think there's really three key components they talked about in the research. Speed. Right. When you compress what can take weeks, sometimes months, into seconds, it's like comparing a sundial to an Apple watch. You know, when you think about just the leap forward cost, I mean, you're literally taking the entire research process that can cost hundreds of thousands of dollars and driving that cost down to zero. And then there's the accuracy part. And we've definitely found that synthetic audiences are in oftentimes more accurate at predicting outcomes in market than humans. When it comes to the body language part, I don't have a lot to offer there. I've never seen a huge consumer insight that transformed the growth of a body brand come through like somebody wiping their brow or. What is this called, when you do this with your arms?
- [03:44]
Angela Voss
It's an audio podcast, Rob.
- [03:46]
Rob DeMars
So I know, but what is that when people do that or when they cross their arms right. In a focus group, I'm like, I don't know, I guess I don't have a strong opinion there. And I'm totally open to being corrected. If someone said that led to a huge consumer insight, but just the mass data that you're able to aggregate and synthesize, it's not even a comparison, in my opinion.
- [04:07]
Angela Voss
Yeah, that is a good point about body language. Like, how good are humans even at reading someone's body language? Well, we're going to talk more about what we found these synthetic audiences are good at. But before we get too deep into that, I thought it might be helpful to talk through how we actually use this technology so that listeners know where we're coming from and when our experience is. So, Ang, would you mind walking through how Marketing Architects is currently using synthetic audiences?
- [04:31]
Alena Jasper
Yeah, I mean, I think we could spend a lot of time in this area, but I think I can hit a couple several key ways to enhance our marketing strategies and optimize client outcomes. One of the powerful applications has just been in that consumer focus group. Right. Type environment for brainstorming and uncovering new insights. You know, it's funny, in the marketing space, somehow we got to a place where we felt like if we were to put 12 people in a room, we're going to produce gold. And it's sort of funny to think about it's. 12 people or it's 20 people, like of all of the consumers, you know, and you go, are you really getting quality out of there? But in this case we can use synthetic groups where AI driven research allows us to rapidly test early stage concepts, campaign directions, emerging trends. Instead of relying solely on that traditional focus group method, we can simulate diverse consumer segments. As we think about broad marketing principles, I think often marketers go, well, I really need to understand my core. That's the most important. Well, why do you go there? Because research is expensive and it takes a lot of time. So of course you focus in on that low hanging fruit, so to speak. But if we need to reach new audiences and create relevance with a broader group of people, this is a great way to do that, you know, so uncovering brand positioning strategies. And I think the biggest Rob was instrumental in this. One of our most innovative applications is ScriptSuth, which is our proprietary, proprietary AI creative pre testing platform that we developed. It's been proven to be highly predictive of in market TV results and I think that's a key piece. And I know we're going to get into trust later, but we were sitting on a bed of performance history in terms of both radio and television. And so being able to kind of validate that, we could leverage large language models and test as many scripts as necessary against synthetic audiences to identify the most effective creative concepts before production. You know, it ensures that the commercials that we bring to market are already optimized for that maximum effectiveness, which helps to reduce risk and just increase that likelihood of campaign success. So that's been a huge one for us and for our clients.
- [06:49]
Angela Voss
And Angie mentioned that we used a lot of our own history to build scripts. I know that a tool like that's custom built, it's pretty complex. Like I know it took us a long time to build it. And the data you put in matters a lot. This isn't as simple as just like asking ChatGPT what commercial is going to perform the best.
- [07:07]
Alena Jasper
Agree. And I think that's where you hit the, the trust problem. This is where people go, well of course I can throw anything into ChatGPT and ask it what it thinks, but how do I trust that it's predictive or it's validated or it's representative of whatever that consumer sentiment might be.
- [07:25]
Rob DeMars
Yeah. When we were developing scriptsuit, there's really three key components that I feel like helped us unlock the technology. One you've mention, which is the proprietary data set, I mean that was really used to calibrate the prompt engineering that works behind the scenes of the tool. Also building a methodology that can withstand the instability of the LLMs because they are their own beasts. And you have to make sure that you're accounting for that or you can get some crazy results. And then I think last, how do you operationalize it in a way that it's easy to use and scalable? You don't have to be necessarily a deep AI scientist to be able to use a tool. The tools that you build have to be usable by everybody on your team.
- [08:08]
Angela Voss
And Rob, I know it took us like thousands of different iterations of this. Right. To get to one was usable like this took us a long time. Because I think one assumption when you use these things that is hard for me when I'm talking about it with marketers is because the output happens so much faster, people assume that there wasn't a lot of proprietary or like, you know, an advantage going into building it. Like building. It's not that simple.
- [08:28]
Rob DeMars
No. It took us at least four months and thousands of prompt considerations for us to get to the ultimate tool that we were able to create. And again, even all of that would have been really just a theory if we didn't have our own proprietary data to stress test everything against. We needed to make sure that we were grading our own homework and that the tool was actually accurate at the end of the day. And to Ange's point, that we could trust it.
- [08:53]
Angela Voss
Yeah, I think that's probably one of the big challenges right now with using any sort of synthetic audiences. How do you know if it's accurate? So that's kind of our favorite example is commercial pre testing because we're a TV agency. But there's a lot of other opportunities for marketers to use AI. And I can just speak to one. One thing we did was every year at our agency, we do an annual survey asking marketers what they think about a variety of topics. So things like what channels are you interested in and where do you look for when you're talking to other agencies? And just a lot of general questions. And this year we ran our traditional survey and then we also ran a synthetic survey just using ChatGPT and plugging in, like the demographics of our audience into ChatGPT. And it was scary accurate. It was accurate with every single question, which we had like 20 to 30 questions. The only ones that it was a little bit off was the average TV test size a marketer's interested in. And then obviously, like any general questions about our brand awareness, it was Very generous with how many people knew marketing architects that I don't think is totally accurate. But besides that, for our survey next year for these general questions, I'm definitely just going to use the synthetic audience because it was so accurate. But Ange, I was hoping you could also share just other ways you think a brand might use synthetic audiences. Not everybody listening to this podcast is running TV commercials, so any sort of quick wins might be helpful.
- [10:16]
Alena Jasper
Yeah, I think even you didn't mention this one. But some of our own marketing, you know, LinkedIn ads, et cetera, we've also used synthetic audiences for and it's been predictive of what we find in terms of in market results. So from a digital perspective that could be helpful. I would say competitive analysis and scenario testing. By training synthetic audiences on real world behavioral data, brands can look to model different landscapes. Competitive landscapes predict how consumers might react to changes in pricing or changes in positioning or even major market disruptions. We're hoping to not have any of those for the near future. But we did have that happen obviously during COVID And then I think to your point, Rob, a quantum leap forward in terms of what we can go do different than maybe what we might have been constrained by in the past in terms of the amount of content that we can push out there. So things like personalization and dynamic content testing. Brands can use synthetic audiences to pre test different variations of personalized ad creative. It could be email subject lines or landing pages to determine which messaging is most likely to drive engagement and conversion among specific customer segments. So there's a lot, I mean there's a lot of opportunity and spending dedicated time as a marketing team thinking through what those opportunities are and sort of charting out, you know, what's the low hanging fruit, the matrix of what we can gain maybe with the bigger moves, but putting those on the roadmap for the future is worthy time spent.
- [11:53]
Rob DeMars
Yeah, I'll say. How do you continue to look at the tools that you do decide to use and stress test them against some of the frontier tools that are coming out. So for instance, Both Google and ChatGPT have released deep research functions now that use advanced reasoning and they're really powerful. And you can even go back to some of the tools you're paying a lot of money for and going, wow, these things are crushing those tools and they cost $20 a month. It's just so pract. I actually laugh at that sometimes. You'll be like, hey, have you tried Google's new deep reason? Well, I don't want to pay $20 a month. It's like, well, that's like the cost of the M and Ms. You spent in the focus group to, you know, to get research that didn't even matter. Like, come on.
- [12:37]
Alena Jasper
Totally.
- [12:37]
Angela Voss
I think one takeaway from this is if you're a marketer and you aren't using either ChatGPT, Gemini, some sort of large language model that you don't have your audience plugged in there, you could be asking IT questions. What do you think about this? That we have a audience within a custom GPT where I could just go ask this quote unquote marketer questions. What would you think of this campaign? What would you think of this email, this messaging? I mean, if you're a brand and you don't have your audience in one of these GPTs, there's just no reason in my mind not to do it because all it's going to do is help you feel more confident about your campaigns. You can ask IT questions anytime. And I think if you don't have that, that's just low hanging fruit for anybody. One thing I wanted to talk about was skepticism because that I think is the probably the biggest barrier right now for marketers trying to use synthetic audiences. And it might be because people think AI lacks human creativity, maybe that the models aren't sophisticated enough. So let's talk about that. What do we think has to change for more marketers and brands to feel comfortable using this stuff?
- [13:36]
Alena Jasper
I think a lot of it is just usage. You know, it's just like anything else. There's an adoption curve. So you've got early adopters. Rob for sure is in that bucket. I would say us as an agency, we are definitely in that bucket. But I think a lot of the skepticism stems from the fear that AI oversimplifies consumer behavior. Right? So reducing people to data sets that lack emotion, some sort of nuance, you know, And I think to overcome that, marketers need that clear evidence that synthetic research can produce insights that are just as actionable and in some cases more precise. They're just getting better and better and better than even traditional methods. And then I think there's the argument out there that synthetic isn't real or isn't human. That just misunderstands how AI models are trained and what they actually represent. AI generated synthetic audiences are not invented from thin air. They're built on the aggregation and analysis of real human behavior. It's just collected at scale through these vast data sets. The foundation of any AI model, whether it's used for research or Prediction or creative generation is historical human data. There's just still a bit of a misunderstanding. When you can do something in seconds instead of weeks, months and you can do it for far less, it feels like it's fabricated or somehow just unreal. And I think usage and becoming more accustomed to understanding the nuance of LLMs and how you need to prompt it is part of on your like we gotta have to come across the bridge. You have to be able to do it well in order to gain the insight that is going to be useful to the brand. But in getting there, I think it unlocks a world of potential in terms of growth, frankly for brands.
- [15:29]
Rob DeMars
Yeah, I think usage is a great one. Ang usage and time. Right. A year ago people thought we had a third eyeball when we talked about scripts, but now it's yeah, of course it does that. You know and I think you've had some of your spicier LinkedIn posts related to synthetic testing. Alina if I remember people get really.
- [15:48]
Angela Voss
Upset about now it's like.
- [15:51]
Rob DeMars
And I was like yeah, of course it is. I mean it was the same thing with generative video when people like oh that sucks.
- [15:56]
Alena Jasper
It's crazy.
- [15:57]
Rob DeMars
It looks like people's faces are melting off their head and now it's like oh okay, yeah, of course, yeah, of course we're going to be making TV commercial. We are making TV commercials with it. I mean it's so it just takes usage and time.
- [16:10]
Angela Voss
I agree with you. It's hard to find data on how many companies allow ChatGPT. The last thing I saw was it's like around 40, 50% just allow it in general. But that I think was an estimate of companies in the US in general. I found another stat late 2024 that 77% of marketers are using something like ChatGPT. That's where. How is it not 100% if you're a company that's not allowing your employees to use ChatGPT? I mean first of all they probably are. They're probably using it without you knowing it. But it's such a miss, you're losing such an advantage. And yeah, I wonder too if there's also part of the challenge marketers face is just C suite buy in to data like this because there's just skepticism. So a lot of marketers I think might not want to trust something like this, run it, present it because they're going to get pushback and people just are more comfortable with data that comes from humans. Even if you can prove it's more accurate there's just that sense of I want the human data. All right, well, so AI, it's changed a lot in general in the past year, the past few weeks, even the last couple days. Everything is changing constantly. So, Rob, I wanted to ask you, what improvements have we already seen with these sort of models that impact synthetic audiences? And what do you think might still be to come in this area? What can marketers get excited about in the future?
- [17:24]
Rob DeMars
I am super excited about what's going to happen with agentic AI. You're already starting to see that happen. If you're not familiar with the term, it's basically like AI being a super smart assignment assistant for you that you're able to go and send off and have it do multi, you know, level tasks. And if you check out the recent release from OpenAI and their operator, you can get a demo on how simple yet effective the technology can be. But when you take that and times it by 10, you can just see all the potential possibilities, things like how can it go and collect data in ways that current search can't accomplish. You can literally send out AI agents to collect and synthesize and analyze consumer data and behavior like we've never been able to do before. Or we talked about focus groups and using synthetic audiences. But now imagine having agentic focus groups where you're able to have different types of agents that are debating each other on particular products, features and benefits. And then also using agents to do real time AB testing on your campaign. So literally to having agents go stress test your flows on your website. I mean, all of these things now are just on the fringes of happening. So I'm really optimistic that it's just going to continue to empower marketers to do things with the same, same criteria we talked about before. Right. Speed and cost and accuracy is just going to get better.
- [18:53]
Angela Voss
Yeah, I think we've got to keep our eyes on agents. I'm so excited. There are a couple use cases in my job that I am just waiting. I know that ChatGPT has operator and you know, we can start testing stuff like that. But very excited for what agents could potentially do. All right, to finish up here, I've put together a little game for us. This one is called Guess the Focus Group Fail. Now, I'm not trying to pick on focus groups, but I thought this would be fun and I think it's good to remind us that AI makes mistakes, but so do humans and sometimes they're big ones. So I'm gonna talk about a focus group sort of Failure and give you three options for what the failure was. And I want you to guess what you think it was.
- [19:33]
Rob DeMars
Okay, let's do it.
- [19:34]
Angela Voss
Okay, first one. This has to do with the new Coke. This was back in 1985. Some say this is the biggest focus group fail of all time. So 1985, Coca Cola, they reformulate their iconic soda based on taste test. But what did these focus groups fail to consider? A, the formula tasted too artificial. B, people actually loved the original Coke's brand and emotional connection. Or C, the formula contained an ingredient that turned people's tongues blue.
- [20:03]
Alena Jasper
I'd have to go B on that one.
- [20:06]
Rob DeMars
It was B for sure.
- [20:07]
Angela Voss
Yeah, it was B. So the focus group, they love the taste, but they didn't realize people had this deep attachment to.
- [20:14]
Rob DeMars
Don't mess with my Coca Cola.
- [20:16]
Angela Voss
No.
- [20:16]
Alena Jasper
Why are you trying to fix. It's not broken.
- [20:20]
Angela Voss
Okay, example two, this one has to do with Pepsi. All right? This is about Crystal Pepsi. In fact, I don't know if you remember.
- [20:28]
Rob DeMars
Oh, I do.
- [20:29]
Angela Voss
Okay. This is in the 90s, Pepsi launched a clear soda called Crystal Pepsi. What was the fatal flaw in the focus group testing? A, people thought clear meant healthier, but it wasn't. B, it tasted identical to Pepsi, making it pointless. Or C, the bottle exploded when shaken.
- [20:47]
Alena Jasper
What do you think, Rob?
- [20:48]
Rob DeMars
This is a good one. I'm gonna go with A. If people thought for some reason it.
- [20:52]
Alena Jasper
Was better, I'll go B.
- [20:53]
Angela Voss
It was A. So the focus groups, they associated clear drinks with healthier beverages, but it was still just Pepsi, so it flopped.
- [21:01]
Rob DeMars
It's like when Miller came out with clear beer. I don't know if you guys remember that. They did the same. I was in college at the time, so it still made you drunk.
- [21:10]
Alena Jasper
Sometimes humans, I don't know. Like, oh, it's clear. It must be like water. It's healthy. Like, yeah, Come on.
- [21:17]
Angela Voss
Yeah. I might think the same thing, to be honest.
- [21:20]
Rob DeMars
Was like, Zima. Is Zima still around? That was like, the clear.
- [21:24]
Angela Voss
Yeah.
- [21:24]
Alena Jasper
You guys don't even know malt liquor. Yep, yep. I know Zima. Used to drop a Jolly Rancher in it.
- [21:29]
Angela Voss
Zima sounds like some sort of, like, disease or something to me.
- [21:34]
Alena Jasper
But I don't know if Zima's around, if they are, like, props to them because I have not seen any marketing. I don't know how they can continue to exist.
- [21:45]
Angela Voss
Okay, I got two more for you. So this next example has to do with Ford. So Ford, in 1957, they had spent millions researching and testing the Edsel Edsel I'm not sure how to pronounce that. Edsel. But it became one of the biggest automotive flops in history. What did the focus groups get wrong? A, the car was too futuristic for the average buyer. B, the focus groups didn't predict that consumers wouldn't like the unusual styling and name. Or C, the steering wheel was designed backwards, making it hard to drive. I think C's answers are kind of throwaways if you haven't guessed.
- [22:18]
Alena Jasper
Or C, the car didn't have wheels.
- [22:21]
Rob DeMars
It's funny, though, because I'm actually. I remember the Edsel was a big flop, but I thought it was because it, like, couldn't. It was really bad at driving. Like, it was a bad car.
- [22:30]
Alena Jasper
It didn't have wheels.
- [22:31]
Rob DeMars
It's almost gonna go with three for a minute going. I don't remember why it didn't work well, but I thought the thing was, like, it was a lemon.
- [22:37]
Alena Jasper
Okay, I'll go B.
- [22:38]
Angela Voss
That was the correct answer. So they thought, like, this unique style and branding would be a hit, but consumers found the name strange and styling unattractive. Probably speaks to the power of consistent, distinctive assets from a brand like Ford. Okay, I got one more. This one disgusts me just talking about it, but this is about Colgate frozen dinners.
- [22:54]
Rob DeMars
Oh, yuck.
- [22:56]
Angela Voss
They once launched a line of frozen dinners. What did the focus groups fail to predict? A, people associated the brand too much with toothpaste, making the food seem unappetizing. B, the meals contain an ingredient that caused mild mouth numbness. Or C, the packaging design made it look like a medical food product.
- [23:12]
Alena Jasper
I mean, just based on Rob's reaction and my internal reaction, I gotta go with number one.
- [23:18]
Rob DeMars
I. I just. I. I think stomachache when I. Yeah. You know, you're not supposed to swallow your toothpaste and yet you're gonna. Yeah, I gotta go with one, too.
- [23:26]
Angela Voss
Yeah, that is correct.
- [23:27]
Alena Jasper
Yeah.
- [23:28]
Angela Voss
I don't know how you get that wrong, but. Yeah, I know. But the lesson is that even focus groups, humans, sometimes we get it wrong and it can't prevent flops. So give AI. You know, give it some grace.
- [23:39]
Alena Jasper
We don't know what the quality of these focus groups are. How many people are we talking to? And are they representative of the broad sample?
- [23:47]
Angela Voss
So there we go. That's it for this episode of the Marketing Architects. We'd like to thank Taylor De Los Reyes for producing the show. You can connect with us on LinkedIn. And if you like the podcast, please leave us a review. Now go forth and build. Great marketing guys are getting good at these games.
- [24:06]
Alena Jasper
I'm old.
- [24:07]
Rob DeMars
I just remember all these things cuz I'm so freaking old.
- [24:10]
Angela Voss
Marketing architects.