← The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
#717: Understanding customers by simulating them first with Mike Taylor, AskRally
00:28:28
Loading summary
Transcript56 lines
- [00:01]
Mike Taylor
The Agile Brand.
- [00:10]
Greg Kilstrom
Welcome to Season seven of the Agile Brand where we discuss the trends and topics marketing leaders need to know. Stay curious, stay agile and join the top enterprise brands and martech platforms as we explore marketing technology, AI, e commerce, and whatever's next for the Omnichannel customer experience experience. Together we'll discover what it takes to create an agile brand built for today and tomorrow and built for customers, employees and continued business growth. I'm your host Greg Kilstrom, advising Fortune 1000 brands on martech, AI and marketing operations. The Agile Brand podcast is brought to you by Tech Systems, an industry leader in full stack technology services, talent services and real world application. For more information, go to teksystems.com to make sure you always get the latest episodes, please hit subscribe on the app you listen to podcasts on and leave us a rating so others can find us as well. Now onto the show.
- [01:08]
Co-Host
Is the most.
- [01:09]
Greg Kilstrom
Effective way to understand real human behavior.
- [01:11]
Co-Host
To simulate it first, agility requires a willingness to test ideas that sound strange at first, like asking bots to act more human by narrowing their point of view, or treating synthetic Personas as real sources of insight. But when applied correctly, this thinking unlocks entirely new ways to scale customer understanding. Today we're going to talk about how synthetic research is reshaping how we understand audiences, and how asking the right questions can make these insights feel far less synthetic and far more human. To help me discuss this topic, I'd like to welcome Mike Taylor, founder and CEO of Ask Rally. Mike, welcome to the show.
- [01:48]
Mike Taylor
Yeah, thanks Greg. Yeah, good to be here and happy to dive into this. There's a lot of opinions online about this topic, so hopefully we can wade through them.
- [01:57]
Co-Host
Yeah, definitely, definitely. And it's something of personal interest to me as well. Definitely fascinated by it. Before we dive in though, why don't we start with you giving a little background on yourself and your role at Ask Rally.
- [02:10]
Mike Taylor
Yeah, so previous to this, I created a marketing agency. It was a growth hacking agency back when growth hacking was. It was cool. And I guess you can kind of think about it as like agile applied to marketing in some ways, but. But yeah, I did that for six years. Grew to 50 people in New York and London and Europe and then I left in 2020. It was like tired of managing people who managed other people who managed other managers, you know, and I wanted to get back to like technical stuff. So I was learning how to code at that time, got access to GPT3 and was blown away. Like, wow, this is amazing. You know so, so dug really deep into that, consulted for a while, became a prompt engineer, or I've called myself that for a few years and created a book on prompt engineering that was published by O'Reilly last year. And then thanks to all that stuff, ended up starting my own business, tech business, Ask Rally, in the synthetic research space at the beginning of the year. So that's what I'm working on full time now.
- [03:18]
Greg Kilstrom
Wonderful, wonderful.
- [03:19]
Co-Host
Yeah, I can empathize with the marketing agency. I sold mine in 2017, so it's definitely fun times. But not always fun times, I guess.
- [03:32]
Mike Taylor
Yeah, we could definitely share some war stories.
- [03:36]
Co-Host
That's the subject of another podcast probably. So let's dive in here and definitely one talk. Want to talk about synthetic Persona? Synthetic research. You've used the term Persona of thought prompting and have scaled synthetic Personas into, you know, some real decision tools for brands. Maybe just to. We've, I know we've talked about this on the show before a couple times, but for those that are less familiar, why don't we start with you just, you know, defining what is synthetic research for someone who, you know, not quite familiar with it.
- [04:13]
Mike Taylor
Yeah. So the way I describe it to my 6 year old daughter is when you talk to the computer, the computer pretends to be real people and then you can use that to ask questions that you otherwise wouldn't be able to ask that many people. So she hopefully gets that. Hopefully that resonates to other people. But yeah, that's fundamentally what's happening here is you will talk to one, you know, 5, 10, 105,000% is the most, most I've had. But you'll have them role play as potential customers or potential users of your product or potential people who you're trying to reach. And you'll do traditional market research like surveys, Maxdiff analysis, usability studies, message testing. Really anything that you would normally do with real people, you can do with synthetics. And the benefit is that it's like a thousand times cheaper and faster than doing it. So even if you can't always get the exact same results you would get if you went out and did a study, we're using this a lot in areas where you just could never possibly do market research at that scale. Test every single ad that you run or explore 100 product ideas, things that you just, it'd just be ridiculously slow and expensive to do normally.
- [05:40]
Co-Host
So it's, Yeah, I mean it's, it's speed, it's scale and I would assume it's, it's Cost too, Right. Cause I mean it's expensive to do focus group research and stuff. So it's really kind of all of the, it's, it's the, the good, fast, cheap kind of the, the antithesis of that, Right?
- [05:56]
Mike Taylor
Yeah, exactly. And, and, and in some cases actually it's not even really comparable to market research because you might not even be able to do market research research in that space. Like, like you can if you're, if you're, your product is mainstream consumer, then you can go and give, give out a hundred dollar Amazon gift cards and get people to join a focus group, but they might not actually even respond in, in a way that's predictive of their consumption habits. Right. So there's all sorts of issues with traditional market research as well. And you know, quite often the people you're trying to reach, they're just not going to be swayed by that Amazon gift card. Like if your audience is like CEOs of you know, 5,000 person companies or whatever, like how do you, how do you go and get those guys into a focus group? You know, so, so, so yeah, the people are using it I think to fill in the gaps around, you know, scenarios where there is no real world comparable.
- [06:48]
Co-Host
Yeah. So for those, you know, skeptical, I mean, I know a lot of people in market research and you know, great, great, you know, super smart people and, and great at what they do. For those that are asking, you know, can the synthetic Persona really, you know, ever represent like niche human behavior? You know, what, what would you say to that? You know, what, what, what are the cases where it can work really well?
- [07:15]
Mike Taylor
Yeah. So just out of the box with the, you know, state of the art models. Today when we compare the AI results to real world studies, like we ask the same questions with the same kind of definitions of who should participate in the study. It's like 50, 60% accurate or similar out of the box. And with some testing and calibration we call it, where we optimize the responses until they get more realistic. We're seeing 70, 80% accuracy. So definitely good enough to be directionally correct. You know, it's never going to be the only thing that you do. And in fact I would say that quite often it's itself like making the business case to do more traditional research. So we worked with a big holding company agency on a credit card project and the reason why they were doing this Maxdiff analysis that we did was they were using it for a pitch. When you're doing a pitch, you can't really I mean, you run a marketing agency, it's like, it's always this trade off, like you can't really just do all the work, it costs too much money. But they want to at least see what type of work you could be doing for them in order to make the decision. So you end up having to do a bunch of work on spec and in this case they're doing the AI version on spec and then pitching for the real study that they're going to scale up. But the synthetic study can then also inform the parameters of the, of the human study. Right. So you know, some of the questions we revised after seeing that the AIs stumbled on those questions or didn't give us the type of responses we wanted. So I think that there's kind of nice symbiosis between them. I don't think it's either or.
- [09:01]
Co-Host
Yeah, yeah, that makes sense. And yeah, I mean, you know, just like there's skepticism, I would say, on the, you know, how accurate the AI is. I think the, the thing that is under underreported or not talked about as much is how unpredictable humans are and biased humans can be as well. You know, so you mentioned the, the incentive. Again, I'm not a professional market researcher, but I'm going to make an assumption that there are certain people that take incentives to answer a survey and certain people that don't. And yeah, again, it's going to completely vary based on what the incentive is and so on and so forth. But it's like there's not, not bias when you, when you ask humans. But I guess that that throws another wrinkle into it as well because how do you account for human bias in something like synthetic research and an AI?
- [10:04]
Mike Taylor
Yeah, exactly. I mean, it's always a difficult thing. And funnily enough, like in this specific AI industry, we have the opposite problem of most AI tools, which is they want to try and remove the bias and we actually want to introduce it.
- [10:21]
Co-Host
Right.
- [10:22]
Mike Taylor
Because we want our AIs to be biased in the same ways that humans are biased because we want to try and predict where they will act in certain situations. So you know, if like a really good example is everyone says they want an eco friendly car and then when it comes down to it, they buy the suv.
- [10:41]
Co-Host
Right.
- [10:43]
Mike Taylor
And you know, there's all sorts of scenarios like that where there's like a gap between the intentions that people have or what they say in their focus group versus what they actually do when they're in the market. And so if you just query the LLMs, they actually share that same bias. Right. Like they say that the eco friendly car is what they would buy. So a lot of the work we've been doing is to try and actually calibrate it towards the point where the AIs are speaking more truthfully about what people actually will do. And that's something that's quite nice because you can't do that with real people very easily. You can't just find the right prompt to get them to tell the truth. Whereas with AI you actually can keep testing until you find the truth seeking prompt.
- [11:27]
Co-Host
Yeah, yeah, well, and I mean, I think that also makes the case for what you're saying earlier, which is, you know, a mix of synthetic and real world. And I would even, you know, I would add to then the, the actual like behavioral activities that they have. Right. Like it all, it all kind of needs to get, get reconciled.
- [11:46]
Greg Kilstrom
Right.
- [11:46]
Co-Host
Because I mean, you know, at the end of the day, how do you uncover the biases that help, you know, point the synthetic in the right direction? Like how, how is, is that, is that you know, kind of observing after the fact?
- [11:59]
Mike Taylor
Yeah, what we try and do is we'll take various studies that we know are problem studies. Like, you know, our customers come to us and say, hey, I don't think this is quite right or whatever. Yeah, we were reading a lot of the research papers to see, you know, there's actually like a real deep amount of research here from social scientists because this is like a playground for them, right? Like, yeah, it's pretty amazing to be able to do this. And, and there's like, there's fewer ethical quandaries than there would be if you were trying to, I don't know, trick real people into things. We're wearing them. That's where a lot of our focus is in trying to figure out what models are better at predicting real world behavior. What prompts, how many examples do we have to give them of what realistic responses look like that sort of thing.
- [12:52]
Greg Kilstrom
Ever wonder how you could improve your marketing technology stack but don't have the time to do all the analysis? I'd like to introduce you to a new tool that we just introduced, martechalyze. Map your martech stack. Look for redundancies. Analyze monthly, quarterly and annual costs. You can even discover new platforms to replace ones that aren't performing like they should. The coolest feature is the mapping tool that lets you visualize the integrations between the tools in your MarTech stack and identify them as active, planned and even highlight platforms you plan to deprecate best of all, you can access Martech allies free and create up to two different martech scenarios for your organization with up to 15 platforms each. Or you can upgrade to Pro and get even more. Finally, there's an easy to use tool to get a handle on your Martech stack. See more and sign up for free at www.martechalize.com. that's www.martechechalyze.com finding great candidates is tough. It can be like trying to find a needle in a haystack.
- [14:01]
Co-Host
You get too many resumes and not.
- [14:03]
Greg Kilstrom
Enough candidates with the right skills or experience. But not with ZipRecruiter. ZipRecruiter finds amazing candidates for you and fast. And right now you can try it for free@ziprecruiter.com work ZipRecruiter smart technology identifies top talent for your roles quickly. Immediately after you post your job, ZipRecruiter's powerful matching technology starts showing you qualified people for it. Ditch the other hiring sites and let ZipRecruiter find what you're looking the needle in the haystack. Four out of five employers who post on ZipRecruiter get a quality candidate within the first day. Try it for free at this exclusive web address. Ziprecruiter.com work again. That's ziprecruiter.com wok ziprecruiter the smartest way to hire.
- [15:02]
Co-Host
Let's go back to kind of that symbiosis between synthetic and real world. So you mentioned, you know, this could be you do synthetic first to get some sort of directional guidance on what to do and then take it into the real world. Is that generally how, like, can you talk through? Like, how does that work best?
- [15:22]
Mike Taylor
Yeah. So you never have the right answer. Like, you never have the complete answer. Nobody's ever going to come to you and say, if you launch the product in exactly this way, it will work, right? So you're only trying to improve the odds. And there are two major things you can do to improve the odds. One is you can avoid costly mistakes. So you can avoid doing something that is predictably bad. Or you could identify untapped opportunities. And so I would say that typically you start with synthetic research just because it's cheaper, faster, and in the early days of an idea, you're just kind of looking for a steer. You're like, there's all these opportunities. Which one should I pursue? You know, I have all these names. Like, which name is interesting? You know, I have all these potential ideas to Test. And, you know, especially if you're using AI to generate ideas as well, you could have hundreds or thousands of potential paths. So a lot of the times it's just narrowing it down and you don't actually know. You can't. You can't go out and test a thousand different names for your brand. Right. You're just never going to have, like, the budget to do that. But you might be able to test 10. Right. So if AI can bring you down from a thousand to 10, then you can build the business case to say, okay, we're going to go after these 10. Or if you disagree with the AI, then it gives you something interesting to explore because you go, hold on. Well, why did I disagree with that? Like, what is it that I like about that name? And then it kind of gives you a bit more of a chance to, you know, react against what the AI is telling you. Like, sometimes our customers get better results just by, like, get having a reaction and going like, okay, like, why do I want to fight for this name? Like, what is it? That is not being said here.
- [17:05]
Co-Host
Yeah, right, right. So how does this compare to, like, using predictive analytics? So, you know, just looking at, you know, current customer available data, let's just say, but, you know, current customer stuff and, you know, running predictive models. Like, what are maybe the pros and cons of doing something like this versus predictive?
- [17:27]
Mike Taylor
Yeah. So GPTs are predictive models themselves, but they've been trained on every single variable that's online. So there are thousands or trillions of parameters for GPT4 or that class model. So you can simplify it and say, I'm going to build a predictive model. I'm going to say that, say I'm like, trying to predict what movie is going to be a big hit.
- [17:56]
Co-Host
Right.
- [17:56]
Mike Taylor
This sub I can go, okay, well, movies about vigilante orphans who are wealthy seem to do really well. Like that's, you know, Batman, Iron man, you know, like counter, Monte Cristo, Zorro. There's a whole deep bench of of movies like that. But then you go, okay, well, how come Iron man was so much more successful than the Batman franchise? It's not just because of the underlying character. In fact, for the longest time, Batman was way more successful than Iron man until Marvel got their act together and did, and now they're trying to revive Batman and make that big. So there are all these smaller variables of timing, current culture, even within the movie franchises. Some of the Batman movies, like Christopher Nolan ones, have done way better than some of the others. So at some point, you just can't really understand all of the variables, but you have access for a couple of dollars to this incredibly predictive model that is to some degree, a human brain simulator that has been trained on every human brain that's been on the Internet. They are just a very good baseline to start with. And you can add traditional predictive models into this as well and kind of weight your decision in some way. But this is like the type of model that you could only dream of as a researcher previously, and now you have it available in an API, and that's pretty powerful.
- [19:31]
Co-Host
Yeah, definitely. So, I mean, yeah, it sounds like in my experience, too, it's like the predictive. Just on a narrow set of data, you're not taking into account all of the other factors because, I mean, we're, as humans, we're influenced by lots of everything, if not lots of things. Right. So it's, it's not just a narrow set of, of characteristics in a, in a spreadsheet or something like that.
- [19:58]
Mike Taylor
Right, Exactly. Yeah.
- [20:00]
Co-Host
Yeah.
- [20:00]
Mike Taylor
And, you know, at the end of the day, they've seen, like, these models have seen how people interact in different scenarios, so they, you know, especially if you don't know that much about the customer, like, you can get a pretty good sense of who that customer is and get, get out of your own head, I think so. You know, would I listen to this model instead of Steve Jobs if he was giving me advice? Right. Or like one of the great, like, you know, product thinkers or marketing thinkers of the world? No, I'd probably wait their advice higher. But do I have access to that? No. So, like, especially if I'm on my own building my business or like I'm preparing a presentation and I don't have any resource where I can go, you know, talk to someone who has deep expertise in that field. This is a really good alternative. Yeah.
- [20:55]
Co-Host
And I mean, even if you could talk to Steve Jobs, it might be interesting to see what this also says. Right. I mean, it's. Isn't it. Because they're, I mean, everyone is, is fallible and there's, you know, there's a lot of. I think, I mean, what I'm hearing you say is, you know, it's not, it's not that there is one single source of truth and, and everything like that, but all these things are very beneficial to be able to look and compare and even to use the, you know, to prompt someone to, you know, ask deeper questions and stuff. Is that, does that. Is that correct?
- [21:27]
Mike Taylor
Yeah, exactly. You can iterate, whereas you can't really do that very easily with a focus group. Right. Like you have to schedule it, you know, talk to people, you know, or surveys. Like wait till, wait till the responses come in and then you're like, oh, I wish I would have asked this question. You know, you had to kind of follow up. Yeah, like one of the things we find is people will take the results of the research they've done already and then create Personas based off that. So like we did 126 customer interviews when we launched and then we have that as like an audience in rally that we ask questions about when we're deciding what to build. So, so, so like it's a kind of a way of continuing that focus group.
- [22:08]
Co-Host
Yeah, yeah. So can you talk a little bit about, about Ask Rally and you know, what, what, how does it work and things like that?
- [22:15]
Mike Taylor
Yeah, so it is basic core level, similar to ChatGPT, except you're chatting with many GPTs at once. So you put out a question, you get a hundred random responses, for example, and you can create those Personas. So you can go in and define your audience and then generate the Personas. We also have a bunch of pre created Personas for you to try in different fields. Like we have one that matches U.S. census data, for example. Like if you're going after the US market and then you can just ask questions, you ask follow up questions. Or you could also do voting as well. So you can do, do polling and say like, okay, which of these options would you choose? You can upload images and videos specifically with the Google models. So we have Google Gemini, we have OpenAI's models, we have some open source models and the anthropic models as well. So yeah, people can switch between them.
- [23:13]
Co-Host
That's great. So as we wrap up here, a couple things for you. What do you think is the biggest misconception or misunderstanding about, you know, synthetic research? Synthetic Personas, you know, good, good or bad?
- [23:27]
Mike Taylor
Yeah, I would say that a lot of the misconceptions about synthetic Personas are not specific to that industry. Either you believe in AI and you use it for everything or you see it as like a threat or a scam. And like there's no way I'm going to convince you other way. Right. Because like I'm, you know, it's not my job to convince people of that. Like they'll figure it out eventually, you know, but, but like I think that you have some people in the industry who have taken a stance against AI generally Speaking like, because, you know, they're annoyed that people's jobs might be at risk, which is, I think, perfectly valid, like response. Or they don't like the copyright angle, the training on copyrighted material or, you know, for one of. Or just like, you know, they've, they made an offhand comment at a conference and people responded well and they thought they're going to make this my whole personality to hate on AI. Right. You know, and that's fine. And you know, at the end of the day people have to make their bets. But, but I think like, if you already believe in the power of AI and like you're using, you know, ChatGPT or Claude or whatever to get feedback, then you just kind of understand this intuitively and you're going to use it. But almost everything that they could complain about with, with AI, like, is also a problem with humans too. That's what I find. So like, whenever I see, yeah, like, they go, oh, what about the hallucination? I'm like, have you talked to like a consumer? Because they bring stuff up all the time.
- [25:07]
Co-Host
Yeah, right, right, Yeah, I love it. Well, last question for you. I like to ask this to everybody on the show. What do you do to stay agile in your role and how do you find a way to do it consistently?
- [25:20]
Mike Taylor
Yeah, good question. I came from an economics background, so, you know, when I was like studying in school, we were learning all about like, Japanese manufacturing was like the cool thing at the time to learn and so. So like that was like ingrained into me like Kanban boards and all that stuff from the very beginning. So I would say I'm an agile native in that respect. That's always just been the way I operate and I don't know if I'd be able to operate in any other way. It's probably why I've navigated towards startups and stuff rather than waterfall processes. So it's almost like I'm like a fish in water and I can't describe the color of water, everything I see. But yeah, very specifically, I would say one of the things I like to do is think about how my beliefs have changed with new information. So like a Bayesian kind of way of thinking is how you might describe it. So I never say yes or no. I always kind of adjust mentally. I think about, I'm adjusting my probabilities. So like, if I run a test and something fails, I don't say I'm never going to do that again, I just say I'm less likely to try that next. Right. Now the other things that I could have done just like got a little bit higher in the list and that dropped a little bit lower. So I think if you think that way gives you the mental agility, I think, to to contradict yourself and you need to be able to do that when you're operating under uncertainty, which is, you know, the case when you're building an AI startup.
- [26:53]
Co-Host
Yeah, yeah. Love it. Well, thanks so much for for joining again. I'd like to thank Mike Taylor, founder.
- [26:58]
Greg Kilstrom
And CEO of Ask Rally, for joining the show.
- [27:01]
Co-Host
To learn more about Mike and Ask.
- [27:03]
Greg Kilstrom
Rally, you can follow the links in the show notes. Thanks again for listening to the Agile Brand brought to you by Tech Systems. If you enjoyed the show, please take a minute to subscribe and leave us a rating so that others can find the show as well. You can access more episodes of the show@theagile brand.com that's theagile brand.com and contact me. If you're interested in consulting or advisory services or or are looking for a speaker for your next event, go to www.gregkilstrom.com that's G R E G K I H L S t r o m.com the Agile brand is produced by Missing Link, a Latina owned, strategy driven, creatively fueled production co op. From ideation to creation, they craft human connections through intelligent, engaging and informative content.
- [27:53]
Co-Host
Until next time, stay curious and stay.
- [27:55]
Greg Kilstrom
Agile.
- [28:01]
Mike Taylor
Brand.
- [28:06]
Tech Systems Representative
Before we continue, I wanted to share a key strategic resource that a majority of the Fortune 500 are already aware of. Finding the best technology, business and talent solutions is not easy. With business demands and competition, competitive pressures mounting, you need to be able to design, deploy and optimize your technology to provide leading customer experiences while driving business growth. Those of you that have been listening to this show for a while know that this podcast is brought to you by Tech Systems, a global provider of technology, business and talent solutions for more than 80% of the Fortune 500, Tech Systems accelerates business transformation for their customers. Whether you're looking to maximize your technology ROI, drive business growth, or elevate customer experiences, TechSystems enables enterprises to capitalize on change. Learn more at techsystems.com that's teksystems.com now let's get back to the show.