← The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
#550: The science of CX with Martha Brooke
00:24:04
Loading summary
Transcript53 lines
- [00:00]
Martha Brook
Mom, can you tell me a story?
- [00:02]
Carvana Mom Narrator
Sure. Once upon a time, a mom needed a new car.
- [00:05]
Martha Brook
Was she brave?
- [00:05]
Carvana Mom Narrator
She was tired mostly. But she went to Carvana.com and found a great car at a great price. No secret treasure map required.
- [00:12]
Greg Kilstrom
Did you have to fight a dragon?
- [00:13]
Carvana Mom Narrator
Nope. She bought it 100% online from her bed, actually.
- [00:17]
Martha Brook
Was it scary?
- [00:18]
Carvana Mom Narrator
Honey, it was as unscary as car buying could be.
- [00:20]
Martha Brook
Did the car have a sunroof?
- [00:22]
Carvana Mom Narrator
It did actually. Okay, good story. Car buying you'll want to tell stories about. Buy your car today on Carvana. Delivery fees may apply.
- [00:32]
Marine Corps Narrator
The wrongs we must right. The fights we must win. The future we must secure together for our nation. This is what's in front of us. This determines what's next for all of us. We are Marines. We were made for this.
- [00:55]
Martha Brook
The Agile Brand.
- [01:03]
Greg Kilstrom
Welcome to Season six of the Agile Brand where we discuss marketing, technology and customer experience, trends, insights and ideas with enterprise and technology platform leaders. We focus on the people, processes, data and platforms that make brands successful, scalable, customer focused and sustainable. This is what makes an agile brand. I'm your host, Greg Kilstrom, advising Fortune 1000 brands on martech, marketing operations and CX. Best selling author and speaker. The Agile Brand Podcast is brought to you by Tech Systems, an industry leader in full stack technology services, talent services and real world application. For more information, go to teksystems.com before we get started, I wanted to let you know that my latest book, Priority is seven Principles for Better Strategies, Decisions and Outcomes is now available. In it, I give ideas and insights for leaders and teams that need to make meaningful progress on their priorities. After all, our priorities are what we do, not what we say we'd like to do. You can find Priority as Action on Amazon or learn more on my website greggkilstrom.com now let's get on to the show.
- [02:11]
Greg Kilstrom (Interviewer)
While we talk about customer experience a lot on this show, today's focus is
- [02:15]
Greg Kilstrom
going to be a little different than
- [02:16]
Greg Kilstrom (Interviewer)
some of our past conversations. Today we're going to talk about adding science to customer experience programs and more specifically, the science of CX surveys. To help me discuss this topic, I'd like to welcome Martha Brook, Chief Customer Experience Analyst at Interaction Metrics.
- [02:32]
Greg Kilstrom
Martha, welcome to the show.
- [02:33]
Martha Brook
Hey Greg, I'm glad we could do this.
- [02:35]
Greg Kilstrom (Interviewer)
Yeah, absolutely. I love this topic and looking forward to it. Before we dive in though, why don't you give a little background on yourself and what you're currently doing?
- [02:45]
Martha Brook
Sure. So my job is founder and like you said, chief Analyst at interaction metrics. That's interaction like what we're doing here and interaction and metrics like the number. And what I do is I oversee the research and analysis this phases for clients like convergex and Yaskawa America and California State Bar. So my, my key, key role is to ensure we hold projects to the highest levels of science.
- [03:09]
Greg Kilstrom (Interviewer)
Yeah, great, great. So you know, certainly there's a lot of measurement, there's a lot of theory, there's a lot of practices with customer experience and customer experience programs. But you know, we're here to talk a little bit about the, the science of it. And so, you know, are CX programs scientific? Are they not? And if not, why not?
- [03:35]
Martha Brook
Well, Craig, that's a very big question with a very big answer.
- [03:39]
Greg Kilstrom (Interviewer)
I know, I know. I give you the easy ones.
- [03:42]
Martha Brook
Okay? But if programs were scientific, we really expect that NPS and AXI scores would be quite a bit higher. In other words, what would be happening is we would all routinely be having good experiences. I guess another way to say it is that customer experience, customer experiences would work. Right. And the best metaphor that I know of is really sort of medical based. Like good seizure medications that are backed by evidence routinely result in patients having fewer seizures. Likewise, if customer experience measurement were really good, we'd routinely expect no matter where we were, that we would have good experiences. So I guess there are some other sort of ancillary backup to why I believe science is not being practiced as much as it is, could and should be. One is that, well, now we like the net promoter question and almost all of our clients want us to use it. That said, it wouldn't be as prolific as it is. It would be used more gingerly, for lack of better words, it would be used as Bain intended it to be used. Which is to summarize now how you feel about the company and not at every single touch point. For instance, I have an interaction with the bank of America call rep. Based on that interaction. I'm not likely or not likely to recommend bank of America. I'm not here to pick on bank of America routinely. Companies are asking it at every touch point kind of willy nilly and it's just not, it's not the way NPS was intended and it's really not a scientific approach. And also, then again, big topic, big, big answer. But I would say that companies tell me all the time they have very low response rates and they tend to hear from very particular kinds of customers, maybe those with more time on their hands. At a conference I spoke at recently, a Company was complaining it's only older customers we hear from, but that's not the entirety of their customer base. And so good science is representative response. So if you're only hearing from a certain kind of customer, well then you're not really getting the full ante of who your customers are. So that's just a little bit about why I believe customer experience programs are not held in general. I mean, ours are, but in general are not held to the highest levels of science.
- [06:32]
Greg Kilstrom (Interviewer)
Yeah, yeah. So given that, I mean, what are a few things that could be done, I mean, to make programs more scientific in their approach?
- [06:45]
Martha Brook
Well, the one thing that I talk about all the time is that companies would work very, very hard to remove leading constructs from their surveys. Right. So a leading construct is one that directs the customer toward an answer you want to hear. So how satisfied were you with xyz? Well, that assumes the customer was somewhat satisfied, right? Yeah, so that's a problem. Nps, I would argue, has bias, like how likely are you to recommend it? It does assume the customer is somewhat likely to recommend, but in any event, and yet again, we use NPS because it's a good benchmarking question when it's used properly. So there'd really be a team approach to scour surveys for anything that's leading customers toward what you want to hear. Customers have priorities. In other words, it's not all equal. And the goal of any customer experience program, and surveys in particular, is to really come up with an accurate measurement of customer experience. So if every question is of equal importance, then you're not really capturing the nature of the customer experience. Right. If some things are more important than others, you have to include that weighting factor. So that weighting factor can be based on asking customers to rate like what is most important in this experience and then using that as the weighting factor. Or we sometimes use correlation analysis to determine a weighting factor. But that's a really important aspect of survey design and survey calculations. And then I would edit surveys for just the bundle of usability flaws that lead to gibberish data. So, you know, that's all there. You know, now, now you've got me on really a topic that I could go on for hours. But you know, examples are like double barreled questions. That's where you ask two things at the same time. So was your server efficient and courteous? Well, what, you know, what are you asking?
- [09:02]
Greg Kilstrom (Interviewer)
So if there were only one of them.
- [09:04]
Martha Brook
Yeah, right. So, and often those are at odds with each other. So you get information, but it's gibberish information. So that you know, the customer is just like, I don't know, eeny, meeny, miny, moe, or insufficient answer options. I think about this all the time because I'm a huge Amazon user. It's just the easiest way in the world to buy stuff. But then because I'm a huge Amazon user, I'm a huge Amazon returner. And so the list of options is never. It doesn't include why it is. It's like, I didn't like it. They don't include that. It's like the website was. Website description was wrong. I almost always pick that, like, okay, I guess the website description was wrong. That's the closest thing to I didn't like it. But whenever there are insufficient answer options, you're going to get gibberish information. Right. Or one of my favorites is not allowing for anonymity. Because if you don't allow for the option for anonymity, you're going to omit a whole group of respondents. That can be as many as 40% of respondents. Just if you're going to name me, which now that seems like now you're going to hassle me if I give you a low score. I'm not going to take your survey. But their data is as important as those who name themselves. I'd say possibly more important. One that we already talked about is using NPS when it just doesn't make sense when the rating scales are off or there's no zero. So we see this in reviews all the time where customers will write, well, the choice was one star, two star, three star, you know, up to five stars. Really. If you'd given me zero stars, that's what I would have given. Like, yeah, you know, zero is. I mean, one assumes that you're somewhat satisfied in a sense. Right. So really the better scale is 0 to 10 or, you know. Yeah, I would say internal language. We see gibberish from that all the time. Like when we review customers surveys and they ask, well, what do you see? Do you think this is ready to go? Cause we do free audits of surveys and we'll say like, you know what, you're in a good place. You don't really need us. We're happy to say that. Or, well, actually, this is not very scientific. Here's some things you want to consider. So in any event, when companies submit their surveys, we'll often see all this kind of internal language. Questions about design, white space, balancing things that you can't expect customers to Know, so they'll just kind of eeny meeny miny, moe. So those are some examples of ways that companies are collecting data. But not all data is good data. So it's sort of gibberish data. And I just, you know, you just hope they're not making business decisions based on that.
- [12:05]
Greg Kilstrom (Interviewer)
I'm sure there's lots of different causes of this, but I mean, you know, a few things. It would seem as a consumer myself, it would seem that it's actually very easy to send surveys. Not, not as easy to construct well constructed surveys. But you know, it, it seems pretty simple to put one together in, you know, in the scheme of things. So it's like, is some of this just. Well, you know, we, we want some information like your, your example about the white space or whatever. So some designer somewhere on of a website is like, you know, I want to know this answer to this very specific question. Not very scientific, not very customer friendly, let's say to, you know, to ask such a kind of a niche question. But you know, is, is some of this just because the tools are so easy to use and there's no, there's, there doesn't seem to be other mechanisms or, you know, why are we getting so many of these surveys? And yet. So the quality is so low.
- [13:03]
Martha Brook
I guess because anybody can buy Photoshop.
- [13:07]
Greg Kilstrom (Interviewer)
Yeah.
- [13:08]
Martha Brook
Doesn't make everybody a designer.
- [13:10]
Greg Kilstrom (Interviewer)
Right, right, right.
- [13:13]
Martha Brook
I think that's, that's the sort of obvious. Maybe it's even a facile answer. I think the deeper issue could be sort of a lack of awareness of science and a lack of awareness of what good data is. And maybe, maybe it's become just a task. Everybody's like, task, I did it, you know, check, check, check the box. And yet really collecting. There's could be nothing, really nothing for any company that's more important than customer listening. I mean, honestly. Right. What could be more important than that? And yet maybe there's kind of a company centricity where they don't really want to know. Like that's sort of a psychological thing. Like maybe they don't really want to know. In some cases they just want to check the box and.
- [14:08]
Greg Kilstrom (Interviewer)
Well, I do feel like in some cases it's like the, the job of CX is to send surveys. I wouldn't say like the, the seasoned CX professionals out there that know what they're doing, like that's their job is certainly not just that, but there are literally people in companies that, that's their job is cx and it's to send surveys. And I, I think to your point, it's. If it is, it requires some more education and some kind of. And I know we're, we're kind of relegating our conversation to surveys. You know, if we open this up to leading, lagging indicators, all that kind of stuff, then it becomes a very, probably even more unwieldy conversation to talk about how those things tie into each other. But if your only tool is a survey, I guess you're going to use it for everything, right? Whether it's, whether it fits or not.
- [15:02]
Martha Brook
Right, Right. Well, there's that and I guess just kind of picking up on what you were saying. The discipline of CX has many methods at its disposal. Surveys simply happen to be the least expensive of those methods. But yeah, there are all kinds of methods. Like, you know, we do customer service evaluations at statistically valid levels. There are customer interviews. Those can also be done at statistically valid levels. So there are other methods outside of surveys that also should be held to the standards of science.
- [15:37]
Greg Kilstrom (Interviewer)
Right.
- [15:38]
Martha Brook
And so, you know, I think the. Maybe, Greg, it's possible the discipline of customer experience is so new that it really hasn't absorbed the science message yet. Like if when medicine first came on board thousands of years ago, I can't say that it was very scientific. Wasn't it like bloodletting or leeches?
- [16:03]
Greg Kilstrom (Interviewer)
Yeah, yeah.
- [16:04]
Martha Brook
You know, so sometimes a dissonance discipline comes on board and it takes a while for it to really catch up to science, which is important. I mean, what we determined in the Renaissance was it really is the best way to understand the world. And so by extension, it's the best way to understand customer experiences.
- [16:24]
Greg Kilstrom (Interviewer)
Yeah, yeah, right.
- [16:25]
Martha Brook
Like it's better than conjecture and belief and it really is the best way we know of to understand what is, what the nature of the world is.
- [16:37]
Greg Kilstrom (Interviewer)
Yeah, yeah, I like that. Yeah. So, you know, moving, moving ahead a little bit, you know, it's. We got to talk about AI, so we're going to. So, you know, how does, how does AI factor into this? You know, we're talking about surveys and cx. Like, is it, is it going to help us? Is it going to hinder us? You know, where do you see that?
- [17:02]
Martha Brook
First of all, I love ChatGPT. We actually have our own ChatGPT engine. So fully bought into AI. Actually, it really should be called large language models right now because intelligence is not where it's at. So it does a lot of great things. So that's the first thing I'd like To say, but it doesn't write surveys, so do not use it for that. Really don't use it for that. It's a large language model, so it's just combing what other surveys are doing. And most surveys are not being held to a scientific standard. Okay, so don't use it for writing surveys. Now, another very common thing that people do with AI is they use it for analysis. And with quantitative information, it just doesn't work. Because the kind of analysis that you want to do for quant is what we call segmentation analysis. That's where you're comparing different populations sort of side by side. Say you have OEMs and distributors and end users. You want to be able to compare each of those populations and how they're responding to each survey question. And so that's a little too complex for any kind of analysis that AI can do right now. Now, another way that companies use AI is for their text analysis. That can be great. But again, remember, right now, AI is not general intelligence. It's a large language model, which means it really has to be trained to be effective. So it's not. Yeah, you can just throw. And we do, we experiment with mostly chatgpt, but some of the other large language models too. Yeah, you can put the text into one of those engines. But in General, especially for B2B, it's just not pulling out the nuances that you need. And sometimes it's hallucinogenic. It's coming up with stuff that sounds really great. Yeah, you know, it's like, we did it recently. And I was like, oh, wow, this is amazing. And then I was like, wait a minute, let's like, really read this. And it was wrong.
- [19:22]
Greg Kilstrom (Interviewer)
Yeah, yeah.
- [19:23]
Martha Brook
You know, it looked really good. Like these were good sentences. And really. And I was, I was just like, oh, done and done. And no, no, no, no, no, no. So it can be quite misleading. That said, AI, very useful if it's trained. So you need researchers working side by side with AI. It's easily confused, even with sentiment, which is the easiest part of text. So sentiment is like, are they happy? Are they sad? How do they feel? But a sentence like love your company in particular, but your customer service is a real hardship. Well, many AI solutions are not going to rate that for what it is. That was a mixed comment. That was not. They're going to say, go with that first phrase. Oh, they're a positive. Love your company. That's not actually what they said. And that's very simple. Sentiment is very, very easy. What's More complex is finding the meaning, the emergent themes, what customers are actually talking about. And so that's much more difficult than sentiment. And so even sentiment has its problems. But, okay, let's put that aside. And what's more important almost is what are customers talking about and how are they thinking, what are the Topics? And so LLMs can be a useful kind of side by side with researchers, but they really do need to be trained. I hope that wasn't too shaggy. Shaggy, Whatever they say Shaggy dog and answer shaggy.
- [21:02]
Greg Kilstrom (Interviewer)
No, no, it's, I mean, we're, yeah, it's still, we're. I feel like we're, I mean, AI has been around for decades, but I feel like we're in early days of this, this wave of, of really using it in these ways. So there's a lot to, there's a lot to do. There's a, I mean, there's a lot of opportunity, but there's a lot to also kind of unpack and, and really understand. And I think, I think, you know, one thing that you touched on was just how it's, it can be really helpful to use AI, but it needs humans to make it better. Just like we can use AI to make us better. It's, you know, it's kind of, it goes both ways. So I think that's where, you know, maybe someday, to the general AI point, like, maybe someday it won't be that, but for the time being it's, you know, it can be really powerful when, you know, either it's a first draft or a second draft or something like that, but it's part of the process, not just some kind of end goal.
- [22:00]
Martha Brook
Right, right. I mean, so the social science way of dealing with text is to do what we call coding the data. And so, so that's where you have a team of researchers, not one researcher, because you need a team to come up with this is verifiable, replicable results. And you go through and tag comments within set protocols and then you can sometimes compare that against large language models and use that to train large language models. But there really are techniques for unpacking what's in a conversation or in a body of text. And these are important, proven techniques.
- [22:51]
Greg Kilstrom (Interviewer)
Yeah, absolutely. Well, Martha, thanks so much for joining. One last question before we wrap up here. You've given a lot of great advice and insights already. But for those that are listening here, know that they need to inject a little more science into their CX programs. What's one piece of advice to where could they start?
- [23:13]
Martha Brook
I think they could take a day and just study the principles of science that so random selection, controlled experiments and then see what of that applies to their survey. You know, I think that would be a day very, very well spent. And you know, of course feel free to reach out to me on LinkedIn or website or however you like to chat because I'm truly always open to that conversation about how do you get evidence based quality data driven information about the customer experience.
- [23:51]
Greg Kilstrom (Interviewer)
Yeah, that's great. Well again I'd like to thank Martha Brook, Chief Customer Experience Analyst at Interaction Metrics for joining the show. You can learn more about Martha and Interaction Metrics by following the links in the show notes.
- [24:04]
Greg Kilstrom
Thanks again for listening to the Agile Brand brought to you by Tech Systems. If you enjoyed the show, please take a minute to subscribe and leave us a rating so that others can find the show more easily. You can access more episodes of the show at www.gregkilstrom.com. that's G R E G K I H L S t r o m.com While you're there, check out my series of best selling agile brand guides covering a wide variety of marketing technology topics. Or you can search for Greg Kilstrom on Amazon. The Agile Brand is produced by Missing Link, a Latina owned, strategy driven, creatively fueled production co op. From ideation to creation, they craft human connections through intelligent, engaging and informative content. Until next time, stay Agile.
- [24:55]
Martha Brook
Brand.