← The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
The Agile Brand with Greg Kihlström®: Expert Mode Marketing Technology, AI, & CX
#685: Using AI in UX research, design, and testing with Jason Bowman, The Office of Experience
00:22:24
Loading summary
Transcript48 lines
- [00:02]
A
And Doug, there's nowhere I wouldn't go to help someone customize and save on car insurance with Liberty Mutual. Even if it means sitting front row at a comedy show.
- [00:12]
B
Hey everyone, check out this guy and his bird. What is this, your first date? Oh no. We help people customize and save on
- [00:18]
A
car insurance with Liberty Mutual. Together. We're married.
- [00:20]
B
Me to a human, him to a bird. Yeah, the bird looks out of your league.
- [00:24]
A
Anyways, get a quote@libertymutual.com or with your local agent.
- [00:27]
B
Liberty Liberty Liberty Liberty.
- [00:31]
A
At Strayer University we help students like you.
- [00:33]
B
Go from Will I To why not? For over 130 years we've been innovating higher education to make it more affordable,
- [00:41]
A
accessible and attainable so you can reach your goals.
- [00:45]
B
Go from thinking can I? To Yes, I Can and keep striving. Visit Strayer. Edu to learn more. Strayer University is certified to operate in Virginia by Chev and its many campuses, including at 212115 Street north in Arlington, Virginia. The agile brand. Welcome to Season seven of the Agile Brand where we discuss the trends and topics marketing leaders need to know. Stay curious, stay agile and join the top enterprise brands and martech platforms as as we explore marketing, technology, AI, e commerce and whatever's next for the omnichannel customer experience. Together we'll discover what it takes to create an agile brand built for today and tomorrow and built for customers, employees and continued business growth. I'm your host Greg Kilstrom, advising Fortune 1000 brands on martech, AI and marketing operations. The Agile Brand podcast is brought to you by Tech Systems, an industry leader in full stack technology services, talent services and real world application. For more information go to teksystems.com to make sure you always get the latest episodes, please hit subscribe on the app you listen to podcasts on and leave us a rating so others can find us as well. Now on to the show. This episode is brought to you by the Office of Experience, a design driven digital first vertically integrated and collaborative agency that believes in the power of ideas and the strength of people. Learn more@officeofexperience.com Agility requires finding ways to stay one step ahead of the competition as well as in anticipating customers needs. So how does a brand maintain the speed and agility in the area of UX design where it has often taken a considerable amount of time, effort and testing to get to a better result. Today we're going to talk about using AI strategically in UX research, design and testing. Tell me to discuss this topic I'D like to welcome Executive Director of Experience Strategy at the Office of Experience, Jason Bowman. Jason, welcome to the show.
- [02:55]
A
Thanks. Thanks for having me.
- [02:57]
B
Yeah, looking forward to. And welcome back to the show, I should say as well.
- [03:01]
A
Yes, yes.
- [03:02]
B
Been here before. Looking forward to talking again. For those that missed when you were here before, why don't we start with you giving a little background on yourself and your role at the Office of Experience. Sure.
- [03:12]
A
So I've been in the digital making practice since the early.com 2.0.0 day. 1.0 days. I've worked everything from agency to in house as like a team lead to startups, you know, mobile, the whole thing basically. I like to say if it's been digital, I probably built a version of it at some point in my life and been at OX for about four years. Started leading the UX team and now I manage the content strategy and the business analyst team as well. Just kind of a nice way of taking that strategy all the way through from concept content and need, all the way through execution, through development. Just kind of keeping that thread tight.
- [03:56]
B
Yeah, love it. Well, yeah, let's dive in here. And so we're going to talk about a few things today, but I want to start by talking about how artificial intelligence is impacting the user experience practice in a few different ways. So first let's start with talking about how AI is impacting research. What are you seeing there?
- [04:17]
A
AI is huge in research. What I find is really speed up the process. So before teams would run to Google, they'd be searching through a lot of things. You'd be looking at best practice documents, comparing a couple of different people's best practices, reading up on things. And what AI is helpful is it starts to either a get you started in one direction, summarize something, or just kind of like even it's getting so good that now it's even starting to analyze things. We started doing some things such as heuristics, where hey, here's a site, here's our framework. You know, it asks a few questions and it starts to spit out results that are about 80% of what we need. So it's really been fun as we've gotten deeper in the agency of training the engine and training the agent to be what we need by kind of creating scripts for it and going through. So that's great with heuristics Personas, it's great at giving you kind of rough ballpark Personas, but also by allowing it to feeding it Personas that you've already created and Then reacting to some of the work or some of the ideas that you're presenting and how it might be better or worse. So that I think we talk a little bit of this later, but it starts to talk about how some of the testing, some of the validation can happen early on. And, you know, even just overall content strategy, we found like, hey, we've got this kind of a site, or here's this kind of a page. What are some recommendations on versions of content strategy? Where might I lean? This way, it kind of allows you to test out ideas. And we tend to use Claude a lot at the Office of Experience, and with that, it can even start to code up a quick wireframe, give you a visual reference. It just kind of lets you play with some things quicker. We always, we, we encourage the use of Claude, the Internet, anything really, to help you create that thing. The only kind of rule we have with our team is make sure that whatever insight you get, make sure you agree with it, you validate it, and you can stand behind it.
- [06:23]
B
Yeah.
- [06:24]
A
Not just say like, because it's like, you know, if you, before AI, you might go to a competitor's website and you could pick up their landing page and design it just like that. And if I ask, why is it this way? Or the client asks, why is it this way? We simply say, well, because they had it right. That's not a good answer. And that doesn't help your case. The same rule applies to AI. It's like, well, why did you do that? Well, the AI gave it to me. Well, again, that doesn't really matter. It has to. You have to ingest it and apply it to your scenario and have it make sense.
- [06:57]
B
Yeah, yeah, yeah, it makes sense. And I mean, even, you know, to your point about AI getting something 80% there or even 50% there, you know, what, whatever it is, it's, it's getting things further, quicker. So the, the key part as well is so that a human can validate it and make sure, you know, leveraging their expertise as well. So, like, I like that dynamic of, you know, let's use AI to get us further, quicker, faster, but let's use the humans to do what humans do best as well. Right, right.
- [07:31]
A
Because if you can get what would take a half day of research and you can kind of compile that into 15 minutes now, it gives you that kind of window or some pathways to do deeper research, maybe to ask, follow questions or dig in something deeper, but you've been able to consolidate and gather more information of much quicker time. It's super helpful in that way.
- [07:54]
B
Yeah, absolutely. So let's talk about the area of predictive validation. And maybe for those a little less familiar with that term, why don't you start with defining what predictive validation is, but also talk a little bit about how AI can help there.
- [08:10]
A
Sure. So in user experience, one of the best ways to learn is actual users. We spend a lot of time creating, but then it's really about understanding how what you created actually applies. Because users are fascinating beasts of breaking things or doing what they're not told, or sometimes you just end up drinking your own Kool Aid and you think that everything you're making is amazing and users tell you different. So what predictive validation does me is it allows us, before we go all the way to publishing or testing, or maybe even before we run a full user test, you're able to just gut check things with that predictive validation. So you feed an artifact to AI, ask it some questions, it feeds it back to you. We talked a little bit about earlier, earlier about that with Personas, how you can say, hey, here's this object based on these Personas, how do you think they're going to react to this content A or B or see this valuable or think this headline, it can start to, to gut check. The funny thing about that was I was talking to one of the coworkers, my coworkers, like, so what happens if, you know, we need to start building in a little more time for those internal cycles? And they're like, oh well see with AI you just change the script until you get the answer you want. So it can be good, but as we, we can also manipulate it far easier. Right. But then with virtual audience testing, same thing with Personas. And then heat maps are really handy because you know, heat maps often tell scrolling and click and eye tracking and you can some really easy to, to shoot up a comp, a wireframe or something and just kind of get a sense of what's where your contrast levels are, where the eye might be drawn. And it doesn't take the place of user testing, but it really does kind of help validate some things internally.
- [10:04]
B
Yeah, yeah.
- [10:04]
A
And then even when doing testing kind of in back to the research, it's great. And a lot of the testing sites already have this engine built in, but if you've got a bunch of recordings, you have a lot of surve great for data crunching, finding trends in large amounts of data, just spitting out roughly what it should do. So again really time saving and compiling and doing Some of those things, yeah.
- [10:27]
B
And I mean, this is what I'm hearing as well, you know, when I talk with, with brands is, you know, it doesn't, they're still doing human research, but what they're doing, they're getting better things to put in front of those humans by doing, you know, synthetic, by doing AI generated research and prototyping and all that stuff. So in other words, their, their results are better because they're, you know, they're 80% of the way there, you know, kind of, kind of to the earlier point. So it just, it not only, you know, all, all of the efficiencies of not having to take a human's time to do all the mockups and the research and all that stuff, but it's also enhancing that, that speed to getting the best solutions. Can you talk maybe a little bit more about that as well?
- [11:11]
A
Sure. I mean, AI, like any kind of machine or something like that, or even just user groups, they're never going to be 100% right or perfect or anything like that. They're going to have kind of an inherent bias. But if we understand that inherent bias and we're using it to help generate content, we know there's a bias, then you can still trust it to test it with that same bias. So you take that in consideration. But it really is that way of, I mean, just for example, so let's just say that we have a disagreement internally or maybe even with the client. But I'll just say internally, it makes it easier. I think something should be one way. Maybe the creative director thinks something should be another way. And there's a little bit of a debate and we don't really know who's right and no one really has the authority to overrule each other. So hey, why don't we just run this through this real quick and see and then you just kind of quickly what we used to do. Well, we'll set up an A B test. All right, let's get the prototypes. Let me spin it up. You could probably set a good A B test up in a couple hours. Then you wait a couple hours to get the results back. Then you got to read through it and you basically spend a day to validate something. This you can probably validate in five minutes. You know, if, you know, kind of, if you're good with prompts or you've prompted enough and you know, hey, here's this, here's this. What questions do you have for us in order to get some validation? Respond that. And then it can Kind of give you guidance and then give you some reasoning behind it. And it just takes some of the bias out of it. Right. It's just doing a general knowledge. Here's what this thing thinks can be wrong and you can override it just like any sort of test, in your opinion. But it really is helpful in that way.
- [13:00]
B
Let's talk a little bit. You know, now that just about everybody, if not everybody, has been using AI to some degree, I assume most people listening to the show have used something at least if not many things. You know, we've all seen the good and some of the not so good. You mentioned some things like, you know, hallucinations, other things like that are possible certainly, but in the good scenarios, AI can be thought of as that collaboration partner, much as you've, you've described so far. So let's talk a little bit about how UX needs to participate with AI to get the best results. And maybe we'll, we'll start with a negative and then we'll, we'll get a little more positive here, but start with the not so good. You know, AI is prone to some odd behaviors like things like hallucinations and other things. What should UX teams know when working with, with AI?
- [13:54]
A
I think we should just remember what the Internet taught us. Like. Right. You just can't believe it. Like you, it's a great starting point, but if it cites a study, you should make sure that it's really citing a study. It didn't just make up a study to cite. You know, you should double check if a fact seems really interesting, make sure you have the source. Right. Or you know, like if it's a study or research, you validate that with some other sources. So that's, that's just super important. I, you know, as we said, you could manipulate it. There's a chance it's feeding you too much what it wants with your prompt. We were joking. Sometimes like if Claude was an actual person in the office, they, they do anything for you. Hey, Claude. Yeah, I love to do that. You know, let me go get you some coffee. And then they come back with like a milkshake and you're like, right. And we're like, well, thank you. This is so handy. I didn't even know I needed this. So.
- [14:45]
B
Right, right.
- [14:46]
A
You know, really kind of just validating it and, and also discipline yourself in that it's not a magic genie. And just like all computers, it needs data and good data to run. You can't just say, tell me my five year projection and not give it any source material. It's going to be like, well, you know what? I don't know what you're saying. So the discipline of even creating a good prompt, asking good question, having an idea what the result should be, and just kind of guiding it down that path is a great way to get the most out of it. But you just, you have to validate. You have to know that it is still learning.
- [15:27]
B
Yeah.
- [15:28]
A
It's not a magic button.
- [15:29]
B
Yeah, yeah. And, you know, in, in addition to, I mean, certainly, you know, a lot of leaders, managers, you know, just companies in general are under a lot of pressure to adopt AI for the, you know, the speed and efficiency things. They're also under a lot of pressure to innovate. Right. And so this, I think this is a, this is an interesting topic here with, with AI, you know, is. Is it really coming up with visionary ideas or is it the humans that are coming up with ideas? You know, what, what role do. Do humans and AI play in innovation?
- [16:02]
A
I think sometimes people use AI in place of innovation, like we're using AI. And that's your innovation button. So there's that. I think, you know, there's also a lot of, we kind of call it the snake oil right now of everyone's like, oh, well, I've got an AI engine that can take a creative campaign from concept, write the brief, build off of the brief, and deliver final copy. It's like, that sounds too good. Sounds great.
- [16:30]
B
Yeah.
- [16:31]
A
But I think innovation still happens. I think AI tends to take all the knowledge it can get, find the themes and build off of that. But sometimes, much like best practices or much like group think or pop music, you could say, you know, it just becomes what people want. And sometimes that's great, but it's not always pushing the boundaries of a new. If you take music, for example, like a new genre, it's not. Someone's not experimenting. AI is not going to necessarily experiment on its own based on your prompt.
- [17:08]
B
Yeah.
- [17:09]
A
You know, and so, you know, asking it, show me something new, it might not really do that. I think there's still that benefit of, of that we're able to do because again, it builds off prompts. And in our heads and in our collective groups, we have hundreds of thousands of prompts that we don't even know exist. Things that we saw 10 years ago.
- [17:32]
B
Right.
- [17:33]
A
A movie I watched, that somehow there's an action in there that makes me think of this, and now I think of something different. And so I think that's where not that Humans are better, but we are still necessary to innovate because I think we don't know all the prompts that we might have. So our brains, you know, we just can't get the AI to act like us because it doesn't have all the background that we have.
- [17:55]
B
Yeah, I think another, another way of, of looking at it too. You know, AI, like generative AI, is designed to predict the next most likely thing to be next to, you know, so in the case of words, it's like, what's the next most likely word to show up? Which is not created. I mean, because you can make the argument that, I mean, there's nothing new under the sun or, you know, whatever cliche you want to use. Like, humans keep reinventing things as well. But I think the, the important distinction is, is what you're saying as well is we draw from a much broader array of things, whereas AI is actually designed to predict the, the most likely and the most reasonable thing to show up next. Because if it didn't, like, it would just be complete chaos and garbage if AI did it.
- [18:43]
A
Right, Right. And it also sometimes will give the same answer to the same question over and over. My sister does some teaching and, you know, she gives a problem to students and she can tell the ones who copied her thing, put it in chat GPT, and it comes up basically the same thing where even if two humans were reading the same material, they might write it differently. More. We would kind of diverge more from that. And as you said, AI is not made to do that.
- [19:10]
B
Yeah, yeah. Last thing I want to talk about here is, you know, just because you can doesn't mean you should. Right. So the same goes for AI Much, much as we've talked about a little bit here already, you know, when's the right time to use AI and when is it not the right time? And, you know, what should UX teams use to guide them here?
- [19:31]
A
I think AI is great. The beginning of, I mean, I think AI is good to use at any time in the process. I don't know that there's a right time or a wrong time. There's times to refine, there's times to look. But one of the things that, you know, we kind of talk about it, it lacks a bit of editing as well. So I can say, hey, AI, build me a homepage or build me, you know, a business analyst analytics tool. Right? And it's like, I want these inputs, I want this, I want this output, I want this, I want this. And it can, you know, code me up a working prototype. But the problem is, is maybe I have too many numbers that don't intuitively make sense. Maybe it looks too much like a spreadsheet. Maybe it looks too much like this. So just because AI created an artifact doesn't mean that it's the right artifact to use. It doesn't mean that it can't be enhanced. It doesn't mean that it doesn't need user experience. Again, you know, there's a lot of brilliant people out there, but they may not be as skilled in one craft or the other so that they need that help editing, right? You know, like, I can watch YouTube videos on how to build a house, but from actually doing it, someone who knows how to build a house has more tricks and tips and can guide me. It's like, I know it says that, but you don't need that. This is what you need for this state, for this instance or whatever, right? So the editing, the refinement, the making sure it makes sense, making sure it's not too verbose, rewriting, checking headlines. I just think there's. There comes a time where it's easier to almost take it away from AI and just work it through than to continue to over iterate with AI because sometimes it doesn't have a history. Like, we'll change that and then it changes five things. Like, no, no, no, go back to the previous version. Change that one thing down to the left, to the right. And sometimes it's just easier for us to do it ourselves.
- [21:22]
B
Yeah, yeah, totally agree. Well, Jason, thanks so much for joining today. One last question for you before we wrap up. What do you do to stay agile in your role and how do you find a way to do it consistently?
- [21:35]
A
I would say one of the ways we do it is I actually just follow my team a little bit. You give the team a little bit of a leash and they start moving in something and always kind of being open to new ideas that different people bring. I think that's when you start building ideas on top of each other. New ways of working, new thoughts come. Something that you never would have considered suddenly comes into practice because someone else is like, well, we just tried this. It's like, you try that and it worked for you. They're like, yeah, it's like, then maybe we should try that again. And just being open to where ideas come from, from all sorts of different teams and disciplines or thoughts.
- [22:10]
B
Love it. Love it. Well, again, I'd like to thank Jason Bowman, Executive Director of Experience Strategy at the Office of Experience, for joining the show. You can learn more about Jason and the Office of Experience by following the links in the show notes. And thanks again to the Office of Experience for sponsoring this episode. Thanks again for listening to the Agile Brand brought to you by Tech Systems. If you enjoyed the show, please take a minute to subscribe and leave us a rating so that others can find the show as well. You can access more episodes of the show@theagilebrand.com that's the agile Brand and contact me if you're interested in consulting or advisory services or are looking for a speaker for your next event, go to www.gregkillstrom.com that's G R E G K I H L S t r o m.com the Agile brand is produced by Missing Link, a Latina owned, strategy driven, creatively fueled production co op. From ideation to creation, they craft human connections through intelligent, engaging and informative content. Until next time, stay curious and stay agile. The agile brand.