← Insights Unlocked
Insights Unlocked
How to build agentic AI your customers will actually trust
00:53:40
Trust, not usability, is the real test for agentic AI. Walmart's Chase Keaton explains what that means for UX research and design teams.
Loading summary
Transcript57 lines
- [00:02]
A
Welcome back to Insights Unlocked. In this episode, we sit down with Chase Keaton, Senior Research Manager at Walmart, to talk about the two competing mindsets shaping how teams use AI today, why the builder model might be missing something crucial, and why trust, not usability, could be the real metric for agenic AI. Chase brings sharp, practical thinking to a conversation every researcher and designer needs to hear. Enjoy the show.
- [00:30]
B
Welcome to Insights Unlocked, an original podcast from User Testing where we bring you candid conversations and stories with the thinkers, doers and builders behind some of the most successful digital products and experiences in the world, from concept to execution.
- [00:49]
A
Welcome to the Insights Unlocked podcast. I'm Nathan Isaacs, Principal Content Marketing manager here at UserTesting and our host today is Mike Mace, a Director of Solution Marketing at User Testing. He's also a longtime tech veteran and one of the areas he works on is helping companies build effective AI products. Welcome back to the show, Mike.
- [01:09]
C
Thanks very much. It's great to be here.
- [01:12]
A
And our guest today is Chase Keaton, Senior Research Manager at Walmart, where he helps teams make smarter customer centered decisions through research. A passionate advocate for human insight, Chase has built his career turning complex customer behaviors into clear, actionable strategies that drive better products and experiences. And as a quick disclaimer, while Chase works at Walmart, he is not speaking for Walmart and he won't touch on any confidential information. Okay, now, welcome to the show, Chase.
- [01:43]
D
Thanks for having me. I appreciate you.
- [01:46]
C
Hey Chase, it's great to talk with you again. So, folks, know the background. Chase and I have talked a few times and I found him to be a really fun guy to talk with about AI things. He's got really interesting, innovative thoughts about what it's doing and also what we can do about it. So today we're not going to be just talking about how you should use AI in your work. I will touch on that some, but this is also what AI is doing to user experience and the opportunities and challenges it's creating for us. And you know, we've talked about this stuff before one on one and I wanted to share it on Insights Unlocked. So. Hey Chase, it's great to talk with you again.
- [02:27]
D
Yeah, I appreciate the opportunity. Thanks for having me, Mike.
- [02:30]
C
So I think to start us off, it'd be great if you could share a bit on your background because you've done a diverse set of things over time that kind of add up to where you are today. Talk to us about the journey you've been on.
- [02:43]
B
Sure.
- [02:43]
D
So a lot of people end up in the UX space through either like an academic lens or kind of like a more of a homegrown natural progression. I have fallen to the latter. Our category. I've been in UX as some form or function for the past 13, 14 years and a lot of that is focused on AI related things for the past five or six years. So it puts me in a unique position to have a really long UX story career across different B2B, B2C channels, different companies, a large diversity there and then also just the ability to kind of talk a little bit about how AI intersects with research, intersects with users, intersects with business. And that's been a lot of my focus for the past five, six years. So I'm happy to share what I've learned. I doubt any of it is going to be world breaking, but I think it's also very important to sometimes just lay things out so that everybody can kind of have a chance to pick apart and get their own interpretation and figure out where to go from there.
- [03:52]
C
Awesome. And specifically what's, what's your exact role at Walmart today?
- [03:57]
D
So currently at Walmart I've my, my title is listed as senior manager and I'm working for the first time in my career under like more of a marketing wing than a strict like product wing or UX wing. And the overlap's very similar. I've got, I work on a fabulous marketing team. We work on kind of all the newest propositions and that's some of, that's very AI adjacent which is, explains kind of the current fit. But we work a lot on understanding what appeals to people from a product standpoint so that that way we can have the right messaging that doesn't damage the brand, but also have our deep understanding of kind of like the psychological motivations. It's often very much about understanding what motivates changes in behavior, what are the frictions when it comes to changing behavior and kind of what are the barriers to like adoption and continuous use. It's been a kind of a refreshing take on a side that I didn't experience a lot of throughout the rest of my career. So it's very similar psychology, very similar activities, but it's applied in a slightly different discipline.
- [05:05]
C
That's cool. So we talked a little bit before this recording about some topics we might cover. And one of the things we talked about was some of the tensions and the questions that you're seeing right now about AI. Could you talk to me about what you're, what you're seeing right now?
- [05:22]
D
Sure, like definitely something that I think is worth surfacing is that AI kind of exists in a weird dichotomy. It almost kind of reminds me of things like code switching where there's kind of like two really prevalent and also somewhat extreme views. One, I kind of think of almost like a corporate flavored like view of AI where you're really encouraged to use it sometimes. In some cases, well this may not be true at my current company. In some cases it's almost like you're mandated to use AI in order to do the, the job in a more efficient, more effective manner. And then there's a lot of like there's no such thing as a bad use for it. Just kind of keep pushing the boundaries, training the models and it's very tech focused, tech forward. And then there's kind of almost an opposing view that's very person centered and a lot of it is kind of groping with if AI were to start eliminating jobs or elimination elimination of portion of jobs or allow for maximized productivity, what happens next? And it kind of, it takes a very like, ethical and like human view of what using AI may or may not look like. And I found that sometimes what has helped me both like in a career and in like my personal usage of AI is to kind of try and keep both those things in your brain at the same time. Right. Like there's a certain inevitability to the usage of AI specifically in like not just corporate cultures. But you're starting to see it a little bit more and more with personal examples ranging everywhere from like help me plan my day to how might I go about understanding these patterns in my relationship and some of that deeper psychology work. And then you've also got kind of like that personal impulse to like what part of this is dangerous? What's the cost? What's the trade off to this? Am I able to use AI in a way that's ethical and pro human and how do I go about finding a way to apply it and kind of activate so much of the empathy that is definitely centric to kind of like that UX career as well. It's, it's, it's hard not to see the echoes and reverberations in different contexts.
- [07:53]
C
You're paid to have empathy and yet there are a lot of people who are in pain around you and, and so you're feeling that as well. I, you know, that really resonates with what I've seen from some folks. I was at the research week conference in San Francisco a few weeks ago and the contrast of Nobody was like, you know, hiding under a rock or something like that. But the contrast of some people who were just on fire about the possibilities and like, they'd get up on stage and say, let me show you my stack. And I use this tool for this and this tool for this. And they were like, off to the races and other people who are much more measured and about, you know, I think of some of my colleagues who, when we talk about AI, they're really worried about, you know, even things like environmental impacts and societal impacts and stuff like that. And it's, it's hard for them to see past those things because they are so passionate about those issues. And yeah, I don't remember ever seeing the tech industry so polarized about anything. Not necessarily polarized in a bad way, but just having all these different aspects that we're thinking about at the same time.
- [09:00]
D
No, it's, it's an interesting point because that polarity informs things because there's almost like a performative element, like, you are going to drink this Kool Aid regardless if you like the flavor, because you're part of an institution or you're part of a movement, you know, and there's also like that deep zeitgeist feeling of almost we'll go with something that's dread maybe, where, like, there's just a lot of uncertainty. Right. And in some ways you could almost like, interpret it as a scale of like, how you're coping with the possibilities, both positive and negative, on that scale of uncertainty. And it's, it's really interesting because I'm. A lot of my colleagues across a lot of different companies in a lot of different spaces have noted there's almost like there's two versions of AI in the room and it's. How do you reconcile that? Right. Or are we dealing with, like, a villainous entity out to destroy every fresh water source that only cares about, like, converting humans to capital and Soylent Green is people and that kind of ideology versus, like, AI can do anything and everything and all you. And you're the only thing standing in your own way from Vibe coding the next biggest thing and why every report you run should be run through AI first and every email you write should be checked, et cetera, et cetera. Right? So there's, there's definitely a balance and kind of like a negotiation between those two things. But what's really interesting is there's some environmental pressure to display one in certain aspects or maybe like, really showcase or maybe upplay another in other aspects. And I, I found that a lot of people get more comfortable with one ideology than another. And oftentimes you can, you can really find some great applications for it when you can kind of just be aware of both those viewpoints. Right.
- [10:56]
C
I like that. You know, it kind of, I think you're suggesting that there's, it's kind of a little healthy to have that dynamic pushback between those things or at least for everybody to be aware that other people may be at different, different points on the continuum.
- [11:12]
D
It's, it's hard because like we, we live in a lot of world of gray and whenever either of those extreme ideologies, if either of them win, it's not going to lead to progress, either corporate or professional. If we completely ignore how AI functions and we just decide to take a very humanistic step like down that road, we're really not leveraging some pieces of the technology that could have ridiculous implications and medicine and like things that could really, truly benefit the entirety of the human race if you just kind of like. But the cost is too high. Right. And then on the other hand, if you go straight through that lens of everything you do should be through that AI functionality, if you take that to its natural extreme conclusion, that's also not great. That's, that's a world where in the next five years there's not really an entry level job anymore. The, the thing that is sometimes whispered in LinkedIn channels, the ridiculous post of entry level job requires five years of experience actually starts to look like a reality in that space. Right. Where we need people who can understand how to operate various AI agents in various different contexts, but we also need people who have done the job before. But does that exist anymore? Because that can be done by AI agents.
- [12:42]
C
So it's a great question.
- [12:46]
D
They're both extreme views, but the tension, the interplay between them can really help inform a way to kind of move forward. And it's kind of okay to have those two beliefs at once, even though they look like they're diametrically opposed. Right? Like the. Because either side, quote unquote, winning wouldn't necessarily cause a net benefit. But there, there is a way to kind of like thread the needle, take a humanistic approach, but also progress because it's, it's progression for the sake of. Progression is cancer. Right. But progression for the sake of benefiting everybody is exactly why a lot of people even choose a career in the space. So.
- [13:33]
C
Yep, yep, I love it. You know, and I think if I think back to the previous big technology transitions that I've lived through. If we had had this sort of balanced thinking in 2000, when Web 2.0 was on fire for all these millennial things. You know, if I had a nickel for every time somebody said the wisdom of crowds and how that was supposed to be a great thing, and yet the very mixed sort of result we've had from that, it would have been a lot better if we'd been thinking through the pluses and minuses more realistically at that time. So we could have planned for it rather than trying to put patches on it later. So, okay, I like the idea of viewing this as it's painful and there are different viewpoints, but that's healthy and we shouldn't be trying to dismiss that away. This isn't a problem we're trying to get rid of. This is a functional thing that actually can help.
- [14:28]
D
It definitely lives in that realm of it's powerful enough that it's scary to touch and it's definitely something that can be overused. But it's like any knowledge, there's a little bit of that. It is the application of the tool that really kind of determines the intent, the impact, the outcome. So being mindful of that application can be very helpful. And that's even more confusing with something like AI because it's essentially a platform. Every UX interaction that you've been trained on throughout the entirety of being a user is very input, output. It's very, I have a goal. This app or this product or this experience will help me address that. And AI is not about that precise goal. It can solve for a lot of precise goals. But AI, as you open it up and it's kind of like that blinking cursor moment is a platform that you can accomplish a lot of different goals in. Right? So it become, it becomes even more important to be centered on what that intent is, what that rationale is, what that instinct is, and just knowing how to leverage that to your best advantage while still keeping in the different ramifications of either kind of extreme measure.
- [15:53]
C
Yep, gotcha. So, okay, so let's go from the philosophical stuff to a little bit more operational. So one of the big areas of buzz right now in the AI world is the builder model. You know, this idea that all of the product related roles, you know, UXR design, PM engineering, all those folks are going to kind of be the same in the future. Everybody's just going to be an AI assisted builder and we'll all just make software that goes out there. And that's the millennial world that we're not millennial generation, but millennial like the end time. That is what we're heading for. What do you see happening around that?
- [16:32]
D
There's, there's a lot of credibility to the idea of the builder model and kind of when you forecast AI capability over like a medium term, something like two to four years. Don't get me wrong, it's always dangerous to try and predict any kind of future. We do our best. But there's a place where that builder model could potentially be a vision of the future where you kind of have one person dedicated to different agents that behave as like a uxr, that behave as a designer, that behave as kind of a product, maybe in some ROI analysis, some of the business in there as well. Right. Also of course engineering, security, a lot of the other kind of like as a service type products that all go towards the building of a product, the building of a thing. And there's. While there is a future where that could be the case, it's what one of the reasons that all those different functionalities exist is, is they create the ideal tension to build a product that's effective for everybody. And one of the common, one of the flaws of AI is that its prime directive is to make its user happy. And the best products are not because product is 100% happy with it. Business is 100% happy with it. UXR is 100% happy with it. The, the best, the best technology and the best innovations often come from everybody being 70% happy and having the healthy conflict between those things play out in a natural way. Right. And you'll see this a lot in different examples where a product will have an what they believe is an excellent idea, they'll convince business to fund it. UXR will do some of the discovery research for it and they will find that not only does it not solve a core problem, it has nothing to do with what the user wants. Right. And then that becomes like attention filled conversation. We have discovered that the user no longer wants this. How do you want to proceed from here? Product. And then it's all about kind of like UXR finding a way to convince product in a way that empowers them to go back to the business to say, hey, actually we shouldn't be building this, but we have discovered that we should be building this. That level of complexity and that many zigzags throughout like Zoom, 0 to 100% done on whatever feature ends up getting funded by that budget. It can't really be handled by a builder model like you can intentionally stage conflict between different agents in a way to kind of like potentially duplicate that. But the problem is there's so many different trade offs and those trade offs are by design fluid. And that, that conflict, that organic process, is always going to give you a big better product than just trying to force a zero to 100 moment because you have the tools that are capable of going from 0 to 100% dot.
- [19:41]
C
Yep, yep, I like that. And the idea that human beings are a much better way to get that dynamic conflict as opposed to AI, where if you ask it to create conflict, what you're really asking it to do is hallucinate about disagreements. Whereas, hey, it's usually better if you're asking AI to hallucinate. I think that's a, that's a great place to use a human being instead because they're much less likely to hallucinate and. Yep, yep. Makes a ton of sense.
- [20:12]
D
Yeah. No, the, the flag for conflict being healthy when it's done with intentionality behind it and leading to unhealthy results when it has some sort of rails or guidelines in that AI focus really can show a lot of the benefit and a lot of the ability to have those trade offs. Like not, not to, not to go. Keep calling back to a point, but the ability to kind of hold those conflicting visions and priorities in your head can really help with when it comes to things like prioritization. Right. And AI is somewhat capable of doing that, but when you really force it, you're in a place of hallucinations, you're in a place of conflict for the sake of conflict rather than productive conflict to show the user point of view or productive conflict where engineering surfaces know really we can't do this without the infrastructure upgrade. Right. And a lot of times those things are what creates the product to evolve to a level where it becomes something that is useful for way more users. Right. So it's the, the, the problem is the point. Right. Like this, the discovery of those problems as you move through the process inform better decision making than being able to glaze right through them.
- [21:36]
C
Yep, got it. The, the problem is the point. I think that, that, that's cool. All right. I'm getting some good, some good little, little here. Back when you and I first talked, you told me at one time you had in your career flirted with the idea of being a designer. And I'm wondering if now that you have all these AI assisted design tools, does that make you want to jump into trying to, trying to be a designer again?
- [22:03]
D
I mean, I've definitely flirted with the idea. Almost every time I've put on a designer hat though, I immediately went to the question of but what do the users want? And I was told, I don't know. And I'm like, do you mind if I research that? And so far throughout my career, almost all roads have led back to research. But when I think about how AI applies to design practices, it's. There is a temptation where like you can go from just the conception of a design, understanding some of the inputs and outputs and you can with a prompt have almost effectively a working AI that's going to be like spit out through the tool that can like already have HTML. They can already be aware of your tech stack. And I can see a world where that might be something that is scary to the design community. But it also is going whenever you use AI to build something from scratch, you're almost always sacrificing novelty and context for speed. And while you can get to an end point that looks like design, smells like design, tastes like design, on that first glance, on that first glow, you'll notice that because everything is built off of pattern recognition, a lot of things will be done right, but they'll be done right out of context. So you might end up with some of the right elements in the wrong spaces. It might be completely inconsiderate of how people block different tasks and organize them when they're trying to do something. It also might over prioritize things like navigation via menu. And we've worked when like in general the industries move way more towards like a search based architecture, right? So like because every design that is compelled via AI is going to be copies of copies of copies of copies, it's going to duplicate a lot of IT best practices on accident, but it's going to be hard to apply your specific context and it's going to get even harder when you start trying to edit things and change things. And as every designer knows, like you live and die by like the system and the replication. Like when I was interested in design, there wasn't a lot of that happening at first where there were still design systems. But there are things like you, if you ended up changing even just like the font of a button, you would have to manually change it across every iteration of that product, right? And you ended up kind of like having to do that. And that's a place where AI can be super helpful as a designer tool, where it can do some of the iteration that you can't always do. Like the component layer that you can't always do like via execution of a design system. So I don't think AI is an imminent threat to designers as long as they keep their seat when it comes to advocating for the customer and creating those healthy tensions where appropriate. But it's also not a seat that I think I ever could occupy because every time I sit in there, I'm uncomfortable enough that I start finding myself more in the psychology of users. And once you're there, it's really hard to do that as often from any other UXC than a researcher.
- [25:29]
C
Got it. So back at the beginning of this year, everybody said this is going to be the year of agentic AI. So here we are halfway through the year now, roughly at the time we were recording this. What are you thinking? What's going on with Agentic and what are you seeing as the trends?
- [25:46]
D
Well, Agentic is interesting because when AI first emerged, it had that very like super Google, super Google formula. Right? Everybody kind of understood it as a search plus, right? Like not only will this address my query, but can ask some complex questions along the way and it might be able to shortcut some of the decisions that are less fun to make and I can actually end up with like a more viable product in a faster amount of time than if I were to search for it myself. Like, you still have to comb that from a data perspective. You still have to understand what's happening there, what's, what's valid, what might be hallucinating. You still have to kind of be able to interrogate that with context, intention, but that's how it essentially functioned. And then the promise of Agentic is not only is AI here and capable of understanding you, but it's also capable of acting on your behalf. And I don't think, I think we're starting to get there, but I still think that there's a lot of resistance when it comes to adopting agents that work on your behalf versus agents that are providing information or providing a one of or two of query. Like you're going to see a lot more design this deck for me, making this argument for these stakeholders based off my Slack messages for the last 30 years for these people. And, and that's still in essence, kind of like an information query in production. Right? Whereas agentic AI is, hey, I need you to buy me everything that I'm running low on for the week and make sure that it's at my doorstep between 7 and 9pm tonight because I know I'll be home. And that way I'll be able to Meal prep for tomorrow around nine. Right. The, that promise is in some ways technologically possible, but there's still a lot of relationship issues because everybody's level of trust and action is different. Right. Like if I were to walk up to 10 different people on the street and I were to ask them if AI could perform a transaction for you, what's the maximum amount of money that you would be comfortable AI using to perform those transactions on your behalf? I would, I wouldn't get 10 different answers, but I would likely get a few. I would likely get a very wide range of. Well, based on my salary and assuming it's just groceries and there are certain limits on it, I'd be fine with it. Spending a hundred dollars a week could live in the same room as somebody who might even be bringing the same take home pay. At a household income level, they might be like zero. I just don't trust it. Like I already struggle sometimes ordering fresh groceries online. Why, why would I trust an agent to now all of a sudden be picking out every bit of my diet even if I give it extensive? So like there's definitely a, a trust threshold, like a trust event horizon. And it's going to be really interesting to explore how to help people if they want to adopt agentic AI. Right? It's, it's going to be interesting to try and kind of find the there where it's at a place where it can start. Let me put it this way. Most human relationships are built off of trust that is developed by alignment of words and actions. That's kind of executed over time as a pattern of behavior right there. If, if every time your friend says they're going to be there, they're going to be there. That's one level of trust. If they're also there when they're helping you move, that's all. That's another level of trust. Right. And all these things kind of cascade and they eventually form a relationship. One of the problems with the adoption of agentic AI is it's kind of the first time that AI is allowed to be an actor. Like it's, it's always had words on the table and sometimes those words work and sometimes they're hallucinations, but it's the first time that actions are on the table too and how the actions align with the words, right? And then at what level of action do you start allowing that trust to grow with AI and some people are very comfortable like there, there's a very wide range of experiences there. But even though the agentic capability is There. There's still a very human nuance that I think we need to understand a little bit more that allows people to get past the tipping point of being able to use agentic AI in a more commonplace manner.
- [30:33]
C
Yep, yep.
- [30:35]
D
It's.
- [30:35]
C
It's interesting. It's. We're so used to building products in the tech industry, and by products, I mean things broadly that could be an AI bot that's on your website or something like that. But we're used to thinking of those things as stuff that we're building out. And it's all about how accurate is it, how much latency does it have, all of those sorts of things. And what you're describing is something where it's fundamentally different. It's not about the product per se, it's about how do we create trust in the product such that they will allow it to do the things that we've designed it to do. Which feels like that's a really different mental discipline that we need to be going through. Right.
- [31:16]
D
It's very much so, because oftentimes when you're interrogating the trust of any experience, you're looking for a, is it learnable, is it teachable, is it repeatable, does it provide the right level of flexibility in the places where it's needed? And you can kind of ask all those questions from AI. But the thing is, when you're doing that with a non AI experience, a little bit more of a traditional interface, oftentimes you're grading things on a binary.
- [31:47]
A
Right.
- [31:47]
D
You're grading things on successful not, successful not, successful not. And when you're looking at AI, there's so much nuance in how it intakes and how it executes that you can't really judge it on a binary scale anymore. Right. And the level of trust that it creates is going to be almost like the relationship that AI develops with each user based off of that user's preferences and thresholds and, and willingness to adopt a new technology. Right. And it's because there's so much more nuance. It's not, it's not a standard usability test where it's like, can you do the thing or not? Right. It's, it's, it's a. And like, what are the struggles and what happens when you're doing that? It's a lot more along the lines of, you tried to do the thing, you mostly did the thing. How did you feel? Do you feel that AI let you down? Do you feel that you didn't provide it enough context and that's why you only got 80% of the result you want? Is 80% of the result you want enough for you to continue using it? It's, it's, it creates a different level of. It's, it's a lot less pass fail. Right? It's a lot more, it's a lot more nuanced on how did it do what it did when it did the thing as an agent for you? Like, how did you feel about that interaction? Would you be more or less confident when you're interacting in that style, moving forward based off of how it performed? And it's, it's an entirely new ballgame because you're not really trying to figure out is the interface in a place where most people can understand it or is the interaction in a place where most people can understand and execute it? It's way more along the lines of is the relationship in the place where both entities want the relationship to be? Because you're not really evaluating that input, output, input, output, input, output anymore. You're evaluating a systemic repetition of different inputs, desiring different outputs, providing different instructions with different context every time.
- [33:59]
C
Yep. Yep. You know, this whole concept of relationship as being key to AI software is to me, a really, really interesting one. You know, I'm, I'm so used to thinking of software as a tool and therefore usability is the most important attribute of it. And what I think you're implying is, hey, it's the trust. It's the trust in the relationship and how the relationship works that increasingly is going to drive the success of AI centric software. Do you have thoughts on, like, how should a company be approaching this? What should they test for? Any, you know, any guidance we could give folks listening to this podcast going, oh, Chase is right now. What do I do about it?
- [34:44]
D
I mean, Chase is right is a bold statement in and of itself. But I think when you're looking for areas where AI intervention makes sense, like, and this is something that you can apply in a lot of different systems. So, like, if you are a business place where you can look for AI intervention is you want to look for places where it can effectively proxy as a person.
- [35:10]
C
Right?
- [35:10]
D
Like, that's kind of like level one. And like the thought process that you're in when you're doing that is, could I trust an intern who's only had a minute of training to do this thing? And no shade to interns. Interns are amazing. It's more of a, it's more of a. Just like a level set. Somebody who has some skills but not necessarily a lot of context or experience because that's oftentimes what you're dealing with. When you're dealing with most AI, could they do it the same way? Right. And that's, that's kind of like level one, that's like a good place to start. And then level two tends to be like something that this audience could likely leverage A fair amount is the idea of like system design operations, both from like design ops and research ops. Like the ability to start thinking in kind of discrete systems and figuring out like where, where you are. Right. As far as like if I'm strong at framing research, but I'm a little bit less strong at recruitment, maybe I could find a way for AI to kind of like bridge that gap. But when it comes to like, sorry, that's a little bit more personal aside, when it comes to more of a business side of things, oftentimes you're looking for places where AI can provide value. But oftentimes like I, you'll see AI be used for things like greeting or item lookup or finding something in the store. And those things are key things that oftentimes people who are actually working in store and wearing their apron or vest or 40 pieces of flare, whatever have you, they're, they're, they're often there to provide that. But when those are often kind of like a side part of their job, right? Like where they're often assigned to like clean different areas or move things around, or find different ways to kind of like keep the, keep the company moving and profitable. And then the customer interactions, there's kind of something on the side. The general impulse has been let's continue having people do those things and then let's let AI handle some of that customer facing element. Right. So if you've got a physical storefront, if it's a walk up, if a customer has a question on how to execute a thing they're trying to do, or if they're trying to figure out something with nuance or they're unsure which product to pick, a lot of businesses are plugging AI into that space. I think that there's a lot of value in kind of inverting that model though, like finding ways where AI can play as something that can perform some of the cleanup, perform some of the routine maintenance and not to. It's almost to free the humans that you already have to do the things that require human intuition, that require the ability to read body language, to infer intent, to ask follow up questions. You're, you're rarely going to have a human, on human interaction where you would classify it as a, as a, like a hallucination. Right. You're, you're never going to walk up to somebody saying hey, where are the flowers? And they're going to be going, flowers bloom best in August. Like you're, you're not the. Although it's, that'd be fun to play out as an SNL skit because you might just be like, you're right to call me out on that. I'll, I'll go you to the flower. I'll guide you to the flowers now. Right. But it, I think that there is a, a little bit of ease because everything's based on language. That most of the jobs that AI is being given are very language driven. But that's also where a lot of the nuance of understanding of human behavior and I'm not sure how you could even begin to train things like that. So I, I see a lot of the first attempt being as a proxy for a person and I see a lot of successful second attempts as a way to free people up to do what they're best at. And that, that, that, that applies really nicely in a lot of different discrete places.
- [39:15]
C
I like that. So bring on the AI driven robots because they can go stack boxes in the background and free up the human beings to go deal with other human beings, which is what they tend to do best.
- [39:30]
D
I mean some of the first conceptions of what AI and that robotica systems look like, I mean this is going to date me quite a bit. Like the Jetsons, like Rosie the Robot. Right. Like the ideal AI was there to help cook and clean so that it could enable us to spend time with our family and do art and engage in things that have a very unique point of view and perspective and things that add value to our lives. And almost from the jump, AI has almost served as an inversion of that where it's generating images and trying to mimic effective art both like the written word in addition to paintings and other things along those nature. But there hasn't been as much of that advent as like this can help me human better because it can do some of the things that don't add a lot of value to my life or at least surface value. Right. Like there's, I don't want to ever demean any type of job or anything like that. There's a lot of. The amount of thinking that you can do when you're working with your body is not to be underestimated. It's just one of those Things where it has always been advertised when it's an imaginative thing as something that frees human capital to do human things. And a lot of its expect and a lot of its execution, especially early on in kind of the advent of AI has been a little bit more flavored of it's doing things that are copying some of those human elements, but not necessarily providing some of the value that we anticipated from when we were conceiving of it.
- [41:09]
C
Yep. You know, I like, I'm glad you mentioned Rosie the Robot, because that's a good, that's a good analogy for me or way to frame it. And I do like the idea of AI evolving to be the ultimate personal butler. You know, the, the thing that like, like one of those British. My wife loves these, these British costume dramas from the 1800s. And there's always the, the butler who understands the person perfectly and can make every anticipate and make everything happen for them in terms of the logistics of their lives so that they're freed to do other, other presumably very fulfilling stuff. And I love the idea of AI doing that. I guess for you as a retailer, if I have an AI who's my super butler, who knows all my preferences, I may be. And I'm not looking for Walmart policy. Okay. Here I'm talking about philosophically, just as a general retailer, I might be interacting with you through my, through my AI bot, my butler bot. That's my interface. How do, how do you feel as a, as a retailer dealing with that? Is that, is that threatening? Is that empowering? What do you do about it?
- [42:21]
D
I, I think it's both. Right. So like there, if, if you have a bot that's kind of executing on your behalf and kind of doing the, doing the butler thing, so to speak, as a retailer, one place where that's exciting is from a personalization standpoint, that butler is going to know your preferences in and out without any of my systems having to know them. Right. And so there's a big inherent benefit for having all of that available context. So that I know as a retailer, let's say that you're shopping for. It's a good one. Let's do, let's say that you're shopping for furniture.
- [43:09]
C
Right.
- [43:10]
D
So as a buyer, as a butler, there's all this wealth of information that my systems can use to try and kind of help surface. And it probably depends on your butler. But like, am I surfacing a decision between multiple pieces for you? Like if you're searching for the perfect couch or end table or does Your butler have such an exact idea of the dimensions that you're playing with, the colors that you prefer, the styles that you're, that you, that like resonate with you and make you feel like, oh, this is going to feel like home. And it's going to be interesting to kind of probe that as we move forward. But the nice part about that analogy or the nice part about that scenario is it does solve kind of the how's context evolving conversation. Right. Because if you only visit my furniture store once every two years, you might have been into one thing back then. And if my systems are pre wired to do that, I might be proposing something that was yesterday's news to you. Whereas your butler might be more aware of that change and that dynamic. Right. And also as a retailer, I wouldn't have to store some of that personalization data. I could just use it when it's needed. And that's, that has a less of a footprint. There's less data centers that have to be there at a whole. Right. It's. There is kind of an interesting thought experiment that comes from that where as personalization becomes more and more important with the advent of AI, duplicative personalization across different preferences could have an astounding footprint over a long period of time. So like having a centralized personalization piece could be pretty interesting.
- [44:52]
C
I like that. You know, and it seems to me, and this is now, okay, a couple people in the industry speculating and you know, with all the. We just said you're not, we're not supposed to do it that way. But I, it feels to me like there could be situations where it is, it is to the advantage of the user, if they're willing to do it, to share a certain level of private information through this sort of controlled mechanism because it will produce much better results for them from everybody that they engage with. You know, I, at some level I want you to know that I've moved from an apartment into a larger house and that I now have much more room for different types of furniture. And that is to my advantage for you to be able to get that information. And it's back to trust. Right. How do we get people to trust being able to interact with this?
- [45:43]
D
Well, it's a fascinating trade off because the more information that you can freely give, the better fits that can be established for you in a variety of contexts. Right. But we're also in kind of like a weird, kind of almost like chicken and the egg moment when it comes to something like agentic AI where the more information and Context it has, the more effective decisions it can more defeat, more effective decisions and actions it can take on your behalf. Right. But you have to have enough trust in the technology and in the ethics of its application to be able to provide that level of personalization to get to the point where agentic might be a reality. So it's a very interesting push pull effect where you have this idea of how do I make, how do I get to that level of customization where it really benefits me before I have full trust in the system. Like what's that leap of faith look like? What's the behavioral barrier that says I can't trust this until X. And that might be the next real thing that needs to be solved for at a grand scale is what are the markings of enough trust that I can take more of a leap of faith with the technology and what are the benefits? What are the goals? What does that look like?
- [47:11]
C
Yeah. Cool. So we're going to start running up against time. I want to be careful with that. But there is one more thing I wanted to ask you. So say there's maybe a UX researcher, designer, somebody like that in our audience on the podcast today. They're working in a mid sized company without a ton of resources, but they're under really huge pressure to make use of AI. Do you have any advice for what they should focus on?
- [47:37]
D
Yeah, I mean this, this goes back a little bit to the conversation that we were having about as a retailer, what are you looking for? How are you looking to implement AI as that like works well in this scale where kind of like at first blunch, at first blush, when you are trying to kind of like dip your toe into effective AI use at work, it becomes one of the first things that you can start stress testing are what are the things that I would feel comfortable assigning to somebody who didn't have a lot of context.
- [48:17]
B
Right.
- [48:18]
D
What are the jobs to be done? And this is actually one place where I think UX and product and a lot more of the customer centric folk are going to be way ahead of the game because there's a lot of metacognition and a lot of self reflection built into the workflow as it is. Right. There's a lot of interrogation for bias and things like that. So the nice part is a lot of your skill set that helps you be successful in elevating the voice of the customer for a product team, for a marketing team is going to be the exact same thing that can help you be successful in implementing AI into your Everyday work, work. I think it's important to try and get to a high level understanding. I always kind of think of it as like bucketing your work, right? So from like a very research centric point of view would be like you've got research intake and then you've got prioritization and then you start choosing methodology and then you develop a protocol or a script. And like there, there are all these discrete steps that happen every time you execute research and there's a little bit of like a shifting like Kanban board there about how many things you can hold in your mind at one time. But as you start decomposing that the better you know yourself and your skill set, you're going to be in really good shape to be like, okay, I, I might be very effective at assigning a methodology, but I really struggle when I'm writing protocol. That's an excellent first step to take with AI when you're like, okay, what are. I'm trying to learn this based on the context of this system. Go ahead and generate a protocol for a contextual inquiry where I plan to spend about 45 minutes per participant. What comes back to you will likely only be about 40% relevant to your task. But what it often will do is it can help you kind of get from inception where like, how do I start this? If that's a weakness of yours to okay, I've got something, it may not be the most amazing thing that I've ever seen, but I can iterate on this, right? I can work on this, I can recontextualize it and it can actually if, if you are really strong at that iteration process of that particular gig of just protocol creation, writing your script, figuring out what your questions and guidance should look like, like you just jump started to there. And even though that might not necessarily be hours and hours and hours, it did the most difficult thoughtwork for you so that you could apply your expertise where you were most effective, right? But what's funny is it's not a one size fits all thing. A lot of your ability to use AI at work comes down to having really honest, reflective conversations with yourself and understanding where you are strong and where you are weak. And I would advise spending time with AI in both places because where you're at your strongest, you will see the flaws because you understand all the different nuance and what's going on at that, at that point in time, right? In that particular expertise where you feel deep on, but where you feel shallow, you won't be able to interrogate the results as much, but you will start to see patterns between what is quality and what is not. And you'll start to be able to understand, okay, it's adding value here, it's not adding value here. And then where most things end up is like it's adding some value. But I need to figure out how to refine it. And that refinement story, that identifying where it's adding value, but it could add a lot more. If you refine it, you're going to find that as a very powerful story for your career moving forward. Being able to tell that will help you with, if not job security, career security, because you understand the value of finding those engines of work and refining them because it's a different muscle than a lot of people have had to use before. So get comfy with that discomfort.
- [52:23]
C
I like that. The better you know yourself, the better you can deploy AI to fill in your gaps. Beautiful. That's a great vision to end up with. Chase, thank you so much for your time today. Really, really appreciate it.
- [52:36]
D
Yeah, thanks. Thanks for the opportunity. I really appreciate the depth of the conversation and it's always fun to explore these things and think about these things and try and understand the multitudes of realities that come into these things. So I definitely appreciate the platform.
- [52:56]
C
Cool. If folks have any follow up questions, I don't want to put you on the spot, Chase, but for me, I'm happy to share my email address which is just Mike M I k e sertesting.com you're welcome to ping me if you got any follow up questions.
- [53:11]
B
Want to keep the conversation going? You can find the show notes@usertesting.com podcast if you haven't already, don't forget to follow us on Apple Podcasts, Spotify, Overcast or Google Play so you never miss an episode. And if you would, if you enjoyed today's show, please share it with a friend or leave us a rating and review on Apple Podcasts. And until next time, this is Insights Unlocked, an original podcast from User Testing.