← Insights Unlocked
Insights Unlocked
AI can build anything. It still needs someone to point the way.
00:53:50
AI moves fast, but humans provide direction. UserTesting leaders unpack AI in the loop, agentic AI, and building trustworthy AI experiences.
Loading summary
Transcript58 lines
- [00:02]
Nathan Isaacs
Welcome back to Insights Unlocked. In this episode, I'm joined by three User Testing voices, Leah Hogan, Amrit Batu and Mike Mace, for a roundtable on what they're actually hearing from enterprise leaders Right now. We get into AI's real bottlenecks, why human in the Loop might have it backwards, and what teams should prioritize heading into 2027. It's a candid, no hype look at where things stand today. Enjoy the show.
- [00:30]
Podcast Narrator
Welcome to Insights Unlocked, an original podcast from User Testing where we bring you candid conversations and stories with the thinkers, doers and builders behind some of the most successful digital products and experiences in the world, from concept to execution.
- [00:49]
Nathan Isaacs
Welcome to Insights Unlocked. I'm your host, Nathan Isaacs, Principal content marketing manager at User Testing. And as we move into the second half of 2026, business leaders are facing no shortage of questions. How is AI changing the way organizations understand and serve their customers? Which customer experience investments are delivering actual value and which are falling short? How are UX design and product teams adapting to rising expectations, tighter budgets and faster moving markets? In this episode, we're taking a step back to look at the bigger picture. Joining me for this special roundtable discussion are three of User Testing's leading voices, Leah Hogan, Amrit Batu and Mike Mace. Together they spend their days talking with enterprise organizations across industries, helping teams make smarter customer centered decisions. They have a front row seat to the challenges, opportunities and emerging trends shaping how companies innovate. Today, we'll explore the conversations they're hearing, most often from executives and practitioners alike. From the evolving role of AI and human insight to changing customer expectations, organizational transformation, research, democratization, and the strategies companies are using to stay competitive in an increasingly complex landscape. Welcome to the show. Leah, Amrit and Mike.
- [02:12]
Mike Mace
Hey, Nathan.
- [02:14]
Amrit Batu
Hi everyone. Hi, Nathan.
- [02:15]
Nathan Isaacs
Hey. You've all been hosts, guest hosts on the show. You all been guests on the show. But just for anybody new to the show, can you just share a little bit about who you are and what you do and what you kind of do day to day here at User Testing? And we'll start with you, Leah.
- [02:33]
Leah Hogan
Great. Well, thank you, Nathan. I've been at user testing for 10 years now, which is, I think quite some time in tech world. But you know, in that time I've spent a variety of time in a number of different roles. Currently I spend most of my time talking with customers kind of across the discipline gamut. So everything from researchers to product to design to marketing folks about these days, mostly AI and how to integrate more into everyday work. From a practical perspective of both delivering value but also just understanding where are we going. I think the other piece is just really trying to understand how research evolves and, and how design evolves as we move into this new paradigm around how it is that we interact with agentic interfaces.
- [03:50]
Nathan Isaacs
Mike, also a veteran of user testing. What do you do here at User Testing?
- [03:55]
Mike Mace
Yeah, so I'm a director of solution marketing and what that means is I work with customers on figuring out some of the biggest problems that they're trying to solve with human insight or the opportunities that they have. So that includes things like scaling and democratization, use of human insight in emerging areas like marketing and product management. And yeah, like Leah, most of the conversations I have with customers these days end up coming around to AI sooner or later. And so a lot of stuff on not just how to use AI in your human insight research, which is a hot topic, but also how to use human insight research in your AI. In other words, how do you make it more effective and more powerful and emorett.
- [04:44]
Amrit Batu
Yeah, I feel like Mike and Leah have got so many more years at user testing than I do. I've been here for six years, which is about six and a half years now, which is a bit scary. I'll give you my introduction I give to all our customers. I've kind of been in the industry for about 20 years now. I worked client side, I've worked agency side, I've been a consultant in a big consultancy firm. I've done my academic years early on in my career and now like I said, the last few years been at user testing and I'm kind of Leah's equivalent in the European market. And what I like to think about is I've made a lot of mistakes over those 20 years of working in the industry and I bring those mistakes into the conversations I have with customers on a daily basis and start thinking about things from a different perspective. How do we stop repeating these mistakes? How do we think about things from a different point of view? How do we do things differently? And as the whole topic is today around AI, it very much centers just now around how we actually using AI within these spaces in less being fearful of it, but using it to our advantage to move forward.
- [05:56]
Nathan Isaacs
Yeah, and I think you know the reason why we're bringing you on. All three of you regularly have one on one conversations with whoever might be out there in the world, whether they're user testing customers or they are prospective customers, sometimes competitors. You know, you're just trying to get a sense of what's going on. We then deploy you to help all these customers and stuff like that and help figure out the problems that they're having or avoid the problems that you've experienced in the past or other companies have experienced. So you, you have your pulse on what's going on out there to whatever extent that anyone can. You have the crystal ball. And so I'm wondering, you've all spent the first half of 2026 talking with these enterprise leaders, attending industry events like Research Week or our own crafted that we had in Seattle just a. A few weeks ago, and working with teams across design, UX and research. What's one topic or challenge in addition to AI that seems to come up in almost every conversation you're having right now. Let's begin with you, Mike.
- [07:06]
Mike Mace
So, you know, I like the framing you said about us being at conferences and stuff, because there is a striking difference between the intense online discussion that you get off of the major podcasts and, you know, the, the big influencers and stuff, especially people within the tech industry. You know, anthropic in particular is very, very noisy. They've, in a good way, they've empowered a huge number of their employees to speak. But when you're a very young company, you know, this is amrit. It was really resonating for me when you mentioned, oh, you've been in the industry for a while and you can see current mistakes and things like that. We get this perspective from very bright, very young, relatively inexperienced people early in their careers who are aligned with the new technology. And what they tend to do is get very, very excited about their theories of what they believe is going to happen. And you get these theories that are being shoved at us from these very, very smart people. And sometimes they're great, they're well worth listening to, but sometimes they're like, that isn't actually how the world works. And one of the intense discussions that's going on right now on that theoretical basis of the influencers is all about, what's the new bottleneck? Right? We've accelerated engineering so we can produce more features. What is the next bottleneck in, in being able to accelerate our stuff? And there are people who say, oh, it's, you know, this other part of development, or it's, you know, maintaining the code as opposed to creating the code. You know, it's, oh, technical debt is building up and they're passionate about all this stuff. And from my perspective, there are some quieter voices that have been saying, hey, you know, actually, we'll probably work out Those other internal bottlenecks as well. The big issue is how do you get customers to absorb change? You know, so you're adding a new feature every week. Do the customers want that? Can they deal with that? Can they learn about that? And to me, it's only a couple of people who've talked about this, but that one resonates as you like. Yeah, been there. And I think that's absolutely true. And the thing is, all of us creating AI tools, whether we're in the tech industry or other companies doing AI experiences, how do all of us help the customers accelerate their ability to absorb change? And that to me feels like a very lasting, critically important issue that would really be great to talk about intensely. And that discussion is not happening yet. And I in for the most part and I really want to see that happen. So I don't know. That's the one that's been eating my brain the most lately.
- [09:59]
Nathan Isaacs
You mentioned that you. We just had, two episodes ago on the show we had Katie. You had a conversation with Katie Robbali.
- [10:07]
Mike Mace
Yes.
- [10:07]
Nathan Isaacs
And. And she brought that up too, right? Yeah. For every new feature, you have to have new check writing about it. You have to have, you have to educate your CSMs about the problem. You have to. If they're CSMs, you also have your customer care people because somebody's going to be calling or trying to reach out and with a problem about it. And I was like, oh my gosh, I hadn't thought about all that kind of stuff. It's crazy. Em, what, what, what for you, the same question, like, what's the, what are the big, the big thing for you that you're, you're thinking about a couple
- [10:40]
Amrit Batu
of things, if I may. The first one that, that really doesn't have much of an answer just now is that expected skills loss as we move forward in this space. If, if everything's being driven by AI and we're needing people with experience to train these AI models in all these different places, there's little space for junior people to come in and take over some of these roles. So as we get to the end of our careers and the likes of us get to that kind of point, who's going to take over from us? Will AI know enough by that point in time? How do we make sure that, as you said there, Mike, when you've got these young stars with their own theories, etc. How do they make sure that they're able to actually fit into the ways of working from that kind of point of view? I think that's a really interesting one. And the other one that is coming up more and more in the conversations that I've been having is the governance, the governance around all these models and how it actually has an impact on the decisions that are made. Now we know that from an AI point of view, you can put the same query into the same model and get completely different answers that's amplified across multiple different models. So if you put something out there that isn't quite right but ultimately violates some kind of governments or legal terms, how do you combat that kind of stuff? So pace is brilliant, the speed is brilliant in many cases, but how do you make sure that you're protecting yourselves as well?
- [12:24]
Nathan Isaacs
The. And in Europe, right, There's so many here in the US a little bit with privacy issues and stuff like that. But in Europe, it's, it's there and we're aware of it. You know, you got GDPR and, and things like that and, and who's to blame? Who's to blame if AI screws up and you roll that out and now you're looking at a massive find, right? Especially if you're an American company trying to barge your way into Europe and you're making a mistake and you're like, oh, sorry, it was, you know, whatever. Frontier models fault the. Leah, what are you hearing out there? What are the thoughts?
- [13:01]
Leah Hogan
So I think it's really kind of a. There, there are a couple of big things that I would add on to what it is that we've heard already. The first is I think that we are starting to come to the end of the era of token maxing. So we are running a gun up against the practical limits of the subsidies that the frontier model companies have been providing to all of us to get us adopting this new technology. It's not free. And one of the assumptions that we've built in as leadership teams is that this is an unlimited resource that can replace people. And we're starting to understand that actually in some cases it's less expensive to run with people. So that has a practical impact. We are just now, in some cases, in some disciplines, getting to the point where we're saying, let's figure out where to use it. Leadership is not providing guidance around what are the use cases, what's in and out of bounds, which models are best to use for which use cases, and then what are the best practices for using AI efficiently and responsibly. And so a lot of what it is that I am doing is coming up with, so what does that look like for UX researchers, you know, we have to consider how best to identify use cases where there is a true value to involving AI in our thought processes. And as well really do some deep thinking about, like, what does the future of work look like for us as people? What do we enjoy doing and what are we good at doing? What are we better at than the technology that's out there? I think it's difficult for us because humans have always been very hierarchical and are more used to leaders saying, all right everyone, this is what we're going to do. Here's how we're going to do it. These are the policies, here's how it's all going to work. We are not seeing that type of leadership from most organizations teams right now. What we're seeing is what democratization really looks like. Is everybody trying their own thing and then giving each other advice about what's working working and what's not working. That's expensive to Amrit's point about governance. And it's inefficient because it basically means that people are probably replicating the same experiments over again and not learning anything new because they aren't even aware that someone else is kind of doing the same thing. So lack of coordination. And then also it really is just everybody is experimenting so much. Like, can we do better to be more intentional about picking those bets to try first as a means of being very responsible, especially from an environmental perspective, electricity use, water use, to find those things that actually make sense for us to use the technology to do rather than make cute cat pictures.
- [16:43]
Nathan Isaacs
Well, Mike and I have a colleague in our team that does those cute cat pictures and we're wondering, like, when you have to actually spend 50 bucks for that picture, then are you still going to be making those cute cat pictures? Right. Going back to what you're saying on token maxims, it reminds me of like I can, I can say I'm going to build a dresser, but after 10 trips to home Depot, I have to stop, right? I have to say, okay, Nathan, you actually don't know what you're doing. You can't keep buying wood and your wife's going to kill you. So the I, I, I wonder. We, we've been a year ago if we had this podcast, we, we say AI is a hot topic. AI is still the hot topic. What's changed in that last year? What, what, what's different this year than, than this time last year with regards to AI and stuff like that? Emorett, do you have a thought on that?
- [17:36]
Amrit Batu
Yeah, it kind of comes off the back of what Lee has just said there as well. I think a year ago, the big thing was the fear of jobs, and we've seen some of that play out with job losses across the industry, etc. I think what we're hearing of more now is people looking for. They've been told to go and experiment and do as much as they can with AI and see where it fits in, but they're now at this place where they're looking for problems to a solution as opposed to solutions to a problem. And this then creates that inefficiency, et cetera. So that's how they're seeing if they can prove their value, by just doing stuff in AI rather than worrying about what their job security is. And that's how they're creating that job security for themselves. So I'm hearing that and seeing that a lot more across various different conversations.
- [18:36]
Nathan Isaacs
Leah, what's changed in the last year?
- [18:41]
Leah Hogan
So everybody, the new buzzword is agentic. And I just had this conversation with someone earlier today where they were making the point that, you know, everybody's talking about mcp and what does that mean for the future of how it is that we interact with all the tools that we typically these days are using on screens. And I think I am starting to come around to the idea that because of the lack of coaching and expertise that is essentially building with especially younger people, that we are going to have to, as designers, make agentic systems, be coaches, because people will not necessarily have developed the taste and expertise to know the difference between slop and quality. And that is, I think, problematic in one sense, because it means that we're seeding as humans some of our. Some of the value of what it is that we bring to the equation of using technology to essentially systems whose motives are a black box, whose ways of working are a black box, even to the people who are designing them. And I think also it means that we risk. And I think everybody has seen this, you know, model degradation after, what is it, six generations of retraining, right? You just get like slopped, right? You're trying to slop with slop. We are going to have to reckon with what. How do we ensure that we don't start to create ever degrading experiences, products, information, because we're handing over experiences to technology, to LLMs. I don't think we have a right answer yet, but I think we have to, like, be really smart about how it is that we think about creating our agentic future.
- [21:01]
Nathan Isaacs
Leah, I want to Go to you, Mike. But Leah, it reminds me of a story my wife talks about. She's an arts education coordinator for the city of Portland and coordinates with seven school districts, if I remember correctly. And one of those districts signed some. There's a threshold where you can just go ahead and make a purchase without having to go to the board and all that kind of stuff. They did it like one grand below that for an AI product that was going to help kids and literacy and understand all that sort of thing. And it was like this comic book program. And you know, after a year, they. You see the examples of what these guys, these students did, and it was truly, truly AI slop kind of stuff. And, and her point of view is like, she's looking at this. It's like, if you had given me that amount of money, $148,000, I could have put art, actual comic artists in all these different schools coming on a program, you know, and, and actually teaching these kids how to be creative and, and have interaction, stuff like that, rather than trying to figure out a prompt that gives you a bad prompt and a bad prompt, and then you have, you know, some comic with a, you know, sho foot in his head or whatever it might be. Anyway, Mike, what are your thoughts on, you know, what's changed in the last year? That's. That's really big.
- [22:33]
Mike Mace
Yeah, man. It's hard to even think back that far because of the. I had to think about it for a minute. So I really resonated to what Leah said about agentic. A year ago, people were saying, oh, here's, you know, conversational AI and the next big thing is going to be agentic. And it was kind of like, well, what do you mean by agentic and what's all that now, at least for those of us who are embracing it a lot, I think we understand a little better what agentic is. And in some ways it's made a bunch of progress. And it's almost like you don't even call out agentic. It's just that's what AI does, is it can do some tasks on behalf of you. On the other hand, we're finding some things that I think are profoundly scary. Not in the, like, scary it's going to end civilization, but scary in the sense of this is really powerful technology and you maybe don't want to trust it too much. If you go on LinkedIn and you search for anecdotes of people talking about using the AI tools to do things for them, and you'll see posts about People saying things like, wow, this is super powerful. I'm using, you know, whichever model, I don't want to call out one of them because it's all of them. I'm using it to do all this Agentix stuff and it's really great. It's saving me a huge amount of time. And it ran wild and it sent erroneous emails to every single person on my email list. And, and that's okay because we should all understand that it's all right to do that occasionally because we're, it's in service of increasing productivity and that's all right. And I was thinking that's all right for a very small number of people who are at the bleeding edge and are willing to make that trade off. But I think to most of us, especially in a company I know, if my email went wild and sent emails to everyone that I have contacts with, including all the customers I talk with, that would not be accepted very well by my employer and I would be blamed for letting that get out of control. And so I think working through the implications of what, what is smart to do with Agentic and what is not smart to do with Agentic, and that sort of stuff is interesting. I think there's an. Something we all need to be thinking about is am I going to have one central agent, one infobutler, which would probably be with your favorite general purpose AI thing. Like, you know, I really like Claude and I built up this whole thing with Claude and I want it to be my interface to the entire world. So I don't want to engage with your company about buying a product from you. I'm going to let Claude use mcp, you know, by the way, model context protocol. In case you're not familiar, it's basically how AI talks to other AI. I'm going to let Claude interact with every vendor that I buy from. And as a result, Claude is my info butler that does my entire relationship with the entire online world. Or will I engage with your bot? You know, whatever you are, say you're, you know, a particular retailer or something like that, you've got your own bot that you want people to talk to. Will customers even come to your bot in the future? Or are they going to work through their own bot talking to your bot, Profoundly different customer experience. Which one's going to win? I don't know. It seems to me like it's really, really important that we all figure that out, which means engaging with customers about how are they using AI, how do they feel about it, how do they feel about using your AI, all of that sort of stuff. So that one's a big one. I think the other one where we're getting some more clarity compared to a year ago is this whole builder model thing. You had some in the spirit of people in inside the industries, the insiders getting into big debates. There was this thing about we're not going to have product managers anymore, we're not going to have engineers, we're not going to have researchers, we're just going to have builders. And I think what we're coming toward is a realization that that was too much a fantasy. There is. There are certain skill sets that you need. You need design expertise and taste. You need research expertise and ability. You need product management, that it's its own set of skills that need to be taught. And the idea that you can do without those skills is crazy. Now, you may get some people who have multiple skill sets, that's okay. But we need those distinct skill sets. They add value despite AI. And in most companies, that means having separate employees for those skill sets, because that's the easiest way to develop those skills and manage those skills. So, yes, you'll have people who can build, but that doesn't necessarily mean their job title is general purpose builder. I think they're still going to have areas of specialty.
- [27:23]
Nathan Isaacs
I think you're. A lot of what you were just saying there kind of speaks to the, the need for a human in the loop. This, this and, and you know, into those skills. You know, a researcher in a loop, a product, you know, owner and builder and, and market owner in the loop. How do you do that? Right. A lot of people are saying, well, you guys are just, you're just saying you want that because you're afraid of AI replacing your job and you're going to push back on it that way. What would anybody have any thoughts on that or. Leah?
- [27:56]
Leah Hogan
I have a strong feeling about this. I think that we have that term flipped. I think it needs to be AI in the loop because the person. There should be a person in the driver's seat and human in the loop. Sounds to me like humans are taking a back seat when that's not the case. Like practically, it should never be the case. It cannot be the case. And actually for very good reasons, especially because for the purpose of accountability, AI will never be accountable. So it's essentially, how do we ensure that humans drive the decision making, the prioritization, the application of taste, and really make some choices in an educated way about how to ensure that humans remain at the center of what it is that we make and build and create. You know, I've been working through many old economy things with elderly people over the last several years. And I'm sitting here and thinking, oh, customer service and, and you know, replacing people with, you know, chatbots. There's no way that anybody over the age of like 70 for core things like wealth transfer or setting up your cell phone will ever go to a computer to do those things. Partially it's because they don't have the judgment, the empathy, the ability to make the right choice when there aren't rules that can be followed, when there has to be a judgment call. A lot of what happens in these conversations is about applying judgment and empathy. So I think that there is still, for some of the mechanical things that can be automated, makes perfect sense. But for those high touch, high empathy, real world interactions, we still will value humans, even though they might get an assist in the back end from some technical system that's making something work more efficiently. Like rebooking a flight, for example. I know we would all love that to happen so much more quickly, but you still want somebody to be there and hold your hand and say, I'm so sorry that you're having this happen to you. Here's your next flight.
- [30:43]
Amrit Batu
Yeah.
- [30:44]
Mike Mace
Or even if AI can do it, you get a competitive advantage as a company by being able to give it that personal edge and touch. And so it becomes a differentiator. When everybody can throw AI at anything, the people who can create a more personalized, more compelling experience, a more humane experience, that actually becomes the big differentiator. It's funny, if you can automate all the process stuff, it's the human touch. And I sound like a self serving. Okay, I acknowledge as a company that does human insight, I sound very self serving about this, but I really believe it. It's why I work in this particular part of the industry. And if everybody can build everything, then it's how do you make that humane? How do you keep in touch with the human beings? Because that becomes the differentiator that's hard for anyone else to provide and that becomes a basis of lasting value if you can figure out how to build that.
- [31:43]
Amrit Batu
Right. I've been using some terminology or trying some terminology with some customers recently and it seems to be resonating. The language that's being used just now is speed. AI brings speed into what we can do, but doesn't necessarily bring velocity in what we can do. And we know the difference between speed and velocity is direction. Ultimately, if we are running at a really fast pace in the wrong direction. It doesn't matter. The human is what brings that direction into that decision making. Both the examples that Leah and Mike have given there, the human brings the direction into decision making that's happening there. So for me, you've got the AI and the human working together. The AI is able to, you can have all these agents working together to deliver incremental improvements, but if you want to innovate, you need the human there to point the AI in the right direction. And that's how you get that, as Mike's kind of put it really well there. That's how you get that competitive advantage. That's how you get advantage over other industries. And it's not just simply putting a human in the seat either. I think, I'm sure you guys have had the same experience as I have, is if you phone up a call center really angry about something, your flight's been delayed or cancelled, and you speak to a person on the other end of the phone and all they're doing is typing in queries to a system and giving you that scripted answer, it increases your frustration. So the human needs to be able to make, as the guys have said there, that judgment call. A great example as I was, I've just been across the States to see Scotland in the World Cup. We did okay, but when we, when we looked at it from, I booked extra legroom seats. I've had operations in my legs. I need a bit more space. Out of nowhere, the airline, and I won't name the airline, canceled my seats and put me just in a, in a standard seat. And when I phone them up and explain the situation to them, they didn't have a way of resolving it. For me. The only way that could be resolved is if I paid more money. Despite it being something that changed on their side and not something that I'd introduced myself. And again, this is where I'm like, the human touch is really important, those kinds of scenarios. And I'm interested to see how people actually take on that challenge and drive forward with that velocity rather than just pure speed.
- [34:36]
Nathan Isaacs
And it goes back to a story we've, we've said as a company and our various identities of different companies that we've all been through, you know, user zoom, user interviews, user testing. You have the one and bad experience, you're not going to go back. Right. Regardless of your AI. And, and I love, Leah, what you're saying, the AI in the loop. Right. Because I will admit I run through, you know, My process of producing the podcast and I'm using the AI to do it. And, and I'm like, okay, it. I'm supposed to go and check to make sure it was right. I'm like, well, it's right most of the time. I won't check this one time. And you know what? It made a mistake, right? It called somebody's name incorrectly or, or gave them some sort of weird name. I'm like, ah, I really need to be more present in it as a parent. I think if you're a parent of a child, you say, yes, yes, yes, yes, because you're paying attention to something else. But when you're present, all of a sudden you realize that your kid is eating paste and you can stop them before it's too late.
- [35:46]
Amrit Batu
Just add in there really quickly. Nathan, there's a quote that I've been using as well with some customers from Warren Buffett and he speaks about it takes 20 years to build a reputation and five minutes to ruin it. So we're at that point now where if you do things, if you think about that, you'll do things differently. So you've got to take that reputational risk into account when you're making these decisions of AI versus human decision making.
- [36:15]
Nathan Isaacs
And to a degree, right? I mean, we're talking about it from our perspectives and, and, and our, our customers perspectives, but there's this big macro thing going on, right? And the, and they're saying the whole world is going to change. Believe us, the whole world's going to change. And for leaders, they've been operating on playbooks that were written 50 years ago, right? All the different business management books and stuff like that. And they're like, oh, there's no playbook for this. We don't know how the world is going to change in a business. So I'm just wondering, like, as we head into the second half of the year, what, what are your thoughts on what companies should be thinking about? What should they be doing? What are some actionable things that you recommend? Mike, let's start with you.
- [37:03]
Amrit Batu
Yeah.
- [37:03]
Mike Mace
I've got one that I want to put on people's agendas and this kind of came out of going to research week. So I'm really proud and excited about the job that the research community is doing on talking about how to absorb AI into their workflows. Intense discussion about that. There were among the speakers some very brave people who were standing up and saying, here's what I'm doing. Here is my tech stack that I'm using Here all the things that I'm doing, feeling super empowered. And it was really, really neat to see. We need to bring that same energy and exploration to a discussion of how we evaluate the experience of using AI. So it's not about using AI in your workflow as a researcher or product manager, designer or whatever. That's kind of self centered. It needs to happen. But there's the. How do we evaluate the experiences that our AI is creating for our customers? And in particular, most of the testing that's coming out of the AI development community is about testing the correctness of AI answers. Do they, do they give a predictable correct answer to a query? And that's very, very important. It's called evals. Super important. There's been very, very little thinking about how do we evaluate the unique aspects of AI and make sure that it's good before we give it to customers and create a bad experience. And it's all about testing and optimizing for relationship and personality of the, of the AI bots. And nobody's used to testing for that because you don't have to do it with traditional software. Traditional software is a tool you test. How easy is it to find the commands within the tool and do those things. AI is a conversation, at least the way it's being implemented today that most of us face. And a conversation is all about how's your personality, how's your engagement, how's your relationship. That needs to be tested very rigorously on all the AI products that we're throwing out at customers. And in the companies that I talk to, there is almost zero amount of thinking about that. It's all focused on the correctness of the answers that urgently needs to change because that's what will drive customer adoption.
- [39:32]
Nathan Isaacs
Leah?
- [39:33]
Leah Hogan
Yeah, you know, I think that's an interesting point because, you know, part of it is there's, I'm, while I'm not an engineer, there's a small part of me that's just like, does it work yet? If we're still working on evals, does it even work yet? And why are we throwing it at scale at people if we can't even confirm that we're consistently getting good results with it? Because you know, to me, if you're thinking about like Maslow's hierarchy of need, it's got to work first before you kind of put the polish on it, or else you're just putting lipstick on a pig. So, you know, I, I agree. I do believe that there will be. I don't think that we should limit ourselves by basically saying we've got to existentially get this thing working first, get the technology working first. We do need to plan for a future where it's going to be working and what do we actually want it to do, trust it to do. What are the use cases that make sense? I feel like we need to, as a discipline, and here I'm speaking as a researcher, focus on the core value that we've always brought, which is what, what do, what are people's goals? What do people really want and need? And rather than focus on the let's throw all the spaghetti at the wall and see what sticks. Get much more intentional about it and again, flip that conversation and say, what problems are we trying to solve for people? Because we've built this really powerful technology, but we still have don't have a good sense for what value people can get from it yet. And I say that because it's not really a matter of perception. A lot of, I mean, we did not know that the smartphone was going to be as powerful a paradigm shift in terms of interaction as it has been. And I think that, you know, especially some of the research that's coming out over the last couple of years or last couple of weeks talking to us about, well, the rise of smartphones seems to be related to the decline in birth rates. Why? So, okay, now we've got, you know, some potential unintended consequences. If we want to create something that's human and that promotes connection between people, happiness, what can we do to ensure that our technology brings that it really is, I think, understanding what people want and need as we have these increased capabilities to deliver these highly personalized experiences to more people.
- [42:35]
Nathan Isaacs
Emmert
- [42:38]
Amrit Batu
Similar to what Leah's been speaking about there at the moment, and we've all kind of spoken about it on this podcast so far as well. Everybody is so in the mindset of how can I build stuff using AI and how do I build the next thing, how to do the next thing? And we almost forget about what AI is really good at just now. It's good at getting to information quickly, getting to data quickly. And for me, for the sort of second half of the year, what I'd love to see within product teams, certainly from a UX research, from UX design perspective, is using that ability to get information, to get data quickly, to close the communication gaps that they currently experience within those product delivery methods, processes, and owning that conversation because we more. More that we own that conversation, the more that we can be involved in strategic calls and what happens with it and then that naturally brings within that organization the human to the center of that decision making process. So I'd love to see it happen more. Like I said, less time building, more time using what's already there to our advantage.
- [44:01]
Nathan Isaacs
Yeah. It speaks to like the need for continuous discovery. Right. This, this idea of. To all three of your points, right Mike, is, is how is AI changing the experience because we've adopted it in our own products or something like that in some way. So how is that. How are people experiencing it? But also how are our customers. It's changing their expectations when they come and use our products. So we need to understand that we have to just understand what the jobs that we need to solve are and are we doing that and then emorett to your point, what are the jobs that we've ignored or because we just didn't have the capacity to. To mine all that data. Right. This is the true. Like if. Because nobody has time to watch 10,000 hours of video or sort through all those surveys and the fill in blank stuff. That's, that's why there was no fill in the blank questions on surveys in the past because nobody had time realistically to review all that stuff. And now we can. So we need more of that. We need more of that discovery type of stuff. Any other last thoughts? Any. Anything. Anything. We haven't talked about any. What's your, what's your predictions for 2027? Anyone?
- [45:30]
Mike Mace
That's. That's six months out, man. That's beyond my planning horizon.
- [45:33]
Nathan Isaacs
I know it's got to be six months. Even six weeks is too far ahead.
- [45:38]
Mike Mace
Well, you know. Or go ahead.
- [45:40]
Amrit Batu
Yeah, I was going to say I think going into 2027 we're going to see a bit of a bounce effect in terms of how people are used within organizations. I think we're at this point at which like we keep on saying everyone's so focused on what AI can do. I think as we head into 2027, probably the second half of 2027, we're going to start thinking about right now we need people to actually drive what we need to do drive. And I'm hoping it doesn't take that long to get that realization. But I do think it may take that, that sort of amount of time to get there.
- [46:16]
Nathan Isaacs
That that's a good point to end on the middle way. Right. It's never any extreme. It's always somewhere in the middle. And that's about everything in life. If people have questions if they want to learn more. Any recommendations for anyone Talk to us. We have Crafted London happening in September, so in a few months, please come to. You can just type into whatever search engine you like crafted and user testing, you'll get to that page. Leah, any, any thoughts, any advice on where they should go or learn more or anything like that?
- [46:58]
Leah Hogan
There are so many, I think, podcasts out there from people who are, who are talking about the technologies and the players and everything else. Like, honestly, I've been listening to Hard Fork a lot.
- [47:12]
Nathan Isaacs
Hard Fork is one of my favorites too. Yeah, that's a good one. To recommend any other recommendation podcast.
- [47:17]
Leah Hogan
Specifically, there is John Whalen, who I interviewed for this podcast, who I think, I think is truly a thought leader in the especially UX research and AI space. Just, he's got some really great content out there, both in his podcast, which is I think called U or AI for ux. And that's a great podcast to listen to because he's talking to so many people, but he's also just a delightful person.
- [47:53]
Nathan Isaacs
Well, and he speaks. We've not talked about this on this episode, and I'm afraid to bring it up, but is a lot of synthetic AI, you know, conversations and stuff like that. So we'll, we're gonna hold down that to another episode. I'll bring you guys back on in a month or so. We could talk about Synthetic Mike, any, any recommendations on where people should go or talk about or podcasts or anything like that?
- [48:20]
Mike Mace
I'll tell, I'll tell you my recommendation. Well, number one, if anybody wants to talk about this stuff, they can reach out to me directly. I mean, part of my job is to have conversations with customers. So I'm mike@usertesting.com Mike. Easy to find me. Glad to hear from you. In terms of the stuff I'm following, the difficulty I got into is there are so many good podcasts and substacks that I want to follow that I wasn't able to do it. So one of my projects with Claude Cowork was building out something that would ingest all of that stuff and summarize it for me, not to replace listening to all that stuff, but to let me find the things that, that I want to go spend time on and focus on, like new ideas and stuff like that. And so far the jury's still out, but it's the only way. Like you said about watching 10,000 hours of video, I can't listen to the number of hours of podcasts that I want to listen to. And so building out something that would scrape the transcripts and tell me what's going on has actually been a good exercise, but it also been just defending myself against how do I stay on top of this discussion when it changes so fast. So if you've been thinking you want to mess around with cowork and see what it can do, that's a, that's a good project or co worker or the equivalent. I'm not trying to take sides. Building that out can be a good exercise and also it helps you stay on top of. Of the huge number of voices that you really need to watch.
- [49:51]
Nathan Isaacs
Right. It's important to be paying attention out there. You know, like you may have various reasons why you don't want to do anything with AI, right? Personal ideological reasons or anything like that, but if you ignore it, you're just going to get left behind. But also there's just so much noise out there. And these podcasts, I mean there are AI podcasts, right? They're just kind of like regurgitating the, you know, slop on slop on slop. So yeah, there's, I'll share this with you, Mike and I have to track it down. The guy who, you know, does a YouTube channel around AI products, right, he's a product guy, but he had to build his own sort of way to sort of vet all that information because part of his shtick is telling you what the latest and greatest is and how do you, how does he separate, you know, the signal from the noise there? Emorett any, any recommendations, any podcast recommendations, e newsletters, anything like that?
- [50:53]
Amrit Batu
Yeah, I was, I'm going to go slightly on a different direction here. So rather than focusing just on podcasts and I think there's a lot of fear mongering that comes from the podcast that we listen to as well. Get to as many community events as you can get, speaking to your peers as often as possible. So not a needless plug here, but crafted in London is going to be a great opportunity for that. Get to some local events, get onto message boards, et cetera. Have those conversations with the people that are in the same space as you because they're going to be experiencing similar challenges to you. And that's where I feel you're going to be able to learn quite a lot in terms of how to approach those things in different ways and go back and share again with them. I think the community aspect's kind of being missed out a bit just now. Let's try and make more of that in general. And as with Leah and Mike and yourself have said there, Nathan, if You do want to reach out where we're all available, whether it's through emails or LinkedIn or anything else. Hey, maybe we can do a Q and A session at one point with some of our customers as well. Let's just get those conversations going like that.
- [52:00]
Nathan Isaacs
I think that's a great idea. We'll try and plan that for later this summer. And we can have a live show with Q and A and we can talk about synthetic users and all that. There's all the links to us will be in the show notes and to these different things. And then just another pitch again on crafted. You don't have to come, but the, the back half of it is free. There's some workshops you have to pay for, but it's definitely. We're gonna feed you. So right there it's a. You look at that. It's like a 10x ROI there. So. And then speaking about community, not just crafted, I think back to the WordPress unconferences. Am I getting that right? When you just showed up and you're kind of like, hey, I'm really curious about this, I'm curious about that. And then you had other people in there and you kind of like all noodle together and figure out best practices and I think we need more of those out there. And I'm sure they're out there, right? So just go ask AI. And with that, I appreciate everyone's time today. Thank you all so much.
- [53:12]
Mike Mace
Thank you.
- [53:12]
Leah Hogan
Thank you.
- [53:13]
Nathan Isaacs
All right, this is audio show, so you have to say yeah. Thank you, Nathan.
- [53:18]
Mike Mace
Thank you, Nathan.
- [53:20]
Nathan Isaacs
Alrighty.
- [53:21]
Podcast Narrator
Want to keep the conversation going? You can find the show notes@usertesting.com podcast if you haven't already, don't forget to follow us on Apple Podcasts, Spotify, Overcast or Google Play, so you never miss an episode. And if you enjoyed today's show, please share it with a friend or leave us a rating and review on Apple Podcast. And until next time, this is Insights Unlocked, an original podcast from User Testing.