Tracy Alloway (2:24)

No. And we've certainly seen various efforts to, I don't want to say catch up, but like to keep pace, I guess.

Joe Weisenthal (2:30)

Keep pace.

Tracy Alloway (2:30)

Yeah.

Joe Weisenthal (2:31)

So of the major legacy companies, you know, which I would say are like Microsoft, Amazon, Meta and Google or sorry, Alphabet I would say clearly like Alphabet is the one that has the model that people are talking about. Gemini. Right. And in a way that's kind of surprising because one common intuition that people have is that companies aren't very good at developing the thing that disrupts their own legacy business. Right. This is just a famous sort of B school Thing that people talk about all the time.

Tracy Alloway (3:02)

Yes. So the issue, one would think, with AI and Google in particular is that Google, you know, famous for its search. No one believes me. By the way, the first time I ever used Google in, like, I guess it was the either late 1990s or early 2000s, I told my parents to invest in the company and they didn't listen to me.

Joe Weisenthal (3:20)

Oh, really?

Tracy Alloway (3:20)

Yeah, they don't believe me either, but I swear I did. The first time I used it, I was like, oh, my God, this is so much better.

Joe Weisenthal (3:26)

So I feel so dumb because I remember the Google IPO very well. And I thought I was very smart because it was like I was 20. I think it was 2004. I was very. I thought I was very smart back in those days. I don't think it now. Now that I'm older, I realize I. I don't know anything, but I thought I was very smart. I was like, oh, these sheep, they're just buying. It's overvalued. Should I short this ipo? It's like a bubble, et cetera. Thank God I didn't, like, put on some short trade and go bankrupt. But, like, I. It never occurred to me to go long because I don't. And I think it's the journalist temperament. We're all, like, very cynical.

Tracy Alloway (3:56)

And that's. Right, you have to be an optimist to be an investor.

Joe Weisenthal (3:59)

Yeah, you have to be an optimist. You have to be willing to be part of the crowd. You have to ride the wave. We don't really like riding waves, etc. And so, like, I was like, oh, should I short this? Anyway, I should have bought it. Okay, well, I should have held it. And you know what? I should have bought it. Even though we don't trade like, should have bought it in, like, 2023 when everyone was saying that chat GPT was going to eat its lunch. Should have bought it a year later when people were saying, oh, Gemini is too woke, et cetera. All these turns, there are opportunities. But anyway, I'm happy to just talk about it.

Tracy Alloway (4:29)

All right, now that we've gone on a very long segue, what we are getting at.

Joe Weisenthal (4:32)

It's a short segue.

Tracy Alloway (4:32)

It's an important segue. What we are getting at is that in theory, AI would seem to pose a threat to Google's core business, which is search. So if you type in a query in Google now, I think I have one open from the last time we recorded a podcast. I don't know why I was Looking this up, it says, can you see tankers physically from the Strait of Malacca? Oh, yeah, in the Strait of Malacca, from Singapore. So if you type that into Google, it used to be you would just get a bunch of search results.

Joe Weisenthal (4:58)

That's right.

Tracy Alloway (4:59)

Now you get an AI overview, which basically pulls in a bunch of results from other pages and gives you a sort of summary. And so the question is, if people are just going to be looking at these summaries instead of actually going to the pages that Google search used to turn out as links at the top of the page, what does that mean for traffic via Google Search?

Joe Weisenthal (5:19)

Well, and then the other big element here, which is that, you know, people stop like clicking as much on links or even seeing links, what does that mean for the advertising business if the expectation is that you, like, see the answer right there, etc. And one of the questions we have about OpenAI in particular is are they ever going to be able to launch, like an advertising business? Can these two things combine, etc. We know that to this day that Google search ads are like the greatest money printer that's ever been invented basically in the history of the world. And so how this is going to interact and how Google is thinking about these questions, as you say, it's a little unclear to me because, yeah, it's nice. I could just put in the same question. Can you see tankers in the Strait of Malacca? I don't know if these are right. I don't know if the AI. I don't know if the AI overview is right, but it's there, it's there.

Tracy Alloway (6:08)

So lots of interesting questions here also. AI slop, right? Are the search results that are being turned up actually of any certain quality? Right.

Joe Weisenthal (6:17)

It's a great point. And one of the reasons perhaps that a lot of people and I would include myself in this use AI more and more, is because I have like, you know, I have some issues, let's say, with search.

Tracy Alloway (6:30)

So anyway, I'm going to clip that. I have some issues. Joe Wiesenthal.

Joe Weisenthal (6:34)

Anyway, we really do have the perfect guest to talk about this. Someone who is like really literally right in the middle of all of this and can answer all of our questions. We're going to be speaking with Liz Reed. She is the VP of Search and at Google, and she has been at the company for over 20 years, has been in the current role, has been on the search team for a few years. And so we're going to hopefully get answers to all these questions. So, Liz, thank you so much for coming on the Odd Lots podcast.

Liz Reid (7:01)

Thank you for having me. Delighted to be here today.

Joe Weisenthal (7:04)

Absolutely. Why don't you just start by telling us, like, what's your role at Google? What does it mean? Okay, you're the VP of search at a company that people know for being a search engine company. But what is your, what does your title actually entail?

Liz Reid (7:17)

I lead the search team that you can think of as covering the product, the engineering, our user designers and data science, the team that fundamentally builds the search product that you use.

Tracy Alloway (7:28)

So how much of your day to day is taken up thinking about AI nowadays versus, like, let's say two years ago?

Liz Reid (7:35)

Well, two years ago, I would say it was also still a fairly large amount. I think AI is a deeply transformative technology in what it opens up, and I think AI has been in search for many years in different forms. It's much more in the forefront these days with things like AI overviews and AI mode. But if you go back several years, it was how we transformed a bunch of ranking with efforts like Bert and Mum that were built on some of the early transformer breakthroughs. But at the end of the day, and if you go back, once upon a time, AI didn't just refer to generative AI, it referred to general machine learning and other things like that. And in the early 2000s, Google had spell correction, which felt sort of revolutionary at the time that used AI. Right. No one calls spellcheck AI anymore. Yeah, nobody calls spellcheck AI, but like it was at the time, right. It just shows you how far the world has come. But the opportunity to really transform search and realize Google's mission at a new level is really exciting and an amazing opportunity and very humbling to do this for a product that so many people use.

Joe Weisenthal (8:39)

By the way, you mentioned Bert, my little software hobby project of training a machine learning model to tell whether something is more indicative of the written or spoken word is based on Bert. So thank you for developing that and thank you for open sourcing it so that someone like myself can train it. But talk to us about just how you're thinking about this core tension that for years Google had a business of you go to a. You put in a term in a search bar and then people click out. And some people, and some of the clicks were to organic results and some of the clicks were to paid results. And now people are, we're entering this world in which people expect to get whatever they want right there from the query without that impulse for a click. And you have a business that is still dominated by that click out one way or another. So just like, you know, and I want to drive into the details, but big picture, is that a real tension?

Liz Reid (9:36)

I think what's interesting about it is that the space of search is very big and what people are trying to do is very big. And sometimes people really want quick answers and they want it right in front of them. And sometimes they want to go deep or they want to hear from particular individuals. Right. I think there's this sort of myth that people want AI or the web that I actually think what we see is that people want AI and the web together. Okay. There are certainly questions for which like, you just want the quick answer and, and then you're done. And that's been true in many ways for years. We'll talk about this with AI, but I bet most of the time you look up the weather, you just want to know what the temperature is and you're done and you don't once. But then you're going to go on a trip and you're going to go and actually dig in more because you're going surfing on other pieces. I think if you think about this, people are like, oh, I have an answer. I don't. So why would I do an ad? Well, the answer doesn't buy the pair of shoes. You actually have to buy the pair of buy the shoes. So you still have to go pick a merchant for that. People care often to hear people's perspectives, you'll talk about, okay, well, we want a bunch of answers. And yet this is a golden age for podcasts. So clearly people sometimes want to spend a couple of seconds and other times they'll spend a whole hour listening to things. And so one of the things we see with the shift with AI overviews is that you get more of this pronouncement with what's your goal? If all you were going to do was go to the webpage, see the fact, and immediately click back, you're going to spend like a half a second on the page. You see those things shift. But if what you were going to go and do is read an article for five minutes, you're still interested in reading that article for five minutes. AIO might help you point to the right page so we see fewer bounce clicks where a user would go and immediately come back because they weren't happy. You see people go though, and they want to hear from other people. They want to hear their expertise, their perspective, their unique take. I take fashion as an interesting example. Sometimes if you hate fashion, then you Love using chatbots to replace the need. You didn't really want to spend any time, that's fine. But if you are someone who is spending a lot of time reading influencers and what their interest in the fashion, you have not decided to replace that with a chatbot. You're still going to want to hear from those fashion tastemakers. And so there's an opportunity with AI overviews to help you get started and then make it easy for you to dig in and connect. And I think people's interest in connecting with other people is just as strong these days in many ways.

Tracy Alloway (12:06)

So just at a simplistic level, can you tell me, like, how does Google determine whether it shows the AI overview or not? So if I type in Corgi into Google search, I'm biased because I have two corgis, but it just gives me a bunch of links to like the American Kennel Club and like the Corgi subreddit and things like that. If I type in what is a Corgi question mark, it gives me the AI overview. Is it just everything with a question mark returns an AI result? Or how are you actually deciding what to present to users?

Liz Reid (12:37)

No, what we try, and an important premise of this is that we shouldn't give you AI for the sake of giving AI. The point is for it when we think it adds value to people. And so it's not really associated with question marks. Question marks are often when people are looking for more of a description, they have a harder question, which maybe a single webpage doesn't answer, or whatever else. But what we're really using is looking at signals from users to say, does the AI overview provide additional value or not? And so most people I don't like, I haven't studied the Corgi query in detail. But for a query like that, probably what we're seeing is that most people aren't just trying to figure out what is a Corgi. Maybe they want to see pictures about it. They want to click on knowing more about the dog breed because they're trying to engage in it. And so we basically learn over time based on user signals. The same way we learn about like, when should you show the weather one box and when should you show local results and when should you use sports? That the AI overview provides additional value. Great. We show it, it doesn't. Then we want to get out of the way. Right? You don't want your search for Wikipedia. You go and you type in Wikipedia, which a lot of people do. They want to get to Wikipedia they don't want to go and say let's give me the history of Wikipedia. That's not why they search for that. If they search for odd lots, they probably want to quickly get to your podcast. And so we have a variety of signals that try and help us understand when is it adding value and not. And we get smarter over time as people both change how they ask questions as the models get smarter. We don't want to put in a overview if we think it's not going to be high quality. So as the models have gotten more powerful, we can cover more cases and just continue to develop really, with the focus being what is the best response to give a user for the question they've asked?

Fidelity Trader Plus Advertiser (14:37)

Introducing Fidelity Trader plus, the next generation of advanced trading from Fidelity Customize your tools and charts and access them seamlessly across desktop, web and mobile for faster trades anywhere you go. Try the all new Fidelity Trader Plus. Learn more about our most powerful trading platform yet@fidelity.com TraderPlus investing involves risk, including risk of loss Fidelity Brokerage Services LLC Member NYSE SIPC Support for the show

Public Investing Advertiser (15:07)

comes from Public Lately it feels like there are two types of investing platforms. Some are traditional brokerages that haven't changed much in decades, and others feel less like investing and more like a game. Public is positioned differently. It's an investing platform for people who are serious about building their wealth on public. You can build a portfolio of stocks, options, bonds, crypto without all the bugs or the confetti. Retirement accounts? Yep, High yield cash? Yes again, they even have direct indexing. Public has modern design, powerful tools and customer support that actually helps go to public.com market and earn an uncapped 1% bonus when you transfer your portfolio. That's public.com market and paid for by

Fidelity Legal Disclaimer (15:50)

Public Holdings Brokerage Services by Public Investing Member Finra SIPC Advisory Services By Public Advisors SEC Registered Advisor Crypto Services By 0/ all investing involves risk of loss. See complete disclosures at public. Com Disclosures so there's a lot of noise about AI.

IBM Representative (16:07)

But time's too tight for more promises. So let's talk about results. At IBM, we work with our employees to integrate technology right into the systems they need. Now a Global workforce of 300,000 can use AI to fill their HR questions. Resolving 94% of common questions. Not noise. Proof of how we can help companies get smarter by putting AI where it actually pays off, deep in the work that moves the business. Let's create smarter business.

Joe Weisenthal (16:34)

IBM I have a question about what you see among user behavior and My question is, do you see the same user or a cohort of people who use both google.com and gemini.google.com and do you see distinct patterns of queries from the same user but different types of searches? Or do people just sort of throw the question mostly in the Google search box and start from there? Or, or do you see people who just use Gemini and do everything there? Like what do you see in terms of emergent patterns about how an individual chooses which of the sites to enter into first and how they do different queries in each one?

Liz Reid (17:17)

Yeah, so, so maybe just so we're all talking about the same thing. Sure. There's sort of your main search page, there's AI mode that's part of search and then there's the Gemini app.

Joe Weisenthal (17:26)

Right.

Liz Reid (17:26)

Right across and I would say like there's a lot of users so their behavior varies across all of them. But there are some patterns. Okay, there's plenty of people who co use across them. There's plenty of people that are actually using several AI products right now. Just in general. Right. Not even just within Google. Across Gemini and search, the more informational ones. If it's an informational query, then the probability that they're using searcher AI mode is going to be higher. If it's a creative query, it's more of a productivity question. I want to please rewrite this to make it sound more formal. Right. Those type questions are going to be more Gemini oriented between AI mode and search. The main SERP page. Some people use AI mode mostly via AI overviews. They start in AI overviews and they transition. For those who go direct to AI mode, they tend to do that for queries that they consider sort of more complex, longer questions, questions where they expect that they're going to do more follow ups versus if you're doing a very browsy query, you might choose to prefer all of the serp. If you know that like your goal is to just get to a particular web page, you're more likely to start with the search result page. But you know, there's obviously overlap in the use cases. But across search and AI mode tends to be sort of more longer complex, more conversational queries versus more traditional queries. And between Gemini and search there's more of a productivity and creativity versus information slant on them.

Tracy Alloway (18:58)

So since we're talking about user behavior, one of the things that seems to be happening now is people will use an LLM, doesn't matter which one, they'll ask a question and then they will go and fact Check the answer that they get on Google. And I'm really curious if that's something that you're aware of as a sort of user behavior. And if the idea is that maybe Google becomes maybe not an AI overviewer per se, but maybe the sort of fact checker of last resort for other LLMs.

Liz Reid (19:30)

I think we're definitely aware that people use Google as a fact checker for some of their LLM use case. I think people have used Google as a place to fact check information pre LLMs for a number of things. A friend tells them something and sort of come. But I think people use search for a lot more than just fact checking or even just looking up facts. They want to go browse what they're going to go buy. They want to go check up the sports score of the latest team. And we do see with AI overviews that with the presence of AI overviews, people are asking more, longer questions, they're asking more conversational questions. And so some of these questions they started bringing to an LLM as we brought AI overviews in, they took on those same types of questions and have brought them to search.

Joe Weisenthal (20:16)

One of the complaints about search, and I would say I've complained about this or complained about is how many of the results in the SERP are like very, almost like too timely. Let's say someone enters the news, there's a headline about so and so I'm like, I'm really curious about this person. And so I like search their name and I get like 10 results or however many results, all from the last day since they made the news. And it actually is difficult for me to find information that didn't have the context of the news. So it was like unbiased in some way. Is that recognized as an issue? Because I certainly feel it as a user. But I'm curious from your perspective at Google whether this is something that you think about as a way in which search becomes less than ideal.

Liz Reid (21:05)

I think one of the things that's generally very challenging about search is that people enter the same query that's often very short with a lot of different intents in mind. Right. And so in a bunch of the examples where you have probably our data says that most people just want to see the recent articles and those are the ones that get all of the clicks. But you didn't. Right. And so how do we figure out across all of the different intents and match them across? And so I think there's this question about both. How do you get the different facets of a Question how can you personalize the results more effectively for people? I do see more. This is anecdotal as opposed to complete data. But to your example, people using AI mode and AI overviews for some of the people queries to understand more about the person independent of the rest of the news articles. Because if you don't know who most of the people searching for it know who the person is, but some of the people searching for it don't. And so you can see behavior like that. But I do think one of the interesting things about the evolution with AI is that that people stop talking just in keyword ease as much and they start expressing more of what they want and then that becomes much easier for us to give an answer, right? So if you say tell me about someone versus like what's new with someone? That's actually much easier for us to figure out how to give better results than if all we just say is someone, right? We used to talk about the an example query in a different way is falafel. What do you want to know with falafel? Some people don't know what falafel is. They want a definition, some people want recipes, some people want to find where to eat, some people want nutritional information. They all just use the word falafel and that's just harder to figure out how across all of them do that. We also see that with tensions on things like, well, do you want video results or do you want more text based results? People are very opinionated about what the right answer is, but they are not very opinionated in the same direction. And so we try and meet, you know, multiple billions of people's needs at once?

Joe Weisenthal (23:10)

Just to be clear, the falafel example is great. Would you say that today you see a greater diversity of falafel related queries, whereas maybe like five or ten years ago you just get falafel and now people know that they could type in where does falafel come from? What is a falafel recipe? Et cetera. Have people gotten more sophisticated over time in their query specifications about falafel?

Liz Reid (23:37)

I don't know falafel very specifically, but in general, yes. Okay. We have seen with AI overviews meaningfully longer queries, we see more natural language queries, but it's also not even something as basic of that. It can also be like you were searching for restaurants. We used to laugh about the like. Before I worked on search, I worked on maps and local some of the intersection with search and People would just be like, restaurants in New York. And you're like, what do you want me to do with that query? Right, okay, the best restaurants in New York are going to take three months and 99.9% of the population can't afford to go to them. Okay, but like are you picking 10 random ones, et cetera. But like part of why people will do that is they had a much more complex. I want a restaurant in this location for five people. It can't be too pricey. I have a vegan member. I also have like, I have kids. That was the question they had in their mind. And in the old world of keywords, that information would be kind of spread throughout the web and so you wouldn't feel confident. You could just put in the question. And now with like AI reviews and aimode, you can start to actually. And you see people do this, they tell you the real problem, right? They don't take their need and translate it to what the computer understands. They try to give the computer their actual need and expect us to do the translation. And I think that's really exciting to see because one, we can be more helpful. But also those are real problems people had. If you go back to the mission, it was organized the world's information and make it universally accessible and useful, like that useful part, right? It's not just that it's organized. Is it useful to you? And I think one of the most exciting things about AI the transformation going on right now is that you can actually make information much more useful to people. And that really opens up, that makes it such that people just ask more questions because we can actually do a better job meeting their needs, does that

Tracy Alloway (25:25)

come with any like complications in terms of privacy or competition for Google? If people aren't using keywords as much anymore, if they're doing basically like query brain dumps into the prompt and saying, you know, I am so and so, I have a kid, I live here, I want to do the following things. This is my issue is that like an added layer of complexity that you have to deal with as like a large search company.

Liz Reid (25:53)

I mean I think people, from a privacy perspective, we give people sort of a range of different things. They can be sort of an incognito, they can be signed out, they can be signed in across. And I think Google has a long tradition of really treating people's data with a great deal of care and having cutting edge security and privacy by design. So I think people are seeing the value and they have continued their trust in Google. I Think it means it's a harder job on quality. You have to take this question that's many parts and you have to figure out how you break it apart. And you have to do work to think about things like latency, because you can't just. If everyone uses the same keyword and it's not personalized, then you can cache it all. If all of a sudden the queries get much more diverse, it has consequences there. But I think we just see that it's very empowering people that it takes some of the work out of searching. I think sometimes people think, oh, a few years ago they said, what more can you do with Google search? But if you actually ask them, okay, when was the last time you spent 20 minutes searching when you would have preferred to spend two? It's actually not that hard for many people. Oh, the last time I was trying to go find a service provider, the last time I was trying to go do these bigger tasks in life. It's been exciting to just make people's lives easier by helping them address their real need.

Joe Weisenthal (27:16)

I have so many different theoretical questions I could ask. Here's one. Actually, you know what? This is something that was inspired by our producer Dash, a conversation that I had with him 10 minutes ago prior to this and related to something else. I imagine in your career at Google, spanning over two decades, at this point, you've been involved in quite a bit of recruiting and recruiting software engineers in particular. And imagine that's a particularly important aspect in some way or another for the VP of Surge, given what we've seen with AI coding and so forth. When you're doing one of these leetcode software developer interviews, et cetera. Is it different today than five years ago? Do you have to think really differently about the battery of technical questions that you would propose to a software engineer today, given just the restructuring of the nature of the job in a world of AI generated code, I think the

Liz Reid (28:12)

process is definitely evolving. I wouldn't say that we have perfected the science yet.

Joe Weisenthal (28:16)

Okay.

Liz Reid (28:17)

But there's two angles in which you're thinking about it, right? One is you don't want to ask questions for which they just go and type in the answer in the chat bot and recite it back to you. Okay. So you need to make sure to the extent that your goal is to understand, are they critically thinking, are they able to think through a problem and do that, you want to make sure that that's actually what you're. And so is it in person. How are you doing that on Some basic way. But there's the other thing that I think. The tools are powerful. You can use them in ways that make you more effective, and you can use them in ways that make you less productive. How do you think as the fluency is changing with AI, the way a software engineer might approach the problem now is different than they might have approached the problem five years ago without some of these tools. And so I think we're all learning. How would it change asking that question, right? Are you building up that expertise? Are you building up that fluency? The fluency isn't fixed. What was possible with the tools six months ago, let alone two years ago, is different than what's possible now. And in six months it will be different. So you have to start thinking about how part of your interview is thinking about fluency with the use of tools. In the same way that when IDs became important or when people stopped using assembly language and they started doing Java, right, You had to sort of evolve the interviews. It's just that it's happening very fast. Right? And so we all have to be on our toes. But exciting. Like, you play with this tool that it doesn't work for something, and then it's not like, play with it two years later, it's like, play with it three months later. Maybe the tool will now work for these things.

Tracy Alloway (29:50)

Okay, well, speaking of tools, I mean, one of the things, Joe, I think you've said this, you've been playing around with Claude code, this idea that actually when you start vibe coding everything and telling your agent to do everything, it feels like you don't even necessarily need, like, a computer, much less a search engine, presumably. So I'm just curious, if you gaze five or ten years into the future, what do you think the default entry point for interacting with the web is actually going to be? Is it going to be a search engine like Google? Is it going to be a specific LLM? Is it going to be my personal agent that I've vibe coded for all of my preferences?

Joe Weisenthal (30:29)

Can I just add on to this question? This is something I think about. Like, if I want to send an email to Tracy today, then what I have to do, like, I find the tab in my browser, I scroll over there, okay, that's my Gmail tab, et cetera, whatever. I would like to just be in my terminal. It'd be so much easier to say here, send an email to Tracy saying this. There's all these steps that I currently do because of the nature of graphical user interfaces, that now that I've gotten in like Claude coding or whatever feel a little clunky. It feels a little yesterday. And so, yeah, I'm extremely curious about will the web with these series of boxes that we drag and drop, et cetera, is that the future or will it just be someone talking in English to their computer?

Liz Reid (31:11)

I don't think like 10 years is a long time right now where the tech is. That's fair. In one year,

Joe Weisenthal (31:18)

we still have browsers.

Tracy Alloway (31:19)

Three months from now, people are like,

Liz Reid (31:21)

okay, we believe in 10 years we'll achieve AGI. Will anyone be doing anything the same? Okay. So with that aside, I think there are some things I believe in and some things I think we don't know. I think you already see if you go back 10, 20 years ago, that the way you interact with the tech has evolved a bunch. Right. It used to be it was just the laptop. Well, now it's the phone. Well, now it's also the watch. Okay. In some cases it's the glasses. This sense that it should feel like the information is at your fingertips in whatever medium is useful. But I don't know that this becomes a. We haven't so far replaced all of the old ones. Right. Like, you use the phone a lot more, but my guess is you're not doing all your cloud code work on your phone and you're doing some of it on a desktop.

Joe Weisenthal (32:09)

That's true.

Liz Reid (32:10)

Right. The introduction of the watch has supplemented, but it hasn't eliminated the desktop. Right. So what's been interesting actually is that it hasn't gone in the direction of converging to the answer. It's actually increased the form factors and so that you want to be able to access this information wherever you are, in whatever form factor makes sense. And so will it be glasses? Will it be something else? Quite possibly. But let's even say it's glasses become a big deal. Glasses are very small screens. Even there, you're probably not going to do your big productivity thing on desktop. So I think what you'll see is that the access point is not confined to one thing, but that the key is to eliminate the friction and the toil. To your point, you had to do six steps. You didn't want to do the six steps. Why should you do the six steps? I think you see that some things are much easier to do with a chat interface and then some things actually a chat interface is a super slow way to go. Do if you have a list and you have to go, say, please remove this long title for the 10th item. That's actually much harder to do with chat than an interface that does that. I don't think it necessarily converges on a single thing. I do think it should feel much more adaptive. To your point about, well, if this is the way you prefer to interact, not just where you are but how you interact and can you customize and can you create, to what extent do the user interfaces look designed for you versus look design for general and can you have influence in them? I think you'll see that. I do think we sometimes we're very aware of what doesn't work well. We're not necessarily aware of what does work well. Companies spend huge efforts working on how do they do shopping carts really well.

Joe Weisenthal (33:55)

Yeah.

Liz Reid (33:56)

This belief that the chatbot will have a more optimized shopping cart for every shopping cart place in the world than the one you go to every day, I don't know. Not clear for those things. But I do think it should feel much more personal, it should feel much more dynamic, it should feel much more ambient and available to you. And I don't think it will be one size fits all, either per person or per form Document.

Public Investing Advertiser (34:35)

Support for the show comes from Public Lately it feels like there are two types of investing platforms. Some are traditional brokerages that haven't changed much in decades, and others feel less like investing and more like a game. Public is positioned differently. It's an investing platform for people who are serious about building their wealth on public. You can build a portfolio of stocks, options, bonds, crypto without all the bugs or the confetti. Retirement accounts? Yep. High Yield cash? Yes again. They even have direct indexing. Public has modern design, powerful tools and customer support that actually helps go to public.com market and earn an uncapped 1% bonus when you transfer your portfolio. That's public.com market ad paid for by

Fidelity Legal Disclaimer (35:19)

Public Holdings Brokerage Services by Public Investing member FINRA SIPC Advisory Services By Public Advisors SEC Registered Advisor Crypto Services By ZeroHash all investing involves risk of loss. See complete disclosure@public.com Disclosures the thing about

IBM Representative (35:34)

AI for business, it may not automatically fit the way your business works. At IBM we've seen this firsthand. But by embedding AI across hr, IT and procurement processes, we've reduced costs by millions, slash repetitive tasks, and freed thousands of hours for strategic work. Now we're helping companies get smarter by putting AI where it actually pays off, deep in the work that moves the business. Let's create smarter business.

Fidelity App Narrator (36:03)

IBM if you follow markets, you know the value of long term thinking. You plan, you diversify, you prepare for volatility. But in life, even the best strategies can't prevent every bad day a fire, a loss, a disruption that demands immediate attention. When that happens, what matters isn't just what you planned. It's who shows up. That's where Cincinnati Insurance comes in. For more than 75 years, they've helped individuals and businesses navigate life's toughest moments with care, expertise and personal attention. Together with independent agents, Cincinnati Insurance focuses on relationships, not transactions. Their approach is grounded in experience, follow through, and trust built over time. Bad days happen, and when they do, you deserve an insurance partner who understands risk, respects what you've built, and is ready to help you move forward. The Cincinnati Insurance companies let them make your bad day better. Find an independent agent@cin fin.com I've been reading some articles.

Joe Weisenthal (37:06)

I think I saw one. There was a big one in the Information recently talking about one of your competitors. Meta yeah, it's a competitor and there was like everyone's token maxing there and there's a token leaderboard and people are competing to show that they're using AI more than others. And from my perspective, that boggles my mind because compute is a cost. And just using compute per se does not strike me as a particularly good way of measuring who is productively contributing to the company. I mean, I could certainly find an easy, quick, recursive way to burn tokens.

Tracy Alloway (37:40)

Generate a bunch of AI images of corgis.

Joe Weisenthal (37:42)

Yeah, generate AI and then tell it. Create one that just keeps telling it to improve itself, et cetera. The flip side, which some people say is like, look, it doesn't matter at this point because everyone has to figure out how they're going to use AI productively in their work. So you know what? Don't even worry about metering AI. Tell everyone to pedal to the metal and AI use. And if someone is maxing out on tokens, it means they're experimenting with something and then they'll find something that really is a productivity enhancer. I'm curious if from your perspective, it makes sense to like essentially see token consumption or compute use as a proxy for someone who's doing their job aggressively.

Liz Reid (38:23)

Well, I think the thing with all of these proxy metrics is if you use them blindly, you're going to run yourself into trouble. If you as a leader, sort of don't use judgment on them, then you get the example of I will just create a job that runs in the background and does dumb things to do the tokens. Right? So as a leader Your job is to sort of use good judgment and not just think about the incentives. So if somebody isn't playing around at all with the tools, when we know that they can improve productivity, then we need to figure out, like, why and how we help support. Like, maybe there's some issue with the part of the system they're working on and we should go fix, or maybe we just need to help upskill them or whatever else the case is. I do think there is a level of experimentation required. Right. So I don't think it works if your answer is like, you need to ensure that all your token use is completely optimized. Like, it's not going to work. Right. People have to learn what's possible. They're doing different jobs. The tech is changing. But it can either be like, don't use the tools or just max your tools blindly. It's a noisy signal, but it's a signal. Right? So go look at it and understand as a place of where to look. Don't use it as a final judgment.

Tracy Alloway (39:40)

So speaking of measures and not oversimplifying them, I want to go back to the sort of core tension that we started the conversation out with, which is the AI results versus people actually clicking through to results and generating traffic. And I know you were talking about AI being expansionary or complementary for Google Search, but I'm very curious how you actually measure that. And the more granular you can get on this, the better. What are you specifically looking at to say that actually this is something that's good for our business versus something that's detracting from the core?

Liz Reid (40:17)

So I guess I would say Google's guidance in Northstar has always been focus on the user. That's our biggest question at the heart is how do we make a great experience for users?

Tracy Alloway (40:27)

Then.

Liz Reid (40:28)

Then you want to be thoughtful, obviously, about other. Okay, you, like, if you don't have a healthy ecosystem, you can't build a service ongoing. So you need to make sure you're nurturing a healthy ecosystem. If you have make no money, then you can't fund this wonderful service. So you have to be thoughtful about those. Okay. But the place you start with is try and build something amazing for users. One of the things we've seen again and again with Google Search is if you're doing a really great job, people will not just do another query, they will come back to you more often. They will take their phone out of their pocket and extra time. That's a high bar, right. It's one thing to go and say I've showed you something. Can you do one more thing while I'm showing it to you? It's another thing to get you to decide. You're going to bother to unlock your phone, you're going to boot up your desktop, you're going to navigate in the browser. And so one of the things we really look for is when we're doing these changes, does it cause people to come to search more often? Not just use search more often, but come more often. We also do various UX research studies and try and understand what are people happy about or not. What are the things they find are frustrating, are more users adopting it, not just how much are they using it. So we look at a bunch of different metrics, but one of the biggest is really like, do you choose to come and ask Google? Like, do you essentially hire Google more often? Right. For things you need. And one of the things that's very surprising to people at times is they think they come somewhere for all the questions they have already today, right? Like maybe they think they come to Google all the time or they think they go to Google+LLMs or Google +LLMs plus TikTok plus whatever. Okay, okay. They think they ask all the questions they have, but that's not true. You actually make a calculation when questions go through your mind of is it worth spending any time to figure out the answer to this question? If the answer is no, then you just don't ask the question. When we talk about AI as an expansionary moment, what we really mean is there's a whole bunch of questions people have. There's a whole bunch of curiosity that people are not exploring. And they're not exploring because they view it as too difficult or too much time or not sure that it will be worth it. And AI lowers that barrier and it can lower that barrier in ways that are sometimes for us, English speaker are surprising. Which is like actually in a bunch of countries, there's not all the content in the web in the language you speak. Okay? LLMs can help unlock that content. AI overviews, because it's using an LLM, can be more multilingual than just the web. Corpus is by default, Right? Okay. So suddenly information that wasn't available to you as a Hindi speaker is now available. It can be visual. You had a question about this flower, you had a question about that cool purse you saw like, where can you buy it? But you didn't know how to describe it. It's possible it can also just be like, my kid has a question, do I say Like, I don't know. Or do I go ask the question, right? You see, with young kids, they ask questions all the time, right? They go, why? Why, why, why, why, why, why? And like, at some point, parents are

Tracy Alloway (43:33)

like, because stop bothering me. Go ask the LLM.

Liz Reid (43:37)

They go, okay, let me Gemini. They do that because from a kid's perspective, they assume adults know everything and it is no cost to them. They're not worried about their time and other things. As an adult, it's not that you're not curious. You just don't think everything is known and you don't have the time. If you lower that barrier, it allows you to be that kid again that just sort of explores all of these things or gets started on those projects that felt daunting or enables you to save or learn a new skill or whatever else. And that's really exciting.

Tracy Alloway (44:12)

But I don't want to dismiss the wonder of being a kid and learning about the world, but I'm going to sound very callous in a second. But how do you make money off of that? Like, how do you make money off of the AI overviews? Is it just customer retention? Is that what we're basically boiling it down to? And then if it is customer retention, then you could have a strong argument for saying that, like, everything can just go through Gemini instead of search.

Liz Reid (44:36)

I think I would say a couple of things. Search only shows ads on the subset of queries, right? You know, like less than a quarter of queries, right? So there's a whole bunch of queries, pre AI overviews, right. That you don't make money on because many of them are not of commercial need. Right? And so you asked a question about tankers earlier, probably pre aio, that query wouldn't have shown ads anyway.

Joe Weisenthal (44:59)

It doesn't show ads now, by the way. Yeah, I checked this too. There were no ads on that.

Liz Reid (45:04)

It doesn't show ads before. There are no ads on that cry. Right? Nobody's trying to advertise something on that. Okay? So those queries, AIO reviews, it doesn't disrupt, right? Then there's a bunch of class of queries where you're shopping the presence of an AIO review or Gemini answers, it doesn't preclude the need to still buy the item. So there's still this huge opportunity with ads because there's all of this choice that's going to. I think you also see that there's an expansion of queries right to this point, so you get more queries. And so some of those queries are more commercial. Some of them are not, but some of them are more commercial. And so those become new opportunities for ads. And there can also be things like when the query is under specified or it's a single query, you actually don't know as much. So you can't maybe target the ads as well. If people start expressing more of their need, if it's more of a conversation, they're going more down funnel. You can actually create better ads. Right. And so you can think about new opportunities for. For ads formats. Right. Some number of years ago people would have said like how can you make money from a feed? Okay, well Instagram ads are very popular, right. And so there's new ads formats as you recognize new technology and new opportunities. But the commercial needs are still often there and the desire for user choice is still often there. And so there's still a lot of possibility going forward. And so it's worked out very well right now with us in the balance.

Joe Weisenthal (46:27)

So I realized that I am a user of both the gemini app or gemini.google.com and just google.com and I have like a sort of an intuition of which one I go to for which purpose. So if I want to look up the capital of Moldova, I'm not, I'll just search capital of Moldova, which I discuss. Do you know what it is, Tracy? Sorry, I'm not trying to stump you.

Tracy Alloway (46:50)

No, I feel like you're going to say it and I will.

Joe Weisenthal (46:52)

No, it's Kisha now. I'd never heard.

Tracy Alloway (46:54)

Oh no, I didn't know.

Joe Weisenthal (46:54)

Yeah, I didn't know that either. But anyway. But if I want to understand what are some academic papers that have been written about why it is that high frequency trading firms tend to not have outside capital, et cetera. And what was the theory for this? I think at this point I would use Gemini for that and hope to. I'd use Gemini or some other ones. But within the context of this conversation, that's more of a Gemini query for me than a Google one at this point. Will there always be two boxes or do you foresee eventually there is just one box and it will just know. This is like why do we need two boxes?

Liz Reid (47:35)

I don't know what life will be like in five years. I think it's very. Sometimes people want sort of an experience. Although the information needs seems similar. They actually want different experiences across. Okay. And so if you take a prelim example, people use YouTube for search. Some in the US they use it, some in India they use it a huge amount. They Bring a bunch of queries that you would bring to Google search in the U.S. right? You could say, okay, well, we haven't. Why haven't we collapsed YouTube and search search box into one search box, right? And do that. And it hasn't necessarily been the case. We have the Google app and we have Chrome. They both allow you to search and they both allow you to browse the web. You have a set of people that love the Google app and you have a set of people that love Chrome and you have a set of people that use both on a phone. But you can't necessarily convince either population that they want to stop using one app and just switch to the other app. So I don't think the space is so huge and it's changing so quickly right now that to sort of be able to know for sure whether or not you can create one sufficiently dynamic personalized experience that one app, one entry point, can truly do it all. I don't think we know yet on that. People come for restaurant searches, they come to Maps and they come to Google Search. We have not collapsed the Maps app and the search app. At some point it becomes big. You're putting all this directions code into the Google Search app. Is that actually useful even if you had a full Maps view? So I think we're just going to have to learn over time about what's good. But the space is really giant and they do have different emphasis right now on what they try to excel at. And you want to make sure that in the attempt to bring things together, you don't become only okay at everything. And you want to make sure that you can shine at all the use cases people need. And that may mean two products or that may not, or it may be a third product. Right. Like, I don't know, in five years there may be a third product that replaces all the products. You have your personal agent, you don't talk to anybody. I don't know.

Tracy Alloway (49:39)

So I realized we kind of promised to talk about AI slop a little bit in this conversation. So one of the things that's happening with AI is not just that I can ask a bunch of questions I might not otherwise have had time or the inclination to ask. But also AI is being used to generate vast amounts of content that are aimed at potentially answering any silly question I or anyone else on Earth might have. Yeah, just churning it out. And I'm very curious how search is weighing, I guess, the quality of its results in the new slop era of the Internet.

Liz Reid (50:14)

I think there's A tendency at times sometimes to think about AI slop as if it's before AI slop, there was slop. Yes, Right.

Tracy Alloway (50:22)

Human generated slop.

Liz Reid (50:25)

There was human generated slop, now there's AI generated slop. So there has always been slop on the web. And so what doesn't really matter at some level is how much slop is on the web, so much as is there great content on the web and can you surface it? Right. And this is Google's bread and butter in ranking and has a long history of looking for spam and trying to drop it and make sure it doesn't show. And like we crawl many, many more pages than we even put in our index. There's pages we put in the index that we never surface. Right. So that we can keep that rate of spam and slop at a very low rate. And it is a constant effort. Right. Like it's not a problem you solve because some of the people generating the spam. Right. There's a lot of financial incentives associated with it. But that is what like we, what people have come to trust Google is that it will show great information and it's the thing that we will continue to put a huge amount of effort in. And so that's the way I would think about it. It's not like how much AI slop or non or human generated slop or whatever automated slop, pre AI post human generated there is, but making sure that the information you do see is trusted.

Joe Weisenthal (51:40)

Liz Reid, thank you so much for coming on Avlas. That was a fascinating conversation. I have like a billion more questions, but man, we'll have you on in three months when the entire world has changed and we'll get an update from you.

Liz Reid (51:53)

Thank you very much. It was a pleasure to be on with you.

Joe Weisenthal (52:08)

I like the point about human generated slap. Do you remember?

Tracy Alloway (52:11)

Yeah, but the difference is the volume.

Joe Weisenthal (52:13)

No, I know, I know, but, but like do you remember Jason Calacandis startup Mahalo?

Tracy Alloway (52:19)

No.

Joe Weisenthal (52:19)

So Jason Kello Kennis, who's been on the podcast, he had this startup for a while. It was like the biggest piece of garbage in the world. It was no for real go. People need to go look at. It was called Mahalo and the idea was like they were just going to have hire a lot of people to like write articles that were not very good to like appear in Google.

Tracy Alloway (52:37)

Oh, I see. And like to swamp this.

Joe Weisenthal (52:39)

Yeah, There was a famous one that my old colleague and I think a business insider, Nick Carlson, discovered. And it was like, if you search like how to play the xylophone. There was a Mahalo article for that. And it was. I swear to God. Okay. It was. Step one, decide if you want to play a xylophone.

Tracy Alloway (52:58)

Well, that's an important step.

Joe Weisenthal (52:59)

Step two, get a xylophone. Step three, learn to read sheet music. Step four, practice reading sheet music and play. So, like, this was actually like this. I just remembering, like, it is. People have been trying to, like, stuff complete garbage into the search results for a very long time. And I always get a chuckle thinking about that example and you should go look for it. And so I'm glad that I am

Tracy Alloway (53:25)

looking for it now. I'm very distracted.

Joe Weisenthal (53:28)

It was so bad.

Tracy Alloway (53:30)

Wait, we're just gonna like laugh about this article for.

Joe Weisenthal (53:33)

Look at this. Just search Mahalo. How to play the xylophone.

Liz Reid (53:38)

Yeah.

Tracy Alloway (53:38)

I see something from Mahalo.com on YouTube that can't see it.

Joe Weisenthal (53:42)

Yeah, yeah. Just search. So, yeah, Business Insider, February Tonya line. Hilariously useless. Mahalo's guide to playing the Xylophone.

Tracy Alloway (53:49)

Did you write that?

Joe Weisenthal (53:50)

No, Nick Carlson wrote it. He wrote it up.

Liz Reid (53:52)

Yeah.

Joe Weisenthal (53:52)

Yeah. Unfortunately, now it's behind a paywall and itself is covered in sloppy ads. So I guess. But anyway, sorry.

Tracy Alloway (54:02)

Decide whether you want to buy a used or new xylophone. Metal xylophones are less expensive than wooden ones.

Joe Weisenthal (54:08)

Please.

Tracy Alloway (54:09)

That's useful. That's a useful tip that I did.

Joe Weisenthal (54:11)

I know, but when you see that, it's like, please, AI. Like, save us from this human generated garbage that was trying to clog search results before.

Tracy Alloway (54:19)

All right. On a serious note, yeah. I did think the point about not customer retention, but like expanding the volume of like user queries on the platform made a lot of sense.

Joe Weisenthal (54:30)

Totally.

Tracy Alloway (54:31)

Which I hadn't like really considered that much before. So even if you do get a bunch of no click users.

Joe Weisenthal (54:37)

Yeah.

Tracy Alloway (54:38)

They are more inclined to come back to the platform in the future and maybe some of that eventually lands in clicks.

Joe Weisenthal (54:44)

The other thing I hadn't thought of, and I thought it was a great point, which is that there are multiple. Google currently runs multiple search boxes. There's the YouTube. Are you looking at the Mahalo article and cracking up right now?

Tracy Alloway (54:55)

Sorry, I am. Step four is acceptable. Experiment with different mallets.

Joe Weisenthal (55:00)

It's so good.

Tracy Alloway (55:03)

I just really want to play the xylophone.

Joe Weisenthal (55:05)

It's so good. It's so good.

Tracy Alloway (55:09)

Step five is practice regularly.

Joe Weisenthal (55:12)

It's crazy. Like, I think AI is much better. Like if I, you know. It's so good, isn't it? It's like the biggest steaming pile of garbage I've ever seen on the Internet 15 years or 10 years before anyone

Tracy Alloway (55:24)

had ever can we actually. Can we have Calican as Jason just to talk about.

Joe Weisenthal (55:29)

We should have J back on the podcast just to like grill him about what exactly he was thinking and his sins against the Internet for having put this on there.

Tracy Alloway (55:38)

Well, he was an early adopter of Non AI Slob.

Joe Weisenthal (55:42)

But anyway, that point about there are multiple search boxes already, right? There's the YouTube search box. You're still left.

Tracy Alloway (55:49)

I'm sorry, I was trying to make eye contact with you and not look at my computer.

Joe Weisenthal (55:55)

Should we just leave it? All right.

Tracy Alloway (55:56)

Okay. All right. Yes. Shall we leave it there?

Joe Weisenthal (55:59)

Let's leave it there.

Tracy Alloway (56:00)

This has been another episode of the Oddhots Podcast. I'm Tracy Alloway. You can follow me at Tracy Alloway.

Joe Weisenthal (56:05)

And I'm Joe Weisenthal. You can follow me at the Stalwart. Follow our producers Kerman Rodriguez at Kermanrman, Dashiell Bennett at dashbot, and Cale Brooks at Kale Brooks. And for more Odd Lots content, go to bloomberg.com oddlots we have a daily newsletter and all of our episodes and you can chat about all of these topics 24. 7 in our Discord Discord GG oddlots.

Tracy Alloway (56:27)

And if you enjoy Odd Lots, if you like it when we talk about AI and human generated sloth, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening. SA.

Public Investing Advertiser (57:17)

When you're running a business, the best days are the ones where priorities stay on track. For midsize and large companies, risk can affect multiple parts of the organization at once, from property and liability to cyber and regulatory challenges. At that level, managing risk becomes an ongoing discipline. At the Hartford, the focus is on helping businesses manage risk before it turns into something more disruptive. And when losses do happen, that work is paired with insurance coverage shaped by years of underwriting, risk engineering and claims experience. Learn more@thehartford.com riskmitigation policies provided by Hartford Fire Insurance Company and its property and casualty affiliates, Hartford, Connecticut.

Cantu Advertiser (57:58)

Cantu heard you loud and clear with their new Ultra Moisture Collection. Powered by Batana Oil and Jamaican Black Castor Oil, this new lineup collection is clinically proven to deliver non stop moisture for up to five days. Five whole days. The shampoo gently cleanses without stripping and the leave in. And stylers keep curls soft, defined and nourished long after wash day. Plus, it's free of sulfates, parabens, silicones, and mineral oils. If your curls have been craving real moisture, this is it. So if your hair's curls have been thirsty, go ahead and try the new Cantu Ultra Moisture Curl Collection. Your curls will thank you. Available now at Walmart.

Okta Advertiser (58:35)

These days, it seems like AI agents are just about everywhere you turn every field and every function. But without identity, you can't trust they'll serve your business instead of jeopardizing it. Fortunately, Okta helps you get identity right by securing your AI agents identities, giving you a single layer of control, a single standard of trust. So whether an AI agent supports a single user or your entire enterprise, with Okta you'll turn risk into opportunity. Secure every agent. Secure any agent. Okta secures AI.