← No Priors: Artificial Intelligence | Technology | Startups
No Priors: Artificial Intelligence | Technology | Startups
Meet Snowflake Intelligence: A Personalized Enterprise Intelligence Agent with Sridhar Ramaswamy
00:42:11
Loading summary
Transcript57 lines
- [00:00]
A
Foreign. Hi, listeners. Welcome back to no Priors Today. I'm here with Sridhar Ramaswamy, the CEO of Snowflake, the former founder of Niva and the SVP of Google Ads. We will talk about his first 18 months of being CEO. The incredible execution over that time in shifting a company at scale to being AI first, where the enterprise ROI is and what happens to the cloud service providers and the ads model in the age of AI. Welcome, Sridhar.
- [00:35]
B
Sarah. Really excited to be back.
- [00:37]
A
Well, it's a pleasure to talk to you as an old friend and colleague. The last time we spoke, you were on the entrepreneurial journey.
- [00:44]
B
That's right.
- [00:45]
A
Doing search still. You're now 18 months into being CEO of Snowflake. It has been a very eventful 18 months. Tell us a little bit just about the journey from taking the mantle from Frank to the first few months to where you guys are today. I think the market's reacted in many ways, most recently incredibly well to the execution. But it's been a journey.
- [01:10]
B
That's right. Snowflake has always been an amazing product company. The original product that Benoit theory conceived of 10 plus years ago was many years ahead of its time and it took the world by storm. And obviously they had the storied ipo, the biggest software IPO at that time. I think what happened was the company was a little slow to reacting to changes from things like machine learning and AI. And that was a little bit of honestly, the reason why Frank voluntarily pushed for the change because. Because he felt presciently that we are headed into a time that was just a lot more tumultuous from a product perspective. And he wanted someone that was product first to be in charge of the company. And the last 18 months have really been about embracing that wave of change. And if you look back to what's happened in the last two years, it's crazy how much change has happened with respect to AI, how it's become commonplace every day in all of our lives and then the speed at which things are still getting driven through. I think the really amazing thing about Snowflake is the company embraced this change, transformed itself, and then showed that not only can we do it from a product perspective, which one could have expected, but we also done significant things to retool our marketing or go to market. Overall, I think that transformation has been pretty amazing to watch, but times can be difficult. Last year there were a lot of doubters, but there were a lot of us who believed both in the value that Snowflake was already creating. And the reason I took this job was because I talked to a whole lot of customers before I became CEO. They all loved Snowflake and that was a big motivation for me to take this job. So I think we have sort of successfully ridden through that and are now at the cutting edge of data and AI for enterprises. It's been an amazing journey to have gone through.
- [03:15]
A
Walk me through just some of the like orientation prioritization you did in the first six months and then you know a little bit more about the long term vision here.
- [03:25]
B
Yeah, the first six months were a lot of tactical changes which primarily around accountability. Like every company that goes through essentially a rocket ship from phase of growth, growing at 100 plus percent year on year, Snowflake had basically specialized at every layer possible. And there was a very long distance between the engineer that did a feature and the customer that made use of that feature. And there were like seven to ten layers of teams that were involved. That works fine. When you have perfect product market fit and you're trying to optimize for every.
- [04:00]
A
Function, you're just the winning cloud data warehouse. Just like, yeah, drive a truck through that.
- [04:04]
B
But on the other hand, if you're working in the world of AI where we can barely tell what's going to come out next month, forget next year, this is the wrong structure to have. So we did a lot of organizing by different areas, making sure that there were accountable people. For example, in product engineering, that was among the first changes organized into different product areas like AI or the core warehousing analytics product. But then we also wanted a straight line over to our go to market team. So we created these specialized teams that work closely with product engineering and marketing to take these new products to market. And that was a lot of the early phase of Snowflake with an emphasis towards faster iteration. This is something that I've believed in all my life, which is speed wins, ability to iterate always trumps carefully laid out strategies. Yes, you shouldn't do dumb things. But on the other hand, realizing any kind of gain requires a lot of iteration. So we made a number of changes on that side, both with respect to how quickly we created products, but also how quickly we iterated with customers. And I would also say we took a little bit of time to find sort of our sweet spot in this AI space. As you know, that itself has evolved a lot. We are not a csp, we are not a foundation lab. So what are we? And there was that discovery of ourselves as the AI data cloud, as opposed to the data Cloud. And I think it's that kind of clear product insight into what value we add that is setting up the stage for the earlier parts of the year or even what we are about to talk about today.
- [05:45]
A
Now you're announcing Snowflake Intelligence. Tell us about that and how it fits into the broader vision.
- [05:50]
B
I mean, first of all, when it came to AI, as I said, we had to look hard at ourselves. Early last year we actually went down the path of creating foundation models. We created a credible MOE model. This was early last year. But we also quickly realized that our ability to compete with the likes of OpenAI or Anthropic was going to be really hard. We simply did not have the capital to be able to invest meaningfully in things like that. So we pivoted away from that into much more of a how does AI massively accelerate what can be done with data that is in Snowflake? And over time that can become a reason to bring more data into Snowflake, which is the phase that we are in now. But a lot of our AI product strategy was actually quite humble. It didn't say we were going to rethink everything. It said enormous number of customers. Something like half of the qualifying Fortune 2000 companies on the planet are Snowflake customers. They have their most valuable data on Snowflake. What does AI mean for that? And so we systematically invested in the components, whether it was Search or whether it was text to SQL, in ways that added on value to the things that people were already doing with Snowflake. And SI is an agentic platform, but it's actually an opinionated agent platform. A lot of agentic platforms, for example from the CSPs will basically say, oh, you can bring in data from anywhere you can imagine any kind of workflow that you want to do, and the one agent will rule them all. Which is nice in theory, but in practice, when you have an infinity of things that you can do, it's also hard to figure out what you should actually do. Snowflake Intelligence is very focused on how do you create value from data, whether it's structured or unstructured, a whole lot faster. And so the kind of use cases that got us really excited, honestly internal ones, were things like if we were to take all of the different dashboards that we used in sales and put it into one single interface, what could that be? We had done two, three versions of this, but eventually that culminated in this internal product. We call it Raven, but it's basically the sales data assistant. Then we started working with early customers, whether it is a Cisco or a fanatics or the USA Bobsled team to figure out what does this all mean for them. And the theme again is get away from the inflexibility of things like dashboards. A Dashboard is a 2D view of a complex surface. It just has no easy answers to the many questions that any reasonable person, you or I is going to have off of that. So we wanted to create something that freed people of the 2D style of thinking, was much more flexible in what it gave people access to, but but also knew its place. This is not a general purpose agentic platform to do it all. This is an agentic platform that lets people realize value from data faster and is a great foundation for people to get value from data, like really quick in a meaningful way. I think having this sort of an opinionated framework for AI has been super helpful for us.
- [09:01]
A
How does a user consume Snowflake Intelligence? Is it like I ask a question, I get push and answer, it builds dashboards for me on like how should I imagine that experience?
- [09:11]
B
Yeah, so we should show a demo of Snowflake Intelligence to you. But yes, it's an interactive interface. You can ask questions. There are a set of canned questions to make sure that you don't have block when it comes to being able to ask questions. You can ask it to say, hey, what datasets do you have access to? What kind of questions can you answer? It'll do a perfectly reasonable job of that. And our aspiration was for this product to be used by every single employee in the company.
- [09:41]
A
So it's not for people who can.
- [09:43]
B
Write SQL, it's not for people that can write SQL. We wanted it to be enough of a daily use product for every single person. There is not a single customer meeting that I'm going to have without quickly checking up on what's the latest with this customer. And so Raven that I talked about, our sales data assistant absolutely has things like what's our relationship with the customer? What kind of contract have they signed? What is their consumption looking like? But also things like what are the most recent conversations that we have had with them? What came out of these? Are there any outstanding tickling issues? And so it is a lot of that. And like many good products, there's breadth, there's value driven to many, many people within a company. But on the other hand, we don't pretend it's a bi dashboard. There are more things that you can do with a Tableau or a Sigma than you can do with Si but that's not the goal because this product also lets you do a bunch of things that you could not easily do in a dashboard and is really meant for any business user. And we place a lot of trust, and we place a lot of emphasis in all our AI products on trust, meaning we, I tell people we need to think of AI the same way we think about software engineering, which is there's a right and there's a wrong.
- [10:54]
A
Okay?
- [10:54]
B
It cannot be this mode of like YOLO AI. You can get some good answers, some terrible answers. It's your problem. And so we very much emphasize you need an eval for every single new thing that you're going to launch. If you want to change the underlying model, you need to be able to quickly verify that you didn't blow up on the things that you are already doing. And so we want it to be the trustworthy product for every employee, which is actually a new thing for us, by the way, because Snowflake, for pretty much all of its history, has always been used by the data team to slap a dashboard on top, which then gets exposed to end users. This is a very different motion. This is why we are working on things like identity provider integration so that you don't have to set up Snowflake accounts for each of the many users you're going to have in your company. We are also by the fact that there is subscription fatigue. And so Snowflake Intelligence is very much a consumption product. People pay for what they consume. And we are experimenting with a bunch of things there in terms of how do we drive broad and deep adoption without having people worry about runaway costs and things like that.
- [11:55]
A
The way you described Raven or the sales assistant agent use case, it sounds like an application or like a lot of applications that I get pitched like, how do you draw the line between like data and agent system and app today?
- [12:09]
B
Back to my point about execution, I tend to be like completely emotionless about there's no line where the strongest current is. On the other hand, it's absurd if we think we are SAP or Salesforce, we are not. There are, you know, somebody managing $100 billion supply chain ecosystem with a complicated software provider isn't saying, hey, I'm going to use SI and I don't need that. That's really not the goal. But on the other hand, I think the line between what an agent system like this is going to be and what like pure software is going to be will be absolutely is going to be bloody. And I can imagine a lot of easy use Cases like my sales team has to go update Salesforce quite often, you know, because we force them to update whenever there's use case transition and stuff like that. Can that be done with APIs? Absolutely. Should you be able to file a vacation on top of workday using our HR agent? I would say that's sort of a reasonable thing. And so we very much take this approach of be opportunistic but again operate from a position of value and strength and not just on naked ambition because I think that doesn't work out. But if on the other hand you focus on how do you like, what does value creation mean, what do these users really want? I think that tends to be much more durable.
- [13:30]
A
So you're describing a bunch of changes for the organization that you executed on very rapidly. Right.
- [13:36]
B
Both in always feels entirely too slow.
- [13:40]
A
Yes, I have always experienced you to be quite impatient. But for scale seems pretty fast. How do you. What, what is one tactical thing you were doing are doing from a leadership perspective in terms of like move faster or communicate new direction internally and get people on board because you, you know this is a, this is a. It's a broader and different vision for Snowflake than before.
- [14:02]
B
Change is hard. You have to acknowledge that. And driving behavioral changes from lots of people is incredibly difficult. We were measured about how we rolled out changes. For example, among the first changes were leadership and alignment changes and clearer accountability. That happened within a few quarters because you're not dealing with as many people. Yes, you organize the teams under them. But I would say that change also what we then called the war room or the pod model of product and engineering and or go to market functions will all work together. That was again an early change and it was done with small groups of people without necessarily disrupting lots of people. I would say other things. For example, rolling out coding agents to our engineers. That was a project. Not everybody wants to do it. Some people are skeptical, some people are not. I'm a big fan of combining bottoms up with tops down approaches. The example with coding agents is that Benoit, our wonderful founder who fell in love with coding agents, did more to drive coding agent adoption with engineers. And any number of words from me, you sort of have to find the right people, find the champions. My take is that every large organization has these forward thinking, curious, I'm going to work over the weekends to figure out how to do something kind of people. You need to find them, you need to encourage them. You need to elev and use that to drive change top down change can be helpful, but it really needs to come from a bottoms up perspective. We've rolled out coding agents to all of our solution engineers and they are excited because that just dramatically lowered the amount of time it takes to create a demo. Usually our demos used to be canned and they were not always customizable to a particular customer. But we can now be like, okay, we know the kind of data we think Ilad and Sarah are going to have as part of their podcast. Let's create a demo with synthetic data sets just for that. I think that's the kind of ability that we have gotten changes hard.
- [16:19]
A
When you and I first met and got to work together, you were an investor, then you were an entrepreneur. Did either one of those rules change the way you are a leader at scale or a CEO?
- [16:29]
B
I think these things are accretive. They add on to things in ways that you don't always appreciate then or ever. It is what it is. I always complained to my family about the 10 years that I spent doing research and getting a PhD. I was like, that was a waste of time, but not really. For example, doing a PhD teaches you to focus on ideas, teaches you to focus on how do you convey them crisply. You often spend like enormous amounts of time writing four line abstracts. But it actually turns out that's incredibly powerful to be able to convey ideas in an easy way. Niva is among the hardest and most heartbreaking experiences of my life. It is what it is. Sometimes you are too early. But on the other hand, I probably learned more about hustling, took success far less for granted, learned more about social or marketing or any of these other things. But you kind of take for granted if you're at Google, at Google, whatever you did. My first launch at Google, which was entirely my work, for three months, one person was covered by the New York Times.
- [17:42]
A
Okay, yeah, so you just have immediate scale with anything. You have distribution, immediate.
- [17:46]
B
You have distribution for your ideas, your products. Doing a startup reel makes you realize that that's actually special. And so I think I bring quite a lot of that when it comes to what does it take to hustle? What does it take to win? Honestly, I think both the Google and the Neva experiences make me somebody that's just a lot more grateful for my job. We talked earlier about how you have to deal with a bunch of stuff that you don't really want to deal with when it comes to doing something big in your life. I'm a lot more gracious about that because it is just such a privilege to be at a place like Snowflake to be having the kind of impact that we have. I remember what you told me after Summit. You said something along the lines of, thank you for inviting me to your rock concert. There was a line that was more than two blocks long of people waiting to get into Javits center in New York for this little conference that we were doing. I'm a lot more grateful for things like that. There's nothing ordinary about it.
- [18:50]
A
You, I think, are perhaps the world's expert on game theory and strategy with the tech elephants because you have led the elephant, fought the elephant and now built on top of the elephant. Right. And so I think the analogy broke down at some point. But in terms of the experience of building Snowflake on the cloud, service providers, and both for you today and then the analogy for anybody building on foundation models as you are as well, how do you think about that? What's a framework for creating durable value there?
- [19:28]
B
I think product market fit continues to be magical. There's a reason that Snowflake exists. Think about it. The three hyperscalers would love to just own the data space like they own any other space, but yet there's no Flake, there's databricks. And so that sort of redeeming value is quite unique and we should all have a lot of humility about what it takes to create that lightning in a bottle. Having said that, I think the model companies, especially OpenAI is super interesting because they are in that phase of their growth where they literally, they don't think they can not do anything. Yes.
- [20:18]
A
Yeah.
- [20:19]
B
I joke to people that these are like empires that have not met their oceans just yet. And so I think you do have to pay attention to what is likely to be in their immediate path. So, for example, I think coding agents are particularly interesting from this perspective because it is very clear that both Anthropic and OpenAI are going to be laser set on having the best one, best one that there is. So I think thinking about what is the likely trajectory of these companies and do they really have a right to win, or is it something that is different enough that you don't really have to worry about? Google, for example, stopped at information. God knows I spent enough time trying to get into physical things like shopping or airline purchases or hotels. We didn't really succeed because we didn't have core competence really in some fundamental way beyond the world of information. I think it's going to be fascinating to discover what that kind of a boundary is for an OpenAI or an anthropic But I think there are lots of areas that can be reasonably guessed at with respect to where they're going to go. I think that's the one that's tough. And early patterns of if you are, for example, a set of prompts on top of one of these models, that's a problematic space to be in. You kind of need to add value. I also think a lot about what differentiates us from these model companies. Is this an area that they're likely to be interested in? How do we make sure that we have distance with respect to what we add and a lot of the urgency and change in the products we create in collaboration with these folks. It all take us towards the data platform as a durable category. But this is also a time where literally no software company can feel secure about their position in the sun. And I actually think that that perhaps is just as important as anything else that I just said.
- [22:16]
A
Do you have that orientation of like, we need to continue to earn it.
- [22:21]
B
We need to continue to earn it. And if there is anything that all of us have learned, say from the CSPs, it is that they have infinite budgets, they have infinite patience and unless you innovate and stay ahead, not just be ahead, but stay ahead, you will be in town. I think that's another really useful lesson to remember as a company like Snowflake navigates the current realm.
- [22:46]
A
And this resonates hugely with me both on the dimension of I'm thinking about one of the founders, CEOs of one of my favorite companies that's now a public company that I thought was unassailable and you know, AI. I hate to be the person to be like, well, this changes everything. But they feel threatened for the first time in many years. I think that's a pretty common experience right now as a software CEO, I think especially when the technical environment is so fluid, defensibility is built, not strategized.
- [23:14]
B
That's correct. It's built every single day. You have to keep moving.
- [23:18]
A
One of the questions I would have on the data cloud is like you, even if you don't have the CSP's budget, you do have the ability to make multi year plans. And you joined Snowflake because you saw a vision for it to be much more than the data cloud. As a technologist, when you look out three to five years, how do you expect people to think of Snowflake both in the ecosystem and then customers to use it most?
- [23:45]
B
Our core strength comes in that data platform layer. I sometimes internally talk about being there for our customers, from inception to insight from when data is first conceived to when somebody gets an insight that feeds back into that system. In fact, the pitch that I make to CEOs or CDOs that I meet for the first time is really that the great companies of this century, a company like a Google or a meta, where more data first companies than like purely product first companies than pretty much any others compared to before you build cars and then you kind of did an instrumentation to make sure that it didn't crash or what the maintenance records for that was. Even products. Think about it. If you built something like Adobe photoshop In the 90s, you did a bunch of research, you built the product and then you sent CDs over to various people and then you waited for some feedback to come back. Data was always like a slow after that. But search ads, for example, was magical because the behavior, what people did, interacting with these ads went back into influencing what happened to that system pretty much like in a few minutes. And my data teams were as large as the product teams. And what I tell our customers is that we want to be that companion for all different kinds of data. We want them to have the expertise that the Google and the metas of the world have. And so we see AI as a massive accelerant for things like that. Because all of a sudden CEOs now realize, wait, this is not just about data modernization. This is not just about me being able to run more queries or perhaps code up a machine learning algorithm. This could influence how my business operations work. This could influence what efficiency means for me as a category. So I think that's the tailwind that we have from AI, because the value of data just got vastly elevated. That's our aspiration. And with respect to the CSPs, my take is that a company like Snowflake, which comes data first, as opposed to a set of services first, with an emphasis on simplicity and integration. You can create as large a database as you like on Snowflake, and it'll be completely shareable within the company, It'll be completely shareable to your partners. And when we talk about AI in Snowflake, it's not as an afterthought. It will work with all of the governance that you have put before. And that kind of an integrated approach, we think will have durability because over time, the idea of buying, let's say like raw compute and storage and writing code in order to solve a problem just, it never gets easier. And so we are a higher level of Abstraction pre AI. This was my main thesis for why I wanted to be part of Snowflake. I said a data platform that especially spans CSPs has the right you have to earn it, has the right to be as large as a CSP itself. And so that was roughly why I joined the company. Our medium term vision for what we want to be, and as I said, AI, I think is a massive accelerant on how do you get value from data faster or how do you get better at acting on data quicker?
- [27:11]
A
When you think about the overall data landscape today, there's the data that's traditionally been in Snowflake and then like, you know, the investments you've made in new partnerships you're announcing. Can you explain, like why SAP and some of the other partners you're working with now?
- [27:28]
B
Yeah, this is a good question. I think for a while Snowflake had a Snowflake centric view of the world. Plenty of people brought in data from SAP or from Workday or from Salesforce. But what is increasingly happening is that all of these companies realize that this is incredibly valuable data. It's not quite their data, it's customer data, but they understand that it is valuable. And they also understand that this line between software and services and software and data is also blurry in a pretty meaningful way. The one quality that I learned from Google, working in areas like payments, which is all about partnerships, was that partnership mentality was how do you pick a set of folks and figure out how you can create value together? Among the earliest places where this went to work was in our relationship with Microsoft, which was okay, but not that great because they had a 1p relationship with Databricks and they're always kind of conflicted about is fabric the answer or Snowflake the answer? Of course, you know, this Sathya is the master of how to create winning partnerships. And so we took a lesson from kind of and how to get out of them and how to. Well, and how to adjust them as you need to. So we've been working on a partnership with them for, you know, the past couple of years. This is both product integration with things like fabric, but also how do the companies work together. I think we're in a much better place now compared to say 18 odd months ago, where yes, there is an understanding that we will compete with some customers and that's fine and we will collaborate on a whole set of other customers where, let's say Azure plus Snowflake is a strict positive, which is the same again, the same attitude that we have with AWS and we are working on a similar sort of arrangement with gcp. I think the software providers are different. As I said, they understand that the world is changing. So with folks like SAP, we are actually thinking much harder about what is that one plus one equals three. With SAP, I think it's going to be absolutely bidirectional data share. But can we also collaborate in the area of analytics and AI and agents and make it easier for people to create these on top of SAP data? I think that can also become a leverage point for us to expand out to more companies because as you know, SAP has incredible presence throughout the globe. And so I think it represents a maturing of how we think about partnerships. We absolutely want to do this with a few other key folks. This is not the kind of thing that you can do with every company. But I think that partnership mentality, create value together is something that'll stand us in good stead and hopefully also be profitable for us. With respect to generating more business, I.
- [30:21]
A
Want to close out with two of the most common questions I get that I think you are more prepared to answer. The first is just with every enterprise customer I talk to, one of the first two questions is going to be where are the highest ROI use cases for AI? For my business. You run a large business, you serve large businesses. What is your Stack rank here? How do you think people should address it?
- [30:47]
B
Every company now has a set of technologists, even if they're not software companies. They need to deal with technology. I would say that coding agents are probably the easiest ROI item just in terms of making new projects faster, demystifying technology so that more people can get at it. As I said earlier, we are large users of coding agents and we're working on coding agents that are going to be part of Snowflake because we want to make it easier for people to be able to use Snowflake itself. What's good for other people. It's also good for Snowflake, I think. Absolutely. Areas like customer support, which fits the pattern of here is a repository of human knowledge. Here is an easy backup in case the AI cannot do something. Plus its ability for AI models to effortlessly tap into voice into typed questions and generate answers. That's an area where there is clearly a whole set of roi. Faster, easier, more seamless access to data, especially when it's combined with I don't have to pay for $50 per user per month license is another easy ROI item. And that's part of kind of our motivation for Snowflake intelligence of democratizing data access. These are among the areas where it's just more or less guaranteed roi. But the other way to think about this is I think obsessing about a lot of ROI too quickly is also a bad idea for many companies because you don't want your first step to be a hundred feet. You want to do a lot of little things that sort of prove value. There's nothing, you know, people can get plenty of value from using the ChatGPTs, even the free ones and tools like that for many day to day things that we do. I think the more companies demystify what it is to use AI, I think the more chance they have of figuring out how to get value because you take the risk. I place a lot of emphasis on how many shots do you take on goal, how many projects can you run very, very quickly so that you get a feel for what's the landscape of change. The sales data assistant had three versions that came before it. They all built.
- [33:09]
A
This one stuck.
- [33:10]
B
This one, well, actually they all stuck. They all added more and more things on top. You did. The first one was just on enablement. The second one was a little bit more about customer information. We also had an app called Customer. It was called Customer360. It was a streamlit app, so Python app that you could get that kind of information from. All of these then culminated into the sales assistant, which is all of these things combined. To me, it's that journey of iteration that's often just as important as I have that big one thing that I managed to launch. I prefer not to take big bets, whether it is in getting engineering projects done or these kinds of projects. I think iterating and creating value every step of the way is the key. And this is the same advice that I give to our customers. I go like, you should not spend a lot of money on AI with Snowflake. You should do it a thousand bucks at a time. And when you have significant value that you feel good about, then you can wrap it up.
- [34:06]
A
This is one reason I'm very bullish on applied companies that know what their immediate usefulness is. Because the landscape of what you could do with AI, if you have customer trust and you understand the workflows is very large. And I think there's just land for the taking for people who have the velocity and sort of also the paranoia to keep expanding into that. But it is also an argument for like why that layer should exist because you're just reducing the time to value versus somebody building it themselves with A generic framework or just straight APIs and engineering work as well.
- [34:45]
B
Yeah. And what a lot of our customers have found out, for example, is that creating something like Cortex Analyst, which in some simple way is text to SQL, is actually a much harder problem than they actually think. There's more trust in Snowflake because we did many, many analyst projects before we ever got into something like Snowflake Intelligence, which is a step level increase, both in capability but also complexity.
- [35:08]
A
And the other question that I get asked a great deal is what do you think happens to ads on the Internet if we have chat interfaces that are much more directed instead of offering you 10 blue links, I have to.
- [35:24]
B
Ask you this is a great question. I think advertising is just an incredibly powerful medium and it's an incredibly powerful business and there are good ways of doing advertising. I'm actually seven years from Google. I'm actually more proud of the work that we did in the search ads team now than I was when I left Google. I think there are good ways of doing it and you know it when you see it. It's a little bit of, you know, it's very clear it will reinvent itself in the chat world. Let's just hope it doesn't become more insidious in terms of discoverability and being able to tell what's an ad or not. It would sure be creepy for you to have a psychiatrist that has like a certain affinity for prescribing one medication versus other. Those are the kinds of unfortunate things that we will discover. But the ad model is here to stay. It will just come in different forms and I think as long as it's done well, it's a reasonable model. And as a consumer you also have to be smart about what's in these things for you. And I think ever more than before, there's an increased premium on preserving our agencies. I think that is what we all have to do as individuals.
- [36:46]
A
I am really encouraged how strongly citations and sourcing in models has taken off in different experiences. I certainly think that the set of things that are being presented to consumers has narrowed a great deal. The fact that people want to go look at primary sources and understand where information has come from, even given all of this reasoning is a useful indicator.
- [37:11]
B
It's a very positive thing, I think. And the nice thing about some of this is that it is not a whole lot of work for you to say take a deep research article written by Gemini, paste it into ChatGPT and ask it to verify all of the links that are there. I think we do have more tools. We have talked about how we did citations at niva, how proud we were then that we launched it in, in early 2023. I think that's an idea whose relevance is still as strong as ever. And I think products like ChatGPT deep research are truly amazing in terms of the value that they can create for anyone. I mean, think about it. You and I can get an expert paper literally on any topic. We just have to have the brainpower to be able to digest it. I think that's pretty amazing. And it's really fun to see some of these core technologies embrace things like that as opposed to just writing like here's this article. Take it or leave it.
- [38:15]
A
I want to ask you one last architectural question. Because you have worked for such a long time on information retrieval and search, you also understand LLMs. You work with a lot of structured and unstructured data today, so you just have a very well rounded point of view. I think there is a contingent of folks that believe that traditional informational retrieval techniques and indexing is less and less relevant as more data is available through the model, even in enterprise use cases or non consumer use cases. How do you think about that?
- [38:48]
B
It is tempting to trivialize things like search as just information retrieval. The insight that powered Google was PageRank. It was a way of harnessing the power of the entire Internet to figure out what was popular and what was not. But PageRank ran out of juice like in six years, like 2004, five. And while Google never really liked to talk about it, the kind of things that became more and more relevant was the click behavior on top of of the search results that Google presented. It was that feedback loop that eventually gave it so much value. And so when it comes to AI systems, including si, remember I talked about eval loops and that's a fundamental construct that you need to be able to launch some meaningful product. But it'll also turn out that that's the construct that you need for that product to get better and better over time. Perhaps we will figure out a way to encode that as well into the context that's presented into the model. But to me right now it's similar to should LLMs be able to do math? Okay, you can argue yes, they should be able to do math, they're so powerful. But as any reasonable person will tell you, a smarter person is going to say no, they should not do math. Instead I should write the two lines of Python, which I know how to direct and run the Python in order to solve the math problem. I think of trust in a very similar way. There are well known solutions for figuring out what is the most trustworthy when it comes to a question that you want to answer. Why would you not use that and think of that as another tool that whatever AI system agent system that you're building is going to use, rather than be kind of like in this maximalist mode of the AI can solve everything. I think all practical people will use the best tools available to them at a given point in time. At least at this point in time there's enough value from these outside tools, including search, that I don't see the point of trying to dismiss it right now.
- [41:06]
A
I mean it's a very principled point of view of like a maximal intelligence will use reliable tools wherever 100% wherever it is available. You cannot be so smart that you don't use the computer.
- [41:19]
B
Exactly. Or there's no bravery in just like hard work without if something can be done easily, you get to focus your energy on other things. So I think that'll very much be the case. And the prevalence of things like search APIs is actually a testament to the fact that all of these models benefit from things like that because they provide that external information that is not easily internalizable just into the AI model just yet.
- [41:47]
A
Sridhar, thank you so much for doing this.
- [41:49]
B
Thank you. Thank you, Sarah.
- [41:53]
A
Find us on Twitter at nopriorspod. Subscribe to our YouTube channel. If you want to see our faces, follow the show on Apple Podcasts, Spotify or wherever you listen. That way you get a new episode every week and sign up for emails or find transcripts for every episode@no-priors.com.