← This Week in Startups
This Week in Startups
Inside Harvey AI’s $8 billion AI lawyer app, PLUS How OpenRouter unites the LLMs | E2207
01:03:04
Loading summary
Transcript117 lines
- [00:00]
A
Hey everybody. Welcome to this week in Startups. This is Alex and today on the show we have a couple of amazing Twist 500 interviews for you. And then at the end, Jason is going to jump on and do a couple of founder questions. So first up, Gabe from Harvey AI. You have heard of this company. It's worth a lot of money, it's growing very quickly. I wanted to learn more about why the law is such a ripe place to apply AI today. Then we are talking to Alex from OpenRouter, an amazing startup. We have used their data on the show a number of times. I was really curious though, what's the business behind the hype? We learned that and more. And then as I said at the end, Jason's going to pop on and answer a couple of questions. All right, let's go. This Week in Startups is brought to you by Lemon IO. Hire pre vetted remote developers and get 15% off your first four weeks of developer time at Lemon IO. Twist. Uber bad data equals bad AI. Your AI is only as good as the data it learns from Uber. That's right. Uber AI Solutions now works with enterprises around the world to source, label, evaluate and scale real world high quality data for every industry everywhere so that you can focus on building the next big thing. High quality data equals Smarter and faster AI. Uber.com and interpret interpret turns feedback noise into customer intelligence so your team knows exactly what to fix and build next. Head to interpret.com twist to book a demo and see it in action. Hey everybody. Welcome back to this week in Startups. This is Alex and we are joined today by an entrepreneur and a founder that you have heard of. The legal profession has become a hotspot for AI innovation. Lots of startups are raising money and growing very quickly, but, but at the absolute top of the heap, there's a firm called Harvey. You've heard of them? I've heard of them. I've long wanted to learn more. So please welcome to the show. It's Gabe Pereira, the co founder and president of the company. Gabe, how you doing?
- [01:59]
B
Good, how are you?
- [02:00]
A
I am, I'm actually quite good. I'm a little bit shocked though that you're not suited up. I thought anyone who interacted with the legal field had to pretend like they just stepped off an episode of Suits.
- [02:10]
C
This is, this is what Winston and I thought when we started the company and really early on we, we had a customer event and we bought suits suits and showed up and they were like, can you guys please just wear hoodies? And so that's why I'm in a hoodie.
- [02:23]
A
Before we jumped on, we were joking. We were going through the fundraising history of the company, and you guys most recently raised 150 million from Andreessen, $8 billion valuation. And I joked, you know, did you buy a boat? I didn't think you actually had, but you let me know that you're. You have a unique sleeping arrangement right now.
- [02:39]
C
Yeah, my mattress is still on the floor, so we gotta. We gotta fix that at some point.
- [02:44]
A
You really don't actually. I mean, what. You seem fine. Company's doing great, so I don't see any issues there. All right. For folks out there who don't know what Harvey does, I think we should start with a little bit of kind of just brass tax. So the way that I understand the company today is that it has two major components. One is a thing called Vault that allows law firms to bring their information, their data, case history, just all of their stuff to the platform. And then you have a tool called Assistant that uses that information and does a lot of work for people doing drafts, that sort of thing. Is that a fair summation of Harvey today?
- [03:16]
C
Yeah, I would think of where we're at today is, I think, kind of how Cursor built this ide, or didn't build the ide, but kind of added all this AI. I think the big challenge for legal was there didn't exist this kind of single workspace where you could work on these. In Legal, they're called client matters. And so a lot of what we've been building in the past past three years is how do we build a single workspace where either a lawyer or an agent could perform a client matter, whether it's a litigation, an M and A end to end? And I think the challenge with that is lawyers use kind of a wide variety of tools. You need case law, you need data rooms, you need ediscovery platforms. And so we're starting to partner with a bunch of these folks, and then we've built some of this functionality. But I think the more or equally important piece is it's not just building the tools for individual lawyers. One of the massive challenge that these large law firms have is how do you manage all of these different client projects? And so if you're one of these large law firms, you're doing tens of thousands of client matters, you have millions of historical client matters, and you need to make sure all that data is separate. And so there's a massive part of the product that I think doesn't get seen as much is all of the governance and admin controls you need. And then another thing we're working a lot on is a lot of this work is not just how do law firm lawyers collaborate with each other, but how do they collaborate with their clients? And so now a large number of our customers are clients of law firms, and we're building a lot of functionality for these two parties to work together on client matters.
- [04:54]
A
So if I'm a law firm and my client is also a Harvey customer, I presume there's a kind of a bridge between the two that lets Harvey kind of talk across the sides. But let's say that I'm. I don't know, a private equity firm, and my law firm doesn't use Harvey, and I do. Does that create added friction or does that still help the overall legal process?
- [05:12]
C
Yeah, I mean, I think what we're seeing. So when we sell to enterprise, I would think of there's two large things that our enterprise customers are using Harvey for. There is a bunch of internal legal work that these enterprises do that they typically don't collaborate with outside counsel. And so things like co contracting and other internal legal processes. And so we're building products there. And then what we're seeing increasingly is when these large private equity firms or enterprises buy Harvey, they're starting to ask their law firms to also buy it so they can work together. And so I think we're starting to see a bit of this network effect from that.
- [05:50]
A
Really push your effective customer acquisition costs down. Or is this more like an edge benefit that hasn't changed the economics of sales?
- [05:58]
C
Yeah. Oh, I think we've seen this make a huge difference. Like, there are. There are deals we get, and we see this actually in both directions, where we see law firms going to their clients and saying, hey, we can work better with you if you buy Harvey. And then we have inbound from enterprises saying, hey, we want to buy this so we can work with our law firm. And then we see it in the other direction, where we've seen some large enterprises buy this and then reach out to their entire panel of law firms and say, hey, can all of you purchase Harvey so we can collaborate? And so I think this is actually quite a big effect.
- [06:27]
A
That's fantastic. I mean, everyone wants distribution. And in the world of selling to large companies, it's very hard to get in the door, let alone to get something close. But I presume that if your law firm's client is saying, please use this, that probably reduces the time to sale by a factor of 10, if not more, just for you guys?
- [06:43]
C
Yeah, exactly.
- [06:44]
A
All right, now I want to talk about the use of AI inside the legal profession, because we all know that it works well in coding because there's a huge corpus of online information. There's tons of code you can pull in and learn from. My thought was that the world of legal stuff is a good fit for AI because it has its own kind of grammar to it. It's structured in a particular way. There's a huge kind of corpus of case law and legal writings out there to pull from. Is that correct? And is there something that I'm missing that makes AI a good fit for AI?
- [07:15]
C
Yeah, yeah, no, I think that's exactly right. Like, I think there's a ton of analogies between code and legal, and these feel like the two domains where you've really seen these types of companies be able to be very successful. And I think it's for the exact reason you said. And then I think the one thing I would add is a lot of the job of these associates is just synthesizing massive amounts of information. And so a lot of times when you get staffed on a litigation or a merger, if you're a junior associate, it's not just, can you understand all the legal context. It is, oh, I'm representing this pharmaceutical company, and I actually just don't know how pharmaceutical supply chain works. And I need to, like, understand that and then understand how that connects to historical litigations. And so there's just this massive, like, learning problem. And I think what we see now with things like Harvey is these associates can use web search to go understand the industry, and then they can use Lexis to do this research, and then they can connect it to, here's a discovery corpus with all the emails, and then use all of this to start drafting, like a motion or something like that. And so it's these really complex workflows across all these different data sources that you need to synthesize.
- [08:27]
A
And you guys recently announced some sort of deal with Lexis, right?
- [08:30]
C
Yeah. So we have a partnership with them for case law, and so we're both working with them to kind of provide this in our platform and kind of expand this as well.
- [08:41]
A
The road, though, to applying AI to the legal profession started off pretty hot. There's a funny anecdote, I think it was on somewhere on OpenAI's own blog, how you guys used GPT3 early on to do landlord tenant questions from Reddit's legal advice subreddit and found that quoting for 86 out of a hundred questions. Attorneys you talked to said they would just send it on, but then later on OpenAI also said that you guys also tried things like fine tuning foundation models and using RAG to try to bring more information into AI models, but found that that was kind of lacking. So it seems like there was like initial fit between existing AI technologies and legal questions and then you ran into some complications along the way. So can you just kind of like chart for me how that progression has gone?
- [09:25]
D
If you're not listening to your customers.
- [09:27]
E
And iterating based on their feedback, your startup will fail. So if you're drowning in feedback from reviews calls, it's time for N terpret. Interpret is the customer intelligence platform that helps you turn random comments and feedback into actionable insights that help grow your company. Their AI system is already trained on your business and your product. It then reads all your support tickets, it reads all your reviews, call transcripts and surveys. Then their agents actually update your team in Jira, Linear, Zendesk and Slack. So this feedback doesn't just sit in some email or messaging thread, but it actually helps keep your team moving the ball forward. Find out why canva, Notion and Perplexity are already using Interpret to stay on top of what they need to fix or build next. So if you're ready to turn feedback chaos into customer intelligence, head to interpret.com twist to book a demo and see it in action. That's E N T E R p r e t.com twist yeah, so.
- [10:27]
C
So I would, I would think of the big difference there is consumer versus corporate law. And so when we started, Harvey Winston and I were really interested in kind of access to justice and could we use this technology to provide legal services kind of more affordably, but for consumers and the models at the time, this was before GPT4 were already quite good. Kind of like you said, if you took a lot of these, hey, I got evicted, here's my lease, what should I do? So much of this is online that the models were quite good. And I think the challenge is there's a bunch of regulatory issues around unauthorized practice of law, things like that, that I think it's something now we're starting to work with, for example the Supreme Court in Singapore and so governments and court systems to see if we can apply this technology. But I think the core business now is more corporate law. And that's where the challenges are much more difficult. Difficult because most of that data is not public.
- [11:28]
B
Right.
- [11:28]
C
If you think of what these very large law firms are doing, for corporations, it's I have a massive litigation, I want to do a very large acquisition. Most of the data you're working there is very sensitive client data that actually never goes into the public models. And so a lot of that is missing. And then the thing that you mentioned we did with OpenAI, that I think is still a really big challenge is we are trying to make these models better at legal research. And so case law is semi public. And so the models have a lot of access to it. And I think what you see is their legal reasoning on things like Supreme Court cases is actually quite good. And my intuition is the reason for that is it's not actually because the Supreme Court cases are public. It's because there's so much public analysis of all of this data online, of all of these legal scholars talking about, here's the implication of this case, here's how it relates to this case. And so a lot of what you see the models doing is being able to interpolate between that and then it looks like this very powerful legal reasoning. And then the issue we run into is when you try to extrapolate that to here's the more nuanced cases that are not Supreme Court cases, but a law firm would be researching when they're doing kind of a specialized matter for a client, there isn't that writing and that work around it publicly. And so the reasoning model, the reasoning of kind of the general purpose models falls off. And so I think we have this big intuition where a lot of the legal expertise is locked up in these law firms. And what we need to do is help them train these systems with their expertise. So each, each law firm can kind of put that into their own system.
- [13:08]
A
So you guys have a history with OpenAI. They were an early investor in the company. And then I think it was in May of this year, you guys added models manthropic and Alphabet to the product, kind of giving yourself a multimodal approach to the market. I'm trying to understand exactly what clients need to do. Do they end up training like their own version of a general purpose model with their own data? Is this entirely rag? How does the architecture work to allow for multiple models given the technical questions you just discussed?
- [13:40]
C
Yeah. So I would think it gets complicated because it depends on if it's a law firm or a law firm's client. And so I would think of in the past three years, not just for us, but I think for most of these domains, a lot of the challenge is still not the post training and personalization of the models, it's really connecting all of the context. And so part of why we've been doing all these partnerships with Imanage NetDocs kind of document management systems of the law firms is it's still very hard to get all of the context that a lawyer needs into the model before you even start thinking of how do you customize this. And so the experience you want if you're a lawyer at a large law firm is you're always working on a specific client matter. And so you want to be able to query that client matter. And so if it's a litigation you want, here's all the emails, here is the ediscovery corpus, the case law, our private documents, and all the work products we're working on. And I want my model, my agent to be able to interact with all of that. And so a lot of what we've been doing is how do you pull all that together?
- [14:44]
A
I thought model Context Protocol or NCP from Anthropic solved that for everybody. And it was a done deal, Gabe. And we were all done.
- [14:50]
C
I think it's a good abstraction. And then the challenge I think with all this enterprise stuff is it's almost never just a technical problem. It is how do you connect all these systems. But it's moving there. So I think with law firms we're actually getting there where we are starting to be able to provide this experience where an attorney has all of that context. Once you have that, the next challenge is we can't train on any of that data. And so we're fully eyes off. We never use any of our client data. But even more so, the law firm can't do that because if you think of the data that the law firm has, it's actually client data from a bunch of different clients. And so the infrastructure we need to build for law firms is how do they partition all of that data? And then for a specific client, can we build the infrastructure where if you have a large private equity firm and you do all their fund formation work, then as a pair you train the system on that. If the client and the law firm are both okay with that. And so we're starting to have some of those conversations. And then if you think of it from the client side, what the clients eventually are going to want is I have one legal model and it won't probably be a single LLM, It'll be some complicated LLM system that has access to all of my company's legal data and it connects to all of my different Law firms. And it can learn from the work that all of my law firms and eventually professional service providers are doing for me.
- [16:19]
A
Because you can probably learn from how people use the product and service, what they're looking for when they look for it, what type of documents, you know, constitute a successful search. Even if you can't look at the actual information in those documents because of privacy, you know, client litigator confidentiality and so forth. That's a big problem Gabe. Cause I think most of the people can, can more easily kind of just hoover up the information and use it. You guys really have to be as delicate as the healthcare industry. It feels like.
- [16:44]
C
Yeah, no, and I think this is something from when we started the company. Like we brought on the CISO of Goldman and we brought on a bunch of these security advisors because I think the really big challenge we're going to solve is for banks, for government. How do you build these systems in a way where you can provide that gen AI experience but it meets all of these kind of regulatory issues. And I think even before you get into the gen AI, there's already these massive technical security, legal challenges. And then when you think of the balance of like we want to use gen AI, we want to get some of the training benefits, but you also have all of those issues, I think that is going to be a really hard technical product business change management problem.
- [17:25]
A
So not to be, not to be a brat. I know there's tons more work to do in the legal space, but it feels like you're building a bit of machinery that could also be applied to other industries where data is quite regulated but they still want to use AI. Is there down the road a copy and paste into a different industry possible or is this going to be so law specific that it actually won't translate as I'm kind of imagining to something?
- [17:49]
C
Yeah, we're actually already doing this in some adjacent industries. And so we do this for tax for example. And so I would think of the problem we're solving and the model where this applies is what are all the industries where you have advisors to companies. And the big technical problem that you need to solve is as a company I need to share my sensitive data with this advisor so they can operate on that data. And this is investment banking, consulting, tax audit, HR compliance, maybe insurance a bit. And so I think this expands to kind of a bunch of these different industries.
- [18:26]
A
Sounds very complicated and tricky. But the good news is that growth has been really, really impressive at the company you guys scaled from. I think it was 50 million ARR kind of at the end of last year, up to 100 million ARR this August. And I know you guys have I think around 500 law firms that are clients. So talk to me about who's buying today. Are they domestic, international? Is this just big law? Because I've had friends that have worked at big law. I'm curious what the customer mix looks like today.
- [18:51]
C
Yeah, so it definitely started our focus was the largest law firms and we actually started outside the US and so we worked with a lot of the large European law firms. Now we're in over 50 of the Amla Hundred. So kind of the top hundred firms and kind of a combination of, you know, a lot of the really large US ones and then a lot of the really large international ones. Mid market is growing pretty, pretty quickly. And so we're starting to sell to smaller law firms and then enterprise kind of the clients of these law firms, private Equity and Fortune 500 is also growing pretty quickly.
- [19:26]
A
It feels like the addition of clients of law firms greatly expands your tam, because each law firm has actually, I don't know the answer to this. Dozens, hundreds of individual clients, tens of thousands. Yeah, see, I was way off, but that's a lot more potential contracted value for the company. Do you think that's going to become the majority down the road?
- [19:44]
C
I think for us, the, the interesting question is figuring out how to balance these parties. And so one of the big problems we want to solve is there is inherent tension of selling to both sides of kind of this marketplace. And what we're starting to figure out with law firms and their clients is, is the ideal win here is we can find a model where for law firms, we can make them more profitable on this platform. And for their clients, they can also benefit. And I think sometimes when people look at this, they say, okay, one of these parties needs to lose. But I think when you actually get into the complications of how these relationships work, we're starting to figure out ways that you can actually make this a win win for both parties. And then I think to your point, it expands the tam because once we go into enterprise, there's stuff we are able to do for enterprises that is not work they use outside counsel for. And so, yeah, things like contracting, tax, HR compliance.
- [20:42]
D
So yeah, with AI, like everything else.
- [20:46]
E
The old saying is so true. Garbage in, garbage out. Without the right training data, you're not.
- [20:50]
D
Going to get great results. But my pals at Uber, they're now working with enterprises all over the world to source, label, evaluate and Scale high.
- [20:59]
E
Quality, real data for your startup.
- [21:01]
D
Uber is one of my favorite companies.
- [21:02]
E
On the planet because they demonstrated their.
- [21:04]
D
Ability to scale and to build great products first in the rideshare market, then they figured out Uber Eats and then they moved on to autonomous driving. And all of this was only possible because of a deep understanding of how to collect, label, understand and analyze data. Now with Uber AI Solutions, you can put the team and tools that turned Uber into one of the world's best companies to work for you, partnering with top talent and experts from Uber's global network. What better partner to help you scale your business than the company that organizes over 36 million rides every single day? Book a demo right now by going to uber.com twist that's u b e r.com twist.
- [21:45]
A
What is the tension though? I'm curious about this because it feels to me that if a law firm uses Harvey and it's more efficient and just better at their jobs because of it, their clients would be happy, not annoyed that they had this technology. And the other way around, if you could work more easily with your clients, why is that a bad thing? So what's the tension you're describing?
- [22:02]
C
Yeah, I mean, I think the simplified version is kind of the billable hour, where when you're charging for work on, on an hourly basis, there is some tension of the efficiency. But I think when you actually look at how these law firms operate. So like one, one good example I use is, is fund formation. And these aren't purely built on billable hour. Like a lot of this work has caps and there is certain of the work that you have to do that gets written off. And so there's all these extra degrees of freedom. But I, I think there is just a challenge, right? It's like if you think of that scale, if you are a very large company and you're spending $2 billion on legal, like these CEOs are giving mandates to their general counsels or their legal divisions and saying, hey, I need you to use Gen AI to reduce legal spend by 10%. Right. And so that obviously is going to put some pressure on the firms. But I think the part that's exciting is we've talked to law firms where they say, okay, maybe we're going to make a bit less on a client matter. But there's firms we're talking to where they say we can actually serve this diligence on a fixed fee or this litigation on a fixed fee, or we can take a bit of a cut here, but we can do this at Scale with better profit margins. And so I actually, my gut of what's going to happen is you will see law firms that are 10 times larger than the current law firms and they're able to operate at a much larger scale because I think Gen will let them decouple revenue from headcount and so you'll be able to get closer to these software margins and kind of like scale beyond your headcount. So I think there is a win, but it's obviously a very challenging thing to figure out at this stage.
- [23:44]
A
I want to squeeze in a couple of quick ones about the market itself. You guys, I think told Fortune that your largest competitor is by far indirectly open AI and that as general model companies get better and better, you guys have to stay, you know, reasonably far ahead because you know you're doing something specific. How much competition do you think actually comes from the major companies versus the startups out there? Because you know, Spellbook and Nexle and Lucio, there's a lot of companies that are coming after parts of the AI legal world. So who's really bigger in your kind of day to day? Is it OpenAI and company or is it more the startups out there that are extra hungry?
- [24:20]
C
Yeah, I think this is the challenge with, with building one of these companies is you have to operate on all these different time frames. So for example, on the day to day now when we go into deals and things like that, especially with law firms, OpenAI doesn't come up very much. Right. Copilot but they, some of the law firms use this but it's really like we need a legal technology vendor because this is such a vertical industry. And then where you see some of it is when you sell to these very large corporates and they're not using OpenAI for their legal but they are starting to think about how do we, you know, replatform our company or connect all of our divisions using this. And I think that's going to happen. Right, right. Like you will as a large company you will have something like a Copilot, a Claude or a ChatGPT available to a large amount of your workforce. Right. And you will connect all of your data systems. And so we don't see the kind of the value we provide as a company as long term the individual assistant. But this creates friction as you want to like sell into these organizations. But eventually what we need to provide and is our value prop that I think will be very different than what the foundation models provide is like what we really need to sell to law Firms is here is how you create a more profitable law firm. And it's more than just here's a product for an individual lawyer. It is, here's all the infrastructure to transform your law firm into an AI first law firm. And so it is the governance and the change management, the training ways to deliver these services in new ways. And so that part feels like that's very differentiated. The horizontal players won't build that and then connected with the enterprises. I think there is short term and we had this historically with the law firms and it's gone away. The self build right where some in house departments will say oh maybe I can just build this contracting solution myself. And my sense is what's going to happen is similar to what happened with Salesforce where a lot of folks were like oh it's just the database table, I can like put all my client data there, I'll just build that myself. And at scale you really need something like Salesforce because of all the things you need to build on top of it. And I imagine that's going to be similar of what happens with these in house legal departments. Most of the how do you optimize your legal spend is not going to be you deploy 10,000 super smart legal agents. It's going to be all the pro process management and the measurement connecting with law firms and things like that. And so I think that's the part where we don't see too many competitors working on that, but I think eventually getting to that. There's friction kind of from all sides and I think everyone faces that.
- [27:04]
A
When it comes to bringing Harvey into clients though, you guys are hiring forward deployed engineers and investing a lot in customer success. I was just curious as a final question here for other founders out there, how much work do you think AI focused startups should be doing to help handhold their clients customers and get them ready to go? And is that amount of work going down over time as AI gets more advanced or is it still a pretty big cost for you guys?
- [27:29]
C
Yeah, we actually historically haven't had a large forward deployed motion and so pretty much 100% of our platform is things that we have built that we are selling to every customer. I think like the way that we are thinking about for deployed is as we go into new verticals, especially for enterprises, how do we figure out what are the right building blocks to put in the platform? So a good example is we are starting to build for private equity outside of legal and a lot of these private equity firms want us to build essentially a fund Operating system system. Right. Like a platform where they can manage all of their funds, they can find all the documents. If they want to create a new fund, they can connect with their law firm. If they want to make an investment out of a fund, they can connect with the firm that does that. But if they, then they can also do the investment diligence and things like that. And I think what we're finding is a lot of these tools that we built rhyme and so a lot of the diligent stuff that we need to build for this is actually things that already exist in Vault. And so really figuring out what those abstractions are, I do think as we scale, especially for these enterprise customers, there is some of this you will need to do. Because historically or the problem we hear from all these clients is just we don't know where our data is both on law firms and enterprise. We have all these custom built systems, we need to connect them. And there is some of this where you can't just go in and say, hey, just put MCP on everything and we'll just absorb it. Like you really do need to sit down with these, with these clients and figure it out. But it feels, it feels very scalable. Like this isn't something that, that I think has been a big issue and like we've sold the same platform to all our corporates and all our big law firms.
- [29:16]
A
Awesome. All right, well Gabe, an absolute treat. Thank you so much for coming on. I presume you guys will be at 200 million ARR by January, February at the latest. So we'll have you back on when it's time. Harvey, AI, what's the website? And just for fun, is there a job you guys are looking to hire for? You want to shout out to the world all roles.
- [29:34]
C
So yeah, please apply.
- [29:36]
A
All right, Gabe, talk to you soon, man. Appreciate it.
- [29:40]
D
Founders, let's be real.
- [29:41]
E
Finding the right developers is really hard when you're trying to run and scale your startup. But Lemon IO can save you time, money and a ton of headaches. They've pre vetted Hill high quality developers and they've ensured that they're experience result oriented and that they're seeking competitive rates. These are handpicked devs with a minimum of 3 years of on the job experience and only 1% of applications to Lemon IO are accepted into their program. So forget the long drawn out hiring process. Lemon's going to match you with a skilled developer in just a few days. And they're not just recruiting talent either, they're helping you integrate them. Into your team. And if anything goes wrong, Lemon will find you a replacement developer. Asav ap. We've had a ton of launch founders work with Lemon IO. So go to Lemon IO twist and find your perfect developer or tech team in 48 hours or less. Twist listeners get 15% off their first four weeks. That's Lemon IO twist. L e m o n IO twist.
- [30:41]
A
AI is an enormous deal. Everyone wants to use it. But there is one startup out there that provides a unified API endpoint, or hook if you will, that lets companies pick and choose their models at will or helps them select. The company's called OpenRouter. It's fantastic. I live and die by its data. So please join me in welcoming Alex Atala to the show, Co founder and CEO at Open Router. Alex, how you doing?
- [31:04]
B
Doing great. Thanks for the intro, Alex.
- [31:06]
A
Yeah, my absolute pleasure, man. All right, so Open Router, I think of it as a unified API that provides access to tons of different LLMs and inference providers. Is that a fair summation, Alex?
- [31:18]
D
Yeah.
- [31:18]
B
There's a new model every 2, 3 days at this point and most people are only familiar with one. So what we try to do is introduce the world to the wide variety of options out there. It's kind of learning that there are like multiple people in the world that you can work with as opposed to just one.
- [31:36]
A
I'm blown away.
- [31:37]
B
And there are true nuances and advantages to exploring the diversity of language models and intelligence.
- [31:45]
A
So when we think about the models in question here, we're not talking about just western closed source models, but also a deluge if you will, of really high quality open source models from China. There's Mistral over in France. Where else are we seeing new AI models pop up from?
- [32:01]
B
Yeah, there's a couple in China that are doing pretty well. There's Kimi K2 thinking, which launched last week, Minimax M2, both really strong reasoning models. We in France, Mistral is really like major contender for European language models we have in, you know, obviously in the US like the heavy hitters OpenAI, Gemini, Anthropic and Xai and, and we're starting to see like a longer tail of mid sized language models that specialize on specific types of tasks. Models are being used more frequently in long running jobs. This could be like building something in the background. This could be using a bunch of resources. And these long running jobs, they don't require deep knowledge about the world, they require resourcefulness and that's kind of a whole new skill and a whole new direction for the AI. Economy to move in.
- [33:11]
A
Now, I thought it might be a good place to start by explaining how OpenRouter is different than Amazon, Bedrock or Azure AI foundry for folks who might be familiar with one particular cloud or inference provider. So for people out there listening, how do you guys compare to those and why are you better?
- [33:27]
B
Yeah, so the primary reason is that there's more options on Open Router and you're able to observe everything going on in one place, regardless of where your credits are coming from. You can bring credits to Open Router all of your existing deals, including things that you do on Bedrock. You can connect into OpenRouter and have it all show up there. So Open Router gives you a single pane of glass for discovering new models, discovering existing ones that you have access to, doing inference the exact same way through all of them. We normalize and standardize and fix errors with all requests, and we upgrade the quality of inference pretty significantly. There are all kinds of paper cuts and ways to shoot yourself in the foot when you switch between providers or try to access a new model. And something we see very often is some company is struggling really hard with like all models today on some problem and a new model pops up and an engineer will be like, hey, I want to try this out and see if it fixes our issue. But oh my, there's a new provider, we have to like set up a new vendor, we have to do a new billing relationship on that, we have to do a new enterprise relationship. Does it have a data policy that like matches with all of our internal policies? And a whole bunch of brain damage happens just to see if the problem is solved. So we delete all of that and just make it so you can instantly test out new models and then use them very efficiently in production too.
- [35:03]
A
So I think a lot about APIs abstracting away complexity, much like how Twilio famously is the UR example abstracted away telecom complexity. It just gave you a single API call. It really feels like you guys are taking a wide number of options providers and so forth in the AI world today and really just making them incredibly simple and easier to consume. And my presumption there, Alex, is that if it is easier to consume, easier to try more things, easier to experiment, people will wind up using net more total LLM inference or just more tokens perhaps is a better way to say that.
- [35:36]
B
That is definitely a part of the thesis. Additionally, when you have more options, you kind of improve your leverage over the whole ecosystem. So, you know, if AI were had zero competition, there was only one company that mattered or only one company shipping models. You know, there isn't a point to Open Router, but the more competition there is and the more like, diversification of features that emerges, the more ridiculous it is for a developer to try to take advantage of all those features. And that's where we come into play. You get them all out of the box without you having to add complexity to the work you're doing.
- [36:26]
A
Also, by making it easier to sample, to switch and so forth, you also effectively increase the quantum of competition inside of the AI market. Because if I had to go to OpenAI only and select them and live and die with them, I'm not really in a competitive marketplace. I have vendor lock. But you guys making it easier to swap between them, it makes your Googles and your anthropics and your mistrals and your, you know, kimmies just more in competition with one another regardless of where they're based and so forth. So it kind of turns vendors into just, just competitive blocks.
- [37:01]
B
Like, we also help bring up distribution for all of our vendors and all of our providers though. So, like, you know, a lot of the people who try out new models when they launched would probably never have done it had it not been really easy to just switch and explore it. So you can kind of, from the provider's point of view, you can think of us kind of like a distributor where we're just helping you discover new users. And once a user gets really, really into a provider, sometimes we'll go to a direct relationship and then connect everything in one place back on Open Router afterwards. But that way we kind of help people grow up and then scale up.
- [37:44]
A
Do AI model companies ever get a little bit annoyed that you're in a way disintermediating them from their customers? I mean, I know everyone wants inference and so they're probably happy to serve the demand, but I would presume that Anthropic would rather have a direct relationship with Alex Incorporated versus me using openrouter's API to access anthropic technology.
- [38:06]
B
So in the early days it was definitely a little tense because there was, it was. We were just kind of a strange new company. It wasn't a category. And that was definitely, I think, a big question. And initially our relations were on the 10th side. And over time it's kind of gone completely 180. Where now the model providers and the model labs see the value in routers and it's now this kind of new AI product category routers are around to stay. We have competitors too. And the product category helps them get distribution. Like I mentioned, it also helps them learn about their models. We do these stealth models where model labs will launch a model ahead of time and learn from our community and our power users and the initial benchmarks that our community runs on the model. And then we did this with GPT5, with GPT 4.1, with. With Grok4, and currently a model called.
- [39:29]
A
Polaris Alpha, which Alex, before the show declined to break confidentiality in his NDAs and tell me what it is. But one thing I did notice just looking into its stats is that it's already seen around, I don't know, 30, 35 billion tokens processed per day. I produce presume that companies come to open Router with a model for some discrete testing because you guys have just a volume of users that will immediately tap into it and put it to work. That's a lot of tokens. Alex.
- [39:53]
B
Yeah, we have a good. I think there's a couple communities that sort of compose on top of each other. For OpenRouter, there's this LLM power user community that we really identify ourselves with. That's kind of how the company got started. And these are. They test out all the models. They're often building things as side projects and sometimes they're running internal evals at large companies and they build a really good intuition for when a new model should be used. We also have a wide variety of bring your own model apps. These apps, some of them you can see on our rankings page as well, are apps where they want users to just be able to choose any model they want and have it work out of the box without the app having to change any code at all. And users love this because I think this is something that the world doesn't quite get yet. Models are like content. Every time there's a new model, it's like a holiday. There's a whole bunch of problems that might get solved. It's a new marketing moment for all the apps that suddenly can support the new model. It's like a new person being born. And when that happens, people develop these personal relationships with the model and then they understand and then they want to go to their apps and they want to bring that with them. The apps that support open Router like that can take advantage of all of those things for free. So those also. That also helps the stealth models really understand the wide variety of use cases.
- [41:46]
A
Because people have so many different things they want to use it with. Bring it to you guys get a diverse use case set it's not like eight of your friends using it all for the same thing. No, that makes sense. On the venture capital side of things, you guys announced about $40 million worth of capital raised across a seed in a series A earlier this year. I think Andreessen led the seed. Menlo led the A. So talk to me about the business model. As far as I can tell, you guys charge a 5 and a half percent platform fee for non free access. Is that the main way that open Router makes money?
- [42:19]
B
So it's, there's, there's three ways. There's this credit top up fee which, which is. It's basically for off the shelf inference. If you want to do volume with us, you can contact us through our enterprise page and get that fee down quite a bit. Okay. The other way we do it is by doing tons of volume on providers and just working with providers to do volume discounts. And then we often give those to users too.
- [42:58]
A
Just to interrogate that, if you're offering, let's say model X from model company Y at market rates, but you guys negotiate a 15% discount that provides really comfortable margin to open router itself. Is that what you're saying?
- [43:10]
B
I won't go into what the margin is, but, but that's like part of how we have a margin. That's right. And then the last component is that there's a whole bunch of inference adjacent software that people really want. Things like observability, web search, like improved team management, ability to limit certain parts of your team programmatically easy like multimodal support, PDF parsing and PDF management, file context management. All this stuff that like, it kind of like it kind of becomes really powerful if you have it managed very closely with your inference. Those things, you know, we, we have like enterprise features that we charge for and that's I think where like the, the margin will shift over time.
- [44:07]
A
Okay, that helps a lot. Now going back to AI models and such, you guys have hundreds of models on the service, you have dozens of inference providers and that has yielded a growth curve in terms of total tokens processed that looks like this. If you're on the audio version, I'm sharing the open Router rankings page. And as we can see here, Alex, pretty quick and steady growth across inference volumes. We're on track to set a new record this week. I think it's about 5.8 trillion tokens. 5.7. Sorry. How quickly does this chart keep going up over the next 12 months in your view?
- [44:40]
B
Well, I think it's absolutely insane how like underutilized LLM still are. There are, you know, I mean, on top of just running into a lot of repetitive tasks that even engineers continue to do today. Like the, I think the most exciting sort of glimpse into how many more tasks can be automated is the new types of training data that the model labs want. And they're really interested in figuring out how to be a product manager. What kinds of things do you need to know? How do you take a bunch of different contexts and distill it into action items or better ways of running agent workloads? Understanding like somebody's calendar very deeply. And these, these tasks, like after a little bit of time they start to show up in the models. And what our job is, what we want to do is make that work visible to everyone. You know, if a model does a really good job managing your calendar, that should be clear in the rankings using empirical data based on both how people are using it and also other signals that we kind of understand over time. So we're like, there aren't that many categories that model labs are known to optimize for today and it's just going to blow up. So I think the, I mean I think it's pretty amazing how much more work there, there continues to be just.
- [46:30]
A
That done there for folks who maybe are a little bit less head in this world as we are. You're saying that LLMs are going to get better at domain specific tasks by better training by their makers. And so essentially we're going to see LLMs perform better at different categories of knowledge, work, product management and so forth and not just drafting legal documents and writing computer code.
- [46:51]
B
Yeah, I mean I'm not training models myself and so I'm not the best person to ask about the exact ways they're going to improve. This is kind of my speculation and hope, but I think the like that's like a big signal that there's room to grow. The fact that nobody is tracking improvement on those things yet. You know, a key kind of muscle to build when you're entering the AI space is benchmarking. Everything is about emails and benchmarks and if you make claims without having an eval or a benchmark, what do you, what are you really saying? And the, there's a lot of like, you know, there's just a lot of value to unlock by creating a benchmark and it also guides the models in, in that direction when you do.
- [47:43]
A
So evals and benchmarks are very useful. There's other vectors you might use to choose which model, which provider at which time you guys built something called Autorouter that essentially does some of that work for your customers. I presume today it's an unbiased setup in which you guys are only selecting for the best option for your customer. But is there a way in the future in which people could like sponsor that to get more people to try out their models for certain use cases?
- [48:11]
B
It's a good question.
- [48:13]
E
We.
- [48:14]
B
So far we've stayed away from letting any providers sponsor more traffic. Our goal is to be like a strong, neutral, Switzerland style gateway that gives everybody a fair shot based on real world metrics and real world benchmarks. So. But I do think that there will be interesting ways to like shift people to new models in the future. We, for example, like, we do a little bit of model recommendation today, but very little. You'll start to see that kind of ramp up in 2016, you know, if.
- [48:59]
A
You'Re 20 or 26.
- [49:01]
B
2026.
- [49:04]
A
I know we're getting old, but like we have to at least get the decade right. You're good. Keep going.
- [49:12]
B
If you. Yeah, yeah. Like there are all kinds of workloads that are just like flooding models that are way too powerful and don't you know where you could save a ton of money?
- [49:22]
A
Yeah, you don't need to use GPT5 high if you can use, you know, 4.1 or whatever.
- [49:27]
B
Yeah, yeah, yeah, exactly. And having the. Having like a really wide range of models and a lot of data about how they get used, I think helps there. But there's also, I think, good ways of making very interesting model routers that we'll experiment with. It's kind of a good positioning for it.
- [49:51]
A
In your Series A announcement, Matt Murphy, one of your board members, said that he believes OpenRouter's value will grow not only because there will be more inference spending, but but also because the startup will collect data on how the models are behaving. It feels like that's aligned with what you're saying here, that you guys will be able to better recommend stuff for users and learn from them and kind of create a virtuous cycle in which people will always have the right model at the right place at the right time.
- [50:17]
B
Right. And today, like our focus so far our focus has mostly been about giving you high quality inference for the models you want to try and to use in production and to help you discover new models based on your preferences. And like model discoverability I think was like the key product strategy for us or goal, I should say, where we really wanted people to discover new models themselves and feel like they understood how to get there and then bring those choices really easily into their products.
- [50:55]
A
Do the models eventually end up so close in terms of performance that they become fungible or commodified? I was just looking at the artificial analysis kind of like leading thinking index and it's amazing how many models are within one or two points of each other. There are vectors where there's big differentiation. Grok code, fast one, ever since that came out on OpenRider is taking a huge market share because it's cheap and fast. But if I kind of just think a year or two ahead, everything's going to be, you know, 10, 15, 20% better, whatever and it all might taste like chicken. Is that where we're going or am I being a little bit too blase about individual labs ability to create something that's distinct from competition?
- [51:35]
B
Well, you know, similar to like, you know what I said earlier about making models shine based on these really like sometimes idiosyncratic skills that they spike on. We were, we're doing the same thing for providers. So a provider that you know is extremely, you know has extremely puts a ton of effort into improving their tool calling ability should be shown accordingly. And that's what we do with our exacto endpoints for example, focuses on tool.
- [52:08]
A
Use at the inference provider layer. Right.
- [52:11]
B
It focuses on, it's based on a lot of different inputs. We look at the model's accuracy when calling tools with the correct schema. We look at the proclivity to call tools. We look at our own users preferences too. We have sort of like doordash style preferences that come in when users are using our API. So you know, we, those are kind of like votes and at scale they become high signal we and then we, we all that data we kind of turn into a big dynamic benchmark. Most benchmarks are static and static benchmarks become worthless after a certain short amount of time. So we believe really heavily in dynamic benchmarks that are constantly updating and, and that like become products as a result. We productize them.
- [53:08]
A
All right, one last question before I let you go. Your, I think co founder Chris Clark, he's your coo, he said that he believes that inference costs will eclipse salaries as the dominant operating expense for most knowledge based companies in the next five or 10 years. I just want to understand this. Is he saying that as AI models writ large become more and more intelligent, we're going to spend more money having them do work? I inference spend then we're spending on, on human salaries because on One hand that's very exciting. On the other hand, I worry about everyone who's 20.
- [53:39]
B
Well, I, I don't know if he meant operating spend inclusive. So I think there's definitely going to be some. This is kind of an ongoing debate in the community how, how much human salaries are going to compare to inference spend. But I think it's kind of like a, more of a question of like what in the economy, like how much value in the economy should be done by AI models if they're improving like this over time, a little bit faster versus humans. And like, it's kind of tough to make any of these statements in the like, current day because you have to kind of like look at the curve and then just look at how the statement changes year to year and what kinds of, what, what sort of domains don't seem to follow the pattern. So I like, it's, it's super early now, but my assumption is that there are going to be some domains where AI models are, are, are just kind of not scaling super quickly. And so we'll have much more human salaries directed at those jobs. And those could be things like actually orchestrating models as an engineer, new jobs that didn't exist prior to AI, just focusing on coding, deep infrastructure work, which the models are, you know, probably weakest at today and might scale up a little bit less quickly. And, and then there will be other jobs that like most, you know, humans mostly do today. They're probably going to be mostly replaced by AI and where that statement will be correct.
- [55:35]
A
Well, again, a super stoked, super worried, but I'm old enough that I'm going to dodge the damage here. I just, I think about the future, but at the same time, people want to use AI because it's great. So who's going to stop them? Well, Alex, it's been an absolute treat when you hit your first 10 trillion token week. I'm going to be screaming about it from the top of my lungs here. Thank you for having the data so public so we can all use it. And it's openrouter AI. And before you go, is there a job that you're looking for, the right candidate for that you want to shout out to the audience so that way they can come find you?
- [56:04]
B
There are a Bunch go to OpenRouter AI Careers. And yeah, feel free to message me on Twitter too if you have a, if you have a, if you want, if you don't fall into one of those roles, but you think you should.
- [56:18]
A
All right, we appreciate it, man. We'll Talk to you again in six to nine months. In the meantime, Alex, thanks for coming on.
- [56:23]
B
Thanks, Alex. Thanks for having me.
- [56:27]
A
Jason, I found a really interesting question over on the startups subreddit. There's a founder that argued that it's hard to build a startup today because everyone is building things so quickly that no matter what you put into the market, people will essentially copy and paste using Vibe coding tools and essentially collapse your pricing ability. People have always done copycats rant famously copied Brex to some degree down the line. So to me, this just sounds a little bit like cope from our friend. I'm curious what you think about Quick Vibe Coded competitors and the potential for it being very hard to get traction when everyone can just roll out the Xerox machine.
- [57:01]
D
You know, as this person, person saying it's now almost impossible to build the moat. Every product mark, every product market fit turns into a red ocean in the blink of an eye. Before you even manage to build an MVP or get early traction, 10 other ventures are already doing the same thing. Burning money and making any road road map to profitability useless. So that is peak bubble behavior, right? Like when we get to a bubble, there's a lot of money in the system, a lot of entrepreneurs, a lot of people chasing the same thing. So you'll see a bunch of knockoff companies. That's pretty normal. What happens with those companies is though, they tend to give up, they tend to not be able to keep up in the race. So a lot of people start the race, right? You might get 10 people who are doing ride sharing, 20 people doing delivery services, and then eventually doordash and Uber Eats run away with deliveries for food and people forget about Postmates or, you know, Sprig and just, you know, a long tail of delivery services, local and all different modalities of it. So then it becomes a marathon. So who wins in a marathon? It's the person who has the most discipline, the most trained, well rested, you know, best diet. And in those cases, it would come down to who has the best team, who is constantly shipping features. And then there's network liquidity. So not everything is about building the product. The liquidity of the network is also important. So if you open up, you know, a ride sharing app and it takes 10 minutes to get a ride, or you open up Airbnb and there's, you know, a hundred choices in a city, you open up one fine stay. And they haven't gotten to Lake Tahoe yet. Okay, you're going to use Airbnb. So the Liquidity of the marketplace becomes defensible. A sales team becomes defensible. If you look at Salesforce, they have many competitors, but. Or Oracle, many competitors, many options, but they had incredible sales teams. So there are many other pieces to the puzzle that can become defensible. In the early stages of a startup. It's really going to be about using capital as a weapon, being able to raise more money than the other competitors and then do marketing and have more features or more distribution. So congratulations. Yes, everybody has the idea. Some people can build it and then a smaller number of people can scale it. And then an even smaller number of people can maintain it, you know, and sustain it. So you each of each of those levels, you lose competitors. Everybody has the idea very few people can finish the product of the people who finish the product. Very few people can keep the cadence up with distribution, sales, right? And then on top of that, very few people can raise money at scale. So all of a sudden you get down to a two or three horse race in every one of these businesses.
- [59:59]
A
So moats then are almost like concentric circles of not only product, but also internal capability, team capacity, and market.
- [60:09]
D
All of these factors become important later on. So distribution's an important one. If you look at a product or service like Perplexity, remember they were doing all these weird deals in the beginning. Like if you, if you buy this, you get a free Perplexity account. There was like five. You get an American Express card, you get a Perplexity account, you get Uber, you get a Perplexity account.
- [60:29]
A
They really were just, you know, throwing. Yes.
- [60:32]
D
And then affiliate programs like just annoying people with these things. They, somebody on that team at Perplexity was very good at doing annoying partnerships to just get the first crank started right. So now the next person who built something that was 20 better than perplexity is sitting there and like, I didn't hire some affiliate marketing partnership program person. I didn't hire a social media person. And this is where startups get really hard in competitive spaces. You have to have, you know, distribution and go to market strategies that beat the other people. It's not just enough to have Betamax versus VHS is an old reference to VHS videotapes. When you watch movies with them. The better technology doesn't always win. Sometimes it's the one that's better marketed and better sold. And marketing and sales, all that is another art in business. You have to learn after product, market fit or, you know, you can. That's the good news is you can buy that one. You can find people who've done it before. So you got to just get help. But it is worse now, so you got to deal with it. It is worse in a hot market. In a quiet market, it's hard to raise money, and it's easier to get attention for your product. In a hot market, it's easier to raise money. But there's so many competitors, it's hard to get attention for your product. So in both cases, it's still hard to get attention for product. In one, it's because you have competitors. It's a competitive market. And in the beginning, it's because nobody cares about your product because it's a down market. Right. Like, people are just not evaluating new products or technologies because their businesses are making cuts.
- [62:07]
A
So entrepreneurship is always hard is what I'm hearing.
- [62:11]
D
Yeah. None of it's easy. I mean, if it was easy, everybody would be doing it. And it's extremely hard. And it is a. It is a death march in some ways, you know, where people. There's just like, Stephen King movie about some. Like, everybody marches, and it's like some.
- [62:30]
A
Sort of weird long walk.
- [62:32]
D
Okay, There you go. So I. I was fascinated by the premise and disturbed. Gotta take this long walk. And. Yeah, the last person standing gets the prize. Everybody else dies. Spoiler alert. It's in the trailer.
- [62:45]
A
I wonder if that was inspired by the Chinese Communist Party's long march. It's an interesting historical question.
- [62:50]
D
It's a. It. This was a Stephen King book, right?
- [62:53]
A
I wonder if he was inspired by that. That bit of CCP mythos.
- [62:57]
D
Wow. Stephen King wrote it when he was but 17 years old. All right, everybody, that's another great question.