← Sourcery
Loading summary
Transcript76 lines
- [00:00]
Gabe
Models are getting more expensive and they're getting better. We're just seeing this huge explosion of usage cost, tokens, everything.
- [00:06]
Interviewer
We need to talk about tokens. Winston mentioned you guys got up to 13 trillion.
- [00:10]
Gabe
We are consuming a ton of tokens for just serving all this stuff. The big misconception right now is I don't think people realize how expensive this is going to get. And they're going to be like, what did my agent do that cost me $10 billion?
- [00:23]
Nico
We launched lab, which is our legal agent benchmark. It's a benchmark for measuring the performance of agents on real world legal tasks. We post our first sort of initial results of how closed source frontier models like those from OpenAI and Thropic Deepine perform on the benchmark.
- [00:50]
Interviewer
Gabe, welcome to Sorcery.
- [00:52]
Gabe
Thanks so much for having me.
- [00:53]
Interviewer
Thanks for having us here. We're in the secret Speakeasy back room.
- [00:57]
Gabe
Exactly. Yeah, our secret.
- [00:59]
Interviewer
So today we're gonna walk through the latest release of your benchmarks, which is such a fun topic. Winston was so excited to talk about it too. So I guess to start, let's break it down. What are you measuring with your benchmarks and where did we get to today?
- [01:14]
Gabe
Yeah, so the thing we open sourced was our legal agent benchmark. And I think the thing that we're super excited about is we have done a bunch of benchmarking in the past. So before Legal Agent Bench, we had big law benchmark, which was more of a chat based benchmark, which was kind of a QA data set. And we shared some of that. We worked with kind of the labs and the providers, but at the time we were a smaller company and we didn't have the bandwidth to fully open source it and support it. And so what we wanted to do with Legal Agent Bench was fully open source this. So anyone working on agents could run their agents and kind of work with us in the community to evaluate these things. And the way I would think of this data set is we tried to really mimic what the coding agent benchmarks did. And the way to think of the coding agent benchmarks is with things like SUI Bench, Terminal Bench, you have a GitHub repo, which is kind of all the code for a project. You have an issue, which is someone said, hey, this doesn't work. Can you fix it? And then you have a bunch of unit tests that will tell you. If you write a bunch of code and run it, the unit test will tell you you broke this thing or this thing passed. And the way the agent gets scored is did it write all this code that passed all the unit tests? And I think there's a bit of a misconception that, oh, legal, subjective, so you can't do this. And I think the thing for especially big law is a lot of the work you can actually quantify. And this is what senior associates and partners are doing when their junior associates do a bunch of this work. And so as maybe an example of a task in the legal agent bench, it would be, can you do diligence, which is you get a data room. So you're trying to buy a company and you'll get all of that company's contracts. And what the law firm needs to do is go through all the contracts and make sure that there's nothing questionable or that might blow up the deal in there. And so the way we set up this data set, like one of these tasks is you have a data room that's all the contracts. You have a partner request, which is a very high level set of instructions you would get from a partner if you're an associate. And usually for something like diligence, it would be, hey, we got the data room, can you look at it? And you would need to infer, I know what I need to do based on that. And then I think part of the like novelty here is we use both of those to create essentially legal unit tests. And so what are all the things that a partner would be looking for when they're trying to assess an associate's work product? And so you would take that data room and you would generate a diligence memo, which is what the law firm gives to their client to basically say, here's all our findings on this diligence and the things you would look for. So one section is going to analyze all the change of control provisions, which is basically, if I'm buying a company, there's going to be a change in control of the company. And many contracts have things that can be triggered and most of them don't matter, but there can be ones where this important vendor contract will change in a material way and you need to flag this. And so that would be like a unit test of like, you want to check that. And then the challenge with scaling this benchmark is that's just for diligence. And then as you scale this to every practice area that these large law firms help their clients in, you have to think about how to do this for fund formation and IP litigation and capital markets. And so a lot of what we're building is essentially this taxonomy of here's all the tasks that a law firm would do, and here's all the subtasks that associates would do and how partners or senior associates would grade that work. And I think what's nice is that framing is kind of the same thing. You need to evaluate these agents to do these tasks.
- [05:11]
Interviewer
Benchmarking makes things a lot more competitive and people love that. I mean, AI is like the most competitive world ever right now. And so I'm really curious on that standpoint, like, how are you then, like, where does this lead to next?
- [05:27]
Gabe
Yeah, so I think the interesting thing, and kind of to your point is, okay, we released this benchmark and now we're going to have all of the labs and everyone else building models competing on it. And I think there's obviously some risk of, oh, what if some lab providers models are the best? Then like, why do they need Harvey to do this? Is maybe one of the potential risks. But what we've seen is actually every different model is good at something different. And so with the initial results we saw, anthropics models are quite strong. But there's areas where 5.5 is better, there's some areas where open source is better. And increasingly it's not just which model is the best, it's which model can solve the task at the lowest price point. Because as these models get better, there's kind of intelligence saturation where it's like, okay, for some simple tasks, I actually want to know that this open source model is good enough. And so a lot of the value that we're trying to provide to law firms is you can't just use one model to solve all these tasks because it's getting too expensive. So how do you think about all the tasks that your law firm does and which model you should use from which provider to solve those tasks?
- [06:38]
Interviewer
I know you've answered this before, but your research partners are your biggest competitors. OpenAI, Anthropic, Nvidia, we have base 10. There's like a handful of across categories. Right. So why open source? It doesn't that pose a risk to you?
- [06:56]
Gabe
Yeah, I think we think of all of these providers as there's obviously overlap with the labs and Anthropic and OpenAI have done kind of like Claude for Legal, Codex for legal. But I think the problem that we're solving is different. I think a lot of the products that they're building are individual productivity and the products we're trying to build are organizational productivity for law firms or large in house departments. And so when we think about Open sourcing. I think there's kind of the initial point that I made of like most of these law firms and enterprises can't rely on a single lab provider. So like, there's a big risk if you're a law firm that if you just use anthropic or you just use OpenAI, you run into conflict risk. So imagine you're using only anthropics models as a law firm and you want to represent OpenAI. OpenAI is not going to let you send their sensitive legal data to anthropics models. And so as a law firm, you need to support all the different models if you want to represent OpenAI and Google and Microsoft and all these providers. So there's the conflict risk, there's like platform risk. Like what happens if you pick one provider and they run out of compute? Their models fall behind. And so you need to kind of abstract all these things away. And so we think of the providers similar to cloud. And I think you see the same thing in cloud with like Snowflake and Databricks and Datadog, like all these companies built on top of the cloud, but also compete with the cloud. So I think it'll look like that. And then I think the bigger point is we don't see the general legal intelligence as where the value of kind of Harvey comes from, because most of the valuable legal data can't go into the general purpose models. And so if you think of these large law firms, the work that they're doing is like a very sensitive internal investigation, like mergers and acquisitions that would move the market and they can't go into public models. And so we think of our strategy as we want to open source the general stuff and work with all of the providers to make these models as good as possible at general legal. And and then we want to build infrastructure for these law firms and enterprises that help them own their own models and build their own systems on their unique data. And that's where we think kind of our unique value comes in.
- [09:21]
Interviewer
And how are you training the models? You can't pull or pool data from your clients, so how are you learning off of them?
- [09:28]
Gabe
Yeah, so the way I would think of the big challenge and where law firms are so different than most enterprises. Like at Harvey, all of the data in our company for the most part is ours. At a law firm, a lot of the data is all of their clients, and you can't mix this. And so a lot of what we're starting to work on with our clients is we have a product called Shared Spaces which essentially lets a law firm and their client collaborate on a legal project. And so you could imagine a world where you have a large private equity firm and their law firm and they're doing all of their fund formation and all of their kind of M and A together in these shared workspaces where you can start training these models is how do you use that relationship to build these unique models? And so a lot of what we want to figure out is given all of the like, ethical wall constraints, regulatory constraints, client data constraints, how can we help law firms use their data in a way that respects kind of all of their client obligations, but still be able to improve their systems based on that?
- [10:33]
Interviewer
So speaking on your background, you were at Google Brain, DeepMind and Meta. What were the biggest fundamental lessons that you learned from each of them that are driving the decisions you're making?
- [10:43]
Gabe
Yeah, I think especially Google Brain and DeepMind. When I was there, this was kind of in 2016, 2017. And so right when deep learning was starting to take off. And I think it was, it was a really cool experience seeing both of those labs because they had kind of opposite strategies of how they approached research, which I think both were correct, but led to kind of different ways of executing on kind of the vision of where they thought AI was going. And so when I was at Brain, it was very bottoms up. It was, I mean, both had some of the smartest AI researchers in the world. But Brain, the approach was, you know, let's get all these smart people, give them a bunch of compute, and then kind of let them do their own projects. And I think part of the outcome there is you had someone like Noam Shazir and Ashish Viswani and kind of the rest of the folks that invented Transformers. And so that approach worked. And then DeepMind was much more top down, where Demis just had this vision of, okay, we're going to create AGI, here's all the things that I think are required. And so they kind of had this tech tree of how they were going to do it. And then let's have all these teams working on research projects that would solve all these milestones to get us to AGI. And I think our approach is a bit more Inspired by the DeepMind one, just because I think because we're in a vertical, the end goal is very clear. Like we kind of know, here's all the legal work that needs to get done. And so you can think about that North Star as you know, here's all the practice areas that these law firms do. Here's how we would build systems that help them do large parts of those with AI. And so it feels much more defined and it's, it's more of an applied problem than it is kind of a pure research problem. But I think seeing that approach really inspired, like how I think about these things.
- [12:35]
Interviewer
How has your role technically evolved over time? I remember I was like, I was just listening to this interview or I forget what it was, but it was, they were talking about Andre Karpathy and how like even back at Tesla, like you, you, you'd think like these roles back then were like super glamorous and stuff. But he was data labeling all day. Yeah, but you know, now he's the almighty God of AI. So like, well, what is the biggest difference over time?
- [13:02]
Gabe
Yeah, I think it's definitely been interesting at Harvey where it's like when we started the company, I was more focused on research than we should have been at that stage of the company where I kind of had a strong sense that what is happening now was going to happen. And so I wanted to figure out how do we train models, how do we scale this? But it became very clear pretty quickly, like, okay, first we need to build an enterprise business. Because we were a bit ahead of the curve of when we first started the company. I was pitching people agents and at that four years ago, people were like, what are you talking about? And so we quickly had to kind of think about what is the right product now for customers that we can make for them that's useful, get revenue and scale up the GTM motion, kind of a traditional enterprise SaaS motion. And Winston and I always had kind of this vision of we're going to need to build two companies in parallel. Like one company is this traditional enterprise seat based business and then the second will be kind of the transition you're starting to see now as these models get better, things move to consumption. And I think you'll need both of these because it takes a while to educate buyers on how to buy this. Like, buying off tokens is insanely complicated. It'll probably at some point go back to something like subscription because it's hard to like forecast these things. But I think my role kind of shifted from when I started the company where I was trying to do research and I was like, okay, we need to do traditional, like build a great product, build a great enterprise business. And then now it's gone back to research where we've put all those things in place. I think a lot of the infrastructure has caught up the agent stuff is happening now that now I'm full time and a lot of it is data labeling and how do we create good data sets, how do we work with all these partners? But I think it's kind of back to doing the things that I really enjoyed when I was@brain DeepMind and meta. And so I think now is kind of the time where it seems like all these application layer companies, the you saw Composer two with cursor, like everyone's starting to make this stuff work and I think that's like super exciting.
- [15:15]
Interviewer
The inference layer has gotten really hot right now. What is your general sentiment about what's going on? How is that going to evolve?
- [15:22]
Gabe
Yeah, I mean I think it's tied to kind of what I was just saying is like as this is starting to work in terms of open source models are catching up, I think a lot of the performance from the models is coming not just from doing this large pre training but all of this test time compute of how do you make these models better at reasoning, how do you build good kind of RL environments for them and good harnesses. And so the more that works, I think the more that kind of shifts the ability to application layer companies to do interesting things. And then for example for us we work with base 10 and fireworks and together AI applied compute trajectory, kind of a bunch of these companies, ngram that are doing either inference RL as a service and I think it just feels like the research is starting again where there's just so many different avenues. All of these Neolabs that we're working with are trying different ways. We're also working with the traditional model providers because they're also making a bunch of progress. And so I think it's just interesting as we're able to build more of these data sets and even we have a bunch of customers that want us to figure out better ways to train on their data. I think kind of all of these things are starting to take off and then I think the inference providers, you're seeing them do very well where there's customers like us, where we do need to move some of our traffic to open source models because it's just becoming super expensive to serve the largest closed frontier model. And so I think it's like a mix of performance. We have a lot of customers that want to own their own models, own their own data and like I think that's kind of also the inferential providers will serve that. And so yeah, I think they're, they're
- [17:07]
Interviewer
in a good spot Sorcery is brought to you by Brex, the financial stack trusted by more than 30,000 companies, including one in three venture backed startups in the U.S. nearly 40% of startups fail because they run out of cash. Rex is literally built to help founders avoid that. Unlike traditional banks that let your money sit idle chipping away at it with fees, Brexit's designed to help you spend smarter and move faster. Their all in one solution combines checking treasury and FDIC protection into one powerful account. You can send and receive money globally at lightning speeds, get 20 times the standard FDIC coverage through their partner banks, and even high yield from day one with same day and even same hour liquidity. Access your funds anytime. Companies like Scale AI, DoorDash, Service Titan, HIMSS, Anthropic, Flexport, Robinhood and Plaid Trust and use Brex. Start today@brex.com sorcery that's B R E X.com sorcery. Turing is training the next generation of AI with tasks that require real expertise and real world judgment. That's why companies like Nvidia and Anthropic, Salesforce and Gemini partner with Turing. Turing builds realistic reinforcement learning environments and data systems based on real operational traces. The kind of infrastructure Frontier Labs need to train superintelligence. Visit turing.coms o u R C E R Y I guess to dig even deeper, architecture is changing really fast. I mean you might have missed the whole SaaS wave.
- [18:45]
Nico
Yeah.
- [18:46]
Interviewer
But starting off in AI, it was all chat based. Now we've completely moved to the Agentic era, which is like a thousand times more cost competitive. You need compute, power, memory, all these sorts of things that are reorganizing how companies are being built and of course the stock market too. So how are you thinking through the agentic layer and building that out?
- [19:10]
Gabe
Yeah, so I think this was like a huge transition we've gone through in like the past six months. Where exactly? Like you said, the original product we built was kind of this chat based copilot for lawyers. And then we built all this product around it that let it work at a law firm on client matters and all these things. And then maybe six months ago when these coding models started getting really good and you could do kind of things like Claude Code and Codex and that started working really well in the command line. We started building a bunch of the infrastructure of how do you get these agents that were running in your command line and could execute all these tools and work really well and how to move that into the cloud? And I think now you're seeing this with these managed agent solutions. And to your point, it's like, now you need sandboxes. The models use way more tokens. We have queries that we have simple assistant type queries where you say, draft me a document that a single query can cost $20. We have like a review product where you can upload 100,000 contracts and ask the models to review them. And some of those can cost $20,000. And so it is just getting incredibly expensive. And I think in parallel, what we're seeing is, I think even two years ago the models were so good with coding that most programmers would just use them and be like, this is super useful. I think we're just starting to see in the past six months that inflection for legal, where the models can now generate entire documents and they're starting to work in a way where most lawyers who aren't using this technology just for fun, they're like, this needs to be so much better than the way that I'm used to doing things for me to change my routine to do it. That like, we're starting to see that absorption. And so a combination of like, the models are getting more expensive and they're getting better, they're using more tokens because they're running for longer and they work so much better that users are using them more. We're just seeing kind of this huge explosion of like usage, cost, tokens, everything.
- [21:16]
Interviewer
We need to talk about tokens. I mean, there's like so many different ways to take this question because there's obviously the margin sentiment on it and then there's also the internal usage and efficiency on tokens. And I think Winston mentioned you guys got up to 13 trillion. So how are you thinking through that and the cost optimization? And then we can also talk about efficiency internally. How do you.
- [21:43]
Gabe
Yeah, so I think this is kind of that inflection point I was talking about where up until recently we've always been capability constrained. And so we always wanted to use the largest model because for most of our tasks and if you look on the legal agent benchmark, even the best models weren't good enough to complete those tasks. And then the usage, we weren't at a scale that the usage, it wasn't something like cursor, where it was so expensive to serve these things and we didn't have the latency requirements. But that has changed in the past six months where now we are consuming a huge number of tokens. I think for some of the labs we are like the largest consumer of embeddings we are consuming a ton of tokens for just serving all this stuff. And so I think it's a good problem to have in the sense of like the usage is so high. We obviously need to think through, you know, how do we serve these customers because if we kind of look at our usage like plots right now, it's like this is just the start. And it's like there's still so much room where it's like if everyone starts using this long horizon agent stuff the way some of our power users are using it, there is just going to be a lot of cost optimization you need to do. And so I think there's a couple things we're thinking through. I think the most obvious is just doing better and better routing of models. So as the largest models get better, not every task needs to be sent to the largest models. And so that's a great use case of the legal agent benchmark. Increasingly like serving more traffic from open source models. Post training models. Like a lot of these like very large frontier models are large because they're good at everything. And so I think a lot of the opportunity for these like specific verticals will be okay. I probably don't need a trillion parameters if I just need the system to be good at diligence. And so how do you build products and models in a way that you can get that frontier performance without kind of all of the frontier costs?
- [23:49]
Interviewer
What do you think the biggest question is? People are not asking.
- [23:52]
Gabe
I feel like that probably the big like misconception right now is I don't think people realize how expensive this is going to get. And I don't think people realize how difficult it is going to be for customers to deal with that. Like I think when I talk with most people they think, oh, just move to consumption pricing and that will solve all the problems. But I think there's going to be this very interesting dynamic where there's kind of a couple ways to price these things. And I think most VCs what they want to see is like can you price the work right? Like can you sell the value of the work you're selling? And then the like that's one end of the extreme and then the other end extreme is just like price by tokens. And I think the problem you run into of pricing the work is, is actually the same problem that law firms run into when they try to do fixed fee pricing, right? They're trying to say, hey, here's the fixed rate of this cost. And I think there's going to be this great irony where when we started the company, one of the questions we got asked the most was, you know, what's going to happen with the billable hour? And I think most people's assumption is just, oh, everything's going to move to fixed fee. So then these law firms can protect their margins. And I think something people don't appreciate about the billable hour and why it's such a good mechanism is it lets you price incredibly complex work at massive scale in a way that the entire industry can agree on. Right? Because all these law firms we've talked to about pricing changes, like whenever we talk to them about fixed fee, they're just like, we just have 10,000 clients, we can't negotiate every engagement and price this. And everyone's different. And I actually think something similar is going to happen with this token based pricing where you've seen a bunch of headlines of like the Uber CTO where he's like, we just ran through all
- [25:43]
Nico
of our
- [25:45]
Gabe
coding tokens for the year, right? In three months. And so all these customers are going to start getting these consumption bills of like $10 million. And they're going to be like, what did my agent do that cost me $10 billion. And if you think about how these law firms solved it, they're like, they got a bill for $10 million from their law firm and they're like, tell me what every associate did for every 6 minute increment on the entire project and they wrote that. And I think you're going to start seeing things like this for token billing where there's going to become this whole ecosystem of like, how do you optimize around this? Because you kind of have these weird misaligned incentives from the model providers, right? Because they're selling you consumption. Like they're somewhat incentivized to have their agents use as many tokens as possible. And then how do you solve that? It's like, well, we can help you benchmark that and route all these and say, oh, actually for this task you don't need to use all these tokens. And so I think there's going to be like, I think this is going to be much more complicated than people realize. And it's the same as like, it's why it's so hard to price legal work and all of this work where it's just so hard to quantify. Like if I think of our token usage for our programming team, it's like, what did they use all these tokens on? I'm like, they're Definitely more productive. But it is very hard to, like, quantify these things. And so I think that will be like a big challenge.
- [27:04]
Nico
Wow.
- [27:06]
Interviewer
I've not heard someone kind of break it down that it'll. I mean, obviously you guys are biased, but that is a very rational way to think about monetization for them.
- [27:16]
Gabe
Yeah, and I think the same way. So I think it's more complicated than that in the sense of, like, when most people look at law firm billable hours, they're like, oh, the incentives here are completely misaligned. This must lead to bad pricing. But I think what people don't think about is like, okay, the billable hour creates some pricing misalignment, but the fact that these law firms need to compete with every other law firm means that that's kept in check. And so it's like this actually converges to roughly the right pricing. And so I think the same thing will happen with the model providers where it's like, if you look at Opus 4.7 and 5.5, Opus 4.7 is three times more expensive than 5.5, but it's 10 or 20% more performant. But these are going to cause pressure on each other and then if open source catches up, you're also going to have pricing pressure. And so even though there is like, if you just had one model provider and they owned all the models, then they would just be like, here's how much tokens cost. My agent's going to use a ton of tokens and you're kind of stuck. But I think this is exactly why you're seeing the inference providers and all the other model providers be successful, because you just are going to need a lot of options the same way. That's how you deal with kind of like pricing with professional service providers. But yeah, it is a weird, like, there's a bunch of different levels of, like, the pricing there.
- [28:38]
Interviewer
So how are you personally keeping track of everything that's going on? Like, what is your research diet? Has it changed over time?
- [28:46]
Gabe
Yeah, I think it's changed a lot. I mean, I think when I first started doing research, the thing that was so exciting about deep learning is everything was open source and everyone published everything. And so I could just go on arXiv for six hours a day and I would just sit there and read papers all day. And you just, you could read every breakthrough real time. And that was the thing that initially, like, attracted me to the field because I was like, this is so interesting and it's so easy to talk to everyone about it because everyone's so open about it. I think that has somewhat changed now, or not somewhat. That has completely changed in the sense like no one really publishes what they're doing anymore and so it is harder to keep track of the research breakthroughs and what's going on. I think the way that I do it now is we work closely with the labs. A bunch of the people I worked with@brain, DeepMind and Meta are at these labs and so keeping up with everyone that way. And then I think the thing that I feel like I'm most excited about, for example, with this benchmark, is we are going to start publishing some of our research that we're doing with the labs or the other providers and open sourcing more models, more of the work we're doing. And so I think there's hopefully as open source catches up, there's going to be more and more of this. And yeah, I think that's kind of
- [30:06]
Interviewer
like the main way Today's episode is sponsored by VCX by Fundrise, the public ticker for private tech, allowing investors of all sizes to invest in venture capital. Learn more@getvcx.com Some of you may not have heard this yet, but our sponsor Public just launched something called Generated Assets and it brings AI into investing in a way I've honestly never seen before. Here's how it works. You type in an idea like AI powered supply chain companies with positive free cash flow, or defense tech companies growing revenue over 25% year over year. Publix AI then dispatches a swarm of agents that scan every single US stock, evaluates them, and instantly builds a custom index around your thesis. What really stands out is how clearly it explains why each stock is included. And before you invest, you can even back test your idea against the S&P 500 so you're making decisions with real, real context, not just guessing. And beyond Generated Assets, Public lets you invest in stocks, bonds, options, crypto all in one place. They'll even give you an uncapped 1% match when you transfer your investments over from another platform. If you want to build a portfolio that actually reflects your thesis, visit public.com Sorcery paid for by Public Investing Full disclosures in the Description Enterprise AI runs on Merge, the AI Infra platform for integrations, AI agent tooling and model orchestration. So your teams ship product, not plumbing. Mistral, Dropbox and Drata already trust Merge in production start building at merge.dev. founders scale faster on deal Set up payroll for any country in minutes. Hire anyone anywhere. Get Visas handled fast and get back to building. Visit deal.comsorcery that's D E-L.com sorcery well, speaking of performance, I have to do a little ad slot. So Brex, who I think you're a customer of, they're all about performance, spending smarter, moving faster. And so with that question I usually ask, maybe this is a layer deeper. Who are the people that you most look up to, that you've learned from, and what are the biggest lessons that you've taken from them?
- [32:14]
Gabe
Yeah, that's a good question. I think Winston's definitely one of them. I think he is in terms of his ability to like scale the company, keep track of everything. I think that's something that just watching him grow with the company has been kind of super inspiring. And I think there's like a lot of things that I learned from him in terms of like when I did research. I think what I was good at is how do I focus on just one thing. And I think scaling a company is being able to keep that one thing in your head, but then solve thousands of these other problems in parallel. And I think as a researcher, like, I was not good at that. I was like, good at like, okay, I'm going to solve like, you know, I'm going to try to figure out AGI and I'm going to ignore everything else. And like, you just can't do that when you're like scaling a company and watching Winston's ability to like do that and also kind of build a team and all these things while keeping that in mind. I think I've like learned a lot from him doing that. Let's see who else. I think my old roommate when I was at Brain, Barrett's off, is someone that I've learned a lot from. He was kind of the best AI researcher that I worked with when I was at Brain and he's now at OpenAI, ran their post training team. And I think kind of similar ability of just being able to keep all of the context in his head, always kind of know the right research directions and things like that. I think that was another person I was super impressed by. And then I think the obvious ones of. I think Jensen is super impressive. Like Demis, I think a lot of these people kind of leading the satya, leading the top labs, cloud providers, all the large players in the AI space. I think there's a ton you can learn from all of these. And then I think kind of the other top application layer companies, the Cursor folks and Brett I think all of these folks are kind of doing super
- [34:13]
Interviewer
impressive stuff given learning from all of them. What are the key traits that you look for in new hires?
- [34:19]
Gabe
Yeah, key traits in new hires. I think the biggest is like, are you just obsessed with the topic? Like the best hires that we've made, it's usually really easy to talk to them about what we're doing because they know so much already. And so I remember like with Nico, Daniel, Spencer, like a lot of our like early hires when I would just pitch them, here's what we're doing. It just, they immediately were like, that totally makes sense. Here's a bunch of spin off idea and we could just like talk forever about this. And so I think that's something that I always look for of like usually if I can have these conversations with someone where we're just building off each other, that's always been a strong indicator. I think it's somewhat unique. But I think Winston and I's hit rate on executive hires is insanely high. And it's usually if we can both have a conversation like that with someone and their background, everything's a good fit. We're like, okay, this person is, is going to work out. And I think that's been like pretty accurate. And you can usually just feel someone who's like insanely passionate or obsessed with a topic versus someone who doesn't know what's going on. And so I think that's kind of one of the biggest indicators.
- [35:42]
Interviewer
What are you most excited for this year?
- [35:44]
Gabe
Training some models.
- [35:46]
Interviewer
Great answer. Nika, welcome.
- [35:49]
Nico
Thanks for having me.
- [35:50]
Interviewer
Congrats on the recent breakthroughs.
- [35:53]
Nico
Thank you. Yeah, we're very excited about it.
- [35:55]
Interviewer
Can you talk through them?
- [35:57]
Nico
Yeah, absolutely. So about two weeks ago, we launched Lab, which is our legal agent benchmark. It's a benchmark for measuring the performance of agents on real world legal tasks. So we designed the benchmark to basically mirror how legal work is done at law firms. And then today we posted our first sort of initial results of how closed source frontier models like those from OpenAI, anthropic DeepMind perform on the benchmark. So we're excited about what the benchmark offers, not only in terms of making model and agent performance accessible to lawyers, but also a number of different research directions that it'll spawn from here.
- [36:39]
Interviewer
So something that Winston and Gabe had emphasized a lot was there was no existing data set for this type of information. So how did you guys create that and how did you train off of it?
- [36:52]
Nico
Yeah, so this was actually, I think one of the most interesting and innovative parts of the project. So obviously legal data is some of the most sensitive in the world. Firms can't just freely share that for something like an open source benchmark.
- [37:06]
Interviewer
Crazy.
- [37:07]
Nico
Yeah. So we had to get creative with how we constructed the data set. And the way that we did it was actually agent led with lawyer review. So coding agents, these Agentix systems are actually so good at generating synthetic data now that even for non public documents, like certain contract types, et cetera, they can generate a pretty good first draft. Obviously not good enough to be lawyer passing. So the way that we approached this was we have a team internally at Harvey that we refer to as Applied Legal Research. They're all former big law attorneys. They come from a variety of different practice areas and subdomains of law. We basically mapped out 24 practice areas that law firms have on their websites that they have lawyers, associates kind of assigned to within the firm. And they went through all of the sort of tasks that the firms do or the associates that the firms do within those practice areas. And they have the priors, right? They're lawyers, they've done this work before to write out. Okay, for this type of work we use these types of documents and they look like this. And this is what a realistic deal kind of looks like. We had agents generate the data from there and then we sent it to sort of like our larger network of lawyers to review the outputs, the rubrics, document quality, all those sorts of things. But it ends up being a pretty human in the loop but scalable way to generate data.
- [38:32]
Interviewer
So for someone reading this, how should they be evaluating the benchmarks?
- [38:37]
Nico
So I think there's two things here. One is just making agent performance legible, right? So there's kind of overall performance we have and we'll continue to maintain a leaderboard of just here's how this model, this agent performs on the benchmark. Obviously higher score is better, but I don't think that these like aggregate average performance measures tell the whole story. Story for two reasons. One, people aren't really thinking about performance in terms of just like quality maxing anymore. Like I need to quality maxing.
- [39:07]
Interviewer
Oh my God.
- [39:08]
Nico
I mean, that's been the story of like the last two years, right? It's just like I'm going to achieve the highest quality possible at whatever cost, whatever latency, whatever burden on my users. But people are thinking about it a lot more now in terms of what is the quality I get for some amount of money spent or some amount of time spent. And so we're able to sort of dissect model performance in these sorts of ways. Right. So you can see what the trade offs are in quality. For a model that costs, you know, one third of the price of our leader, which is Opus 4.7 right now, or like Gemini 3.5 flash, it's like seven times faster at completing work than some of the frontier models. So you can look at it that way to make like real kind of production agent decisions.
- [39:52]
Interviewer
And where are you getting outside of this? Where are you getting all of your research information? Diet?
- [39:57]
Nico
Yeah, so I think there's sort of three, there's three kind of avenues here. One is I did come from research, so I still have kind of a fondness for academic research. Like we obviously have partnerships with the labs. They're not publishing as much, but a lot of my old collaborators are still publishing archive papers and Neurips, iclr. All these academic conferences, even new conferences are spinning up around agents. Twitter is just like the real time, really. Twitter feed it directly? Yeah, I think you have to have a great filter. You have to, you have to understand like the signal to noise ratio of Twitter. But if you're following folks like Karpathy, Noam Brown at OpenAI, Sholto at Anthropic, some of these kind of leading voices within the labs, as well as some of these academics who have a presence on Twitter as well, it can be helpful.
- [40:51]
Interviewer
You talked about the biggest takeaways from the benchmarks, but what are you most excited about?
- [40:57]
Nico
Yeah, so what I'm most excited about is so we get this cost, we get this latency, we get this practice area breakdown. Right. That basically tells us where do we need to invest to improve performance of our product for our customers. I also think that the benchmark is a great kind of way to ground research directions that we want to invest in. Right. And so we've seen an insane amount of activity from the open source community, both AI researchers and legal tech, which believe it or not, has a bustling sort of open source community to investigate a number of areas. Right. So like post training open weight models, I think we're starting to see early results that show that open weight models when post trained, kind of close the gap with the closed source frontier. The agent harness is like the buzziest topic.
- [41:45]
Interviewer
Yeah, tell me more about that.
- [41:46]
Nico
So the harness is essentially the term, the term of art that Twitter has adopted for the infrastructure and scaffolding that goes around the model. Right. So you have some model, it's making decisions, you give it some number of tools like I Can read a document, I can write a document, I can do a web search, I can write code. And the harness is basically the way that you define all of these tools, skills, how agents can delegate to other agents to complete tasks. It's basically the infrastructure and scaffolding around the models. It's a really interesting area of research for us though, because the thing that we're seeing over and over again is that specialization matters and domain expertise matters. Right. And so we can have our lawyers, the open source community, our AI researchers, all working on legal specific skills and tools that make the agents better at these tasks that lab and our customers are sort of experiencing.
- [42:43]
Interviewer
So you're even more bullish on Harvey now?
- [42:45]
Nico
Yeah, I've been bullish since. Since day one. I think it's only increased.
- [42:49]
Interviewer
I know. I think I saw you have a bobblehead behind you. I know they have the funko pop, but yeah, seems like you've been here for a bit.
- [42:56]
Nico
That is one of the most unique three year anniversary gifts I think I've heard of from a company. But yeah, three years in about a month.
- [43:04]
Interviewer
Exciting. Okay, so as we close out, how many times are you going to beat this benchmark?
- [43:09]
Nico
I do think the benchmark is now a target.
- [43:13]
Interviewer
You created your own competition internally. Is this good?
- [43:16]
Nico
We did. My perspective on the entire kind of like AI benchmarking and hill climbing exercise is the extent to which everybody is investing in making models and agents more capable for legal. That is purely to our benefit because actually the innovative work that we want to do is not just base model capabilities. And so you ask, what is the thing I'm most excited about? I think this year? One, I do think the benchmark gets saturated within a year at least. But two, I think this year is the year that we see intelligence at an individual level kind of brought into intelligence at an organizational level. And so for Harvey, what that means is moving one layer of abstraction up in decision making. Like how do lawyers collaborate with lawyers on our platform? How do human agent teams collaborate? And I think there's a lot of really interesting product infrastructure, but also AI problems to explore there that frankly we haven't been able to explore yet because the models are not good enough. Right. So if we make the models good enough, then we can do really innovative research, in my opinion.
- [44:21]
Interviewer
So the collective IQ will go up 50 points at least.
- [44:25]
Nico
Yeah.
- [44:25]
Interviewer
Okay. Well, Nico, thank you so much.
- [44:27]
Nico
Thanks for having me.
- [44:28]
Interviewer
Hey, it's Molly. If you enjoy our interviews, check out our newsletter, Sorcery vc, where we deliver a once a week, top deals and tech headlines. Email and also go deeper on our podcast interviews. Subscribe to Sourcery Today and don't forget to subscribe to the podcast on YouTube, Spotify, Apple or wherever you listen. Link in Description to sign up.