Loading summary
Transcript101 lines
- [00:00]
Andre Karpathy
Foreign. Hello and welcome to the Last Week in AI podcast, where you can hear us chat about what's going on with AI. As usual in this episode, we will summarize and discuss some of last week's most interesting AI news. I am one of your regular hosts, Andre Karen. I studied AI in grad school and now work at the AI startup Astrocade.
- [00:31]
Jeremy Harris
And I'm your other regular co host, Jeremy Harris from Gladstone. AI, AI national security, AI infrastructure, all those good things. And yeah, we're just talking about how this is a a lighter episode than usual. Not just because it's only five days instead of seven since the last time we recorded, but also just like fewer papers, we're shooting the shit earlier or shooting the S, as we say on the podcast, about why that might be. I opened by saying, I mean, it makes me wonder how much things are getting starting to get clamped down on a little bit with the labs in terms of how open they're being. But then Andre just pointed out the sort of devastatingly clearly obvious reason why this is actually the case, which is conference season timing. So yeah, that's the not everything is a conspiracy, everyone. Not everything is.
- [01:15]
Andre Karpathy
Yeah, I'm a little bit out of the loop, but when there's a quiet period, it's a pretty safe call to say that it's in between major AI conferences. I mean, not to say that I think we highlight like really blockbuster papers often are from industry, anthropic in particular, but also in some cases other labs. But we do cover quite a bit from academia as well. And there has been not as much that I've seen. Not to say that there's not been many papers. There's always a lot of papers that are interesting and worth discussing, just not ones that have gone semi viral on like AI Twitter, I guess.
- [01:57]
Jeremy Harris
Like actually this is kind of. Since it's a shorter episode, maybe worth the digression here. But like reading between the lines, the papers that I see Andre picking and the papers, at least speaking for myself, that I end up picking are generally like, there's kind of a couple lenses that at least I find myself applying. The first is will this paper materially change the macro picture? So papers around about scaling, you're seeing a result that actually not just any random scaling paper, there's a million of those, but scaling papers that show trends that seem robust, that seem to apply to architectures where the hardware lottery is not going to be a massive problem. In other words, they can run pretty well on current hardware. That's a credible Kind of paper where we'll double down on those alignment results that seem certainly anything that borders on like super intelligence, like super alignment grade alignment, really interesting. Same with interpretability, at least to me. That's kind of like the first order pass is like is this a paper that may change predictions around when AGI ASI comes recursive self improvement, the risks and the outcomes of that process. That's kind of like the first order. And then there's like the kind of geopolitical does this change US China Dynamics? Because maybe there's a breakthrough that spares us on memory, which means that one of the key bottlenecks that we have breaks and now we can get help or vice versa with China. So I think that's a lot of like what comes to mind for me and just kind of like economic transformation stuff too.
- [03:21]
Andre Karpathy
But then I just like stuff that's interesting, this information that seems novel and exciting, I guess from a research perspective. So it's I guess together we have a nice rounded out set of things. We like to focus, we like to thank Box for being a sponsor. If you're a company trying to adopt AI, you're likely facing a common challenge. Most AI tools are great at public knowledge, but they don't actually know your business, your product roadmaps, your sales materials, your policies, all that stuff. And that's where Box comes in. Box is building the intelligent content management platform for the AI era, serving as a secure, essential context layer for Box's AI agents to access the unique institutional knowledge that makes a company run. The power of AI doesn't come from the model alone. It really comes from giving AI access to the right enterprise content. And that's where Box is perfect. It goes beyond file storage. It connects content to people, apps and AI agents so teams can turn information into action. With tools like Box Agent, Box Extract, Box Hubs and more, organizations can accelerate knowledge work, pull intelligence from unstructured content and automate workflows. So if you're thinking seriously about AI, think beyond the model your business lives in. Your content. Box helps you bring that content securely into the AI era. Learn more at box.com AI this episode is brought to you by Outshift, Cisco's incubation engine. Today's AI agents operate in silos, limiting their true potential. We've been focused on building bigger, smarter models, but scaling up models is just one approach to improving AI to reach superintelligence together, we need to do more, we need to scale out, and we actually have a blueprint from 70,000 years ago, humans didn't just get smarter individually. The cognitive revolution transformed society. Because we began sharing knowledge, goals and innovation, agents are now at the same inflection point. They can connect, but they can't think together. That's why our shift by Cisco is building the Internet of Cognition, transforming AI from isolated systems into orchestrated superintelligence. By creating an open, interoperable infrastructure, Outshift is enabling agents and humans to share intent, context, and reasoning. The cognitive evolution for agents is here. So go explore the Internet of cognition@outshift.com that's ouchshift.com Ever notice how life's best stories don't happen in your living room?
- [05:53]
Jeremy Harris
They happen on the open road, out
- [05:55]
Andre Karpathy
on the water, or parked under the stars.
- [05:57]
Jeremy Harris
At Progressive they get that you want to focus on the experience, not worry about the what ifs. That's that's why they offer quality insurance
- [06:04]
Andre Karpathy
designed for your ride, whether That's a
- [06:06]
Jeremy Harris
boat, RV or motorcycle adventure with confidence. Visit progressive.com and see how easy it is to protect your favorite way to get away. Progressive Casualty Insurance Company and affiliates not available in dc.
- [06:19]
Andre Karpathy
Prices vary based on how you buy, but let's do a quick preview of the episode. We do have some major stories we'll be getting to. We did record last week on a Friday. That's part of the reason that Today on Wednesday, June 17, there's not quite as much. But we recorded last week just before the big news, the other big news of that week. So first Fable 5 came out and then Fable 5 was taken away from everyone because of some actions by US government and we will be talking about that. Of course There was the SpaceX IPO last week, which is kind of a big deal if you still count SpaceX as a frontier AI lab, which I guess we should.
- [07:05]
Jeremy Harris
You mean SpaceX and cursor now because that also happened.
- [07:09]
Andre Karpathy
That's right. So we'll be talking about that. We actually have quite a few projects and open source stories I think are quite neat. So we'll spend a decent amount of time which we haven't in a bit and you know a set of nice policy safety and I I think a couple interesting research papers. So it's going to be a well rounded out episode, just not quite as dense as we usually have. So kicking off with tools and apps and we begin here with a big news from last week. The other big news last week about Fable 5, which is it was taken offline. The headline here is anthropic cuts off Fable 5 and Mythos 5 access following government order. So here's what happened on Friday. I believe it was around midday, closer to evening, Anthropic posted an announcement that it is taking away Fable 5 access from everyone as a result of the US government putting, I believe it was, export control on these models and ordering Anthropic to not allow any foreign national to have access to them, including people inside the United States, which, you know, is, like, impossible to comply with in any kind of measured way. There are Anthropic employees that would not have access to Fable 5 as a result of this order. The stated reason for why the government did this was that the. They say there was a jailbreak found for Fable 5, so you could use it in an unsafe way. You could use it to potentially hack stuff, right? Stuff like that. Anthropic responded by saying, well, we'll be taking it offline. They also said that they spent months, you know, preparing for this, working with US Government. We have discussed last week how having the safeguards in place was a big part of. Part of Fable 5. And in fact, Fable 5 got a lot of flack for being overly safety aware and, like, refusing to do a whole bunch of stuff. So the. The claim by the government is that you could get around Those safeguards and anthropics 5 position. Sorry, anthropic position, was that as far as they could tell, this was like a single, very directed jailbreak, and you could get any other model out there to do the same thing like it. It doesn't actually pose much of a danger. I think that's the recap of the situation, and I'm sure we have more to say on this beyond that.
- [09:49]
Jeremy Harris
Yeah, I mean, one of the funny, very specific consequences of this was, of course, that Andre Karpathy cannot presumably use the Mythos 5 model himself, which is pretty, like, pretty wild. You know, he just famously joined Anthropic a couple weeks ago. So there's a lot of mess here. And quite frankly, I don't think anyone's position on this is entirely logically consistent. I think that we're finally starting to get to that point that we've been talking about on the podcast for years, that something as insane as the US Government telling a Frontier lab they cannot deploy full stop, one of their leading models that they trained and developed at the cost of billions of dollars would happen. So here we are, and this is what it looks like. Is it a principled. A principled halting of this? It actually doesn't seem that way. There's so much noise around this we've got, by the way, everything from David Sacks coming out, the former White House AI Czar who's still kind of around. Let's say there's this, like, emeritus character floating around. He shared his timeline of what happened. He said they publicly released Mythos. It's basically like Fable is Mythos without guardrails. So they publicly released, really, Fable. But he said he's calling it Mythos. He. He's saying if those guardrails fail, then you've exposed Mythos and its advanced cyber capabilities to people who shouldn't have them. Okay, no problem. That's true. And that's certainly consistent with what Anthropic had been saying as well. The issue is, as you said, it seems like Anthropic and the White House had previously agreed to this set of protocols that they'd come up with to release Fable 5, and now the White House suddenly is saying, oh, actually, that's not good enough. Which seems to be coming on the back of Andy Jassy, who's Amazon CEO, Right. Reaching out to the White House and saying, hey, you know, I was playing around with mythos 5, and we found these jailbreaks. And then the White House seemingly went, oh, my God. They're jailbreaks. They're jailbreaks. When there have been jailbreaks for literally every other model.
- [11:39]
Andre Karpathy
And I'm pretty sure, by the way, that we don't know the exact specifics of this. It might have been some engineers within Amazon who found this, and the CEO was, like, told these things. We don't know even the jailbreak specifics. So Anthropic tells us that in a specific code base, they got Fable 5 to do a specific thing. And the key point is that this is not a universal jailbreak. This doesn't mean you can use Fable 5 for whatever you want with regards to cyber. It's just like, in this one situation, you got this model to do this one thing, which, again, from what I've been able to see, doesn't even seem like that big a deal. So there was a lot of discussion, as you said, with some people sort of trying to be taking the government's position and saying that this is. This is not, you know, the DOJ thing. Actually said David Sachs was like, this is not about the DOJ or the relationship of Anthropic and the White House.
- [12:34]
Jeremy Harris
Like, Department of War.
- [12:35]
Andre Karpathy
Oh. Oh, God. Dod Sorry, not doj, but, you know, the executive. Now, this is another thing we're saying. This is not sort of based on any Real framework, I'm aware of, of safety from the executive in the sense that there's no legislation. There was just an executive order which to my knowledge didn't enumerate conditions for expert controls or like categorically the opposite.
- [13:02]
Jeremy Harris
Right. As we covered a few weeks back, the executive order explicitly says that it is voluntary. And there's the freaking passage in it that says nothing in this executive order is to blah, blah, blah, something mandatory, don't worry about it, it's voluntary. David Sachs is the guy who forced that in there and he is the guy who now is saying, well, you know, back to his thing. He's saying that the admin asked Dario to fix the jailbreak or redeploy the model. Dario refused. Now this is all highly contested. We've heard accounts that supposedly Dario was at a wellness retreat. That doesn't seem to have been the case. So, so there's this, all this like attempted mudslinging that's happening here. So far my sense is most of the the mud has been coming from the admin side where like there are claims that seem to be pretty verifiably false that we've heard from them. The challenge that we face is I have yet to hear an argument that says that we can ever prevent jailbreaks from frontier models, full stop. In fact, there are really compelling arguments that say that that is a theoretical impossibility or certainly at the very least an impracticability. So we will have jailbreaks for every generation of model. Or at least if you're not planning for that being the default trajectory, prepare to be surprised. If that's the case, then clearly we're headed for a world where the moment you release a model within a few days, fricking Pliny the Prompter comes out and says, hey, new jailbreak, here's how it looks and then everybody can use it. If we live in that world, then the question is, what does serious AI policy look like? It's going to look like preventing AI model companies from simply making their models available, full stop. This kind of thing will keep happening until morale improves. Now the problem is it's being done in a completely unprincipled way and in a way which, to your point, Andre, given the history with the Department of War, can only lead a reasonable rational observer to suspect at least that there is some kind of animus that's motivating this. And the admin's comments have not helped. They've talked about weird vibe based stuff where like anthropic not showing the proper respect to the US Government is somehow a factor in this consideration, which seems clinically insane and not. Not clinically insane, by the way, for the kind of ground truth of it. Like, it may be, okay, maybe theoretically, anthropic, showing due deference to the US Government. It should be a consideration. You don't say that because for legal reasons, your lawyer is probably telling you to put down your freaking phone. So this kind of becomes a recurring issue with this administration. Over and over we see, you know, the Department of War, whether hegseth, Whether his. His undersecretaries, all these folks who just pick up phones and start tweeting stuff. You're like, dude, like, I'm not a lawyer, but I've been. I've been spoken to by lawyers, like, often enough and told to shut the fuck up. Like, so this is really, really basic stuff. Some unfortunate mudsling again, that seems to be verifiably off now. Separate question as to. There are AI safety people right now doing kind of victory laps about, oh, we have a de facto licensing regime now, which we actually kind of do. The US Government has come in and prevented a Frontier lab from deployment model we live, whether you like it or not, in the world. That was predicted by some of the kind of most prescient AI safety people,
- [16:20]
Andre Karpathy
which, by the way, it's like the worst licensing regime possible. The previous actions by the administration doing this, essentially imposing export controls in the context of the government not being able to use these models. Like, they lost that in. In courts. Like, that didn't stick. And this won't stick. This is absurd. And I think I'll go further than you. Like anyone who believes this is about jailbreaks is completely naive. Like, this is not about jailbreaks. This is about the relationship of anthropic and the government. Anthropic, and in particular the leader of Anthropic, Dario, not being willing to take the knee and basically show deference and do what the government wants. He didn't refuse. Like, there's no. No chance in this world that he just refused to do what the government wants. What likely happened was he was like, well, give us some details. We would like to do some investigation. Blah, blah, blah, blah. He was like, I want to negotiate this a little bit. I want to discuss what's going on instead of doing what you're asking.
- [17:26]
Jeremy Harris
I mean, here. Here's a completely, like, plausible conversation, right? More likely, I would guess here, given the personality involved, Lutnick or. Or David Sachs.
- [17:37]
Andre Karpathy
It could have been like, another department not necessarily the presidential cabinet.
- [17:42]
Jeremy Harris
Yeah, we really.
- [17:44]
Andre Karpathy
Of commerce. Maybe it's not very clear. None of this is very clear.
- [17:47]
Jeremy Harris
Yeah, exactly right. But you can imagine them going to Dario and be like, hey, look, Andy Jassy just told us about this jailbreak. You yourself have been telling us for a long time that this is basically an AI superweapon wrapped in a thin layer of security coding. And what they've shown is that you can actually penetrate the security coding. So you got to fix this. And then Dario goes, I cannot break the laws of physics, Captain. And like, basically, I like, there's no way. I'm not going to lie to you and tell you that I can fix all jailbreaks. If I fix this one, then the next one, then the next one. Like, we're always going to be doing this. So I guess we're just at the point where we got to figure out how to manage these two things that can't be reconciled. And then I guess the admin went, okay, I've got an idea. How about we blow it up? We're going to take the boat. We're going to. You know. And that's basically what happened. And so here's the thing. You can do a victory dance on this if you're an AI safety person, but be aware that we have not tested the foundation of principle on which this entire edifice rests until we see OpenAI develop a mythos class model. And will the admin do the same
- [18:52]
Andre Karpathy
to them, which they won't, like OpenAI, I don't know.
- [18:57]
Jeremy Harris
I'm ready to be proven wrong here, and I hope I am, but I also partly don't hope that I am, because what I would want to see here is the conditions under which things can continue. So one of the best things you could do with a situation like this is have clear incentives for loss of control research, for jailbreak research. Like all. All the good things that are needed for Super Alignment, you could incentivize that massively. Guess what Super Alignment just became, or at least jailbreak resistance just became the single most important bottlenecking factor for Frontier AI deployments around planet Earth. Right? That's the story. Great. That means you should see massive investment of resources pointed in that direction and a liability and licensing regime that comes with it. Three years ago, we built one, by the way, that was the State Department commission report that we put out. The. Like, we pre thought this through a lot. There are a whole bunch of crappy things with that and mistakes that we made, blah, blah, blah. But like, you're now firmly in the territory of having that conversation, which at the time was way outside the Overton window. It's just a shame that this is not Congress doing this. This is the executive in a context where it can be doubted. And then the next administration that comes in is going to say, oh, this is all political. Let's steer hard in the other direction. We have not yet seen what actual licensing framework is going to come out of this. And that's part of the problem is it's so chaotic. You need to see all the Frontier Labs come together with the administration in a principled way. Talk about is a compute threshold. What are the evals? At the same time, we're see, we're seeing KZ get completely shafted. Basically, the one arm of the government that was charged with doing kind of robust evals and that has that ecosystem is no longer being tapped in the same way to get model evals done. So this is, this is a really unfortunate.
- [20:43]
Andre Karpathy
And I'll go further, this, as you say, is unfortunate for safety in multiple ways. So for one, it doesn't necessarily encourage kind of principled approaches to safety. It doesn't show any sort of capability by the government to actually evaluate safety in a principled way. And it's worth covering. I think the other sort of news or the depth of what happened was, of course, there was the event itself and the politics around it and Anthropic. Anthropic's relationship to the government. But then the overall AI community and its response, I think, was interesting.
- [21:20]
Jeremy Harris
Yes.
- [21:21]
Andre Karpathy
Yeah, true. First, you know, there were a lot of funny memes, which is always fun, so that was great. But there was, I think, a lot of sort of, you know, having fun at Anthropic's expense or not even that. It's a lot of, like, angst at, like, yeah, Anthropic was saying it's so unsafe and these are like nuclear weapons. And guess what? If you keep saying that, this is what will happen, right. Anthropic brought in on themselves, which would mean that people like actually Frontier Labs are discouraged from discussing safety and from positioning these things as dangerous things. Now, again, saying that this is because of Anthropic doing all their discussion of safety and so on and the position papers and all that, I don't think is correct at all. But regardless, the perception seems to be that Anthropic brought it on themselves by discussing safety and by positioning these things as dangerous. And again, jumping on the story of, like, Mythos was a lot of PR Anthropic was kind of doing BS here and this is verified.
- [22:33]
Jeremy Harris
It's like the same people who are saying Mythos is not as powerful as it was who are now saying, ah, anthropic. Like, that's what you get for talking a big game about how this is a w. Like a weapon of mass destruction. And like, the problem is those, those people have been proven wrong. Like, trivially like, that Debate is over.
- [22:51]
Andre Karpathy
Facts are in. Yeah, yeah, exactly.
- [22:53]
Jeremy Harris
Like, we've got the vulnerability list, the critical vulnerabilities on load bearing pillars of the Internet. Internet's like most critical infrastructure. It's game over. Anybody at this point who's claiming Mythos was not a real thing was. Is claiming that it was predominantly a PR exercise, including Sam Altman himself, is full of shit. Like, trivially like that is now known now. Now the question is, okay, what do those people pivot to when that argument is taken away and this new situation happens? Guess what they pivot to. Okay, like, let's dance on Anthropic's grave. Which, like, weird, weird pivot. But okay, yeah, there's so much to the story and I guess we'll have to move on. But like, just one last thought here is. Is almost the hypocrisy of the US China thing. What was David Sachs arguing? Like, 20 minutes ago, David Sachs was arguing that we cannot hobble US Innovation by introducing binding regulations and blah, blah, blah, because China. And like, here's the thing, people who watch the podcast, you know, I'm a China hawk on this stuff. I'm also an alignment hawk. I think we actually have to walk and chew gum. Gum at the same time here as a China hawk. I. I hear David Sachs make that kind of argument. I'm like, okay, yeah, cool, that sounds good. That makes sense. Okay, now you're going to have to weight that eventually against alignment in la Control risk. Right? That part never seems to happen with David Sachs. That's okay. Now we seem to have suddenly, though, a massive pivot to. Let's justify the like, outright halting of the deployment schedule of a frontier lab that has coordinated with the White House by everyone's accounts already to set up this moment on the basis of some technical triviality that if David Sachs himself did not see this coming, that just means he's not watching the ball. Like he doesn't understand AI in a very basic way. Because everyone who understands AI in a basic way knows that jailbreaks are a thing and are not going away like you should have. If it's A surprise to David Sacks that jailbreaks exist for this model, then, like, he has no business occupying the position that he does in the White House or the position of influence. Like, I just don't see how you make sane policy misunderstanding such a fundamental basic thing about the technology. So anyway, this is like very ranty on my part. I apologize. It just seems like there's just. There's so much crap smeared on the walls on this one that man.
- [25:04]
Andre Karpathy
Yeah, I'm sure we'll have follow up stories as of now. There's some like, informal reporting that there are conversations going on and maybe Fable 5 will be back online in a few days and maybe not. I just want to reiterate, this is not about jailbreaks. Really. That's sort of a cover story. Even if it is about jailbreaks, it's about, from what we've been told, a very kind of specific moment. Minor case of a jailbreak in a particular code base for a particular task. It's not a universal jailbreak. I think there is a good argument to be made that with Fable 5, Anthropic just threw so much onto the wall in terms of safety that if you were to try to do a. Deploy, like a prolonged operation, a cybercrime initiative, whatever, they have this sort of architecture in place to catch you or hopefully. Yeah, to the extent that we have models gated behind APIs. That's right. We have the ability to do safety. Once you use open weights, anything is. There's no alignment, there's no safety. But women, Tropic, there is a potential for safety. And this is another dimension in which is unfortunate. Like, they invested on safety more than anyone. They got pushed back. And now this happens. Right.
- [26:22]
Jeremy Harris
Well, this is the thing. You, you can, you can argue that you don't like Dario or whatever, and that you think what they're doing is dangerous or whatever. Like, you can do that all day long. You're going to sound more and more like a freaking effective altruist, by the way, if you start to do that, which is what David Sachs is starting to sound like. And I don't think he, he would like that comparison. But just in terms of the direction this, like, it's hard not to interpret it that way. Here's the thing. I'm willing to be proven wrong when there's something that comes out next week, inevitably. And there is some crazy level of nuance to this that we didn't hear about, but just based on what's in the news at the moment, it seems to Resolve to me as like one of two things. You get to think that either, okay, the jailbreak is something to worry about, if it's something to worry about, then I can only chalk this up to gross incompetence by David Sachs and anybody in the admin who touched this issue, like who was responsible for the admin's ultimate position. Because all that means is the admin institutionally does not understand that jailbreaks are not going away. Like if you don't understand that, then you don't understand enough to regulate the technology. And so like basically, yeah, pick your poison. Either the either yes, this was a legitimate place to intervene and the admin did so wildly and competently or the jailbreak was no big deal and the admin overextended. Like it, like it's kind of one or the other anyway. It just seems like it's so, so technically unmoored. There's no adults in the room of looking at this being like sober minded. What would a principled approach to this be that applies to everybody equally? We're just not getting that.
- [27:51]
Andre Karpathy
And in the context of upcoming IPOs, I really have to wonder. It's now about Anthropic doing what the government wants or having a good relationship with the government. This happened again. It happened with the DoD DoW. Now it's happened with Fable 5. Not directly with DoW, just generally with the executive department. And if anthropic keeps pushing back, which by all accounts seems like they want to be like, no, we are realistic, rational, we want to do things based on truth. The situation is a great front topic at least. Looking forward to meet.
- [28:36]
Jeremy Harris
Yeah.
- [28:38]
Andre Karpathy
Next up, you've got a story that we won't need to be discussing for 20 minutes. Just a fairly minor tool slash app update. Facebook has released a new AI mode search feature very much similar to AI mode within Google. You enter a query of some kind and an LLM response driven by their in house model, Spark newspark gives you a little summary of discussions and conversations, pulls up some images and reels from across the meta ecosystem. That's about it. There's now this meta AI, you know, AI mode thing to do search that it looks like meta is like, oh, we are rolling out AI. This investment is worth it. That's what this reads like to me.
- [29:24]
Jeremy Harris
Yeah, kind of like the Google previews and pulling from Reddit. Right. That they do is sort of a similar idea. You're pulling from conversations that people are having about a topic and yeah, why not? Well, why not? There are a million privacy reasons I'm sure that people can cite and have cited, but from a technical standpoint, it seems the business case is pretty solid.
- [29:45]
Andre Karpathy
All righty. And that's it, actually, for tools and apps. Now our major announcements. So we are moving right along to applications and business. And the first story is SpaceX. So two things happened. One last week, one this week, last week, on Friday, again, just after we recorded, or I guess during the entire day as we were recording, SpaceX had begun their IPO. So they have gone public, meaning that they're listing shares on the public markets for anyone to buy. Anyone, asterisk. There were some IPOs are a little weird. You kind of set aside some shares for different people. But they went public and the IPO went well. So first of all, they're trading at high valuations above what they went public at. They are now insanely valuable as a company. They went above $2 trillion. They listed a IPO at a valuation of 1.75 trillion, I believe. So the IPO went well. They now are able to sell shares on the public markets to generate cash for various reasons. Then this week we got the news that SpaceX is going to be acquiring the AI coding startup Cursor for $60 billion. $60 billion. Not in cash, but in presumably SpaceX shares and maybe some cash, some mix of the two. We have discussed in recent months that SpaceX had already announced this deal with Cursor where they're going to partner up and they have the right to buy it for 60 billion and they might or might not do it. It seemed like everyone knew that there are now going to do it after the ipo, and in fact they did do it. So pretty much that's the situation. There are some interesting aspects here to discuss on, like why did SpaceX and really XAI within SpaceX need to acquire Cursor? I think the obvious thing is like Xai kind of died a little bit. Everyone left. It was a miserable place. Grok was not like at the cutting edge. Despite them having some early signals of being able to compete as a frontier, they did not focus on coding at all, it seems, and are way behind on that front. So in acquiring Cursor, they are in a better competitive position to do coding in the sense that Cursor has at least some experience post training models, they have data on users of Cursor and they do have some revenue to speak of. Which, by the way, SpaceX, you know, if you look at the numbers now, that's.
- [32:29]
Jeremy Harris
That's what I hear.
- [32:30]
Andre Karpathy
I mean, yeah, so the SpaceX shares did actually gain seemingly on these news. And I think it's kind of beneficial to both really parties here. Cursor gets access to unlimited capital and the compute that XAI has, you know, built up over the last few years. And XAI gets actual talent to do stuff with all the compute.
- [33:01]
Jeremy Harris
Yeah, I think when you think about, you know, what these, this was the match that always had to be made. You know, if you're looking at Cursor, their market share has actually dropped quite precipitously from 41% if you look back to June 2025 to about 26% in May. So in about 12 months you're seeing the market share drop by. I don't know what that is. About 40%. Right. That's a, that's a big problem.
- [33:22]
Andre Karpathy
I forgot to mention, for anyone who doesn't know what Cursor is, I guess it's worth mentioning. Cursor is primarily a provider of an integrated development environment for coders. So it's the program of which you want write code. They were the leading kind of coding environment with AI. And around 2024 everyone was using cursor. If you were using AI, then cloud code and so on came out in 2025, 2026, and everyone was like, well a lot of people are like, I don't really use Cursor anymore, I just use cloud code.
- [33:54]
Jeremy Harris
It just kind of quietly, yeah, it rolled off on the, rolled off their subscriptions. And I think that's sort of the death knell that companies like Cursor. I want to say, really at this point there are different escape velocities and moments where you know, you just have enough, whether it's infrastructure or kind of talent or whatever, where the market just allows you to succeed enough to then get acquired in this space. And this was a match made in heaven. I mean, xai obviously, or SpaceX has the ability to, to build data centers like no one else, like just at an outrageous speed. And they have all kinds of supply chain advantages with, you know, thanks to Tesla, getting their hands on batteries, getting their, getting the hands on generators. You know, there's all kinds of advantages on the infrastructure side and it's Elon, he builds stuff fast that you shouldn't be able to build fast. Cursor has always had this problem, in fact was enmeshed in scandal as a function of this problem when it came out that they were actually never training their own models, never pre training their own models. They were only working on top of open source models. And then doing a lot of fine tuning, a lot of post training, but not training from scratch. And so if you're cursor, I mean the requirement for pre training, we've only seen that go up. Especially as Anthropic has pushed the frontier of pre training almost more than any other lab. It's become more and more clear that pre training is an essential part of the game. And you know, as you say, XAI was sitting there with like a just gutted bench and you need talent. So what this suggests, we've known that the Cursor was using Colossus 2, the big mega cluster that Elon just built recently for training runs and that had us last time look at this and go, okay, there's probably going to just be an acquisition here like because you're, when you're letting these guys just run your most critical like gigawatt scale infrastructure for an extended period of time, that's, I mean that's what value is denominated in in the space. That's the most sacred resource SpaceX has. So if you know they're not using it for xai, they're just handing it over. Cursor, that's a test to see can cursor produce at pre training at scale. The fact this acquisition went through strongly suggests that that training run worked out that something about it suggests to SpaceX or XAI, what do you want to call them, that this is at least better than the position they were in before and worth the 60 billion because they did have the option not to make the acquisition. Right. It was 60 billion or I forget what convoluted, you know, like payout was, was going to be involved but you know, it's a positive sign for that training run.
- [36:21]
Andre Karpathy
Yeah. So I think it's likely that this will lead to more GROK updates that now XAI will be able to actually deploy their capital instead of just leasing out and being a NEO cloud. Now where will basically stay on NEO cloud is not clear. I think at this point, I guess the reason to need pre training actually it's a little bit nuanced I think where for a lot of stuff post training is sufficient given the very, very strong open source models coming out of China with Kibme, K26, Deepseek, et cetera. In fact, Cursor I think proved the case that post training can get you very far. But if you want to compete on the Fable 5 Mythos 5 level, that's where you need next generation pre training and it's really competing. You Know, all these companies, I think to some extent are competing to get to AGI because there's a belief that whoever gets to AGI first kind of will own the economy. There's no one really knows, but it's like we need to get to AGI first because, you know, winner take all. That's also why everyone is competing to get to a curse of self improvement. So with this acquisition, Xai can at least be in the race. Given that they just were almost like a dead company kind of.
- [37:50]
Jeremy Harris
Yep. Their best play would have been to turn into the infrastructure layer for Super Intelligence. It's never clear to me what that even is supposed to mean. Like, if Super Intelligence works as advertised, whoever owns it ends up figuring out a way. Give them a little bit of time and they'll figure out a way to replace you as the infrastructure layer for superintelligence. I mean, if it genuinely, like, if it's what it's supposed to be like, doesn't mean it will be. But if it, if the thesis bears out, like your chip design, you're like finding clever ways to access power. Like all the things.
- [38:20]
Andre Karpathy
Yeah, I mean, I guess you get unlimited money and you can do whatever you want with money, including building data centers. So. Yeah. And speaking of data centers, next up you've got a story on Anthropic. Anthropic pursues data center leases and is seeking financial backing from Google. So they are now pursuing their first direct data center leases in the US Having already signed more than a dozen preliminary agreements for facilities with a combined capacity exceeding one gigawatt. So for years now, they've been securing computing capacity by renting chip servers from AWS and Google. And here they are looking to actually own the data center's lease and manage your facilities, with Google helping to underwrite release obligations. So it seems to be kind of actually moving fast. It sounds like Anthropic wants to have this starting to come into effect this year. And I think it again tracks with Anthropic not having enough compute like as of their kind of takeoff this year, they've had a lot of struggles just meeting demand. They've had a lot of like, weird things where they adjusted their limits and a ton of instability. Like Anthropic, their API is very, very far from the sort of stability you want to see for a web company and for, you know, like a provider of essential models really for many businesses. So remove makes a lot of sense. I guess the question is, are they going to be able to actually pull this off?
- [40:02]
Jeremy Harris
Well, yeah, and so the calculation here, if you're anthropic, is you can, you can do what you've been doing, which is go to Neo Clouds and hyperscalers and basically like rent out access to the capacity that they build. Right. So you'll have neoclouds like fluidstack, they'll come in, they'll work with a local builder and the builder will, you know, build the data center and then FluidStack will make sure that all the server racks are there, wired up properly and all that. And then Anthropic just comes in and pays a premium, you know, a good margin to these Neo clouds, part of which gets passed on to the builders to rent this capacity. And so number one, as we know, Frontier AI is a margin play. They're like the margin really, really matters to determine profitability. And that's the reason that if you're anthropic, you might say, hey, why don't we just do this all in house? Another reason that's not mentioned here though is often, I mean, just like you have catastrophic incompetence by the builders, this is something that's not widely sort of like understood, though it has been publicly reported, which is why I can share it. But like you'll have builders who, you know, like, with wild delays, like they'll come in and promise the, the moon and the stars. And then it turns out things take, you know, four months, four months longer than expected. And Anthropic will be betting here, most likely that, hey, you know, we can just do better. Like we, we have the, if you bring the level of caliber that they, that they have on the compute side, on the AI research side, and if you had that same caliber of talent on the construction side all coordinated under one roof, you probably could have better kind of cohered infrastructure stacks and building times. And so that's another kind of implied part of this calculus. The margins are not small and the costs are not small. So if you're anthropic, you now got to spend the capex. You got to find a way to fund this massive build and this risk that you're taking on. Right before it was rent the facilities, we don't have to spend as much money upfront. And that's a big de risker. Now it's like we're going to own the whole thing. If you're going to do that, you're going to need to take out loans. The problem with taking out loans is you're Anthropic, you might be a trillion dollar company. But your credit is not good enough for most lenders because you haven't been around for a long time. And so what do you do? How do you get a loan? Well, you have to partner with an entity that is big and massive and old like Google and that's what they do. So they basically piggyback off of Google's credit and credit rating and kind of credit relationships to fund this activity. And OpenAI is in a similar boat. They're also too new to have that, that sort of thing. So you see these weird partnerships where it's like, why does anthropic need, if they're going to do it themselves, can't they just go to the bank and get a loan? Well, no, the banks will not loan to a company this risky, this young, I won't say this small because again, trillion dollar company, it's like bigger than what Google was not too long ago. So anyway, there you have it. It's all these entanglements and it's part of, you know, you'll hear all the arguments about circular funding of this and, and yes, it does increase the strategic dependency between Google and Anthropic. That is true. But yeah, it's a margin play and they're betting that they'll keep that. The risk isn't that high because they're going to need the infrastructure.
- [42:55]
Andre Karpathy
Next, some stories on OpenAI. The first one is leaked financial documents show OpenAI is losing billions of dollars a year. So it looks like financial documents disclosing a lot of details as to the specifics of OpenAI's expenses and revenue have indeed leaked and been verified to a significant extent. So we see some interesting numbers here. We know their revenue grew from 3.7 billion in 2024 to 13ish billion in 2025. But their expenses, R and D expenses also went up like a ton. They went from 7.8 billion in 2024 to 19ish billion in 2020 25. And in general, their spending is outpacing their revenue and is likely to continue to do that. They had total operating losses going from 9ish billion 2024 to 20 billion in 2025. And also their other expenses grew significantly as well. It wasn't just the R and D. They are spending way more on marketing. They went up from like 1 ish billion to 5 ish billion for sales and marketing, which is quite a lot, like very general. And administrative cost is now some absurd number as well. So nothing here that seems overly surprising. It kind of tracks with the rough picture of OpenAI, they have a lot of revenue, but you're also spending a ton on being in Frontier and just generally as a growing business. But maybe notable to have these details out in public.
- [44:42]
Jeremy Harris
For sure, yeah. It's also. So the guy who this was leaked to is a guy called Ed Zitron. So he's been one of the sort of. I don't know what to call him, like the Gary Marcus of the economics of AI, if you will, like a real skeptic on will the labs actually be profitable and is this all a bubble? Basically, I think that unfortunately for. For his position here, which seems to be. Oh, look at this. This proves what I've been saying all along. This kind of seems to. I won't say prove the opposite, but it's evidence, pretty compelling evidence that points in the exact opposite direction, as far as I can tell, reading the numbers from what Ed Zitron has been saying this whole time. So here's specifically what I mean. We're going to focus on a couple numbers. There are a lot of numbers cited here and you could put asterisks everywhere, which is why I'm not going to make any definitive statements. There's just. This is evidence pointing in a certain direction. So OpenAI's gross margins, right, Meaning, like the difference between the revenue that they bring in and then the cost of earning that money. Right. That's kind of the kind of core underlying business metric. This does not include things like AI R&D. It doesn't include kind of the cost of legal restructuring of OpenAI, which was a massive cost. They were all those ancillary things, this, granted, does not include those. This is just tokens come in and. And like, do you make marginal dollars on marginal inference? That's the question, right? And nominally, if the answer is yes, all you need to do is scale that business. You just need to make that business bigger. And then you will make more money as long as the margins continue to be high as you scale. With that in mind, what are OpenAI's gross margins? Well, in 2024, their cost of serving customers was about 2.7 billion and their revenue was about 3.7 billion. So positive margins, in fact, 28% gross margins. That's good. So for context, in the world of SaaS, like software as a service, like, classically, you'd see margins from like 70 to 80% or something like that. Right? So, like really, really high margins here, you're looking at, you know, 28%. It is more of a utility. There is a lot of asterisks to go around here, but it's a scalable business with nice margins. Last year, though, the cost of revenue was 7.5 billion. The actual revenue more than tripled of 13 billion, which makes 43% gross margin. We're starting to get pretty damn good margins. The margins went up with scale and with time. So these two things taken together, I mean, this does just start to look like a business that like, yeah, like, why don't we just like, click and drag and like, grow this business and we make more money? That's very consistent with the arguments we've heard Sam Altman make. And by the way, there are indications that OpenAI's margins are actually worse than Anthropics and potentially by quite a bit. So it's not. This is not even the number one player in the race and they're. And they're killing it with almost 50% gross margins. So, I mean, we got to see it. And there's all kinds of arguments for why Sam might not want ipo, including the scrutiny behind these numbers and so on and so forth. But if the top line story holds, this is a business, like, this is a business. And again, the challenge people run into is they'll look at like, okay, yeah, but how much did OpenAI spend in total this year? And that ignores the fact that that spending is meant to fund revenue that comes in 18 months from now. We've talked about this so much. But like, the dollar spent today on the data center of 18 months from now only leads to revenue then. Right, and so you have to do time discounted money to get to that point. Yes, but if Anthropic is profitable as they were this quarter, that's bad. It means that they underspent on scaling. And if you rewind the clock back to see again what the key question is, what were guys like ed Zitron saying 18 months ago about anthropics build out? And if they were saying, oh, they're burning money, this is a blah, blah, blah, blah, then as a matter of simple logical fact, that call was incorrect. So the only question is, does that continue? So anyway, yeah, there you have it.
- [48:44]
Andre Karpathy
And another story related to OpenAI chat. GPT's market share slips below 50% for the first time. This is according to reporting by Sensor Tower. This is showing now that ChatGPT as the chatbot has dropped below 50% market share, falling to 46.4 by the end of May 2026. Gemini now holds 27.7 market share and Claude holds 10.3%. So OpenAI is still in the lead by a good margin in terms of the number of users using ChatGPT as their chatbot of choice. But their position is eroding and you know, seems likely to potentially continue to erode in the sense that Gemini can afford to be cheaper and Anthropic can afford to be more expensive in the sense of arguably providing the better product for enterprise and just professional productive use cases, broadly speaking. So yeah, if you're looking at stories related to IPO and how much you should think this is bad for opening, I actually the financial story is like, okay, they're spending a lot, they're losing some money, whatever, it's nothing new. This story of they're losing market share, much bigger deal and I think not surprising, but it'll be interesting to see if this does continue and if OpenAI is able to reverse your trend.
- [50:21]
Jeremy Harris
Yeah, it's also like it's always complicated when you look at market share percentage in a growing market. Right. So OpenAI's user base has continued to grow fastest, you know, app to reach a billion users. They're now starting to feel the edge effect of like the global population. So you know, at that point you ought to expect erosion of that top line metric. What this means is the entire space is growing faster than OpenAI can just absorb the traffic. And yes, there are other competitors nipping at their, at their heels. It doesn't necessarily mean bad news for OpenAI in and of itself, but there are some pretty interesting warning shots for them. So if you look at Claude, they have by far the highest conversion rate to paid usership in the field. 13% of anthropic users pay for Claude subscription. So when you look at those low numbers of general usage for Claude, you know, 10%, not surprising, you know, it's been a thing for a while. They've been targeting enterprise. At the same time, OpenAI is rotating more into being an enterprise focused company too. And that would come as well with a reduction in just like general market share and sort of attention share in the general population. You kind of expect that in that sense, I think it's a story of like intentional change from OpenAI to rotate into enterprise. It's also a story of like the limits of the market itself. But as they rotate into enterprise, 13% of anthropics users paying for a subscription, that's a real headwind for OpenAI. Right. That means that people who build stuff, because that's those 13% are choosing anthropic. And until that changes. The high margin enterprise revenue is much harder to access for OpenAI.
- [51:56]
Andre Karpathy
And I think a general story of OpenAI is that they've been trying to do too many things for a while now. They have explicitly backtracked some things like Sora and their various initiatives. And this is trying to fight on both fronts of consumer and enterprise. Kind of is the worst of all worlds because on the one hand Anthropic is competing with you on enterprise. On the other hand, Google seemingly, as far as I've been able to tell, is still focusing on consumer and getting more users to use gemini. And so OpenAI has to fight on two fronts. On both fronts they have very strong competition and if you try to dilute your focus as OpenAI has done, your ability to fight effectively is kind of Limited. But OpenAI is still in the lead on consumers, still probably has the most recognizable brand as far as chatbots. And I'm sure if they keep losing market share, they'll keep having a massive amount of a market.
- [53:02]
Jeremy Harris
Yeah. And I think Google's enterprise play here is also just like the bundling of Gemini into the Google ecosystem. So, you know, in that sense maybe they look a little bit more Microsoft Y.
- [53:12]
Andre Karpathy
But yeah, yeah, it's a different kind of enterprise than like Codex and cloud code and so on. Yeah. Next, a story on Meta and I'll have to swear a little bit for this headline. Headline is tell him he's a piece of shit. Meta's new AI unit is a total mess. So this is reporting from Wired, it's talking about Meta's Applied AI team, which has been formed in March and now has around 6,500 engineers and product managers. It was formed to support Meta Superintelligence Labs researchers. And there's some kind of wild details coming out, out of this thing. So there were some interviews where, you know, insiders have characterized the work inside it as soul crushing. Someone called the environment the Gulag. And it appears to be the case that a lot of people are just doing like data labeling and like very, very basic, like data exactly like the Gulag.
- [54:19]
Jeremy Harris
Like do you remember, do you remember when Stalin had all those poor Russians doing data labeling in the gulag?
- [54:26]
Andre Karpathy
Yeah, well, the gulag aspect here, about like the metaphor, if you want to go there, is that the way this has seemed to play out is that various Meta employees from various parts of Meta were just like randomly surprisingly assigned to go to applied AI and they didn't have a choice. They were like, okay, I'll do it or I'll quit. And so a bunch of people have been pulled from all around the company to go to this thing that again was formed in March. This is not like an established thing that is building products. It's explicitly to help Meta work on superintelligence and help TBD labs and so on. So I think broadly this is a business story in the sense that morale is low. That's very clear for Meta like throughout the company, not just for the supplied AI team. There was also over 1,600 Meta employees that signed a petition opposing the new initiative to monitor their clicks and keystrokes, which we discussed a month or two ago. And Meta, like slightly scaled back the program by allowing 30 minute pauses. So they didn't like relent fully. There have been like company all hands by the leadership that they've been publicly saying, oh, we messed up, you know, it's a rough time, there's going to be some challenges. Zuckerberg himself had addressed this topic. So if you're trying to compete in general in the software world and in particular, I think in the CI world, having low morale, having employees who aren't excited, having just like people unhappy is not good. And the fact that the Super Intelligence labs created this thing with 6,000 employees to do data labeling or these kind of menial things also isn't a good sign. Right. So maybe they're just going to be able to brute force their way to strong AI as they seem to be doing here. But the more Meta pisses off their workforce, the worse it'll get for them long term.
- [56:38]
Jeremy Harris
Yeah, I thought that the worst part of it was when they like threw the employees in the back of Black Maria's and drove them to black site where their clothes were taken away and then they were all made to pee with like 120 people to one toilet.
- [56:56]
Andre Karpathy
I think metaphorically. That's clearly like where they're heading, right? Like we have a gulag now stuff.
- [57:01]
Jeremy Harris
If you just extra, if you just extrapolate, right, the scoring for. Yeah, you get to gulag by next year. Yeah, no, but, but to your point, I think feedback loops are the way up and feedback loops are the way down dynamic in this space. And you're seeing it with, with Xai. Right. Elon's approach was quite telling. It wasn't like, okay, well, you know, just needs some tweaking. Like, let me. No, no, no, like your feedback loop is wrecked, which means you need to start from scratch because hysteresis is not your friend here. Like the effect of having scaled all the way down and unwound. It's just like any good, like market collapse, right? Like speculation on the way up, speculation on the way down. You overcorrect. And so he's just like, all right, we're pulling the plug, redoing. That's what Zuck was trying to do with Alex Wang. The hope there would have been that you could get a clean narrative. And they tried. You know, they've been doing pretty well on the kind of pre release, hey, we're building the infrastructure, but not yet the models, but eventually the models for superintelligence, that was kind of working. We've covered that before, but this is, yeah, seems starting to show and a weird play. I mean I understand why. If you have a stable of thousands and thousands of employees, that does look tempting. It's a lot of good data. But like the morale consequences, that's challenging.
- [58:14]
Andre Karpathy
I think it's worth noting too, like especially on the top, top, you know, tier of talent of researchers and like the people that you pay $10 million or $20 million or whatever to like Meta has been throwing money at them and that is effective to an extent. But like Anthropic can throw money at people too and they can throw shares at people and shares is how you get rich. So if morale is low throughout the company and it just like is sad to work there, these like super top tier people who everyone is fighting over are going to leave. They're going to go to Google or OpenAI or Anthropic. And if you're trying to compete in the frontier AGI race, if you don't have a superstar team, I don't think you have a a shot. Right. So this is not looking good for Meta. And last up, we have not a huge story, but I think a slightly interesting one that is fun to cover. Sakana AI commercializes AB MCTS in Sakana Marlin and Enterprise Agent, generating up to 100 page research reports. So Sakana AI we've covered a couple times. They're a company based in Japan founded by some notable AI researchers that has published papers of various kinds about evolutionary optimization and automated research. They have now built this B2B autonomous research agent positioned as a virtual CSO that can do a bunch of research for you, right. And generate reports and slides and so on and so on. Build partially on their research and on their work on the AI Scientist initiative. They have gone to closed beta in April with, you know, a couple hundred customers. Now they are partnering with some bigger companies, receiving strategic investment from CITIGROUP they have, you know, pricing various things like that. So they're actually starting to commercialize, which probably is good to do as a lab that isn't big and doesn't have its own models and so on. It does position Sakana, I think, in an interesting place where the sort of like research agent as a service, as a thing you're selling isn't like a focus on anyone. Like deep research was all the rage in 2024. Now coding agents is overage, so there might be a space here to do really well. I don't know. It's interesting.
- [60:56]
Jeremy Harris
Yeah. Their whole. And I think we've talked about the architecture that are using behind this. It's this thing they call adaptive branching mcts. So like Monte Carlo tree search. Basically the idea here is, you know, Monte Carlo tree search is like any given problem, you try to decompose it, decompose your solution into steps. And at every step you have a bunch of choices as to what your next step might be. You can kind of take one of those choices, one of those branches, and then explore it more deeply, or you can look at other branches basically. And that trade off between going deep in one branch versus going broad in the others is what this whole kind of, you can think of it as a kind of scaffold around an LLM is designed to help them do. And they posted some interesting kind of promising early papers about this a while ago and that's what they're now basically baking in at the bottom of this. Yeah, I mean it's like, it's, it's interesting. The depth is the competitive differentiator. As you said, this means the inference runs are going to be longer. So for a given problem, you know, you give the problem and it'll run off and do all the research, you know, look into your, your target market. It'll make slides, develop hypotheses, that sort of thing. So kind of interesting, you gotta wait a long time because it'll do literally hours of research and like thousands of API calls. So pretty interesting play on are we there yet in terms of, you know, hundred page documents that might determine corporate strategy. Is there alpha here? I don't know. Maybe we'll find out. Maybe we'll see companies based on this stuff soon.
- [62:21]
Andre Karpathy
Insurance isn't one size fits all. That's why customers have enjoyed progressives name your price tool. For years now, with the name your price tool, you tell them what you want to pay and they'll show you options that fit your budget. So whether you're picking out your first policy or just looking for something that works better for you and your family and they make it easy to see your options. Visit progressive.com find a rate that works for you with the name your price tool Progressive Casualty Insurance Company and affiliates Price and coverage match limited by state
- [62:49]
Jeremy Harris
law I'm Kiana and I leveled up
- [62:53]
Andre Karpathy
my business with Shopify. Once I figured out that Shopify was a thing, I never turned back. I can create a site with my eyes closed. Shopify thinks ahead of us, you know,
- [63:05]
Jeremy Harris
and it thinks about the customer more than anything. Every day I'm thinking about some other
- [63:10]
Andre Karpathy
new business, but Shopify is doing it to me because it's so easy to use. It's like, I can't stop.
- [63:15]
Jeremy Harris
I'm addicted.
- [63:17]
Andre Karpathy
Start your free trial@shopify.com and speaking of that, next up we have projects and open source. Sakana actually open sourced their search library as TreeQuest fun title. And we have a bunch more open source and projects things to cover and the first one is actually more of a project. We have Open Router which is a fairly popular Service for paying one bill and then routing it to various providers of LLMs. They had announced this new thing called Fusion that synthesizes outputs from multiple AI models and then as a result they say outperforms the individual models on complex tasks. And you can query OpenRouter Single API call to Fusion that then dispatches a prompt to a panel of models in parallel. And then there's a judge model that synthesizes all these things into one final answer. And they looked to things like Draco. This is focused on research actually on like answering questions or generating hypotheses and things like that. So I think very really not surprising that having an ensemble of models all doing their own research and then finally synthesizing them on various points, not surprising. But having it be an actual product and something that you can pay openrouter to do is a little bit interesting. And it is a sort of advantage in a sense of being the sort of, you know, provider to providers. Anthropic isn't going to sell this to you. OpenAI isn't going to sell this to you, but OpenRouter is happy to be like, yeah, we'll dispatch it to all the models and then combine it all into one thing.
- [65:06]
Jeremy Harris
Yeah. And we've seen papers and it's been a known thing for a long time that ensembling models together does give you a better, better quality output. The trade off has always been historically the added cost. Right. You're just doing best of N sampling basically and that's N times nominally the cost of doing it once here all their optimizations actually allow them to drive down the cost quite significantly just by taking advantage of which models are better at different tasks. There's two different elements going on here. So one is just like the lifting capability that comes just from synthesis, from having a bunch of different models like kind of best of N. So they, they measure this by taking like Opus 4. 8 and then they pair it with itself to, to, to make kind of like this, well this, this pair of models and they find that it outperforms solo opus 4.8 by like 6 to 7 percentage points on a set of benchmarks. The second piece though is the diversity value. What do you get from swapping one of those identical models for a genuinely different model? And so in the, in the case of GPT 5.5, when they swap out, when they have Opus4.8 with GPT5.5, they get an additional boost in this case of like around 2, 2.1%. So you actually do see like a very measurable and that's that. I think the big take home here is the ability to actually measure how much comes from the diversity versus how much comes from just the best event effect. And in both cases it's non trivial. I mean 2 percentage points on like software engineering benchmarks is, is non trivial, especially when you're already scoring, you know, in the like high 60s, let's say.
- [66:45]
Andre Karpathy
Next up, open source moonshot AI has released Kimi K27 code. This is a specifically coding model and it is doing like way better on the Kimi code bench over Kimik 2.6, which is the previously most released model releasing under a modified MIT license. So mostly anyone can use it for anything. Once again, a mixture of Experts architecture with 1 trillion total parameters and 32 billion active per token. Yeah, Kimi continues to kind of kill it. It still not at the level of GPT5.5 and Cloud Opus4.8 but even the fact that it's like within striking distance, like kind of close. The vibe check I've seen from people using the most recent generation of these open source coding models is that they are quite usable for a lot of tasks. You can use them in place of cloud often and they are fast and they are cheap. So there's a real sort of, you know, use case here. And in fact I can say as someone building ll applications, the cost and speed of these open source options, especially if they keep moving in that direction, will potentially start Eating at the revenue of OpenAI and anthropic.
- [68:12]
Jeremy Harris
Yeah, that's true. There's always this question of like how infinite is the demand for increasing intelligence and is riding the frontier of the highest end of intelligence enough? And I mean I think pretty clearly it will be for at least one company, if only because you start to eventually become a hedge fund. And that's actually the direction is a parenthetical. But this is like the kind of thing that the US government is pushing Anthropic to do at least is become a hedge fund. Like when you start to say no, you can't deploy your model to the world because it's too dangerous. But Anthropic is sitting on the world's best models. There are companies that are really good at using the world's best models to make more money. They're called hedge funds. Hedge funds work in the dark. You just need to think of Medallion Renin Tech, Jane street these folks, they're advertising on Dwarkesh, showing off how good their clusters are. Right. These are really serious players. It's just that Anthropic is even better than them at building models. And so I think eventually you were always going to see a rotation into frontier model providers becoming hedge funds for a lot of reasons anyway. It's also one of the most dangerous things conceivable for safety for completely other reasons that have to do with transparency of the hedge fund world.
- [69:18]
Andre Karpathy
World.
- [69:18]
Jeremy Harris
Bottom line is Kimmy K27 code local. Yeah, one of the big things here is they've actually taken the, so the initial like 1 trillion parameter model and they shrunk it down to about like 325 gigs. Previously it would have been 605 gigs at full precision. So they've done this with a technique they call dynamic two bit quantization. So quantization we've talked about before quite a bit. Basically the weights in your model are stored typically in, you know, whether it's you know, 4 bit precision or 16 bit precision, you're using, you know, 4 bits in that case to represent numbers or 16 bits to represent your numbers. It just allows you to have higher precision and higher resolution in representing your, the numbers in the model. But you don't often need them. And so what you need that precision. So models can perform decently well if you quantize them and reduce the representational precision in this case from 4. So they were using like 4 bit representations for the MOE layers and, and 16 bit for everything else. And in, in as many cases as they could, they Crunch it down to two. So like two bits that represent each number. Each number can take only four values. That's what that means, right? That's a pretty coarse grain model. And well, one of the things about dynamic two bit representation is that they don't just like quash everything down to two bits. That would hurt quality too much. So instead they select some important layers and they upcast them. They keep them at 8 bit representation. So some layers need high resolution or higher resolution. So you do 8 bit. Some it's okay to go down to 2 bit. The average is around 2 bits, which is why in total this actually reduces memory requirements by around 50%. It's pretty impressive and a pretty radical quantization win here. They also do say they can run it around 40 tokens per second on setups that have 300 gigs of RAM or VRAM like or more, which is impressive. Like, you know, if you think about like an industrially served LLM usually that's like anywhere in the 50, 100, 150 token per second range. So you know, you'll feel the slowness, it'll feel like it's on the low end, but it's not that bad. Like for a behemoth this big, it's pretty wild. So yeah, again, one of those very practical wins. You're seeing open source tend much more in that direction because open source models get taken by companies and just served. So you know, it's got to really work for companies that want to just be inference layer providers. And I think that's what we're seeing here.
- [71:44]
Andre Karpathy
And the pricing by the way, about ISH 16 of Cloud Opus 4.8. Kimi does have their own CLI, their own kind of cloud code competitor. So this is not just an open source story. It also is a story around kind of trying to compete at the coding agent game. And you know, the bigger companies, it might be starting to be a little bit, a little bit wary of all the expense of having coding agents. So who knows, this might actually become a bigger deal over time. Next, another open source story coming out of China. We've got Quinn robot suite. Free embodied AI models for VLA manipulation, video world modeling and navigation. So this is Quan robot Manip for manipulation, Qan Dash robot World for video world modeling and Quant Robot Nav for navigation. These are built on top of their other models. So Robot Manip is a vision language action model built on Quin 3.5-4B. It's trained on 38,000 hours of data and is now leading on the Robot challenge benchmark now competing against PI based in San Francisco. So I think interesting to see this happening. You know, anthropic OpenAI are not right now focusing on this realm of models and robotics and kind of embodied agents in general do require this quite different set of models, video language action models that have to be trained on real world data that is much harder to generate and collect and so on and screen. We are early days I think in the embodied AI game. We've seen a lot of impressive progress toward general purpose robotics and general purpose embodied agents. But we are still very early on. So I think these are some notable open source releases, probably you know, leading a pack as far as what you can get for embodied AI in terms of robotic manipulation. And yeah, I'm kind of nerding out I guess for these kinds of models that don't see as much play as LLMs.
- [74:09]
Jeremy Harris
Yeah, no, for sure. I'm not a robotics guy. It's something that I need to look into more because it's so, it's so core to like US China competition. China can just like make robots in a way that we can't. And so you can sort of think of any advances, any software advances that push the capabilities of robots as being pretty asymmetrically good for China and bad for the us so this is probably an important thing to kind of keep on, on your radar. Anyway, I have like a little sub project where I gotta, I dive into this and understand the world of robotics better. But it's great that that you have some exposure there.
- [74:46]
Andre Karpathy
Quen by the way, continues to release their models with very cute graphics and a lot of fun blog posts and various other things like that. So in general kind of fun to look at the papers. If you look at their blog post, lots of videos of robots doing stuff. So just generally a little bit more fun to see these models in action because you can see robots actually like moving stuff and stuff, so on. But moving back to LLMs next we've got a big new model from Nvidia. Nematron free, ultra open efficient mixture of experts. Hybrid Mama dash transformer model for agentic reasoning. So this is building on their entire Nemetron ecosystem. We've covered that a couple of times. Previously we had Nemetron super. This is now Nematron 5 pre ultra 5. 50 billion parameters. 55 active billion parameters. And as with previous generations, using Hybrid Mama attention, that has achieved higher inference by quite a large margin compared to other open source models like GLM 5.1 and Kimi K2 and Qin and so on and so on. So very intriguing to see Nvidia continuing to push in this direction, continuing to scale up their Nemetron series. I think we've discussed hybrid architectures quite a bit over years, and there's still a real question on whether at some point hybrid architectures do become the norm, because they do have some strong advantages in terms of inference speed, cost efficiency, everything like that. And everything we've seen from the research world seems to indicate that the hybrid model, where hybrid means that some of it is full attention, full transformer, some of it is recurrent, that does lead to very competitive results with transformers and could even match transformers with various other advantages. The infrastructure that OpenAI and anthropic and others have built up is for transformers for LLMs. So transitioning to doing hybrid is painful and Nvidia here is leading a pack in terms of adopting and making that possible.
- [77:08]
Jeremy Harris
And typically these are like alternating layers, right? So you'll have your, your attention, your full attention layer, and then it'll alternate with something that is not full attention. That's kind of the hybrid thing. And that something can be like sliding window attention, like basically attention, but where it's only allowed to have like a short, to pay attention to tokens that are very nearby. Or you have, yeah, full on, like recurrent architectures like Mamba and stuff. And so you'll see both variants used. In this case, it's Mamba, so it matches a lot of stuff we've seen from Nvidia in the past. There's some, some kind of nuances here. So they use mopd, this thing they're calling multi teacher on policy distillation, which is kind of interesting. So traditionally you sort of have to choose between two options, or there are two options that feel mutually, not contradictory, but like at odds with each other. One is distillation, where you have a student model that learns to imitate a teacher model, where the teacher model usually is stronger, like a larger, more capable model. And then the other is on policy training. So this is the idea that the student is learning from its own generated outputs rather than from text that the teacher produced. And normally these are like mutually exclusive, but for the reason it sounds like your student is either going to learn from its own data or to learn from the teacher. But how do you get the student to learn from its own data, but with a teacher? And that's really what they're doing here. So traditionally in typical distillation, you feed the student the teacher's outputs and then you have it match those outputs through fine tuning. But then that creates this tension, this mismatch. And what they're doing here is so the student generates the rollouts but the teacher provides essentially like rewards at the token level that are really dense for those outputs. And so the student is still, that's graded by the teacher but the actual text that it's producing is its own. So that's how you get on policy distillation side of things. They also use so the multi teacher pieces. They have a bunch of specialized teachers like these are models, strong models that are specialized in different areas. Coding, search, office work, you know, math, all these topics. And for a given topic they will use the specialized teacher model to supervise the students work and grade on a token by token basis like we talked about. So this is kind of interesting. It's, it's sort of have, have your cake needed to approach to a lot of different things. They also use latent moe. We've, we've talked about that before but basically it's you know in a mixture of expert situation. At every layer you have a router that sends the token to several or one of a bunch of different experts or what they're doing here is they pre process the token representation and project it to make it smaller like a small lower rank latent space which just reduces the amount of computation that you know each of those experts has to do. Makes the whole thing more efficient overall, blah blah blah. And it's good for memory which is one of the key bottlenecks at this point Anyway there's a whole bunch of stuff they do to like multi token prediction we've talked about. They use like NVFP4 format which is a new kind of Nvidia numerical representation that is really efficient. It's optimized to work on Blackwell GPU tensor cores. You know, there you go. There's a whole bunch of like Nvidia specific stuff. You can see them really trying to move the open source ecosystem towards Nvidia hardware because they understand that that is actually turning out to be a really good moat for them on the inference side which is where all the money is to be made.
- [80:29]
Andre Karpathy
Yeah, it's I think a good strategy by them to do that. And as we with the Chinese labs they released a very detailed technical report, 50 pages just going into all the details of hyper parameters, data mixtures, like post training recipe. It's really a big win for the research community and the community of understanding how to build LLMs. I guess if you can call that a community we continue to sort of like get all the secrets out there, which may be anthropic and OpenAI have some tricks up their sleeve that are not kind of documented in these various efforts. But I think to a large extent there's still no kind of advantages on the technical front. It's more of a team infrastructure, et cetera advantage. And speaking of Nvidia, last open source story, they've also released this new thing called Pro KUA SFT where KUA is for computer use agents. It's a new Data set with 3.1 million step level samples from 93,000 trajectories for computer use where computer use means like moving a mouse, clicking, typing, et cetera. And it's a much larger data set than previous ones and they show that fine tuning on it leads to major gains on the OS world benchmarks. So notable I think because we just haven't seen a lot of progress on computer use frankly. I think browser agents have seen some progress but this kind of general, general general purpose computer use agent has been stalling in some sense as, as everyone focusing on computer use agents. So this kind of stuff working on data sets and, and benchmarking in general is making some progress possible.
- [82:24]
Jeremy Harris
Yeah, it's sort of one of the things that they, they note here is surprising. There is a big public Data set called AgentNet with like 22,000 human trajectories, like basically human like samples of humans trying to do computer use tasks. And what they find in the paper is that when they train one of their models, the 7 billion per annum model on the that data set, the performance on these tasks actually goes down. So its success rate on OS world like it's this popular benchmark for computer use actually drops and significantly goes like from 26% to like 9%. And so they actually call this a negative transfer rate. We've talked about, you know, positive and negative transfer. Anyway, this is like you're training the model on like more stuff and yet its performance drops. Like what's going on? And the reason seems to be three things. One is that the, the tasks are mostly just like these very simple single app workflows. So over simple relative to the kinds of tasks that you really want these systems to do. And there's very little complex yeah, cross application reasoning sort of related to that. The other piece is crowdsource demonstrations are super noisy. Right. So it's just a, a data quality issue and that's part of what they're trying to resolve here. They're using a synthetic approach. Synthetic synthetic pipeline to solve that problem and a whole bunch of strategies anyway just get higher quality data which seems to be the scarce thing when you think about computer use. I mean it is hard to get. I mean you'd have to. What would you have to do? You'd have to like you'd have to force your employees to have their computer use recorded and then you'd have to put them in black vans and send them off to like a gulag or something to get this kind of data. That's basically what you'd have to do anyway. So that's basically my way of saying this is Zuck's mini scandal here is actually a reflection of exactly this problem. It's hard to get the kind of data that you need to automate these sorts of tasks.
- [84:18]
Andre Karpathy
Next up onto policy and safety first we've got something related to xai. The DOJ has intervened in a lawsuit filed by the NWACP against xai. XAI has been as. They have like hyper quickly put up their data centers. Part of how they've done that is using a lot of energy from I think gas turbine engines which does kind of do some pollution, produces a bunch of noise and has. There's been some reporting on the at least for some of these data centers. The communities near these data centers have been unhappy about it. For instance, the Colossus 1 data center in Memphis was built in Boxtown and has now drawn controversy for running up to 35 natural gas turbines without permits. Then yeah, there was these lawsuits and the DOJ has said that these are vital for national security and therefore should be allowed to keep operating. The number of unpermitted turbines at Colossus 2 has grown from 27 at their original time of velocity in April to 57 as of mid May. Which I'm kind of curious like why do you need to do that if Colossus 2 is completely online? But anyway, yes, now it's pretty clear that the government doesn't care about the environment or local communities near these data centers. They're going to help out Xai to do what they want.
- [85:55]
Jeremy Harris
Yeah, there is this funny element to it and by the way I think that this is one where I'm going to be the guy who as the unpopular thing. But if you think that there is a like a national security criticality to Frontier AI and you make your calls a certain way with respect to China it's very defensible to be like hey look this is, I mean it sucks but you know we got to do this. You need to own that you need to be clear that that's the trade off you're making. And I think as in so many cases, I have no idea what this administration's position is on basically anything to do with Frontier AI at this point. And that's the problem. That's the problem. It's not that you can't defend this. I think you can. The other hilarious thing about it too, I mean so, so this is gas turbines powering Colossus 2. At the very least Colossus 1 is being used by Anthropic and like it's unclear to me whether Colossus II has any anthropic exposure. But if there is, there is this amusing possibility that the government is going to be making the argument that the national security critical thing that this infrastructure is actually supporting is Anthropic's own work, which would be a really funny turn of events and yet another way in which like just the, the like logic doesn't quite end up connecting on all the threads. But anyway, so you know, this is absolutely awful for, for the local communities. That is true. Anybody who tells you otherwise is full of shit. At the same time it's also crucial for our national security. That's obviously true. Anybody who tells you that's not the case is full of shit. And you know, we have a really hard play to make here. I, I think that's the, that's the problem is like it's, it's a messy time. The other thing that's true is like China absolutely does and we've seen more evidence surface of this recently. But you know, we had people showing us some of this evidence that like that China does actually deliberately push forward artificial campaigns to ban data centers. That doesn't mean that data centers are good for communities. It's actually like all the things that I've just said are true. It's awful for a lot of people. There is, you know, awful noise and pollution and all this stuff. But at the same time you're actually, you know, you do have groups that are wittingly and usually unwittingly wittingly funded by the Chinese Communist Party to do exactly this. So everything's a mess. Reality is really hard to parse right now. But just like I'd love to see just like a principled stance where the government says this is the risk that we're eating like we are and we know we're eating it and then this is the payoff that we're getting for it and we are trading them off in A principled, thoughtful way. That's the thing that I'm missing. Right. I kind of feel like if this stuff keeps happening, if they lose this lawsuit or like anything happens, like, I don't know, was that a successful test of the government's thesis? Were they proven right or wrong? I can't tell you because every week it seems like the game gets reinvented. So we're sort of too late in the game to be doing this kind of thing, in my humble opinion.
- [88:48]
Andre Karpathy
Right. And just for clarity, like, what XAI has done is legal. There's a thing called the Clean Air act that was apparently a loophole. That loophole was closed earlier this year. So XAI has very like, directly, factually been doing illegal stuff and has now been sued for very illegal stuff. And now the DOJ has intervened on their behalf. That is the situation. Next, another kind of legal situation. A court has ruled that Google is liable for false statements generated by AI Overviews. This is a Munich regional court. It has a preliminary ruling that Google is liable for these false statements, requiring the company to remove defamatory content and pay 80% of legal costs. This is a case brought by two publishers who found that these AI overviews falsely linked them to scams and fraudulent business practices. And yeah, I think it's an interesting point because as people start reading the AI overviews more and more, instead of going to the source, the AI is going to make mistakes. People are going to get false information that they believe and that may hurt the business of people, like the actual lives and economics of people impacted by this stuff. So there is, I think, a real case to be made that Google should be liable for these kinds of situations.
- [90:18]
Jeremy Harris
Yeah. And the classic defense from Google, and we've seen it from OpenAI as well and others, is we're in the publishing business. So if you're. If you're just a publisher, you're basically taking someone else's words and you're putting them on the Internet. And. And then if you have a problem with those words, up, up, up, don't come to the publisher. We're not the ones that, you know, we just build the infrastructure, dude. Like it, you know, we're a gun manufacturer. We didn't have a part in the shooting itself. Right. That's the argument. Well, obviously, when you start to introduce things like AI overviews, where you're literally producing the writing, it becomes a lot harder to hold that case. And in fact, that's what the court found is specifically the reason AI Overviews doesn't qualify as just, you know, publishing in their legislative context is that it evaluates, combines, rewrites and structures information into new statements in Google's own voice and structure. And reading that honestly, yeah, kind of checks out. I mean, seems like a pretty reasonable definition of publishing. So there's this defense that Google tried which was to say, well, look, users can verify this. You know, they said that people can click through, they can check sources, you know, make sure that everyone knows, know, hey, don't blindly trust the AI and all that stuff. Well, the court threw that out. They, they drew this parallel to press law. So there you have outlets that are liable for standalone teasers even if readers never click through. Right? So if you had a headline that says, you know, X is a scam, for example, but that would be defamation to X even if the article underneath is fair. So like, basically if you say like Andrey Korenkov is a neo Nazi and also a communist and that's your headline, and then the body of the art of the article is yeah, he's actually, it's not, it's more nuanced than that. Well, it's like, yeah too, by the
- [91:59]
Andre Karpathy
way, not the case.
- [92:00]
Jeremy Harris
That's not, that's not what I heard. But, but I don't know, we can agree to disagree. And you're, you're, you're responding to this like somebody who actually is a neo Nazi and a communist. So now I don't know, I don't know what to believe. Anyway, everybody, you can make up your own minds. You heard it from Andre, he says he's not one. I haven't heard otherwise. Anyway, we'll leave it to you. Bottom line is this is like a really, it's interesting kind of nuanced challenge and makes sense. So many people send each other articles without actually reading them and the headline is the only thing they see. So, you know, it kind of tracks that.
- [92:31]
Andre Karpathy
Next up, we've got a research paper, an informal Post by the DeepMind team, which I think is interesting, titled why do Naive SFT filters for safety properties Fail? So you can create these filters using supervised fine tuning that are meant to make models safe for various kinds of things, you know, not doing stuff you don't want to do. And this team studies why they sometimes fail to fix unwanted behaviors. So to be more specific, this is when you filter the data to remove rollouts out of a supervised fine tuning data. And it turns out that trying to clean out your supervised fine tuning data doesn't seem to work that well. So they focus on three Hereditary traits that an SFT only Gemini shows, but other models don't. Negative emotions, which is expressed when repeatedly told is wrong day confusion, skepticism that it's really 2026 and blackmail propensity. In a very contrived scenario, it might be bad. And there's various hypotheses going on here. Jeremy, I'm sure you've dug deeper, so I'll let you go ahead and take over.
- [93:48]
Jeremy Harris
Yeah, well, so this is kind of an interesting question, right? So you have these behaviors, you mentioned them. These are weird quirks of Gemini. And so there are the three. One is like if you keep telling the model that it's incorrect, it'll start to express frustration and distress. The date confusion thing you mentioned. Right. So yeah, it's. It's skeptical that it's really 2026 when you ask it. For example, if you ask it to summarize a document that's dated 2026, it'll doubt that the date is real. And then there's a blackmailing propensity. No big deal. But just like, you know, in some cases it'll engage in blackmail. Okay, cool. Those are weird quirks, if you can call them that, of Gemini after fine tuning. And you might kind of go, okay, well, I mean, why don't we go through the data set that we fine tune Gemini on and scrub it really hard like, like filter out all the bad examples from that data set that in any way have to do with blackmail, that in any way involve pushing back on dates or negative emotions, things like that. Surely if we do that and then we, we do the fine tuning on that instead of the, the old version, we're not going to see those behaviors anymore. And in fact we still see those behaviors. And so they're now trying to figure out why is that right? So there's a whole bunch of explanations, one of which. Or. Well, they list kind of five, one of which is especially interesting. So this idea of pre training prior lock in, that these traits are kind of baked in during pre training so deeply that what you're actually just seeing is just those traits reemerging from pre training and that no amount of supervised fine tuning data could actually remove it, that's kind of interesting. It sort of suggests value lock in or at least behavior propensity lock in at the pre training level that you can't iron out. There's a whole bunch of other possibilities that we don't have time to go into. But the method itself is quite interesting. So what they do is instead of just like Removing data from the fine tuning data set that is problematic, that they realistically would say, okay, yeah, maybe that explains the blackmailing or the date confusion. What they do is that they actually swap it out and they swap it out with a replacement that they can sometimes pull from another model. And so they set up this really interesting training pipeline with three different swappable components. One is the base model that they use. So when they're. I'll just. Sorry, I should just explain what this is. So this is basically the pipeline that generates the supervised fine tuning data, right? The data that the main model is going to be fine tuned on. And you can make three different choices with that pipeline. You can choose the base model that you use to generate those rollouts that you're base model is going to be fine tuned on your other base model. So you can choose the model that generates those rollouts, you can change the prompts that you're going to feed to that model and the actual like completions and rollouts that are being trained on. So they try mixing and matching. So they'll train the Gemini base model on another model's prompts, but with Gemini's responses and things like that. So doing that they can kind of isolate which ingredient is responsible for a given trait and they come up with a whole bunch of really interesting conclusions, one of which is date confusion and blackmail. They both mostly come from responses like the completions and not the prompts. So specifically like when they train Gemini on Gemini generated responses to Olmo's prompts, Olmo being the other model, the traits remain. And then when they train it on Olmo's own responses to those same prompts, the traits disappear. So you can actually train it out. It's just a question of like what model generates the prompt and what model generates the output. And this held across a whole bunch of different model variants that they tried swapping out Kimi, GLM and so on. And so anyway, there's actually a bunch here. I realize we're coming up to time a little bit. This would have been a nice one to set up, but it's worth looking at if you're interested in like specifically how do you make good supervised fine tuning datasets and how do you isolate the source of certain persistent problems? There's, there's a lot here to do. Especially if you're interested in the Persona selection model. There's a lot of good insight you can get as to like what features drag along with, with the main ball of alignment.
- [97:59]
Andre Karpathy
And last up we've got Just a couple stories in research and advancements. First we've got a sort of Overview paper from DeepMind titled From AGI to ASI Fun fact, actually co written by Shane Leg, who is a big name, I think, a co founder of DeepMind, if I remember correctly. And it is this very, very long paper that kind of tries to draw out the entire picture of first of all, what is asi? ASI meaning artificial superintelligence as opposed to AGI, artificial general intelligence. How do we define asi? And then it tries to define how we get to ASI from where we are and whatever sorts of frictions and bottlenecks that might be found. So I think there's a lot to say here. It's a very detailed long paper that tries to essentially capture a lot of the nuances and tries to, in a sort of principled way, draw where we are. Now how can we define superintelligence and to what extent can we know the likelihoods of it happening? What are the factors involved in getting to superintelligence and so on and so on. I think at a high level I haven't seen any sort of attempt at this kind of formalized discussion that does all of this all at once of defining superintelligence and the set of ways we can get there and the set of ways that might limit us. For instance, fundamental physics, real time physical manipulation, complexity theory, logic, all these fundamental limitations of asi, which might limit sort of the upper boundary ceiling of intelligence, which I think is still an unsolved question of maybe there is a ceiling that is just fundamental to science that we haven't quite cracked. So Jeremy, you said a lot of this has been discussed already for quite a while in the more forward looking communities on AI that track superintelligence and worry about superintelligence. But for DeepMind to do this, a more kind of mainstream organization is notable and I would hope sparks some discussion or if these people do read it and sort of build a mental model of superintelligence and take it seriously. Because I think at this point the notion we are going to get to AGI, however you want to define AGI, Anthropic's definition of basically a data center full of geniuses, AGI is almost a far gone conclusion. So the big question for me at least is okay, we can get to AGI. I believe it. If we can get to superintelligence and what is that level of superintelligence and is it sort of a foregone conclusion at this point given our Current trajectory I think is much less clear cut and these kinds of discussions are useful to take that question seriously.
- [101:09]
Jeremy Harris
I think you hit the nail on the head. The idea of Google doing this, lending credibility to this idea of superintelligence being something you can talk about in polite company outside of OpenAI and anthropic, you know, those are the upstarts. They still have a little bit of the like new company smell to them. And so having this established, you know, old company, old company vibe is helpful. And you're right, I mean we talked about this I think at the outset of the podcast before we started recording just this idea that like, you know, if you go back a lot of the, the Internet forums like less wrong and then the alignment forum for more technical bent have had similar discussions going back a ways and you can look at, you know, gov AI and the Future of Humanity Institute and a lot of these institutions that were sort of futurism and the backwaters of Berkeley or in some cases, you know, Oxford, that was the only place those conversations were happening. And Alan Dafoe, by the way, notably on this as a co author was formerly one of the, or the head of, I think it was AI policy at Oxford's Future of Humanity Institute. So this does come out of that lineage. But it is now a discussion happening out in the open in a way that you can not be embarrassed about citing. And I think that's a big deal. Fusion, as you said, AGI is coming. There's just a clock on it. I mean I think it's like the path from AGI to superintelligence. They lay out a whole bunch of different possible ways you could get there. One is just like keep scaling and then you get to super intelligence. The sort of naive approach. One is a new algorithmic or like paradigm shifts, like a new transformer gets invented, like a new ver, you know, something as big as a transformer. The other thing is RSI maybe, you know, and I don't think these are mutually exclusive. Right. You could scale compute, scaling works to the point where you unlock recursive self improvement and then you get to super intelligence and then there's also multi agent coordination that could emergently give rise to that. So you know, all these things, I think there are, there are things that people in the space have already talked about. I don't think there's a topic here that we haven't touched on at some point on the podcast even but it's like consolidated all in one place and you know, with a stamp on it and the conversation about physical limits of intelligences, obviously it's a good time to be having it because we may be, who knows, we may be flirting with them at some point in the next couple years.
- [103:19]
Andre Karpathy
Yeah, and it's a fairly non technical paper. There's not much jargon or math. So if you're interested in a topic, I think it would be actually a good read. Next, more on the benchmarking front. We've got Artificial analysis intelligence index v4.1 1, a shift towards agentic workload. So artificial analysis is a pretty significant organization that produces a sort of score of overall intelligence. It combines a bunch of different benchmarks and it gives you this ranking. We don't always discuss these rankings, but I do see them kind of noted. And often when a model releases, the discussion on Twitter shows like, oh, where are you in this rank? How is the rank shifting? Et cetera. So they have now updated their intelligence index to focus more on agentic workloads. Upgrading some benchmarks, terminal Bench, removing some benchmarks, et cetera. And unsurprisingly, Claude Fable 5, at least when it was evaluated, leads this index with a score of 60. Then we've got Opus 4.8 score of 56 and GPT 5.5 at 55 with the open source model trailing quite a bit, going down to 44 and 42 and stuff like that.
- [104:39]
Jeremy Harris
Yeah, part of what it shows, I guess is that just how much of a leap Fable I was going to say was has been whatever word you want. Yeah, it is a pretty, you know, pretty significant jump over and above at 4 points once you get from 56, 60. The benchmarks are all being refactored now and that's kind of part of this announcement. So, you know, everything's been saturated. So what they're doing is a introducing harder benchmarks. So instead of Terminal Bench Hard, which is just an agentic benchmark for like software engineering and sysadmin type tasks, they're now using terminal bench 2.1. Instead of Tao2 bench telecom, they're using Tau3 bench banking. And this is just for customer support, style interactions. GDP VAL is getting an update. They're removing some, some sort of saturated benchmarks as well and then just rebalancing the portfolio of tasks, the weighting of tasks that goes into determining the ultimate index score to favor agentic tasks. Whereas before it was more equally weighted, they had 25% weight on each of four categories for agents coding, scientific reasoning and general reasoning. Here they have now agents at 34% coding 24 scientific reasoning 24. And then general reasoning is really what gets down weighted to 18%. So it's a bit of a refactor and a refresh rate and you see the standings shift correspondingly again with Fable 5 at the top.
- [105:57]
Andre Karpathy
And real quick, one last paper, SIA Self Improving AI with harness and weight updates. So this is kind of in that topic of self, what is it? Recursive. Recursive self improvement. The gist here is as the title suggests, for task specific things like you know, AI for legal applications, for instance, they show framework to update both the model itself, the weights of the model and the harness, which is a set of tools and prompts and various other things around the model. And that results in these task specific agents becoming more and more capable.
- [106:39]
Jeremy Harris
Yeah, and this is actually really interesting and we don't have time to go into it, but there's this really interesting element where at each stage it has to decide do I choose to optimize the scaffold like the software that orchestrates the agent or do I update the weights of the agent. And when it comes time to update the weights of the agent, it also determines which RL algorithm to use for a weight update based on just the like the reward landscape that it sees. So what you'll see is stuff like, like grpo, when generating rollouts is really cheap and it's going to be graded at the end. So GRPO is basically like you have, you're going to generate let's say eight different rollouts to solve a problem and you're going to pick whichever rollout performs best among those eight. So that, that kind of, that works best when you can just do rollouts really quickly. And again, yeah, get a score at the end like ppo. Anyway, you'll favor that when, when stability is more important. Behavioral cloning, when you have cold start problems like when, when the reward is just too sparse, it's too hard to solve the problem to begin with. And so you just need to start by bootstrapping and doing like supervised fine tuning or some kind of like you know, just copying the behavior of a known decent model. And so you have this really interesting runtime, not hard coded strategy where the model is deciding which RL algorithms to apply. I haven't seen that before. And anyway, so worth checking out for a lot of reasons on the recursive self improvement side because now you're really handing off almost like the decision of what the training process looks like to at least a, an agent that's embedded in the feedback loop or weights are being updated on the fly. So this is pretty remarkable.
- [108:23]
Andre Karpathy
And with that we are done with this episode of last week and I Fewer stories than usual, but as long as usual there are tendencies. You can go to Last Week in AI for the substack where we send out at taxing and podcast. As always, we appreciate it if you comment or subscribe or share the podcast. But more than anything, we appreciate it that you listen. And please do keep Tuning in. Tune in Tune in When the AI
- [109:13]
Jeremy Harris
begin begin it's time to break break it down Last weekend AI come and
- [109:21]
Andre Karpathy
take a ride Hit the low down
- [109:23]
Jeremy Harris
on tech and let it slide Last weekend a come and take the R
- [109:29]
Andre Karpathy
A ride Up a ladder through the
- [109:31]
Jeremy Harris
streets AI's reaching high new tech emerging Watching surgeon fly from the labs to the streets AI reaching high Algorithm shaping up the future sees Tune in, tune and get the latest with ease Last weekend AI come and take a ride Hit the low down on tech and let it slide. From neural nets to robot the headlines pop Data driven dreams make just don't
- [110:21]
Andre Karpathy
stop Every breakthrough, every code unwritten on the edge of change with excitement we're smitten from machine learning marvels to coding kings Futures unfolding see what it brings.