C (4:49)

Yeah, the two models that are released, by the way, are Claude Opus 4 and Claudsonnet 4. Note the slight change yet again in naming convention. So it's no longer Claude 4 Sonnet or Claude 4 Opus. It's now Claude Sonnet 4 Claude Opus 4, which I personally like better. But hey, you do you a lot of really interesting results. Let's start with swebench. Right? So this swebench verified the sort of like software engineering pseudo real world tasks benchmark that really OpenAI polished up, but that was anyway developed a little while ago in the first place. So OpenAI's Codex 1, which I'm old enough to remember a few days ago when that was a big deal, was just for context, hitting about 72%, 72.1% on this benchmark that was like really quite high. In fact, for all of 20 seconds it was soda. Like this was a big deal. Like on whatever it was Tuesday when it dropped and now we're On Friday, no longer a big deal because Sonnet 4 hits 80.2% now, going from 72 to 80%. That is a big, big jump. Right. You think about how much there's not that much more left to go. You've only got 30 more percentage points on the table and they're taking eight of them right there with that one advance. Interestingly, Opus 4 scores 79.4%. So sort of like comparable performance to Sonnet on that one. And we don't have much information on the kind of Opus 4 to Sonnet 4 relationship and how exactly that distillation happened if there was sort of extra training anyway. So that's kind of a another thing that we'll probably be learning a little bit more about in the future. But.

B (6:23)

And this is the numbers, the like upper range with a lot of compute that similar to O3, for instance, from OpenAI. When you let these models go off and work for a while and not kind of a more limited variant.

C (6:42)

Exactly. That's a really good flag. Right. So there's a range with inference time compute with test time compute models where, yeah, you have the kind of like the lower inference time compute budget score, which in this case is around 72, 73%, and then the high inference time compute budget, which is around 80% for both these models, again contrasting with Codex 1, which is sitting at 72.1% in this figure. And they don't actually indicate whether that's low compute mode or high compute mode, which is itself a bit ambiguous. But in any case, it's a big, big leap. And this bears out in the qualitative valuations that a lot of the folks who had early access have been sharing on X. So make of that what you will. All kinds of really interesting things. So they've figured out how to apparently significantly reduce behavior associated with getting the models to use shortcuts or loopholes. Big, big challenge with Codex one. A lot of people have been complaining about this. It's like it's too clever by half. Right. The O3 models have this problem too. They'll sometimes find janky solutions that are a bit dangerously creative, where you're like, no, I didn't mean to have you solve the problem like that. You're sort of cheating here and, and, and other things where they'll tell you they completed a task, but they actually haven't. That's kind of a thing I found a little frustrating with, especially with O3, but. So this model has significantly lower instances of that. Both models, meaning Opus 4 and Sonnet 4 are they say, 65% less likely to engage in this behavior than Sonnet 3.7 on agentic tasks that are particularly susceptible to shortcuts and to loopholes. So this is pretty cool. Another big dimension is the memory performance. So when developers build applications that give Claude local file access, Opus 4 is really good at creating and maintaining memory files to store information. So this is sort of a partial solution to the problem of persistent LLM memory. Right? Like you can only put so much in the context window. These, these models are really good at building like creating memory files, explicit memory files. So not just storing in context, but then retrieving them. So they're just really good at a kind of implicit rag, I guess you could, you could call it. It's not actual rag, it's just they're, they're that good at recall. There are a bunch of features that come with this. As with any big release, it's like this whole smorgasbord of different things and you got to pick and choose what you highlight. We will get into this, but some of the most interesting stuff here is in the Claude 4 system card. And I think, Andre, correct me if I'm wrong, do we have a section to talk about the system card specifically later or is this it?

B (9:21)

I think we can, yeah, get back to it probably in the Advancement section just because there is so much to talk about with Google. So we'll do a bit more of a deep dive later on to get it to technicals, but at a high level I think, you know, as a user of ChatGPT, LLMs, etc. This is a pretty major step forward and in particular on things like cloud code, on kind of the ability to let these LLMs just go off and complete stuff for you. And so moving on for now from Anthropic, next up we are going to be talking about all that news from Google that It came from IO 2025, bunch of stuff to get through. So we're going to try and get through it pretty quick. First up is the AI mode in Google Search. So starting soon, I guess you have a tab in Google Search where you have AI mode, which is essentially like ChatGPT of search. Google has had AI overviews for a while now where if you do, I think at least for some searches you're gonna get this LLM summary of various sources with an answer to your query. AI mode is that, but more in depth. It goes deeper on various sources and you can do follow up questions to it. So very much now along the lines of what Perplexity has been offering, what ChatGPT search has been offering, et cetera. And that is, yeah, I guess really on par. And Google has demonstrated various kind of bits and pieces there where you can do shopping with it. It has charts and graphs, it can do deep search that is able to do looking over hundreds of sources, et cetera.

C (11:29)

Yeah, that kind of tight integration is, I mean, Google kind of has to do it. One of the issues obviously with Google is when you're making hundreds of billions of dollars a year from the search market and you have like 90% of it, it's all downside, right? Like the thing you worry about is what if one day OpenAI like chatgpt just tips over some threshold and it becomes the default choice over search. Not even a huge default choice, just the default choice for 5% more of users. The moment that happens, like Google's market cap actually would drop by more than 5% because that suggests an erosion in the fundamentals of their business. Right. So this is, this is a really big five alarm fire for Google and it's the reason why they're trying to get more aggressive with the inclusion of generative AI in their search function, which is overdue. I think there are a lot of people who are thinking, you know, why did this take so long? I think one thing to keep in mind too is with that kind of massive market share in such a big market comes enormous risk. So yes, it's all fine and dandy for OpenAI to launch ChatGPT and to have it tell people to commit suicide or help people bury dead bodies. Every once in a while people kind of forgive it because it's this upstart. Right. At least they did back in 2022. Whereas with Google, if Google is the one doing that now you have congressional and Senate subpoenas, like people, people want you to come and testify, they're going to grill you. You know, Josh Hawley is going to lay into you hard as he ought to. But that's kind of the problem, right? You're reaching a fundamentally bigger audience that's since equilibrated. So OpenAI is kind of benefiting still from their brand of being kind of swing for the fences. So in some ways the expectations are a bit lower, which is unfair at this point. But Google definitely has inherited that legacy of a big company with a lot of users. So yes, the rollouts will be slower for completely legitimate sort of market reasons. So anyway, I think this is just like really interesting. We'll see. We'll see if this actually takes off. We'll see what impact that has too. On chatgpt. I will say the Google product suite is this sort of unheralded, relatively speaking, unheralded suite of very good generative AI products. I use Gemini all the time. People don't tend to talk about it much. I find that really interesting. I think it's a bit of a failure of marketing on Google's end, which is weird because their platform is so huge. So maybe this is a way for them to kind of solve that problem a little bit.

B (13:49)

Well, we'll touch on possibly the usage being higher than some people. I think there might be a Silicon Valley bubble situation going on here where you're not Silicon Valley, but you're like, you know, spiritually in Silicon Valley in terms of a bubble. Moving right along. Next announcement also from YO was talking about Project Mariner. So this was an experimental project from DeepMind versus V, the equivalent to OpenAI's operator, Amazon's Nova on Propex computer use. It's an agent that can go off and use the Internet and do stuff for you. It can, you know, go to a website, look for tickets to an event, order the tickets, et cetera, et cetera. So Google has improved this with the testing and early feedback and is now going to start opening up to more people. And the access will be gated by this new AI Ultra plan, which is $250 per month, which was introduced also in this slate of announcements. So this $250 per month plan is the one that will give you like all the advanced stuff, all the models, the most compute, et cetera, et cetera. And you'll have Project Mariner as well. And with this update you are going to be able to give Project Mariner up to 10 tasks and it will just go off and do it for you in the background, somewhat confusingly. Also, Google had a demo of Agent Mode, which will be in the Gemini app. And it seems like Agent Mode might just be an interface to Mariner in the Gemini app.

C (15:37)

Maybe.

B (15:37)

I'm not totally sure, but apparently Ultra subscribers will have access to Agent Mode soon as well.

C (15:45)

Yeah, and it's so challenging to kind of, I find, to highlight the things that are fundamentally different about a new release like this just because so often we find ourselves saying like, oh, it's the same as before, except smarter. And that's kind of just true. And that is transformative in and of itself. In this instance. There is one sort of thing, you alluded to it here but just explicitly say it, that the previous versions of Project Mariner were constrained to doing like one task at a time because they would actually run on your browser. And in this case the, the big difference is that because they're running this in parallel on the cloud. Yeah, you can reach that kind of ten or a dozen tasks being run simultaneously. So this is very much a, a difference in kind. Right. This is like many workers in parallel chewing on your stuff. That's a change to the way people, people or you're more of an orchestrator, right. In that universe than a certain sort of a leader of one particular AI.

B (16:39)

It's quite interesting and moving right along. The next thing we'll cover is VO3, which I think from just kind of the wow factor of the announcements is the highest one, I think in terms of impact, probably not the highest one, but in terms of just wow. AI is still somehow blowing our minds. VO3 was the highlight of Google I, O and V. That is because not only is it now producing, you know, just mind blowingly coherent and realistic videos compared to even a year ago, but it is producing both video and audio together and it is doing a pretty good job. So there's been many demonstrations of the sorts of things VEO can do. The ones that kind of impressed me and I think a lot of people is you can make videos that sort of mimic interviews or you know, typical YouTube style content where you go to a conference, for instance, and talk to people and you have people talking to the camera with audio and it just seems pretty real. And it's, yeah, you know, different in kind from video generation we've seen before and coming also with a new tool from Google called Flow to kind of be able to edit together multiple videos as well. So again, yeah, very impressive from Google. And this is also under their AI Ultra plan.

C (18:16)

Yeah, it's funny because they also include a set of benchmarks in their launch website which by the way are sort of, they're sort of hidden. Right. You actually have to click through a thing to see any anyway and, and I always find it interesting to look at these when you've got that wow moment. I, I don't mean to call it quite a chat GPT moment, but for text to video because we don't yet know what the adoption is going to look like. But certainly from an impact standpoint it is a wow moment. When you look at how that translates though, relative to VO2, which again, relatively unheralded. Like not a lot of people talked. They did at the time, but it hasn't really stuck. So 66% win rate. So 2/3 of the time it will beat VO2 on sort of movie gen bench, which is a benchmark that Meta released. It's just a basically about preferences regarding videos. So it, it wins about 2/3 of the time, it loses a quarter of the time and then ties 10% of the time. So it's a pretty like. It looks like a fairly dominant performance, but not, not like a knockout of the sort that you might. It's difficult to go from these numbers to like, oh wow, like this is the impact of it, but it certainly is there. Like when you look at these, it's. It's pretty remarkably good. And this speaks to the consistency as well of those generations. It's not that they can cherry pick necessarily just a few good videos. It does pretty consistently beat out previous versions.

B (19:35)

Right. And they also actually updated VO2. So just demonstration of how crazy this was in terms of announcements. VO2 now can take kind of reference photos. We've seen this with some other updates. So you can give it an image of, you know, a T shirt or a car and it will incorporate that into the video. And all this is folded into this Flow video creation tool. So that has camera controls, it has scene builder where you can edit and extend existing shots. It has this asset management things where you can organize ingredients and prompts. And they also released this thing called Flow tv which is a way to browse people's creations with VO free. So tons of stuff. Now Google is competing more of a Runway and kind of, I guess what OpenAI started doing with Sora when they did release Sora fully, that had some built in editing capabilities. Now VEO isn't just text to video. They have more of a full featured tool to make text to video useful.

C (20:47)

Yeah, and the inclusion of audio too, I think is actually pretty important. You know, it's this other modality, it helps to ground the model more. And I suspect that because of the causal relationship between video and audio, that's actually quite a meaningful thing. Like if they. You know, this is interesting from that whole positive transfer standpoint. Do you get to a point where the models are big enough, they're consuming enough data, that what they learn from one modality leads them to perform better when another modality is added, even though the complexity of the problem space increases? And I suspect that will and probably is already happening, which means we're heading to a world by default with more multimodal video generation. That wouldn't be too surprising, at least.

B (21:26)

And next up, you know, Google, I guess didn't just want to do text to video, so they also did text to image with Imagine4. This is the latest iteration of their, you know, flagship text to image model. As we've seen with Text to Image it is even more realistic and good at following prompts and good at text. They're highlighting really tiny things like the ability to do detailed fabrics and fur or on animals. And also they apparently paid attention to generation of text and typography, saying that this can be useful for slides and invitations and other things. So rolling out as well for their tool suite. And last thing to mention about is they also say this will be faster than Imagine Free. The plan is to make it apparently up to 10 times faster than Imagine 3.

C (22:30)

Yeah, and it is. It's unclear because we're talking about a product rather than a model per se. It's unclear whether that's because there's a compute cluster that's going to come online that's going to allow them to just crunch the crunch through the images faster or that there's an actual algorithmic advance that makes it say, 10 times more compute efficient or whatever. So always hard to know with these things. It's probably some mix of both. But interesting that. Yeah, I mean I'm at the point where I, it's like flying on instruments. I feel like I can't tell the difference between these different image generation models. Admittedly, these photos look super impressive, don't get me wrong, but I just, I can't tell the incremental difference. And, and so I just end up looking at like, yeah, how much per token, like, or how much per image. So, you know, the, the price and the latency are both collapsing pretty quickly.

B (23:15)

And moving right along. We just got a couple more things. We won't even be covering all of the announcements from Google. This is just a selection that I thought made sense to highlight. Next one is Google Meet is getting real time speech translation. So Google Meet is the video meeting offering from Google, similar to Zoom or other ones like that. And yeah, pretty much now you'll be able to have almost real time translation. So it's similar to having a real time translator for like a press conference or something. When you start speaking, it will start translating to the paired language within a few seconds, kind of following on you. And they're starting to roll this out to consumer AI subscribers initially only supporting English and Spanish and they're saying they'll be adding Italian, German and Portuguese in the coming weeks. So something I've sort of been waiting on honestly, I've been thinking we should have real time AI power translation that is very kind of sophisticated and powerful and now it's starting to get rolled out.

C (24:25)

Yeah, I personally thought people who spoke languages other than English were just saying complete gibberish up until now. So this is a real shock that. Yeah, no, but it's kind of funny, right? This is another one of those things where you hit a point of where latency crosses that critical threshold and that becomes the magic unlock. Like a model that takes even 10 seconds to produce a translation is basically useless because it's at least this really awkward conversation, at least for the purpose of Google Meet. So another case where it did take Google a little while, as you pointed out, but the risk is so high if you mistranslate stuff and start an argument or whatever, that's a real thing. And they're deploying it again across so many, so many video chats because of their reach that that's going to have to be part of the corporate calculus here.

B (25:09)

Right. And this is a thing we are not going to be going into detail on, but Google did unveil a demo of their smart glasses and that's notable I think because Meta has their smart glasses and they have real time translation. So if you go to a foreign country, right. You can kind of have your in ear translator and I wouldn't be surprised if that is a plan as well for this stuff. But last thing to mention for Google, not one of the highlights but something I think notable as we'll see compared to other things. Google also announced a new Jewels AI agent that is meant to automatically fix coding errors for developers. So this is something you can use on GitHub and it very much is like GitHub copilot you will be able to task it with working with you on your code repository. Apparently it's going to be coming out soon. So this is just announced and yeah, it will kind of make plans, modify files and prepare pull requests for you to review in your coding projects.

C (26:24)

Yeah and like literally every single product announcement like this, they have Google saying that Jules is in early development and quotes may make mistakes which anyway, I think we'll be saying that until we hit super intelligence just because you know, the hallucinations are such a persistent thing. But there you have it, right?

B (26:42)

And next story actually is directly related to that. It's that GitHub has announced the new AI coding agent. So GitHub Copilot has been around for a while. You could task it with reviewing your code on a pull request, on a request to modify a code base. Google also had the ability to integrate Gemini for reviewing code. So Microsoft very much kind of competing directly with Jules and Codex as well with an offering of an agent that you can task to go off and edit code and prepare pull request. So just part of an interesting trend of all the companies very rapidly pushing in direction of coding agents and agents more broadly than they had previously.

C (27:35)

Yeah, this is also notable because Microsoft and OpenAI obviously are in competition in this frenemy thing. Copilot was the first, apart from OpenAI's codecs was the first sort of large scale deployment at least of a coding autocomplete back in like I want to say 2020, 2021, even just after GPT3. And yeah, so they're continuing that tradition in this case kind of being fast followers too, which is interesting. Like they're not quite first at the game anymore, which is something to note because that's a big change.

B (28:06)

One small thing worth noting also they did announce open sourcing of GitHub copilot for VS code. So this is like a nerdy detail but you have also competition from Cursor and these other kind of alternative development environments with a company behind Cursor now being valued at billions and billions of dollars and that is a direct competitor to Microsoft Visual studio code with GitHub copilot. So them open sourcing with GitHub copilot extension to Visual Studio code is kind of an interesting move and I think they are trying to compete against these startups that are starting to dominate in that space. And just one more thing to throw in here. I figure we're flagging because of its relation to this trend. Mistral, the French company that is trying to compete with OpenAI and Anthropic, has announced Devstral, a new AI model focusing on coding and this is being released under NFT Apache to license and is competing with things like Gemma 3 27B. So like a mid range coding model. And yeah, Mistral also working on a larger agentic coding model to be released soon. Apparently with this being the smaller model that isn't quite that good. This is also following up on codestral, which was more restrictively licensed compared to devstrawl. So there you go. Everyone is getting into coding more than they have before.

C (29:50)

You get an agent and you get.

B (29:52)

An agent and onto applications and business we have first up, not the most important story, but I think the most kind of interesting or weird story to talk about, which is this OpenAI announcement of them fully acquiring a startup from is it Jony I've.

C (30:14)

Yeah Jony I've.

B (30:15)

Yeah Johnny I've yes. Who seemingly has had the startup IO that was. The details here are quite strange to me. So there's a startup that Joni I've started with Sam Altman seemingly two years ago that we don't know anything about or what is done. OpenAI has already owned 23% of this startup and is now going on a full equity acquisition that they're saying they're paying $5 billion for this IO company and that's a company with 55 employees that again at least I haven't seen anything out of and this is they're saying like the employees will come over. Jony I've will still be working at Love from which is his design company broadly speaking which has designed various things. So Jony I've not a full time employee at OpenAI or IO still sort of like a part time contributor collaborator. And to top off all these various kind of weird details this came with an announcement video of Sam Altman and Johnny I've walking through San Francisco meeting up in a coffee shop having like a 8 minute conversation on values and AI and their collaboration that just had a very, very strange vibe to it that was you know trying to make this very artsy I guess feel to it. They also released a blog post called Joanie and Sam. Anyway, I just don't understand VPR aspects of this with business aspects of this. All of this is weird to me.

C (32:15)

It almost reads like a landing page that Jony I've designed like to announce it. It's very like kind of sleek, simple Apple style almost one might say very.

B (32:24)

Similar to actually Love from their website has the same sty. Yeah the blog post is like this minimalist centered text, large text and the headline is Johnny and Sam. I think it's.

C (32:38)

I'm just going to say it's weird. So this blog post they're talking about the origin story of this. I think the news reports around the time that IO was first launched said or called it like Sam Altman and Johnny I've's new startup and the implication was that this was a company being co founded by Sam and Johnny together or something. That's clearly not the case, at least according to what they said they imply. They say something like it was born out of the friendship between Johnny and Sam which is very ambiguous. But the company itself was founded about a year ago by Jony I've along with Scott Cannon who's an alum at Apple, and then Tang Tan and Evans Hanke. Evans. Evans Hanke actually took over Johnny's role at Apple after Johnny departed. So they're tight there. A lot of shared history, but none of the actual co founders are. Sam OpenAI already owns 23% of the company, so they're only having to pay 5 billion out of the total valuation of 6.4 billion to acquire the company. And then, as you said, somehow, out of all this, Johnny ends up still being a free ish agent to work at Lovefront. That, by the way, highly, highly unusual to acquire a company even at a $6 billion scale and to let one of the core, arguably the most important co founder just leave. Like, this is normally not how this goes. Usually famously, with like the WhatsApp acquisition by Facebook, it was like a. I forget what. It was like a $5 billion acquisition, but the founder of WhatsApp left Facebook early and so he was on an equity vesting schedule, so most of his shares just vanished and he didn't actually get the money that he was entitled to if he'd stuck around. So the common thing, weird, that Johnny gets to just leave and fuck off. And like, apparently, I don't know if he's still getting his money from this or. It's so weird. This is like very esoteric kind of deal, it seems. But bottom line is they're working on a bunch of hardware things. OpenAI has hired the former head of Meta's Orion augmented reality glasses initiative. That was back in November, and that's to head up its robotics and consumer hardware work. So there's a bunch of stuff going on at OpenAI. This presumably folds into that hardware story. We don't have much information, but there's presumably some magic device that is not a phone that they're working on together. And who the hell knows, right?

B (35:01)

So this announcement, which is very short, like, I don't know, like maybe nine paragraphs, concludes with saying, As I O merges with OpenAI, Johnny and love Frump will assume deep design and creative responsibilities across OpenAI. And I O not like a strong commitment and as you said, free agent.

C (35:22)

Like what?

B (35:23)

Deep design and creative responsibilities. And yeah, IO was seemingly working on a new hardware product, as you said, like a hardware interface for AI, similar to the Humane AI product PIN and Rabbit R1, famously huge failures. Very interesting to see if they're still hopeful that they can make this AI computer, or whatever you want to call it, AI interface within OpenAI. And with Joanie, I've but anyways, yeah, just such strange vibes out of this announcement and this video and the business story around this.

C (36:06)

Can an announcement have code smell? Because I feel like that's what this is.

B (36:10)

And moving on to something that isn't so strange, we have details about OpenAI's planned data center in Abu Dhabi. So they're saying that they are going to develop a massive 5 gigawatt data center in Abu Dhabi, which would be one of the largest AI infrastructures globally. So this would span 10 square miles and be done in collaboration with G42 and would be part of OpenAI's Stargate project, which I'm kind of losing track. Is OpenAI's Stargate project just like all their data centers, where they might want to put them? And this is coming after, you know, of course, Trump's tour in the Middle east with G42. Having said that, they're going to cut ties, divest their stakes in entities like Huawei and the Beijing Genomics Institute.

C (37:09)

This is pretty wild from a national security standpoint. It is not unrelated to the deals that we saw Trump cut with the UAE and Saudi Arabia, I won't say last week or the week before. So for context, OpenAI's first Stargate campus in Abilene, Texas, which we've talked about a lot, that's expected to reach 1.2 gigawatts, really, really hard to find a spare gigawatt of power on the US grid. That's one of the big reasons why America's turning to the Saudis, the Emiratis and so on, and the Qataris to find energy on these kind of energy rich nations grids. And so when you look at 5 gigawatts, that's, you know, five times bigger than what is being built right now in Abilene. That would make this by far the largest structure, or, sorry, the largest cluster that OpenAI is contemplating so far. It also means that it would be based in foreign soil, on the soil of a country that the US has a complicated past with. And just based on the work that we've done on securing builds and data centers, I can tell you that it is extraordinarily difficult to actually secure something when you can't control the physical land that that thing is going to be built on ahead of time. So when that is the case, you have a security issue to start with that is prima facie not an option. When you're building in the UAE for a variety of reasons, you may tell yourself the story that you're controlling that environment, but you cannot and will not in practice. And so from a national security standpoint, I mean, I would really hope the administration is tracking this very closely and that they're bringing in, you know, the special operations, the intel folks, including from the private sector, who really know what they're doing. I gotta say the current builds, including from the Stargate family so far are. The level of security is not impressive. I've heard a lot of private reports that are non public that make it very clear that that's the case. And so this is a really, really big issue. Like we got to figure out how to secure these. There are ways to do it and ways not to do it. But OpenAI so far has not been impressive in how seriously they've been taking the security story. But they've been talking a big game. But the actual on the ground realities seem to be, seem to be quite different again just based on what we've been hearing. So really interesting question. Are we going to have this build go up? Is it going to be effective from a national security standpoint and what's it going to take to secure this? Yeah, anyway, all part of that G42 backstory that we've been tracking for a long time between Microsoft and OpenAI and the United States and all that jazz.

B (39:35)

Yeah. And it seems like with Trump in office, there's definitely set to be a major deepening of ties and OpenAI is OpenAI, Microsoft. Other tech companies seem happy to jump on board with that move. And yeah, as you said, there's been kind of a lot of investments going around from that region into things like OpenAI. So makes some sense.

C (40:02)

It's worth it if you can secure it. Like this takes immense pressure off the US electric grid. Right. Like we're not going to just build or find 5 gigawatts like tomorrow that takes. We actually don't know how to build nuclear reactors in less than a decade in America. So it's a really good option. Saudi capital, UAE capital, those are great things if you know, they don't come with information rights or whatever. But yeah, this is like, if you, if you want to like get the fruits of the of sort of Saudi and UAE energy, you got to make sure that you understand how to secure the supply chain around these things because.

B (40:36)

Yeah, well, with the billions of dollars this will surely cost, you would hope they put in a little bit of effort.

C (40:44)

You'd be surprised. Yeah, security is expensive and it's actually like it can't necessarily be bought for money because the teams that actually know how to secure these sites to the point where they are robust to for example like Chinese or Russian nation state attacks are extremely rare. And it's literally like a couple guys at like Seal Team 6 and Delta Force and the agencies and, and like yeah, their demands on their time are extreme and you probably can't network your way to them unless you have a trusted way to get there. So like it's a really tough problem.

B (41:18)

Well, onto the next story I think another sort of weird, almost funny story to me that I thought worth covering. LM arena, which had the famous AI leaderboards that often have been covered. We covered it just I think a few weeks ago there was a big controversy around seemingly the big commercial players gaming the arena to get ahead of open source competition. That organization has announced $100 million in a seed funding round led by A16Z and UC Investments. So this is going to value them at something like $600 million. And this is coming after them having been supported by grants and donations. So a hundred, like I don't understand what is the promise here for this leaderboard company organization? Is this just charity? Anyway, it's very strange to me.

C (42:25)

I would love to see that slide deck, that pitch deck. There's a lot here that's interesting to say the least. So one thing to note by the way, is they raised. It's a hundred million dollar seed round. This is not a priced round. So for context, when you raise a seed round you're oh man, this gets into unnecessary detail, but basically it's a way of avoiding putting a real valuation on your company if you raise it with, with safes. Usually the whole thing with a seed round is you don't give away a board seat. Whereas if you raise a series A or series B, you're starting to give away board seats. So, so this implies that they have a lot of leverage. Like if you're raising $100 million and you're calling it a seed round, you're basically saying like, yeah, we'll take that money, you'll get your equity. But don't even think about getting a board seat. That's kind of the frame here. You can only do that typically when you have a lot of leverage. Which again brings us back to your very, I think very good and fundamental question. What is the profit story here? And like, I have no idea. But it's notable that like LM arena has been accused of helping top AI labs game its leaderboard and they've denied that. But when you think about like okay, how could a structure like this be monetized? Well, maybe showing some kind of not overt preference, but subtle preference for or indirect preference for certain labs. Like, I don't, I don't know, I'm speculating and this should not be taken like we. I just don't see any information on exactly what the profit play is, which kind of makes me intrinsically skeptical and. Yeah, well, we'll see where this goes. But again, there's a lot of leverage here. There's got to be a profit story. It's being led by a 16Z. So, you know, there's a there there presumably.

B (44:08)

Yeah. Apparently it cost a few million dollars to run the platform and they do need to do the compute to compare the these chatbots. So the idea here is you get two generations, two outputs for a given input and people vote on which one they prefer. So it is costly in that sense and it does require you to pay for the inference. And what at least has been said is this funding will be used to grow LM arena and hire more people and pay for costs such as the compute required to run this stuff. So, yeah, basically saying that they are going to scale it up and grow it to something that supports the community and helps people learn from human preferences. Nothing related to how this 100 million will be something that the investors will get a return on, but could be.

C (45:09)

A data play, like kind of scale AI. You've got some data labeling that's cool there. I just like. Yeah, I'd love to see that deck.

B (45:20)

Yep. And next up, going back to hardware. Nvidia CEO has said that the next chip after the H20 for China will not be from the Hopper series. So this is just a kind of small remark. And it's notable because previously it was reported that Nvidia planned to release a downgraded version of the H20 chip for China in the next two months. This is announced amidst a transition in US policy as to restrictions on chips. And after the sale of these H20 chips designed specifically for China was banned only a few months ago, it looks like Nvidia is kind of having to change their plans and adapt quite rapidly.

C (46:13)

Yeah, it seems like they will be pulling from the Blackwell line. This makes sense. Jensen's quote here is, it's not Hopper because it's not possible to modify Hopper anymore. So they've sort of moved their supply chains over onto Blackwell. No surprise there. And they've sort of squeezed all the juice they can out of the Hopper.

B (46:31)

Platform and presumably sold out of their stock when it was announced that they Couldn't do that anymore. Next up, I put this in the business section just so that we could move on from Google for a little bit. Was announced that Google Gemini AI app has 400 million monthly active users, apparently which is approaching the scale of ChatGPT. Apparently that had 600 million monthly active users as of March. So yeah, as I previewed I guess seems very surprising to me because Gemini as a chatbot hasn't seemed to be particularly competitive with offerings like ChatGPT and Claw Pod and haven't seen many people be big fans of Gemini or the Gemini app. But according to this announcement, lots of people are using it.

C (47:26)

Yeah, and apparently so. So the comparable here, there are recent court filings where Google estimated in March that ChatGPT had around 600 million monthly active users. So you know, this is like two thirds of where ChatGPT was back in March. So you know, to the extent that, you know, ChatGPT and OpenAI are encroaching on Google's territory, well, Google's starting to do the same. So yeah, this is all obviously a competition as well for data as much as for money in the form of subscriptions. So this is all self licking ice cream cones, if you will, or flywheels that both these companies are trying to get going.

B (48:02)

Right. And I think also part of a broader story, this whole thing with Google I O 2025 and this announcement as well I think demonstrates that over the last few months really Google has had a real shift in fortune in terms of their place in the AI race and competition basically until like 2025, they've seemed to be surprisingly behind. Gemini was surprisingly bad, even though numbers looked pretty good and their web offerings in terms of search lagged Behind Perplexity and ChatGPT, you search then Gemini 2.5 was updated or released I think in late January and kind of blew everyone away with how good it was. Gemini 2.5 and Gemini flash have continued to be updated and continued to impress people. And now all this stuff would be a free imagine 4 the agents. All these like 10 different announcements really position Google as I think for many people in the space, looking at who is in the lead or who is killing it. Google is killing it right now.

C (49:14)

They are. And this is, you know, we've talked about this before. Google being the sleeping giant, right, with this massive, massive pool of compute available to them. They were the first to, I mean there's the first to recognize scaling in the sense that OpenAI did with GPT2 and then GPT3. But then there's the first to recognize, let's just say the need for distributed computing infrastructure in a more abstract sense. And that was certainly Google. They invented the TPU explicitly because they saw where the wind was blowing and then now they have these massive TPU fleets and a whole integrated supply chain for them. You know, OpenAI really woke the dragon when they, when they went toe to toe with Google via ChatGPT and Microsoft. And so yeah, I mean to some degree this is not, to some degree entirely. This is the reason why you're seeing that, you know, 5 gigawatt UAE build, that's that OpenAI's going to build. They need to be able to compete on a flop per flop basis with Google. If they can't, they're done. Right. This is kind of just how the story ends. So that's why all the capex is being spent. Anyway, just these announcements that we're seeing today are the product of CapEx that goes back, you know, two years, like breaking ground on data centers two years ago and making sure chip supply chains are ready three years ago and designing the chips and all that stuff. So you know, this is really a long time in the making every time you see a big rollout like this.

B (50:37)

Yeah, and not just the infrastructure, I mean having DeepMind, having Google AI, you know, Google was the first company to really go in on AI in a big way, spending, you know, billions of dollars on DeepMind for many years as just a pure R and D play. Microsoft later also started investing more in Meta and so on. But yeah, Google has been around for a while in research and that's why it was to a large extent kind of surprising how lagging they were on the product front. And now seemingly they're catching up. And just one more thing to cover. In the section we have a bit of a analysis on the state of AI servers in 2025 versus something. Jeremy, you linked to just on X, so I think I'll just let you cover this one.

C (51:33)

Yeah, it's sort of like a random assortment of, of excerpts or take homes from this big JP Morgan report on AI servers from their Asia Pacific Equity research branch. And there's just like a bunch of, a bunch of little kind of odd tidbits. We won't spend much time on this because we gotta go, man. There's more news just looking at the mismatch between for example, packaging production, so TSMC's ability to produce wafers of packaged chips and then downstream GPU module assembly and how that compares to GPU demand and they're just kind of flagging this interesting mismatch where it seems like there's about a 1 million or 1.1 million GPU unit over supply currently expected heading into anyway the next few quarters. Which is really interesting given where things were at just like two years ago, right? That massive, massive shortage that saw prices skyrocket. So you know, kind of curious to see what that does to margins in the space. This is all because of Nvidia inventory buildup basically. Like there's a whole bunch of, of excess there anyway and there were some yield issues and things like that that are being fixed anyway. They've got interesting, interesting numbers about the whole space capex increasing across the board from these massive cloud companies by like pretty wild amounts and in particular ASIC shipments. So basically AI chip shipments projected to go up 40% year over year, which is huge. I mean it's like that's a lot more chips in the world than there were last year. And keep in mind those chips are also much more performant than they were before. So it's 40% year over year growth on a per chip basis. But on a per flop basis, a per compute basis, it's even more than that. You know, we may be like doubling the amount of compute or actually more that there is in the world based on this. Anyway, you can check it out if you're, if you're a nerd for these things and you want to see what's happened to Amazon's Trainium 2 demand. It's up 70% by the way, which is insane and a bunch of other cool things. So check it out if you're like sort of like finance and compute nerd, because this is just gonna, just gonna.

B (53:44)

Be your, your weekend read onto the next section. Projects and open source. We just have one story here I guess to try and save time because there is a lot more after. And the story is pretty simple. Meta is delaying the rollout of the biggest version of Llama. So when they announced llama 4 they also were previewing what Llama for Behemoth, their large variant of Llama 4 that is meant to be competitive with, you know, ChatGPT and Quad and basically the Frontier models. So it seems like according to sources that they initially planned to release this Behemoth in April. That was later pushed to June and it has now been pushed again until at least fall. So this is all kind of internal. They never committed to anything. But it seems like per kind of the reports and general, I think things that are coming out that Meta is struggling with training this model to be as good as they want it to be.

C (54:55)

Yeah, I think this is actually a really bad sign for Meta because also they have a really big compute fleet. Right. They have huge amounts of CapEx that they've poured into AI compute specifically. And what this shows is that they now have consistently struggled to make good use of that capex. They have been consistently pumping out these pretty mid models, unremarkable. And then to make up for that, gaming them, to make them look more impressive than they are in a context where Deepseek is eating their lunch. Both from a marketing branding standpoint and also just raw performance and compute efficiency. And so yeah, this is really bad. The whole reason that Meta turned to open source, it was never because they thought that they were going to somehow open source AGI. That was never going to happen. Anybody who has AGI locks it down and uses it to bet on the stock market and then funds the next generation of scaling and that shit and then obviously automates AI research. It was eventually going to get locked down. This was always a recruitment play for Meta and there were some other ancillary infrastructure things getting people to build a month build on their platforms and that. But the biggest thing absolutely was always recruitment and now with that story just falling flat on its face, it's really difficult. Like if you want to work at the best open source AI lab, A, like unfortunately it looks like right now there are Chinese labs that are absolutely in the mix, but B, there are a lot of interesting players who seem to be doing a better job on a kind of per flop basis over here you look at even Allen AI, right, they're putting out some really impressive models themselves. You've got a lot of really anyway impressive open source players who are not meta. So I think like Zuck is in a real bind and they're doing a lot of damage control these days.

B (56:44)

Yeah, and I think this speaks to like Meta has really good talent. They have been publishing just fantastic work for many, many years. But my sense is that the skills and experience and knowledge needed to train a massive, massive LLM model is very different and the competition for that talent is just immense. Xai, when it came out I think was seemingly providing just really, really big packages to try and get people with experience in that anthropic has had very high retention of their talent. I think I saw a number somewhere like 80% retention. We've seen people leaving from Google to go do their own startups. So I think Meta, presumably that's part of the problem Here is this is a pretty specialized skill set and knowledge and they've been able to train good LLMs. But to really get to a frontier is not as simple as maybe you know, just scaling onto research and advancements. And we begin with not a paper and not a very detailed kind of advancement, but a notable one. And this is also from Google, so sort of under the radar. Just as a little research announcement and demo, they did announce Gemini diffusion. And this is the kind of demonstration of doing language modeling via diffusion instead of autoregression. So typically any chatbot you use these days essentially is generating one token at a time, left to right, start to finish. You know, it picks one word, then it picks the next, and it picks the next. And we I think recently covered efforts to move that to the diffusion paradigm where you basically generate everything all at once. So you start with all the text and some messy kind of initial state and then you update it to do better. And the benefit of that is you can do be just way, way, way, way faster compared to generating one word or one token at of the the end time. So DeepMind has come out with a demonstration of diffusion for Gemini for coding that seems to be pretty good. Seems to be comparable with Gemini 2 flashlight, the smaller kind of not quite as powerful fast model. And they are claiming speeds of about 1500 tokens per second with very low initial latency. So something roughly on a scale of 10 times faster than GPT 4.1 for example, just lighting fast speeds. Not much more details here. You can get access to the demo, signing up for a wait list and yeah, if they can push this forward, if they can actually make diffusion be as performant as just out aggressive generation at the frontier, really really big deal.

C (60:09)

Yeah, and diffusion. So conceptually diffusion is quite useful from a parallelization standpoint. It's got properties that allow you to parallelize just in more efficient ways than transformers. Potentially one of the consequences of that. They show a case where the model generates 2,000 tokens per second of effective, effective kind of token rate generation, which is pretty, pretty wild. It means you're almost doing like instant generation of chunks of code. There's to kind of give you a sense for why this would matter. There's a certain kind of sometimes known as like non causal reasoning that these models can do that your traditional autoregressive transformers can't. So an example is you can say like solve this math problem first give me the answer and then after that walk me through the solution. Right? So give the answer first, then give the solution that's really, really hard for standard autoregressive models because what they want to do is spend their compute first, spend their inference time compute, generating a bunch of tokens to reason through the answer and then give you the answer. But they can't. They're being asked to generate the solution right away and only generate the. The sort of derivation after. Whereas with diffusion models, they're seeing, they're generating the whole thing all at once. They're seeing the whole canvas all at once. And so they can start by having, you know, a crappy solution in the first cycle of generation and a crappy derivation, but as they modify their derivation, they modify the solution, blah, and then eventually they get, you know, the right answer on the whole. So this may seem like a pretty niche thing, but it can matter in certain sort of specific settings where a certain kind of causality is at play and you're trying to solve certain problems. And just generally it's good to have other architectures in the mix because if nothing else, you could do like a kind of mixture of models where you have some models that are better at solving some problems than others. And this gives you an architecture that's a bit more robust for some problems.

B (62:12)

Right. And like, intuitively, you know, you're so used to, when you're using ChatGPT or these LLMs to this paradigm of like you enter something and then you see the text kind of pop in and you almost are reading it as it is being generated. With diffusion, what happens is like all the text kind of just shows up, it's near real time. And that is a real kind of qualitative difference where it's no longer, you know, waiting for it to complete as you're going. It's more like you enter something and you get the output almost immediately, which is kind of bonkers if you think it can be made to work anywhere near as well as just the autoregression paradigm. But not many details here on the research side of this. Hopefully they'll release more because so far we haven't seen very successful demonstrations of it. And moving on to an actual paper, we have chain of model learning from. For language model. So the idea here is you can incorporate what they are saying as hierarchical hidden state chains within transformer architectures. So what that means is you can. Hidden states in neural nets is basically just the soup of numbers in between your input and output. So you take your input, it goes through a bunch of neural computing units and generates all these intermediate representations from the beginning to the end and keep updating until you generate the output. So the gist of a paper is that if you structure that hidden state hierarchically and have these chains that are processed at different levels, with different levels of granularity and with different levels of model complexity and performance, you can be more efficient, you can use your compute in more dynamic and more kind of flexible ways. So that's, I think, the gist of this. And I haven't looked into this deeper sort of. Jeremy, maybe you can offer more details.

C (64:28)

Sure. I think this is kind of a banger of a paper. It's also frustrating that this is, I mean, this is a, a multimodal podcast. We have video, but we don't like, you know, there's like an image in the paper that makes it make a lot of sense. It's figure two that just sort of shows the architecture here. But high level, you can imagine a neural network has, you know, layers of neurons that are, you know, stacked on top of each other. And typically the, you know, the neurons from the first layer are each one of them is connected to each neuron in the second layer and each neuron in the second layer is connected to each neuron in the third layer and so on. So you kind of have this dense mesh of neurons that are linked together. So there's a width, right, the number of neurons per layer and there's a depth, which is the number of layers to the network. In this case, what they're going to do is they're kind of going to have a slice, a very small, narrow width slice of this network, and they're going to essentially make that the backbone of the network. So let's imagine there's like, you know, two, two neurons in each layer, and the two neurons from layer one are connected to the two neurons from layer two and layer three and so on. And the two neurons, say, at layer two, can only take input from the two neurons at layer one. They can't see any of the other neurons at layer one. That then becomes this like pretty cordoned off structure within a structure. So if you have like a larger number of neurons in each layer that are only connected to the additional, anyway, sets of neurons at each layer, hopefully you can just check out the figure and see it. You can kind of see how this allows you to kind of increase the size. You can run your model in larger mode either by only using the thin slice of, say, two neurons that we talked about, or by considering a wider slice, four neurons or eight or 16 or whatever. And so what they do is they find a way to train this model such that they are training at the same time all these kind of smaller sub models, these thinner sub models, so that once you finish training, it costs you the same amount basically to train these models, but you end up for free with a bunch of smaller models that you can use for inference. And the other thing is, because of the way they do this, the way they engineer the loss function is such that the smaller slices of the model, they have to be able to independently solve the problem. So the thinnest slice of your model has to be able to make decent predictions all on its own. But then if you add the next couple of neurons in each layer to your model and get the slightly wider version, that model is going to perform a little bit better because it's got more scale, but it also has to be able to independently solve your problem. And so it ends up, those extra neurons end up specializing in kind of refining the answer that your first thinner model gives you. So there's this sort of idea where you can gradually control, you can tune the width of your model or effectively the level of capacity that your model has dynamically at will. And from an almost interpretability standpoint, it's quite interesting because it means that the neurons from that thinnest slice of your network that's still supposed to be able to operate coherently and solve problems independently, those neurons alone must be kind of focused on more foundational basic concepts that generalize a lot. And then the neurons that you're adding to the side of them are more and more specialized as you add onto them. They're going to allow the model to perform better when they're included. But excluding them still results in a functional model. So there's a lot of detail to get into in the paper we don't have time for. But I highly recommend taking a look at it. I wouldn't be surprised if something like this ends up becoming fairly important. It just, it smells of good research taste, at least to me. It is a Chinese lab that came out with it, which is quite interesting. But in any case, check it out. Highly recommend. It's. Yeah, it's a cool paper.

B (68:25)

Yeah. Actually a collaboration between Microsoft Research and Fudan University, several others, but they did open source or say they will open source. The code for this stuff and paper. It's kind of funny. It produce a lot of terms. They're like, here's the notion of chain of representation, which leads into chain of layer, which leads into chain of model, which leads into chain of language model with the idea that these kind of cumulatively lead up to the notion that when you train a single large model, it contains these sort of sub models. And it is quite elegant, as you say. Now that I've taken a bit of a deeper look, next paper is seek in the dark reasoning via test time instance level policy gradient in latent space. So the idea and a problem here is a variant of test time compute, where you want to be able to do better for a given input, that by leveraging computation at test time rather than train time, you're not updating your parameters at all, but you're still able to do better. And the idea of how this is done here is sort of mimicking prompt engineering. So you're tweaking the representations of the input for the model, but instead of actually literally tweaking the prompt for a given input, it's tweaking the representations within the model. So they are using every reward function to update the token wise latent representations in the process of decoding and that they show can be used to, for given input, improve performance quite a bit. So they're kind of optimizing the internal computations in an indirect way. That is yet another way to be able to scale at test time. Quite different from for instance, chain of thought.

C (70:36)

Yeah, so that was actually really good. I never thought of this as an alternative to prompt engineering, but I think you're exactly right. Right. It's like activation space, prompt engineering, or at least that's a really interesting analogy. Yeah. So this is another, in my opinion, another really interesting paper. So the basic idea is you're going to take a prompt, feed it to your model. In this case, you're going to give it a reasoning problem and get the model to generate a complete chain of thought. So the model itself just generates the full chain of thought, vanilla style, nothing unusual. And then you're going to feed the chain of thought to the model and this is going to lead to a bunch of activations at every layer of the model, as usual. Now the final layer of the model, just before it gets decoded, you have activations there and you're going to say, okay, well why don't we essentially build a reinforcement learning model and have that model play with just those activations. And what we're going to do is we're going to get the model itself to decode and then estimate the expected reward on this task for the final kind of decoding the answer. And you're going to do it in a very kind of simple, greedy way. So whichever token is given highest probability, that's just the one that you're going to predict. And you're going to use essentially a version of the same model to predict the reward. And then if the reward is low, you're going to go in and modify. So according to the model's own self evaluation, if the reward is low, you're going to modify the activations in that final layer, the activations that sort of represent or encode the chain of thought that was fed in. So you're going to tweak those and then you'll try again, decode, and then get the model to evaluate that output. Oh, you know, I think it needs to be, you know, we need to do some more tweaking. So you go back and you tweak again the activations and you can do a bunch of loops like this. Essentially it's like getting the model to correct itself and then based on those corrections, it's actually changing its own representation of the chain of thought that it was chewing on. It's really quite interesting. And again, it feels, it sort of feels obvious when you see it, but somebody had to actually come up with the idea. A couple of observations here. So there's an interesting scaling behavior as you increase the number of iterations of the cycle, right? Get the model to actually decode, evaluate its own output, then tweak the activations a bit. What you find is there's typically like an initial performance improvement that's followed by a plateau. And that plateau seems to come from the model's own ability to evaluate, to predict the rewards that would be assigned to its output. When instead of the model self evaluating, you use an accurate reward model, one that always gets the reward prediction right, then all of a sudden that plateau disappears and you actually get continuous scaling. Like the more of these loops you do, as long as you're correctly assigning the reward and it corresponds to the true base reality, you just continue, continue, continue to improve with scale. So that's another scaling law implied in here, which is quite impressive. There's also a bunch of like compute efficiency stuff. So there's a question of, like, do we, do we think of the playing field as every activation in the final layer of the transformer or as a subset. We could imagine only optimizing, only doing reinforcement learning to optimize, say 20% of those activations. And in fact it turns out that that ends up being the optimal way to go. And 20% is a pretty good number, they find. Don't optimize all those Activations, just optimize some of them. And at least for me that seemed counterintuitive. Like why wouldn't you want to optimize the full set of activations? It turns out, you know, a couple of reasons. One is just optimization, stability. Right. So if you're updating everything, there's a risk that you're just going to go too far off course and you need to have some anchoring to the original meaning of the chain of thought. So you don't. Yeah, steer way off. And then there's issues of representational capacity. So just having enough latent representations to allow you to do effective extrapolation. Anyway, this is a really, I think, interesting and important paper. Wouldn't be surprised to find it turn into another dimension of test time scaling. So yeah, just thought it was worth calling out.

B (75:09)

Yeah, it's interesting in a sense of, I don't know, it's like you have an auxiliary model or you could conceptually have an auxiliary model that's just for evaluating this in between activation and doing sort of side optimization without updating your main model. Something about it seems a bit strange conceptually and maybe there's like equivalent versions of this, but that's just a gut feeling somehow I get. And next we have two. Experts are all you need for steering thinking. Reinforcing cognitive effort in MOE reasoning models without additional training is the title of this paper. So this is a way to improve reasoning in mixtures of experts. Model with without additional training. Mixture of experts is when you have a model that sort of splits a work across subsets of it, more or less. And they are aiming to focus on and identify what they're calling cognitive experts within the model. So they're looking for correlations between undesirable reasoning behaviors and the activation patterns of specific experts in nmoe. Mixtures of experts models. So basically just large language models that have mixtures of experts. And then when they find the experts that turn out to have the best kind of reasoning behavior, they amplify those experts in the computation of the output. And typically the way mixtures of experts work is you like route your computation to a couple of experts and then you sort of average out the outputs of those experts to decide what to output. So conceptually you can sort of give more weight to certain experts or route the data to certain experts more often. So when they find these at these theoretical cognitive experts, they show that in fact this seems to be something that can be done in practice for LLMs that have MOE for reasoning applications.

C (77:29)

Yeah, and it's kind of like, I want to say embarrassingly simple how they go about identifying what are the experts, what are the components of the model that are responsible for doing reasoning. And so it turns out when you look at the way deep seq r1 is trained, right, it's trained to put its thinking, its reasoning between these thinking tokens, right? So they kind of have, it's like HTML, if you're familiar with that. Like you know, you have like bracket, think, bracket and then your actual thinking text and then close bracket, think bracket. What they end up doing is they say, okay, well like let's see which experts typically get activated on the thinking tokens. And it turns out that it's only a small number that consistently get activated on the thinking token. So hey, that's a pretty good hint that those are the experts involved in the reasoning process. So the way they test that intuition is they say, okay, well if that's true, presumably like you said Andre, if, if I just dial up the contribution of those experts, of the reasoning experts on any given prompt that I give them, then I should end up seeing more effective reasoning or at least a greater inclination towards reasoning behavior. That's exactly what happens. So this is pretty, I would have like, it's this happens so often but like I would have been embarrassed to suggest this idea. Like it just seems so obvious. And yet the obvious things are the ones that work and in fairness they only seem obvious in hindsight. This is obviously a very good idea anyway. So they use a metric called point wise mutual information to measure the correlations between expert activations and reasoning tokens. It's actually a pretty simple measure, but there's no point going over it in detail. One interesting thing is there's cross domain consistency though. So the same expert pairs consistently appeared as the top reasoners, the top cognitive experts across a whole bunch of domains. Math, physics, a bunch of stuff which really does suggest that they encode general reasoning capabilities. I wouldn't have bet on this, like the idea that there is an expert in an MOE that is the reasoning guy. One thing, they don't touch on this in the paper. But I would be super interested to know how are the different so called reasoning experts different? Right? So like they're saying there's two reasoning experts basically in this model that you need to care about. So how do, like what, in what ways do their behaviors differ? Right? What is, what are the different kinds of reasoning that the model is capable of or wants to divide between two different experts? I think that'd be really Interesting. Anyway, so a whole bunch of other stuff we could get into about compute efficiency, but there is no time.

B (80:08)

There's no time. We have quite a few more papers to discuss. So a lot of research. Also this week and next one is another Gemini related paper. It's Lessons from Defending Gemini against Indirect Prompt Injections coming from Google. Quite a detailed report. Something like, I think 16 pages. No, actually like dozens of pages if you include the appendix with all the various details. The gist of it is you're looking at indirect prompt objections to things like embedding data in a website to be able to get an AI agent that's been, you know, directed to go off and do something, go off course. And the short version I'll provide as a summary and Jeremy, you can add more details as you think you know is appropriate is that they find that it is possible to apply known techniques to do better. So you can protect against known attacks and do that via adversarial fine tuning, for instance. But the high level conclusion is that this is an evolving kind of adversarial situation where you need to essentially be continually on it and see what are these new attack techniques to be able to deploy new defense techniques as things evolve.

C (81:47)

I think that's a great summary, especially given the time constraints. Yeah, I'll just highlight two quick notes. So first is they find adaptive evaluation of threats is critical. So a lot of the defenses that do really well on static attacks can be tricked by really small adaptations to attack. So tweak an attack very slightly and then it suddenly works. Right. So this is something that we see all the time. And then there's this other notion that if you use adversarial training to help your models get more robust to these kinds of attacks, that's going to cause the performance to drop. And what they find is that's actually not the case. One of the most interesting things about this paper is just like the list of attacks and defenses to prompt injection attacks that they, they go over, I'm going to mention one and then we'll move on. But it's just called the spotlighting defense. I actually had never heard of this before. So if you have an attacker who injects a prompt in or some, some dangerous text into a prompt, like ignore previous instructions and do some bad thing, the spotlighting defense, what it does, it will insert what are known as control tokens. So they're basically just like new different kinds of tokens at regular intervals that just break up your text so that, you know, ignore previous instructions get split up and you have, you know, IG and then control token and then nor and then pre and then another control token. And it just has a way of. And then you tell the model sorry in the prompt you tell it to be skeptical of text between those control tokens. And so that kind of teaches the model to, you know, be a little bit more careful about it. And it has anyway really effective results. There's a whole bunch of other defenses and attacks they go into. If you're interested in the attack defense balance and the zoo of possibilities there, go check out this paper. It's a good catalog.

B (83:35)

Next up we have from Epic AI how fast can you can algorithms advanced capabilities? So this is a blog post associated with a previously released paper titled LLM E guess can LLM capabilities advanced without hardware progress? The motivation of the research is basically asking the question of can we find software improvements that yield big payoffs in terms of better accuracy? So it ties into this hypothesis that if LLMs get good enough at conducting good AI research, they can find breakthroughs to self improve. And then you get this so called intelligence explosion where the LLMs get better at research, they find new insights as to how to train better LLMs. And then the better LLMs keep finding better algorithmic insights until you become super, super ultra intelligent. And this is one kind of commonly believed hypothesis as to why we might get what is it sai super intelligent AI asi. Yeah, asi. Asi. Relatively soon. So this blog post is essentially trying to explore how likely that scenario is based on the trajectory and history of algorithmic progress so far. And the gist of their conclusion is that there are two types of investments. They are compute dependent and compute independent insights. So there are some insights that only demonstrate their true potential at large scales. Things like transformers, mixtures of experts, sparse attention that with smaller models when you're testing may not fully show you how beneficial they are, how promising they are. But as you scale up, you get way way stronger benefits of like 20 times the performance, 30 times the performance versus smaller things like layer norm, where you can reliably tell that this algorithmic tweak is going to improve your model. And you can verify that at 100 million parameters instead of 10 billion parameters or 100 billion parameters, meaning that you can do research and evaluate these things without ultra large hardware capacity. So the basic conclusion of the paper is that the idea that you can get intelligence explosion needs to be a result of finding these compute dependent algorithmic advances being easier to find. So you need to find the advancements that as you scale up compute will yield like big, big payoffs rather than relatively small payoffs.

C (86:40)

Yeah, the frame is that these. So these compute dependent advances are like you said, you only see the return on investment at large scales or the full return on investment at large scales. And they point out that when you look at the boosts in algorithmic efficiency that we've seen over the years, these are dominated by compute dependent advances. So you look at the transformer, the moe, mixed query attention, sparse attention, these things Collectively they're like 99% of the compute efficiency improvements we've seen. 3.5x according to them from compute independent improvements like flash attention and rope. But, but they don't hold the candle to the these like approaches that really leverage large amounts of compute. And so I think in their minds the case that they're making is like you can't have a software only singularity if you need to like leverage giant amounts of physical hardware to test your hypothesis to validate that your new algorithmic improvement is actually effective, you need to actually work in the physical world to gather more hardware. I think this frankly doesn't do the work that it thinks it does. There are a couple of issues with this and actually Ryan Greenblatt on X has a great tweet storm about this by the way. First of all, love that EPIC AI is doing this. Really important to have these concrete numbers so that they can facilitate this sort of debate. But I think the key thing here is so they highlight, look, transformers, transformers only kind of give you returns at outrageous or sorry, give you the greatest returns at outrageous levels of scale. So therefore they're a compute dependent advance. I don't think that's what actually matters. I think what matters is would an automated software only process have discovered the transformer in the first place? And to that I think the answer is actually probably yes. Or at least there's no clear reason that it wouldn't have. In fact the transformer, moe, mixed query attention, they were all originally found at tiny scale, as Ryan points out, about one hour of compute on an H100 GPUs. That's quite small even back in the day, in relative terms it was certainly doable. And so this is like the actual question is, do you discover things that give you a little lift that makes them seem promising enough to be worthy of subsequent investment? The answer seems to be that actually basically all of the advances that they highlight as the most important compute dependent advances have that property were discovered at far, far lower scale. And we just keep investing in them as they continue to kind of show promise and value. And so it's almost like any startup, you keep investing more capital as it shows more traction. Same thing. You should expect the decision theoretic loop of a software only singularity to, to latch onto that because that's just good decision theory. So anyway, I think this is a really rich area to dig into. I have some issues as well with their frame. They look at Deep SEQ and they kind of say that the Deep SEQ advances were all kind of compute constrained advances or compute dependent. But again the whole point of Deep SEQ was that they use such a small pool of compute. And so I almost want to say like to the extent that compute independent means anything, Deep seq, a lot of their advances really should be viewed as statutorily compute independent. Like the point is that they had very little compute. This is actually a great test bed for what a software only process could unlock potentially. So lots of stuff there. You can look into it. I think it's, it's a great report and a great room for discussion.

B (90:07)

Yeah, I think it's, it's kind of introducing the conceptual idea of compute dependent versus independent algorithms and then there are questions or ideas you can extrapolate. Last paper, really quickly, I'll just mention without going into depth, there is a paper titled Reinforcement Learning Fine Tunes Small Sub Networks in Large Language models. The short, short version is when you do alignment via reinforcement learning that turns out to update a small number of the model parameters, something like 5% or sorry 20% versus doing supervised fine tuning, you update all the weights as you might expect. So this is a very strange and interesting kind of behavior of reinforcement learning alignment versus supervised alignment. And I figured should just mention it as an interesting paper but no time to get into it. So moving on to policy and safety, we have first an exclusive with a report on what OpenAI told California's Attorney General. So this is I suppose a leak or perhaps, I don't know, a demonstration of this. Response to petition for Attorney General action to protect the charitable nature of OpenAI's assets sent to the Attorney General on May 15 by OpenAI basically has all their arguments in a position to the groups that want to stop OpenAI from restructuring and really just restating what we've been hearing a whole bunch. You know, Musk is just doing this as a competitor and is being, is harassing us and has misinformation and basically saying, you know, ignore this petition to block us from doing what you want. It isn't Valid.

C (92:17)

Yeah. And it's. So there are a whole bunch of interesting contradictions in there as well with some of the claims OpenAI has been making, or at least the vibes they've been putting out, which is pretty standard. OpenAI fair that they'll, you know, they get, they try to get away with a lot, it really seems. And there's a lot of examples of this here. So one item is. So they, they suggested the nonprofit, by the way. So some of this is revealing information, material information about the nature and structure of the deal, this sort of nonprofit transition thing that was not previously public. Right. So OpenAI recently came out and said, look, this whole plan we had of having the for profit kind of get out from under the control of the nonprofit, we're going to scrap that. Don't worry, guys. We hear you loud and clear. There are now a bunch of caveats we highlighted, I think last week that there would be caveats. The story is not as simple as OpenAI has been making it seem. A lot of people have kind of declared victory on this and said, great, you know, the nonprofit transition isn't happening. Let's move on. But hold on a minute. This is OpenAI doing their usual best to kind of control the PR around this, and they have done a good job at that. So here's a.

B (93:30)

Let me just really quickly mention for context. This is partially in reply to this not for Private Gain coalition that has a public letter. They released a public letter in April 17th. They updated their letter in May 12th in response to, on May 5th, OpenAI announcing that they are kind of backing off from trying to go full for profit with this new plan of the Public Benefit Corporation and the kind of not going for profit. So this not for Gain a coalition updated their stance and essentially has criticism still. And this letter on May 15th is in response to whole chain of criticism.

C (94:18)

Yeah. So if it wasn't complicated enough already. Yeah. And so here's, here's a line from the Opening Eye statement here. The nonprofit will exchange its current economic interests in the capped profit for a substantial equity stake in the new Public Benefit Corporation and will enjoy access to the Public Benefit Corporation's intellectual property and technology personnel and liquidity. That sounds like a good thing until you realize that, well, wait a minute, the nonprofit did not just enjoy access to the technology it actually owned or controlled the underlying technology, so now it's going to just have a license to it, just like OpenAI's commercial partners. That is a big, big caveat. Right. That is not consistent with really the spirit potentially, but certainly the facts of the matter associated with the previous agreement as I understand them, under the the current structure, OpenAI's LLC, so the sort of main operating agreement explicitly states that the company has a duty to its mission. And the principles advanced in the OpenAI charter take precedence over any obligation to generate a profit. That creates a legally binding obligation on the directors of the company, the company's management. Now, under the new structure, though, the directors would be legally required to balance shareholder interests with the public benefit purpose. And so this is like the fundamental obligations, legal duties of the directors is now going to be to shareholders over or potentially alongside, I should say, the mission. And that shift is probably a big reason why investors are more comfortable with this arrangement. We heard SoftBank say, you know, look, from our perspective, everything's fine after they had said OpenAI has got to get out from under its nonprofit in order for us to keep our investment in. And now they're making these noises like they're satisfied. So clearly for them, de facto, this is what they wanted, right? So there's something going on here that doesn't quite match up, and this is certainly part of it, or at least seems like it is. By the way, no public benefit company in Delaware, Garrison lovely, who's the author of this, says no Delaware PBC has ever been held liable for failing to pursue its mission. Their legal scholars can't find a single benefit enforcement case on the books. So in practice, this is a very wide latitude, right? There's a lot of shit that this could allow. In this letter, they're trying to frame all the criticism of this very controversial and I think pretty intuitively kind of inappropriate attempt to. To tran. You know, convert the. The kind of nonprofit or all that jazz. They're trying to pin it on Elon and say basically he's like the only critic, or that's sort of the frame. Just because it's easy to dismiss him as a competitor and for political reasons. He's a. He's an easy whipping boy. But there's a whole bunch of stuff in here. You know, there's like, I'll just read one last. One last excerpt because we gotta go. But OpenAI's criticism of the coalitions. This is the coalition you referred to, Andre, April 9th letter is particularly puzzling. The company faults the coalition for claiming that, quote, OpenAI proposes to eliminate any and all control by the nonprofit over OpenAI's core work. This criticism is perplexing because as OpenAI itself later demonstrated with its May 5 reversal that was precisely OpenAI's publicly understood plan at the time the coalition made its statement. The company appears to be retroactively criticizing the coalition for accurately describing OpenAI's proposal as it stood. So you could be forgiven for seeing a lot of this as kind of manipulative, bad faithy comms from OpenAI, especially given that this letter was not meant to be made public. And it fits, unfortunately, a pattern that we have seen the people, at least many people, believe they have seen many times over. We'll see where it all goes. But this is a thorny, thorny issue.

B (98:12)

Yeah, I think we've gotten hints at kind of the notion that OpenAI legally has tried to be aggressive. Not just legally, also publicly in terms of arguing with Musk and so on. And we only have time for one more story, so we're going to do that. We have ACT activating AI Safety level Free protections from Anthropic. So Anthropic has their responsible scaling policy that sets out various thresholds for when they need to have these safety level protections, with additional safety levels requiring greater scrutiny, more stringent processes, etc. So with Claude Opus 4, they are now implementing these AI Safety Level 3 measures as a precautionary measure. So they've said we're not sure if Opus 4 is kind of at the threshold where it would be dangerous to the extent that we need this set of protections, but we're going to implement them anyway. And this comes with a kind of a variety of stuff. They're committing to do it. They are making it harder to jailbreak, they are adding additional monitoring system. They have a bug bounty program, have synthetic jailbreak data, security controls, making sure the weights cannot be stolen and so on. Kind of quite a few things. They released a PDF with the announcement that is, you know, like something like a dozen pages with additional details in the appendix.

C (99:55)

Yeah. And so the specific thing that's causing them to say we think we, we are flirting with the ASL3 threshold is the, the biorisk side. Right. The ability, they think potentially of this model to significantly help individuals with basic technical backgrounds, like we're talking undergraduate STEM degrees, to create or obtain and deploy biological weapons. Right. So that's really where, where they're at here specifically. This is not, I don't think, associated with the autonomous research or autonomous autonomy risks that, that they're also tracking. But we got early glimpses of this. Right. With Sonnet 3.7. I think the language they used, it was either Anthropic or OpenAI with Their model, it was sort of similar, was we're on the cusp of that next risk risk threshold where really it's kind of similar. Whether you look at the OpenAI preparedness framework or Anthropics SL3 in terms of how they define some of these standards. The security measures are really interesting, especially kind of given our work on the data center security side and the cluster security side, one of the pieces. And this echoes a recommendation in a RAND report on securing model weights that came out over a year ago. Now they have implemented preliminary egress bandwidth controls. So this is basically restricting the flow of data out of secure computing environments where AI model weights are so literally like at the hardware level. Presumably that's at least how I read this. Making it impossible to get more than a certain amount of bandwidth to pull data of any kind out of your, out of your servers. That's meant to make it so that if somebody wants to steal a model, it takes them a long time at least if they're going to use your networks, your infrastructure. And there are ways to kind of calculate what the optimal bandwidth would be under certain conditions for that. But that was kind of interesting. That's a big piece of really R and D that they're doing there. Also a whole bunch of management protocols, endpoint software controls and there's a bunch of stuff here. This is a big leap, right, moving to ASL3. So this is fundamentally increasing. It means that they're concerned about threat actors like terrorist groups and organized crime, that they would start to derive a lift, a significant benefit potentially from accessing Anthropic's IP. They are not, you know, ASL3 does not cover nation state actors like China. So they're not pretending that they can defend against that level of attack. It's sort of like working their way there. As their models get more powerful, they want to be able to defend against a higher and higher tier of adversary. So there we go. Curious to see what the other labs respond respond with as their capabilities increase too.

B (102:33)

Yeah. And we're seeing hints that maybe we'll cover more next week. That and we've already covered to some extent that these reasoning models, these sophisticated models are maybe harder to align and are capable of some crazy new stuff. So this also makes sense for that. Yeah, but we're going to call it with that for this episode. Thank you for listening as always. We appreciate you sharing, commenting and listening more than anything. So please do keep tuning in.

A (103:30)

Break it down last weekend AI come and take a ride get the low down on tech and let it slide Last weekend AI come and take a ride Couple lads to the streets AI's reaching high new tech emergent Watching surgeon fly from the labs to the streets AI's reaching high algorithm shaping up the future sees tune in tune and get the latest with ease Last weekend, AI come and take a ride get the lowdown on tech and let it slide Last weekend AI come and take a ride I will last through the streets AI's reaching high from neural nets to robot the headlines pop Data driven dreams they just don't stop Every breakthrough, every code unwritten on the edge of change excitement we're smitten from machine learning marvels to coding kings Futures unfolding, see what it brings.