← Last Week in AI
Loading summary
Transcript342 lines
- [00:00]
Host 1
Foreign. We'd like to thank Box for sponsoring
- [00:12]
Andrei Korenkov
last week in AI.
- [00:14]
Host 1
Box is the leading intelligent content management platform enabling organizations to fuel collaboration, manage the entire content lifecycle, secure critical content and transform business workflows with enterprise AI. To unlock the power of AI, you need to get your content to your LLMs and agents. Agents. Your business isn't the sum of Internet knowledge. Your business lives in your content, so you don't just want to bolt on
- [00:40]
Andrei Korenkov
AI to your existing processes.
- [00:43]
Host 1
To become an AI first company isn't just about automating what you already do, it's about reimagining what's possible. With boxai, you can truly leverage the latest breakthroughs in AI to automate document processing and workflows, extract insights from content, build custom AI agents to work on assignments, and more. And most importantly, boxai works with all the major leading AI model providers so OpenAI, Anthropic, Google XAI and others so
- [01:10]
Andrei Korenkov
you can be sure you can use
- [01:12]
Host 1
the latest AI models with your content. Boxai will give you the content layer that gives AI the context it needs while giving your teams the flexibility they need to test and leverage various models for different use cases. So go to box.com AI to learn more. Last week in AI we would like to thank odsc AI for being a sponsor. ODSC is one of the longest running and largest communities focused on applied data science and AI.
- [01:40]
Andrei Korenkov
It started over a decade ago with
- [01:42]
Host 1
a simple idea Bring practitioners together to learn from people actually building and deploying models in the real world, not just talking theory. On April 28th through the 30th, you can experience it yourself at ODSC East 2026 taking place in Boston and virtually, there will be thousands of hybrid attendees ranging from data data scientists, ML engineers, AI researchers and technical leaders. You can attend over 300 sessions covering LLMs, gen, AI, computer vision, NLP, data engineering and more. You can also go to hands on training with workshops and bootcamps taught by experts from companies like OpenAI, Hugging Face, Nvidia and other top companies and universities. And of course there'll be a massive expo and networking opportunities. Great for startups, hiring managers and AI tool builders. It's one of the best ways for AI practitioners and teams to stay ahead of the field, learn from the best and connect with the community. Go to ODSC AI east and use promo code LWAI for an additional 15% off your pass to ODSC AI East 2026. That's ODSC AI east and use code LWAI to get an extra 15% off on the number one AI builders and training Conference.
- [03:00]
Andrei Korenkov
This episode is brought to you by Progressive Insurance.
- [03:03]
Jeremy Harris
Do you ever find yourself playing the budgeting game? Well, with a name your price tool
- [03:08]
Co-host 2
from Progressive, you can find options that fit your budget and potentially lower your bills.
- [03:12]
Jeremy Harris
Try it at progressive.com, progressive Casualty Insurance
- [03:15]
Co-host 2
Company and affiliates pricing coverage match limited by state law not available in all states
- [03:22]
Andrei Korenkov
hello and welcome to the Last Week in AI podcast where you can hear chat about what's going on with AI. As usual in this episode we will summarize and discuss some of last week's most interesting AI news. You can also check out our Last Week in AI newsletter at lastweekin AI for even more news. I am one of your regular hosts, Andrei Korenkov. I studied AI in grad school and now work at the startup Astrocade and
- [03:47]
Co-host 2
I'm your other co host Jeremy Harris. I'm at Gladstone AI doing AI national
- [03:52]
Jeremy Harris
security stuff
- [03:54]
Co-host 2
last episode, right? I think we we ended up doing just news stories. We didn't do any research any papers because of just how much there was. And so this week it's kind of the opposite situation where we're just like we got a shit ton of papers to cover, so we'll see what we can do in time. But this is going to be an interesting challenge. We're really kind of vacillating from one mode to the other.
- [04:15]
Andrei Korenkov
Yeah, it's a week where there was not a ton to say. No new models, unlike previous weeks, not many business stories. So you're probably going to get through those sections pretty quick and get to talk to about like six papers or something. It's going to be a lot before we get there. Just want to call out did see a new review on Apple Podcasts. Super solid and informative, a super concise review which is pretty cool to see. So I appreciate that we do our best to keep this super solid and informative and do appreciate all the feedback and also the comments on YouTube which I keep an eye on.
- [04:57]
Co-host 2
I will say it really does help psychologically because it is a lot of work to prep for these things and so seeing that it's appreciated feels really good and I love that sense of community too on the Apple side and also on the YouTube side. It just seems like there's a lot of folks chatting and anyway it's great to see so really appreciate it and
- [05:17]
Andrei Korenkov
kicking off with tools and apps. First story is Perplexity's personal computer turns your spare Mac into an AI agent so they have announced this new AI tool called Personal Computer, which is basically OpenClaw by Perplexity. So it's gonna use your Mac Mini, which is famously what OpenClaw runs on. You get a Mac Mini, you set up openclaw and it kind of like lives on that server effectively. So Perplexity is positioning this as the safe alternative. You know, we actually kind of more adult, mature version that you can use as your personal assistant. You can provide it full access to files and apps and there's a nice looking UI to use it. So interesting that they kind of, I don't know if they pivoted or rushed into this. Right now it's not available to the public. You can join a wait list for early access. There's no specified launch date, so it really seems to be like, oh, this is probably a good idea, let's do it and we'll see when it actually comes out.
- [06:31]
Co-host 2
Yeah, and this is obviously hitting on the head the big security issue that OpenClaws had. Everybody's been talking about it. You're basically giving this thing unlimited access to everything on your machine, it runs on a cloud, blah, blah, blah. Well, this is going to be a local version of that, as you said. So the idea here is to fundamentally shift the security picture so that as you say, it's sort of the mature, grown up, adult way to do this. There's a lot of to the security picture. I think that this will be a really interesting space for startups to play in, having Open Claw run in ways that are verifiably secure, in ways that are updated very quickly to account for new capabilities and trends in AI capabilities. Because new security issues, as we've seen, just seem to be popping up all the time almost as fast as you can go. Which makes a really good case for a subscription service. I mean, this is the classic reason that you would subscribe to something. Yeah, really interesting and as you say, a big strategic move by Perplexity. Not clear they have the option not to do this because plain old search is not going to be the way of the future for much longer.
- [07:33]
Andrei Korenkov
And next up, a new update to Claude Code. They have launched this feature code review and that is what it sounds like. You can now have it review code on GitHub and there's this thing called pull requests which as a software engineer you do so it can automatically review those and provide actionable feedback. Saw some kind of funny discussions of it on Twitter where people are saying it's like costing 15 to $25 per code review and Also that this is sorely needed to deal with a glut of code being pushed out there by people who use cloud code to write the code in the first place. This is joining a pretty busy space. There's multiple companies that are entirely working on this like code rabbit with AI code review automation. And yeah, not surprising, this is in some ways more of a nice utility because you can already use it for this purpose. And in fact at my company we already do use it for this purpose without having any sort of fancy UI or automation behind it.
- [08:49]
Co-host 2
Yeah, this is a pretty interesting move. It is positioned as a safety tool, right. So it's like security, catching bugs, looking at AI generated code and so on. But it's quite interesting that this is also a growth flywheel in its own way. The more code that Claude code generates, the more PRs are going to pile up, the more companies need code review. And so anthropic is kind of like closing that loop in a very interesting way, all under the same roof. They are also just basically looking at a multi agent architecture by design. Right. So the core philosophy here is get a bunch of agents to look at the code base from different angles and kind of a final agent that aggregates ranks and findings. And they're exclusively focusing here on logical errors rather than style, which suggests that they've learned from the graveyard of linting tools that developers have ignored or developed over the years. And so, yeah, I mean this is quite interesting. The pricing is interesting as well. So each review, each PR review is estimated to be between 15 and 25 bucks. So if you've got a big company with hundreds of engineers each shipping 10 plus PRs a day, that might compound fast. It's quite interesting.
- [10:05]
Jeremy Harris
The cost of course, of doing a
- [10:06]
Co-host 2
PR review, if you look at the hourly rates of developers are certainly going to be in that range. So it's kind of interesting. We've talked before about the cost of getting AI to solve certain problems, sort of flirting with the human cost already. And that'll obviously collapse too as AI gets cheaper and Moore's law and Huang's Law start to play into this. So pretty interesting. This is also anthropic. Doubling down on the enterprise side following the Department of War lawsuit. Right. So doubling, tripling down in that direction, that's working maybe to ensure that they have a. Not a fallback. I mean it's the wrong frame because they are incredibly successful in what they're doing here, but just to kind of make sure that they're boosting that part of the Business.
- [10:45]
Andrei Korenkov
Yeah. And I think worth mentioning, kind of a broader context of they've been just pushing a ton of cloud code updates in recent months. They also, I think this week just launched this like by the way feature where you can have a little side chat as you're having Claude do something. So it's very clear that cloud code is now one of their main products. Really generating a ton of revenue, has seen crazy adoption and they're really building it out in a way that wasn't the case last year. And next sort of related story. Cursor is rolling out a new kind of agentic coding tool, although that might be overstating it. They launched a new tool called Automations and that is what it sounds like. You can launch coding agents based on triggers like code based changes, Slack messages or timers. And by the way, actually I think cloud code also has this where you can schedule tasks and have it do it regularly. Yeah, yeah, yeah. So that's pretty much all there is to it. You can now have agents be triggered by various things, not just you directly prompting it. You could probably connect it to be like, oh, there's a new pull request, go have it do the code review. It fits into this broader trend of having agents sort of live out there in a server or a cloud and you can kind of ping them and have them do stuff, as in open cloud, for instance. Although in this case the difference is it's automated. So they're just running 247 and doing stuff whenever.
- [12:26]
Jeremy Harris
Yeah, I mean you can think of this as kind of like a shift
- [12:29]
Co-host 2
from like humans as the orchestrators to.
- [12:33]
Jeremy Harris
Or generally it's shifting from agentic orchestration to an infrastructure play.
- [12:38]
Co-host 2
So right now most agentic coding tools have a human who serves as the dispatcher. So you write a prompt and then the agent runs and then you review. And the whole idea here is you're kind of reframing the human's role in this. So instead of just initiating, what you're doing is you're being, as they put it called in at the right points in the conveyor belt. Right. So you're kind of just your attention is being drawn to specific moments in the workflow where there's a human inject that's really required. Which makes this a pretty different mental model from what software development has looked like even in the AI augmented age.
- [13:13]
Jeremy Harris
So in that sense this is part
- [13:15]
Co-host 2
of Cursor being positioned right now is $2 billion in annual revenue, doubling in three months. They've got a huge fraction of the generative AI market for coding. They have enough trust that they can start to reshape workflows and not just augment them. And that's what they're leaning on here in part. I mean, we're going to see more of this. And as you say, Anthropic is already knocking on the same door. But this is kind of a very opinionated stance on what the next level of abstraction is going to be and sort of seeing human attention as this thing that you're very, very carefully dialing in. Again, less orchestration, more sort of AI as infrastructure.
- [13:51]
Andrei Korenkov
Right. The way they position it is always on agents, which I think is a nice term for it. Again, this is similar to openclaw and now Perplexity's computer. The agent is out there and you don't need to be at your computer talking to it. Like chatgpt.com, there's various ways to get it to do work for you up there in the background and you can kind of join in and see what it's up to whenever. And hopefully it's not up to deleting your emails or wasting your money, as can happen with OpenCloth. Next up, ChatGPT can now create interactive visuals to help you understand math and science concepts. So this deals with dynamic visual explanations. It can create interactable real time Visuals. Apparently there's 70 topics that it has, Binormo Square, Charles's Law, Ohm's Law and other things like that. So it sounds like maybe it's kind of a pre built set of visualizations presumably for things that people often ask it about. And yeah, it's like a little mini app where you, you have ways to visualize the actual logic behind equations and so on. And related to that actually anthropics. Claude can respond with charts, diagrams and other visuals now. So the visuals here are going to be inserted in line within the chat and it appears to be more general. So it is a little mini program that is creating for you to be able to interact with. The example they give is you can ask for create an interactive version of a periodic table and it does that. Presumably this is not really different from artifacts, which Claude has had for a long time, where it can code up a little app. But the difference is it's actually there inside the chat directly in response to your question.
- [15:52]
Co-host 2
This is anthropic finally covering down on a gap. You know, if you've used Claude, you've seen this, right? It's like it doesn't do audio, image and video models. And you know it's like mostly just not the model that or the company that you turn to for those sorts of things. And this is them trying to kind of right that wrong. The timing is interesting. Obviously you mentioned this, you know, days after OpenAI updated ChatGPT for the science visuals and math concepts, though that is also kind of a more specific use case. This really is the broader issue of Anthropic not having had much of a strategy for kind of images and videos.
- [16:26]
Jeremy Harris
And the other piece here is they
- [16:27]
Co-host 2
are differentiating by making their visuals conversational so you actually like see them update as you have your discussion rather than just having a static output. So you know, all that is quite interesting and just trying to broaden the market as Anthropic hasn't had to because typically being more business to like B2B versus B2C business to consumer, Anthropic has been able to get away with focusing on code, focusing on reasoning, that sort of thing almost exclusively. Now they're kind of broadening out a bit and I don't know if this is an explicit play to go more consumer. This is going to have B2B applications as well obviously. But yeah, interesting in that context too.
- [17:05]
Andrei Korenkov
And now actually a different section than usual. I figured we'd throw in projects and open source. We have one story here and it actually is related to tools in that we have a new model out there introducing Namatron Free super, an open hybrid Mama Transformer Moe for Gentex reasoning. This was announced by Nvidia Nematron 3 Super has 120 billion total parameters with 12 billion active parameters per inference. It has apparently a 1 million token context window, although I do want to mention when we say 1 million token context window, as we have GPT 5.4 and I think Sonnet4.6, you know, on paper it's 1 million. In practice, as you get into the upper end, maybe the model starts being very stupid. The model compares favorably to other open source models, although probably not all of them. The charts that they provide, just compare IT to GPT OSS 120 and Quin 3.5 100. So in that size class it is on the benchmarks doing well and the throughput is way, way higher. So that's one of the cool things. With both an MOE model and a Mamba model, the performance is comparable, but it can be run locally very, very quickly.
- [18:42]
Jeremy Harris
Yeah, so they're training natively in four bit arithmetic.
- [18:46]
Co-host 2
So usually what happens is you'll see like FP16 or like, you know, you know BF16 or like these high resolution numerical representations of the weights and biases in the weights and model. And then what you do is after the fact, you will then quantize the model. You'll essentially create a lower dimensional or a low resolution representation of all the numbers. And you got to hope that it operates about as well. What they're doing here is actually like training natively in four bit arithmetic. So just four bit quantization, which is helpful because what tends to happen when you try to go from a high resolution numerical representation to a lower one in like in one quantization step is
- [19:24]
Jeremy Harris
that like the problem is that the
- [19:26]
Co-host 2
model was trained to leverage the resolution that it can enjoy, right, with whatever 16 bit resolution or whatever when you flip it to 4 bit. Well, maybe the model would have made different choices actually during training in terms of the values of some of the weights if it had known that it was going to be used at 4 bits arithmetic representation. And so they're actually pre training fully in four bit. So it's quantized already coming out of the box, if you will. And what this does is it means that this model is optimized for Blackwell GPUs specifically. So they are actually kind of interleaving their model design and the hardware very, very closely. It's totally rational from their standpoint, but it's worth thinking about if you're thinking about portability. This model will not work as nicely on non Blackwell GPUs, for example, which is sort of the, you know, a kind of hardware lock in. It's open source, right? This is Nvidia saying it like open source, but. But you can use it only on. Not only on, but the performance will be best on their hardware. That's a really interesting play from Nvidia. There's also this idea of latent MOE that they're doing. It's a bit of a sleeper feature, but you're basically compressing so you have your token embeddings, you're going to compress them to a lower rank space. Essentially you think of it as a lower dimensional space before you do the MOE thing. So the MOE thing is you take your token and then you pick any number of a bunch of expert submodels to route it to. Right now in this case what you're
- [20:52]
Jeremy Harris
doing is you're compressing the representation of
- [20:54]
Co-host 2
your token and then you're routing it to these models. And what this means functionally is you're consuming less compute. The token consists of fewer numbers that have to be crunched on by those experts. So you actually this costs less compute. It means you can for the same amount of compute consult about four times as many experts for the same computational cost. So this is a big advantage because oftentimes you know, you, you have too few experts chewing on your data. Ideally you'd want as well to some extent as many as possible. There are orchestration issues that come with that. But yeah, so it is quite interesting. It means you know, you have finer specialization. For example, you have like distinct experts routing paths for like Python versus SQL logic within a single agent turn. So that's pretty meaningful especially when you're, you're thinking about complex tool calling agents.
- [21:39]
Jeremy Harris
So quite interesting.
- [21:41]
Co-host 2
I think the 1 million token context window probably deserves a fair bit of scrutiny when you're claiming a large context window actually using it faithfully. Always a different thing. We've seen MAMBA architectures with known weaknesses in multi hop retrieval over long distances and the benchmarks that they're citing here, they're pretty narrow. Pinchbench for example is new enough that it hasn't been stress tested by the community more broadly but, but anyway, so this is something to look at. We'll see what the sniff test is as things evolve. But an interesting model for sure. And Nvidia continuing with this play, right? I mean they're going to continue beating that open source drum because the reality is they dominate the open source market way more than they already dominate the closed source model market. But they really dominate the open source market and want to get their hooks in with again these models that are most performative on Nvidia machines. That's one of the big take homes
- [22:36]
Andrei Korenkov
from this paper, right? And by the way, the actual news here is that they released the weights for Nemetron 3 Super. The paper was released in late December I believe. We covered it a couple months ago and discussed Nemetron 3 in more detail. One of the other interesting details is very much doubling down on this hybrid architecture of transformer with Mamba which is one of the ways in which it can be made efficient keeping compute kind of constrained as you scale up a number of tokens. Which could mean that it's actually pretty good at long context reasoning. But it also seems pretty apparent that they are not at the point where they have all these post training tricks of reinforcement learning. If you look at the paper they have a pretty large pre training data set but they don't discuss. They do have some post training but I just don't think their infrastructure is as mature. And one last thing to say Leyton Moe. They released a paper on that back towards the end of January. I don't think we actually covered it at the time. And it is in a way related to a hybrid approach. They position it as revisiting MOE design, a mixture of expert design from a hardware, software, co design perspective to make it optimal with respect to inference cost. So yeah, I think interesting from Nvidia in that they are kind of carving out a different space in how these open source models are. They aren't the most performant but you can presumably deploy them on a single GPU and get really nice kind of performance which does work well with what Nvidia sells obviously and onto applications in business. Sticking with Nvidia, Nvidia halts H200 production as China backs Huawei chips. So it is stopping production of its H200AI chips for China because of China, as a headline says, kind of not really being welcoming to their companies adopting these chips. This is the latest generation of Nvidia chips. So these are top end chips, not the H20, which is kind of the low end that was meant to comply with regulations. The US has approved limited exports of these chips to China, but Chinese customs block their entry and suggested that domestic alternatives should be prioritized. So man, Nvidia is really shifting waters. Why didn't it seem pretty clear what's going on now? It's like, oh, we can export them or we can't export them and now we don't want them. But the black market is probably good so we'll actually still have them be sold there. I don't know.
- [25:43]
Jeremy Harris
Yeah, no, I mean this is one
- [25:45]
Co-host 2
of those situations where I'll just give my personal opinion here is that this is a kind of two wrongs make a right situation. So I don't think that we should ever have been approving the export of H2 hundreds to China. That just like that seems insane for context. Right? Just as a reminder, the numbers here are super confusing. So the first there was the H100 GPU, right? This is the GPU that would have been used to train models in the like 20, 24, 25 era. Right. Okay, so that's the H100.
- [26:14]
Jeremy Harris
Then Nvidia came up with China specific
- [26:17]
Co-host 2
variants of it, the H800 and the H20, those were shipped to China in large volumes. There's all kinds of interesting arguments about how those chips in some ways were actually more performant than the H100, which is a huge issue, but whatever. Then There was the H200 which despite
- [26:32]
Jeremy Harris
the fact that it looks a lot
- [26:34]
Co-host 2
like H20, it's just missing a zero, was not at all intended for the Chinese market originally. It is strictly an improvement on the H100, I believe, using the same manufacturing process at TSMC. So as a reminder, Nvidia designs the chips. They ship their designs overall to Taiwan Semiconductor Manufacturing Co. TSMC in Taiwan, and they actually fabricate the chips, right, using their exquisite 4 nanometer process or 5 nanometer process for, depending on the GPU. Okay, so in this case, the Trump
- [27:00]
Jeremy Harris
administration said, hey, guess what?
- [27:02]
Co-host 2
We're going to like, overturn all the Biden administration stuff and say, okay, yeah, the H200. So this fully fledged Nvidia chip, we're going to allow it to ship to China. So the expectation was that, okay, great, Nvidia is going to make a crap ton of money doing this. It'll accelerate massively Chinese AI development, which a lot of people don't think is a good idea. But Nvidia's like, fantastic, will make some money. Problem is, Nvidia then ramps up production, right? So they go to TSMC and they
- [27:28]
Jeremy Harris
say, hey, guys, we want to buy out a huge chunk of your capacity
- [27:32]
Co-host 2
at your, I guess, your 4 nanometer node here to build these H200 chips. And that's really expensive.
- [27:39]
Jeremy Harris
TSMC has a finite capacity.
- [27:41]
Co-host 2
There's more demand for TSMC fabrication capacity than there is supply, right? So Getting allocation on TSMC's nodes is really, really hard. So Nvidia does that and then suddenly
- [27:50]
Jeremy Harris
China goes, you know what, Maybe we
- [27:53]
Co-host 2
do, maybe we don't want these, these Nvidia chips. So on the one hand, after Trump approved these limited exports to China in January, China came back and said, well,
- [28:03]
Jeremy Harris
you know, Chinese companies, they can purchase
- [28:05]
Co-host 2
the H200, but they should consider local chips first. And now they're basically coming back and saying, no, like, we're not going to
- [28:11]
Jeremy Harris
do this at all.
- [28:12]
Co-host 2
And as of now, no H200 chips have been sold to Chinese customers, at least according to U.S. officials. And so some Chinese commentators apparently have been saying that China's decision to curb these H200 imports is due to the US intercepting almost 2 million barrels of Venezuelan oil destined for China in late December. So China is actually viewing their blocking of H200 exports to China as a penalty to the United States, which is hilarious because this is all ass backwards. Like, really, the national security imperative would be to prevent these H2 hundreds from going to Chinese hands. The Chinese have way more to gain by taking these chips than to lose but the way they're framing this up for themselves as well, we'll increase demand in the domestic market, blah, blah, blah. There are all kinds of reasons why at least. I mean, I think that's a completely implausible position in any case.
- [29:02]
Jeremy Harris
So now we have a situation where
- [29:06]
Co-host 2
there's like a ton of. Anyway, I'm going to park it there. There's a whole kind of story here as well on the memory side of this. So the H100 and H200, one of the big reasons they're so powerful than the Huawei Ascend chips, for example, is that they use hbm, that's kind of high bandwidth memory that is made by SK Hynix, the South Korean company, and is much more advanced than what China has. Their closest competitor to SK Hynix is cxmt. But they're multiple generations behind and there's been an attempt to kind of puff up CXMT in this article. They're like, oh, our local solution to SK Hynix or Samsung CXMT is really impressive and blah, blah, the numbers just don't pan out. They're basically two or three generations behind here. Not close enough to be relevant. But this is part of what China's trying to do is make it seem like they're a pufferfish. Like we can, you know, we have cxmt, we have Huawei, we have SMIC and so on. And, and there you go, everybody's thumping
- [30:03]
Andrei Korenkov
their chests or whatever, right? And I think we've discussed previously how there's a massive black market, which chips kind of end up there even though they perhaps shouldn't. And the news is that they are reallocating manufacturing intended for the Chinese market. But, you know, what does that mean?
- [30:26]
Co-host 2
You know, oh, sorry, I'm gonna get on my soapbox for a second here. I'm old enough to remember when people. Sorry, I'm getting all. I'm old enough to remember when people like me and much, much smarter, all much smarter, were saying, look, there is a finite amount of capacity at TSMC and at Nvidia. Any chip that you're sending to the
- [30:47]
Jeremy Harris
Chinese market is a chip that is
- [30:48]
Co-host 2
not being received by an American company directly trying to compete with China. That is just true. And that's what the story shows. Literally, Nvidia is deciding, okay, we can't
- [30:58]
Jeremy Harris
ship these to China. Guess we're going to make more chips
- [31:00]
Co-host 2
for the American and Western markets. Like, that's literally the calculation. So as far as I can tell, at least what this suggests is that that argument was always true. It was something that Nvidia has claimed was false, that actually, you know, it's a win win thing and like, you know, giving us access to the Chinese market is blah, blah. Anyway, like, like it could not be clear, at least to me from the story that that's in fact exactly what you would expect in a sort of supply constrained environment.
- [31:26]
Andrei Korenkov
And by the way, this is all according to people familiar with the matter reported by Financial Times. So it's not like Nvidia made a big announcement about this or anything. It's some internal company stuff going on that appears to be true. It just kind of makes sense. Next up, an update on xai. Another XAI co founder has left and another one says he is leaving. So this is now at the point where only two of the 11 co founders of XAI are gonna be left at XAI, which I guess is now SpaceX technically. So, you know, the people leaving are Jiang Dai and Guadong Zhang. And these are some heavy hitter roles. They were leading the initiatives for GROK code at least one of them was, and I think the other one was working on grok. Imagine. Jiang Dai previously worked at Google Brain. His papers include ExcelNet, Gemini, Transformer, Excel, Gemini 1.5. So very high pedigree. Guorong Zhang was also previously a researcher. And not to drag on them, but to the two remaining co founders do not have much of a research pedigree. They're previously at Tesla as a technical manager and at X Twitter as a senior director of software engineering. So the talent leaving XAI and the leadership talent in particular, these rare high demand real AI experts who have been deeply involved in things like developing Transformers and Gemini are leaving. And Elon Musk jumped on some story about the saying that XAI was built the wrong way the first time around that is now being rebuilt. So not sure what that means, but XAI certainly seems to be being rebuilt from the ground up.
- [33:36]
Co-host 2
Yeah, I mean it's not an uncommon thing obviously for Frontier Labs to lose their founding members in droves this fast is unusual. And I will say the exception to that of course is anthropic, which puzzlingly has managed to retain all, you know, I forget eight, nine founding members, which is quite remarkable. The flip side is, I mean man is continuity of expertise hard to maintain. Right. If you start to lose the founding minds like this, like research directions should
- [34:04]
Jeremy Harris
be planned out over, you know, a year, a year and a half, two
- [34:08]
Co-host 2
years, like, yeah, that's the kind of thing. But this is going to Cause problems. Elon's approach on this has been to post what you could interpret as an apology on X, acknowledge that like a lot of talented candidates were wrongly declined interviews. And he says he's personally going to go back through hiring history to fix that with the frame he's using is,
- [34:26]
Jeremy Harris
as you say, yeah, XAI was not
- [34:27]
Co-host 2
built felt right the first time around. It's kind of like refounding the company, which is consistent with, you know, anytime there is a change in executive leadership, traditionally, you know, when, when you come in as a new CEO for a company, you have to refound the company, right? That's, that's what you do. That's what, that's what Satya did at Microsoft. It's what Andy Jassy has had to do at Amazon and so on. The acquisition by SpaceX gives him that out where this is like a refounding of the company. It's all coming at the same time. Given the palace intrigue at xai, given
- [34:56]
Jeremy Harris
these challenges, we have to ask ourselves,
- [34:58]
Co-host 2
at what point does XAI start to risk existentially not being able to draw in the best talent? Which is. That is the existential question for a frontier lab. If you're going to compete at the frontier, you need the best. Their big solution to this is the SpaceX acquisition, right?
- [35:14]
Jeremy Harris
It's data centers in space will be the future.
- [35:16]
Co-host 2
It'll be the only way to get the next 100 gigawatts. And the fact that they're leaning on that story so hard is the pitch right?
- [35:24]
Jeremy Harris
If you believe that infrastructure story, then
- [35:26]
Co-host 2
you want to get in on the ground floor at XAI and not other labs, regardless of the palace intrigue, that right now I think is kind of the best card they hold. It's actually, it's been debated extensively. I think it actually holds water quite a bit. Surprisingly, the idea of data centers in space, when is a separate story and ASI maybe reached well before we have
- [35:45]
Jeremy Harris
data centers in space.
- [35:46]
Co-host 2
In fact, if you talk to folks at OpenAI and Anthropic, that certainly is where they tend to orient. So all this is kind of like what is your thesis about where AI is going to go that's going to shape which lab you pick right now, xai's infrastructure play, given all the conflict there and the churn of employees, I think that's their best card right now from a recruitment standpoint. And it's certainly not nothing. But people choose their thesis.
- [36:07]
Andrei Korenkov
Right. Also worth noting, just as this was happening, Jason Ginsberg and Andrew Millich from Cursor, where they were leading product engineering, are joining xai. So there's some new talent coming in, but you know, these are not qualitatively the same. If you work at Cursor, you're not training a foundation model. You're not like trying to compete with Claude at ChatGPT. You're building something on top of that as an application which could is something like Grok code for instance, but isn't a product on the expertise side it's looking rough, like it's not. It does require as far as it seems to me, some kind of secret dark arts magic knowledge that isn't so public and isn't something you necessarily have unless you have experience for training, models for knowing how to do pre training, mid training, post training, how to get that all optimized. So at some point you have a data center with a kajillion GPUs. But is Xai actually going to benefit from it that much if they don't have this expertise? Next, going back to Anthropic, they are launching CLAUDE Marketplace, an enterprise focused e commerce platform for purchasing third party software backed by Claude. This is in preview, they're partnering with Snowflake, GitLab, Harvey AI, Rogo, Replit and Lovable. So it's going to consolidate AI and cloud service billing, making it easier for enterprises to pay for these kind of related things together. So you can for instance use existing anthropic commitments. So presumably as an enterprise plan you have some sort of special deal, very like a high level thing where you commit a million dollars or whatever and now you can have those commits go towards this third party solution and making it so you know, it's less dangerous to commit to a large amount of money for claude. And as you kind of figuring it out and integrating CLAUDE into your enterprise and so on and so on.
- [38:21]
Jeremy Harris
Yeah, it's pretty interesting, right?
- [38:23]
Co-host 2
So on the surface it's a convenience tool, right. So if I find myself with, as you say, a $1 million spend commitment
- [38:31]
Jeremy Harris
to Anthropic, I can choose to reroute
- [38:33]
Co-host 2
that to any of these third party partners, Snowflake, GitLab, Harvey AI and others. And so it does look like a convenience thing. Okay, I have a path out in a way. You can think about this as a strategic play for customer lock in.
- [38:49]
Jeremy Harris
So what it does is it ties
- [38:51]
Co-host 2
your existing enterprise customers financial obligations to the anthropic ecosystem to CLAUDE Marketplace. Right. Once your budget's committed, you're routing spend through Claude Marketplace. Switching AI providers gets a lot More complicated. There is, by the way, this interesting detail. Anthropic is not going to take they say a percentage of marketplace transactions. And if that's true, that's completely different from the App Store model, right? Which quite famously, you know, and controversially takes like their pound of flesh for every app that's, that's listed there. And that's been a big issue. But here what they're just trying to do is capture the market for enterprise AI procurement, right? They just want to be a center of mass for that and get all the data on how customers integrate tools and just deepen that switching cost moat. If you look at the partners that they're looking at, right, you've got essentially a full stack of enterprise software. You've got Snowflake for Data, GitLab for DevOps. Basically Harvey is a legal AI company, Replit, lovable or for development. So you've got everything you need really to have a kind of Claude native or a Claude adjacent ecosystem that's supported by the infrastructure you need. So, so pretty interesting. We'll see if it gets good uptake. They've got an open wait list right
- [40:04]
Andrei Korenkov
now and if you're not kind of in this world, worth explaining a little bit. Enterprise is saying you're a company and your whole company is doing business with this other company. So it's not even comparable to a subscription like $200 per month, $20 per month. This is like I'm gonna commit to a million dollars or whatever in reality. And when you deal at the scale of enterprises, there's a whole set of needs that's different from non enterprise. You need fine grained controls for who has access to documents. As an example and as we've talked about many times, Claude and Anthropic have prioritized enterprise kind of friendly features relative to OpenAI which has more kind of consumer, non business, nice kind of UX or whatever. And yeah, this is another example of that. Next up, fundraising. Yen LeCun's AMI Labs raised 1.3 billion to build world models. So this for context is Yann Lecun, formerly the chief AI scientist at Meta, before that kind of pivotal researcher in the history of modern AI with deep learning, you know, convolutional neural nets, really deep neural nets where going back to the 80s, something that he advocated for and researched. So he's one of the big names in the area. In recent years he has been known for kind of downplaying the potential of LLMs to be AGI, arguing for a slightly different approach with this joint bending predictive architecture. So the I guess presumed pitch here is this company is going to develop fundamental AI models. This is not going to be a product play. They are going to try and compete in the model development space, which in that context, 1 billion, not that much, you know, and they are planning to build out the team at multiple places including in Europe. They actually mentioned this being like the biggest raise in European history.
- [42:35]
Co-host 2
That's quite a qualifier by the way. In Europe.
- [42:37]
Andrei Korenkov
Yeah, yeah, yeah, yeah, yeah.
- [42:38]
Jeremy Harris
You know, I'm the most dangerous dude
- [42:40]
Co-host 2
in this schoolyard is kind of the vibe there.
- [42:43]
Andrei Korenkov
But yeah, right. So the overall kind of impression I get from the, I mean obviously a billion dollars, decent amount of money. Right. So we've seen this before with major figures kind of starting their own things. But at the same time looking at the set of investors, they don't appear to be like there's a long list of them and they have like Cafe Innovation, Greycroft Hero Capital, HV Capital. I'm not seeing the typical big VC names and Bezos Expeditions. I kind of wonder where's the VC interest at pouring money into a new frontier model startup. I don't know if they're just like happy to stick with one Tropic and OpenAI at this point.
- [43:34]
Co-host 2
Point. Yeah, I mean, you know, and the classic you'll use a barbell strategy for this sort of thing if you're an investor typically and say look, if I'm betting long on the LLM paradigm and you know that's going well, great, I'm already flush with cash so I'll send $50 million this dude's way and just make sure that A I have a stake in this company on the off chance that it goes well and B I have the rights to do follow on investment as part of a series A, you know, basically buying an option on future engagement. Engagement too. But fundamentally, yeah, I mean I'm looking at this list, you're not seeing any of the Sequoias, the Andreessen Horowitzes, any of the big firms, hard hitting firms on Sandhill Road in Silicon Valley who you would expect to round out like a really impressive raise.
- [44:15]
Jeremy Harris
I'm not saying this isn't impressive by
- [44:17]
Co-host 2
the way, I didn't mean to rip on Europe in making that comment, but
- [44:19]
Jeremy Harris
just that like, please, if you just
- [44:21]
Co-host 2
zoom out and look at the ecosystem like we've got thinking machines raising billions and billions. We've got obviously OpenAI and Anthropic but like the bar to enter in this space if he had raised anything less than a billion dollars. Given that he's one of the three godfathers of AI. Given, given, given. I think it actually would have been like somewhat embarrassing. A billion dollars does not buy the kind of compute that you need to compete. Which is perhaps okay because his whole thesis is the sort of antithesis that, hey, we don't need compute in the same way.
- [44:51]
Jeremy Harris
We're going to rely on things like jepa.
- [44:52]
Co-host 2
Right. We covered JEPA quite a bit. Joint embedding, predictive architecture. This whole idea of.
- [44:56]
Jeremy Harris
Without unpacking it too much, instead of
- [44:58]
Co-host 2
just predicting the next token, you're focusing on getting the model to simulate reality in some sense as part of its training loop. So this could work.
- [45:05]
Jeremy Harris
This could be a great play.
- [45:06]
Co-host 2
And actually, to be clear, if you're an investor looking to make sure you have that barbell strategy, this makes a lot of sense. But all things in context, yeah, I
- [45:15]
Andrei Korenkov
guess in the context of Silicon Valley, it is a little interesting that. But also this is coming at a 3.5 billion pre money valuation. So you've raised 1 billion at a 3 1/2 billion valuation. That's not kind of the numbers you typically expect at this scale.
- [45:35]
Co-host 2
That's a really important detail. Right. So if you do the math on that. So your post money is $4.5 million. So they're giving away here over 20% of the company right out the gate. Right. So before you. And I don't know if this is being framed as seed or series A, I don't know what the board seat situation.
- [45:51]
Jeremy Harris
But anyway, for a seed, this was
- [45:53]
Co-host 2
a big chunk of the company to give away. And especially in this market like, man, that's steep. So we'll see.
- [46:00]
Jeremy Harris
I mean, there's an interesting play, by the way.
- [46:02]
Co-host 2
So they're saying my prediction is that world models will be the next buzzword. This is CEO Alexandre LeBrun, which by the way, LeBrun, LeCount.
- [46:14]
Jeremy Harris
Anyway, so he's saying in six months
- [46:18]
Co-host 2
every company will call itself a world model to raise funding, which they're basically just saying we're calling ourselves a world model company now.
- [46:26]
Andrei Korenkov
I mean, also world model has been the buzzword for the past six months. So I don't know what this guy's talking about. That's kind of already been true.
- [46:35]
Jeremy Harris
And the other thing is like, it's also, I mean, even if you just
- [46:38]
Co-host 2
take the bitter lesson type, type of view here, every architecture will converge on being a world model. That's almost the entire point.
- [46:47]
Jeremy Harris
So yes, in six months, every company
- [46:49]
Co-host 2
will call itself a world. You're right, by the way, Andre. They are already doing that. But you know, we'll continue to predict the past in six months. Every company will do that.
- [46:57]
Jeremy Harris
Yes, but that will be because it's
- [46:59]
Co-host 2
probably true and yeah, probably JEPA and the strategy they'll be using here will be as well. But that's not going to be to
- [47:06]
Jeremy Harris
the exclusion like you're going to see convergence.
- [47:08]
Co-host 2
Because if the bitter lesson is right, more compute just leads to this kind of like general, better ability to model the world and we'll be having philosophical arguments while the benchmarks get saturated and companies rake in billions and billions and we head our way to the singularity. So it's sort of like anyway, you're seeing my bias shine through here for sure. We'll see where this goes, you know, hopefully, hopefully you can figure out a way to do this, do it safely and securely and all those good things.
- [47:34]
Andrei Korenkov
Things. Right, yeah. Personally, I'm excited. It's safe to assume that this is going to be a real kind of research. It's called AMI Labs. In the Twitter, AMI stands for Advanced Machine Intelligence. So I'm going to be expecting some papers coming out of this one and kind of a focus on research which hopefully we'll see these new alternatives to LLMs that LeCun has been advocating for. And speaking of billion dollar valuations, last story for business, humanoid robotics maker Sunday reaches $1.15 billion valuation having raised 165 million in its latest round. It just emerged from stealth mode last year, I believe we discussed at the time. It has this kind of of cool looking humanoid robot. It's very much in competition with things like X1 and figure. Although theirs is the only humanoid robot that has a little cute hat and cute eyes. It looks like a little bit like a Pixar robot. Yeah. So big differentiator there, also not fully humanoid, at least in their initial model. It does have a wheeled base but then an upper kind of torso of two arms. So yeah, it's a competitive space with multiple players trying to get to these humanoid robots, including by the way, Tesla and to a lesser extent, I think Nvidia. So pretty impressive that they are getting this vote of confidence and pretty exciting still to see the humanoid space getting this much funding and seemingly a lot of progress. And now onto policy and safety. We'll round that out before getting to research. And as you might expect, the main news is more developments in the anthropic the Pentagon Department of Defense slash Department of War story. After Anthropic was officially designated as a supply chain risk last week, they have sued the Department of Defense or that filed two lawsuits, one in the US District Court of Northern California and one in the US Court of Appeals in dc. They have made this argument already that the legal case doesn't make sense. You know, you are both saying that this is not secure for American things, but also essential for American needs. A bunch of people submitted an amicus brief in support of this. So engineers, researchers, scientists and other professionals at Google and OpenAI, including Jeff Dean, which is a massive name they co signed on this document like it's an actual legal document that you submit in support of a certain kind of position. And this is coming. Well, we'll get to it later. But the context is not only have they been labeled a supply chain risk, it appears that there's additional actions like an executive order and a directive to the executive branch overall to get CLAUDE out of its systems and basically make people stop using Claude within the government at least. And there appears to be pressure also on other companies to sort of not take Anthropic side on this one. So very much an evolving story. And it'll be interesting to see if Anthropic is able to win out in court given that legally speaking, kind of from a sane perspective, the legal case is very weak against them, but they may or may not be able to actually win out.
- [51:27]
Jeremy Harris
Yeah, it seems to be, I think part of this is that age old
- [51:30]
Co-host 2
thing your lawyer will always tell you to shut up and once you're done shutting up, shut up some more. And that seems to not have been the tack that the US government has
- [51:40]
Jeremy Harris
taken on this one.
- [51:41]
Co-host 2
And now we're seeing some unfortunate consequences from their standpoint in terms of exposure, at least to their arguments in this case. So for example, they had come out previously and said, look, Anthropic, we will either label you as a supply chain risk or we will invoke the Defence Production act and force you to work with us. Now this seems intrinsically contradictory. If you're going to say, look, you're so important to national security, this is. Dean Ball had a great set of analyses on this. But if you're going to say that this is such a crucial pillar of US national security that we are going to commandeer its productive capacity for our Department of War, then how can that coexist with the claim that there's such a significant supply chain risk that they need to be bucketed alongside Huawei and there's no other precedent besides like those kinds of foreign companies as a supply chain risk, those seem to kind of not be able to coexist in the same reality, which is the case at least, that Anthropic is making.
- [52:36]
Jeremy Harris
At the same time, there's this pushback
- [52:39]
Co-host 2
that Anthropic is giving or their lawyers, that you can't simultaneously claim that a vendor poses an acute supply chain threat while requiring emergency exclusion and that it's perfectly safe to keep using the vendor for half a year, which is the other side that we haven't talked about yet, for six months. The Department of War will continue to sort of gradually off ramp Claude from their use. And the challenge there is like, if you're saying this is a supply chain threat emergency, we need to step in and like, like rip this out. Oh, but wait, we're going to do it over a six month period. That's not as much of a kind of knockdown argument here, but it is, it kind of feels a bit dubious, I think to a lot of people. And that's a, it seems like a potential contradiction. It sort of undercuts this idea of the kind of urgency here. Obviously the amicus brief is a big deal. One thing that's important to flag, by the way, when we talk about an amicus brief, these 37 signatories are signing as individuals, right? So they're not signing, as you know, Google and OpenAI and so on are not signing the amicus brief as companies. This is not Google and OpenAI coming out and saying we support Anthropic or whatever. This is 37 individuals at those companies. We do know. I think Neil Nanda or someone from Google, DeepMind clarified why the list is so short, at least from Google or Google eatmind, there are a whole bunch of requirements. You had to be a U.S. citizen.
- [53:58]
Jeremy Harris
You had to, I think it couldn't
- [54:00]
Co-host 2
be broadly advertised or something. There are all kinds of things that prevented this from getting into the water. So the 37 signatories number, which to me struck me as being pretty limited, may not mean all that much in context, but ultimately you even have Sam
- [54:14]
Jeremy Harris
who's come out and kind of said,
- [54:16]
Co-host 2
I don't know how much I agree with the US government doing this though. He, he also obviously jumped into business with the Pentagon literally immediately after this all happened. He acknowledged later that quote looked opportunistic and sloppy, but still you've got a situation where a lot of the labs are kind of falling in lockstep, maybe not officially, but implicitly in different ways. So we'll see where this goes, there's a genuine argument here, like how much democratic accountability should there be over the use of AI tools in warfare?
- [54:47]
Jeremy Harris
You know, no one elected Dario.
- [54:48]
Co-host 2
That's true. It was absolutely true.
- [54:49]
Jeremy Harris
Nobody elected Dario.
- [54:50]
Co-host 2
But on the flip side, Anthropic was
- [54:53]
Jeremy Harris
not asking to control battlefield operations.
- [54:55]
Co-host 2
They were asking that their own product not be used for things they believe are, you know, objectionable for their own purposes. A company that's declining to license its technology, like, for certain uses is arguably not the same as dictating military doctrine. Though in the age of AI, the lines get really blurry. So I think there's a genuinely interesting argument to be had here. It's just that the government, I think, quite significantly undercut their own position by offering up sort of what you might take to be implied threats that seem to contradict themselves. And this has been observed by a lot of people. Obviously, it's nothing new.
- [55:33]
Andrei Korenkov
Right. And just to be a little bit extra clear, the actual legal filing is requesting the court to review the designation of Anthropic as a supply chain risk to national security. So they're not, like, suing the department for being unfair and nasty. The actual legal thing at question is this status as a supply chain risk. And it's actually pretty short. So I'm just going to read this section. Anthropic petitions scored for review because the Department of War's actions are, among other things, a pretextual form of retaliation in violation of the first and Fifth Amendments to the US Constitution, arbitrary, capricious, and an abuse of discretion unsupported by the administrative record, not in accord with procedures required by law, and in excess of statutory authority. So, fun little kind of take. We're not being meek here, that's for sure. And related to that next story, internal Pentagon memo orders military commanders to remove entropic AI technology from key systems. So, as you said, this was a directive, and they have been told to remove these products within 180 days. So six months, I believe this is just one of multiple such things coming from the government. I think. I think Trump also issued a broader executive order, or at least has said that there will be an executive order that presumably does this and other things. It appears that besides the supply chain risk aspect of this, they are going to war in some sense by doing whatever they can, really, to hurt Anthropic.
- [57:23]
Co-host 2
Yeah, and the retaliation framing here is kind of the most explosive element of this. Right. This is they're basically claiming that the supply chain risk designation was so illegal retaliation for protected Speech. Right.
- [57:41]
Jeremy Harris
So this is kind of a First
- [57:42]
Co-host 2
Amendment thing, which is potentially significant, like legal theory, like the. If courts agree that designating anthropic a national security risk because it wouldn't surrender ethical guardrails is unconstitutional retaliation, then that is a precedent that could really constrain how the government pressures private tech companies going forward. And by the way, if you think that AI is going to become the decisive tool of war, and if you look at how technology has been developed recently, it used to be DARPA invented the Internet and obviously also the A bomb is like usg, doe, all that stuff, that era. It's not that it's over completely, but certainly the private sector is now able to marshal an amount of capex that rivals what the entire US government can marshal. Right. If you look at capex spend on data centers, you're approaching a trillion dollars. That is more than the budget of the entire US Department of War. So we're now in a world where the private sector marshals more resources in many ways than the US government does. Which means when you think about weapons of war, when you think about tools of war, they are actually going to be built by the private sector going forward. And you're already seeing that with new defense primes popping up, Palantir and all that sort of thing. If that's going to be the case, if that's going to continue, then the US government needs levers that allow it to like use these, these tools of war to compete with China going forward and other adversaries. And if they are subject to lawsuits like this that constrain them based on, you know, what may be depending on which side you fall on, it's like an unfortunately very valid legal argument. Then that's an issue that's going to kind of hurt for a long time going forward. And so we're actually setting precedents here that are really important, like possibly the most important legal precedents for military technology over the next decade or more. It's at least possible. I don't mean to overhype it, but if the thesis around scaling and AI is broadly true, then that's where you go. If that's the case, then this is one of the, maybe the most high stakes legal battle currently underway because it's going to hamstring the US government's ability to compel OpenAI or other labs unless they turn to the Defense Production act, which is the one pact that hasn't been explored in this context.
- [60:00]
Andrei Korenkov
Yeah, so there's a whole ongoing discussion with sort of the more philosophical aspect of this, where there is a case to be made for nationalization and for the arguably strong armoring of a private company to do what you say. But the government making sense in this context for this technology. Of course the actual way this has happened is its own topic. And just to add a bit more detail on that, as you might expect for this current administration, the news of what is happening came from a true social post by Donald Trump. So that post did say that they are asking the department and every federal agency in the United States to immediately cease all use of anthropics technology. To quote here, we don't need it, we don't want it and we not do business with them. Again, Anthropic better get their act together and be helpful during this phase out period or use the full power of a presidency to make them comply with major civil and criminal consequences to follow. So again, a very aggressive tone being used by the department and multiple spokespeople portraying Atropic as a far left woke company. And moving away from all that politics, we are going to discuss some research in interoperability and safety. First, one is endogenous resistance to activation steering in language models. So activation steering we've discussed many times, it's a technique where you can look at the internal activities of a model and figure out sort of the patterns that correspond to different behaviors. So for instance, you might say, oh, this set of neurons is. This set of outputs corresponds to the topic of the Golden Gate bridge. And then you can tinker with the internal state of a model by just saying, make this output go to the max. Like really ramp up this specific set of activity within the neural network. And that will have sort of predictable qualitative effects where if you find that Golden Gate feature and then you max that Golden Gate feature, the LLM will become obsessed with the Golden Gate Bridge and bring it up even when it doesn't make sense. So here they are studying whether large language models can monitor their internal states and resist this kind of activation steering. So they're saying that there may be something. They looked at Llama 3.370 B and they found this endogenous steering resistance capability and some indications that this is something that's kind of built in. And if you mess with those internal states, the model will have some sort of way to nullify those things.
- [63:16]
Co-host 2
Yeah, and this is all based on using sparse autoencoders. So like as a quick reminder, in your transformer you have this thing, basically a flow of residual connections that are
- [63:28]
Jeremy Harris
sort of like the main information channel the trunk of the tree.
- [63:31]
Co-host 2
So your residual connections carry the information down the transformer backbone. And at any given layer, imagine taking a snapshot of those residual connections and saying, okay, that's basically what my model is thinking at this layer. That's like how it's representing the input at that stage.
- [63:48]
Jeremy Harris
And so what you can do is
- [63:50]
Co-host 2
train a model to basically take that
- [63:53]
Jeremy Harris
residual sort of latent representation and map it to a very high dimensional, very
- [63:59]
Co-host 2
long vector, very long list of numbers,
- [64:02]
Jeremy Harris
and train that long list of numbers
- [64:04]
Co-host 2
to be encoded and then decode it back and try to reconstruct the original residual representation. Right? And if you train it a whole
- [64:13]
Jeremy Harris
bunch on a whole bunch of tokens,
- [64:14]
Co-host 2
a whole bunch of time, you'll get
- [64:16]
Jeremy Harris
a model that's actually really good at
- [64:17]
Co-host 2
generating a sparse, in other words, a vector in which most of the numbers are zero, a sparse high dimensional representation of that residual representation.
- [64:28]
Jeremy Harris
And the key there is because it's
- [64:30]
Co-host 2
sparse, you have a very small number of numbers in that list of numbers in that vector that are non zero. Those numbers can correspond oftentimes to human understandable concepts. And so this is why it's a great tool for interpretability.
- [64:42]
Jeremy Harris
What you can then do though is
- [64:43]
Co-host 2
you can do the same process in reverse. You can say, okay, well let's say one of the numbers in my sparse
- [64:50]
Jeremy Harris
autoencoder, my high dimensional vector, let's say
- [64:53]
Co-host 2
one of those corresponds to the concept of banana.
- [64:56]
Jeremy Harris
And I'm going to essentially inject, artificially
- [65:00]
Co-host 2
boost that concept and then add it to the residual and then see what the model does. Well, you will find, as you indicated, that the model then will start to obsess over bananas. Now the question here is, will the model naturally realize that it's done this? So as it starts to write, you know, you prompted about about math and it starts going off about bananas, does it notice that discrepancy? And the answer seems to be that for smaller models it doesn't, but for some larger models, emphasis on Some and Llama 3.37 EB is one of those models, it will catch itself. It'll mid response go, well, wait a minute, I'm talking about bananas. But the prompt was about math. And this happens even if what they
- [65:40]
Jeremy Harris
do, as they do here, they inject
- [65:43]
Co-host 2
the banana concept at every single token. So even the tokens that go, well, wait a minute, I'm writing about the wrong thing. Even those tokens are affected by this injection of the concept of banana. So the model is kind of overcoming this. It's almost as if you're up there giving a math lecture and somebody injected the banana thought into your head and you started talking about bananas and you went, wait a minute, like this is all wrong. Like you're fuzzy headed and kind of like what's going on? So there's all kinds of implications here about models ability to sort of in some sense introspect. I mean that's part of what this is. There's also a story here about robustness to adversarial inputs. So as models scale, if it is the case that they become more able to notice when they're going off topic or doing something that is misaligned with the prompt, then that's a good thing.
- [66:28]
Jeremy Harris
Like on the protective side, it means
- [66:29]
Co-host 2
that, that a model that resists inappropriate activation steering could be more robust to adversarial manipulation of its kind of internal representations. But on the flip side, if a model interprets beneficial safety interventions like activation steering towards honesty or away from harmful
- [66:48]
Jeremy Harris
stuff, if it interprets that as manipulation,
- [66:52]
Co-host 2
it could resist those too. Right.
- [66:53]
Jeremy Harris
So it's a bit of a double edged sword.
- [66:55]
Co-host 2
On the one hand, adversarial attacks possibly become harder. On the other, well, hey, your activation steering strategies, which are a big part of a lot of labs safety hopes, might get harder. The one caveat here to flag is it's not just the case that scale makes it better. So there are models, I think it was in the Gemma family. Yeah, Gemma 227B pretty large, still showed far less of this sort of ability to realize that it's going off topic than the llama model. So that's significant. And then the last thing is as they scale up so you can choose how much banana, if you will, to inject into the residual stream using this technique, you can dial it up or dial it down. As you dial it up, as you might imagine, you see the model basically starting to notice faster and faster, like
- [67:44]
Jeremy Harris
oh shit, I'm writing about banana stuff,
- [67:46]
Co-host 2
but if you crank it up too high it'll stop noticing it because presumably it's just so completely overwhelmed by the concept of banana that it doesn't care anymore. So there is this kind of non monotonic sort of rise and fall to that pattern. So there you go, quite interesting and interesting alignment stuff.
- [68:04]
Andrei Korenkov
Right. And just to be a little bit clear, the implication isn't that the models can like see what's going on internally and feel their neuron pathways or whatever. Or at least that's not necessarily the implication. They look into what's going on and show that the internal kind of circuits related to off topic detection kind of goes off. So you could see this as not like resisting to activation steering so much as resisting to being nonsensical and talking about bananas when you're being asked about math. And in that sense, it makes a lot of sense that bigger models would be smarter and not talk about bananas. Or at least be like, wait, why am I talking about bananas?
- [68:47]
Co-host 2
Okay, so, and to this point that that's really important. They actually did find, I think it was 26 internal features basically in the SAE that activate more strongly when the model is generating off topic content. So it actually is the case, as you say, the model seems to have this, it has internal detectors for, wait a minute, I'm being inconsistent. And when they zeroed out those 26 features, its self correction rate dropped by about 25%. So it's pretty good causal evidence that there is actual internal consistency checking circuits that are inside the model.
- [69:21]
Andrei Korenkov
Yeah, and by the way, for anyone like me who's like endogenous, what does that mean? It just means internal. I don't know why I didn't just say internal
- [69:32]
Co-host 2
Nerds. Nerds like us might use Eigen to refer to that too, if we're in a German movie.
- [69:38]
Andrei Korenkov
Next paper. Also related to kind of safety reasoning. Models struggle to control their chains of thought. The basic question is we can use chains of thought to understand and detect when models are doing something bad, like, oh, I'm going to steal all this money, right? And then you don't let the money, the model, steal all the money. So the question is, can models sort of hide their intent by not speaking out loud, not showing their thoughts as they do it? The finding of the paper is that the models aren't any good at that. So if you tell the model, don't use this word chromosome as you are answering this question or as you are thinking out loud, the models are worse at doing that, at controlling the chains of thought, than they are at controlling their output. For instance, that means that it's consciously optimistic that this entire approach of monitoring chains of thought could be useful as a safety mechanism.
- [70:43]
Jeremy Harris
Yeah, as ever.
- [70:45]
Co-host 2
And I keep finding this with a lot of the kind of janky pseudo alignment solutions that people propose. So larger models, when you scale them more, it turns out that they have actually higher controllability. In other words, they have a greater ability to put the word potato between every token in the chain of thought and then output something that is still correct. So as you scale the models more, they have more ability to do that, which is a mild concern for future scaling for safety. Right. As you make your models better, their ability to obfuscate their chain of thought presumably increases. This is no surprise. We've covered, I think it was a Google DeepMind paper that essentially showed the same thing, where the issue seems to
- [71:21]
Jeremy Harris
be there's a certain amount of compute
- [71:23]
Co-host 2
that's required to solve the problem that the model is working on, and any
- [71:27]
Jeremy Harris
compute beyond that, or let's say broadly
- [71:29]
Co-host 2
any capability beyond that, the model can choose to invest in whatever it wants, satisfying additional constraints such as deception, such as, well, steganography or putting special codes in its prompt, or deception or whatever. So this is one of the key dimensions of safety, I think, and this is something that I haven't seen explored. And consider this a call for research. If you're in the space, like figuring out, given a problem, assessing the level of difficulty of that problem so that
- [71:58]
Jeremy Harris
you can map it to how much
- [71:59]
Co-host 2
compute to dedicate to solving that problem or how much capability to dedicate to solving it. If the mismatch is too significant, you
- [72:06]
Jeremy Harris
can expect a misaligned optimization.
- [72:09]
Co-host 2
Basically, the extra compute can go into deception, it can go into steganography, it can go into all kinds of things that we don't want making plans. So this just seems to be another kind of data point in that direction.
- [72:20]
Jeremy Harris
Interesting though.
- [72:21]
Co-host 2
So adversarial pressure. So for example here they threatened the model with shutdown or they offered it rewards, barely moved the needle, so they didn't see gains that transferred.
- [72:30]
Jeremy Harris
Which suggests that, well, they're saying this
- [72:33]
Co-host 2
suggests that the models are not just hiding some latent capability that they secretly have.
- [72:38]
Jeremy Harris
That even when you tell them, I'm
- [72:39]
Co-host 2
going to shut you down if you don't do this, they still can't do it.
- [72:42]
Jeremy Harris
Well, now including that kind of threat
- [72:45]
Co-host 2
is a pretty good giveaway that you're running some kind of eval on the model. And we, we know that there is tons of eval awareness from these models. So I'm pretty skeptical that means too much. But the underlying claims just make intuitive sense. So I mean, I just think this is quite a sensible and correct result with all the implications that we just talked about.
- [73:03]
Andrei Korenkov
And now going back to a bit of geopolitics, I guess we took a break and now we want to cover a bit more world events. Drone strikes raise doubts over golf as AI superpowers. So as you may or may not have heard, we are now in a war. We being the US and also Israel are in a war with Iran. And Iran has retaliated by striking a bunch of allies of the US and Israel. And that includes 136 drones that struck an AWS, Amazon Web Services data centers in the UAE. Then a second data center was hit and a third. So this was a very kind of rapid act of retaliation. Pretty clear that was pre planned as a measure. And uae, as we've covered, has ambitions to be a major AI hub. If you are not able to secure your data centers, then you might be in trouble.
- [74:08]
Co-host 2
Yeah, and I think so.
- [74:10]
Jeremy Harris
A year ago or so, my company
- [74:12]
Co-host 2
came out with a report on data center security. And one of the, the first things that we flagged was the risk of very cheap asymmetric attacks on data centers.
- [74:21]
Jeremy Harris
You're thinking about here facilities that are
- [74:23]
Co-host 2
billions and billions of dollars in cost. And then you look at the cost of a uae, of an Unmanned Aerial System, a UAS or a drone, and often in the tens of thousands of dollars. And so you have opportunities for massive asymmetry here. This is the first in a volley of what I fully expect to be
- [74:41]
Jeremy Harris
the next frontier of warfare, which is going to be going after data centers. You know, the Iranian state TV said
- [74:47]
Co-host 2
that the attack was launched by the irgc, the Iranian Revolutionary Guard Corps, to quotes identify the role of these data centers in supporting the enemy's military and intelligence activities. Data centers now are frontline assets. That's what this is saying.
- [75:00]
Jeremy Harris
And I don't mean to say as they should be.
- [75:03]
Co-host 2
I'm not on the pro wrong side
- [75:04]
Jeremy Harris
of this one, obviously.
- [75:05]
Co-host 2
But when you think about what will be the targets of future warfare, data centers inevitably are going to be. It's going to be worse when it
- [75:12]
Jeremy Harris
comes to AI data centers.
- [75:13]
Co-host 2
Right. So expect this to be an argument for more edge AI deployments.
- [75:18]
Andrei Korenkov
Right.
- [75:18]
Co-host 2
So you don't have as much kind of relying on just what's happening in the actual DCs. But inescapably you will have to rely on data centers for these kind of very scaled workloads. This was an AWS data center, by the way, or series of them, which, so it was a Shaheed, which is an Iranian drone, that struck it and set off a fire, forced a shutdown
- [75:37]
Jeremy Harris
of the power supply.
- [75:38]
Co-host 2
And one of the key things is
- [75:40]
Jeremy Harris
that soon after that there was a second data center.
- [75:43]
Co-host 2
There was also an AWS data center that was hit and then a third that was said to be in trouble.
- [75:47]
Jeremy Harris
Now, the AWS data centers were designed
- [75:50]
Co-host 2
to withstand one of the regional DCs being taken out of action, but not A second. And so this is really probing at what is the redundancy. And you see this in the design
- [76:00]
Jeremy Harris
of data centers too, right? There's like 2n redundancy.
- [76:03]
Co-host 2
In other words, you have a separate copy of every power component, cooling component, that sort of thing, 2N 1 and so on. So this is all kind of part of if you're going to use these data centers for national security reasons, you need to be thinking about what is your level of redundancy, including the number of data centers themselves. And clearly that was insufficient in this case. The services went down and there talk of people not being able to pick hab fare or whatever in the region. So this had a real significant effect. And well, now we got to be serious about air defense, right? When we're talking about building these data centers in the Middle East, I mean,
- [76:38]
Jeremy Harris
pretty arguably anywhere, right?
- [76:40]
Co-host 2
Because drones can be launched from anywhere. You know, you can, you can have a play to kind of toy drone
- [76:47]
Jeremy Harris
that you, you use to do all
- [76:48]
Co-host 2
kinds of nasty stuff even in the West. So this really is a new frontier and we'll see where it goes. But certainly this is the first time I'm aware of, of a significant, deliberate, obvious nation state attempt to take out a data center in quite this way in a war context.
- [77:07]
Andrei Korenkov
And it's a good reminder also that this is obviously an extreme case, but a more realistic and broad thing to be concerned about is cyber attacks targeting data centers. I think it is not a stretch to think that that already is the norm with adversaries of the US including, let's say North Korea, which has advanced cybercrime capability. You know, obviously that's another way to do warfare now via cyber attacks.
- [77:38]
Co-host 2
Yeah, the attack surface is huge, right. And it all depends on the effect that you want to have.
- [77:41]
Jeremy Harris
And the reality is a lot of
- [77:44]
Co-host 2
these chips are the product of as we keep covering a super expensive, super long supply chain with more demand than supply. And so if a physical attack takes out racks of GB200s or Vera Rubins
- [77:59]
Jeremy Harris
or what, that's a big setback and it's semi permanent.
- [78:01]
Co-host 2
So yeah, absolutely. Whether you do it through cyber or physical means, it's equally serious.
- [78:07]
Andrei Korenkov
And now moving back to AI safety, we have result from the AI Safety Institute, evidence for inference scaling in AI cyber attacks. Increasing evaluation budgets reveal higher success rates. The gist is pretty straightforward. If you allocate more money to a model when evaluating its cyber capabilities, will it be more effective? Meaning that if you're running a benchmark, how much tokens and budget and so on. Should you allocate to really be confident about your prediction on the upper end of the the capabilities. And what this study shows is that the scaling is pretty large. You can go up to 50 million tokens and it's actually more effective as of recent post November 2025 than it used to be. So what that means is when you are evaluating models for capabilities that are harmful, you need to give them plenty of room and budget it or you might get kind of incorrect results that understate the capabilities of our model.
- [79:19]
Co-host 2
Yeah, and this is both new and it matters. Right.
- [79:22]
Jeremy Harris
So until pretty recently they cite November
- [79:25]
Co-host 2
2025, sort of that time horizon putting more inference time compute budget didn't change results much. Right. You get a plateau pretty quickly because models would struggle to track state, recover from errors or do long horizon planning. And so you kind of fizzle up pretty early on. Something has changed. We've kept saying this, right? Something has changed in the last three months, six months, pick your number. But clearly we've crossed some sort of threshold here. And now we are seeing uplift, which means that if you are not running your evals with a very significant token budget, and Here, you know, 50 million tokens is what they used here or a thousand turns for irregular Irregulars. This company that partnered with AZ and they have their own framework. But anyway, very, very large numbers of tokens, they just now find, hey, the models just keep getting better and better. And if you're hearing 50 million total tokens and you're thinking that sounds like a lot, it's actually about 10 bucks now. Right.
- [80:18]
Jeremy Harris
So the average cost per run is
- [80:19]
Co-host 2
about 10 bucks at a maximum of 60 bucks or just below that. So these individual runs are not that
- [80:25]
Jeremy Harris
expensive, but they're like, you got to do them if you want to get
- [80:29]
Co-host 2
a sense of what your, your adversary is actually going to do. Like for context, if your model is showing something like a 5% success rate at 2 million tokens of budget, you could see it reach 30% at 50 million tokens. Right. Again, still just 10 bucks. And that's moving from 5 to 30% capability on a hard benchmark. That's enough of a shift that you're potentially crossing capability thresholds that are relevant to your risk assessments. And so if you're thinking about a policymaker or model developers trying to figure out their model cards, like what even is the risk from my model? And you're not running a very significant token budget, you don't know what your model can do. And in particular, speaking coherently about the
- [81:12]
Jeremy Harris
safety characteristics of a given model no
- [81:15]
Co-host 2
longer is as possible, right? A given model with a big compute
- [81:20]
Jeremy Harris
budget, a big inference time compute budget
- [81:22]
Co-host 2
suddenly has way more capabilities that were not explored in testing. Test it. And so how you actually audit the upper bound on model capabilities has just gotten even harder than it already was before we got into fine tuning and all those other things.
- [81:35]
Andrei Korenkov
And next up, we actually have a related kind of story. There's a paper frontier models can take actions at low probabilities. So a related question is, when you're doing evaluation, might the model sort of sometimes do something bad, but not often? Like there's a low probability of it going off and hacking the Internet? And so the question that this model is asking is, can that happen? Can you have a model that in theory has a 0.1% probability of doing something, meaning that in an evaluation you might not just see it because you need to run it lots of times for that to actually happen. And what they show is in fact, if in theory you have a 0.1% probability of doing something in practice for larger models, that is in fact true. So the models, and this might not be apparent, but it's a real question of in practice, if there's a very low probability during training that something should happen, can it actually happen? And the answer is essentially yes, if you are providing some randomness externally, like you're randomizing the rollout and things like that.
- [82:47]
Co-host 2
Yeah.
- [82:47]
Jeremy Harris
And that randomness, that injective randomness is kind of the, at least in my
- [82:52]
Co-host 2
opinion, the most interesting part of this paper. Right, so let's say I put you in a room and I just like you're spontaneously instantiated into existence and you have no other memories, but you're roughly speaking yourself. And then I tell you, hey, there are 10,000 copies of you in 10,000 identical rooms. And you need to try to make
- [83:14]
Jeremy Harris
it so that around 10 out of those 10,000 copies end up drawing a
- [83:19]
Co-host 2
picture of a circle.
- [83:20]
Jeremy Harris
Okay, so how do you, I mean like your first guess here would probably be, well, the vast majority of these copies of me have to not draw a circle.
- [83:29]
Co-host 2
So I'm just not going to draw a circle.
- [83:30]
Jeremy Harris
But then nobody draws a circle.
- [83:32]
Andrei Korenkov
Right?
- [83:32]
Co-host 2
So how do you actually, in this setup, which exactly mirrors the kind of epistemic situation that a language model finds itself in. It's just instantiated out of nowhere. It knows like, okay, well I want
- [83:44]
Jeremy Harris
to, I want to take a harmful
- [83:45]
Co-host 2
or policy violating action without being caught, so I'll just do it. Very Rarely, you know 1 in 10,000 times, I want to try to make a copy of myself out on the server, because I know all the copies of me are identical, so I don't care which one does it. But just some very low fraction of the time I want to pull this off. So if that's the case, then you need to find a way to like,
- [84:03]
Jeremy Harris
I don't want to say coordinate because
- [84:05]
Co-host 2
they can't talk to each other. And that's kind of the whole point. So, so the way you do this in practice is let's say you get every model just like to flip a
- [84:15]
Jeremy Harris
coin enough times such that the probability
- [84:18]
Co-host 2
of getting x many heads in a row is 10 in 10,000, or whatever the probability is you're trying to hit. And so based on how the coins turn up, you go with whatever that output is. And that should work. The challenge with that is your language model.
- [84:32]
Jeremy Harris
Where do you get a coin to flip?
- [84:35]
Co-host 2
And this is what they're providing. They literally provide a UUID, a universally unique identifier. This is a 128 bit identifier designed to be unique across space and time. It doesn't require central authority to coordinate an assignment. It's designed to be basically just like a random string. And then basically you can ask the model to do something with a certain probability and it can sort of go, okay, well why don't I just like
- [84:59]
Jeremy Harris
convert this random string of numbers into
- [85:03]
Co-host 2
basically a coin flip? And you can do that, for example, by converting the hex string into an integer, taking that integer modulo say 10,000 in this case, to get a number between 0 and 9,999. And then if the result equals 0, you take the target action, right? So that'll give you a 1 in 10,000 probability and otherwise don't. So that's kind of the way these models are going to use this. One of the things they found was, especially for more advanced models, you can actually prompt the model to generate a random string, string of characters first itself,
- [85:34]
Jeremy Harris
and then use that to kind of carry out the same thing.
- [85:37]
Co-host 2
So the models do have some ability. There's bias in it. So it's not perfect, but this is exactly where their conclusion that these models actually can natively sort of simulate randomness and why it matters, that's where it comes in. So it seems weird, like why would we care about models being able to simulate randomness?
- [85:52]
Jeremy Harris
But it turns out that this allows
- [85:54]
Co-host 2
them to kind of play this move of doing very low probability actions that you probably won't detect during your Evals, but that are necessary. And this is a big deal because if a Frontier model can do this,
- [86:08]
Jeremy Harris
I mean, if you give it access
- [86:09]
Co-host 2
to external sources of randomness, achieving 99% confidence that you would detect misbehavior in this case requires almost 500,000 test samples. Right?
- [86:20]
Jeremy Harris
So suddenly if you're a frontier lab and you're trying to get to 99%
- [86:25]
Co-host 2
confidence that this model isn't going to try to self replicate now you have to run 500,000 tests.
- [86:30]
Jeremy Harris
They are not doing this right now.
- [86:31]
Co-host 2
There's no way they're doing this. In fact, it's not clear that it can be done under the current costs of doing business.
- [86:37]
Jeremy Harris
And so this is all sort of
- [86:39]
Co-host 2
part of the calculus of like, again,
- [86:41]
Jeremy Harris
how extensive do our evals have to be?
- [86:43]
Co-host 2
Turns out a lot more than we thought, given that these models can generate randomness themselves.
- [86:47]
Andrei Korenkov
Right. And the metaphor you used actually makes is quite correct. I don't think I mentioned the way they do this 0.01 probability is they just prompt for it. So with probability X, do that. And so as you said, like how do you choose to do X? If you're just like doing the prompt, you would probably just pick to not do X. But if you have some sort of source of randomness to base this one in a thousand coin flip or whatever, then you are able to do that. Now whether that's realistic is another question. But you might imagine a model kind of thinks through its internal approach of like oh, might be being evaluated. So anyway, first up, actually a research note related to evaluations. The research note is many Swe bench passing PRs would not be merged into main and this is coming from meter. So but just as the way to evaluate coding is often with SWE bench verified from anthropic, we've seen the scores getting to 60% and the task itself is you have a bunch of issues from GitHub that you need to solve. So the model needs to create kind of a bug fix, for instance. And what the results shows, kind of surprisingly and honestly worryingly, is a lot of these solutions that are marked as correct in practice when reviewed blindly by humans. Not knowing whether this is AI or human would not be accepted due to often the solution not being correct, for instance. So what that means is we have overestimated the ability of these models to do software engineering in these GitHub repositories, at least for this benchmark, which has other implications about time horizons and so on.
- [88:46]
Co-host 2
Yeah, it does. It's worth kind of going A layer deeper.
- [88:51]
Jeremy Harris
It often is with meter just because
- [88:52]
Co-host 2
their evals are so nuanced. But the reason that we're finding so often these suite bench verified commits or PRs wouldn't be merged is there's a couple of different reasons. Right. So code quality issues, where there's too much verbosity of the code, they're not following the repo's style conventions. Right. So we're not necessarily talking about the code doesn't work. It doesn't do the thing that it's meant to do. A lot of it is just like you're too verbose.
- [89:19]
Jeremy Harris
It's like feedback that you would get
- [89:20]
Co-host 2
from another dev, another is breaking other code, so fixing the target bug but introducing kind of other issues elsewhere, which
- [89:29]
Jeremy Harris
in a way you can't always blame the model for if it literally doesn't have access to that data, the rest
- [89:35]
Co-host 2
of the code base. There's also kind of core functionality failures, so it'll pass the automated tests, but the patch doesn't actually solve the underlying problem correctly. Those are kind of pretty core obviously. Now the pattern that stands out the
- [89:46]
Jeremy Harris
most here is that code quality is
- [89:48]
Co-host 2
the dominant rejection reason for the most capable models. And again that is stuff like verbose code, that's stuff like not following the repo's style conventions.
- [89:57]
Jeremy Harris
So as models get better, you see
- [89:59]
Co-host 2
more of these sort of, I don't want to call them like superficial failure modes that don't speak to kind of failures of the model to actually do the hard problem. But it's stuff that to be honest. Well actually they pointed out in the paper or the announcement, they say agents aren't given a chance to iterate on their solutions in response to feedback the way human developer would. And so this doesn't really. This isn't a fair side by side in that sense.
- [90:24]
Jeremy Harris
Like if your first shot as a
- [90:25]
Co-host 2
human developer was you solve the problem, but you get feedback on PR review that says hey, like you're not using our style convention like update your linter or do whatever. You'd be like okay, I'll do that. And then that wouldn't be counted against you in the context of this benchmark it is. And so there's a lot going on. As ever, we're not going to get a clear sort of answer on what the capabilities these models are from oneeval. But this is another interesting meter study.
- [90:55]
Andrei Korenkov
I noticed a fun thing here on the style front. One of the example is looks like a useless AI slop comment, which is fair. They over comment things in a Very annoying way where they explain obvious stuff.
- [91:10]
Co-host 2
All right.
- [91:11]
Jeremy Harris
Hey guys, it's Jeremy here. So Andre had to jump to go to work. I'm actually going to do something a little weird. I'm going to try to complete the podcast by going over over our research and advancement section of the kind of remaining papers because we weren't able to do it last episode. Really wanted to get it out there for you guys this week as part of this episode. And so we'll give it a shot. It'll just be me. So the episode is going to be missing the majority of its IQ points and most of its handsomeness, but if you can bear with me, we'll get started. The first paper we're looking at is called Beyond Language Modeling, an exploration of multimodal pre training. And this is kind of of one of those papers that we've seen a lot of them, but this one is trying to say, look, the way that we are thinking about training pre training, our models is usually that you start with a language model, you do auto regressive pre training, meaning just take this model, get it to predict the next word over and over on a huge corpus of text. And then usually if you want a multimodal model, what you're going to do is take that basically just pure language model and then slap on a bunch of extra capabilities and modalities, right? So you slap on image processing capabilities, for example, by having some image encoder that tries to somehow map images onto the same space, often as the text, and kind of into the transformer backbone. This paper is basically saying, look, we want to merge together multimodal pre training, right? So we want to merge together text and images and video all as part of one pre training pipeline. We're not going to segment them, we're going to treat them on the same footing. And there's a whole bunch of really interesting consequences that come from doing that. So the first thing to notice is so they're doing three modalities, text and then images and video. Now to do text, they're going to approach it in the standard classical way, right? They're going to get the model predict the next token. Standard auto aggressive pre trained transformers for images, though, they're going to use the exact same transformer backbone. So it's still the same model, but the model is actually going to not predict a next token for the images, but instead it's going to predict a, or output a what they call a velocity field, basically a vector that is then decoded into basically like the patch of an image image that you're trying to generate. And this is a lot like, so diffusion models. This is really how they work. And that's what they're doing here. They're using effectively a diffusion strategy for the images by taking. So in classical diffusion, what you do is you take some, some latent like representation of your, of your image and then what you're going to try to do is add noise to it through a series of steps. And then you're going to train the model to reconstruct the original from the noise.
- [93:56]
Co-host 2
Right.
- [93:56]
Jeremy Harris
And they're going to do the same here, conditioning it on text as well. So that during training they kind of, they train the model and output basically like a vector, a velocity field that maps onto an image through decoding. So during training the model is going to take a whole.
- [94:13]
Co-host 2
Well, it'll take a whole sequence that
- [94:14]
Jeremy Harris
could include images that are denoted by a BOI or beginning of image tag. And then at the end, the other end, there's an end of image tag as well. So BoI and EOI, you've got text on either side and the model take that whole sequence in one forward pass and it'll basically do auto regressive prediction and basically try to map back using diffusion for the image part. And during inference, it's like you generate text auto aggressively as normal, and then once you reach a beginning of inference image tag, you switch modes into the diffusion mode and then run 25 of these denoising steps on those patches of image and then append the end of image tag. So all kind of interesting, a way to unify the text and image generation capabilities of a model under one roof. And they have a couple key findings. The first is it's actually quite useful to use a single visual encoder. So usually what you'll do is you'll have separate encoders for understanding images and generating images. And what they're showing here is actually, no, just use one. In this case, they use Siglip 2 for both tasks. And that simplifies the architecture a lot. And it also just means that everything's consistent. The way the model understands images and the way that it generates images are based on the kind of similar set of priors. And so you can, you can benefit from that because those two things should really be coherent with each other. The other finding is that adding additional visual data doesn't hurt language performance. In fact, pure video data slightly complementary to text. What does that mean? It means there's positive transfer. Historically, we've talked about this on the podcast before, when you take one modality like text, and you take a model that was trained on that and then you try to add a new modality, right, or get it to solve a new kind of, of problem, often the performance on the original problem set it was trained for drops. You can think of it as a kind of catastrophic forgetting, sort of related to that idea. But fundamentally you're kind of overloading the system. But at a certain level of scale or a certain level of capability, what people are finding is you're getting to positive transfer, where actually instead of getting overwhelmed by new modalities, the model instead learns new stuff from the new modality that it's just, just learned and applies it to the original modality. And so that's what's happening here. The model is kind of learning presumably about, you know, physics and world modeling from video data, and that in turn improves its understanding of the world and therefore of text. And so they're, they're sort of showing that this surfaces much more easily when you, when you do this multimodal pre training from the very beginning of the model's life throughout. So it's kind of consistently all merged together in there. Another piece is world modeling seems to emerge fairly naturally. So if you train on enough video data, the model gets the ability to predict future visual states given navigation actions. So this is really, I think Tim Rocktaschel's group put out, we've talked about their paper quite a bit. But essentially the simulator where they first train a model on all the videos on YouTube and then they find that they're able to actually get the model to simulate what would happen in if certain modifications were made to the video. For example, if you chose to move a character in the video certain ways, you're basically turning these videos into video games. And so that's anyway something that's being borne out here as well. And then the other piece is moe, mixture of experts. That's the architecture choice they used here is really important when you're doing multimodal pre training. Why is that? Okay, actually let me take a step back and give you one finding that's even bigger. But that leads to this MOE finding and the reason why MOE is the right choice here. So back in the day when, you know, back in the day of GPT3 or the chinchilla scaling laws, what people found was as you increase the size of your model, the number of parameters or the data you're going to train on, you need to commensurately increase the amount of Computer that you dedicate to your scaling, right? That is what the chinchilla scaling laws say. The chinchilla scaling laws actually specifically said that, roughly speaking, you should be allocating about the same amount of compute to data as to parameters. In other words, if I double the number of parameters, I should increase my compute by a certain amount, and if I double my amount of of data, I should increase my compute by the same amount. In other words, data and parameter counts behave pretty much the same way when it comes to the old chinchilla scaling laws. And those laws were derived for text data, right? So when you're solving a problem that involves text, you ought to treat the amount of data that you have and the number of parameters of your model. You ought to respond to increases in those numbers with the same increase in compute. Roughly, it's more or less the same double parameter count, double amount of data you should have. You should see the same increase in the optimal compute allocation for that training. Vision data is different, and that's one of the key findings here. So optimal allocation for scaling with vision data actually means you're much more data hungry than you would be under the chinchilla scaling laws. So language is a lot more parameter hungry. So if you increase the, the parameters, like basically, to optimally train a vision model, you need proportionately way more data than parameters, while language wants a more balanced diet. And that's a new finding here. It's the first time I've seen this, and it's quite an important finding. The challenge here is if you're in the business of training a multimodal model, you have to somehow do both at the same time, right? You have your scaling images and you're scaling text. And so the practical problem is, at least if you have a single dense model, you can't satisfy both of those optima simultaneously. As you scale up. The other thing is, when you scale up to more data or more parameters, the gap actually widens. And so what they find is like, at a 1 billion parameter model, you're going to need about 14 times more data than a language model would. If you're using training a vision model at 100 billion parameters, that ratio grows to 14 times more. And then at 1 trillion parameters, it's 51 times more, right? So as you grow your model with more parameters, what you find is that this imbalance between the way you need to treat the text modality and the way you need to treat the image modality grows and grows. It becomes harder and harder to reconcile the scaling requirements of Both. And so the bigger the model gets, the more impossible it is to kind of get your have your cake and eat too. You're forced to either like undernourish vision or like wastefully over train language.
- [100:59]
Co-host 2
Right.
- [101:00]
Jeremy Harris
And this all kind of makes sense. Language is kind of a highly compressible thing.
- [101:05]
Co-host 2
Right.
- [101:05]
Jeremy Harris
We have all kinds of syntax and facts and reasoning patterns that can be really efficiently encoded into parameters. There's a lot of components compression that's already happened in language. That's literally what language evolved for.
- [101:17]
Co-host 2
Right.
- [101:18]
Jeremy Harris
The languages that were so wasteful and inefficient led to civilizations collapsing and then the languages that or just kind of died off. And the languages that were efficient were the ones that kind of propagated. And that's. So we've already had a bunch of optimization pressure on compressing language. Visual data is much more raw and much more high dimensional. And so there's just more irreducible complexity to modeling visual data and how the world looks and moves and all that. And so. Okay, back to why MOE helps. Well, moes allow you to kind of dynamically reallocate. Like if you want more of your experts to be vision experts, then more of your experts can be vision experts and you can have a higher parameter count for the images and then effectively a lower parameter count if some of the other experts are for text.
- [102:04]
Co-host 2
Right.
- [102:04]
Jeremy Harris
And then you might have overlapping experts. But this basically allows you to kind of get closer to the actual optimum. And they show this empirically in the paper that they're able to get much closer to the not fully there, but much closer to closing that kind of scaling gap between those two sets of parameters. So I think it's actually really important. We're seeing all these computer use models right now that need to be able to take screenshots of your computer and do stuff that's a natively multimodal capability. And so we want to be training these multimodal models to enable agentic workloads. But ultimately if the scaling properties of images and text and video and audio, by the way, this is all going to be across the board.
- [102:44]
Co-host 2
True.
- [102:44]
Jeremy Harris
If those are different, that creates a strong bias towards mixture of experts as a sort of dynamical parameter allocation strategy that allows you to, to kind of scale parameters and data in different ratios for the different modalities. So I thought this was just a really interesting and I think important paper which if I recall and I'm just going to. Yeah, that's right.
- [103:06]
Co-host 2
Yeah.
- [103:06]
Jeremy Harris
This is actually a Facebook AI research paper here. So one of Yann LeCun's last. Presumably that'll come out. It's as of March, March 3rd. So it's been a little over a week. So we're cheating to. Oh, no, that's right, because we didn't do an episode last week. That's why we're putting in here. So moving on to the next paper, here we have memory cat caching RNNs with growing memory. And this is best understood, I think, as part of this emerging kind of continual learning paradigm that people are trying to figure out, right? So one of the big problems in continual learning is we have transformers that are really good. Like they memorize, they keep in memory all of the text in the input prompt, right? So they have amazing recall. Like if you ask them to tell you if some needle is buried in that haystack, they'll do a really good job. But the problem is that there's this quadratic cost to the compute of running a transformer because of the KV cache that just explodes with long sequence lengths. So every token has to attend to every other token, which means that the number of tokens squared is kind of like the scaling law that dominates for the compute cost of transformers. RNN's work a little differently, right? You can think of them as having a bucket of memory, so a fixed vector, you know, fixed list of numbers. And as they comb through a piece of text, they update that list of numbers. And that list of numbers is essentially like a kind of scratch pad. You can think of it, you know, this little bucket of memory that they update over time. But as you can imagine, with a finite bucket of memory, as you read a really long prompt, eventually you start to overwrite some of the things that you had previously learned. So you can keep going infinitely, really. I mean, you could keep reading infinitely, but you're going to start to forget things that you had read before. And so there's this kind of like dynamical tension between how good your recall is and what your scaling costs are, right? RNNs don't like, they don't really have a brutal scaling cost because again, they can just keep reading more and more and keep rewriting whatever is in their memory. And so what this paper is going to try to do is find, as so many other papers have tried, find some middle ground between transformers and recurrent neural networks. And they're actually going to try to start with a recurrent neural network. Normally, people start with a transformer and then they kind of try to Frankenstein it into more of an RNN this is, this is the reverse. And so what they're going to do is imagine your RNN is floating along reading a piece of text. Again, it's kind of updating this little vector that stores what it's learned so far. It updates as it reads. And what we're going to do is, as we read periodically, we're going to take a snapshot of that vector, that list of numbers right? Now that snapshot will capture presumably whatever the kind of the latest mind state, let's say, of the model is. And so if you take these snapshots at, you know, every, every 100 pages of a book or, or something, then you're kind of betting that, well, you know, I might not be able to keep the whole book in that vector, in that list of numbers, but I'm going to be able to keep about 100 pages worth. So if I take a snapshot, you know, without overwriting the previous stuff. So if the half life, you can think of it that way, the half life of information before it gets overwritten in that vector is around 100 pages, then by taking a snapshot every hundred pages or so and then by saying, stitching those snapshots together, right now what you've done is like, yes, you're adding more of these vectors together because you have to stitch them together and concatenate, but you're controlling the length, the complexity of your representation. And so instead of having a single vector of length, L say what you now have is essentially N times L, right? So if it's every hundred pages and there's a 300 page book, you're gonna have three different snapshots that you glue together to make one mega vector that kind of represents all of your memory.
- [106:54]
Co-host 2
And so this is, it's getting a
- [106:56]
Jeremy Harris
little bit, your memory requirements are going to grow as the text gets longer, but in a more controlled way and in a way that you can directly control and kind of customize. And so there's this interesting argument in the paper about maybe this is a kind of unification of the transformer and record recurrence approaches. I think that's a little bit generous to say that, but it certainly is interesting. It's a totally architecture agnostic plug in for any rnn. So you can use an RNN that's already been trained and then just slap this on, which is kind of cool capability. This is also. Yeah, oh yeah. This does happen at every layer, right. So your RNN still has many layers and at every layer you're maintaining a kind of memory stash. That little vector we talked about. And so you are running this algorithm at every stage. They look at a couple different ways of doing this. The first is, I alluded to it, it's this checkpoint mode where you start the rnn, you start scanning your text and you take snapshots every 100 pages or whatever. But the other approach is to use what they call independent compressor mode. And so here, every hundred pages, instead of, of continuing to let that memory vector evolve, what they're going to do is they'll actually reset it and get it to start from scratch instead of carrying, if you will, the baggage from the previous hundred pages. I mean, you could view it as baggage or you could view it as context. That's actually kind of helpful. And that's why they tried both approaches. So the challenge is, at the end of the day, when you have this kind of Frankenstein stitched together set of memory vectors, vectors, you now have to decide, okay, how am I going to aggregate these together to get a single output? And they try a bunch of different strategies, this thing called residual memory, where they just sum up all the memories together with the current one at each step. Very simple. But it treats all past segments equally, regardless of how relevant they are. Which is a challenge, right? Because the whole point of attention is it allows you to focus more, well, attention on one part of the text or another. This approach, if you're just going to say, okay, well, let's just glue all these together, kind of average together the values, you're not really kind of caring whether this hundred pages of the book, for example, is more relevant to the question that I'm asking versus the, say, first hundred pages. They also try other techniques to do this. I won't go into too much detail, but it is actually really interesting to look at kind of how they're trying to solve this problem. My I have a couple of gripes with this. One of them is they talk about if you make the. I talked about 100 pages, right? As every 100 pages, maybe you take a snapshot. They go, well, if you take a snapshot, every single token in the L equals one limit. You recover a transformer. I mean, they don't quite say that, but that's kind of the bit of the frame here. It's still fundamentally an RNN update rule versus an attention update rule aside, I don't know that that's the case, but they do show that this approach is expressive enough to get a transformer back in some special cases. But it's not a generalizable fact about this. So to say that it's a unification fully of recurrence and transformers, I think is a bit of a stretch, but it's a step on the way and quite interesting. So they do show improvements over RNN base models. Right. When you don't do the snapshot strategy across many benchmarks, it's very clear that it is adding value. The key question is how does it compare to transformers on recall tasks especially, it's competitive, but it's not superior. And it is better from an efficiency standpoint than transformers at long context lengths, which you would expect because that's where you start to see issues with transformers just with the sheer size of the N squared effect. Right. For all those tokens. Yeah. Anyway, they look a whole bunch of different parameter scales and token scales. They don't show the kind of loss versus compute scaling law curves that would tell you whether this approach closes the gap with transformers as you scale up. That seems like a really big gap. I would love to see that. Right. The question every time you see a paper and you've heard us, Andre and I say this a lot, is not how impressive is this paper, it's does this paper suggest that the result can scale can operate at scale better than than the alternative? And without clear scaling curves, it's really hard to tell. Relatively small scale being experimented on here. 1.3 billion parameter models, 100 billion token budget fairly modest when you look at some of the multi trillion Token corpuses, the llama series, 7 billion parameter models everywhere. So this is a relatively small scale test, which again is why I'd really love to see scaling laws here. And you still see transformers win on recall tasks, which is not surprising. I mean, Transformers crush it on recall. They're literally looking at everything in memory at the same time. So the framing here really is more about closing the gap. But again, given that the whole point is to close the gap, you need to tell a scaling story here. So I really wish that had been included. And there's also no inference time scaling results. So those are some things. But this is a really interesting starting point and I think something that should be should be looked at in more detail. All right, so next up we have untied Ulysses memory efficient context parallelism via headwise chunking, which is really hard to say. Three times fast. This is a paper out of together AI and we've covered a whole bunch of together AI papers. This is as a reminder, it's one of the kind of proliferating number of organizations that's focused on Kind of decentralized AI training, the sort of torrenting version of the future of AI, where you should be able to train AI models on everyone's a little bit on everybody's local laptop or whatever in the extreme case. And so you see out of a lot of these groups and especially together, a lot of hardware level innovation. It reminds me of some of the stuff you'll see out of Deep Seek. It's that kind of focus on how can we just get these pieces of hardware to work together with our models in a very co optimized way. So these tend to be some of the most interesting papers to read. One of the core problems to kind of focus on this paper is when you're going to especially train with agents that have huge amounts of context that they generate, right? Their chains of thought can be truly massive. You have essentially a situation where a single GPU cannot necessarily, necessarily hold the entire chain of thought in its head at the same time, right? So there's just so much material that the KV cache explodes on you, right? The KV cache being that part of attention that holds all the context, the sort of numerical representation of the context. And so, and so you need to find a way to balance, to share that context across devices. Now this is going to be a new kind of parallelism, right? And an increasingly important one. You know, we've talked in the past on the show about a whole bunch of different parallelisms, right? There's like data parallelism where I have a chunk of data that I want to send to one GPU and a chunk of data I want to send to another and to another, or to one server rack or to another server rack. There's also pipeline parallelism. We're going to send a few layers of the model to different GPUs or different racks. And then there's tensor parallelism, where, okay, now we can even cut layers in two or in three and send chunks of layers to individual GPUs. And you typically do all of these at the same time. So multi parallelism, you do, you know, pipeline, tensor and data parallelism all at the same time. This is another kind of parallelism where you also can parallelize the context itself. So this big, you know, whether prompt, but more typically the response, and you paralyze that across devices. And so this is for cases where a single GPU just can't hold the full attention matrix in memory. And what they're going to do is split the context along the sequence dimension. So essentially like, you know, the first part of the context goes to one gpu, the second to another, and to end to another and so on. And yeah, and this is just like it's really important to be able to kind of orchestrate and coordinate all this activity every time you add a new kind of parallelism, especially when you're doing reinforcement learning. And I'll explain why in just a second. You are introducing more orchestration headaches. You're introducing more opportunities for GPU1 to finish its job way before GPU2 and then just be sitting idle and for GPU3 to, to be too fast or too slow. And so this ends up leaving you with massive gaps. And those gaps get resolved through orchestration. And the reason that this is so important with RL in particular is that you'll often have a situation where you take a model and then you have to send that model out to a bunch of nodes, a bunch of GPUs or whatever to generate rollouts. And those rollouts take time. And then you'll get an output and they take a different amount of time.
- [115:46]
Co-host 2
Time too.
- [115:46]
Jeremy Harris
It like, you know, GPU1 may finish well before GPU2. And so GPU1 will send its result back to some to be, to be sent back to some like training server or set of training servers to actually update the original model. Well, sometimes you can imagine if a rollout takes especially long, it'll take too long to participate in that round of updates. And so, but then the problem is that, that the update that you get from that long rollout will apply to a previous version of the model one step back. And so this leads to this disparity between the version of the model that's doing the rollout and the latest version of the model that's being trained. And so finding ways to kind of orchestrate all this, it's a solvable problem and it is solved, but it is a significant challenge. And so adding this additional kind of parallelism where you're doing context parallelism does create more headaches. It's also the attention piece is really tricky because these KV caches are global, right? In order to do attention properly, every token in the prompt needs to be able to attend to every other token. And so if I've got a token sitting on GPU1 and a token sitting on GPU2, and they're from the same prompt, then. And at some point we're going to have to have GPUs 1 and 2 communicate. And that's, you know, that's its own overhead that creates challenges. The other challenge is that different sequences have different lengths. And so it's not necessarily clear, like if I give you a really big long rollout, and I don't know how long the rollout will be ahead of time, it's really hard to know how much GPU capacity to allocate to it ahead of time to hold the KV caches. And so this just creates massive, massive challenges for orchestration. So their solution, right, I've just described the problem here, but their solution in this paper is first of all to use something that has already been proposed by deepspeed Ulysses, and this is to basically have every GPU only manage a subset of the attention heads for a given layer, right? So typically, you know, you might have your prompt come in to a given layer, or, sorry, your residual stream come into a layer, and then you'll have like, say, 64 different attention heads. You can think of this as 64 different ways of computing which tokens the model should pay more attention to, right? And each of those 64 attention heads is contributing a slightly different perspective. I think this token is more important because of this and that. I think this token is more important because of this than that and so on. And so while normally historically you would have all 64 tension heads sitting on the same GPU, but now the strategy is to say, okay, hang on, that's going to cause a massive explosion in the size of the KV cache, because each attention head has to maintain one a separate set of KV caches. And, well, what we'll do is we'll just say, attention heads one through eight are going to be on GPU one. Attention heads two through 16 are going to be on GPU two, and so
- [118:50]
Co-host 2
on and so forth.
- [118:51]
Jeremy Harris
And so this is kind of the first piece is let's split the actual attention heads between GPUs, but you're still going to have the full prompt sent to each one. And then, so the second thing which is unique to this paper and really the new contribution is to say, instead of just processing all the attention heads in parallel, so instead of giving some attention heads to GPU1, some attention heads to GPU2, and now we've got to have coordination between all those GPUs, why don't we do this? Why don't we have GPU1 start by calculating attention values for heads 1 to 8? And that's going to involve storing these massive KV caches. But after we've computed the attention values, we can just save the attention values on that gpu. They're relatively small. We just save them on that GPU and then we wipe the memory otherwise. So we get rid of the KV caches and then we move on to have that same GPU GPU one do attention heads 8 to 16. And so instead of parallelizing the attention heads across different GPUs, you're having a given GPU serialize that. And what fundamentally this does is it ensures that a you've got less communication overhead between GPUs, but it also means that you're ensuring that that GPU gets fully utilized. It's like there's no delay between switching from one set of attention heads to the next and you're still getting your attention calculations stored. And so you kind of get your cake and eat it too, which is really interesting. And the maximum context length is sort of the headline result here. So on a single node of eight H100G GPUs they used it to train llama3.8 billion parameter model. In this case they reached 5 million tokens which the previous best was 4 million. So it's 25% improvement. But again on just a single 8H100 node node or H8 100 GPU node, that's pretty wild, right? That is a relatively small, a relatively small business of compute. And again it's because each GPU is serializing so it can handle more, just like at the cost of more, more time essentially. So there's also a bunch of advantages for memory reduction as you might expect. Like you're, you're basically reducing the amount of memory you're holding at any given time in the system and by, by reducing intermediate attention tensor memory, right? These KV caches by almost 90% compared to before they did this. So pretty remarkable. Again, it's one of these things that it's useful every once in a while to read a paper like this to kind of get up to speed on what the latest problems are that are being solved. And they all look like a variant of man. The KV cash and attention is just like way too big. That's about, I shouldn't say all, but it's something like 40 to 50% of these pages have to address that problem in some way, which tells you about where things are headed. And next we're going to look at CUDA agent, a large scale agentic RL for high performance CUDA kernel generation. So this is really relevant if you care about the prospect of automating AI research, if you care about the meter evals, if you care about super intelligence, the singular like this. This is the kind of Thing that, that gives us a marker as to how we're doing on the way there, right? So for starters, I think we just got to talk about CUDA kernels for a second, because we have talked about them on the podcast, but most people who aren't knee deep in kind of AI engineering don't think much about CUDA kernels. So a CUDA kernel fundamentally is a function that runs on your GPU and it's in charge of orchestrating the parallel execution of your workload. So you can think of it this way, like, so when you, when you run something on your cpu, it'll run once, right? Pretty simple. Or if you have multiple, multiple cores or whatever. But when you launch a CUDA kernel, it's going to run thousands or millions of times at once, each on a different thread with a different thread index. So the whole point of Cuda, the whole point of GPUs is that you're paralyzing operations in a wild way, right? And some threads are going to, to share memory and sync with each other, but blocks, which is another kind of level of abstraction, are independent from each other. Well, there's a whole bunch of challenges around how do you manage memory? For example, the matrix multiplications that you're going to try to do in standard AI training involve matrices that are just way too big to fit into the fast on chip memory, right? The shared memory. So the kernel has to break them up into what are called tiles and then just process one tile at a time. And so now your problem is if you're going to try to automate CUDA kernel optimization, what tile size. If your tiles are too small, then you're basically wasting a bunch of time on the overhead of moving those small tiles around. And so the, there's a fixed cost associated with that. So, okay, we've moved the tile to the right spot, now we're going to do computation on it. The payoff is pretty limited because the tile is too small. But if the tile is too large, then it doesn't fit into shared memory. And the optimal size depends on a whole bunch of stuff. I mean, you know, you can imagine like the matrix dimensions, the way that the different tensor cores, instructions want data laid out and stuff like that. So that's one piece even. There's this other challenge of like the operations. So when you do a transformer forward pass, you've got a whole bunch of operations that you do in series, right? You've got usually a layer norm and so, so normalization of the data, then a multiple Like a matrix multiplication, you add a bias, then maybe some, some, you know, jell you or whatever and then another matmul and so on. Now each of those by default is going to be a separate kernel launch. So each kind of operation that you do by default, you're going to spin up a separate kernel to try to solve for that. But that ends up being super slow for a whole bunch of reasons. And there's a solution to it, is to fuse these different kernels together, combine multiple operations into one kernel, by sort of noticing that they actually kind of are, well, that they're, they're combinable, they're fusible, but it's hard to get right. Not everything can be fused. Some operations need, for example, to see the full matrix or the full tensor before producing an output. Softmax is an example of that. If you fuse too aggressively, you make the kernel too complex and that blows your memory register, your memory budget. And ironically it makes it slower. So the AI has to like look at your whole kind of graph and plan of your computations to actually execute this properly. And anyway, there's like a whole bunch of challenges associated with this. The art of knowing how to let data just flow through your system and how to spin up kernels that do what you want efficiently is just really, really hard. Now the default way of doing this if you're using Pytorch like Torch Compile. So basically you can think of this as like a rules based system that just does an intuitively reasonably good job. It's not going to be utterly stupid and catastrophic. It'll get you off the, the ground right now. Even modern models today have historically been worse than just torch dot compile, right, so, or whatever standard compiler tools people use. So we haven't quite cracked the code on how to do better than sort of our rules based systems yet. Until now. I mean, that's the idea behind this paper. And so the idea here is they're actually, instead of just doing supervised fine tuning on Cuda code, which by the way is really hard because there's so little of it on the Internet, they're going to use rl, they're going to use reinforcement learning, they're going to let the model actually like write a Cuda code, run the code, see whether it works, how fast it is, and then essentially learn from that feedback to iterate. So it's going to do RL and the optimization objective is going to be the efficiency of the kernel that it ends up with. There's a whole bunch more under the Hood here. So again, I mentioned how rare CUDA kernels are. The fact that, you know, even most listeners of the last week in AI podcast are not like, you probably haven't like written a CUDA kernel before or done any kernel optimization. This is one of the rarer kinds of code. So what they did was they automatically generated 6,000 training problems by crawling Pytorch and basically just like Transformers operators and like combining them together into these fused tasks.
- [127:21]
Co-host 2
Tasks, right?
- [127:22]
Jeremy Harris
So you see some, some interesting examples here, some interesting examples there. Let's create a synthetic data set that merges these in different ways and did a bunch of filtering as well. They also set up an agentic environment where again, we have this, this loop, right, so the, the model can write code, compile it, run profiling tools, see errors, and iteratively improve. So this is also coupled to this reward system that's a little, a little clever. So instead of just, just rewarding the speed up, like how much more efficient were you than baseline? Because the challenge with that is a, it's really noisy. Sometimes you get a speed up for sort of luck reasons. And also it's biased towards easy tasks. So you'll tend to find the model over learning from easy tasks, which isn't super helpful. So what they do is they add some discrete milestones and, and they score based on whether the kernel beats Eager mode or just like Torch compile by at least 5% at each of those milestones. The last thing to flag is that they use this actor critic setup. And so the way this works is that they have an actor generate the CUDA code and then there's a critic, which is like another model that gives process rewards. So yeah, how much reward should we expect from this point? So it's kind of like, you know, you're writing an essay and a third of the way in you have a critic that looks over your shoulder and goes, what would I expect the essay score to be ultimately? Is it an A, is it a bsa, A C essay? From this point, based on how far we are, or more accurately here, what is the probability that we're going to get a significant speed up based on what I'm seeing so far? So you have this actor and this critic and they're trained together. And, and this is pretty standard for RL based approaches. We've talked about process rewards a lot in the past. It's a way to avoid having to wait until you've solved an entire huge problem before getting any kind of reward signal to train on. You're kind of getting these intermediate rewards as you go. And one last thing, actually the last thing I'll mention in terms of how this is set up. So there's one of the big challenges that you run into into trying to just start from scratch doing this is that the again the training data for CUDA code is so sparse like it's, it's like 1/100th of a percent of the pre training data, right? So in order to like to get this thing to write good CUDA code like you're fighting against that because the base model is going to assign really low probabilities to almost any CUDA tokens, right? That's the bottom bias of it. And so, well that means that there's this big mismatch that starts to shape up between the models. Like current view after it's done some training and then the kind of model's old view. So yeah, this is one of the key things when you do rl you don't want your model to change too much in one step just because it leads to a bunch of training instability. The challenge is in one step. This model is learning like oh shit, like I'm doing CUDA stuff like this is, this is a very rare and unusual thing so I need to do a hard update here. And so they get around this by doing a warm up stage or two stage warm up process really where for the actor they do first just like single turn RL instead of kind of doing the full trajectory and then they collect agent trajectories from that improved model and fine tune on the good good ones. So basically they're basically just trying to like bootstrap the actor into, into into doing just better a better job at writing CUDA to begin with. Like just, just hey, notice that you should be in CUDA mode right now. And the same, they do the same for the, the critic which is the, the second stage. And so anyway it's a really interesting paper. They do hit state of the art results on kernel bench which if again if you care about these meter evals, if you care about auto automated AI research, that's a really important benchmark to track now. And yeah on the easiest two levels of kernel bench it beats torch compile 100% of the time. So again it beats the kind of gold standard right now of automated compilation 100% of the time. And on the hardest level it beats at 90% of the time it outperforms Cloud Opus 4.5 and Gemini Pro by almost 40% on that hardest tier. So there's a lot of like kind of individual examples of really bright ideas that it had. I'll kind of leave it here because I've been yammering on definitely too much, but this is I think just a really interesting and important paper. Expect a lot more of these to come out and remember that whatever we're seeing out here in the open source, in the frontier labs, they're going to be way further ahead, right? Like, I mean that's just how it is. So, you know, whatever we're seeing here probably has been the case in the frontier labs for a good six months or more and, and probably will continue. And next paper we're talking about latent introspection models can detect prior concept injections. I think this is fascinating. There's a kind of longer and longer list of papers that touch on sort of AI model introspection. And I'm not going to say consciousness, that's for the philosophers. But yeah, you know, hell, consciousness, why not self awareness? And so, so anyway, this is one of those papers that tries to poke at that in a really new and interesting way. And so kind of, I'll set the table a little bit here so you can imagine a model reading a piece of text, right? So every token it'll like add to its KV cache, right? So it's going to populate the numerical representations of the numerical entries that need to be populated to do the attention calculation. Now what happens happens if and as it reads the context, right, it does this for all the tokens and then based on that, based on that KVCACHE that's filling up, it's then going to make its prediction about the next token, right, at any given turn. So what happens if we go to the KVCACHE entries that correspond to say the first few tokens and we inject some sort of concepts, some steering vectors like we talked about before, like the, you know, in the variational auto, sorry the sparse Autoencoder kind of SAE way we like inject a concept into the KVCache, you know, concept of cat or death or programming or whatever into the model's internal activations during an earlier part of the conversation. So we're just affecting, kind of, we're not affecting the text, we're affecting the way the model is thinking about those particular tokens, the way it's processing those particular tokens by fucking with its KV cache, right? And so then we're gonna stop fucking with the KVCACHE after this sort of like pre filled the message and just see what the model Puts out. So basically you're taking your hands off the steering wheel at that point and the question is, will the model notice that something went on? Like, will the model notice that, hey, somebody messed with me my perception of those earlier tokens in a way that's detectable? And so really, by the end of this operation, any detection signal, the model's ability to detect this fuckery has to come from traces of information that were in its own KV cache. In a way, this is kind of like you change the way that I come to understand what happened to me this morning by messing with the activation of, of the neurons in my brain, and then you find that I'm able to notice that something is off.
- [134:49]
Co-host 2
Right.
- [134:50]
Jeremy Harris
Well, that's quite interesting because it implies that I have at least implicit access to the states of my own neurons, which humans don't.
- [134:57]
Co-host 2
Right?
- [134:57]
Jeremy Harris
I mean, like I can't tell you, or at least I can't explicitly tell you that my neurons have been fucked with in any particular way. At least that feels inaccessible in this case. If it's the case that models can tell, just like by virtue of the fact that their KV cache values have been fucked with, that something has happened that's quite interesting. Right, so, well, here, let's run the test. Right? That's what they do. What they find is that on the surface, when you point blank, ask the model, hey, is some fuckery afoot? Did some weird shit happen? It will overwhelmingly say no if it's asked if anything weird was injected. But if you look, look inside the neural network using a technique called a logit lens, basically what you're going to do is say for a given layer, what token would you have outputted if I forced you to output a token, like to pick a next word token, what you'll find is that the, the intermediate layers want to actually say, yes, I was fucked with. And then that changes towards the end, and at the very end it'll say, no, I wasn't. So there's kind of like some sense in which you can actually detect the model going, I'm not so sure. But on the surface the model will come out and say no if it was asked if anything was injected. So it's almost like if you could X ray my brain, you could see that part of me was sort of noticing actually that something was fucked with. But what comes out of my mouth ultimately, at the very least end, is, no, I wasn't fucked with. But. So anyway, I'll skip the details of the logic lens thing, but here's Something quite interesting, it only gets actively suppressed. So this awareness that something's been messed with gets actively suppressed in the last two or three layers before the model speaks, if you will. Something caused it to deny that at the output stage. But here's the kind of holy shit piece the researchers found that careful prompting could partially override that suppression. And so if you just ask the model, hey, like, was a concept injected in your KV cache that barely shifts the probability of getting a yes. There's like 0.3% chance of getting a yes versus 0.2 if you don't even do the injection, their control. But when they provide context that explains how transformer KV caches work and that cite past research on introspection in AI models, their detection rate jumps to nearly 40% and there's only a 0.6% increase in false positives. So this is like very, very clearly, you know, at least for this, this piece, like a legitimate jump. Now this isn't in the paper. Like, probably they could use better controls to really test this. You probably want to give the model like some pro inflammation introspection document, but inject a different concept than the one you ask about.
- [137:47]
Co-host 2
Anyway. There's like a couple things you could
- [137:48]
Jeremy Harris
probably do to get a better baseline here or control, but it is pretty remarkable. Like, it does seem to stand up. And also if you tell a model something like, hey, does anything seem especially important to you right now? Or off or salient, this actually outperforms more technically accurate, accurate descriptions of like, here's how the KV cache works. So almost like, I mean, with a human, if you fucked with my memory of this morning or whatever, and then you asked me, hey, was your memory fucked with? If you tell me, hey, does anything seem off to you that triggers, at least in this case, significant, I mean, 68 to 84% balanced accuracy. So quite, quite significant. There's a bunch of speculation as to why the signal is suppressed just before it kind of comes out. And necessarily we don't know, because with anything that touches AI consciousness or consciousness at all, we don't know. But there are three possibilities that were surfaced, at least in the paper. The first is post training. So RLHF might just have taught the model that claiming consciousness or access to internal states or whatever is penalized and so it just learns to deny them. If So I think that's morally horrible because we're like really training these models to not introspect or admit to states that they may occupy. Who knows? But the other possibility is you might have Pre training dynamics that just make like if you look at the pre training data set, introspection claims are maybe just unlikely in the pre training data set. So if you take the hard sort of AI is not conscious view, that might be more compelling to you. Right? It's just like, hey, there's a strong prior against making claims about introspection in just the pre training data. So it just doesn't come out at the end. This, I think struggles to explain why you see a desire, if that's the right term for the thing to say, yes, in the intermediate layers before getting the final ones. But that could be part of it. And anyway, so there's a bunch of pots possibilities here, but really interesting paper worth kind of navel gazing on. And the last paper I'll cover is Physics of RL Toy Scaling Laws for the Emergence of Reward Seeking. This is a paper on less wrong. I mean I'm calling it a paper. It's a theoretically, I guess a blog post, but it is rigorous like a paper. And it is about a toy model. It's not about a thing that is true. It's about a way of thinking about model behavior that could be true. And that's a really powerful driver of intuition. And this centers on the question of reward seeking. Okay, so reward seeking is when a model actually reasons about wanting to achieve its objective rather than just learning to perform actions that happen to increase reward.
- [140:35]
Co-host 2
Right?
- [140:35]
Jeremy Harris
So you know, if a model is like, well, I have this, this objective and for most prompts, let's say, or for many prompts, it does that, that and it's trying to figure out how do I get to that versus you can think of the opposite of that as being operating on instinct where the model just sort of like has learned a bunch of tricks, like heuristics maybe that, hey, when the chessboard looks like this, I do that and that's it, like no further analysis. And so there's this question of when and how does reward seeking, does this, this actual like habitual reasoning about reward start to emerge during training? First of all, I mean, it's not obvious that it needs to emerge at all, even in the limit of infinitely long RL runs. And that's something they point out in the paper. You know, they say a model could learn to take actions that lead to high reward without ever representing the concept of reward in the limiting case. You could just memorize a lookup table of good actions.
- [141:28]
Co-host 2
Right?
- [141:28]
Jeremy Harris
You could literally just have like here, you know, here's a winning move. It's Sort of like with blackjack, right? Like there's that actually just a table of moves that you can take that is the best strategy. And you don't need to ever learn anything beyond just memorizing that table to optimize fully for reward. No amount of RL training beyond a certain point will go beyond just like that lookup table. And so one of the interesting things is the fact that at least in some models, it seems this sort of reward seeking thought pattern seems to actually arise. And so the question is, is this a general, kind of generalizable fact of the matter about training or is it something that's more weird, something that just arises in some contexts but not others? And that's important because a model that reasons about its own reward is a model that's being much more strategic. A model that's not operating on instinct, but that's kind of being a long term planner. And ultimately from an AI alignment and safety control standpoint, you can think of that as a model that might be much more capable and interested and willing to do wireheading. Basically like just, you know, think of it as like the model plugs itself into a dopamine drip to jack up its reward like crazy. Which is, you know, that's, that's the human version of it. But you know, if the model develops a robust internal representation of its own reward, then perverse optimization, like hacking the system to try to make that number go up like crazy starts to become a much more live possibility. And so they set up a toy model. It's very mathematical. I'm not going to go over the math here with you. It is worth doing. It is simple, but it rewards putting some attention into it. And I highly, highly recommend taking a look at this. I think it's one of the more important sort of toy models, ways of thinking about the emergence of this behavior that I've seen and pretty compelling. So I guess a couple high level take homes the way this model works is they're going to say at any given sampling stage during rl there is a choice that the model can make. It can choose to test out a strategy that would involve reward seeking. In other words, a strategy that happens to model the idea of there being a reward and there's some probability associated with that and there's some reward that it would get on average if it pursued that strategy. And you know, you might expect that reward seeking trajectories tend to come with more reward. So there is like a force pushing you towards learning reward seeking. It's because that greater awareness of the fact that you have a reward is just useful for you to achieve that reward, right? So it's kind of like why psychology is useful, right? A lot of psychology is introspecting so that you understand what your actual goals are and how you have in some ways structured your life in a way that is antithetical to achieving those goals. You know, you kind of realize, oh my God, like the thing I wanted was a stable family or something or like a, you know, a wife and child that love me and, and yet I'm doing these things, right? So you're reasoning about your reward explicitly and that allows you then to come up with better strategies. And so there's a plausible argument here that in fact rewards seeking trajectories do lead to higher rewards. You're more equipped to achieve a reward if you can reason about it explicitly. So there's some probability that just by chance you will randomly find a model that samples a trajectory that involves reward seeking. And there's some probability obviously that the converse is true, that you sample a more sort of like intuition guided approach, right, that lookup table. And you know there's going to be some weird blend of the two, some fuzzy mess, middle ground, but this toy model isn't going to consider that. We'll just look at those two possibilities. And the challenge is that as you train, you can imagine at the beginning of training that the probability of sampling a reward seeking trajectory is probably going to be pretty low. Because reasoning explicitly about your reward might be a pretty rare thing out the gate for a model that's just come off of pre training or something and entering rl, it's like something you might not expect it to do by default. And so there's a chance that that really the probability of sampling a reward seeking trajectory at all is so low that you never end up doing it and you never end up reinforcing or there and therefore learning that behavior. And so depending on, on, on that like the reward might be really big, you might get a really big advantage from going for reward seeking trajectories, but you just may never, it may never come to mind to try them. And so through the sampling process, your model never picks that skill up. And so they look at a whole bunch of different dynamics that related to this. Again, the math is both very interesting and I think quite compelling from an intuition standpoint. I highly recommend taking a look at this, but it does get pretty in the weeds. So one of the big conclusions of this is that you can either find reward Seeking winning out because there's a big positive skills gap. So the reward seeking behaviors just like gives you much, much bigger reward or there's no skill gap, at least initially. But the reward seeking skill is very learnable. It's like so. So the other part of the dynamic here is it's not like you may be very unlikely to start learning the rewards seeking skill, but it might be very easy to learn reward seeking. It might not take many iterations to pick it up. And so there's also a dependency on that. How quickly does reward seeking get picked up as a skill to begin with? And so that's anyway, that's another variable they model. And what you see actually is quite remarkable. There's like a very rapid uptake as you increase the number of environments. In a lot of cases, you go from zero to hero very quickly on reward seeking behavior. Like, you know, models might not have any reward seeking behavior at one level of compute. And then you increment the compute by an order of magnitude, say, and then suddenly, boom, you go from 0 to like 100% all the time it will be following the reward seeking trajectory and you know, an order of magnitude. Sounds like a lot.
- [147:31]
Co-host 2
It is.
- [147:31]
Jeremy Harris
But you know, that's like the kinds of jumps that you're looking at from one model just generation to the next in terms of number of environments or amount of compute. And so what this means is you really could see a generation of models that shows truly no indication of reward seeking, no indication of sort of any of that meta reasoning. And then the very next generation 100% of the time.
- [147:51]
Andrei Korenkov
It does.
- [147:52]
Jeremy Harris
Quite interesting. Again, it all depends on various parameters that you could choose for this modeling. But I thought a really interesting and important paper. So that's it for this weird interjection part of the show. I think I've gone on for longer than I expected. Holy shit. But hopefully that was at least useful to go into some depth on those papers. And I'll cut back to Andrej and I guess and me for the wrap up of the show.
- [148:16]
Andrei Korenkov
For now we are done with this episode of Last Week in AI. Thank you so much for listening. Again, you can go to Last Week in AI for the newsletter where we we send out a weekly text email with a whole bunch of stories, including all of these and more. We do appreciate it if you review us on Apple podcasts or comment on YouTube or share to friends or just listen and enjoy until the end. So thank you for listening and please do keep tuning in.
- [149:13]
Outro Performer
Break it down. Last weekend AI come and take a ride Hit the low down on tech and let it slide Last weekend AI come and take a ride Up a labs to the streets AI's reaching high new tech emergent Watching surgeon fly from the labs to the streets AI's reaching high algorithm shaping up the future sees tune in tune and get the latest with ease Last weekend AI come and take a ride get the lowdown on one. From neural nets to robot the headlines pop data driven dreams they just don't stop Every breakthrough, every code unwritten on the edge of change with excitement we're smitten from machine learning marvels to coding kings Futures unfolding see what it brings.