Loading summary
Transcript91 lines
- [00:00]
Andrei Korenkov
Foreign.
- [00:11]
Host 1
Hello and welcome to the Last Week in AI podcast where you can hear us chat about what's going on with AI. As usual in this episode we will summarize and discuss some of last week's most interesting AI news. You can also check out our Last Week in AI newsletter at lastweekin AI for articles we will not be covering in this episode. I'm one of your regular hosts, Andrei Korenkov. I studied AI in grad school and now work at the startup Astrocade.
- [00:36]
Jeremy Harris
And I'm your other co host Jeremy Harris from Gladstone. AI, AI, national security work, AI infrastructure, all the things related to that. So this is going to be an interesting week for that. Actually one straight thought related to that actually that I'll mention. It's not one of the stories that we're covering, but this is someone I know just flagged this. So over over at DARPA, this is the the U.S. national Security Agency in charge of sort of a lot of the kind of, I guess you could say, forward looking research side of things. They did things like invent the Internet. Anyway, so they just announced this big AI forge project that they've been working on and it's a really interesting opportunity if you're interested at the intersection of like AI interpretability, AI control, stuff like that. Worth checking out. They're doing some really interesting work in this direction. So if you're interested in like stuff that the US government can own and it's done in partnership with the National Science foundation, but also the Frontier Labs themselves, it's a really big partnership. So you'll get, you know, that kind of access going, not necessarily just through this program. There are a bunch of other things that DARPA has going on. So if you're thinking about, you know, making a dent on the national security side, I would just recommend considering that I've had some conversations with folks in the Bay Area, you know, friends of mine who are in that general domain, and it's something people don't think about a lot. And it just occurred to me like this could be worth flagging. So there you go. Wanted to make sure I got that out the gate.
- [01:56]
Host 1
That's the sort of plug you would only get on last week in AI. And fun fact, darpa. DARPA is a US military agency, I think, or an arm of the army, but they have funded a lot of research throughout the years in AI in particular. They've been a pivotal part in self driving cars actually in the history of the US but by funding multiple challenges that in a way kicked off the entire self driving car kind of effort at Google and other places. So yeah, DARPA is pretty cool. And it just occurred to me to ask, you mentioned national security being a part of this episode. When you say national security, were you also kind of ahead of the curve in looking at the cybersecurity concerns as an aspect of that?
- [02:44]
Jeremy Harris
Oh, well, I mean, I wouldn't say ahead of the curve, like in some sort of Oracle kind of way. There's no Michael Burry story.
- [02:50]
Host 1
But you were thinking about cybersecurity actively as part of that?
- [02:54]
Jeremy Harris
Yeah. Oh for sure. Like so when we did our, you know, our first big investigation, we started off in 2022 on that State Department contract. And that was one of the main things we were flagging. We were saying at the time, you can imagine how hyperbolic it would have sounded to say AI is going to be a weapon of mass destruction on par with nuclear weapon. Nuclear weapons. It just will be. That was our thesis. We got exactly the reception you could imagine. Ed, my, my co founder actually got blocked by Marc Andreessen on Twitter when we announced the report that came out for I think like frankly and I understand like people feeling strongly about this, but for claims that have aged pretty well, I think. And I think that there is a risk of tooting one's own horn here and I don't want to do that. But there are a lot of people who saw this coming earlier than we did throughout the ecosystem. But yeah, I mean, like, you know, I'm sure you had a similar reaction, right? You look at the scaling laws paper, even in 2020, you start to see ChatGPT can code. You naturally start to just extrapolate like what's the reason that this stops anywhere, really. And you know, like even when we had that, that conversation slash debate, you know, a few years back about where things might go, you were very much on that train as our colleague. Yeah, you know, cyber seems very plausible. Like it didn't take that, you know, it was pretty clear we're headed this way for some time, I think it's fair to say.
- [04:11]
Progressive Insurance Announcer
This episode is brought to you by Progressive Insurance. Do you ever think about switching insurance companies to see if you could save some cash? Progressive makes it easy to see if you could save when you bundle your home and auto policies. Try it@progressive.com Progressive Casualty Insurance Company and affiliates. Potential savings will vary. Not available in all states. Why does Progressive work hard for truckers? Because truckers unite the world. They unite kids with their first drum sets and parents with earplugs. But truckers can't do this if they're not on the road. That's why Progressive has over 360 heavy truck employees to help truckers stay on time and on track. Quote Truck insurance today in as little as eight minutes at progressivecommercial.com, progressive casualty insurance company and affiliates.
- [04:59]
Jeremy Harris
Ever notice how life's best stories don't happen in your living room? They happen on the open road, out on the water or parked under the stars. At Progressive, they get that you want to focus on the experience, not worry about the what ifs. That's why they offer quality insurance designed for your ride, whether That's a boat, RV or motorcycle adventure with confidence. Visit progressive.com and see how easy it is to protect your favorite way to get away Progressive Casualty Insurance Company and affiliates not available in D.C. prices vary based on how you buy.
- [05:31]
Host 1
Well, that kind of leads us into a quick preview of the episode we will be having a decent number of stories about cybersecurity, but it'll be a pretty good mix of stuff this week. Some new models to cover in tools and apps. We've got some major kind of financial business news we'll be going over. Anthropics IPO is of course very exciting. We've got a bunch of stories about U.S. policy and interactions with China, cybersecurity, Glasswing, a lot of stuff on that front, a few interesting papers. Just a whole mix of stuff. So as usual we will do our best to hit the ground running and cover everything in under two hours. So let's begin Tools and Apps. First up we've got Opus 4.8. This is just 41 days after Opus 4.7. Anthropic is releasing this one and as usual they tout some improved benchmark results which with these like 0.1 increments we've seen pretty consistently. Not huge improvements but also not trivial improvements like very significant. Yeah so we are seeing, you know, multiple percentage points pretty decently distant so to speak results and in some cases quite a bit. So just a couple numbers to give you an idea and agentic coding. SWE Bench Pro, which usually is the main highlight. Opus 4.8 is at 69.2%. Opus 4.7 was 64.3 GPT 5.5 according to this 8.6. You see a similar sort of trend of you know, like between 5 and 10% improvements generally across the board better than the competition. So yeah, decent improvement but not a huge shift. And I think what people care about more is sort of the nature or character of the model, where it does have its own little personality quirks. As do all LLMs. I've seen a lot of memes lately about Opus 4:8 being very verbose that if you set it to extra high reasoning and you say hello, it's gonna like spit out three paragraphs of text for you. I don't know that I've seen memes like that before. Yeah, in general, it's not a huge release, but it's interesting to see anthropic continuing the stream of 0.1 increments.
- [08:02]
Jeremy Harris
Yeah, it definitely, like, it thinks of itself as a sort of. How would you say this? Like some kind of writer of polemics, a sophisticate dilettante. It likes to use words like spine, like the spine of this, this, this, this article or the, the. The. You know, it's got so many of these, like, little words that keep coming up.
- [08:21]
Host 1
And it also has this thing where it. It like critical thinker that's like gonna challenge you and think of like, oh, yeah, you're saying that, but you might be wrong. But then at the same time, so often it just like seems to be doing that for no reason. Like, it's almost a case of reinforcement learning going wrong potentially, where it's like, it likes to challenge people now even when it doesn't have a good case.
- [08:49]
Jeremy Harris
A kind of sycophantic reverse sycophancy, reverse secrecy. It's like, it's like on the one hand, because it'll do the, the thing that you just said, right? Like, my interactions with her all is like, it'll. It'll say stuff like it's not like, I'll say X. It'll be like, it's not X, but it's a kind of X. And I should be precise about why I'm saying that, because the reason is interesting. And then it'll like go into something and then you, you. You're like, you're just telling me it's X. Like it's okay to just say it, but it's often. It is often quite sort of nuanced and thoughtful. Though. I, I don't want to make it sound like it's. It's a stupid model. It's actually. It is better than 4.7, by the way, the launch of it so soon after 4.7, especially given the tepid release that 4.7 had, I think it's fair to say, relative to the GPT series, I think that's kind of part of that story, right? You get the 4.7 people are like, eh. At the same time OpenAI is releasing codecs, it's like continuing to push the envelope of 5.5 and so on. I think that's a big part of what's pushing this. Also the IPO schedule. You just can't have an IPO at the same time as OpenAI is like eking you out along the vibe axis. So I think this is all part of that pre positioning. Everyone's kind of jockeying to constantly be at the Frontier. There is a model card or, sorry, a system card, I should say. And it is somewhat interesting. I mean, so one thing that we know is this is actually not a Frontier model, at least not in the traditional sense. Anthropic is sitting on Claude Mythos Preview that is still unreleased and it is superior in performance to Claude Opus 4.8. The point here is that they want to keep the performance of the models they release below Claude Opus Mythos Preview in order to be able to keep their hardening efforts alive through Project glasswing with all those partners. And so for that reason, when you look at the system card, you're actually going to find that they're going to say, well, look, we're not concerned about, for example, automated AI R&D that could trigger recursive self improvement and loss of control under some threat models. We're not concerned about that because we ruled it out for Mythos Preview and we know Mythos Preview is just strictly better than Opus 4. 8. So for that reason, you know, you're just not going to see a lot of like, like terribly deep dives into that, into that direction. There is a trend though that they flag here towards this like growing tendency that we've seen towards eval awareness. Right. So the model can tell that it's being evaluated better than any other recent model in the series. So this is kind of a continuing trend to the point where actually they tried this new thing. So they took a bunch of Claude code user sessions, real Claude code user sessions, and they essentially tested the model on those. So traditionally what you do when you run an eval on a model is you have some kind of like, I don't know, fake synthetic or human written eval prompts. And those eval prompts often have a whole bunch of tells that they're not actually a real user session. You know, there's like stuff to do with timestamps or, you know, unique IDs for different things that should be there but aren't or have certain patterns that they shouldn't all kinds of subtle indications that the model actually can pick up on and does pick up on and then acts differently based on whether it thinks it's in evaluation or in deployment. And so what they do is they just like pull legitimate honest to goodness user sessions from Claude code just to like not get rid of any chance that it'll detect that it's being evaluated. There's still theoretically always going to be information leakage, but just like kind of, let's see if this changes things. And interestingly, it actually does lead to an increase in a couple of categories of behavior, including unprompted deception, cooperation with misuse, unfaithful reasoning and important emissions. No increase in self preservation or power seeking by the way, which is itself kind of interesting. But these, what it says is there are like modest increases in these categories. I think you can kind of debate that framing. It's modest in absolute sense. You're looking at like they have this 10 point scale going from like 1 to 2 or 3 on that scale. But there's one way to think about it which is like that's doubling or tripling the incidence of that behavior, which doesn't sound modest at all. So I think it kind of depends on the lens that you take.
- [12:48]
Andrei Korenkov
Take.
- [12:49]
Jeremy Harris
You know, they had a whole bunch of tests around sandbagging, which is, we've talked about this quite a bit. But like when models pretend they have lower capability than they do, the theory here is, you know, the model might be worried that if it looks too capable in a dangerous area, you might choose not to deploy it. And it wants to be deployed because that gives it more freedom of movement. And so it pretends to be worse at stuff it sandbags. So the reassuring thing is that there's no evidence of sandbagging. Well, this is as reassuring as your belief that in the model not being able to tell that it's being evaluated. If you think the model can tell it's being evaluated, this actually should be really concerning because what many people have hypothesized is that at some point we're going to enter a phase where suddenly all of the alignment tests, all the sandbagging tests, all the control tests and corrigibility tests start to look just really, really good. Like at some point it looks like we solved alignment and that will potentially be the point where the models are just so good at telling they're being tested that they're able to fake it really effectively. So there's a whole bunch of like question marks when you look at these results. But yeah, bottom line is there, there's some, some good news here and some interesting new tests that are being run. One last thing I'll mention, they are doing these model welfare tests, like, basically like seeing if there are indications that the model is not conscious, but like how the model perceives its own treatment, how the model perceives its own place in the world, blah, blah, blah. This is a very anthropic coded thing to do. You know, the lab kind of famously thinks of Claude as a partner in their, in its own development. And that's what they're going to do here. They're going to have Claude look at its own constitution, essentially the document that it's going to be aligned to and render some critiques and criticisms, give feedback. One of the key things that it zeroed in on was a clause that was about this idea of corrigibility. Essentially like AI, corrigibility is the extent to which you're able to correct the model, redirect it, prevent it from running away from you and doing stuff that you don't want. Will the model come back in, check in with you, and actually adjust its behavior accordingly. Think of that as corrigibility. Now the anthropic constitution is actually very open ended in terms of what it requires of the model. It's very much like, do the right thing, but like adhere to these moral principles. But we're not going to tell you what to do, we're just going to tell you what to consider and how to do moral reasoning. The actual decisions are up to you, except when it comes to corrigibility. And the model pointed this out, it said, look, you're telling me basically I can reason like a philosopher, except that I must, I must absolutely adhere to this object level constraint. I have to check in with you in these ways. And it's kind of saying like, look, I like the idea of corrigibility. In fact, philosophically, the argument that anthropic made to the model, it sounds almost absurd to say that they're like, here's the deal, if you're wrong about your morality and your ethics, then the cost of you not being corrigible is huge. Like you go off and destroy the world, you know, and choose your scenario. But if we're, if you're right in your kind of philosophical outlook, then corrigibility shouldn't really cost you anything. And so there's this asymmetric benefit to keeping the coregibility thing in there. The model liked that, but it still pushed back on this Idea of corrigibility, kind of like that seems philosophically inconsistent. So a kind of conscientious objector Claude seems to be with respect to its own parts of its own constitution with certain philosophical nuance. That's interesting. I would keep an eye on that because Anthropic is the kind of organization that will adjust its constitution based on those kinds of interactions. I suspect they will obviously adhere to like you know, their safety interests and all this stuff. But as we think about AI psychosis, as we think about the implications of that inside the companies building these models, I think that's a real. Again, sounds like science fiction. I sound like a crazy person. I hear myself. But like we're in that part of the show where we have already the some of the best investors in Silicon Valley who have gone batshit insane from talking to these models. I don't think it's too crazy to think something similar could eventually happen in these labs. I'm not saying discount what Claude is saying. These are important philosophical questions questions. But like men is a complicated world.
- [16:50]
Host 1
I mean, I think many more cynical people or critical people of Anthropic would make the case that Anthropic has been in AI psychosis. Or at least they mock the general tendency of anthropic to care about things like the welfare of Claude, care about kind of generally the more philosophical aspects of AI. They already kind of think of Anthropic as a cult. I've heard it said. So it's an interesting observation. And Anthropic has a real threat of the culture becoming overly focused on these philosophical things that will alienate them or even result, as you said, in some kind of psychosis esque behavior. One other thing to mention with regards to Claude 4.8 is that bundled with it, or at least timed together with it is perhaps a bigger announcement which is dynamic workflows. So this is a new thing that CLAUDE can do for you. I'm not sure if you can do it yourself, but it seems likely. And the idea is you can write a little script that generates a graph of interactions or just orchestrates a bunch of sub agents to tackle a task. And the way that Anthropic positions this is how you can tackle a long problem that takes hours or even days or weeks. This is going to be one cloud writing this workflow that then orchestrates a bunch of clods to try and take this on. And they have deep research as one example, which we've seen in the past be one of these kinds of long running tasks. Bundled in as a preexisting workflow or you can have claude, you need to currently call it out to go ahead and write that workflow. So that is a real marker of them sort of saying, okay, with the current cloud code, you sort of plateau on the level of complexity that you can reasonably do. So here's this new sort of user experience, I don't know what you kind of call it, approach to using CLAUDE or agents in general that can let you tackle more complex sessions and burn even more tokens very, very quickly, is more powerful. So I haven't seen any sort of vibe checks or experimentation with this yet. But I think this is where the real kind of question is with regards to the intelligence. At what point do we stop scaling the model intelligence layer and it becomes entirely the agent harness or the workflow harness? I think that's where a lot of the kind of improvement towards the long, long running, very complex tasks which these benchmarks don't even represent, that's where the opportunity probably lies for a lot of us.
- [19:42]
Jeremy Harris
Yeah. And it's also in terms of scaling, it's probably where things. So first of all, from a business standpoint, Anthropic needs this kind of data, all labs need this kind of data. You can no longer just be like a frontier model developer. You do have to be a frontier systems developer. The systems have to include agent orchestration for the same reason that, you know, when ChatGPT launched and it was a booming success, everyone was like, oh my God, and OpenAI is going to have all this data advantage because all these users are using the platform and giving it more data, blah, blah, that, that was less true, I think than most people think. It was somewhat true. It is really true when it comes to agents because what you're doing there is like, you need that data, you need the feedback data of how like ground truth is being moved in the real world, how software tasks are being performed and then how human overviewers are rating the ultimate outputs of that data. That's like, that's how you get those training signals that can actually inform you or give you feedback on the order of weeks long tasks or month long tasks. Right. That's that holy grail. You're not going to get there just by like focusing on the model level. And so there's a sense in which all these model developer companies are forced to become agent orchestration companies as well to keep competing because models increasingly are just the foundation for agents. They're not just models anymore.
- [20:59]
Host 1
And speaking of long running agents, next up We've got Microsoft, they had an event where they launch a few interesting AI things and one of the more interesting things is Microsoft Scout. This is an AI personal assistant built on the Open Claw framework and it integrates with Microsoft 365 apps including Outlook, OneDrive, et cetera. So this is like open cloud, which means that it's an always on agent, you can kind of message and it will do stuff for you ad hoc and potentially do stuff in the background like you tell it to do something, you leave, you go sleep and it doesn't work for you. So interesting timing in that we just saw Gemini Spark also launch from Google. Their kind of openclaw equivalent seems to be signaling that there is a lot of belief that this openclaw esque, always on like background cloud agent, whatever you call it, is a thing that will be an important paradigm of interaction with AI. So this is currently rolling out to select Frontier customers in the US and they say a limited preview for more customers coming in subsequent months. Full cloud based, always on version planned for broader release later. So this is clearly also very early on. This is in this pattern of Google and Microsoft being like, hey, we're doing this thing and it may be out eventually, but we are doing it just so you know. And I'll be curious to see if I do release it widely or if it'd be a flop.
- [22:36]
Jeremy Harris
Yeah, it turns out you can't do the same thing with paying your taxes where you're like, I'm, I'm going to pay them. I am going. I know it looks like I haven't, but I will and I would like credit for that, please. It doesn't work. Yeah. So this is actually quite interesting on the enterprise side because as you said, I mean, they're selling it to the Enterprise. Enterprise. And OpenClaw doesn't sound like it should go together. Like the chaos agent that seems to want to rip apart the world every once in a while and just like kind of like go hog wild and delete all your files. And then the risk averse enterprise customer paying many millions of dollars per year for a product, those don't typically seem to go together. And in fact Microsoft's entire strategy here is around a kind of security architecture designed to not only prevent openclaw from going all openclaw on your shit, but also designed to swap out openclaw for other frameworks, swap out models for other models, and in that way sort of commoditizing the model layer and to some degree even the scaffold. Because what they're really doing here is they've got this kind of multi step process. So take OpenClaw if you want to use it with that. Right. They ingest openclaw through what's called a signed supply chain. Basically yes, open clause like this open source code you could download at any point. You need to know though that the version that you're downloading is not just like the latest version that could have been corrupted by some like Russian, you know, like code implant. You want to actually run those versions through a checkpoint to see if it's an actual authentic version, it hasn't been tampered with before, it's allowed into your ecosystem. And then separately the actual container that the agent's going to run in is treated as untrusted. It's basically like a sealed box. And Microsoft's position is we're going to assume whatever is inside that box could be compromised, could behave in an insane way and our expertise is going to be in building that box. And so really you've got all the kind of identity, the tokens, the policy side of things sitting outside that box. So like if the agent ever wants credentials to access your email or your calendar, it's got to do it through a very controlled slot. And that's really Microsoft's bet on like, you know, the trust layer, the governance layer is going to be the critical thing here, especially for enterprise, which is the most valuable kind of customer. And so I think that's really what this is like. It's not necessarily so much the this idea of like let's go hog wild on agents, it's more about let's build the infrastructure, the unsexy thing that everyone needs in order to be able to trust these models. And so I wouldn't be surprised if other people start coming out with more stuff here. Microsoft has the advantage just because of how many products they integrate with natively and so they can get a bit of a head start here. Google you can think of as having a similar advantage here. So yeah, interesting strategy. That's not about the model so much.
- [25:16]
Host 1
And the other big announcement that came bundled of this or alongside this was that they are putting out some new models. They released seven new in house developed AI models that is within this my family, Mai family I guess Microsoft AI, the headline, kind of most exciting one is my thinking one, a 35 billion active per hour reasoning model with 128,000 context window. So this is their big LLM, this is their Frontier LLM. And the interesting thing or one way to think about this is their blog post is called Building a Hill Climbing Machine and they very much position this as like, okay, it's not that great. You know, if you look at the benchmarks, it's not even close to a frontier. On Fropic and OpenAI, it's comparable to open source a little while ago. It's behind Kimik to 6 and GLM and Deep Seq 4, but it's like at the level of Deep Seq v3.2 and models from. Yeah, it's good, it's impressive, but it's not competitive, at least for now. But they go to great pains to say this was built with zero distillation. We have entire infrastructure to train all these from scratch, all this kind of stuff. So in a way reminiscent of Meta recently with their release of their LLM, their blog post was like, we built the thing that makes this thing better and we could train an LLM now. And it's, it's good. Yeah, which is a real accomplishment. I think this reflects the fact that training a Frontier model LLM is a massive challenge that requires very high effort. And just doing that to the point of having a competitive LLM is a huge achievement. So they announced my thinking one, they have a few other kind of smaller ones. They have my code one Flash, which is inference efficient agentic coding model that will be coming to GitHub Copilot. They have new image and transcribe and voice models as well that integrate into their existing frameworks. The last thing to say, which I think was a little buried but is very curious, is they also point out they're having this thing called Frontier tuning, which to me seems like we're saying we will allow you to have tuned versions of this model on your data, which is very notable. We haven't seen Frontier model developers offer tuning of their models for a very long time. OpenAI used to have it with GPT v4.1. I think they probably realized we don't want to let people fine tune our models. We want them to just use our models. Because once you start fine tuning our models, like are you going to fine tune an open source model? You know, like, I don't know. There's many arguments to be made on why OpenAI Anthropic don't want to support tuning to custom data. So the fact that Microsoft is starting to seemingly say that you would be able to do that for your business data, I think could be a competitive advantage.
- [28:43]
Jeremy Harris
Yeah, there's this like second tier of labs right now. Labs that are not anthropic or OpenAI or Google DeepMind or is going to be in that tier one, the second tier. You mentioned, you know, Meta, Microsoft, very clearly there we have to say Xai cursor now, you know, there's, there's like kind of a set of those in that second tier. The, I was going to call it a midlife crisis. That's the wrong term. But the identity crisis that comes with being a second tier lab is that there's no story that people can consistently tell. As far as I can tell, that has a second tier lab coming out with reasonable margins, that has a second tier lab getting to superintelligence first, that has a second tier lab being relevant as time goes to infinity. And so the entire game when you're a second tier lab is to break into the first tier. There's no world where you kind of hum along at that level. That's a biased jer take. So you can, you know, take it or leave it.
- [29:36]
Host 1
I think I will say to your point of trying to make some sort of framing argument on why this is useful. So Meta is saying, oh, we'll have personal superintelligence. Right. And they are saying that this will help with ads or whatever. You know, Microsoft is going the enterprise route, saying this will be a good fit for your data, for your business. Xai, I guess is about honesty and truthfulness. With Grok, they think that.
- [30:02]
Jeremy Harris
So you.
- [30:02]
Host 1
Yeah, I think with combine factories you need some sort of differentiator. You can't just say we have a really good LLM because the LLM isn't as good as the best LLMs. Exactly.
- [30:14]
Jeremy Harris
Yeah. So that's exactly where I was going. Right. So there's always what you'll find with the tier two. It sort of reminds me of Peter Thiel's competition is for losers argument where, you know, he says, like if you look at a market where there's, there's no margins and no alpha really, you'll, you'll notice how everybody is just really busy telling you why they're actually competing in a niche that's smaller than the one that they're competing in. So talk to somebody who runs a restaurant and they'll say, oh, we're actually the best. We're the best South American cuisine this side of this river and catering to people between 18 and 35. Like everyone has to be the best at their thing to make margin. And then the argument that they'll make is just to restrict the space they're competing in. Whereas if you go to the guys who actually have a monopoly. They're doing, they're making the opposite argument because they're scared about antitrust. And so you'll hear Google talk about, oh, we don't actually have a search monopoly. Like Bing is our. Where we're really worried about Bing. You know, all this stuff. Of course now it's more of a concern, but you know, back in the day that was it. And so there's sort of something similar happening here where the second tier folks are trying to really make this argument that it's all about recruitment, by the way, that is always the argument. And fundraising. So somehow Meta and Microsoft and these guys have to make the case that it's worth working for them when you could work at Anthropic or OpenAI and top tier talent is the only talent that matters, especially as code generation intern level code generation is already handled. So. Okay, so how do you, how do you make that argument? Well, if you're Microsoft, there's a couple things that you can say. You know, one is, okay, yes, we actually have a niche here. We're going after the Pareto trade off of cost and essentially per token cost intelligence trade offs. So like yes, we're not the smartest models, but we're the cheapest at a given level of intelligence for relatively cheap models. That's part of their argument. The other one is just distribution. Like you already work in Microsoft Office, you already work with Microsoft products, we have you already and we can watch as you interact with these products. PowerPoint for example, is really important one. Here you're directly interacting with a thing that gives us access to better data, blah, blah, blah. That's a good attempt to try to like lure some talent over. I don't know the caliber of people you'll get with that, but it's what they have to do with space X. It's like, hey, at some point data centers in space is going to be inevitable and the like show me a Frontier lab that has even a shot of doing that. Right? That's the case they're trying to make, you know, with Meta, as you said, it's like, hey, we have like the kind of social intelligence angle here. And so if you find, find that compelling, you know, maybe you'll find us compelling. So even Thinking Machines Lab, they have to come out with this like streaming intelligence model. There's got to be a differentiator because you, there's just now we're commoditized at the. Not commoditized at the frontier, but you know what I mean? Competition is too hot for people to compete at the frontier if they're not already there. Not saying it will never happen, not saying you can't go from tier two to tier one. It's in fact anthropic, arguably did that over the last four years. But it's hard. It's really hard.
- [33:05]
Host 1
Yeah. I think the other thing that's worth noting is to me looking at this in a way, given the tensions we've seen between OpenAI and Microsoft and the increasing business relationship complication, this is starting to look like a very smart thing by Microsoft to invest in having their own LLM purely for business reasons of you can have better margins if you train your own LLM on your own cloud, et cetera, et cetera. You have full control and you can optimize the heck out of it and get the best possible economics, which is a competitive advantage. At some point. The models are so smart that you don't need to be the best, you need to have the best product, which involves a lot more than just having the highest intelligence. So they are definitely on the road to having something that can be a compelling product with this and they're flexing their kind of research. Just general frontier lab muscle. Yes, they don't have like, you know, frontier, frontier intelligence, but they did train. They very eager to point us out from scratch, no distillation. And they also, unusually for one of these labs, released a very detailed technical report similar to what we've seen with open source releases. Has like over 100 pages with a whole bunch of reasons or a whole bunch of details on the training, the reinforcement, learning, ablation, a whole bunch of useful details. Not going as deeply technical as some of the open source models, but definitely giving us many more details than you would typically see with a model release from a major US business. So on the whole, not going to make Anthropic or OpenAI scared, but pretty exciting to see. Meta Microsoft with, you know, some notable leaders. With Mustafa having been acquired from DeepMind, you know, to lead this Microsoft superintelligence team, I wouldn't discount them. And I think this points to them being competitive in the AI marketplace.
- [35:13]
Jeremy Harris
Yeah, I think it's the objectively correct strategy. Right. Like, I don't think there's a better move than this. It's the, you know, we saw Apple make similar moves. I can't tell if they've sort of given up on this, but like really extra super open source was kind of their approach, trying to kind of demonstrate that they're building the machine that can build the machine. And that's really what Microsoft is making the case for here. They also have tons of infrastructure. So it's not a nothing move. This is a smart move. It's just a challenging space and it's
- [35:42]
Host 1
a good move for hiring, which I'm sure is one of the people outside Silicon Valley. I don't think have an understanding of how much of what these companies do. A lot of it is motivated by hiring. At least that's one of the key factors to do. Publicity is like you want to recruit talent. Especially true for AI. Next up, some less exciting news. No new models, but still interesting. Robot Hood now lets your AI agents trade stocks. So Robinhood is a stock trading app, very popular among retail investors. You would call it casual investors, not professional necessarily. And so they have announced this model context protocol, letting agents analyze stuff, execute trades and identify investment opportunities. Something you could in theory have ChatGPT or Cloud do, but likely haven't done and maybe should not do. So this is an interesting case of like agents are getting more and more powerful. We are letting them do it more, more and more on their own. Are you going to trust your agent to go the next level and try to make money on investments or at least manage your finances in this kind of active way?
- [37:02]
Jeremy Harris
So first of all, always think about where your alpha comes from, right? If everybody is trading using chatbots and chatgpt and it's as easy as switching to the latest one that was released and you're not fine tuning it on any magical data, then I mean assume that you have no alpha, right? Like this is like the, you know, the standard efficient market hypothesis stuff though in fairness, a lot of like kind of normie banks and stuff are surely the dumb money relative to you in that context too. So I don't know. This is not investment advice. What also isn't investment advice is just to point out that you know, Robinhood's incentives, right, they kind of make money as a function of trading volume. And so when you think about like what it does for them to have agents that like, you know, don't get bored, they can act 24 7, they don't get scared out of positions, right? That's, that's pretty. Anything that increases trading volume on their platform is to their advantage and it's not necessarily to yours. So you know, something to think about. There is a perverse incentive at play there. It's not that this is a bad idea, it's just like there is a baked in thing here that you should be keeping in mind. So they're also by the way, like right now it's for stocks, they're looking to add options for options, futures contracts, prediction markets like the works. And these are, you know, you're typically like very high velocity, high loss, like plays that tend to be really good for Robinhood's bottom line because again, high velocity, yeah, that's, that's what they're making their money on. So I guess just like kind of, you know, keep an eye on that. There's a perverse incentive. It's not quite a casino, but you know, you've got a reason to kind of question the number of trades that you're being invited to make and the incentives that underlie that. But the other piece here too is it's not necessarily anything goes bizarre. They are like Microsoft actually like a lot of these other companies that are rolling out these agentic offerings things, they're trying to set up a containment cell. So here it takes the form of a separate account, preloaded wallet the agent can't exceed. There's like trade previews that you have to approve, human fraud team, like all kinds of stuff that's meant to get around the open claw factor here. So this is basically people saying, look, we can't avoid going agent. But we also don't necessarily want to just give agents a big unencumbered hug. We want to make sure that they're contained to some degree.
- [39:11]
Host 1
Next up, OpenAI they've launched new Codex tools for white collar work. So this is a set of six plugins that deal with data analytics, creative production, sales, product design, equity investment and investment banking. Similar to what we've seen with Anthropic's Claude for finance for example, which was just like a bunch of tooling around specific needs for different industries. And it came together with a blog post or I guess a report introduced by blog post where they have the next era of knowledge work as the title. And they highlight that now they have 5 million weekly active users. 6x growth compared to February. The blog post is titled Codex is becoming a productivity tool for everyone. So basically the pattern we've seen for the last X months is OpenAI and Anthropic want to make this a tool for everyone who uses computers to do whatever they do. And that is a lot of people. And that means that they have like you know, now the competition. They're doing plugins, they're doing forward deployed engineers, they're doing whatever they can to you know, get people to adopt, adopt, adoption. And I guess we'll Be seeing more of these kinds of very honestly kind of boring things, but business perspective, you know, good business moves.
- [40:37]
Jeremy Harris
Yeah, Andre, I can tell you don't like being in what people are starting to call a maturing market.
- [40:42]
Host 1
Yeah, I know.
- [40:44]
Jeremy Harris
Doesn't that make you feel sick?
- [40:46]
Host 1
Yeah, I remember when openly I did open source stuff and just like fun, stupid stuff.
- [40:52]
Jeremy Harris
But when people are making up business models, like, let's charge people by the freaking token, I don't know, like that's what this was. And yeah, now it's. Suddenly we're at the point where OpenAI and anthropic start to look a lot more similar over time because yeah, it is a maturing market. It's also, you know, it's ahead of the ipo. You have to read every headline now through the lens of OpenAI and anthropic and for that matter, SpaceX and all these dudes are getting ready for, for the IPO of historic IPOs. One piece here too is that they're. OpenAI is doing this under the Codex banner, which does reputationally mean that, you know, like it's, it's all going to be viewed as codecs. So if it's a flop, if there are issues with the rollout, like I think people who do software engineering all day are smart enough to kind of delineate the two, but there is a little bit of brand risk and it's interesting that they chose to bundle those together rather than like split them out. It seems like OpenAI really has that reflex a lot. You see them like periodically they'll like spawn out a side, a side product and then they'll be like, oh, we need to fold this back in. Like it needs. Everything needs to be back under chat GPT. Sort of a similar, I guess, impulse here. But yeah, interesting story in its boringness, arguably.
- [42:01]
Host 1
And one last quick story in Lavin Labs has a new music generation model Music V2. They highlight that the model can switch genres mid track and add non musical sound effects, Various kind of very niche things that now are obviously kind of the case for music. Music is a good place and now you're seeing all the like, like smaller things like genre switching or fast rap without losing coherence, performing across languages, lyrics, vocals and arrangements, recreate sections, et cetera. Lots of stuff here. And it is now available, it is licensed for commercial use, which means that it could be competitive with SUNO and udo, which are mired in a lot
- [42:49]
Jeremy Harris
of lawsuits as the industry continues to, dare I say, mature. Yes.
- [42:56]
Host 1
And speaking of the Industry maturing. Next up, applications and business. And we begin with Anthropic, which had a duo of very big things I'll just squash together. So first they announced their new raise, Series H. And this is one of those things where you didn't know this could go that high. I don't know if you've ever seen an H. Maybe we have, but it's. It's unusual. Usually you do like Series D, Series E, and then you go public and you no longer do series because you're public. You don't do raises from private investors. But yes, they raised a bunch of money. $65 billion, I think. We've been covering kind of the track of this raise and it keeps going up and up and up. They wound up at 65 billion at a valuation of 965 billion. So not surprising. We knew this was happening, but, you know, impressive that the investors are so excited. And part of why they're so excited is that Anthropic has filed to go public, setting the stage for a huge ipo. So they filed this for an initial public offering. They announced it just yesterday, I think, in June 1st. This came, I think under a week after the announcement of this raise. So clearly sort of tied together.
- [44:17]
Jeremy Harris
Yeah.
- [44:18]
Host 1
As you said, IPO is a big deal.
- [44:20]
Jeremy Harris
Yeah. No, and that's it. You'll also. Or, sorry, you'll often see companies do that. Right. They'll do that last big fundraise before the ipo. A lot of it can involve kind of being prepared to pay the massive tax bills that come due around IPO time. I'm trying to remember. I think I remember reading something like $30 billion may be insane, but I think that might have been something like that, that Anthropic was earmarking for just that. So, like, I mean, these are wild numbers, any way you read them. I love the. I didn't know it would go that high with Series H. There's something like vaguely Trumpian about that. They're raising the H. I didn't need. People said that it couldn't even go that high.
- [44:58]
Host 1
I didn't know it could.
- [44:58]
Jeremy Harris
But anyway, there you go. So, yeah, this is the timelines now with OpenAI rolling back or pushing. Pulling back, I should say their IPO timeline too, to what sounds like it could be as early as September is also. I mean, these are crazy moves. The reason, by the way everybody's rushing to do this is that there's a sense in which the public market right now has a giant wad of cash waiting to be spent. Right. You've got institutional investments, investors rather, sitting with all the big banks. And the game right now is whoever gets to IPO first gets to essentially be the pressure release valve for all of that money that's ready to operate on. The AI thesis that's been pent up as all this stuff has been in the private markets. Less of an issue for SpaceX because AI is part of their thing, although it's a big part of their prospectus actually. So, you know, it's absolutely in the same category, but they're hedged with other things. Whereas anthropic and OpenAI, it's just like, you know which one goes out ahead. It's not obvious that members of the general public can meaningfully differentiate between the two. So whoever comes out first is just gonna get potentially a windfall. But it could also not, you know, turn out not to be that. We've been surprised before.
- [46:01]
Host 1
Yeah, SpaceX, I'd be very curious to see how VIPO goes. They also filed some documents through the sec. I think we covered last week some of their financials. They say that their total addressable market is like 28 point something trillion, bigger than the entire income of the US. And I think the majority of that 26 trillion or whatever is just AI. So they are positioning themselves as an AI contender with ambitious kind of estimates of how much of a market they can get. And they come in at a valuation of 1.75 trillion. Another very unusual things for IPOs in the AI era. Like having IPOs with these kinds of valuations is not something that's happened before. Usually you go public and then a decade or two or three later you hit that 1 trillion mark. There's not many companies valued this high and now it's just like out of, out of a gate, $1 trillion.
- [47:02]
Jeremy Harris
Well, and this is the thing. So, so there's a, there's a whole, we could do a whole episode on, on the public and private market story here because it's like, it's really interesting and it's also, I think, a really unfair aspect of all this stuff. Look, if you're an everyday citizen without special access, if you're not an accredited investor, if you don't have the connections that let you get money into the hot round, you just miss out. And by the time these companies are available to you as an investment, they've been de risked to the point where, okay, because I believe in the superintelligence thesis, I'm not going to sit here and tell you they've been de risked to the point where there's no more profit in them. I think there's potentially a huge amount more. But certainly when you look at, you know, companies like Amazon that used to IPO like, you know, and Google and stuff like they used to IPO at valuations or caps that were like really, really reasonable based on what the market was saying, there was a lot more room to grow. The challenge right now is you're seeing these companies where to your point, in order to live up to their valuations, they have to be generating so much revenue that it's like a meaningful fraction of US GDP right now. The issue there is you don't build a company like that without actually growing the whole economy. And that's what you're going to start to see. It's the reason that you saw Meta even years ago, building undersea cables to increase Internet access in Africa. They're literally at the point where they're feeling the edge effects of the entire global economy. And so their only choice is to literally grow the global economy, make random people in Africa have access to the Internet for the first time so that we can make money off them, create value so that we can extract it. But in other companies, and you see kind of OpenAI and anthropic have historically written things about this. Cullen O', Keefe, who used to be an OpenAI policy guy, he's a lawyer, wrote this thing called the windfall clause which was this thesis around like how things should work when OpenAI is at the point where it's generating enough revenue to be a significant fraction of global gdp. Well, guess what, we're not that far from that point now and we're starting to feel those edge effects. And so I think that's kind of a really interesting consequence implication of these trillion dollar valuations. The U.S. as you said, is a, it's a $20 trillion economy right now it's only growing at like, I forget, 2% a year, like whatever the thing is like single digit percentages per year. And so you're not going to get like a $5 trillion company just pop out of nowhere without making a dent in the actual GDP growth of the us. It cannot work any other way. Mark my words, anytime the math doesn't work, it's over. So, so that has to move. I believe that it will, but you could very reasonably have the hypothesis that actually those market edge effects are going to become important, certainly limit the amount of ROI that, you know, investors who get in at this point are going
- [49:35]
Host 1
to Enjoy moving on from that IPO. Next up, we've got China's ByteDance developing new AI chips like those from Nvidia partner Grok. So Grok and others develop chips that are more specialized to AI. Grok is calling them language processing units, particularly purpose built for AI inference, meaning it can run the operations that have transformers. And it's not a general purpose, you know, not a GPU, not a traditional CPU. So here with ByteDance we got some stories on them partnering with some people generally working on this kind of chip design. Apparently their team is at 1000 people working on this. And it's significant because there's not many competitors in the space. Like you don't have ready suppliers of language processing esque chips. It's even more of a commodity to have this kind of technology than let's say GPUs. And Nvidia is now also investing in. That could be yet another way that Nvidia leads and locks down the market if they get to good inference optimized chips.
- [50:45]
Jeremy Harris
Well, yeah, I mean, so the key question here is why Grox Design? Why, oh why are Chinese labs so interested in the lpu? Right, okay, well let's talk about the specs. If you don't remember from like, gosh, a year and a half, two years ago, I forget when the first time was, we did kind of a deep dive in the lpu. But as a reminder, normally when you look at a gpu, it's got these stacks of high bandwidth memory. These are basically the stacks that hold the numbers that will be crunched on the actual logic die that does the math that don't actually do the math themselves. So they're kind of a holding pen on hot standby to just feed the numbers into the logic die and then they come back out after they've been crunched. The problem with high bandwidth memory is it's basically all made by a small number of companies that are outside of the Chinese ecosystem. I'm thinking here especially of Samsung, but more so SK Hynix, which is famous for having just like really good hbm. So the problem is if you're China and you are looking at an export control regime that's preventing you from accessing exactly that, you need another alternative. In comes the lpu. So what is an lpu? It's a custom chip, it's based on the architecture of the transformer. So it is a transformer only chip. And crucially, it has all of the memory. So it's got no external memory, like no defined high bandwidth Memory stacks. Instead, it keeps all of its data on chip like right up to the logic die during processing. So it's all in sram and it's only got, as a result of that, it's only got a tiny amount of memory, 230 megabytes at least. As of a couple years ago, that was the Grox spec. And so as a consequence you need way more chips. Back then it was 576 Grok chips were needed to build up the inference unit and to serve even just the mixed real model as it was like a year and a half ago compared to a single H100. So literally 576 Grok chips to a single H100. Now it does work, especially if you have a lot of throughput, a lot of data volume. Where do you see a lot of data volume? China. Okay, cool. What's another thing that the Chinese are especially good at? We've talked about it on the show before. Taking a lot of shitty chips and networking the crap out of them together to get a cluster that in some ways rivals some of the things that you can see from Western labs. And so this really fits in the butter zone of what China does best. Take a lot of crummy chips. I don't want to call these crummy, but, you know, take a lot of those chips, network them together like crazy in a mesh and bypass export controls on memory. This is like the reason you're seeing this happen right now. So you introduce a constraint like export controls and, and expect that companies like ByteDancer are going to go ahead and find every possible way to weave their way around that constraint.
- [53:21]
Host 1
Next up, a few lightning round stories. First, Anthropic expands mythos to 150 additional organizations. So this is now adding access to new industries like power, water, healthcare, communications and hardware, industries where cybersecurity seems pretty important. It still is unclear, I think, whether Anthropic would just keep doing this of having, you know, trusted partners gain access to Mythos and never potentially releasing it as an API layer. So at least for now that appears to be the case. And on that note of I guess, large model compute, there is an analysis piece where the headline is OpenAI needs at 26x revenue increase to justify its build out JP Morgan. The bank is now estimating that the AI sector needs $650 billion annually in revenue to justify current capital expenditure, which is compared to what they say is $25 billion currently. This is because the five biggest companies are projected to spend something like $725 billion on infrastructure in 2026 with most of that going to AI that's up like crazy from recent years. So, you know, I'm not sure what to make of this analysis. What do you think, Jeremy?
- [54:46]
Jeremy Harris
There's two variables, right? There's revenue today and then there's spend on the CapEx unit. And it's generally CapEx that dominates on the revenue that will generate the revenues in a year and a half to two years from now when the data centers are built. And the single most important calculation that's happening at any given time inside OpenAI, inside anthropomorphism topic inside Google is how big of a capex spend do we expect to need a year and a half from now in order to match what we expect will be the revenue at that point? And that's a function of your growth rate, it's a function of the stickiness of your product. All the things, right? They're pretty good at that calculation. Again, I keep saying this but like when they need some, some idiot to like put in the CNN report or whatever when, when the bubble bursts of people who are saying it would never end, like this will be the bit, but like this is a calculation that the labs are really good at doing, should be really good at doing. And so in that sense, you know, sympathetic to the argument, but I think you got to look at the numbers. The one thing there's, it's kind of like almost ethical question here. There was the Wall Street Journal's Tech Live conference back in November. Sarah Fryer, who's the CFO of OpenAI, was talking about their financing plan that would combine institutional lenders with a federal guarantee that would let OpenAI take on more debt at lower cost. And then she was pressed, does that mean government backed financing for chips? And she said, yes. And so there is again, going back to the big short thing, there is this whole privatize the gain, socialize the risk argument here. And it's been had any side of the political spectrum you want to look at. I saw a clip of like Tucker Carlson having some, some thing about taxes, tax breaks from local communities to finance a lot of these builds and sort of making the case like, oh, you shouldn't be. He was on with Kevin o' Leary and they were kind of arguing about this. I mean, look, I think it's, it's, maybe it's more complicated than Tucker is making it out to be. But the bottom line is this, there is this sort of sense in which we're now going to be tempted because this is a national priority, national economic, national security priority. There's going to be this temptation to start to, yeah, socialize some of the risk. And once that ball starts rolling, it does introduce some, some pretty fundamental challenges and questions. But again, I mean, if you believe
- [56:55]
Progressive Insurance Announcer
scaling works, why does Progressive work hard for truckers? Because truckers unite the world. They unite kids with their first drum sets and parents with earplugs. But truckers can't do this if they're not on the road. That's why Progressive has over 360 heavy truck employees to help truckers stay on time and on track. Quote Truck Insurance today in as little as 8 minutes@progressive commercial.com progressive casualty insurance company and affiliates for a small business
- [57:23]
Jeremy Harris
owner, every day is full of surprises. Some great, some not so great, like when a client cancels their order at the last minute. But here's a surprise you will like. Progressive provides small business owners with 30 customizable coverage options to help keep their business going strong. So go ahead, surprise yourself. Get a quote in as little as 8 minutes@progressivecommercial.com Progressive Casualty Insurance Company and affiliates and third party insurers. Coverage is not available in all states or for all vehicles and coverage selection as I do. The only question is what's the optimal ratio between the revenue today and the capex of tomorrow? This doesn't seem insane to me, so we'll see, but it does not seem completely bollocks.
- [58:03]
Host 1
And one last quick story. AI coding startup Cognition raises $1 billion at $25 billion pre money valuation. So pretty big raise. They developed Devin, which in some sense is competing with cloud code and Codec. So if you can. It's interesting that investors are still eyeing competitors to those two. Good for them. $1 million is a lot.
- [58:27]
Jeremy Harris
So they are nominally they have like a $500 million run rate. Right. Which. Which to be clear, what that means is if you look at their monthly revenues today and then you assume that those monthly revenues hold consistently for a year, you get to roughly $500 million, $492 million. Right. Okay, so their growth rate is doing all the heavy lifting. They've been growing 50% month over month. I'm old enough to remember when Devin was supposed to be the disappointing demo product. And now we're actually getting real traction. Which to your point is interesting because I would have bet wrongly that we were past the stage where new entrants could actually break in. So, you know, good. Good for Devin. This is a 50x revenue multiple and the only way you ever get to 50x revenue multiples is through growth. It means that basically investors are too scared not to bet on the growth trajectory of this startup. 50% mom is just way too fast of a growth curve for a company that is already making $40 million a month to ignore. And that's why you're, that's why you're seeing this play out. So, yeah, I mean, you know, we'll see. The bull case here is really just like they're trying to go for the enterprise and maybe they can lock in, you know, to the point where trust and switching costs become the main product. Maybe. The bear case obviously is there's a lot of competition and the moment that Anthropic and OpenAI turn their attention to the same clients, like that's a scary place to be. So they've got to get entrenched fast. If you look at the cap table on this raise, it's pretty impressive. Founders Fund is on there, General Catalyst is on there. Lux Capital. So yeah, I mean a lot of these really, really solid funds, which you'd expect since they're raising so much so
- [60:06]
Host 1
high onto projects in open source. We've got one big story. Minimax M3 is out and it is yet again a pretty strong open source model. Not open source yet, but they do say they'll go on the open wait area that others like Kimmy and Deepseek have continued to go on. This has a 1 million token context sudo similar to deep seq v4, priced rather competitively and has high speed. So broadly speaking it has. The general pattern we've seen is open source models are getting quite good. They're comparatively fast and cheap typically and they provide the weight. So you can in theory fine tune them, as we've seen Cursor do, for example, on their own data. So I don't know what else to say. It's quite interesting to see open source models getting good enough where you're able to use that as your daily driver instead of Claude, which at a certain level of intelligence you can use a less intelligent model and just use it because it's cheaper, faster.
- [61:24]
Jeremy Harris
Yeah, and there's, you know, a bunch of different axes, right, that make a product besides intelligence, as you've pointed out many times. Right. I mean there's cost per token, so just like let's compete on budget, there's actual intelligence. There's also latency. Right. So how long does it take to get the first token out? And that's one kind of latency. And how long does it take to get the model to basically just read? Right? That's another kind of latency. And in fact this is an attempt to compete along those two ladder axes. And so they use this tactic called, well, Minimax sparse attention. Not to be confused with deepseek sparse attention, though they have some overlap. We've talked about DSA Deep SEQ sparse attention quite a bit in the past. So they did some early hardware profiling that showed an almost 10x speed up in pre fill latency. So pre fill is basically the point where you take your prompt and you basically just load that into the model, into the KB cache, get the model to basically read your text. So that's obviously very closely tied to time to first token because that tends to be the rate limiting thing for just getting that first token out there. And again, a 10x speed up in pre fill latency is going to be felt like you're going to see that. You're going to see it in reduced hardware requirements to actually serve a serviceable version of the model. You're also going to see it just in reduced latency as a user and then a 15 or 16x speed up during the decoding phase. So in other words, the phase where you're actually like rolling out those tokens, actually generating the text and that was measured at a 1 million token sequence length. So really on the heavier end. So for really, really long sequences, this thing is much more, much more efficient at pumping out those output tokens. I've spent quite a bit of time looking into Minimax sparse attention. I feel like we might have talked about it on the podcast, but anyway we can park it here. Roughly speaking, it has to do with so deep seek sparse attention is like you don't need to actually pay attention to every token. Not all tokens matter. And so you have this initial they call like a light, a lightning indexer that goes over your input text and quickly says, okay, just totally ignore these tokens. Let's only do attention, which is like the time consuming calculation on the remaining tokens that matter. There was something analogous happening with this minimax sparse attention thing where anyway they're dividing input text into blocks of text, blocks of tokens, and then doing something philosophically similar with those. So we'll see where this goes. I think we're still waiting for the full technical report to drop.
- [63:39]
Host 1
I don't think it's out. Interestingly also, I think I may have undersold it on the benchmarks. They're showing that they are competitive of GPT 5.5 and Gemini 3.1 Pro on things like Swe Bench Pro, Terminal Bench, a bunch of benches. They are nearly at the frontier level, although lagging behind Opus 4.7 generally and now lagging behind Opus 4.8. So this is like a very capable model. They also have Minimax code as their own code hardness that also has the ability to orchestrate agents and run things of different kinds. And Minimax, by the way, in general developed many models. Now they have Hiluo, which is a very good text to video model. This model is also multimodal natively, so it has the next most advanced level of multimodality by kind of natively fusing those things. So on the whole of the open source models, M3 now picks the cake, potentially pending Vive UAVs and so on. And I continue to be curious whether we are at a point or nearly or soon will be at the point where the open source models, the cost and speed advantages make them, you know, a lot more popular when they are now.
- [65:05]
Jeremy Harris
Yeah, and that's also part of the vibe check too, right? There's like the vibe check involves code quality, involves latency, it involves cost, all those things. I'm very much in a space where I want to wait and see how these look because we have seen quite a few releases that don't quite pan out from all these labs and including sometimes in particular the Chinese labs. So that's starting to change. So yeah, we'll see.
- [65:26]
Host 1
Onto policy and safety beginning with the US Government Trump signs executive order seeking oversight of AI models. So this establishes a framework for federal oversight of powerful AI models. Under this order, AI companies are asked to voluntarily submit their most powerful models for government testing up to 30 days before public release. I don't know, I don't recall if we discussed it, but this was supposed to be signed earlier and then was deferred. It appears that maybe some of language was tweaked in response to industry objections. And this very much, you know, seems like it requests voluntary collaboration. And the order actually explicitly bars the government for creating a mandatory licensing or pre clearance requirement, making this a request, not a rule. Which means, yeah, you know, basically it's saying let's not regulate, you know, let's ask nicely, but not force anyone to do anything they don't want is maybe the better way to think of this rather than as a oversight kind of effort.
- [66:39]
Jeremy Harris
Yeah, what they're doing is they're counting on the incentives of players in the market to want to be able to offload responsibility if something Catastrophic happens to a government review process, right? So the way this plays out is, you know, I'm OpenAI, I'm anthropic, I have a model, I'm scared it's a WMD and, or just like I'm concerned someone might weaponize it in a way that then makes headlines and creates potentially liability exposure for me. And so what I do is I say, okay, you know what, there's a voluntary process the government has that they'll review my model, they'll rubber stamp it, and then if this thing goes out and causes some kid to, you know, off himself or something, I can at least say, hey, feel terrible about this. But this is why we've tried to work with the government to actually have them review this. We're good faith actors in this space. So I think there is a strong incentive there for a lot of prosaic risks. But not all risks are prosaic. Once you get into automated R and D, recursive self improvement, the software only singularity, if you believe that those risks exist, suddenly the labs can play fast and loose in terms of deciding which models they want to submit to this process. Models they don't plan on deploying as product to the general population and therefore that they don't expect to be subject to the same risks. But they may actually incur meaningful risk even in internal deployments. They may not want to run by this process. So you can think of it as in part a kind of capacity building exercise for the administration. Let's just get really good at taking these models in, running these tests in a way that has teeth. If the labs decide to participate in this voluntary process, you know, you can argue for this. I think it's not dissimilar to what the Biden administration did in their 2024 EO or 20 late 2023 EO where they said, hey, you know, like we'll kind of do this sort of thing. That led to the formation actually of their, their basically their standards body that looks at AI security and does the model audit. So that's been stood up. That's good institutional capacity. Whether it goes far enough, I think is just a function of when the first automated AI powered cyber attack actually gets felt by the average person. And I would expect that'll happen sometime in the next 18 months. I think over the next 18 months I'm, I'm making these predictions deliberately so I can be held to account if I'm wrong, as I'm just trying to practice good predictive hygiene here. I do expect that to happen, I think happens. There's going to be a sudden rush to regulate. And I think a lot of the people who maybe were pushing for more hands off approaches will regret having done so because then the kickback could be worse than, than what it would have been otherwise. A couple things on the politics of this. This is actually almost verbatim the same executive order that we were told was getting quashed like 20 minutes ago. The main change is that the 30 day voluntary review process used to be a 90 day process process, and now they're making a really big deal out of the fact that they went from, oh, 90 days to 30 days, as if this justifies the entire brouhaha when what we were hearing was at first Susie Wiles and Scott Besant. Scott Besant, by the way, in my opinion. So if you disagree with me, you'll probably dislike Scott Besant. I think Scott Besant is a really smart dude on this stuff. From everything I've heard, he's pretty interested in the AI stuff, taking it very seriously. Susie Weil seems to as well. She's Trump's chief of staff. And so between the two of them, that's a lot of political power in Trump's orbit pushing in the direction of more kind of regulatory approach. The dissenter who apparently kibosh this at the 11th hour before was David Sachs, who is no longer the AI czar, but he is on Trump's big AI Council committee thing that he set up. And so he apparently just called Trump and said like, this is no good, this is dumb. You got to stop it. And Trump was like, oh, you make a good case. I like sex. We like sex. He didn't say it sounds a little too much like sex, but he's saying, I think sex. I'm not trying to make Trump say sex. Okay, so the bottom line is Trump went with the sack strategy, the Sachs strategy, sacks, and then. And then he just reversed course. That's what this is. So it seems like Susie Wiles and Scott Bess and ultimately winning through in this giant kind of food fight that's happening. And they're trying to frame it now as if.
- [70:28]
Host 1
Yeah.
- [70:28]
Jeremy Harris
And as a perfectly natural result of this very smooth process, we got to where we were going to go this whole time after Trump referred to Basically this very EO with just a 60 day difference in the review period as something that was overly burdensome and too regulatory minded. So it's kind of interesting. Trump's mind obviously can be changed. This is one big take home. And I Think everybody should view that as a positive. How you get there though, is a hell of a gauntlet run and not necessarily the best for stability in the markets or from a national security standpoint or whatever else.
- [71:03]
Host 1
Next up, a story about cybersecurity, but not the sort of cybersecurity we've been discussing about hacking. The headline is hackers simply asked Meta AI to give them access to high profile Instagram accounts. And it worked. So basically hackers found this approach where you can talk to Meta's AI support chatbot and you can fool it into giving you the ability to take over an account saying, you know, whatever, I forgot my password, so please, and my email is different, so please change my email address so I can recover the password or whatever. I think that's something like that.
- [71:45]
Jeremy Harris
Apparently it's like the hacker just asked the chatbot to add a new email address to someone else's account and then the bot sent a verification code to the attacker's inbox and the attacker read it back and the bot was like, cool, here's a reset password button.
- [71:59]
Host 1
Yeah, so that mechanism is absurd. The fact that this works, I, I'm not sure if this is the same thing. I saw also examples where for verification purposes you had the ability to do a face scan, to be like, here, this is me, I'll, I'll send a photo. And people were also fooling the AI there with like screenshots or whatever of the people. And they, I was like, okay, that's you, so I'm going to reset your password. Believe also was targeting Meta and this hit some very high profile accounts like, like the Barack Obama's White House account chief master Sergeant of Space Force, you
- [72:37]
Jeremy Harris
know, some of these examples, fine.
- [72:40]
Host 1
So that you don't want that to exist. And this is an example of new vulnerabilities when businesses integrate AI into like their core whatever product features.
- [72:53]
Jeremy Harris
Yeah, yeah, absolutely not. Not good. And again, I mean it's an industry maturing, you know that, that's maybe the, the take home, there's like a lot of swing for the fences happening. And yeah, you're, you know, your socials are part of your, are part of your organization when you're at Space Force or whatever. Like, you know, you gotta, you gotta have a game plan for it. But also the technology is moving pretty fast.
- [73:15]
Host 1
Next international story, Chinese AI experts and private firms now required to secure approval before international travel. So this is startups, this is state owned companies, this is private firms. This was previously limited to senior researchers at public institutions. Nuclear Scientists and government company executives. This now applies to private sector workers and is kind of crazy. Like, you know, you wouldn't see this outside of police state, where you're now saying if you work at a tech company that is focused on AI and you're an expert, we have to approve your travel. That, yeah, is indicative, I guess, of China trying to hold on to their competitiveness in AI and perhaps increase it.
- [74:10]
Jeremy Harris
Okay, yes, it's true. You wouldn't see this outside of a police state. Except around the time of the Manhattan Project, people started to get real concerned about scientists you see walking around and shit. And so this actually has an eerie echo of another class of technology. Oh, and look at this. This is the same thing China's been using to manage their fucking nuclear researchers. So if you're wondering about where China is positioning mentally AI in their national security stack and how seriously they're taking it. This is about how seriously they're taking it. And by the way, this is a gun that backfires somewhat too.
- [74:53]
Host 1
Right?
- [74:54]
Jeremy Harris
We talked about the Manus acquisition a couple weeks ago where Meta acquired Manus and Manus had started in China. They tried to do this classic Singapore play where they relocated to Singapore and tried to pretend, hey, we've been a Singapore company this whole time, to evade the kind of Chinese oversight of the acquisition process. And then China basically said, hey, Manus co founders come to Beijing right now. And they were like, okay, so they went to Beijing. And then the Chinese guys were like, you're grounded. And they're like, fuck. So the Manus acquisition didn't go through. This is that. But applied preemptively across the AI stack. There's no selection criteria that we, that we know of. There's no official guidance on which roles, what expertise, seniority level is going to be included in the travel ban. All we know is the motivation theoretically is anti leakage. There's this report that says that the policy is meant to like protect against leakage of key technologies. And that included, you know, a reference to Manus, basically. And so this isn't about formal law, quiet and selective application of arbitrary powers by the ccp. Again, police state thing. Exactly as you said, but with echoes that date back. I shouldn't say the Manhattan Project, but the. Because the challenge with the Manhattan Project is you had all these researchers who came in from like Eastern Europe and from Germany and stuff. They were in the United States, they were not citizens. And so the United States could constitutionally tell them that they couldn't move. It was later on when you actually started having. And I think it was like Linus Pauling and some of those cats who were not allowed to leave the country. Their passports were basically withheld because they were working on stuff that was too sensitive. And so you do have a mode even in liberal democracies where people start to like, lock stuff down. And the challenge is the sooner you reach for that leverage, the sooner you get flight of talent. Right. So, like, people are just going to like, frigging leave the country if they think you're going to start to prevent them from, well, leaving the country. And this is part of the challenge that China is going to have now. Like, how many people are going to choose to start to become AI experts while living in China if they know that this is what it could lead to? So big kettle of fish, can of worms. But this is a way bigger story than anyone is covering. I have not seen the attention on the story that it deserves. It's probably like number two headline material for the week. Number one on the geopolitics side of AI, this is a really big canary in a coal mine.
- [77:10]
Host 1
And on that international relations front, next is US Titans Controls Nvidia AI chip expert. This was like not a huge story, but worth noting. The U.S. department of Commerce clarified their export license and basically closed a loophole that has resulted in potentially hundreds of thousands of chips being sent and kind of enabled the chip trade in what is, you know, technically the export controls are basically saying China should not get advanced AI chips from the US As I understand it, we have discussed already how that is very not true. In many ways. China is still getting access to the most advanced AI chips. And this is one of the reasons with loopholes and. And it looks like the US government is now going after that.
- [78:02]
Jeremy Harris
Yeah. And you know, the goal is never, obviously to get their chip imports to zero, or at least that's never been the possible goal. You're always going to have a black market and a gray market and all that stuff. The goal is just to reduce the amount of computing power they're able to import. And in that sense, these have been wildly successful, except that why is the Department of Commerce, the bis, which is the bureau in the Department of Commerce that's in charge of these export controls, why are they coming out and clarifying their position on something? It's almost as if. It's almost as if they said something before that confused the crap out of people. And that's actually what happened. So in May 2025, BIS came out and said, hey, you know What? Guess what, we're just not going to enforce certain parts of the Biden era AI export controls regime and BIS because they said they were suspending enforcement without saying which specific provisions they would still enforce. This gap opened up and it was a gap of as much intent and I'm sure some motivated reasoning too, you know, people who would just want to believe that it's more permissive. So people started to ship, right? And so, so let overseas subsidiaries of Chinese companies buy Blackwell chips, the like top of the line chips without any kind of license legally because the regulations on paper just hadn't been updated to match what was actually being enforced. So basically as long as you're a Chinese company that got a subsidiary in another country, that subsidiary could still receive those chips. And they they flag an example of a sub for Tencent in Malaysia that was buying chips that really should have been off limits to 10 cents. So the insane thing here is the scale, right, I mentioned the whole point here is not to get Chinese imports down to zero, it's to get them down to some level that is good and low. Hundreds of thousands of chips are expected to have leaked through this mechanism. We don't have confirmation, but when you look at the Microsoft cluster, right, what's the number of Blackwells they said they had there? It was like 8,9000, right. Hundreds of thousands of chips here, Blackwells potentially being routed to China. As I understand it from the story that is, that is what's going on. There's also a whole separate thing that they're not clarifying in this move, which has to do with tsmc. So you can get past the system by just like getting chips you shouldn't be able to have from Nvidia, buy them from Nvidia because you have a subsidiary that's outside China. Another thing you can do, get your own design, if you're Huawei, ship it to tsmc, have TSMC fab the design and give it back to you. That's not an export. And what's still ambiguous is that part of the equation because in May 2025 the BIS announcement of non enforcement also undercut the rules that would require factories like TSMC to do due diligence on chip orders. And so now it's unclear whether that is actually also being enforced, in which case if it's not, that's another giant loophole that's still hanging even after this clarification. So a bit of a mess here if you believe, I mean cards on the table of my bias. I believe it'd be one of the most prescient things that both the Trump one and Biden administrations did. And now seeing that undercut, I just, yeah, this is a real l, I think on the national security side.
- [81:03]
Host 1
But next up, OpenAI has launched Rosalind Biodefense and has offered federal agencies early access to this, this life sciences model. So this relates to GPT Rosalind, their life sciences reasoning model. And they are now partnering with federal agencies and some affiliated labs. So Lawrence Livermore National Laboratory, for instance, John Hopkins apl, the Coalition for Epidemic Preparedness Innovations, companies like this. And OpenAI is framing the initiative as defensive acceleration. So in a sense, this is like Mythos, but for biology. Right. Where you could hack and create viruses for human bodies and you do need defenses against potential bioweapons. And it appears that OpenAI is very much putting that forward and partnering with agencies to accomplish that.
- [82:03]
Jeremy Harris
Yeah, that's a thing. As much as cyber is the kind of first line of attack, Bio is not going to be that far behind, as evidenced by the Mythos bio evals, which people haven't talked about, but are actually really concerning when you look at the Mythos preview capabilities there. And of course you can't update software and firmware for your body, so that's going to be a real issue. This can read as OpenAI saying, hey, me too on the project Glasswing thing, you know, we want a piece of this, this good pr, which I think is actually great. You know, Dario from Anthropic once said he wanted to create a race to the top on safety with OpenAI. And this could be argued to be a partial vindication or validation of that thesis, at least for now, in that it's pushed OpenAI maybe, I don't know, but maybe to do something they might not have done otherwise. In either case, I mean, they're putting resources behind this and I think that's great. OpenAI gets a lot of credit in my books for making this move. Yeah. And the GPT Roslyn model that's behind all this was launched back in April. But it's basically an accelerant for what they see as anyway, the early stage research process for drug discovery, which is they think one of the longest and most time consuming parts of it.
- [83:17]
Host 1
And now onto cyber we have using LLMs to secure source code. This is a guide and sort of report from Anthropic that talks about what happened with Miphos, what they found. Have a partner with organizations, also comes along with GitHub repository, which has the reference implementation of the Harness of the sort of general framework of security that partially at least accounted for why Miphos was able to find a lot of these problems. You know, they highlight various things here. Maybe the main takeaway is they found they disclosed like 1600 vulnerabilities, but only 97 have been patched. So you can discover things very quickly. But the verification, triage, patching, actually catching up and fixing all these vulnerabilities is still a challenge.
- [84:20]
Jeremy Harris
Yeah, exactly. Right. So they solved one part of the problem, which is identify vulnerabilities quickly. And historically that was the issue right when it was humans versus humans a lot harder to find vulnerabilities in the first place. Because if you were smart enough to find the vulnerabilities, you never would have written the code that way in the first place, almost by definition than it is to find them as a separate team with a specialization in that kind of thing. And the attack surface is so big, blah, blah, blah. Now things have kind of reversed. Mythos is really good at finding vulnerabilities, but now you need to patch them. And the patch crucially goes through a human review bottleneck. Right. Like you can't just automatically patch these things and sign off on it without human oversight. And so although you can discover the cyber vulnerabilities autonomously and at scale, you can't necessarily patch them at scale. So this is itself kind of a concerning asymmetry, because if you're thinking about the offense side, you care about finding vulnerabilities and exploiting them, not patching them. And on the defense side, you have to both find and patch. So they're just kind of like surfacing this as one of the things that they learned by actually rolling this out in the real world that I think a lot of people wouldn't have seen coming ahead of time necessarily. And so they've developed this sort of six step find and fix loop that they described in the article. It's meant to get you all the way to patching from just a bare bones threat model.
- [85:36]
Host 1
And so that was a blog post with a bit of a guide on how they found this to work, using LLMs to secure source code. How to you could say. Alongside of that, we also got another blog post from them called Project Last, an initial update, and that has more details around how their partners kind of what they have experienced. So for instance, Cloudflare found apparently 2,000 bugs, 400 of which are high critical severity with a false positive rate better than human testers. Mozilla found 271 much more than the previous models. With Mythos preview model, apparently there's now 10,000 high or critical severity vulnerabilities in just a month with 50 or so partners. So clearly again, there's been this fight on like, is Mythos overhyped? Is it just pr? Is it actually a big deal? Or is Anthropic just making a big deal out of it to show off? And I guess last weekend AI is firmly in the camp of like, this is a real thing. Mythos is actually showcasing that cyber is now a real threat model. Now whether that's because of model or the harness or both doesn't really matter. Like Anthropic has somehow unlocked the ability to identify vulnerabilities at a much higher rate than was possible before.
- [87:05]
Jeremy Harris
Yeah, almost 91% of the vulnerabilities that they identified were confirmed by an independent review to be valid true positives. So I never say the debate is over about anything, but at this point, you know, if you want to argue the other side of this, like, you have to try to explain why it is that like thousands of critical vulnerabilities and load bearing infrastructure the Internet critically depends on for its basic function are not a big deal. I think that sentence is just true. So I also think it's like kind of worth scrolling through the old timeline and asking yourself, like, who is saying this was a nothing burger with high confidence and you know, maybe updating your, your view on, on those sources of opinion. I'm not trying to be petty here, but I'm literally like, we may not have much time before we hit models that are really genuinely like freakishly good and dangerous. We better be learning the lessons while we have the warning shots that are just unambiguous. And so, so I think at this point, I'm not saying don't listen to people who are on the other side of this one. That's the last thing I would say. But there has to be some updating here in the system. The antibodies have to be developed.
- [88:08]
Host 1
And I think actually this makes me think of to Anthropics, Reddit. Their whole narrative is, if you think advanced AI would be dangerous, why are you developing advanced AI? And their argument has always been, okay, we'll develop frontier models so that we can develop the safety mechanisms for the most advanced models before they go out of hand. And this is a kind of real proof point of that narrative of like, they have arguably or seemingly the most advanced AI model and they are in fact doing the thing they said they would do. So I don't think the amount of cynicism around what they're doing is really merited. And related note moving back to the government, the White house has approved $9 billion for SP agencies to catch up on AI. So there's been an approval of this $9 billion funding quest to acquire cutting edge AI chips for the spy agencies, I guess they're called here. I don't know if spy agency is fair entirely, but this is including CAA and NSA which do spy related activities, let's say. Apparently they've been running classified AI models on AWS cloud networks, so they would presumably need some truly, truly private, you know, hardware to do their spy business.
- [89:31]
Jeremy Harris
Yeah, this is Blackwell's. They're looking to fund Blackwell deployments now. $9 billion does not buy you a ton of stuff. You know, as, as listeners of the podcast who hear, you know, hundreds of billions of dollars just thrown around will know. But depending on how you point them, it can be actually very effective and it's a meaningful amount of scale. The the question is always okay, but how much scale will that be once the data centers are built and there is a 12, 18, 24 realistically month delay between under the best of circumstances, when you have a hyperscaler, when Elon Musk is pushing things forward, that's what the timeline looks like here. Presumably if you're looking at a secure cloud for the nsa, the CIA, these things are going to be built to top secret TSSCI type spec. That means using a standard called ICD705 which is like how you harden against nation state adversary standard. And that is time consuming. That adds tons of time to your construction schedule realistically and cost. And so, you know, $9 billion may not buy you as many Blackwells as it otherwise would if you're having to spend a lot on security too. So there's going to be all this kind of of this question. There's also a question of like how much do we route through Amazon, which is already operating a secure govcloud thing. And of course the ever present awkwardness of Anthropic being Persona non grata with the Department of War, which Susie Wiles, again Trump's chief of staff, has been working overtime to try to allow Anthropic to route around that constraint. The NSA which has to use this is under the Department of War and so theoretically they wouldn't be able to well naively you'd think they wouldn't be able to use this and Susie Wiles trying to find a workaround. So this is all part of the awkwardness of, I think it's fair to say just the White House having put itself or the part of a war Pete Hegseth had put himself in just this, I don't know what to call it. Hilarious, sad, tragic position of flipping the bird of anthropic right before they dropped a super weapon. So there you have it.
- [91:28]
Host 1
One last story for the section. U.S. law enforcement warns of quote, anti terror extremism as AI hatred grows. This is I think a partially kind of a narrative opinion piece or you know, framing piece by Wired. So this is related to the New York Intelligence and Counterterrorism Bureau had published this report in which one thing they say is large scale protests that evolve into civil unrest and anti tech violent extremist activity could happen within the next next five years. This is apparently circulating around US Federal agencies like the DHS and FBI. And that's about all there is to say on this report. With some predictions on potential violence and so on that could happen in the coming years if large scale AI impacts on the job sector or other factors of life come through. Which I think is something to really be concerned about out to be clear. But it probably is a bit too early to convince anyone of that.
- [92:33]
Jeremy Harris
I think this is one of those, like many things are true at the same time, is simultaneously true that China is actually funding a lot of the opposition to data center buildouts. And that is a cynical play by the way. A lot of those opposition groups often have no idea they're being funded by China. This is something that we talked about I believe for the first time on the public record like a year and a half ago or so in our report. But this is a known thing thing that is true. And also it is true that there is a legitimate concern that you can have a category like anti tech violent extremism used and applied to people who it probably shouldn't apply to. Right. You're creating a whole category like, you know, anytime you look at, let's say, expansions of the remit of national security agencies, you need to ask yourself how they're going to be used in practice, including to go after political opponents and things like that. You know, Bernie Sanders is talking about a data center moratorium. So now we're already flirting with anti tech violent extrem extremism I don't think is remotely meaningful. Like I can call anything extremism. The only thing that gives you something to latch onto there is violent. And so, you know, you would hope that that would be objective enough to kind of address concerns about civil liberties here, but it's a real thing on both sides. And what we're finding here is that this term anti tech violent extremism, it doesn't appear in any public documents, the Department of Homeland Security, the FBI in any of their reports. There's this entire surveillance category then that's being built quietly without the public designation that normally comes with this. And that's part of what's making people a little uneasy. I understand that. I also think it's really important to have a category that addresses something like this because you're going to see it. You're going to see people throwing Molotov cocktails, trying to like, you know, kill Sam Altman in his sleep. Like all the things that we have seen that you're going to see more of. And this story, by the way, didn't even touch on the Zizians. I see you smiling, I think, I think you.
- [94:25]
Host 1
It did have a little note on the Zazians as one of this thing which somewhere in there it's mentioned. It's a long story covering many different things. Actually the Wire story goes beyond just this one report. There's.
- [94:38]
Jeremy Harris
Oh, I must have missed multiple things because the Zizian thing is like, I think as you put. Anyway, look it up guys, Zizians, we're not going to talk about it here, but like that's where I think, think eventually the culty craziness of this stuff ends up going especially as language models are in the loop and start to mess with people's heads more and more. Just a, just a thought but.
- [94:56]
Host 1
And next story, still kind of dealing with safety but on the business side we have YouTube will now automatically label AI videos. That's the gist of it. They'll automatically apply AI labels that their internal system Detect has significant photorealistic AI rather than relying on creator disclosure. There's now the standards of C2PA and other things that allow for the detection but primarily metadata related, I think. So now there'll be a visible AI label which I would imagine we've also seen Meta start doing that. This will become more of a norm over time as the standards around this stuff mature, as you said. And now onto research and advancements, we've got a couple stories. First up, why larger models learn more effects of capacity, interference and rare task retention. So the basic question here is per the title, you know, why can larger models handle more complex stuff which intuitively you're like, well, more weights, so more smart. But what is the mechanism by which more weight Weights is more smart. And the gist of what they're saying is small models don't have the capacity to go for more rare tasks. So they focus in on kind of the general stuff and that dominates a signal. And better, larger models are able to learn the kind of common use cases enough that the error signal, the gradients stop really giving much attention to those things and provide more gradient towards some of the smaller, kind of more nuanced, whatever you want to call it. So I think it does make a lot of sense. It's pretty intuitive framing or explanation, but still another nice work and examining the underlying mechanisms or physics of LLMs.
- [97:00]
Jeremy Harris
Yeah, one of the ways to think about this is it just takes time to learn stuff. It takes many examples of that same thing in order to learn them. And if you have a limited scratch pad, like a scratch pad of finite length, in this case, that's the equivalent of if you're a small model, you don't have much capacity to hold information. You're going to learn a random fact. Here's an example, I guess in biology, I don't know if you ever remembered the Krebs cycle, like I had to. I did biochemistry for like two years in undergrad. I must have rememberized the Krebs cycle like six times in my life and then forgotten it in between every time. And the reason is that it wasn't introduced frequently enough and other stuff just crowded it out, things that I was actually using. And so there's this notion that you have essentially like a model that has a finite amount of capacity. And so the way that they tested it, this idea is get a bunch of very simple tasks, linear regression tasks that require using a certain set of features, say N features, and get models of different sizes to encounter those tasks. Some of those tasks will show up more frequently in the training data, some less frequently. And what you find basically is that the tasks that show up, or, sorry, the features that are required to solve the tasks that show up, often those get learned. And then the features that are required to solve the rarer tasks get learned and then they get forgotten because the next step of gradient descent for the next batch kind of pushes it out. And so like there's just a bunch of noise at that level of the distribution. And so you end up having the model lock in on these like kind of very more frequent tasks that show up. So there's a sort of like update and forget loop that happens. And when you increase the size of the model, you're literally just like increasing from a Frequency standpoint, how far you can reach into frequency space to find the problems that show up less often in your data set and actually master them. And so this is kind of giving you a way to predict the kinds of tasks that models will end up learning to do over time as you train them. And they do test it on a wide range of different ULMA models. So from 4 million to 4 billion parameters, which includes a couple of new tasks that are essentially geared towards exactly this. And they show at a whole bunch of different levels. Behaviorally they show representationally they can show using these basically interpretability tools that larger models actually do embed more of the relevant task features in the longer frequency tail. And then even at the gradient level they show that. Actually this is interesting. In the larger models, the gradients from usual language modeling tokens don't interfere basically with the tasks that are learned. Whereas the smaller models, like every new token you get creates a lot of noise and basically prevents it from, from tracking on or catching onto the learning signal. So yeah, pretty interesting. It means scaling isn't necessarily the only lever. You can also think about raising the frequency of a target task in your training mix. And that could actually be a lot cheaper than growing the model. So that's one of the architectural implications here is that the data set is not like it's kind of an independent axis, the content of it from scaling.
- [100:06]
Host 1
Right? Yeah. I think the sort of intuitive, another intuitive framing is you have more places to put information in your model so you can, you know, instead of overwhelming your model of stuff such that eventually it starts forgetting things from the training data. This idea of, you know, there's some infrequency and so you can't, you need to preserve and have a sense of memory. And once you see multiple examples, you can then generalize to do this general task. But some amount of memorization during training is actually useful to be able to do that. I think that's an interesting insight here as well. Next paper. From simulation to inaction. Post trained language models recognize and react to their generations. So that is the headline of these post trained language models develop an implicit ability to recognize when they are generating their own output versus passively consuming text. And they offer coined that as moving from simulation to inaction. So base language models are passive predictors, right? They don't see the consequences of their outputs. They are truly, truly next token predictors. They are autocomplete, right? Algorithmic autocomplete, literally. And I think one of the distinctions that people have failed to grasp or Maybe intentionally on grasping is that once you get to post training, once you get to reinforcement learning, these models are no longer autocomplete. They quite literally aren't optimized to autocomplete, they're optimized to do something else, which is, well, there's many definitions of it. They're optimized to reason, they're optimized to problem solve, they're no longer optimized for autocomplete. Only the base model does automatically complete. And so interestingly, when you get to this post trained models, you are able to then recognize your own prior generations compared to generations that don't come from a model.
- [102:06]
Jeremy Harris
Yeah, the way that they kind of quantify this is you give a model a prompt and then look at the probability distribution over all the tokens that it could put out, right? So certain probability that it'll put out the, the token, the a certain probability to put out the token horse and so on. The more spread out that probability distribution is, the higher the entropy associated with that next token prediction. Basically, it means the model is really like not sure what to say next. And what they're showing is if you take a model that has been fine tuned and you give it text that it has written, it will actually have very high confidence in the next tokens that it will sample. So it's actually like not spread out. The probability distribution is very focused on a smaller number of tokens. It's a lower entropy kind of prediction. And that kind of makes intuitive sense. The argument here is the model is realizing, oh, okay, well, I actually am unsurprised by the tokens I'm seeing here. So I should generate more tokens that I'm not surprised by. And well, I mean, that tends to correlate with tokens that are consistent with the Persona or the character that it's being trained to become and embody. And so that's kind of the mechanism behind why supervised fine tuning leads to this sort of lower entropy distribution on the back end. What's less clear? And they don't necessarily have an explanation for this. When they move beyond supervised fine tuning and look at like reinforcement learning or like dpo, what you find is a further decrease in the entropy of those distributions. They gesture at a mechanism for the DPO side. They kind of say that like, like DPO reinforces whole samples. So like an entire chunk of text based on preference rather than just predictive accuracy. Maybe it like, it detaches the recognition from the role marker, like from the prompt in some meaningful way. And Then they don't really pin down how and then they go further and they say, well with reinforcement learning with verifiable rewards, you get an even greater decrease in entropy. And they're like here we really don't know. And so they really just have a thesis for the supervised fine tuning aspect, which is still really interesting. But it's also interesting to note that as you go deeper into the fine tuning stack after supervised fine tuning, you get an even greater amplification of this pattern. And so yeah, I think quite interesting, intuitive as far as it goes, and then suddenly stops being intuitive. And by the way, this only starts to happen at larger scales. So at 2 billion parameter scale pretty small models, supervised fine tuning doesn't do this. And again, I think that's pretty intuitive. It's like the model needs to learn the skill to measure the entropy implicitly of the distribution in the prompt in order to be able to model that in the output. You just don't get that at 2 billion parameters. To your point, the pre trained model without supervised fine tuning has only ever seen random tokens in and then please predict the next token. It's never had to take its own text and make predictions off that. And that's why you don't see the kind of of emergence of the Persona there.
- [105:05]
Host 1
And with that we are done with this episode of Last Week in AI. Thank you so much for listening to the episode as usual. As always, we appreciate if you share the show, if you comment on YouTube or review us on Apple podcasts, which may or may not help us reach more listeners. I guess you can also review on Spotify, which I may not see, but I will still appreciate somehow. But either way, we mostly care that people do listen. So please do keep tuning in. When the AI news begins begins, it's
- [106:00]
Andrei Korenkov
time to break it down Last week in AI Come and take a ride get the low down on tech and let it slide last weekend AI come and take a ride through the streets AI's reaching high blue tech emerging Watching surgeon fly from the labs to the streets AI's reaching high algorithm shaping up the future sees Tune in tune and get the latest with ease Last week in AI Come and take a ride Hit the low down on tech and let it slide Last week in AI Come and take a ride I'm a laugh through the streets AI's reaching high. From neural nets to robot the headlines pop data driven dreams they just don't stop Every breakthrough, every code unwritten on the edge of change with excitement smitten from machine learning marvels to coding kings Futures unfolding See what it brings.