Loading summary
Transcript91 lines
- [00:00]
A
Foreign. Hello, welcome to the Last Week in AI podcast, where you can hear a chat about what's going on with AI. As usual in this episode, we will summarize and discuss some of last week's most interesting AI news. I'm one of your regular co hosts, Andre Karen. I studied AI in grad school and now work at the startup Aspirate.
- [00:29]
B
I'm your other co host, Jeremy Harris. I'm wearing a hat because I got back from taking my daughter to the park.
- [00:34]
A
What?
- [00:36]
B
And yeah, actually I'll be away, by the way, for the next episode because I'm on a trip to the States doing a bunch of AI security stuff, but excited to do this one. We punted the recording from Wednesday this week. We're now recording on Saturday and we were just talking about how a bunch of stuff came out in the interim, which is really great. Challenging a little bit because the way my schedule works out, I'm not able to like, spend the same amount of time between Wednesdays and Saturdays. So I'll be a little lighter on the details for some of these things, but I've still tried to prepare a bunch of material for it, so be warned. Andre is going to be carrying some of the weight for some of the more recent releases. But yeah, hell of a lot happened.
- [01:12]
A
Yeah, it's one of these things where it's like a thing in the AI industry where we get waves of releases because I guess everyone wants to be one, upping each other and everyone kind of knows when each other's stuff is about ready to go. So everyone just sort of like preps releases and announcements and then they just do it around the same time. And that's exactly what happened in this episode. We'll be talking about GPT 5.6, we'll be talking about Grok 4.5, Muse, Spark 1.1, a few other models on top of that. And beyond that, the big story, I think, will be in policy and safety with some new vehicle research from Anthropic and a new kind of major think piece from AI 2040. So should be a good mix of content in this episode. We'd like to thank Box for sponsoring us. Box is building the Intelligent Content Management platform for the AI era, serving as a secure essential context layer for Box's AI agents to access the unique institutional knowledge that makes a company run. And that's the key idea. The power of AI doesn't come from the model alone. It comes from giving AI access to the right enterprise content. Box's recent State of AI in the Enterprise report found that 96% of organizations say agents need access to company specific content, but only 36% have connected agents to trusted content across many use cases. And a trust bit is very important. Box is built of security, compliance, governance and threat protection in mind, so employees and agents only access the information they're authorized to use. If you're thinking seriously about adopting AI in your company, think beyond the model. Your business lives in your content, and Box can help you bring that content securely into AI era. Learn more at box.comlwai. this episode is brought to you by Outshift, Cisco's incubation engine. Today's AI agents operate in silos, limiting their true potential. We've been focused on building bigger, smarter models. But scaling up is just one approach to reach superintelligence together. Together we need to do more. We need to scale out and we actually have a blueprint from 70,000 years ago. Humans didn't just get smarter individually. The cognitive revolution transformed society. Because we began sharing knowledge, goals and innovation, agents are now at the same inflection point. They can connect, but they can't think together. That's why Outshift by Cisco is building the Internet of Cognition, transforming AI from isolated systems into orchestrated superintelligence. We're creating an open interoperable infrastructure. Outshift is enabling agents and humans to share intent, context and reasoning. The cognitive evolution for agents is here. Explore the Internet of cognition@outshift.com that's outshift.com
- [04:11]
B
I started Ornod in 2013 and we make bike apparel. The best part of Shopify for me is our ability to run the business as essentially non technical people. We're able to admin everything on the back end, front end and sell things online easily. If Shopify were a bike accessory, I think it would actually be the bicycle. It's the thing that you do the thing on. We run the business on Shopify.
- [04:36]
A
Start your free trial on shopify.com let's kick it off in tools and apps. First up, we've got OpenAI rolling out GPT 5.66 after Government Greenlight and announcing ChatGPT work. So they've now publicly rolled out GPT 5.6 including Sol, Luna, all the variants of a model. I think we already discussed previously how the model was. Sol was a little bit confusing in terms of its benchmarks, but generally seen as like fableish sort of frontier competitive with anthropic. And I suppose the bigger news aside from these models being public is this ChatGPT work thing where they are updating the ChatGPT desktop app, which used to be sort of just mirroring the website where you have a chatbot interface into just codecs basically into an agentic coder. And then they kind of hid the actual chat functionality very kind of subtly in the ui. So they're trying to onboard whoever is using this desktop app into using agents like hardcore. Right. Which is an interesting strategy by them. And also rebranding to ChatGPT work from Codex, which seems pretty smart given like, everyone knows ChatGPT. I think not many know about Codex.
- [06:02]
B
Yeah. And I think in and amongst all this too, the whiplash on the model release permissions, and I will call them permissions from the US Government, because that is what they amount to, is pretty, pretty wild. So, you know, the US government just greenlit the release of GPT 5.6. Of course, the request that they had made was for a review period for the US government to be able to look at GPT 5.6, assess its capabilities and clear it for release. The strange thing that we're getting is there was a report out in Axios that said that the government's position on the GPT 5.6 SOL release was that no permission for model releases is required or granted with respect to 5, 6. And the decisions about the scope of these releases quotes rest entirely with the companies. Now, in theory, of course, this is maybe the case. There is no law on the books. There is no regulation on the books officially.
- [06:57]
A
They could have just done it and then who knows what would have happened?
- [07:01]
B
You know, here's an excerpt and I'm pulling this all from. I, I should. I can't remember whose. Whose tweet I'm getting the. This particular excerpt from, but it was really, I thought, good sort of context for this whole discussion. So if you look at the letter that Lutnick sent to Anthropic when he was approving Fable going back on the market. Ooh, Kind of weird thing to even say in this context, isn't it? How is Lutnick approving Fable if there is no permission required or granted? Like what? Anyway, here is the wording that Lutnick used at the time. He said, anthropic has committed to work with the US Government on protocols and standards and releases for the quotes covered models. He says, I reserve the right to reevaluate and adjust the scope of license requirements on the covered models should circumstances change. What?
- [07:45]
A
And this is the Secretary of Commerce, by the way, not like some sort of oversight thing for AI at all. It's they ad hoc, essentially. All this stuff.
- [07:55]
B
Sam himself said of the 5.6 SOL release says bad news. This is on Twitter at the request of the US Government. It's. It's launching today in limited preview instead of the open access launch we were planning on. We're working with the government to get to general availability as fast as we can. This does not sound like a company CEO who has a free hand to do what he wants. It sounds exactly like someone who understands that in reality, what's happening is the government, yes, will allow you to do it, but then they'll smack you back by having the BIS export control the crap out of your company and effectively grind your operations to a halt with some of your most important customers on a whim. That's the reality here. So there's sort of this funny thing. I have no objection to licensing regimes. Like we, again, we put one together, like three years ago when we predicted this moment would come. It's been obvious we would get to a moment like this. And frankly, a lot of people in the Trump administration in particular have spent the last many years saying that's ridiculous, it would be irresponsible. This is craziness, this is doomerism, all this stuff. And yet here they are doing exactly that, ostensibly, but just not calling it that, which I think is just an intellectually dishonest play. I'm being quite forceful on this because I think we need to force a conversation at the legislative level to have actual regulation to do this with principle. I'm not saying any answer here is right or wrong, but it's like, it is just obvious to all concerned that the current approach is still wildly untenable and just false. We're getting like, lies. There are lies in practice, if not in theory, they are lies in practice. When you look at, like, how is this stuff actually being regulated? There is a licensing regime today in the US for Frontier AI, full stop. You can dress it up, you can say that it's just like, oh no, it's like. But like, ultimately it's also inconsistent. OpenAI was just asked to, I guess, just show do deference to the feds. Whereas Anthropic was export controls. Different mechanisms are being used here in what seems to be a fairly ad hoc way, and they're being very weakly and poorly justified. So anyway, there is also all kinds of undisclosed model loopholes, by the way. So if, if you, if you have a Frontier model that's built by one of these labs, and it's just not Announced some of those models, the internal deployments, if you're concerned about loss of control at least, or weaponization, the internal deployments are as dangerous as the external deployments for a huge fraction of the risk. Surface. China can steal the model, you can lose control of the model, a rogue insider threat in your company can weaponize the model. None of this changes and yet there is literally no effort being expended, seemingly no requirement to disclose internal deployments. So not only are you seeing a totally unprincipled, uncoordinated, seemingly semi thoughtless approach being applied here that's rattling markets, causing all this confusion, but it's also failing to grasp that there are that the exact risk they seem to be concerned about, which is AI superweapons, still exists like fully for a wide range of the kind of threat vectors that you'd be concerned about. So anyway, I just think this is like a really classic example by the way, is say superintelligence, right. A lot of people think that they have something close to frontier potentially. If that's the case, I mean, totally undisclosed. They don't even plan on launching a product. They may have these things running around on their servers. China all in there, Russia all in there, whatever. Yeah, who the hell knows, right? So this is just, I mean it's going to get fixed over time, but there's a pretense here that there is in any way a kind of like thoughtful approach being applied that I think is just laughable at this point. This is a, a chaotic mess and it needs to be fixed. We need to be able to look back, like literally just, I'm just asking for two weeks of look back where the government looks half coherent in a two week window. Please, just like give me two weeks where I can go back to look. What, like, what did the Commerce Secretary say? And okay, yeah, that, that's aligned with what we heard the White House, you know what I mean? Like, we're just not getting that this is, this is just crazy talk at this point.
- [11:49]
A
Wow, Jeremy, it sounds like you're getting to understand US Politics. Because
- [11:56]
B
what I, what I will say is it used to be more coherent, like when the White House put out the action plan, you know, it's this roadmap. They more or less stuck to it for like, I don't know, six months or whatever. And then the anthropic thing happened and I feel like everybody just like got brain cancer or something. And now like I have no idea what the hell anyone's position is on anything. And they Keep saying stuff that's like, dude, like, do you actually have a preferred lab? Because it really sounds like you have a preferred lab and that's not good. It's bad for your standing in the courts when you inevitably have to litigate this stuff. It's. It's bad for the market, it's bad for the coherence of your security and safety argument. This is really bad.
- [12:33]
A
So, as usual, we had to touch on the politics of this. And one more thing to say on that front is an interesting like, as a comparison to what happened with Anthropic, I saw after this release that UK azs, which we've talked about, I think, quite a few times, there was a post saying that when they got access to evaluate GPT5 SOL, they were able to get universal jailbreaks within hours of testing, which is what was not shown to be the case with Anthropic. Supposedly the problem was like this very directed single jailbreak that kind of wouldn't give you much versus universal jailbreaks is basically, we can make this model do whatever you want. Either way, it's very clear there's fewer safeguards in place than what Anthropic had. Anthropic had a very restrictive set of things even with the initial launch that would like, roll you back if they detected that you were doing cyber stuff or whatever. To my knowledge, there's no or there are still cyber things in place by OpenAI, but I think if you compare them, it's very different. So good for us to get GPT 5.6. So we got to vibe check, I guess, of a community. And it's a good model, People like it, it might be Fable level, it might be better than what Entropic has. So it's very nice to see the outcomes of competition and OpenAI sort of regaining its focus and getting back to providing a challenge to Entropic, which we have been sort of focusing on other stuff for a while.
- [14:15]
B
On that note, right? If it is Fable class, one of the things here that is also pretty bullshit is that Anthropic was held back on their release and therefore their monetization of Fable for weeks. Meanwhile, like I want to see the side by side of how long OpenAI was forced to delay 5, 6. If the gap is less that basically the US government just handed them weeks and therefore hundreds of millions of dollars. That's what those weeks bring in in profit. And so, you know, this is just another way in which this unprincipled approach of seemingly, I think to many People picking favorites almost shamelessly. When you look at the language that's used by just some of these, like, it's crazy anyway. So, yeah, it's not a fact.
- [14:58]
A
It's like crazy if you don't live in the United States that this could happen. But why we keep talking about it is that it appears likely that this is just going to be the situation going forward. So this is not a one off case of it happened with these two models. It looks to be the case that the administration is at the point of saying, okay, there's going to be these powerful new AI models and we have a say in what these companies do and we can do whatever we want because there's no rules in place or no policies and so on and so on. So we can expect to keep talking about this unfortunately, as new models from anthropic and from OpenAI get announced.
- [15:45]
B
Yeah, just to also like, you know, people listening, you know, I have been a defender of this administration's approach to AI for quite some. Like, basically for the entirety of their first year at this. You can go back, I mean, I think to a fault in some cases. But, you know, I was trying to find the logic where it was, you know, in Silicon Valley, there's an unfortunate bias in the other direction and I think that needs to be corrected for so you can sort of see the signal and the noise. I think the challenge is this was all kind of cozy as long as an ideologically aligned entity was in the lead. I think that mythos just like broke everyone's brains. I think there was also, you know, David Sachs has no business having an opinion when it comes to AI, like Mark Andreessen. These are people who fundamentally, like, do not actually understand the technical side of the safety argument and the security pieces. And I mean, like, you can go back and see like individual things that show that, but, you know, you can hope that there is enough and I think overall saner minds will prevail to be clear. Like eventually it becomes just like, these are bioweapons, these are cyber weapons. Like, no one can really deny that this is a thing. And I think at that point you're going to see Congress step in, you're going to like, AI 2040 and all that stuff is making is calling the very easy prediction in that respect. And we've been talking about that for years on the podcast alone. And so sorry, I mean, on the podcast alone as opposed to other venues. So anyhow, I just think we're going to get to sanity. It's just like people have to be moved different distances to get there.
- [17:13]
A
Let's say it's, it's a bit of a mess, this whole thing. But if you don't want to hear about the politics, I guess this is good news because GPT5.6 is out. They have Sol Luna and their small model and it is a cheaper alternative to Fable. While you know, at some point first person accounts being potentially better and one of the weird parts with these frontier level models is they do seem to be diverging a little bit in their flavors of intelligence. So these models always had a sort of character to them and slight quirks and differences, but in terms of their capabilities they felt similar to me. And it now feels like there's this high level differentiation between GPT 5.6 and Fable where when doing very advanced problems of like math and so on, they have different sort of high level pitfalls and ways of reasoning. And looking at the problem, it's quite an interesting phenomena. Next up, another new model. This is Grok 4.5 from SpaceXai. So they are calling this an OPUS class model and if you look at the benchmarks it's, you know, generally competitive. Opus4.8, terminal bench, deep swe. This is their coding model alternative for both GPT5.6 and what anthropic is providing. And the big news, I guess besides that they have a new model, people are saying this is a pre trained model. So this is like cursor jumping in and training brand new GROK that is much better at coding. It's very cheap relative to what the alternatives are. There is the pricing is $2 per million input tokens and $6 per million output tokens. That's I would have to check. But maybe two and a half times cheaper than Opus if I remember correctly and I don't even know how much cheaper it is than Fable. So it's a relatively very cheap model that appears to be pretty capable. And from what I've seen of people's conversations, people seem to be saying that it is pretty capable in practice, not just on the benchmarks. So I think we'll talk about some other model releases too. The trend is definitely moving toward a price war where token prices are going to be pressured down. Especially now where Fable you can use no first class model for your everyday needs like the frontier frontier models you probably won't be using as your default. And that means that this price war might get nasty.
- [20:08]
B
Yeah, it's actually quite interesting to think about where exactly you're going to see the profits accumulate for that reason in the space, like there's this argument that I think is implied, maybe has been made explicit. I just haven't heard it quite that way. People say there's infinite demand for intelligence. Yes. Is there going to be infinite demand for the frontier of intelligence or does the frontier of intelligence end up getting consumed by the labs themselves in pushing recursive self improvement? What's the shape of that pyramid? Basically? I think that's a question we don't yet know the answer to. And it's I think more non trivial than I think a lot of people make it out to be. Myself included. Myself from like three years ago, certainly included. So we'll see. But the price wars are definitely going to be an issue increasingly. I do think the frontier will continue to command a premium and I do think that it's going to be hard for players like ROK to actually end up leapfrogging anthropic. We're seeing a lot of stuff like some of the benchmarks that were shared originally showing things that, you know, may have worked their way into the training corpus that were like, you know, these cursor oriented benchmarks and things like that. But one interesting note about this too. So we don't have, unlike previous GROK announcements, what we don't have is a model card that looks at all the, all the usual risks and so 4.5 references a bunch of benchmark comparisons, but we don't have this like structured principled assessment of how, you know, how things are going to work. On the, on the safety side, we know there are some cybersecurity safeguards that were added. They're kind of hand waved and I mean ultimately this kind of hand waving happens when you're concerned about US government stepping in and saying hey, like you can't actually release this man of which there is plenty of precedent now. So anyway, it's, it's also meant for like these long horizon tasks, right, with nominally full autonomy. So the stakes do go up. Whether it can actually execute, you know, on the kind of meter evals type plots that, that everybody cares about on cyber and on, on AI and autonomy remains to be seen. But kind of interesting, it does seem to have at least some pretty good roughly Pareto capability. So it may not be the most capable, but when you look at capability and cost, there's a point on the curve where you could quite plausibly put Grok 4. 5.
- [22:23]
A
To be fair, in the announcement post they do have a single sentence saying we've also added new safeguards reflecting the model's cybersecurity capabilities. So I think they've addressed it pretty well with that one sentence.
- [22:35]
B
That's a fair point.
- [22:36]
A
I retract everything and to correct my previous statement, it's 1/4 the price of opus and 1/8 the price of Fable. So much, much cheaper than the models from Anthropic. Also cheaper I think than GPT 5.6. So very interesting point in time where we haven't seen getting into this price for territory I think until now and in fact with the next model we'll have even more to say. So moving right along, we've got Meta and they have released Muse Spark 1.1 which is essentially just their update of the model that is good at coding or at least like somewhat good at coding. Enough good at coding to make a claim, but you can use this as an agent and potentially part of your workflow. And they are also jumping into a price wars. I have to double check the prices but essentially it's very, very aggressive pricing relative to Anthropic and OpenAI. It makes sense because nobody would even try this as an option unless there's a strong incentives in place. And I've not seen many vibe checks of people trying it. But the benchmark numbers are looking fairly good and I think it would not be surprising if in fact it is sort of starting to compete at least with open source models like GLM and deepsea just given, you know, they collect data, they have the infra. They should have gotten to this point given the amount of resources they just shoveled into this.
- [24:17]
B
Yeah, it's, it's, it's pretty interesting when you look at like the broad fingerprint of capabilities that's advertised here. So the first thing by the way from a. You can tell what part of the model announcements I rush to immediately when I see these cyber capabilities because that's actually going to determine how much money you can make, it turns out in this current domain. And by the way, don't think that Bio isn't coming soon in exactly the same way for exactly the same, for even actually worse reasons because you can't patch human anatomy anyway. So they do these dangerous capability evals. They say they can't rule out the high risk threshold for, for both chemical and biological and cybersecurity. And they say that when they bring in their mitigations they can get the risk down to moderate or lower, which is, you know, the kind of acceptable to release range by their assessment. When you look at cyber though, the capability jump is huge. So Cybench which is the, is one of the classic cyber benchmarks. The score goes from 65 to 93%. That's insane. Like, and remember, right, going once you, once you're pushing 85% or even 80% on a benchmark, depending on what it is, it gets a lot harder to score that incremental point. So getting from 65 all the way to 93, this is a really, really big jump.
- [25:27]
A
And this is relative to the previous Muse Spark 1. So this is true basically for all benchmarks, at least with respect to coding. All of them have very, very significant jumps, I guess Cyber being a part of that. So it kind of tracks. And to your point, they did release a hundred page safety evaluation report. So they do seem to have their safety framework actually being put into practice.
- [25:55]
B
It's weird to see Meta doing more safety work now than Grok, than SpaceX, but that's where we are. Yeah, I mean Alex Wang I guess is scale pilled and that's why he's putting it out. But yeah. And then so there's also Cybergem, which is more of a real world vulnerability discovery benchmark. And that's interesting because so 59% is a score that Muse 1.1 gets on that benchmark that is well below Opus 4.8. OpenSport is like 79. Right. So like huge, huge difference, like 20 points GPT 5.5 is like 82. So in a lot of these cases you see kind of very uneven footprint here certainly relative to the other frontier models. It's impressive like you said and I think it is, you know, you can think of it as in some domains, probably glm 5.2 ish, kind of adjacent, which is impressive. Crappy today is impressive two months ago. Like so, you know, it's worth, it's worth patting them on the, on the back for this. But yeah, roughly GPT 5.5 comparable on bio. So anyway, all of which is to say I think Meta is going to enter the chat on the regulated AI model stuff if this trend continues probably sometime in the next like three months I would guess. So you know, this is going to be a, an industry wide thing. It's no longer going to be just anthropic and OpenAI the President of the United States is going to have to stare at the markets and say not today. And at some point that's going to have to happen and there's going to be a big market correction. I say that as somebody who is in various ways invested as an angel investor, as an investor in Hedge funds. As an investor in various things, I have stakes on this, but I have obviously more on our survival species. So I hope the President is able to say no to the markets when the time comes. And it's going to be a really tough. I mean, you know, then there's the national security picture, and that is a huge challenge. Like that's. It's not. You can't say no to the national security picture. You need to say how. And that's a conversation for another day.
- [27:46]
A
Yeah. And worth noting too, that I think it's fair to assume that there are relationships in place and some people have more leeway. And I would imagine it's true that Meta, as one of the big tech companies, with tech in general having made a big, big turn in Silicon Valley, it used to be the case that tech was progressive, tech was sort of aligned with the left a little bit. That has changed. Tech is aggressively aligned with whoever is in power and are very happy to work with Trump and basically cozy up to him and gain favor. So I would expect Meta in a way to have a bit of an advantage with respect to getting permission to not have oversight and so on and so on. Going back to the pricing, so they are pricing 1.25 per million input, 4.25 per million output. So cheaper than Groq4.5 even. What is this 1:6 price of Opus 48 getting towards the territory of open source models, I think potentially even at the same level, which is used to be like super cheap. So again, part of this price war trend.
- [29:01]
B
Yeah. And one last thought too is there's always this question of what's the personality going to be like once you have a new entrant in the frontier kind of space or potential. One question obviously comes up like, okay, Grok is, is going to, you know, do anything for you. Claude is well, Claudi. And then ChatGPT is ChatGPT. It's going to write you some lists. The trade off here seems to be or the personality trade off seems to be towards over caution least that's one part of the personality here. So their false refusal rates on benign requests are higher than peers considerably in some cases, especially on things like Cyber, higher than GPT5.5 and Opus. So it definitely it's tuned conservative, not politically, but like it's tuned to refuse to answer questions when there's any doubt. So it's I, I find that kind of interesting. This is like a complete pivot from like the Yann LeCun era of Facebook Meta to now. Alex Wang, this starts to look a lot more like a, I don't know, like a cousin of Anthropic. That's a real stretch, but there's a flavor of it here anyway.
- [29:59]
A
And speaking of Meta, we had some other news as well. They introduced Muse Image and Muse Video, their new synthetic media generation models. A bunch of screenshots and videos that showcase that these models are quite good and potentially competitive with a frontier. Also benchmarks showing with their quite strong. And the Muse Video was just in preview. Muse Image was actually rolled out and made usable across their platforms, resulting in backlash because one of the things you could do with Muse Video museimage was use it directly from, I believe Instagram. And if you simply tagged someone who has a public account, you, you could make AI generated images of them. Like it was made incredibly easy to make edits of any account on Instagram. And the backlash was strong enough that they backtracked the release two days or three days after it came out. So yeah, kind of a fumble on Meta's part here where they appear to be making good progress. Muse Image and Muse Video, as far as I can tell, are quite good synthetic generation models. And this is a place where, you know, There's Google and OpenAI competing, but anthropic doesn't focus on this stuff. And not a Banana is a little bit old. ChatGPT is not updated so they could get a real advantage here given the amount of data they have of Instagram. I would imagine that just the training volume is ridiculous, but the product side release here was a bit of a message. And next, going back to models, there's a story Chinese AI models gain ground with US companies as cost surge. So this is a bit of a trend check where it's showing that the share of tokens that are coming from these open source Chinese models, DeepSeek and Z AI has gone up significantly. It has now gone up above 30% weekly since February 2025 on open router. Open Router being one service where you can kind of query and direct your queries to different LLMs and there's some data there. We don't know how many requests are hitting the official APIs. So again, another trend of what we're seeing where GLM 5.2 is incredibly cheap. It's still, I don't know if it's cheaper than Metaspark 1.1, but it's same territory of much, much, much cheaper than anthropic and OpenAI while being very capable. GLM 5.2 had a very positive Vibe check from computer that you can basically use this as your cloud code driver for the most part. And it is sort of like 1/10 for ish, 1/10 ish for price. So we have some data here from open router showing that it is starting to gain effect. Well, OpenRouter of course is a bit of a niche. We don't know how big the impact is, but it's happening.
- [33:12]
B
Yeah. Keep in mind also that China is going to be clamping down on open source at some point in the next year or two. So many of these trends are I think properly viewed as transients where it just becomes untenable for, you know, like cyber weapon models or models that can even be used just to amplify the freedom of the individual. If you're thinking about China in ways that the CCP is not going to like, you are going to see clampdowns. They'll be justified in various ways. They're not always going to be justified. It's like, oh, this is, you know, we're clamping down to clamp down, but the effect is going to be that. But in the transient this is actually really important. Like if you're trying to think about an offensive cyber campaign as part of the Chinese state and you want to like get all up in people's systems in the west we have agents, like AI agents that you can make sleeper agents and you know, our ability to control those, those agents is only going to increase over time. And so while you have Chinese agents from deep sea or you know, GLM 5.2 running on your infrastructure like that is an insider threat and that is a way that people need to start thinking about it. And I mean, I know that they are in various quarters, but increasingly even at the corporate level, I think people are going to start thinking about it that way. Sounds like science fiction, but like that's just where all the scaling lines are pointing. So I think it's an important flag that like just in the same way that people are suddenly get feeling a shock from overreliance on the frontier labs and like, oh man, the US government can just like shut down my access. Like this is no good. Yes they can. They will continue to do this like until morale improves, the government will keep clamping down on models. But the same is going to be true and perhaps ought to be true from the open source side. Increasingly over time, you know, if you, if you don't want your, your like ip, your commercial secrets to be like sent overseas, increasingly you're going to want to like look at a model that was made, you know, with the hardware that you trust or software trust.
- [35:00]
A
Yeah. A couple more things I'll say on this front. I think it's a nuanced picture and I think it's interesting with these open source models because they are open source so you can take the actual weights that are public of the model and just look at their performance. And this is why having Safety Institute and cyber kind of protection institutes is actually very beneficial I would imagine is if there is a sleeper agent component. Well it would be quite a big failure of the safety institutes to fail to detect this potential in these models. And secondly because they're open source, I mean you can align it all you want after with just some post training. Right. So the picture is a bit muddled is all I'm saying. It is because they are open source, the models don't have to be used as is once they hit west and there are providers on the western front of just inference that you would probably go through rather than going straight to Z AI and so on or trying to host a model yourself, you would go to Fireworks or Groq. And they typically put these models on their platform with some caution. So it's a bit of a nuanced picture and we can expect for the near term for these models to keep coming out and keep providing pricing pressure.
- [36:22]
B
Absolutely. On the pricing pressure. And I very much agree with like your structural assessment. Right. Like having the weights definitely reduces the risk considerably. One of the challenges is like we start to flirt with. We basically need a solution to the alignment problem at a certain point because we still don't know how to even detect sleeper agents, let alone, you know, excise them from these models. And there's like maybe that'll change. But there are a lot of people in the alignment community who actually see the sleeper agent kind of intervention as being equivalent to a big part of solving the larger alignment problem, which a lot of people don't think is theoretically possible. But like whatever. So we fork out. And you're right, like there's a world where we certainly benefit either way. Having access to the weights, how much we benefit is a really big, challenging, thorny, open question. And I think ultimately it's a leading question that nation states will be asking themselves as they inevitably start to weaponize these models like it's it, you know, this is going to be part of the future of subthreshold warfare.
- [37:18]
A
Yeah, we have seen by the way. I mean this is not just speculation. We do know that these Models, if you ask them about things that relate to state sanctioned discourse. If you ask about Chinese history, the models are going to give you the Chinese history from the government's perspective in every case. So these models are very much already being influenced by policy and it's not. Yeah, I totally agree that there is a real potential for them to be, you know, the sleeper agent thing sounds ridiculous, but it's possible. Right onto applications and business. And first we got Meta. Again, the news is they are planning a cloud business to sell AI computing power. This is according to Bloomberg. They are doing this cloud infrastructure business internally called Meta Compute that would sell outside customers access to AI computing power. And Meta's own AI models competing directly with aws, Microsoft Azure and Google Cloud. Very interesting. Kind of mirrors what happened with XAI or Space XAI now, where they were a model company and more so beyond being a model company, they spent absurd gajillions of dollars building up data centers that were kind of leading cutting edge and they built as much computing capacity as was possible basically as quickly as possible. And were they able to use all that compute, who knows? It's actually, you know, to ramp up your team and your training processes and the infrastructure and so on and so on and so on. Having four data centers suddenly doesn't mean that you can utilize all four data centers effectively. And I think there is a case to be made that Meta, as an actual business that needs to make money and doesn't get kajillions of dollars from VCs is starting to look at their balance sheet and saying we are spending kajillions of dollars on these data centers and what are we getting in return? Nothing. And that would mean that there's a real case for them to seriously move in this direction.
- [39:31]
B
I like that the word kajillion has now become a unit of measure. On the last week in AI podcast, we've given up now the economy is unmoored. Yeah, there's a lot going on here. One piece is. It is absolutely true that Meta. So every company has this SpaceX, I'll call it problem or incentive where if you're trying to compete at the frontier, you're going to try to buy hundreds of billions of dollars worth of data centers a year and a half from now. You're going to build them and then you wait and if it turns out that you fucked up and you don't have a genuine frontier model because you're SpaceX, because you're Meta, because you're whatever, now you have a bunch of Excess capacity. And like you don't have an economically valuable use case for it. What is an AI model? It's a funnel that turns compute into intelligence. That's what it is. And so there's two kinds of value. There's the funnel and then there's the compute. And then you put those together. Some companies have one, some companies have the other. So Meta here does not have the intelligence, they don't have the frontier models, but they have the infrastructure. So yeah, that pushes them towards becoming a NEO cloud. But the challenge they have with SpaceX is that SpaceX is differentiated by, by its institutional and cultural capacity to build infrastructure really, really fast. Meta has historically not been scale pilled. It has historically not been the company that believes in superintelligence through scaling. It has historically not been the company that like makes bets that are of the same shape as the anthropic OpenAI bets. And so I think like right now a lot of the story is framed around excess capacity that Meta happens to have right now. So it's unclear whether they're actually going to like pivot in a long term capacity. And the same question by the way has been raised about SpaceX. Like, you know, Colossus 1, they rent that out. Colossus 2, you know, they give it yes to cursor, but then they acquire cursor. And so like yeah, it's kind of in house. Maybe they are trying to data centers in space.
- [41:22]
A
Right, That's a new thing.
- [41:23]
B
Yeah, exactly. Like so but the question is like whose model runs on an infrastructure? And what's becoming pretty clear, and that wasn't at least, certainly wasn't obvious to me like three years ago, was that we may end up living in a world where you have competition among the frontier labs at any given time to figure out who's got the best model. And then like once it's clear that let's say Claude is the best model now everyone reorients their inference compute to service Claude because that's just like the most financially. Like that's the highest ROI thing for the, from the market standpoint, it's just rational. Now everyone who wants to be a frontier lab continues to compete along the training compute axis because they still, they want to get that best frontier model for the next generation. You know, two weeks later. We want everybody to be competing to run our model. But you could imagine at least the incentives, one part of them that is legible to me at least right now, points in this direction where you actually might have more fluidity. Everybody's competing to build compute. It's unclear who wins, but whoever wins they'll be happy enough to like have their model run on other people's infrastructure and rent that and everybody else will be happy to to amortize the cost, the ungodly eye watering cost of this infrastructure. Like people aren't going to pay us to use our shitty model. We might as well like get money from the leading lab so that this whole thing wasn't a giant waste. And so the question really is where does that margin go from the actual model training part of the stack? And that's where you know for a while whatever company happens to be in the lead at that time gets to just earn a lot more margin. And so I think that's potentially part of this. You could ask yourself like what does meta have over a typical NEO cloud in this space? And like the answer isn't great. They don't have like the tpu. They don't have like these massive like internally developed really high performing ASICs. What they have is financial capital. They're less risky than a core weave because they don't have like crazy debt to equity levels. They have an actual cash flow business in advertising that can massively support this. And so if they wanted to, literally just because they're a floating giant ball of cash with a decent technical bench to them, they actually could move into this market. It's just a question of like how scalable they are and where they see themselves stacking on the kind of model part of the set.
- [43:29]
A
Yeah, I think I'll, I'll say meta to me looks like it have a little bit more of a advantage in the sense that they're a mature company that has data centers for their websites. Right. So they are in the data center business, broadly speaking. They have however many, they have built many of them. So they have the structures in place to do that and a talent in place. They also have worked on custom hardware for inference which we've discussed. I think it's moving along to a decent place. The real challenge is they are not in the cloud business and have not been in the cloud business. So they're competing with Google Cloud AWS which have been clouds for a very long time and have served GPU capacity for a very long time. One other thing I'll say is to your point of we've seen this play out earlier this year. Anthropic had explosive growth with cloud code. Explosive. I Forget it's like 10x30x ridiculous in the span of months and they ran out of compute. It was like a ridiculous situation where suddenly you could barely use cloud code if you were on the $20 plan or whatever because you constantly hit limits. And the models got worse. It was very, very obvious they just did not have the compute to serve the demand and they had to scramble to make a deal with XAI and so on. And if the projections about GDP and AI adoption and AI capabilities hold, where we have another 30x jump in demand in like a year, which is plausible if you think that cloud cowork and so on will continue to be as good for professionals as they have been so far, the picture of being a provider of a data center starts looking very appealing because Anthropic will have to pay you whatever you're asking to continue to serve their customers.
- [45:32]
B
Yeah, you're right. And it's sort of this game of chicken, right, that they're playing. It's like, it's a horrifying, like, proposition for both because while Anthropic is absolutely desperate to open their mouth wider to be able to suck up all the sweet users. That sounds horrible anyway, you know. You know what I mean? Yeah, let's pretend I didn't say that. But like, just as Anthropic's desperate for that, on the back end, you're, you're the, let's say you're SpaceX. Like you're sitting on these, these data centers that you built. You're sitting on a massive capital outlay that you just put out and it's
- [46:03]
A
not making you any money and it's depreciating rapidly.
- [46:06]
B
Exactly, exactly. It's so, so you're actually like in a game of chickens that you like, nobody wants that to, to go on for very long.
- [46:12]
A
If you don't make money from it in a year, like, you're screwed. You messed up GPUs are outdated. Like you wasted your money, right?
- [46:20]
B
Yeah. If you have a gigawatt, you know, $100 billion, like a gigawatt cluster, roughly ballpark is $100 billion or so. If you have that just like hanging out. Like its lifespan is like, I don't know what, call it like 10 years or something like order magnitude. So like, yeah, yeah, why don't you go and like light $10 billion on fire. You're obviously not going to do that. And so you're going to make whatever deal. Anthropic wants the benefit, but you're dealing with the loss aversion at that point. And so anyway, it's a, it's going to be an interesting economic time. That's why I think the economic forces pushing for this kind of call it like compute infrastructure fluidity where it chases like whoever happens to have the frontier model in that moment. The argument I think is pretty strong. I'm, I'm very curious what that adds up to for the neo clouds and stuff, but seems like it's there.
- [47:05]
A
And I think this also adds interestingly to the price war picture where to do a price war you need to be able to drive down your costs, ideally to be able to make this profitable. And one of the interesting things with anthropic is it appears that they have become profitable. They've managed to make this business model work, but with rather expensive models, the most expensive models on the market. When you have more supply of data centers, and this is just speculation on my part, but you would imagine that if you have more supply of compute in general, hitting a market. And I think all these companies OpenAI meta, all of them have made very, very deep commitments in capex to continue building data centers. So I would imagine the compute is coming online and will continue to come online in the next year or so. And it's a staggered process. So as you get more of these data centers finished up and ready to serve demand, you have more supply of compute. And that means that the tokens can be produced probably cheaper, and that means you can charge less and great, now it can compete in a price war and the consumers win, basically. But let's see how it goes.
- [48:22]
B
It also, it depends on so many factors. How does dollar denominated demand track with the actual intelligence per token? And then also like, can you grow the data center base fast enough for that increased demand? What Claude suggests is like actually in some cases you suddenly hit a point where the answer is no. And so this is like really it's hard because it's a function of what new companies you can create as a function of breaking through those new levels of intelligence per token. And also a lot of this is just almost literally impossible to predict. This is what markets are made to discover. But yeah, it could work out in either direction. And lots of room for speculation.
- [49:00]
A
Beauty and markets. Discovering what prices should be truly a miracle of, I don't know, modern business summer routines live or die by how easy they are. And honestly, if something takes too much effort, I'm out. That's why Groons is my go to. It's my one daily pack of gummies covering my greens, vitamins and minerals all. Plus it has 6 grams of prebiotic fiber, which is more than 2 cups of broccoli. No mixing powders, no giant pills, no hassle. I just rip open the pack and I'm done. They taste good and they make it easy to stay on top of my health even when life gets busy. Save up to 52% with the code podcast at Grooms Co that's Code Podcastruns co Next story Related to Data centers as well like US Energy Regulator Sets Ultimatum for Data Centers so this is ferc, the US Energy regulator issued tailored shock show cause orders under some section of the law to each of the six regional grid operators under its jurisdiction, directing them to justify or reform the rules that govern how the data centers, manufacturing facilities and other large energy users connect to the electric grid. And this is a very notable point, Jeremy, I'm sure you know more about this of at some point the data centers are exceeding what the energy grid can provide. In the case of Xai, they have a bunch of gas turbines that are spewing pollution and so on that are operating to produce electricity. So yeah, what is the picture here Jeremy?
- [50:34]
B
Yeah, I think you hit the nail on the head. It's, you know, a lot of these data centers increasingly are moving to behind the meter energy generation, which means they're not on the grid. And this is happening really as much as anything. Well not partly it's happening for political reasons because it's untenable to have people's energy prices go through the roof and then you get political pushback and blah blah blah like no data centers in my backyard Bernie type stuff. But then also there are other advantages to it. And so now you have regulated energy regulators like, like PJM is an energy as a grid operator that recently had to basically order generators to run at maximum output and bring idle power plants online because there was a heat wave and they couldn't support the heat wave because of all the new load on the grid. So like this is having an impact where you're flirting with brownouts right now. And you know, if you look, if you plot out the rate of energy consumption from data center demand, it's like right now it's in the single digit percentages of total US Energy production that is only going to increase. And so you're going to put stress on the grid and eventually you're going to find brownouts become a thing. And there are all kinds of ancillary issues where the regulators are now getting their feathers all ruffled. You know, one example, and we're doing a like a pretty deep dive on this right now that'll be public in the next couple months probably. But one thing to keep in mind is like, so a lot of the infrastructure, the energy infrastructure that these data centers use is like designed to trip on the same signals. And so if you have, you can imagine like basically a correlated risk where one kind of signal on the grid basically shuts down a whole bunch of data centers at the same time. So you've got this on the one hand like super high correlated risk because this has been a totally unregulated market in a lot of ways or in this way. And at the same time you've got the risk of brownouts and things like that. The regulators, like Department of Energy is going to have to step in. They're going to start to insist on big changes. And so again, this is one of those things where the past trends are real, but also we're going to learn a lot about the impact of government regulation in the next couple months.
- [52:31]
A
By the way, does anyone remember climate change? That climate change is a thing and pollution and carbon? No, I guess it's just me. Yeah, not helping all this stuff.
- [52:43]
B
Yeah, I mean, you know, and, and I, I think it's the thing you could make the argument that like for almost arbitrary climate related problems, having better AI that can help us solve these problems is a thing whether it's anthropogenic or not. Also because like you know, either way, if they heat, you know, heats increasing in shit like carbon is science, you
- [53:01]
A
produce more carbon, it's gonna make things worse. And you know, but just like you
- [53:06]
B
could find ways like regardless of sort, like you could find ways around it that are more creative with it like blah, blah, like. Yeah, like yes, yes, yes. But it is funny how the discourse
- [53:15]
A
is like the point also is, is I guess if you have a rush to add more capacity and build data centers and so on, you're not going to get clean energy, renewable energy to drive these data centers. And that's already been shown to be the case. Google Meta, et cetera, have made deep investments in powering their data centers with renewable energy. And that is sort of like that by the wayside. It's not a big priority.
- [53:42]
B
Well, nuclear, nuclear is coming online, suit like that. I'm a huge fan of that. I don't know how the hell we got it in our heads that like nuclear is bad, especially given the politics. Yeah, yeah, yeah, exactly.
- [53:52]
A
I mean it's, yeah, it's starting to actually. I don't know how far along that is. We've covered many times sort of mini Nuclear and sort of promising looking technology. But is it getting to a point where it could be powering data centers?
- [54:08]
B
Yes, there are a bunch of data centers that are like have plans for nuclear buildouts. SMRs are probably going to be the biggest kind of sort of shift. Expect the first sm. Like these are small modular reactors. You can expect the first to come online sometime. Like the most optimistic I've heard people like who are actually like in the data center building space say is like 2331, maybe 2030. So it's like you know, if you believe in Leopold and whatever AI 2027, like we're way into super intelligence territory by the time that matters. But. But who knows?
- [54:39]
A
Yeah. And you know how much does super intelligence help you with stuff building nuclear safely anyways onto projects and open source and we do have a couple of interesting new models that are open source starting with nemotron-labs- diffusion a tri mode language model unifying out regressive diffusion and self speculation decoding. This is from Nvidia in their Nematron model family of models and the short version per the title here is. So you can do decoding, you can do language models in different ways, right. The standard way is autoregressive, meaning that it you output one token at a time effectively. So you provide the input, you give the output and you keep sort of doing this. Loop diffusion as we've discussed many times is something typically done with images where you sort of like produce the entire thing at once over multiple steps and sort of make it better and better. And you could do it with text interestingly. So instead of going left to right word by word, you kind of produce an entire paragraph at once and denoise it until it's correct. And then there's self speculation decoding which is kind of like standard auto aggressive but you have within the model some shortcuts more or less that let you be quicker. With diffusion also can potentially be quicker. So in this try mode approach they trained a single model that can do any of these and also has been trained on each of these objectives and it allows you to switch between causal and bidirectional attention. Meaning that you can use these two training objectives and they're releasing here base instruct and vision language variants at 3 billion, 8 billion and 14 billion parameter models. The models are very quick so 6x more tokens per forward pass than for instance grandfree 8 billion while being comparable. And I don't know how we got to this point where Nvidia has produced the most interesting Research on neural net architectures and potential alternatives to Transformers, I guess we've seen from academia, Mamba and so on. But as far as like large scale AI model demonstrations of something that could be better than Transformers, I mean this is pretty exciting stuff.
- [57:12]
B
Yeah, I mean, coming from, I guess, a company that knows the hardware so deeply, you know, if your hypothesis is that progress is hardware is determined by hardware, this may feel like a natural place. But we just hadn't seen it to your point. Like, you know, the last time when we started covering the big Nvidia open source releases, I think you had to go back years to Nematron, Turing and lg. The kind of big Microsoft video collaboration from like, I want to say 2021 or something was like the latest, you know, the biggest case that you could point to. And now suddenly here they are. This is a really interesting paradigm. I mean, as you say, it's this combination of this think of it as like auto regressive modeling left to right and the diffusion piece. And they do a couple things to kind of make it all work together conceptually. So the first is they do still have to use just standard traditional autoregressive pre training to get the model up to speed, right, to set its priors. And then they turn on this dual track autocomplete end diffusion thing. But you see that a lot. Like there's often a need to just use auto aggression to get these things off the ground. Once they turn on though, this dual track thing, they do it in blocks. And so instead of having your entire output window, the way diffusion works is you kind of start with random noise for all the tokens in the output field and then you kind of gradually populate them and get more, more confident as you do more steps of diffusion and you zero in on what the text needs to look like. And of course you get the benefit of being able to look at the whole body of output text at the same time. Whereas with autoregressive modeling, the challenge is once you choose a token, now you're stuck with that token for the next one. And so if you go off in the wrong direction, you can't go in like 20 tokens later be like, ah, that's no good. And so you'll see the models kind of doubling back a lot as a result of that, because that's the only way they can get themselves back on track. And so what they do here is they're going to do what they call blockwise diffusion. So you split the sequence into blocks and the Model denoises one block at a time. It kind of treats the earlier blocks as clean context. And so it's bidirectional. It looks in other words at the whole chunk of text within a given block, but it does lock in past blocks as it goes. So it's causal, it's sometimes referred to as causal.
- [59:23]
A
It's kind of like a mix of the two basically. And this is something we've seen already as a potential pattern. But this is to my knowledge the like biggest demonstration of this idea. And in general like this hybrid approach.
- [59:37]
B
Yeah, and, and they also, they even have this like. Interesting. So, so the backbone of Transformer is the residual stream. It's kind of like the, the, the, the vector that gradually gets modified at each layer as, as more layers add sort of more, I'll call it context, but I, you know, more, more information to the vector and let it evolve. And what they have is what they call dual stream attention. So they have this like this clean stream that's just causal, it's just used for the autoregressive part of the modeling. And then they have a noisy stream which they use for the diffusion loss and they're processed together in the same forward pass. And so both of those objectives, the diffusion objective and the autoregressive objective, they're computed at the same time. And so that's how you kind of get these to work together. They find that there's this like number that they use to vary the balance of how much autoregressive to how much diffusion to put in the objective, which is a very common thing. The interesting thing is that they find that the losses peak at the same, they kind of like rise and fall together, both the diffusion and the auto aggressive losses, which is not intuitive and very interesting. It suggests that they're complimentary rather than fighting for model capacity in some sense. Kind of interesting across the board and conceptually I think one of the deeper attempts that I've seen to marry these two concepts together
- [60:52]
A
and one more open source release, this one a bit less to say, but still interesting. Tencent has released high free and opened 295 billion mixtures of experts model with only 21 billion active parameters. So the short version is this is not as good as GLM 5.2, but pretty decent, pretty decent intelligence. And it's smaller and it's just another model that's like pretty solid, it's completely open source, it's pretty cheap, you can use it potentially even in a home compute cluster. So the models for now are continuing to be released, but they may or may not keep doing that. Onto policy and safety. And we begin with some interoperability research, effectively from Anthropic and potentially some kind of safety philosophy, whatever you want to call it. And this is, I think, the second most exciting news of the week, in addition to the model releases. So we'll probably spend a small amount of time getting into the details. So the name of the research for Entropic is verbalizable representations from a global workspace in language models. And I'll try to provide the quick TLDR and see if I can manage it. So they begin anthropic with kind of laying the groundwork conceptually of what workspace is. And this is a model that has come out of modeling consciousness in humans, where one aspect of it that you can pinpoint to is consciousness effectively is attention, mirroring attention, kind of directing it. And you have this sort of workspace where you slot in some stuff to be consciously aware of. Most of the processing of information, sensor data, et cetera, is unconscious. You're not aware of it, but it's happening. And then you kind of bring in bits to be looking at and thinking about, like the active thinking you're doing, the active verbalization of things and things you're keeping in mind to draw on. Kind of at the tip of your consciousness is this workspace notion where a bunch of stuff is happening that you're not conscious of. And then there's this thing that you can kind of use consciously. And that's effectively what Anthropic is, at least comparing this method to. So the gist of the method is this kind of idea of, Via some cool math that relates to Jacobians and whatever else you want to get into. The gist is you can find the tokens that the model seems to be kind of potentially ready to say that are on its mind, so to speak. So this is very different from the traditional interoperability technique we've talked about a lot, where you can look at vectors within a model that you can try to map to concepts. The difference here is instead of looking at vectors that sort of roughly map onto concepts, which is sort of this, like, unconscious, diffuse, messy kind of soup of stuff going on inside your model. This is telling you, for a given token or a given token sequence, this is sort of what explicitly in terms of tokens, in terms of words, is going on inside the model. And there's a whole lot to say about the technical aspects of this, the interoperability aspects. We will not be able to get into all of it. But I'm sure, Jeremy, you have a bunch more to say.
- [64:28]
B
Yeah, everybody be prepared to roll your eyes for the annoying Fermi technical, the classic things that you see. So, you know, imagine every. We talked about the residual stream being the backbone of the transformer, right? This, like, vector that keeps getting updated for a given token, every layer, until finally you get to the last layer. And at the last layer, that vector gets decoded into a probability assignment for every possible next token. So you have this vector, this list of numbers that in some sense encodes the meaning of the vector you're about to predict. But then you need to multiply that vector by a matrix, a decoding matrix that turns that into concrete probabilities for the word, the. The word apple, and so on. And so historically, what people would do to try to answer the question you just raised, right? What is the model thinking at some middle layer, which is kind of where you want to. If you want to monitor the model for suspicious sketchy planning, if it's going to kidnap your children and murder you in a dark forest, you want to look and be able to tell, is it thinking those thoughts somewhere intermediate in its intermediate layer, Basically, what is it not telling you?
- [65:35]
A
Right, yeah. Inside of it, aside from the output, what is going on?
- [65:39]
B
Exactly, exactly. And so one historical naive approach people would take is they would take the decoding matrix at the end. They would literally just like, use it to decode immediately whatever the residual stream was or the residual vector was at any given layer. So let's look at layer number 27. Let's just whack it with the same decoding matrix that we would use on the very last layer. The problem here is that that decoding matrix was never optimized to work with that particular layer. It'll give you probability assignments over next tokens, but it's not obvious what the hell those are actually supposed to mean. So somehow what you actually want is the ability to say, okay, if I took this residual vector at layer number 27 and I modified that residual vector a little bit, what would the impact be downstream on the final predicted token? Let it propagate all the way through those other layers. And like, if I just modify, you know, the residual stream at layer 27, how then does that change the output all the way up to layer 78, or however many layers there are when it actually gets decoded? And instead of whacking it with a matrix that was only optimized to work with like layer 78 or whatever. Instead, try to come up with some layer that approximates all of the shit that happened between you. And like that final layer. That's what this is. So that's what the Jacobian here is doing. It's an approximation and a pretty good one, a pretty principled one. Of all the crap, all the transformations that happen between, you know, call it layer 27 and the final layer and the output. And so it actually does give you, in some sense, the best estimate that we have as to what the model was thinking, let's say how it was trying to nudge the final output.
- [67:21]
A
Yeah, I think the verbalization aspect of this, because it is nuanced sort of, you can look at the embeddings, you can look at the activations at layer whatever and sort of say, what is these activations telling us? And that's the traditional way. So the way this is different from the traditional interoperability techniques is this verbalization aspect where you look specifically at the token outputs, the words the model would lead to, as opposed to whatever soup of stuff going on inside of it. And at least the argument that Anthropic is making is that this is akin to this workspace model and is actually interpretable as something akin to like what the model is conscious of. Now, whether that's true is a whole other topic with different perspectives, but it is a compelling and kind of intriguing argument.
- [68:16]
B
Yeah, exactly. The connection here to the whole consciousness thing, because it may not be obvious from our description itself, are like, how does this actually tie into that? There's a couple pieces of evidence. So one, if you ask the model what it's thinking. Okay, so if you look at the space of words that the model's thinking about at layer 27, that you can, you can figure out using this technique, if you swap one sort of one vector for another. So. So you know, if it's thinking about cell or switch it for rugby, that will actually result in changes downstream that match exactly that. So you can do these pretty surgical interventions. So. So this means that there is like some, some like causally connected thing here. They also do stuff like tell the model to not think about certain concepts, and they find that actually these intermediate layers are forced to think about the concept, which is sort of funny. It's like humans, if I say, don't think of a pink elephant, like, first thing you're going to do is think of being elephant. But the key thing is. So the argument for the consciousness thing is that there is a kind of reasoning that we cannot access, but there's also a kind of reasoning that where basically like all thoughts get promoted to some shared reasoning space that the full human brain can access. And that that kind of shared workspace is what consciousness is about. And this is essentially that allows them to sort of do a. A dissection of a language model and show that something similar is happening there where you actually have layer like the earlier layers. Just like the model can't access the thoughts that are going on there, it's just too early. The processing of the residual stream isn't yet to a point where it's thinking coherent thoughts. The thoughts are too raw. But once you get to those middle layers, you actually do get to that workspace. And they're able to show through a bunch of experiments that this kind of maps onto intuitively how you might think about that in the context of, you know, the workspace associated with consciousness. So quite interesting. Quite deep when it comes to that. I wish we had more time for this one, but hopefully this gives a general sense.
- [70:09]
A
Yeah, honestly, we could spend like two hours typing deep into this and maybe we need to interpretability episode. We keep saying we'll do one one day, we'll do a deep dive episode again, but it's hard to find the time on the consciousness piece. So I think it is important to note nuance here where there's kind of a few things you might be referring to when you say consciousness. Right. There is sort of the functional aspect of consciousness, which is like there are thoughts, there's like a process going on versus how consciousness and thinking and reasoning is working. And global workspace theory is part of that. It's sort of trying to map out, okay, here's how thinking happens. Right. A completely different aspect of it is like the experience of consciousness with kind of phenomenal aspect of like, I feel that I'm conscious and I'm aware of it and so on and so on. And these two things are distinct. We don't know why, you know, we know our brain is doing stuff. We know that thinking is happening in some sense. Reasoning, data processing. Why that data processing seems to result in our kind of conscious experience of the world is the deepest, hardest problem of philosophy. The hard problem of consciousness, anthropic is not making a statement as to that case. So I think there's a lot of backlash against anthropic whenever they do this kind of stuff, which I'm getting very tired of, of like anthropocators being like anthropic always mention safety, always goes on about consciousness. But when you say consciousness and global workspace theory and so on. What we are claiming here is that functionally it looks similar and you can make a case that there's these kind of like thinking esque workspace esque mechanisms within the model, which is plausible, right? Because for various reasons at least you can say we have this model of how thinking happens in humans. There might be a similar model you can apply to AI. And this is an intriguing aspect. Whether they should have framed it this way in terms of research is a kind of thing you can argue about. But yeah, very interesting kind of new tool in the interoperability toolbox and I think another pretty powerful tool potentially, if it turns out to be reliable. Next, Beijing is looking at curbing overseas access to China's top AI models, according to sources. So Chinese authorities have held meetings over the past month with top tech firms including Alibaba B&C AI about potentially restricting overseas access to China's most advanced AI models, including those not yet released. This is according to free anonymous sources. These are led by China's Ministry of Commerce. All secret, but wouldn't be surprising, right, if this were to come to pass. China has nothing to gain basically from releasing open source models to the rest of the world. So we are getting some indications of it at least being floated and the pieces being put in place.
- [73:28]
B
Yeah, I mean it's, you know, we talked about this idea of sleeper agents and I think that will, will start to, I mean if you start to see the Chinese, the, the CCP actually endorsing the continued release of very powerful open source models, that should increase your baseline belief in the, the probability that there are sleeper agents built in there or that there is some kind of game afoot. It doesn't mean that it's guaranteed. It's just like, just think about the incentives and that's where you land. They do want to have more market share. Obviously on the open source side it is helpful in a whole bunch of ways. But here the Chinese Communist Party is clearly making that calculation themselves and going, well, you know, we're not so sure that we, we like this. So I will say, you know, for people who model the Chinese Communist Party as being this benevolent, this sort of like benefactor of the open source space and how wonderful it is that all the open source models are Chinese. Yeah, I guess hold your breath for the next two years because you're going to end up seeing a lot of similar moves just as everybody realizes the strategic significance of these models. And again, it tells you something. If they keep publishing this that tells you only one thing. I mean, could tell you other things, but it may tell you something quite interesting about what they're putting in those models. So, you know, we don't know how the restrictions would work. There was a May roundtable of a bunch of Chinese legal experts who produced some kind of summary of another proceeding. And in that case, they were talking about a tiered system with like basic open source models that would be subject to simple filings, you'd have more advanced technology that would face security reviews, and then the most sensitive frontier models would just straight up be barred from public release or restricted to domestic use. So everyone seems to kind of be converging on the same obvious truth, which is you can't have weapons of mass destruction in the form of like really, really powerful AI, cyber and biomodels just in the hands of random people. And again, it was foretold five years ago by a lot of people, and a lot of people said it was ridiculous at the time, but like, if you just kept plotting the scaling curves, we were always going to get here. So this is in some ways the least surprising, surprising news of the week.
- [75:27]
A
Right. And I think even aside from government interference, Deepseek is looking to IPO Tencent and then Bay Dance are massive. So we might just see the same thing that happened in the US happening in China. Right? Where at some point your models are good enough, where it doesn't make sense to open source anything, you're competing with other Chinese companies, and if you have the best model, you keep it to yourself. And so either way, I think this open source gravy train is probably going to end as it has in the US and next up, the ex OpenAI employee behind AI 2027 recommends a rosier path. So speaking of China, this is about AI 2040 Plan A, which per title is from some of the same team who did AI 2027. So, quick recap, AI 2027 is kind of a narrative that tried to sketch out why you should be afraid of misalignment and potential X risk and, and generally how things would go that could lead to a catastrophic outcome or potentially a good outcome. And it made a bunch of predictions in a sort of narrative framework, some of which have been very accurate about, for instance, the rise of coding agents, government starting to play an active role in mid-2026, things like that. And AI 2027, it does predict a takeoff scenario where at some point we get recursive, self improvement, something, whatever you want to say, and then the models get crazy Crazy good. Everything is turned upside down and AI can kill us all. Right, so this is the follow up to it that kind of tries to lay a narrative of a path where AI maybe can not kill us all, ideally. And the gist of it, that's the pitch, folks.
- [77:22]
B
That's the pitch.
- [77:24]
A
And the very high level plan is like, okay, so we want AI not to kill us all. We also expect AI to become super intelligent probably. So what we need to do is make sure we have alignment figured out before AI becomes like too hard to control and can kill us all. And the way they say we need to do that is slow down development, right? You need the models to not self improve and explode in intelligence super quickly. We need to be like, don't make us smarter until we figure out alignment and then they can be smart and solve all our problems. And Most of this AI 2040 thing is devoted to, well, okay, let's say we want to do that. How do you do that? And in particular, how do you do that in both US and China? Right, because you can do something in the U.S. but then if China keeps racing forward, then U.S. won't do that because you don't want to stay behind. And so a lot of it is spent kind of arguing or laying out a story, a narrative of potential cooperation and mechanisms to slow down AI development, which have things like monitoring access to compute, like pretty strict direct government oversight of AI model development. A bunch of responses to this AI 2040 similar to AI 2027 again, could go on for an hour and a half about it. High level. You know, it is interesting to read about, to see one sketch of how things, how people are looking at the future. And if you are a believer in the fast takeoff scenario, then it's probably nice to read about a potential optimistic outcome. Probably an unrealistic way to see things in many ways. But hey, I mean it's good to be optimistic and say maybe AI will not kill us all.
- [79:17]
B
Yeah, absolutely. And so full disclosure here. My co founder and brother Ed and I both had a look at AI 2040 before it came out and gave some feedback on it and stuff. We have our quibbles with it, as everyone does. I think the top lines generally make sense. I think this is a really thoughtful team that's worked on this. I mean Thomas Larson and Daniel Pokotayo in particular, who, like, I've just known them longer than the other folks who co authored it. But like, very, very thoughtful people, they've been right. I mean like if you look at AI 2027 more than anybody else at the level of detail that they've offered, they've knocked it out of the park. Like there's like weird levels of correspondence that we see between what's happening right now and what they predicted.
- [79:55]
A
And 2027 is a while ago. I forget when but it was like early 2025. Ish.
- [80:00]
B
Maybe exactly a while ago. Something. Yeah it was something like that. Where to the point where it's like, you know, people were, were, were laughing at it in ways that now are just like oh I guess the US government is just like doing this. Okay. Yeah, like wake up. That's like wake up and smell the scaling curves. That's where they've been pointing. The funny thing with scaling curves is that something that compounds in like 10xs every year is something that like worry yeah the 10x will actually happen. And like it's, it doesn't do you much good to, to believe it when the numbers are small. It's when the numbers are big that you're actually testing how good is your kind of empirical extrapolation. So anyway, so it's aged very gracefully. What they do is they lay out a couple scenarios. Won't go into detail but like they've got this plan, a scenario that they've presented in a lot of depth here where they're talking about yeah the US and China get together and they're both really scared about loss of control and other things. So they actually set up this like international way of coordinating around this stuff. They've got a plan B that's like China won't go willingly and so we need to make them slow down. Plans A and B together are like exactly what we've been focused on over the last like several months as we've been talking to like literally the diplomats who led us China engagement on WMD risks to see like what is China like at the negotiating table, what's realistic. But also some of the hardware like compute assurance people who, they've obviously spoken to a lot of the same people, they're proposing the same solution of network taps and recomputations. The trouble is when we take that to the intelligence community and ask them about it, it's, it kind of looks like a bit of a non starter. So there, there's a bunch of things here where we're going to be coming out with our report pretty soon. Our differentiator is our ability to like actually talk to the intelligence community, talk to the diplomats, talk to the people who do the builders of the data centers. And to take these kinds of scenarios, just like we did with situational awareness, and be like, okay, but how does this make contact with the actual reality of institutions and sort of special operations and that sort of thing? So we're going to be doing the same thing with this by accident. So I'm not going to kind of like, give my full take on this, but I think it's actually really, really thoughtful. It does an excellent job. And any quibbles that I might have with it are like, within the range of, like, things could go one way or the other, and I don't think that they're meaningful. So they got a bunch of, you know, plan C and plan D. Things get increasingly shitty as, like, different governments don't take it seriously enough, early enough, and then we end up in really bad positions. It's worth at least skimming, especially if you are compelled by what AI 2027 said and how well its predictions have aged. I would expect this to age similarly gracefully. So check it out. I highly recommend it more than anything we've covered in the last sort of like three weeks or so. I would recommend taking a look at this.
- [82:32]
A
Yeah, exactly. I think if you look at the discourse, as with AI 2027, the mainstream, there's a very strong tendency of kind of the mainstream of AI commentators, whatever you want to call them, AI community, roughly, to be at best dismissive of these kinds of discourses around X risk and around catastrophic outcomes, around super intelligence. A lot of skepticism, in some cases, kind of constructive pushback, in some cases just like making fun of it and being like this is silly. And in many ways it's been shown that kind of the dismissal of these kinds of safety concerns is wrong. These are real things to consider. I am someone who is more on the extra skeptic side. I think both AI 2027 and AI 2040 very heavily lean on a takeoff scenario of exponential self improvement. And that's the key question. There's no question we get to AGI, but does AGI get us to sai, and does that get us to models that are impossible to control? I think nothing in our current scaling curves or science really tells us much about fast takeoff is my personal stance. But you can make arguments either way
- [83:56]
B
anyway, and that'd be a good episode for us to do at some point. That's one of the things that I appreciate about the podcast, because it's like there is. You do get one takeoff. I think we're both. I may be more actually outright like X risk pilled, but you Certainly are more kind of a moderating influence in that respect. I think that's important because it's. The reality has been rough around the edges. Like one of the key things that people like me have been wrong about is if I had been correct, you know, five years ago, we'd all be dead by now. I think that's very fair to say. There are other aspects of the story that I was telling myself that sounded like science fiction of the time and like were exactly right. Right. Like, you know, the mythos thing as an example. But it's, it's never as clean as you want it to be in either direction. And, and this is just, we have to proceed accordingly and it's not obvious what that means.
- [84:46]
A
Yeah. The other thing that Silicon Valley is very bad at is converting the technology progress into societal impact. And it's very easy to overestimate the style of the. So you know, if you were, I would have to go back and look at the 2027. But part of these projections is like insane changes in GDP and everyone being automated and every job being automated and often it's a bit slower than you would imagine.
- [85:09]
B
I think they're pretty much on point. Weirdly on the GDP and workforce side, the big numbers at this stage, as I recall from AI 2027 are more like capex spend. That's the stuff that, that sounded crazy. And now it's just like, yeah, you know, of course we're spending like trillions of dollars on capex. Like why wouldn't you.
- [85:28]
A
That's fair, that's fair. But like GB beyond just like building more AI, I guess.
- [85:33]
B
Yeah. I can't remember if they made this specific prediction, but it's in the same ballpark. It's like basically AI is the dominant driver of US GDP right now. In the same way exponentials always look cute when they're, when they're young and then they become monsters very quickly.
- [85:45]
A
GDP growth.
- [85:45]
B
Right, yeah, GDP growth. Yeah, yeah, sorry, sorry. Yes, gdp. Good point. But, but that's, you know, that's, I think in line. I'd have to go back and check. But like Daniel posted recently, I think he said something like by his estimate we're about, you know, 75% running at about 75% of the speed of AI 2027. And that matches my rough hand wavy sense of it. But you know, there are quibbles at the margin everywhere. Yeah.
- [86:10]
A
And we're going to have to close it out there. Real lighting paced episode. So much fun stuff to discuss. Thank you so much for listening to this week's episode as usual. Caveat for me. I'm sorry for real releases being a bit choppy and the timing, it should be getting back on track as my startup gets a bit less crazy. We appreciate you listening, sharing, rating the podcast commenting we'll respond to comments probably next week. And more than anything, please keep tuning in.
- [86:55]
B
Tune in, Tune in When the AI news begins begins it's time to break, Break it down Last weekend AI come
- [87:10]
A
and take a ride get the low
- [87:12]
B
down on tech and let it slide
- [87:18]
A
A ride through the streets AI reaching
- [87:22]
B
high New tech emerging Watching surgeon fly from the labs to the streets AI's reaching high algorithm shaping up the future
- [87:32]
A
sees Tune in, tune and get the
- [87:35]
B
latest with ease Last weekend AI come and take a ride get the low down on tech.
- [88:04]
A
From neural nets to robot the headlines
- [88:07]
B
pop data driven dreams they just don't
- [88:10]
A
stop Every breakthrough, every code unwritten on the edge of change with excitement we're smitten from machine learning marvels to coding kings Futures unfolding, see what it brings.