Loading summary
Transcript85 lines
- [00:00]
Andrei Karlenkov
Foreign.
- [00:11]
Jeremy Harris
Hello and welcome to the Last Week in AI podcast where you can hear us chat about what's going on with AI. As usual in this episode, we will summarize and discuss some of last week's most interesting AI news. You can also check out our Last Week in AI newsletter at lastweekin AI for articles we did not cover in this episode. Lately, I've not been very prompt with that. So as I often do, apologies the startup life is a bit of a grind, but I'll get back to it right away. And I am Andrei Karenkov. I'm one of your regular hosts. I studied AI in grad school and now work at the Startup Astrocade.
- [00:48]
AI Policy and Research Expert
And I'm your other co host, Jeremy Harris at Gladstone. AI do AI national security, AI infrastructure stuff, full stuff, all the usual fun stuff.
- [00:57]
Jeremy Harris
You were safety guy, other guy, you know.
- [01:01]
AI Policy and Research Expert
You know, Andre, since the new political winds have shifted, safety now is a bad word, you see. So even though I was always doing like equal parts safety and security, you know, I mean, more like thing from before, I will refer to myself as a security guy. Yeah, yeah, you got to, you know, when in Rome, when in Rome, you got to do thing.
- [01:23]
Jeremy Harris
But like when you poo. Tweet. Yeah. Safety is not the nice word, but security now that people take seriously.
- [01:30]
AI Policy and Research Expert
Oh yes. Suddenly.
- [01:32]
Andrei Karlenkov
Yeah.
- [01:32]
AI Policy and Research Expert
If I say that the AI kidnapping your wife and kids is a. Is a safety problem, everyone's like, it's not, not an issue. Everybody would just calm down. But the minute we say it's security now, now it's real. So.
- [01:43]
Jeremy Harris
Yeah, well, we were just saying before starting here that it's actually for once a bit of a light Newsweek, in part because we recorded the previous episode on Saturday and had to include like a million stories. We are recording this on Wednesday, July 1st. So we've got like half a week worth of news and not too much has happened. So this will be an episode with just a couple big stories on the business and politics and tools front. We'll probably spend a bit more time on open source and research and advancements. So if you're a big fan of that, expect us to, I guess, take our time possibly, since we just fill up whatever time we have with whatever stories we have. So we'll see if we actually go short this time. We'd like to thank Notion for being a sponsor. With the recent launch of Custom Agents, Notion became the collaborative AI workspace where teams and agents work side by side. And now their new developer platform is turning that workspace into infrastructure. Developers can build on the platform gives developers and coding agents the ability to extend beyond what's possible on Notion by connecting to external systems, bringing context in, taking actions across your tool stack, and exposing custom agents capabilities to any system that needs them. Teams can use it to sync any data source into Notion, build any tool for your Notion agents and orchestrate any agent any in Notion via things like the cli Notion hosted sandboxes to run custom code with workers, an external agent API and an agent SDK. It makes it so the workspace and the platform you build on are the same thing and Notion is already built for teams. It has permissions, context, visibility and Governance from day one. Learn more about Notion's developer platform today at notion.comlwai that's all lowercase notion.comlwai to try Notion's developer platform today and when you use our link you're supporting our show notion.comlwai we'd like to thank Box for sponsoring us. Box is building the Intelligent Content Management platform for the AI era, serving as a secure essential context layer for Box's AI agents to access the unique institutional knowledge that makes a company run run. And that's the key idea. The power of AI doesn't come from the model alone. It comes from giving AI access to the right enterprise content. Box's recent State of AI in the Enterprise report found that 96% of organizations say agents need access to company specific content, but only 36% have connected agents to trusted content across many use cases. And a trust bit is very important. Box is built of security, compliance, governance and threat protection in mind, so employees and agents only access the information they're authorized to use. If you're thinking seriously about adopting AI in your company, think beyond the model. Your business lives in your content, and Box can help you bring that content securely into AI era. Learn more@box.com LWAI and let's kick things off with tools and apps and the first story is Trump drops restrictions on Anthropics, Mythos and Fable models so this just happened? Or just saw this tweet from anthropic Claude Fable 5 will be available again globally tomorrow. Let me just read this verbatim since this covers pretty well. After a series of productive conversations with the US Government, we are redeploying the model with a new set of classifiers to target and block more cybersecurity tasks. In the near term, some routine tasks like coding and debugging will fall back to Opus and 4.8. We'll continue to refine these classifiers over the coming weeks to reduce false positive and better distinguish genuine misuse from legitimate requests. We've also begun drafting a consensus framework with Amazon, Microsoft, Google and other Glasswick partners for assessing the severity of AI jailbreaks and how AI developers should respond to them. Finally, we're scaling up our collaboration with the US government on model testing and safeguards. This will include pre release access to models and safeguards for evaluation, information sharing on jailbreaks and misuse, and dedicated resources for joint research. Yeah, that's the story. It appears to be. They talked it out and they decided to slap on some more safety classifiers that are probably overly safe given the language here of like you'll probably be rolled back to 4, 8 on routine debugging and hopefully this indicates that there is going to be a more productive relationship between the US government and anthropic, certainly, but maybe also just the broader AI ecosystem.
- [06:48]
AI Policy and Research Expert
Yeah, I have no confidence whatsoever in this possibility, but it seems like it's a real possibility that actually the new safeguards and security measures that are being layered on these additional classifiers and so on are a sort of just giving the US government the ability to say they got something, a manufacturing concession. When we talked about it before, nothing has changed whatsoever, at least in my analysis of the situation from two weeks ago, from last week. We remain in a world where no one knows how to stop jailbreaks. It is 100% guaranteed that US adversaries, including specifically the Chinese are will find ways to jailbreak mythos and any relevant OpenAI models of similar capability. 100% guaranteed when they are released.
- [07:40]
Jeremy Harris
We had the discussion, I think, and I'll reiterate that the one thing I'm not certain about is targeted jailbreaks, like one off jailbreaks getting a response out of a model. I wouldn't be surprised if that's true. It's impossible to fully resolve that. But sort of agentic jailbreaks where you have a model do something complex over a long horizon, given that there are these online detection mechanisms. So the model may be jailbroken, but there are detection mechanisms on top of that that would just be like the model. Shut up. You're not supposed to do this. That I think probably Anthropic has put in enough work on to make it viable.
- [08:23]
AI Policy and Research Expert
And this is the thing, this is the core of the thing. Right. So like the framing of like this is all about the jailbreak in general, that whatever the scary jailbreak was that Andy Jassy flagged. But when the mythos thing first blew up. We still don't know what that is, and we still haven't had the principle articulated. So we still don't know what the principle is. All we know is that there's like, a new layer of jailbreak prevention. That's probably stuff Anthropic would have put in place, absent this whole hoopla.
- [08:49]
Jeremy Harris
So.
- [08:49]
AI Policy and Research Expert
So it's, like, really unclear that there was marginal value added here beyond just Andy Jassy flagging this to Anthropic, and then they would have covered down on it. But, like, this whole thing seems like very much to be a. It's crazy to me that Anthropic wouldn't have covered this down pretty immediately after having it flagged if they'd known that the alternative was the USG is going to come in and say, absolutely not. You may not release this whatsoever. So to me, this still smells like were giving the administration the option to declare victory and move on here, which maybe that's what needs to be done, given the personalities involved. But like, to first order, that's what this smells like. It is noteworthy, by the way. So OpenAI we were talking about, well, will the administration also block, you know, GPT 5.6's release? They kind of did, but they did it through a different set of mechanisms, let's say. I won't even say authorities, because everything here is sort of muddied with, like, is it voluntary or isn't it? So the Commerce Department and the export controls Framework was what was used to prevent Anthropic from releasing Mythos. What the administration did with GPT 5.6 is that they went ahead and asked OpenAI to voluntarily just hold it back so the US government could assess it.
- [09:58]
Andrei Karlenkov
Right.
- [09:58]
AI Policy and Research Expert
So this is the first time that there's been a preemptive request by the US Government to a frontier lab to say, hey, hold on now. The interesting thing is the skep, or people who want to sort of criticize the administration on their treatment of anthropic run into two problems. Right, so. So there's two issues here. Number one, the administration did actually tell OpenAI, after all, to hold on releasing GPT5.6, which I think is a positive update for evenhandedness. The other positive update for evenhandedness is the fact that the request for OpenAI to not release GPT 5.6 has not been lifted, even though Mythos, or rather Fable, is now clear for release. So you actually do have Anthropic regaining a little bit of a lead time that they would have had, and I think the argument there is pretty clean. Like, Even if the US government is basically just going to say, hey, OpenAI, you have to hold on your release for long enough that Anthropic can recover the lost advantage that came from those deliberations, in principle, I would actually see that as like, I don't think that's the.
- [10:58]
Jeremy Harris
I think that's one read. I think there's an alternative read. So, first of all, the administration Lutnik had cleared mifos for select White House approved customers, which it sounds like OpenAI had the same. I think there's a different read here, which is this kind of administration, the modus operandi is to just seek power. It's like a wild AI model where they just will kind of want people to bend the knee. So there is an increasing escalation of the sorts of things that this kind of administration will ask of anyone and everyone. And this includes AI. And given the precedence of Anthropic and the kind of them telling Anthropic to do things, anthropic saying no, OpenAI being told to do this is just kind of another pattern of, okay, well, we've seen this thing where we couldn't exercise our power. Now we need to set a precedent that we can do this. So I will say, like, regardless of fairness or not fairness, maybe they didn't single Anthropic. But whether this is actually for safety or this is like trying to set a precedent of we can tell you what to do that you can have
- [12:13]
AI Policy and Research Expert
different reads on, I think there's a very strong argument to be made that this administration temperamentally does the knee jerk sort of control thing. And I think you could, you could argue it the other way, but I think that argument is. Is harder to make, to be honest. So I generally do agree. I will say, though, while it's true that mythos 5 is only being allowed to be used by specific US governments that are involved in defending critical infrastructure, Fable 5, which is the one I mentioned in the key one that is going to be approved for general rollout. Right? So that. That's the key one from an economic standpoint that Anthropic was only ever planning on releasing broadly, at least at the moment. So just as a general principle, zooming out a bit from the mess that we're currently in, because it's a mess and there's so much blame to go around, especially on the U.S. i mean, the administration's been catastrophically incompetent in managing this. Whether you say it's because of malice or Just like ridiculousness. The language they used, the approach they used like it was incoherent and, and just like, I don't think anyone should be proud of themselves for how this was handled on the administration side. However, one interesting principle that may accidentally have come out of this or may have semi intentionally come out of this is the thing I'm trying to gesture at here, which is, okay, so the administration came out and said, hey, Anthropic, you can't launch Fable even though you pre cleared it with us, blah blah, blah, blah, blah, we're just going to do this knee jerk thing and stop you. Well, Anthropic very reasonably might say, look, our advantage comes almost entirely in the time delay between the moment we release a new frontier model and the moment OpenAI catches up. Or at least that's a huge fraction of our ability to get users on the platform. Okay, so if they did get there fair and Square and OpenAI is then presumably still working on their roadmap and gaining time, well then you need some principle that allows you like, if it takes the government 20 days to figure out an entirely new approach to a new level of capability, to regulate a new level of capability effectively, then you kind of have to like apply that same stop time to the big competitors. You have to turn to OpenAI and say, hey guys, like look, if there were no regulation, you would be 40 days behind. So we're going to require you to stay 40 days behind. We'll impose the same gap on you that we imposed on Anthropic. It's janky, it's messy. But like as a principled way of kind of like continuing to reward innovation, while also, I mean you can think of it as like a, it's almost like what patents do. You get to an innovation first, you get 20 years to just like have a monopoly on it. And, and that's a government endorsed monopoly designed specifically to encourage R and D, to encourage innovation. That would be the same thing here. If you do this thing where you say for, you know, Anthropic, guess what, or whoever gets there first to a scary new level of capability, blanket ban, and then like everybody gets to catch up, then there's no, no incentive to keep pushing the envelope. And that is again the most profitable period in a frontier company's life, is that that crucial gap. So I think that's kind of like this important question. How do you deal with it? And like we may have stumbled into, into a pseudo, I won't say resolution, but a mitigation here.
- [15:12]
Jeremy Harris
Yep. And once again, apologies for kicking off of politics, but it is what it is, right? And yeah, I think the final bit I'll say on this front is there's been discussion, I think you may have mentioned it, of another dimension at play here is China. And we've been discussing GLM 5.2 has gotten tons of play, lots of discussion among the AI community as a viable competitor for coding. Like even this hasn't been true before, but you could potentially go to one of these Chinese competitors just to lower prices. So I think to your point of anthropic having this position of we need the lead time, you can't block us or whatever, the same can be said of Chinese models. And the argument there I think is even stronger to the US government that if you, you know, limit us while the Chinese will just keep keeping up and you don't want that, right? U.S. government, so please let us do what you want to do.
- [16:17]
AI Policy and Research Expert
Starts to, starts to feel an awful lot like a treaty with China is going to have to be on the table. Like if the US government wants to be able to keep doing this, there is literally there's two options. You either like have a treaty with China that's enforceable and verifiable, very hard or you degrade, deter, disrupt China's ability to, to build AI stuff domestically. I will conjecture that there is no third option, so place your bets accordingly.
- [16:45]
Sponsor/Advertisement Voice
People.
- [16:45]
AI Policy and Research Expert
This is not investment advice, but if I'm predicting the future, same as I like you know, going back to the scaling days when it was like obvious we were going to do scaling when like it was obvious the nation state security was going to be important for this stuff. Now we're in a world where it is obvious that if we keep building Mytho style WMB cyber weapons in people's basements and sooner or later you're either going to need an international treaty or sub threshold warfare to like literally hamper each other's capabilities to do this. Again if you see a third way, I'm super into it.
- [17:14]
Jeremy Harris
But yeah, I mean also the second way of I won't not warfare. But for years and years these export controls that have been, well with Trump it's been back and forth but already this has been kind of a norm for a while. Which actually going back, we might round back to this later but I wonder. Yeah, just like the actual export controllers have been around for so long. I don't know if the original argument was about AI or just generally high tech.
- [17:46]
AI Policy and Research Expert
Yeah, it was actually about AI back in the like the amount of prescience that like the U.S. government, I think I may be crazy, but I feel like 2017, 2018 you started to see this and certainly like in the GPT3 era, it was like dead serious and huge credit to the Department of Commerce. I mean, wow. You know, they were scale pilled basically. And I think initially it was, you know, through the lens of more narrow military applications, computer vision systems for, you know, weapons platforms, stuff like that.
- [18:17]
Jeremy Harris
There was deep learning pre GPT. People forget now but the 2010s had a lot of AI hype. Deep learning.
- [18:25]
AI Policy and Research Expert
GPT3 was the third one actually.
- [18:28]
Jeremy Harris
Yeah, there was a big Alexnet DeepMind got bought by Google in 2013.
- [18:35]
AI Policy and Research Expert
Yeah.
- [18:37]
Jeremy Harris
Well, next up, another strong anthropic. They are launching Claude Sonnet 5 Cloud. Sonnet 5 is their first Sonnet since 4.6. They've been focusing on their opuses and their mythoses. Some of the big news is the price. It's priced at $2 per million input tokens and $10 per million output tokens through August 31, after which prices rise to $3 and $15, which is a bit of a new thing. This makes it cheaper than Opus 4.8 GPT5.5 Gemini 3.1 Pro. It as you might imagine, has pretty significant improvements over sonnet 4.6, especially on agent decoding terminal bench 2.1, which seems to be the norm there. Of course anthropic highlights improvements in generally across the board I guess is where the factors so misaligned behavior is a bit down relative to Sonnet 4.6. They have actually I haven't seen this before. A cool graph of pass rate as opposed to cost per task on a log scale. So you can see them plotting the effort levels and then instead of output tokens, it's actual cost and they show that, you know, per effort level 4 pass rate. Sonnet 5 is less expensive by a decent amount than 4.8 while having a higher pass rate than Sonnet 4.6. And the last bit is they do highlight the ability of it for cybersecurity. They show that it's not that good. Slightly better than Sonic 4.6, but much, much, much below Opus 4.8 and Mythos 5. They say that they are still going to be launching with cyber safeguards enabled by default, even though it's, you know, not that much better than opus, than Sonnet 4.6. So I guess this might be de facto a safeguarder that'll be in place from now on.
- [20:36]
AI Policy and Research Expert
Yeah, it's. It's funny you look at some of the plots that they put out, it's rare to see this sort of thing, but they have the performance of Sonnet 5 versus versus GPT 5.6 on like a cyber eval. And like very deliberately you can see like Opus 4. 8 is like below and then you have Sonnet 5 and then. Or sorry, Sonnet 4.6 is below. I'm sorry. And then you have Sonnet 5 and Then above that, very important, above that level of capability is GPT5.6. Okay, guys, don't worry. GPT 5.6 is more dangerous. It's, it's important to know that we are better than the previous, but we're worse. We're worse than our competitor here. We're, we're bad enough that, that we can legally be used in the United States. That's kind of the message at this point, which is sort of like this funny reverse marketing thing that's starting to happen. And this is also another artifact of this funny sort of stutter step with the Mythos 5 or sorry, the Fable 5 release is Sonnet 5 is now starting to Dog Food Opus 4.8 a bit. You look at some of these benchmarks and like, you know, granted on agentic tasks, Opus 4.8 is still better. But like on knowledge work related stuff, you're seeing Sonnet 5 actually exceed Opus 4.8. So they really are dogfooding and starting to kind of compete with themselves there. So it's sort of interesting and again, a function of like this wouldn't happen if we had a Fable 5 like a larger scale model that, you know, knocked everything out of the park. So kind of interesting and a weird sign of the times, let's say.
- [22:02]
Jeremy Harris
Yeah, not much more to say on it. I will say there's been some discussion on the cost per intelligent index from I think it's artificial analysis where according to this other benchmark, Sonnet 5 is actually more expensive than 4.8 max, in the sense that the cost isn't just about per million tokens. It's also about how many tokens your model uses. And Opus, I think it was Sable 5. One of the improvements of it was it used fewer tokens to make more progress. And so it was overall cheaper than Opus 4.8, I think. And Sonnet 5 seems to be the opposite of it uses more tokens for a given task and therefore it is, at least on this index, more expensive. Although on the benchmarks that Tropic highlights with os, World Verified and some other stuff, it is not more expensive. So worth noting that little bit. The vibe check is mixed so far so. And to wrap things out we have some actual fun app kind of stories, no US politics, no nuance or you know, anything scary to discuss. First we have Google's Notebook LM can sum up your research in a TikTok style clip. So remember Notebook LM, it was like all the rage a year ago. It came out initially as a way to generate podcast style discussions of anything. You give it as a sort of learning tool as a way to catch up with some topic you want to learn about and it kind of went mini viral. It was like this labs thing from Google. It was kind of thrown out there without much fanfare and it got big and it's been big. If you're a, you know, coding kind of guy like me, you mainly use the serious models. You don't do that much of a consumer AI side despite where I work. But Notebook seems to be quite popular to me and it's interesting to see this sort of non chatbot tool, this tool that you give it some document and it creates these podcasts and now these TikTok style clips. 60 second vertical AI videos summarizing uploaded research, has custom AI art paired with narration and it is now rolling out. So I think it's one of the rare examples of something that's not a chatbot becoming popular and sort of showcasing a different way to use an interactive AI. And one more story from Google, they have also released a faster, cheaper image generator with nano banana two lite. So this one is going to be costing 0.$34 per 1000 images. It is going to be replacing the original Notobanana. It's available for API, et cetera, et cetera, et cetera. And it's been quite a while since the original Nado Banana. It was like November, late 2025 is my memory. So it's been over half a year. Kind of surprising that they haven't doubled down. Initially it was a big deal, but regardless, makes sense that we have a new generation here and out to applications in business. First up we've got etched pools of 400 plus engineers from Nvidia, TSMC and more to build a new frontier inference cluster for AI which is already worth $1 billion in demand. A very long and detailed headline. So etched is a new AI infrastructure startup that again has pulled all of these engineers. They've raised 800 million across four financings and they are saying they have over 1 billion in customer contracts and that will be Shipping their first racks this summer, they have completed a successful A0 tape out on TSMC's N4P process. I'm sure, Jeremy, you can explain what that means and I guess, yeah, maybe you can talk through what makes this interesting.
- [26:15]
AI Policy and Research Expert
Yeah, so Etched is one of those companies that they are new in terms of like seemingly coming out of nowhere. And they have been working at this for about three years, but that's a really short period of time for being as far along as they. As they are or appear to be. So currently they are basically switching from the narrative they'd had previously, which was they're building a transformer only asic, basically this transformer only chip. The whole architecture is baked into the silicon like it's, it's. It wasn't meant to run anything else. And that was the whole pitch. And so people looked at this and were like, well, you know, if transformers end up being the thing, I can see why this would work. But like, what about moes? What about state space models like Mamba, can you do this? And they're coming out here and saying, well, guess what? Actually we absolutely can run, you know, Deep Sea, Quin, Mamba, Llama, like all these MOE models, we can do basically all this stuff. Right? And so it's a bit of a pivot announcement at the same time as they're coming out and saying, we just pulled off a really remarkable feat, this tape out. So a tape out is this moment where you kind of finalize the layout of your circuit. So your integrated circuit. This is like, you know, all the, all the little wires that describe how data has to flow through your chip and you then hand it off to the foundry, in this case TSMC for manufacturing. And so the A0 part refers to what's called silicon revision or stepping. So chip revisions, they're named with like a letter number. So you'll have like a zero, A one, B zero, B one, and so on. And so usually the letter gives you like, is it like a major revision that you're flagging like a new version number? And then the number is more of a minor thing. So A zero is the very first kind of design that they have and they're announcing an A0 tape out. So this is a really big deal because usually these sorts of things will take like many, many years to pull off. I think right now I had in my notes somewhere, did they do this in like, I think three years end to end? So basically they're using the, yeah, the M4P process nodes to be like, 4 nanometers. They got there under three years from seed funding, which is pretty wild and it's really rare. Like, like most chips also need multiple kind of reactions, Re spins they're called. Basically they have to go back and forth with a foundry multiple times before they get a functioning output. And so this they did in the first pass. So the first time they shipped it, they got it all to work, which is a huge, huge positive signal. It's falsifiable, like it's an engineering result. It's not just like some qualitative statement. This is a big, big deal, assuming it's true, which it'd be hard to fake. So this is a really big achievement for a very young company that is trying to compete in a way that's just shifted. Right? So they're no longer just the transformer only company. There's a lot of ambiguity about what they're actually doing under the hood here. But they're rotating into more of hey, we do the whole stack. They're a systems and manufacturing company now instead of just a chips company, which is a pivot because hey, the transformer only chip thing clearly was not the way. You know, we've seen how we're seeing these layered mamba. Attention transformer architectures where seeing MOE take over, like they were going to have to change that. But now what they're doing is number one, look, we have the fastest chip design or one of the fastest design processes, which is amazing, but also full stack co design. So they're doing racks, right, like you mentioned, all the way up to the rack level. They're doing their own circuit boards, their own cooling plates, networking like all this stuff. And this essentially means that they're now, you know, going to be shipping stuff to data centers that's like, you know, a lot more packaged and complete. Well, they kind of need that sort of play because right now they're arguing for a $5 billion valuation. They haven't shipped a single unit. There's got to be a really, really big tam, like a really big market story that they're, they're going after here. So this is a very like, it'll be very interesting if it works out. They've got investors on board that are kind of tilted towards quant trading firms and chip trading, kind of the semiconductor industry. So you know, Jane Street, HRT, Jump, Two Sigma, all these like quant trading firms. The reason that that's significant by the way, is a lot of what they're after is low latency and quant firms really care about that, right? They want very quick outputs because you know, that's, that's what matters when you're making trades super fast. And the other thing is they have a strategic investment from Venture Tech alliance which has these deep partnerships with foundries, including tsmc. So when it comes to this industry, your ability to compete is in large part a function of how much allocation you can get from tsmc. If TSMC says, hey, we'll reserve a chunk of compute, like basically if you make compute, people will buy it. Like this is not an issue. Like everybody is compute hungry right now. You'll be able to sell your chips for good money. So convincing TSMC to leave capacity on the table for you is the key thing. That's where having that Venture Tech alliance investor partnership is a big deal. Last thing I'll mention is related to that. They currently are saying like, hey, we have a billion dollars of signed customer contracts which like that's cool, but you have to fulfill them. And this is a space where if you tell people you have compute, they will, they will buy the compute. That's not an issue. Like the hard thing when you're kind of supply constrained isn't getting customers to say they'll pay you money for a hypothetical chip, it's actually producing the chip. And so we now have to see how it works end to end. It's going to be a question delivery, delivery quantities, delivery timelines and yields. That story is yet untold. If it works out, these guys could suddenly become a really interesting force in the ecosystem.
- [31:51]
Jeremy Harris
And I think it's interesting that they have this many orders given they're pretty new and they're competing with Nvidia and so on. $1 billion in customer orders is nothing to scoff at. Are demand constricted on chips from the big data center builders? So maybe this is helping meet that demand.
- [32:13]
AI Policy and Research Expert
Yeah. And I think from a practical standpoint, they are doing things that are supposedly performance breakthroughs, right? So they're advertising like low voltage inference. So basically just using way, way lower potentials to power their chips. And then also what they call cluster scale memory. And they hand wave. I mean they say stuff like we get multiple times the flops. Density 80, like 80% peak flop, sorry, an 80% improvement in peak flops. So, so like this is good. But they're deferring, sharing any of the hard data for later. They're saying their performance roadmap or, sorry, their performance numbers and their roadmap is going to come up this summer. We don't know exactly when but yeah, I mean like strategically too. If you're tsmc, one of the things you're trying to do is make sure that the people who build your build stuff with your leading nodes are building the most useful stuff possible. Because that shows the world that you can do a lot more with TSMC fabrication processes, which in turn drives up the price and demand for TSMC chips. So even when they're supply constrained, there's an incentive for them to like allow new players to run little tests using their allocation. Like just set aside allocation for competitors to Nvidia for example. Like these guys to come in and just see what they can do with it in the hope that there's a more competitive ecosystem that drives up demand in the long run for TSMC's nodes. So anyway, it's a really interesting story. I don't know where this all shakes out because we have to see delivery, we have to see performance numbers and all that jazz, but yeah, it's a pretty big deal.
- [33:47]
Jeremy Harris
And related story about chips DO rallies on AI chip IPO report Their shares rose 7% in Hong Kong after reports that its AI chip unit, Kun Long Sheen is targeting that Hong Kong IPO at a $50 billion valuation. It looks like they filed back in January. They already have customers including Tencent, a competitor of Baidu. And they've been around for a while. They started as Baidu's internal AI chip unit and 2011 and have since expanded to external customers Tencent and Binadance apparently. And for context, I think I mentioned this on enough. Baidu sort of like China's Google so tech giant. And this is, you know, you could make the analogy to TPUs within Google, not exactly the same but you know, this is clearly something that can be deployed at scale. They have said that they have multiple 10,000 card clusters that were already used to train Baidu's earning models. So kind of makes sense that this would be a big deal.
- [34:59]
AI Policy and Research Expert
Yeah, and they are in some ways old. They go back to 2011 when they were being used internally at Baidu, but they do now already service external customers. And the interesting weird thing about this IPO is they're asking the investors that are coming in to buy their chips at like alongside their investment at rates that are around three to seven times their intended IPO share subscriptions. So you know, let's say you want to buy, you know, I don't know, half a, half a billion dollars worth of IPO share subscriptions and your call it like I don't know, 10 cent, then they're going to come in and say, hey look, you need to buy say three to seven times you needed to buy $1.5 billion at least of our chips alongside your purchase of shares. And like, any time you see weird stuff like that, the appropriate response is to be like, how clean is this IPO? Like, you know, can you really defend that 50 billion? Like, is this a way of trying to manufacture demand to drive up to justify that, that $50 billion price point? And I mean, it's a weird arrangement. We'll see if people actually bite. There's an almost infinite demand for chips, especially in China where obviously they're starved for chips and not energy. So chips are really the bottleneck. Anyway, it's interesting and weird. We'll see what the stock actually does once it gets listed.
- [36:24]
Jeremy Harris
Next up, going to robotics. Agility Robotics plans to go public via SPAC in a $2.5 billion deal.
- [36:32]
Sponsor/Advertisement Voice
Deal.
- [36:32]
Jeremy Harris
Agility Robotics is one of the competitors in the humanoid AI space with plans to deploy robots in warehouses. They have already secured over 300 million in multi year orders for its next gen Digit V5 and 30 plus potential customers. Evaluating large scale deployments and in going public with spac, SPAC is like a way to basically merge with an existing public company and become a public company and avoid all the IPO business and make it faster. So they'll be trading under ticker and they'll be able to generate lots and lots of money, presumably to then be able to expand digit V5 production capacity. So I'm not sure how to read this in terms of if it's a good sign or a bad sign for Julia Robotics, but they are one of the major players in the space and have been targeting creating these robots for manufacturing purposes for a while.
- [37:33]
AI Policy and Research Expert
The SPACs are things that you used to hear about during the frothy 2020 and 2021, like the COVID era market insanity. And they involve basically it's like a merger between a private company and one that's already public and then they form this blob and it leads to less regulatory scrutiny than the usual IPO route, which can well have pluses and minuses. It's a reason to be more skeptical about the, the listing just because, well, you know, there's a reason you would go with a SPAC over a traditional IPO route. It's interesting. There's, there's a lot of hype around humanoid robotics and this is definitely like you could view it as a way to capitalize on that hype. They claim to have good customer traction. So, you know, there's a lot of obviously humanoid robot demos we've covered a million times where you see a bunch of promo videos, you don't see actual commercial deployments. That seems to be the case. They're claiming $300 million in orders. They're claiming a 30 customer pipeline, but, you know, $300 million versus a $2.5 billion valuation. Keep in mind the margins are a question mark here. Like when you're doing hardware, hardware is hard. Margins tend to be lower unless you're Nvidia. So usually if this were a software business, traditional software business in the SaaS era that we are just leaving now, you know, $300 million in revenue would mean something like, you know, 200 and some million dollars in actual profit, but here it's going to be much less. And so, so using that to back a $2.5 billion valuation, that's almost like software multiples. So, you know, maybe it holds, maybe it doesn't, but we'll, we'll have to see and the proof will be in the pudding. I mean, I feel like this year is going to be a big year for clarifying separating the boys from the men a little bit on the robotic side where we start to see, okay, you know, who's actually with, with these customers. You know, they say they've got, you know, Toyota, a Canadian manufacturing plant for Toyota, Mercado Libre, a whole bunch of really big companies. So we'll see if that translates into enduring demand.
- [39:35]
Jeremy Harris
And last story for the business front, China's Deep Seek plans to at least double staff in all departments. So this was an announcement that they are planning to do this. They posted on WeChat targeting technical engineering roles. So I guess it's like we are hiring. Please apply. And this is of course, after They've raised approximately 50 billion yuan, US$7.4 billion, one of China's largest ever startup funding rounds. So, you know, maybe that's surprising. Deep Sea, kind of a big deal.
- [40:10]
AI Policy and Research Expert
Yes, that's it. You heard it here first, guys. Deepseek, kind of a big deal. It's actually kind of interesting that, you know, so Deep Seek is famously very, very, very lean, right? They obviously used to be high flyers. So Deep Seek initially was just a, like a hedge fund. And then now they're sort of discovering, wait, we can't keep ignoring the fact that like the most profitable part of our business is actually Deep Seek. So let's just like make that a thing. And this is them really building an institution. Around, around Deepseek in a more enduring way. They used to have 150 to 170 people. So when they're saying they're doubling headcount, they mean they're going up to 300. So compare that to OpenAI or Anthropic and it's like peanuts. But doubling headcount is a thing. One historic challenge that Deepseek has had that's kind of amusing is poaching risk from other labs. Like you can imagine if you have literally like just 150 people that are responsible for backstopping China's leading AI lab and like competing semi directly with Western Frontier Labs. Like, yeah, every individual on that team is a load bearing contributor to that company. Like there's no two ways about it. And so you're gonna have massive poaching attempts. So a lot of what their fundraise was about was literally just defending their talent and offering compensation packages that mean that, you know, people aren't gonna, aren't gonna poach as much. I don't think I'm hallucinating this. I think I remember us talking about the Chinese Communist Party, basically kind of telling the AI industry in China, hey guys, cool it with the poaching on the deep seat, okay everybody like fucking cool it. Like, we need this national champion to persist. Don't rip them apart and sell them for parts and stuff. So if that's the case, it's kind of interesting. I don't know if it's the right long term play for China, by the way, because one of the things we've seen in the west is when you do break a frontier lab apart, OpenAI, you end up with a lot of other competitors, some of which turn out to be even better than the original. Like Anthropic seems to be relative to OpenAI now potentially. So long term, a lot of the heavy handed regulation on the Chinese side may backfire, who knows, but, but also it may not. And, and so, and in fairness, I don't know if that is an old story now. And, and you know, maybe they've changed their tune, but yeah, kind of interesting. There's a, a whole big pivot. I mean, you talked about the roles they're emphasizing, right? Data engineers, development engineers. This is not a pure research shop in terms of the hiring spree, right. They're looking for people that are going to turn this into a product. And well, it's an analogous to what we saw with OpenAI actually back in the day. Anthropic, right. They used to be A research lab and then it turns into a product shop. So anyway, something to keep an eye on is that direction and we'll see if the valuation rises from 50 billion onward. It's surprisingly low in a way, but also chips are constraints, so there you go.
- [42:52]
Jeremy Harris
Yeah. I have to wonder if this will be an accelerant or a way, something that slows them down in terms of product development. If they go agentic AI and start having a popular coding model, well, that's a lot of data. So redeems may keep getting better. We'll see. And speaking of models out of China and projects and open source, we've got Longcat 2.0. Oh, this is from Meituan, a Chinese technology company that does things like on demand food delivery. Presumably a big one given that they've been working on some advanced AI. This is a large scale mixture of Experts language model, 1.6 trillion total parameters and 4.8 billion activated parameters per model. Apparently both the full training run and the large scale deployment are built entirely on AI ASIC superpods, ASIC being kind of a very custom chip for a given task, not general purpose like GPUs are. So it's the first time I've heard of this. And it, as we've seen with recent Chinese models, is, you know, starting to be competitive with Western models on a bunch of benchmarks including Terminal Bench 2.1. Their benchmarks are saying that it's comparable to Gemini 3.1 Pro, close to Opus 4.7, but behind 4.8 and on many other benchmarks like SB Bench Pro, Browser comp writing, et cetera. It's pretty high up there. I mean it's pretty impressive that they are comparing their model to Western models and not other open source models. And this is open source. So somehow I forgot about longcat and Matewan as a factor. But we've got so many of these. We've got Kimi, we've got deepseek, we've got glm, now we've got longcat. The models just keep coming out.
- [44:58]
AI Policy and Research Expert
Yeah, and then this is another one of those. You might put it up with the deepseek R1 almost in terms of the depth of like infrastructure advancement that's involved here. And sort of setting that aside from the, the conversation around the actual capability, like the raw benchmark scores and all that stuff, just like what they are demonstrating they can do here. So there's a lot that we're learning from this paper in terms of the algorithmic improvements that make this thing tick. You mentioned you know, massive model, 1.6 trillion parameters, 48 billion activated per token. I mean 48 billion. I remember when 48 billion parameters for just a dense model was considered a lot. Here we are, you know, at scale and again all on Chinese hardware nominally probably Cloud Matrix 384 superpods with Ascend 910Cs. That's, that's the most likely. And this also Leonard Heim has, has. I think it was Leonard who came out with some, some top line analysis on the numbers here. His estimate I think was around 40 to 80 megawatts was the scale of this thing. So 48 megawatts, just to put that in context like you're having, that's basically maybe a year behind where we're at right now in terms of scale, maybe a hair more in the west. And the fact that they're able to fill that out now, the megawatts don't turn into compute as efficiently in China because their chips are worse. So that's another thing you got to factor in. But this is a really like big honking training run anyway. So, so this is like approximately the scale of like Deep Seq V4 Pro. So it's been done before in China. But it means like you're dealing with you know, 50,000 or so chips. You're doing a really, really big, big thing here. They brag about the training stability. So they talk about for example this was trained on 35 trillion tokens and stable throughout. Now that's not a, like they're telling you that for a reason. When you do training runs that long, if you're able to have like stability throughout, it means this was not just some flexibility flash in the pan, one off win. This is actually an enduring sort of technical breakthrough that is robust. It's a big engineering success that's repeatable and you see that. I mean there's so much in here about what they're doing for like you know, fault tolerance and basically just the robustness of the infrastructure. There's also stuff that they do just for efficiency. So key thing is because you're dealing with a 1 million token context window, you run into all kinds of attention related issues like deep seek sparse attention which has been just the thing that people use to help with attention scaling in China especially lately is you take. So for a given token that you've got that you're, you're trying to let's say process in the, in the residual stream, that token needs to attend to all the other tokens in the context. Right. So if you've got a million token context window. You've got to do well for, for each token you've got to attend to a million different tokens. But you have to do that for every token. And that's how you get to, well, N squared or a million squared, which is just explosively huge. Now the thing that deepseek sparse attention does, it says, okay, instead of applying the full attention mechanism to do that attention calculation, what we'll do is use a really, really, really, really light model called the lightning indexer. This is a tiny, very efficient model and it's just going to do a rough chop. It's going to roughly try to estimate what the attention scores would be for all those inter token attention values. And it's only going to keep like the top K of those values and then you can do the downstream analysis on just the top K. And that's usually a huge efficiency boost. The problem is that when you do that, you're still having to do a million times a million, like N squared basically for the lightning indexer, even though it's really lightweight, it's still having to attend every token to every other token. And so that's now becoming a bottleneck. Once you get to a million token context windows, even the lightweight lightning indexer from Deep Seq, sparse attention no longer, no longer cuts the mustard. And so what you have to do is find a way to make it even lighter weight. You, you do a couple different things. So they use what's called cross layer indexing. So when the indexer kind of decides, okay, these are the top K tokens at one layer, it's going to reuse those same identified tokens and say, okay, well presumably for the next couple of layers, those same tokens will continue to be the ones with the highest attention scores. We actually covered a similar strategy to that, I think last week or the week before. It's becoming a thing where people are reusing those attention approximate attention score calculations across many layers. Even if it's like technically going to be a little off, it's more or less right. And that's good enough for deep learning. As we keep seeing. The other strategy is they'll use this thing called hierarchical indexing. And what this does, it tackles the N squared problem directly. So what they're going to do is instead of looking at like, okay, we have a million tokens, they actually start to create these kind of bulky kind of blocks, if you will, of, of tokens. You, you group a bunch of tokens together and then you do attention between Blocks of tokens as a first pass. So you're, you're actually reducing the N instead of doing, you know, just a lightweight pass on the full N squared. So there, there's a bunch of other stuff too. Like, you know, when you do this top K token selection, you're like, I'm only going to pay attention to the tokens that are. You have the K tokens that are expected to have the highest attention values. This creates a really big problem computationally because you end up having like this very scattered set of token positions. Like, if you have a given prompt, you might want to attend to tokens like, you know, 17 and like 12,000 or something. And then like it's zeros everywhere else, right? And then it's completely different for the next prompt. And this discontinuity between one query and the next just means that there's no regular pattern here. And it's just really tough on AI accelerators, GPUs and ASICs and stuff because they're better at reading contiguous blocks of memory than pulling together a bunch of scattered memory addresses. And so these contiguous reads you can pull down with one big high bandwidth pull, right? And the scattered ones are just like this tiny scattered reads that waste most of your bandwidth. And so they come up with this thing called streaming aware indexing, which doesn't matter, but basically it addresses this problem, kind of reshapes the budget for token selection, for memory, address selection. In a sense, all of these steps are independent, they're orthogonal. You can choose to apply one or the other, and they're part of the core breakthrough here. There's also this other thing we've seen a lot called ngram embedding. So typically a single token is like a syllable or, you know, 2/3 of a word or something. But often phrases themselves carry intrinsic meaning that's not captured by any given word. It can be helpful to have an embedding at the phrase level. For some chunks of engrams, an N gram is like, say, a number of tokens. So a 2 gram would be 2 token word. A 5 gram is a 5 token word. They go all the way up to 5 grams. So an engram of size 5 and, and they have like dedicated embeddings for some, obviously not all, because that'd be combinatorically insane. But for some 5 grams and, and 4 grams and 3 and 2 grams, they actually have dedicated embeddings. And they show that this creates a whole bunch of beneficial effects across the. Across the thing. Last comment. Sorry. This is like because there's so much here and it's, it's so, so much of it is strategic. Parallelism, like the ungodly scale of parallelism, just gets even ungodlier again. So when we talk about parallelism, we're talking about what are the ways that you can take this problem of training an AI model or doing inference on an AI model and break it up into different sub problems that you can farm out to different GPUs. And traditional version of this is like, you know, data parallelism, for example. You take all the prompts that are coming in, you give some prompts to some machines and other prompts to other machines. That's pretty clean. You can get them working in parallel, but you can also chop up your model. You can chop it up layer wise. So literally just hold only some layers of some Experts on some GPUs and not others. And then you can even carve them up within layers. So you can take one layer from one expert, from one model and put it onto a GPU and, or a chunk of those, those layers and just basically break up the model piecewise like that. They're also using context parallelism because when you get to a million tokens of context window, your kvcache explodes and you need to actually spread your context across machines. So now one GPU might see a fraction of a layer of an expert of a model for a fraction of a context window on top of all that, and on top of that they're adding this parallelization for these engram embeddings. Because the engrams basically just like, we don't have to go into the details, but the bottom line is they just get really big. And so you, you, you can't hold them all in memory on one machine. And, and so now it's like just every end of incremental GPU is holding such a small fraction of the problem. This thing really is starting to look like it's a hive. It only makes sense when you zoom out and kind of look at it at massive scale and something like a 512 way parallelism. So across 500, basically you need 512 machines to keep straight in your head in order to do a coherent piece of work. That's the scale they're talking about here. And there's tons of other stuff too that we're skipping. But this is a remarkable feat of engineering. I'm sure similar things are being done, by the way, in Western labs. We're just not seeing it published. And so that's a Big play here. But man, I mean this is a really serious, for numerical stability, for you know, bit level determinism so they can keep rerunning their experiments and get the same outputs. That's surprisingly hard to do on this kind of hardware. So anyway, I'll stop talking, but man, what a behemoth.
- [54:40]
Jeremy Harris
Yeah, I'll say. For those in the audience who are not engineers, kind of zooming out these kinds of skill sets of going super deep and like optimizing inference for specific kind of compute tasks, this is like deep magic. You're not going to learn this in school. This is not like standard engineering. This is the reason why deepseek people are so valuable. This is advanced, advanced and a niche stuff that most engineers are not capable of. I'm sure there are computer engineers and there's been many, much work on chips in general, but this is pretty specific to AI. And all these like index and attention optimizations and all these things is like cutting, cutting edge. Right? This is not something that's standard or any, any textbook. Eventually it will be, but for now this is all completely new. Like and presumably like very, very impactful. So worth kind of keeping that bigger picture in mind.
- [55:45]
AI Policy and Research Expert
One qualitative thing about it too is like each individual thing you look at and like if you, if you go into dd, if we had a five hour podcast and we cover this in detail, I guarantee you each individual thing you'd be like, oh, okay. Like that, yeah, that's kind of obvious though. And like, yeah it is. You know, lightning indexer. Instead of looking at all the tokens, just like pick the top ones. Like what's, what's so magical about that? It's just the taste of guessing which ones will matter and quickly iterating and experimenting on these things to find the right ones and then how to link them together without having them fight with each other. It just shows you how much low hanging fruit there is in the space that ideas this simple are what the field is made of. Like it's not. There's, I don't want to say there's no advanced calculus, there's tons of advanced calculus. But like a lot of these ideas are. It's like when you see a magic trick done under the hood and you're kind of disappointed and you're like that's all it was. That keeps happening. That's what the space is made of. And anyway, sorry, I just like an important qualitative lacquer there now I gotta
- [56:47]
Jeremy Harris
say is on the pricing front it is fairly cheap. So it standard price is $0.75 per million input, 2.95 per million output. So that's almost I think a fifth of standard anthropic pricing for Opus 4.6 and so on, which is comparable quality. And it is more than 50% off for a limited time discounted price. And this is something we haven't discussed, I don't think explicitly, but there's been this ongoing sort of price war going on in China of like slashing of prices by 50%, by more than 50% for some models where they are now like 30x cheaper than Western models or something like that in some cases. And there is a real kind of concern I think now for Anthropic, given this price discrepancy and the models catching up, like we just said that Sonnet 5 is launching at a discounted rate, which is not something Anthropic had done previously. I think we may be at a point where there is going to be a price war for token prices, at least not at the very, very frontier. So Fable 5 can afford to be expensive. Sonnet 5 Opus 48 Opus 46 GPT 5.5 I don't know if Western, if OpenAI and Fabic will have to start actually competing with Chinese labs and companies.
- [58:22]
AI Policy and Research Expert
Yeah, at the enterprise tier where you know, things are stickier and there, there's like less risk appetite. So am I going to send my data? One would be, you know, am I going to send my data to China? Like okay, well I mean oftentimes it's not that you take open source models, you deploy them, you know, through someone or on your infrastructure. But even just like the use of Chinese models doesn't come with zero regulatory risk, it wouldn't be insane, especially given this administration's behavior so far, to see some kind of weird ban on Chinese open source models being used by, you know, by companies or you know, for certain things. And so like yeah, I think there's like they enjoy that advantage, so they're fortunate in that sense. But like what you're pointing to is absolutely correct. Like there is price pricing pressure and there's a question of if and not when at a certain point the pricing pressure wins that debate. Right.
- [59:11]
Jeremy Harris
That's just like the pricing pressure also by the way, not just from China, like there are Western companies like Geber AI that are serving these open source models on their infra. And you can also just pay for this API. And you know, even at our company we've started looking at GLM 5.2. And because the open source is cheaper and is good for a lot of stuff. And even beyond the coding agents, like if you want to build a product that uses millions of dollars in tokens, you got to start thinking we might want to move.
- [59:47]
AI Policy and Research Expert
Hey, if you're looking to run a Chinese influence operation having literal AI agents that are running with pretty wide latitude on Western servers doing pretty critical things, this is what you give up when you don't do open source stuff. And it's part of the reason why as a second place player, nominally China benefits from the open source strategy, right? I mean it's like it's a huge advantage.
- [60:10]
Jeremy Harris
Summer routines live or die by how easy they are. And honestly, if something takes too much effort, I'm out. That's why Groons is my go to. It's my one daily pack of gummies covering my greens, vitamins and minerals. Plus it has 6 grams of prebiotic fiber, which is more than 2 cups of broccoli. No mixing powders, no giant pills, no hassle. I just rip open the pack and I'm done. They taste good and they make it easy to stay on top of my health even when life gets busy. Save up to 52% with the code podcast at Gruns co. That's code podcast R U N S co. You know,
- [60:41]
Sponsor/Advertisement Voice
this is one of those topics that no one really wants to think about. But if you're a parent, a homeowner, married, or helping take care of aging parents, it's something that honestly deserves a little attention. I'm talking about estate planning. And before you tune out thinking it's complicated or expensive, that's actually why I want to mention trust and will. Trust and will makes estate planning feel way less intimidating than it sounds. They offer modern attorney designed wills and trusts that you can create online. And the process is surprisingly straightforward. In fact, you can create a will in as little as 30 minutes. That includes things like guardianship for your kids or pets, how your assets are distributed, and even healthcare directives. Don't wait until it's too late. Protect your loved ones with trust and will, the most trusted name in online estate planning. Go to trustandwill.com future to get 20% off. That's trustandwill.com future for 20% off. Trustandwill.com F U T U R E
- [61:41]
Jeremy Harris
and next up, we've got a couple of benchmarks to cover. So first up we've got OS World 2.0 benchmarking computer use agents on long horizon real world tasks. So this has 108 long horizon real world computer use tasks that are addressing limitations of previous benchmarks like OS World 1. Leading agents now score over 83% on those previous benchmarks despite the task being short and narrow. So on this one, the median tasks takes a skilled human about 1.6 hours of active operation. That's like 50 times longer than Oswald one. And the leading agents average more than 300 steps per task. That's 10x a previous benchmark. So more of a realistic evaluation. And this has been a trend we've been covering of like there's been benchmark after benchmark after benchmark. That's like this is a real world long horizon task. And I think there's been a rush to release these kinds of benchmarks because like this is what people are working on. They're working on coding agents, they're working on like now with Cowork also computer use agents, browser use agents, and the benchmarks just don't exist for that to be evaluated realistically. And these kinds of things make that possible. Where in this context computer use will be like actually going to a gui, clicking around, writing emails, you know, you can go to a paper and see screenshots of actual computer use.
- [63:17]
AI Policy and Research Expert
Yeah, and this is also, this is something that I actually want to reach out to some of my friends at meter about this because nominally this seems to contradict, there's all this nuance.
- [63:25]
Jeremy Harris
Right.
- [63:25]
AI Policy and Research Expert
So it's obviously not going to turn out this way, but it seems to contradict some of the meter curves that we've seen. Like if these tasks actually do take, you know, an hour and 6, 1.6 hours, I should say, to complete on average for humans or median time for humans, you know, and we're seeing in this case. So the best model was Claude 4.8 Opus on maximum thinking. So it completed about 20.6% of tasks with what they're saying, 54.8% partial score. That's much, much, much lower than.
- [63:57]
Jeremy Harris
Yeah, it's worth noting the thing with computer or what is this OS OS world is this is graphical user interface usage. So across the board every AI model is worse at this than at sort of terminal like pure coding.
- [64:12]
AI Policy and Research Expert
No, and you're right, I mean that probably is the whole thing actually, especially for claude where multimodality seems to be more of an issue. Though interestingly, GPT 5.5 plateaus around 13%. So it's interesting. GPT 5.5 is more token efficient, but it plateaus near 13%, which is consistent with my internal caricature of OpenAI versus anthropic, where OpenAI is better at the multimodal stuff, but they're worse at the reasoning, the kind of like thinking part of things. And so you see it takes more tokens for an anthropic model because a lot of those tokens are just being dedicated to being better at the multimodal side. But ultimately the reasoning allows them to get to a higher plateau. That may be wrong for a whole bunch of reasons, but at least it's my like rough sketch in my head right now what's going on here.
- [64:59]
Jeremy Harris
So.
- [65:00]
AI Policy and Research Expert
So the kinds of tasks that they're reorienting towards are more what they think are like more real world workflows, sort of like underrepresented in, in the past versions of these benchmarks. So a lot of like what they call interaction design challenges, streaming interaction or like environments that change on the fly. And then also things like cross source reasoning. So you know, it's not enough to just pull up one one source, you have to actually check it against something else, whatever. So there's a lot of this sort of like walk and chew gum at the same time stuff that, that seems to be the focus of the benchmark.
- [65:32]
Jeremy Harris
Next up, another benchmark bench Terminal use agent bench a benchmark for general purpose terminal use agents. So we've been ment terminal bench bunch that one is focused on engineering coding. The big deal with this one is expanding beyond that to many other tasks. So this would include office and productivity, web and information, even image editing, video, audio editing, video understanding, diagram drawing, you name it, as a whole bunch of different versions of tasks. And now this is related to with things like cloud code. One of the things people observed is it's not restricted to coding. You can like actually do video editing with it just because it can, you know, leverage various tools. So this is now a benchmark to demonstrate that. And they've got a whole bunch like multimedia science, engineering things outside of coding, quite a few office tasks as well for spreadsheets, documents, presentations, emails, et cetera. So there you go. And this is a collaboration between Meta, Duke and so Stanford. So I wonder if Meta is like starting to put some of their research that they're using to develop their systems as well.
- [66:49]
AI Policy and Research Expert
So by the way, again a case where we see Opus 4. 8 winning at maximum reasoning effort. So their score here is roughly 66%. The like best setup and this is the setup most favorable to that kind of a model. And you're still seeing failure at about a third of the tasks that were, you know, they were adversarially designed but meant to be realistic. So it really does seem like there's more, you know, more to be squeezed out and curious to see if there's an eval on this for Mythos or sorry, for Fable 5 when it comes out. But yeah, I mean, I think there's this interesting question. These two evals back to back, show us the contrast, right? It's like, is this going to be sort of like the argument between humanoid robotics and non humanoid robotics in a way where, you know, like, are we going to reshape the world for the bots or are the bots going to have to accommodate the way the world was already built for humans? In other words, is computer use is just like being given a GUI and click around and do things like a human and add that layer of constraint. Is that the future of agentic interactions on the Internet or is everything going to be terminal based? And the real question is how do we refactor the way we serve up applications to agents to make them natively, say terminal based? Natively kind of writing based. And you know, looking at these two, it's kind of, I don't know if you design it this way, but like having the back to back benchmarks here really does kind of drive that home. And you see the gap, like roughly speaking, something like a 40% performance gap between, you know, what you can do from the terminal and what you can do when you're looking at computer use. And that's huge, right? You're like on your way to saturation with the terminal side, whereas just, you know, nowhere near it with just adding the gui.
- [68:29]
Jeremy Harris
And interestingly, another benchmark also from Meta SWE together, evaluating coding agents in the interactive user sessions. And for the title, the key difference here is that this is evaluating task where a human is in the loop. A human simulator is going to be able to interject, ask questions, redirect the agent, et cetera. There's 109 tasks here and is one of these cases where it's a more nuanced evaluation. And compared to things like SWE bench, we see the models struggle a bit more kind of to execute effectively. Sometimes they have to be corrected. Actually there's, you know, a decent amount of room to improve across the board. Opus 4.8 again leading, but GPT 5.5 very close. So not a huge difference between them.
- [69:26]
AI Policy and Research Expert
Yeah, part of this is kind of the product management side of a coding assistant chip, if you will. So you know, typical coding benchmarks are going to involve like Suite Bench is a great example, right? That whole, that whole category or Terminal Bench, it's very static. Like the agent gets a prompt and then it generates code as the output. And then there's this evaluation of the final code and it's all static, right? But if you code, you know, even if you vibe code, that's not how it really works. Usually there's a whole back and forth, you know, you're, you're incrementally going like, hey, that's not what I wanted. I wanted this. Or the model has to ask you follow up questions. I mean this is why like the usual way, you know, when I do, when I do vibe coding it, like I have the model interview me to start, right? Like do a product management type interview where I'm the customer. And so this is what this is getting at. It's trying to look at can we assess, can we benchmark performance on the kind of interaction quality and the amount of human effort that goes into this. And so what they have is a bunch of recorded human kind of user agent sessions. And they basically analyze them for like, did the agent actually do what it was asked? Like, and stay anchored to the user's intent in a way that's. Anyway, they've got this two axis evaluation thing where it's not just about correctness. They also like look at essentially the user correction rate, how much corrective steering the agent needed or picked up. And then also a fidelity check on whether the simulating agent like faithfully re. Expressed the user's original intent back at them. Sort of like what you do again in a product management interview, like confirming, hey, let me like repeat it back to you to confirm my understanding. Like, here's what I think your problem is, right? So anyway, again, 4.8 knocks it out of the park in first place. Percent pass at one rate. And you know, you've got, you've got here three for three for Opus 4.8. It's not even their best model. So kind of interesting. I mean opus 4.8 relative to the models in its generation, you're really seeing always on a bit of a delay because the evals take a while to come out. But you're, you're seeing why Anthropic seemingly was pulling away in the race until everything was frozen.
- [71:37]
Jeremy Harris
And now onto policy and safety. We've got only one story unusually. Taiwan raids Super Micro and two supply chain partners in widening Nvidia smuggling probe. So this is Taiwan's Keelung District Prosecutor's office. They raided Super Micro's Taiwan office and their partners expanding an investigation into the diversion of Nvidia AI chips into China. Super Micro shares fell 8% and various other impacts. Apparently Taiwanese law doesn't classify unauthorized AI chip exports to China as a crime, but there's been charges of forgery and fraud stature here. The six individuals were summoned for questioning over document offenses. And this, as said, is an expansion of previous efforts. So Taiwan is now getting more involved, it looks like, and is considering new legislation that would restrict AI chip sales to all Chinese customers, not just blacklisted firms.
- [72:45]
AI Policy and Research Expert
Yeah, it's kind of interesting. Who's that guy? Al Capone. Right. They famously got him on tax evasion instead of murder. That's the same thing here. And now, admittedly, as you said, Taiwan's kind of realizing, oh crap, you know what, we should probably make it illegal to skirt US export control laws too while we're at it. So that will change. There was an initial sort of investigation into these 50 or so super microservers that had, you know, top of the line Nvidia chips that were in Taiwan and were exported, as you said, with falsified customs declarations, towards China. That last piece is the real crime, at least in Taiwan, that the, the Taiwanese are, are pushing on. Super Micro is this interesting kind of embattled company. So it has a history of a lot of what my grandfather might have called jiggery pokery. You wouldn't have said it quite like that, but you know, it's pretty close. Yeah. So like, you know, they, they've had, call it credibility deficits. So in the past, in 2018 they were delisted from the NASDAQ. In 2020, there was an accounting fraud settlement with the SEC. So now you've got the appearance of the company being behind some kind of scheme to ship, I mean, pseudo illegally, I mean flagrantly illegally. If the, if the customs document stuff is true to China and you know, their Stock fell like 7 to 9 percentage points based on that. The ticker, by the way, is smci, so not smic, don't confuse those two, don't confuse those two AI hardware related companies. But smic, which is the Chinese tsmc, also don't confuse that, okay? If you, if you confuse those letters, you must be stupid. There's no other possibility. Yeah, it's got nothing to do with the fact that we're almost identical. Smci, Super Micro here. The interesting thing is that there's like a weird attempt they're making to say, okay, the people who did this were not Super Micro it was just the. Who is it? They, they name a couple people. So like criminally charging individuals here and not the. Yeah, yeah, so, so, yeah, Liang. Okay, so, so no one beyond the indicted employees. And I, I, I have to dig to find the set of employees. Basically there's like a small number of employees that Supermicro is claiming were the ones, the baddies who did this. It's got nothing to do with super micro parent like the company itself, they're claiming has nothing to do with this. So that, that's sort of like an interesting defense. We'll see how that plays out. But right now, like 12 raids or something. Yeah. For locations tied to Supermicro does not sound like it's an investigation constrained to just a couple people at Supermicro. It really kind of seems like it's the company. So that's why you're seeing the stock, the stock plummet. And yeah, it's a good test like of whether AI export control laws actually work. Right. Or export controls, I should say, generally. Can you do this by relying on the domestic company to, you know, pull things off like this? Taiwan is doing it. They want the partnership with the United States. They see them as their stable ally. There is a violation of their own domestic laws too, which probably helps here, but yeah, kind of interesting. They've historically stayed out of this like the Taiwanese government. And so now seeing them draft their own restrictions and conducting visible raids is a really interesting next step.
- [75:52]
Jeremy Harris
And on to research and advancements. First paper, auto data, an agentic data scientist to create high quality synthetic data. And again from meta here, which I keep wondering if this research is just showing some of the internal developments like the research is being aimed a little bit towards what the product needs are for creating their frontier models. But regardless, the story here is a more advanced way to generate synthetic data for training and evaluation. The gist is that this is using an AI agent acting as a data scientist to create high quality synthetic training and much more data. It iterates through data creation, analysis, performance measurement and recipe improvement to produce increasingly better data. And on top of that, there is a meta optimization component of the entire agentic system. So the data scientist agent improves over time at its task of creating high quality data. And there's details here of kind of the specifics. So there's some initial data that goes in, there is a main agent and it talks to multiple other agents and it has a challenger LLM. And anyway, it's one of these system papers. When you've seen a lot of these for things like research where you take a thing and you create an agent and there's some other agents and there's a whole workflow and the output of a workflow is supposedly some good training data or some good research or whatever. So I kind of take these things with a grain of salt. I'm generally skeptical of synthetic data in general because it's very easy to sort of show good examples, but then if you try to do it at scale it does more of harm than good. But regardless, this is a pretty serious effort at building something that provides useful synthetic data.
- [77:51]
AI Policy and Research Expert
Yeah, I guess it depends on. I guess you mean synthetic data at the kind of pre training step rather than like the rs like the RL rollouts.
- [78:00]
Jeremy Harris
Everything. Everything including rl, you know. Yeah, I think is it's easy to mess it up and it's only as good as the producer model to some extent. Like there's many issues of it in terms of variety generation, in terms of correctness. It's, it's very tricky in my experience to make it useful.
- [78:21]
AI Policy and Research Expert
But yeah, training stability is like super hard for like RL rollouts and things like that for sure. But like you know, you do see especially when you have like verifiable rewards and things like that used, you know, like you'll see in terms of task generalization and like other. Anyway, that's actually a really interesting thing I feel like we could have a good discussion about. But to your point, like this is, you know, Meta is releasing this. It's a very scale pilled thing to do, which is maybe consistent with like Alex Wang's general position. You know, he's kind of known to be this very scaling pilled fella. So what do you do when you're scaling pill? You try to get AI agents to train or to produce the AI data that you'll use to train AI agents. So that's kind of on brand. One of the things that this is doing is it's really leaning into a kind of lesson that we've really been learning for the last, I would say three, four years. Very clearly that you want to calibrate your training data to a difficulty level that your model is ready to tackle. Right. So this is really where you mentioned that they have this like weak solver model and a strong solver model. So the weak solver model is a crappy, it's like Quinn 3.5, but it's a 4 billion parameter version and the strong solver is like a 400 billion parameter version. So like two orders of magnitude Larger and more capable. And so what you do is you basically like have a, they call it the challenger model, but basically it's some model, it generates some training examples, some potential training examples. The weak and the strong solver both try to, both try to solve the problem that the challenger just generated. And what you want is a situation where the weak solver fails, but the strong solver succeeds because that tells you, hey, we've got a problem here that's like, you know, training example. It's in that butter zone of hard enough to have something to teach, but not so hard that it's hopeless. And historically we've seen things like in the 50% success rate kind of order of magnitude be that sweet spot. And so, so that's really what they're targeting here. Except they're seemingly doing it at pre training and then they do it across a whole bunch of different domains where they see different kinds of failure modes. They do computer science research questions and there they run into the problem. That typical approach to generating problems produced problems that were way too easy. So like the weak solver would already get 70% success rates. And the key thing there was the gap, right between the weak agent and the strong agent. And so the gap there was just a 2 percentage point gap. So it almost didn't matter whether you got the weak or the strong agent to try to solve the problem. And so what they found this loop solved that was by driving the questions towards more detailed questions about algorithm design, numerical claims. And that ended up creating like a much wider, like I think it was like a 30 point gap. So you see this over and over across their benchmarks, different ways of trying to maximize the gap between the weak and the strong agent. And the design questions that like kind of right in the middle.
- [81:12]
Jeremy Harris
Next paper. Reinforcement learning without ground truth solutions can improve LLMs. So the idea of reinforcement learning is you sort of learn by trial and error, right? So you're given a problem, an input, you produce an output and then you get a reward that can be positive or negative 1 or 0, depending on whether your output is correct. Roughly speaking, there's some nuances there. But with regards to LLM agents, typically that is the norm where you produce some answer in terms of like verifiable rl, for instance, you can say this solution is correct or the solution is incorrect. And that has been the norm. I think of using RL in the context of LLMs. So what this paper is introducing is a new framework called river ranking induced verifiable reinforcement learning. Learning where you don't have one solution. There's no right or wrong necessarily, but there are different levels of correctness or different levels of quality. And you may not even know kind of a ground truth optimal solution, but you can rank them. You can say you know the solution is better than that solution. There's some, you know, fancy ish math, how you can use that to produce the reward signal to then be able to train your agent. So to me this reads like a fairly useful way to do rhl because often there is no one solution. Actually for most things, maybe for a lot of things, there's no one solution. There's many solutions and some solutions are better than other solutions. So that is at a high level, exactly what this is demonstrating as something you could do, which again you've seen some examples of things where you've tried to do some ambiguous reward. But this certainly seems like something relatively novel.
- [83:07]
AI Policy and Research Expert
Yeah, yeah, it's actually kind of an interesting take. Let's say you almost call it a take or an implementation of GRPO like Group Relative Policy Optimization. So the way that works is like so you've got your pre trained model and then you produce these rollouts. You're at the reinforcement learning phase. So usually what you do is for a given prompt, create n different rollouts. So maybe eight different rollouts and then you're going to score each one. And usually what you do is you're going to score each rollout on maybe like on a bunch of, let's say different, we might call instances on a bunch of different problem configurations, maybe a bunch of different unit tests or a bunch of different input types to kind of get at different dimensions of that, of that rollout. And then you average those scores together and you get an average score for each rollout in that way. And then what Group Relative Policy Optimization does is it basically looks at, okay, which of the out of these eight rollouts which is performing better on the whole? Like how? Like let me create a reward function based on how different samples, different rollouts perform relative to their peers in this group. And so that's useful when there's really no like concrete correct answer that you have or when otherwise it's advantageous to just compare the quality of rollouts to one another and hill climb on that. What this does is it says, okay, we're going to find basically a fancy way to convert the scores of each rollout. So if you look at each rollout and you're going to rank the rollouts based on which one did better along this dimension of scoring. And then they have a fancy formula that converts those ranks into what they call a winner heavy shaping formula that it preserves the relative ordering of the ranks, but it kind of gives the winner this amplified score and it heavily penalizes rollouts that are straight up invalid. And then there's this kind of middle band that's reserved for everybody else. So rollouts with intermediate levels of performance. And again, that winner kind of gets this very amplified reward signal. And this is all just a way of making sure that you get around a problem where when the model samples a batch of solutions, often it'll put together a lot of near duplicate copies of a pretty mediocre strategy, and only occasionally it'll stumble on a genuinely really exceptional solution. And the problem is, because RL updates can give credit per sample in usual grpo, those many copies of a kind of mediocre solution can outweigh the single kind of exceptional one. And so this is a way of amplifying whatever the very best answer out of the eight or the 16 or whatever samples is. You want to make sure that it, it can kind of be more influential. And then the last piece too is another reason this is important is this whole idea of let's focus on the rank, relative rank of these different rollouts instead of just their, their raw scores is really important because often different test instances give you completely different ranges of raw scores. Like one could produce scores that are in the thousands, others in the tens. And so this is a way essentially of normalizing those scores to the same axis, the same scale. Anyway, they do find that it works well. They get increases in performance by like on this thing called Ali Bench.
- [86:34]
Sponsor/Advertisement Voice
It's.
- [86:35]
AI Policy and Research Expert
Anyway, you won't have to get into it because this is kind of more the technical nuance side, but you do see meaningful increases on coding benchmarks on the order of like 9ish percent, which is pretty big. Anyway, it does seem to reflect what like kind of they added here as words that, you know, I am diarying
- [86:52]
Jeremy Harris
out here a couple last stories in synthetic media and art. First, a bit of a follow up from last week, Neon Buys Artificial, a film about OpenAI after Amazon dropped it. So this is independent film distributed Neon. Artificial is director Luca Guadagnino's upcoming drama about OpenAI CEO Sam Altman that got dropped despite being nearly complete. It stars Andrew Garfield, big actor, following the events surrounding Altman's brief dismissal and reinstatement as OpenAI's chief executive in 2023. So this also has people starting as Mia Moradi, people Starring Ilya Sutskever, Geoffrey Hinton, Elon Musk. Like, I want to see this movie so badly, so I want to cover this because I'm excited to know that it is not dropped and this will indeed come out and hopefully be good.
- [87:52]
AI Policy and Research Expert
What do we know about Neon? Like the studio, because Neon is a
- [87:55]
Jeremy Harris
fairly reputable, high quality A24s. It releases some more of the artsy indie type stuff. So no blockbusters, but generally, yeah, produce a lot of good movies.
- [88:07]
AI Policy and Research Expert
That's interesting. I mean, it's kind of one of those things where the smaller players who have the willingness to swing for the fences and take risks may start to benefit now from the industry's angst, unease about that. That's cool.
- [88:21]
Jeremy Harris
It would be a big deal with regards to popular culture and AI. There's a lot of backlash and this certainly is a good story. You know, this whole drama around OpenAI and Sam Altman being like, fired as the CEO with a very cryptic announcement and being reinstated like four days later with a lot of backroom politics, which now we know a lot of details of from the trial with Elon Musk. I'm excited for a movie.
- [88:49]
AI Policy and Research Expert
So it's also. And maybe, maybe you'll see the first ever Last week in AI movie review on the back end of this. But the, the, the, the, the thing I hope they get into is that there is a way in which Silicon Valley works. There are group chats that, that run the valley and, you know, names of people that most people haven't heard of. Guys like Ron Conway like that, that just can plow over layers of, of objection and, and issues that otherwise companies might have. You know, Peter Thiel, obviously people have heard of him, but, like, how does the engine work? And a lot of people don't understand that when you work in the space, you get used to it. You take it for granted. Like, oh, yeah, okay, you got an intro to Ron Conway. Okay, yeah, I'll take that. What it means to have SV angel on your cap table, even though he's left now, and all this stuff. I hope that the movie kind of gets to that. I think that the way that Silicon Valley works is largely really good. But that's a long conversation. The specific way that it, it worked in the context of the Sam Altman thing, you know, different story potentially, and certainly obviously the firing itself, whether that was handled competently.
- [89:57]
Jeremy Harris
Yeah, I mean, that was a case of not business as usual. It was a very strange episode. And Silicon Valley largely operates in a very, I guess, traditional business. But In a very loose relative to sort of traditional established businesses. There's a lot of international relationships and chats and you know, stuff like that. But this was certainly much crazier than anything you would usually see going on in companies. So anyway, there are companies that are not afraid of going against big AI, I guess. And last story, Tidal won't pay royalties on AI generated music, but isn't banning it outright. So Tidal is music streaming service and it will label tracks identified as 100% AI generated. Not sure how they can do that with a special icon that they will try to also expand to substantially. I generated again, it will be not monetizable, but it will allow it. Spotify has also addressed similar concerns with a verification program given real artists a green check mark, while Deezer has developed a detection tool. So I think this in general is a little bit interesting. It's not like a huge deal, but AI music is kind of this weird thing where we've had concerns about AI art displacing artists and so on. And largely it has been true, certainly with, you know, movies or tv. But music is much easier to miss, like when you're listening. And I think there has been a lot we may not be realizing of. There has been some stories of people pushing out thousands of songs and earning millions of dollars. And I think increasingly this will be a problem for platforms like Spotify and Tidal and so on. So it's interesting to see them starting to respond to it. Well, that is it for this somewhat short episode of Last Week in AI. Thank you so much for listening. Thank you for commenting and subscribing and sharing the podcast. We do appreciate, appreciate people tuning in. And please do keep tuning in as we continue to try to post every week and cover all the things happening in AI.
- [92:33]
AI Policy and Research Expert
Tune in when the AI
- [92:38]
Jeremy Harris
begins begins
- [92:41]
Andrei Karlenkov
it's time to break, break it down Last weekend AI come and take a ride Hit the low down on tech and let it slide Last weekend AI come and take a ride Up a ladder through the streets AI is reaching high New tech emerging Watching surgeon fly from the labs to the streets AIs reaching high algorithms shaping up the future sees Tune in, tune in get the latest with ease Last weekend AI come and take a ride Hit the low down on tech and let it slide Last weekend AI come and take a ride I'm a Last week the streets AI's reaching high. From neural nets to robot the headlines pop data driven dreams they just don't stop Every breakthrough, every code unwritten on the edge of change with excitement. We're smitten from machine learning marvels to coding kings futures unfolding, see what it brings.