Loading summary
Transcript78 lines
- [00:00]
Sebastian Mallaby
Monday.com AI agents took over my work
- [00:03]
Prof. G
and I absolutely love it.
- [00:04]
Sebastian Mallaby
Chasing deadlines, writing status reports, updating stakeholders.
- [00:08]
Prof. G
Agents handle the daily grind.
- [00:09]
Starbucks Announcer
Now.
- [00:10]
Botox Announcer
I stay in the loop only when it matters.
- [00:11]
Chevrolet Announcer
Create your own AI agent in minutes
- [00:14]
Sebastian Mallaby
on Monday.com when you need to build
- [00:17]
Indeed Sponsored Jobs Announcer
up your team to handle the growing chaos at work, use Indeed Sponsored Jobs. It gives your job post the boost it needs to be seen and helps reach people with the right skills, certifications and more. Spend less time searching and more time actually interviewing candidates who check all your listeners of this show will get a $75 sponsored job credit at indeed.com podcast that's indeed.com podcast terms and conditions apply. Need a hiring hero? This is a job for Indeed Sponsored Jobs
- [00:48]
Chevrolet Announcer
Chevy is called the heartbeat of America for a reason. With SUVs made to move with your rhythm, the versatile Equinox tackles your entire day. The spacious traverse fits your crew and your whole weekend and tracks bring style with balance value you can count on, all infused with tech that has your back so your drive always hits the right. Cord Chevrolet Together, let's drive.
- [01:27]
Prof. G
Welcome to Prof. G Markets. Cracks are forming in the OpenAI story. Last week, the company reportedly proposed giving the US government a 5% stake worth roughly $43 billion, as a way to share the upside of AI with the public. Critics, however, argue it would amount to a government bailout and see it as a troubling Signal for both OpenAI and the broader AI boom. That news came after reports that OpenAI had pushed back its IPO plans until 2027, adding to concerns about the company's financial position. So we wanted to speak with someone who has spent years studying the history of AI and who also believes that OpenAI could run out of money in the near future. Sebastian Malaby is a prominent journalist, author, Pulitzer Prize finalist, and senior fellow at the Council on Foreign Relations, and today he's joining us to discuss what is next for OpenAI, what is next for the AI industry, and what investors should be watching. Sebastian, thank you so much for joining me on the show. I'd love to start with an article you wrote back in January that was titled this is what convinced me OpenAI will run out of money. And you said back then, quote, My bet is that over the next 18 months, OpenAI runs out of money. We've been seeing a lot of red flags since then, the delaying of the ipo. Later we saw this proposal for the US Government to take a stake in the company. I guess I'll just start with what do you make of the recent news. And do you hold to your prediction?
- [03:07]
Sebastian Mallaby
Yeah, I do hold to my prediction. Back in January, the burn rate was just crazy. So that although OpenAI had good products and quite a lot of traction, 900 million consumers, they won't be able to charge money for the product. Like 5% of the retail consumers were actually paying. If you look at a chart of where these users are, the US is the number two market, India is first, and the next three are kind of Brazil, Indonesia and so forth. So these are not rich consumers. You can't charge them very much money. So they had a business model that imagined that they could throw money in all directions. A collaboration with Jony. I've to have a new form factor which would supplant the iPhone serving Sora video generation models and all this stuff, all of which is very expensive and yet the revenue side simply wasn't there. So the burn seemed to me to be totally unsustainable. And even though Sam Altman is a magician when it comes to raising money, he wasn't going to be raising $660 billion, which is what the internally projected burn rate was for the next five years when you looked at the documents back in January. Now, since then, what's happened is some good news, right? Because OpenAI, I think, has recognized that it had to get the burn rate down. It's pulled out of a bunch of data center building products, stargaz, gate, all that stuff. It's canceled sora, the video generation model, which was a total money loser. And it's tried to impose some sort of order on the chaotic management, but it's only been kind of half successful. And in the meantime, OpenAI is squeezed between anthropic, which is much better at the frontier enterprise applications like coding assistance and cybersecurity stuff and agentic stuff. And then on the other hand, it's squeez by the Gemini model from Google DeepMind, which has now reached more retail consumers and is way better at monetizing from that because Google has plugged AI into its search advertising business and that business is now doing more revenue than ever. So I just think that OpenAI is technically a good lab, but it's very hard to monetize when you have a product where there's a lot of competition and it's kind of a commodity and they're not terribly well managed and they've relied too much on the fake it till you make it Silicon Valley tactic of kind of weird smoke and mirrors fundraising gambits. If you look at the fundraising they did and announced earlier this year, the headline number they raised was $122 billion, which is an astronomical amount. But when you dig in, and I'm amazed the press didn't point this out more, about two thirds of that amount was kind of promises in the future conditional upon having a successful IPO or payment in kind, like access to compute. The actual real money was a small share of the total fundraise. Which raises the question, why announce this massive 122 billion headline number when anyone who digs into it can see it's rubbish? Well, the answer is they're trying to head fake investors into putting more money in, trying to persuade people they have momentum. They don't. And this news that you pointed out just recently that they have delayed it seems their IPO into next year is just the latest icing on the cake, the latest evidence that they talk a big game, but they are behind where they say they are.
- [06:40]
Prof. G
Do you think that the delay of the IPO was in large part because of all of this? Because perhaps Sam Altman and the company know that as soon as Wall street actually gets like an audited review of their financial statements, then suddenly the tide will turn on this company. And suddenly people will say, sure, you might have a great product, but this is not a sustainable business model. Do you think that that was the concern, that people might actually see how the company actually works?
- [07:09]
Sebastian Mallaby
100%. I mean, everybody remembers the WeWork story when WeWork was this rocket ship back in like 2019, and it went out with a prospectus to do the IPO and people looked at it and said, this is a joke. And nobody wanted to buy the shares and the IPO never happened. So you can fail in going for the IPO. And OpenAI is in this very tough position where on the one hand it needs the IPO because it can't hope to raise enough money if it stays private. On the other hand, if it tries to do the ipo, it may not succeed. And then it's really cooked just looking
- [07:50]
Prof. G
at how much they are spending at the moment. We just saw the financials that were released by Ed Zittran, who's this independent journalist. He got his hands on the Numbers. They generated $13 billion last year in revenue. They spent $34 billion, which means that their operating loss was $21 billion. I mean, we could talk about the net loss, which was even higher than that, than that number. But that seems to be like a good roughly estimate of how this business is actually doing. And you mentioned that they're stuck between on the one hand, Gemini and then also Anthropic. Anthropic is an interesting one because we also don't know much about that business and we know that they're unprofitable and we know that they're in a similar business to OpenAI. And as you say, this technology is becoming increasingly commoditized. They said there were reports that maybe they were coming up on a, on a quarter of operating profitability, but I think we probably have to take that with a grain of salt because we don't know how they're doing their accounting. My question how does anthropic compare to OpenAI from a business model perspective?
- [09:09]
Sebastian Mallaby
I think you're making a good point and I agree with you that we don't know as much as we would do if Anthropic were a public company or if the prospectus was public. What we do know is that Anthropic has always targeted enterprise customers, which means the type of customer that actually pay for the product. And we know that it's been ahead on stuff like coding assistance and cybersecurity AI. Not that OpenAI is bad by the way. I mean, it's not far behind, but I think Anthropic is the cutting edge on those particular applications that enterprises are really willing to pay for. And meanwhile, Anthropic has not been sort of distracted into announcing a whole suite of retail oriented business initiatives which came to nothing. I mean, OpenAI announced it was going to do shopping at one point and that doesn't seem to have happened. It said it was going to do ads. I'm not sure they've got terribly far with the ads. It did generate the Sora video model which just was a huge money loser, whereas Anthropic never went down that path. So I think Anthropic has been way more laser focused on the part of the market that makes sense, which is the enterprise part, and just better managed. The other point I'd make is that Anthropic amongst all the frontier labs is known as the one where the churn in terms of the scientists is the lowest. People go there, they believe in the mission, they believe in Dario Amadeus, the leader, and they tend to stay there, they don't hop around, whereas all the other labs are subject to job hopping and that's obviously disruptive.
- [10:55]
Prof. G
I think one of the questions that is in investors minds, especially if you're worried about the potential of an AI bubble and the potential that an AI bubble might pop, is is it an OpenAI problem or is it an AI problem? Is it that OpenAI is just bad at managing their finances and they pursued all these side projects and they don't really know how to get their spending under control? Or is AI as a business model just too expensive relative to the amount of revenue that could be generated by charging customers for using ChatGPT or charging enterprises for these larger enterprise wide AI contracts? What is your view on that debate? And just for context for our listeners, you wrote the Power Law, which is one of the most famous books ever on venture capital. And it's kind of about how venture works as a business model where you do lose money for a number of years and then you figure it out eventually. Is AI going to be that story or is this different?
- [11:56]
Sebastian Mallaby
My view is that we have an OpenAI bubble, but not a general AI bubble. So I think OpenAI for the reasons we've discussed, is a 5050 look. It might work. I'm not saying I don't know that they're going to fail. I'm just saying there's a 50% chance that by next summer we'll find they couldn't really go public in the private markets. They can't raise enough money and they have to sort of sell themselves at some sort of discount to another company. It could be Amazon or Microsoft or some other big company that wants an AI team because technically OpenAI is a good team right. Now, on the more general issue, yeah, there's debate at the moment about whether enterprise customers are having a oh my God moment where they think, oh, these tokens are just so expensive. Now I've spent the last 18 months telling my teams that they should just go out and go wild with AI and experiment and do whatever they feel like and token max. And the more tokens you use, the better of an employee you are because you're showing that you're AI forward. And now, wow, this is expensive. And I've hadn't seen a productivity gain yet. And so what am I doing here? I have to rationalize this and there are lots of stories out there about how companies are imposing a sort of middle layer between the user in the enterprise and the models. And the middle layer is there to switch a query so that if it's a simple query that I'm asking, it gets routed to a cheap, low token consumption model. And then only if it's a seriously difficult one, will it go to a fable or something expensive. So I think there's some sort of sensible rationalization about how the AI customers are spending money on this technology. But fundamentally, fundamentally, if you look back at what's happened since the release of ChatGPT, the clear story is this is unbelievably exciting, fast progress in the tech. I mean, when ChatGPT came out, the thing hallucinated nonstop. Then when GPT4 was plugged in six months later, it basically stop like 80% of the hallucination. Then you got very long context windows so you could put a whole Tolstoy novel into the model and then query it. Then you got these reasoning systems that could do math and logic, which had been impossible before. Then you get agentic system, then you get coding assistants, then you get cybersecurity systems. Now you've got bespoke AI, autonomous scientists emerging. This is unbelievably fast progress. So I fundamentally think that AI as a sector, and therefore the demand for the semiconductors, the data center businesses, all these things that people worry about, I don't think that's a bubble. I think that's for real. And it's going to take a little bit of time for companies to figure out how to ration their people's use of tokens. So it's sort of sensible, but basically they're going to consume a lot of tokens.
- [14:53]
Prof. G
Another data point that the Bears might present that we saw last week is Meta launching their cloud business. And this would be the argument against what you're saying, which by the way, I agree with, but I want to play devil's advocate. The very thing that Meta said they wouldn't do, they're now doing. They said that they would only launch a cloud business if they had, quote, overbuilt. These were Mark Zuckerberg's words just a few months ago. The plan was, let's build out all of these data centers, build out all of this compute capacity, because we within the Meta organization need it so desperately, because we're going to build all of these internal AI products and we're going to AI turbocharge our business. And then they turn around and say, actually we don't have the demand internally that we thought we did. And so we're going to sell it to someone else and we're going to let someone else figure out how to sell an AI product and how to make that a profitable business. Which seems quite bearish from a bubble perspective because it basically said, I mean, who else but Meta would be the one to build out their own suite of AI products if Meta can't crack it, if OpenAI is struggling to crack it, TBD on anthropic, then who's going to crack this? Who's going to make this not just an interesting technology, but an interesting technology that makes money.
- [16:23]
Sebastian Mallaby
And we've seen the same with SpaceX, of course, that they also decided to sell their compute capacity to Anthropic and others because xai, their own model hasn't really got much of an uptake, and so they don't need all the compute they've built for their own model, therefore they're sending it to others. So you could view this as a bear signal, as you've just described, or you could view it as a bull signal because it means that you've got some consolidation going on in the frontier model space, and less competition means better margins for the remaining participants. It means that maybe there will be more pricing power for the ones that are less standing. So I don't agree that that's. I think that's the proper reading. The proper reading is we have a rationalization of the market. If you looked at the whole sort of US ecosystem three, four months ago, you had XAI trying to compete, Meta trying to compete, and then on top of that you had the big three, Google, DeepMind, OpenAI and Anthropic. So that's five. And that's before you count me, Strat in France, Cohere in Canada and all the Chinese models. Right. That's a lot of competition. And I don't think that this thing is going to consolidate down to a winner takes all sort of 2010s social media platform or something. But I think some consolidation is in order such that it looks like cloud computing, where there's kind of three or four big providers. So now we've got three leaders who are still standing within the us, plus the foreign ones. That feels good to me in terms of the future business stability of the sector.
- [18:03]
Prof. G
If OpenAI runs out of money, as you say, per your prediction, what do you think the outcome would be? One of the things that you wrote is that maybe it would be absorbed by another company. How does that play out if indeed what you're saying might happen does happen?
- [18:23]
Sebastian Mallaby
So look, I think we've seen lots of examples of either acquisitions or more recently, acqui hires, where you have a smallish AI company like Inflection, which Mustafa Suleiman was running, and then it got sort of sucked into Microsoft or like character AI which got sucked back into Google. So there's a playbook here. Now, OpenAI is a lot bigger than either of those two, so it would be a more complex playbook. But basically, it seems to me that the demand for AI talent and for AI products, and therefore the compute infrastructure that serves that demand, I don't think that's going away because fundamentally I think this is useful stuff that people are going to figure out how to use productively. And so I don't know whether the whole of OpenAI gets bought by Amazon or Microsoft or some other acquirer, or alternatively there's some kind of fancy acquihire deal where part of OpenAI is sucked into a big company, or alternatively that there's a bit of a splintering and the staff, the technical staff at OpenAI get individually hired into other labs. Who knows, right? What I'm saying is that there's a fundamental problem with the way they're going about their business model. I think they understand that, which is why they pulled out of data center building and various other things in the last six months. But they've got some way to go to fix things and patch it up. And one of the lessons about how you do startups coming out in my previous book, the Power Law, is that when you have a very high valuation, a down round is super painful. They're valued in the last round at $852 billion post money, and in the secondary market, they're trading for a lot less than that. And if they were to just say, okay, we really worth 600 billion, the hit to everybody's equity options inside OpenAI would be horrible and they would lose people. And the hit to investors who had believed in OpenAI would be bad and they would get pissed off, and the whole momentum machine that Sam Altman has built would really go through a convulsion. Now, it might be what you have to do to make this thing sustainable, because. But my point is, once you ratchet all the way up to this very high valuation, it's difficult to climb down. And that is why I think he says, why doesn't the government have 5%? Because a strategy to get out of this box that he's in is for ALTMAN to give 5% to the government. And then the government will say, right, OpenAI is too important to fail now because we own 5% or 10% or something, and they'll do what they did with intel, which they took a 10% stake in last year, and next thing you know, the Commerce Secretary Lutnick is like calling other tech companies in the Valley, saying, you're going to do a deal with intel, you're going to bring intel in as a partner on your next project, blah, blah, blah. And so you've got the US Government, a Trumpy US Government, strong arming other companies into giving business once they're in your corner. So I think that is what Sam Altman's strategy is here, to kind of recruit the investment banker to whom you can't say no, the US Government, which
- [21:50]
Prof. G
seems like he's basically just trying to take some sort of workaround shortcut around capitalism. And it seems like we are increasingly seeing that, like, if you can't figure it out in the free market, then, oh, let's just go over to Washington, walk in the White House, kiss the President's feet, and then hopefully he'll save us. And we are increasingly seeing that that is what is actually happening. We're seeing the government taking up stakes in multiple companies. We're seeing the odds that the government will take stakes in even more companies. Those are going up. They may indeed take a stake in OpenAI. Last I checked on the prediction markets, the odds of that happening were more than a third. It's possible that they would do the same with Anthropic, with Palantir, with Andrew. It makes me very upset because I think of it as cheating. I think that you're kind of cheating the game of capitalism. I'd be curious to get your views there. And then following up on that, that if that actually happens, say OpenAI is running out of money and then Trump just bails them out in whatever way. We use taxpayer dollars to just continue to subsidize the business. What comes after that? Does that mean that OpenAI is fine? Does that mean that the rest of the AI industry is on shaky ground? I'm not quite sure how to even model out that potential scenario.
- [23:15]
Sebastian Mallaby
First of all, I think your formulation that they're cheating capitalism and they're going to the government and doing an end run to run capitalism, I think that's a good, perceptive and quite amusing insight. So thank you for that. I also, though, would say that this is just the way the world is going, or at least the US Is going. So if you look at the number of American companies in which the US Government has announced, either done a deal or has announced the deal and it's yet to be consummated. A colleague of mine called Jonathan Hillman at the Council on Foreign Relations did a formal count which just went up on the Council of Foreign Relations website, and the answer is there are 30 of them, 30 such companies since the Trump team came into Power in January 2025, where there's an equity stake from the US government in a private company. So this is where the world is going. And I think this trend has been very much encouraged by the deceptive example of Intel. So in the case of intel, if you look at what the performance has been since the government took a stake last August, it's been fantastic. I mean, it's been way better than the Philadelphia Semiconductor Index, which is the normal index you would look at as a kind of comparable for how intel has done. Intel, I think is up like almost 400%. The Sox or the semiconductor index in Philadelphia, that's up like 150%. So intel has done incredibly well since the government came in. And I think people just lose sight of the fact that, yeah, it did well because you've got the Commerce Department calling up other companies and ordering them to do business with Intel. So intel gets a whole bunch of contracts and it's like turning its game around because you've got the government behind. Picking a winner. Now, it's one thing to say the government might have a justification picking a winner. When we have a problem with all of the cutting edge semiconductors being made in Taiwan. We don't want to be reliant on an island that could be invaded by China, and so we want domestic US Semiconductor manufacturing. I get that argument right. I don't, I don't believe in extending the same argument to OpenAI, which is just one of multiple American foundation model builders. We don't need OpenAI for any strategic reason. So there would be no justification for picking a winner around OpenAI. So I think that capitalism is sometimes justifiably twisted because you have a national security reason to do so. Backing OpenAI would not be a justifiable instance.
- [26:02]
Prof. G
Well, I could imagine that the justification that would be floated is OpenAI isn't systemic to the real economy, but they'd maybe try to say that, but it's systemic to the stock market because Microsoft's future revenues depend so heavily on OpenAI. So do Google, Amazon XL. I mean, basically all of the hyperscalers, Oracle, a lot of these companies are very, very important to portfolios. They are what make wealthy people wealthy in a lot of cases. And maybe the argument for Trump would be, oh, well, we need to keep this thing afloat, otherwise people's stocks are going to go down. What would you make of that argument?
- [26:48]
Sebastian Mallaby
I'd say welcome to China. I mean, that's the kind of thing the Chinese government would do, is prop up the stock market with government intervention of that sort. I mean, look, in the United States. When the Federal Reserve operates a policy that looks like it might be about stabilizing the stock market, people freak out and say, well, that's a Fed put. And that creates bubbles, more bubbles in the future. And capitalism doesn't work unless there's real risk involved. And that's the Fed. If you have bunch of political types in Washington, the Commerce Department and so forth picking winners and distorting outcomes in the market, you don't have a market anymore. It's not a free market. Your point about this isn't run around capitalism or to say the same point differently. This is an end run against the notion of a fair level playing field on which different companies compete fairly and then the most efficient ones win. That's what we're supposed to believe in as the wellspring of efficiency in American capitalism. Well, if you start de leveling the playing field by picking OpenAI as a winner, you've just trashed the that.
- [27:57]
Prof. G
We'll be right back after the break. And if you're enjoying the show so far, send it to a friend and please follow us on YouTube, Spotify, or wherever you get your podcasts. Support for the show comes from Deleteme. The Internet makes it easy for scammers, stalkers and hackers to do their thing. That's just a fact. But it's important to remember that there are ways to protect yourself. That starts with protecting your data. And Deleteme can help you do just that. Deleteme does all the hard work to wipe your personal information from data broker websites. It was named the top pick for data removal services by Wirecutter. And our producer Jennifer Sanchez has tried Deleteme. She was surprised to find out how much personal information was actually already out there there. But Delete Me made it very easy for her to clean it up. Take control of your data and keep your private life private by signing up for Delete Me now at a special discount for our listeners. Get 20% off your delete me plan when you go to JoinDeleteMe.com Prof. G and use promo code Prof. G at checkout. The only way to get 20% off is to go to joindeleteme.com Prof. G and enter code Prof. G at checkout. That's joindeleteme.com ProfG. Code Prof. G.
- [29:23]
Starbucks Announcer
Tomorrow morning is knocking. Stock your fridge now. How about a creamy mocha frappuccino drink? Or a sweet vanilla smooth caramel maybe? Or a white chocolate mocha? Whichever you choose, delicious coffee awaits. Find Starbucks Frappuccino Drinks wherever you buy your groceries.
- [29:38]
Shopify Announcer
When it's time to scale your business, it's time for Shopify. Get everything you need to grow the way you want, like all the way way. Stack more sales with the best converting checkout on the planet. Track your Cha ching from every channel right in one spot. And turn real time reporting into big time opportunities. Take your business to a whole new level. Switch to Shopify. Start your free trial today.
- [30:12]
Prof. G
We're back with Prof. G Markets. This is a good segue into China, which is a topic that you also wrote about recently. The title of your piece was quote, I went to China to see its progress on AI. We can't beat it. I was in the New York Times recently, your op ed. What did you learn about Chinese progress on AI? And why do you think that we can't beat it?
- [30:36]
Sebastian Mallaby
We haven't mentioned this yet, but I'm going to mention it now because you've given me the excuse. I published a book this year called the Infinity Machine about Demis Hassabis.
- [30:46]
Prof. G
I was going to get to it.
- [30:48]
Sebastian Mallaby
There you are.
- [30:49]
Prof. G
I'm glad you mentioned it.
- [30:51]
Sebastian Mallaby
Go read the Infinity Machine, folks. But no, seriously, the thing about China is it does everything faster. And so they published, although they got my manuscript last, and then they had to translate it into Chinese and then they wanted photographs and other embellishments. They produce in a way, a more complex product, but they actually published it before Penguin Press in the United States or any of the other deals I had in other countries. So I go to China right at the beginning of my book tour. I spend eight days, you know, going to four different cities. Hangzhou, Shenzhen, Shanghai, Beijing, talking to both computer scientists at the private labs like Huawei and Ant Group and so forth, and then also talking to academic computer scientists from universities. And what struck me about these guys is that first of all, they talk about safety. They just bring it up. The notion, which I've heard from friends in Washington, that the Chinese don't give a damn about AI safety is just not true. They do talk about safety. Now, I'm not claiming that the government's policy is to pursue safety or that the majority view in China is that they want safety. China is like the US China has some accelerationists and some people who want to go slower because they're worried about the safety issue. That's the same as the US So neither side is going to de escalate and start going slower unless the other one does as well. But what I'm saying is because to caricature China as like only accelerationist 100%. That's just wrong. And so there is scope to talk to them about safety and maybe we'll come on to that. But the other thing which I observed is that China is very good and very focused on applications. And so if you go to a company like hikvision which is under US sanctions and it's kind of a out of body sort of double take experience when you go there because on the one hand it feels like an American tech company. I love tech companies. They're kind of all about building cool things things and making the world better. I kind of buy that, I drink that Kool Aid, I kind of believe in it. I like technology, right? So I see these people trying to build cool technology and they show me stuff like for example, there is an AI kind of scanning camera thing and you point it at some water and you get a reading on the pollution count in the water. And because they've created that, guess what? There is an internal market in water pollution reduction between different Chinese cities. So if you're the downstream city city, you will pay the upstream city to reduce the pollution in the water that's going to come downstream to you. And so you can do pollution reduction when you can measure the pollution. And this is what they're doing at this company, this is what they're building. But they're also under sanctions, these guys by the US because the US says, and historically this was actually true, that they're building other kinds of cameras which are good for surveillance of civilians and so forth in Xinjiang and whatever. So they're both bad guys and they're cool guys. It's a difficult thing to figure out. But whatever they think, whether they are bad, bad or cool, they ain't going away. These guys are for real. They are building cool technology. You go to Huawei, they've got application after application. Here is our special AI to service the bullet train between Shanghai and Beijing every evening. We used to have human technicians, mechanics who would go under the train and make sure it's all fine. Now we just have AI cameras and a couple of robots and they fix the train for you. They are doing this, we're not stopping them. We have imposing closed chip export controls on China to try to hold them back. It hasn't worked. These guys are moving ahead. And the latest thing, as you probably saw, is this model from a group called Zhipu in China, which isn't quite as good as Mythos from Anthropic, but it's pretty close. So we are kidding ourselves if we kind of assume away the reality of China being a technology superpower and we need to to, on the contrary, get our heads out of the ostrich position in the sand and start talking to China about what happens when they have a mythos level model which could hack every single bank in the global financial system and wreak havoc. We need to persuade them not to release it on an open source, open weight basis, because then any criminal can do it and there won't be an off switch.
- [35:08]
Prof. G
What is your view then on AI policy with China? Obviously, the big debate is should we, should we have these export controls? Should we sell chips to China? Are we selling weapons to our enemy? Or do we need to sell dumber chips, basically dumber weapons to the enemy, or should we not have these export controls at all? Do you think that we should have a policy or is the path forward more of a method of diplomacy?
- [35:34]
Sebastian Mallaby
I believe in American power. First of all. I work at the Council on Foreign Relations in New York and we do geopolitics all day long. And I believe that US power is generally a force for good. So I would rather that the Chinese were behind on AI. Okay? And so to the extent that a chip export ban helps us to be ahead, I support it. And indeed, when it was first announced in 2022, I wrote a massive long essay in the Washington Post about why this was a good idea. But the reason I've had my doubts recently is that I look at the results and I'm not seeing that Chinese models are that far behind. And in the meantime, because they're not far behind, I think we have to reckon with the reality that they are building models which are going to destabilize the global cyber system. And unless we persuade them not to release them on an open weight basis, which is what they're doing at the moment, we have serious trouble on our hands. Like everything in cyberspace will be destabilized and we need a policy to deal with this proliferation. And I would be willing, I'm in favor of the chip export ban if we could have it for free and there'd be no downside. But if the effect of having chip export controls is that we can't talk to them about an agreement on not doing open weight mythos, then I'm willing to trade a bit on the chip export ban.
- [36:59]
Prof. G
Just looking at some of how these models have affected the ecosystem. And something we were saying earlier in the us You've got anthropic, you've got OpenAI, you've got Gemini. Those are kind of the heavyweights in the US right now. But it does seem that as pricing becomes more of an issue, companies are more interested in cheaper models, which usually means Chinese models. And indeed that is exactly what we're seeing when we look at OpenRouter, which is basically a tracks developer marketplace for AI models. Chinese models went from less than a third of developer traffic in late 2020 to 60% by mid-2026. There are some companies that American companies that have started using Chinese models. Cursor, Airbnb, Shopify, Uber, Microsoft is currently testing Deepseek. What do you make of this transition over to the Chinese models, specifically the cheap Chinese models? And what role does that play in potentially the policy discussion? Discussion?
- [38:03]
Sebastian Mallaby
Well, I mean, it shows you that they make good models that serious American companies are thinking of using. And so that's another argument for why you can't just pretend that China can be beaten and that's the end of it. And these guys are for real and we have to work with them, not just against them. Now, I think it's useful to just for a moment think through the lens of the Cold War, where when there were nuclear weapons in the Cold War, there were two kinds of big risk, right? One was a nuclear conflagration between the Soviet Union and the United States. And the way we prevented that was through mutually assured destruction, basically close to parity in the power of the two arsenals and therefore deterrence. On the other hand, there was a different category of risk from nuclear weapons, which was the proliferation of these systems to rogue states or terrorists and so forth. And we dealt with that with a separate mechanism which was the non proliferation regime. Now the point is we were both competing with Russia having an arms race with Russia having a Cuban missile crisis with Russia being told by the Russians at the United nations, we will bury you, as Khrushchev said when he banged his shoe on the table. So there was deadly serious competition between the two superpowers, but at the same time there was cooperation on a non proliferation agreement. And the way I see the future with AI is that we'll do the same. We will have inevitable competition between China and the U.S. but we'll also, I hope, have collaboration because the proliferation risk is too awful to contemplate unless you have some collaboration.
- [39:45]
Prof. G
It seems though that what they're doing is basically stealing what we have. People are calling it distillation. And you wrote about this and your definition of distillation, quote, every time a US lab produces a cutting edge model model, Chinese rivals quickly reverse engineer its capabilities and build a copycat version. The follower has the advantage. And when I look at how, I mean, companies are switching to Chinese models because Chinese models are cheaper. And as you say, maybe they'll use the more advanced, cutting edge models in America that are more expensive for certain tasks, the Chinese models for other tasks. Which essentially means that we are kind of, maybe we're collaborating, but also you could say that we're sort of ceding advantage to, to the enemy, to the Chinese players in the AI ecosystem. And it seems that the reason that those models are good, cheap is because of distillation. That is theft. I don't know if I'm being crude by calling it theft. I don't think I am. And I think the Chinese have shown a pretty strong track record of stealing intellectual property from the US and then going out and monetizing it on their own terms. I mean, what do we know about this process of distillation and what do we know about why and how the Chinese models have gotten so cheap and therefore so successful on a global scale?
- [41:10]
Sebastian Mallaby
So distillation is a process which involves asking a very strong model. Like a new American model comes out, a Chinese copycat would ask a ton of questions to that model and get the answers. And the answers amount to training data such that you can train the Chinese model. Like if the question is like this, the answer should be like that. And when a frontier, like a first mover, an American lab has to train the model in some specific, very complicated frontier expertise, like let's say, quantum physics, they expensively hire a bunch of quantum physicists and engage them in creating problem sets and model questions and answers. And generating that training data for the AI is a super expensive, time consuming, painful process. But once you've created the AI that can replicate all those quantum physicists, the Chinese can come along and not hire the human quantum physicist, but just, just query the machine equivalent. And that's what distillation is. Now when these Chinese companies do this, it is not illegal, but it is against contract. In other words, when you sign up to use an American model, you check some boxes and you sign an agreement saying, I'm not going to query you gazillion times and then train my own model by copying what you've done. And so they are violating contracts, contract, but not sort of federal law. That's my understanding of it. Now, whatever the legal niceties, the question is, can you stop it? I mean, I'm all in favor of stopping that if we can. And it seems to me that Anthropic and Google and OpenAI have all of the commercial incentives in the world to put in anti distillation safeguards if they can come up with some. So I think this is a surface self solving problem insofar as it has a solution. And by the way, I should add, Elon Musk the other day or a few months ago casually admitted that his company Xai had distilled from one of the US frontier competitors. So it's not just the Chinese who do this. But look, this is the rough and tumble of the marketplace, it's not nice. But the practical question is, you know, let's stop it if we can. But insofar as we can't, we have to live with a reality on the ground, which is that the Chinese models are good.
- [43:50]
Prof. G
Something I can't figure out is, I mean, if these AI labs are as capable as they say they are, can they not figure out some cybersecurity method to stop the distillation from happening? If Mythos is the most powerful cybersecurity technology and software that the world has ever seen, but we can't figure out how to get these Chinese developers to stop querying and replicating the same software. I'm sort of like, surely you guys can, can figure it out. I guess my follow up would be say they do figure it out. Say we do put an end to Chinese distillation of US AI. Would that not one, solve America's problems in one fell swoop and two, kind of put an end to Chinese AI or at least the progress that they have been making? I mean, is that not kind of a poison pill for China?
- [44:46]
Sebastian Mallaby
I'm not sure is the answer whether if you could stop distillation. And by the way, I think the latest anthropic models do have some anti distillation technology built into them. So we'll see how effective that turns out to be. It's going to be obviously a, you know, cat and mouse, both sides trying to get smarter on this one. But to answer your question, let's posit that US labs figure out a way to stop distillation. Would the Chinese fall behind like a lot or just a bit? I'm not sure anybody really knows the answer. I kind of suspect that if they needed to generate their own data, they would, and they would pay more money and it would be more expensive and it would take them a bit more time, but they would get there because they've got plenty of extremely smart Chinese scientists that they could engage in generating training data.
- [45:42]
Prof. G
We'll be right back. And for even more markets content, sign up for our newsletter@profgumarkets.com.
- [45:58]
Shopify Announcer
This episode is brought to you by State Farm. Listening to this podcast instead of doom scrolling Smart Move. Another smart move getting help from one of State Farm's 19,000 local agents when you choose to bundle home and auto bundling. Just another way to save with the personal price plan. Prices are based on rating plans that vary by state. Coverage options are selected by the customer. Availability, amount of discounts and savings and eligibility vary by state.
- [46:24]
Starbucks Announcer
This episode is brought to you by Palmolive. Family time isn't just the big moments. It's weeknight dinners, sitting around the table, everyone talking all at once. So when the plates are empty and the sink is full, use Palmolive Ultra. Palmolive's most powerful formula removes up to 99.9% of grease, leaving your dishes sparkling clean. And the new convenient pump makes cleaning even easier so you can spend less time tackling dishes and more time together. Shop now@palmolive.com chronic migraine 15 or more
- [46:55]
Indeed Sponsored Jobs Announcer
headache days a month, each lasting 4
- [46:57]
Shopify Announcer
hours or more can make me feel
- [46:58]
Indeed Sponsored Jobs Announcer
like a spectator in my own life.
- [47:00]
Shopify Announcer
Botox Anabachulinum toxin a prevents headaches in adults with chronic migraine. It's not for those with 14 or fewer headache days a month. It's the number one prescribed branded chronic migraine preventive treatment.
- [47:11]
Botox Announcer
Prescription Botox is injected by your doctor. Effects of Botox may spread hours to weeks after injection, causing serious symptoms. Alert your doctor right away as difficulty swallowing, speaking, breathing, eye problems or muscle weakness can be signs of a life threatening condition. Patients with these conditions before injection are at highest risk. Side effects may include allergic reactions, neck and injection site pain, fatigue and headache. Allergic reactions can include rash, welts, asthma symptoms and dizziness. Don't receive Botox if there's a skin infection. Tell your doctor your medical history, muscle or nerve conditions including als, Lou Gehrig's disease, Myasthenia gravis or the Lambert Eaton syndrome and medications including botulinum toxins, as these may increase the risk of serious side effects.
- [47:48]
Sebastian Mallaby
Why wait?
- [47:49]
Shopify Announcer
Ask your doctor. Visit botoxchronicmigraine.com or call 1-844botox to learn more.
- [47:58]
Prof. G
We're back with Profg Markets. Okay, let's turn to the book for a moment. Your most recent book was the Infinity Machine, Demis Hasabis, DeepMind and the Quest for Superintelligence. There was one. One quote from the book that really stood out you said, quote, if you couldn't negotiate safety mechanisms inside one company, what chance would there be to negotiate common safeguards among multiple labs in multiple countries? Which really relates to kind of what we're discussing here in terms of AI safety and AI policy. But also there's an important implication in that, which is that safety mechanisms will were not able to be negotiated within a company. What did you learn about the inner workings of these AI labs and why can't they figure that stuff out?
- [48:47]
Sebastian Mallaby
Embedded in the story of Google, DeepMind and Demis Hassabis is this sort of morality tale about somebody who really wanted to make AI safe. And that was his sort of driving passion from the time he founded DeepMind. In 2010, he bonded with his co founder Shane Legg at a safety lecture lecture in which they discussed the potential for AI to attack humanity by the year 2030, which turns out to be perhaps a prescient projection. At least the capability is going to be there. Whether the AI attacks is a different question. But I mean, anyway, the point is Demis Hassabis was thinking about safety since the beginning. And so when he sold his company to Google in 2014, a condition of the sale was you've got to give me safety and ethics over sight board. I can't let AI be rolled out into the world just on the say so of the Google corporate board. There has to be these independent people from outside. They mentioned Barack Obama as an example. When Barack Obama was leaving the presidency, could we have somebody of that stature who would be on a board saying when it's safe to roll it out? That was Demis vision and it was coupled with a new another hope which was that all of the major scientists would come together in one single effort to roll AI out into the world so there would be no competitive pressure to go unsafely and too quickly. And it turns out, and this kind of transpires through the story that I tell, that all of Demis optimistic stories about how he was going to make AI safe, they all crashed and burned. The idea of just one lab building AI turned out to be a pipe dream. It turns out that humanity is too tribal and competitive and disputatious. There will be multiple labs. When you are kind of confronted with the prospect of being able to build a God machine, there'll be plenty of different sects of worshippers trying to do that. Right. And the idea of oversight within Google. Ultimately the Google board would not agree to giving some outside grandees a veto over how they used this technology that they were spending billions of dollars on developing. They weren't going to do that on a fiduciary basis. Obligation to their shareholders. They couldn't. They felt right. So the point being that the experiment that Demis ran at DeepMind and I discovered all these internal documents which had the back and forth between the red lines from one team of lawyers to the other team about the exact safety mechanisms that they might use, and all secret strategizing that Demis did to threaten to spin out of Google if he didn't get the safety oversight he wanted. And then the Google DeepMind general counsel threatened me and said I wasn't allowed to publish any of this. And I said, the heck with you, I'm publishing it anyway. So it was all quite dramatic. But the bottom line of the story is it turns out to be impossible to impose safety restraints within one AI lab when that lab is in a competitive posture with respect to others. And we saw the same thing play out, of course, at OpenAI, but more in public when the safety board temporarily fired Sam Altman for, like, five days. So what this shows us is that if you want to stop a race which has multiple players, you need the government to enforce restraint on all of the players at once. And if there are players in China, you need the Chinese government to buy in and also agree to put restraints on their guys. When the U.S. puts restraints on labs within the U.S. france, Canada, that's fine. Basically, the U.S. can compel compliance in those places because Cohere in Canada or Mistral in France depend on American technology and the American market to function. But with China, you can't compel them. So there needs to be two countries, two governments that do a deal deal where everybody agrees to put some caution and checking of models before they're released.
- [52:57]
Prof. G
It was the policy of the U.S. government that they were going to do none of that. And they said, I mean, they even issued an executive order banning states from trying to regulate AI in their own way. But then it seems like they've kind of turned on this. Last month, Trump signed a new executive order where he basically asks companies to hand over their models to the government, let the government check them, and then kind of greenlight them. But, I mean, on the one hand, it's progress in the direction that you think is the right direction, but also it's not very harsh or strict or strong. It's basically just like, hey, could you please send your model over? We'd appreciate that. What do you make of Trump's AI policy at this point in terms of government oversight? Over these AI models and their safety.
- [53:56]
Sebastian Mallaby
Given my perspective that government needs to get involved, I've been very much cheered up by what's happened since April, when Mythos first came on the scene and galvanized the US Government into paying attention and restricting the release. Because also, although you could argue correctly, that on paper, the executive order is kind of a voluntary collaboration system with Frontier Labs, blah blah, blah, blah. The reality is it's not voluntary in the least. Right. I mean, Commerce recently called up Sam Altman at OpenAI and ordered him to seek government permission before he gave his latest model to any customer.
- [54:37]
Ryan Reynolds
Right.
- [54:37]
Sebastian Mallaby
Government has to sign off on each customer, customer by customer. This is extremely heavy handed handed. Right. So I think they're in it for real, the government. They have realized that they can't let this stuff disseminate around the world without being controlled by government. And so we're going to get pretty tough controls. I think the gap in the system is that they're not talking about doing this in coordination with China, because the US policy world has two kinds of China expert that you've got. The kind of people who are always hawkish on China, and then the people who used to be a bit hopeful about collaborating with China, but then Xi Jinping rose to power and seemed to kind of frustrate all those hopes of collaboration. And so that group of former doves flipped and became uber hawks on China. So you've basically got the traditional hawks and the new hawks, but they're both hawkish and nobody wants to say they want to talk to China. This is the, the problem. This is the huge gap in the posture because the US government has done a 180 on domestic regulation of domestic models. And I welcome that. The next thing that's going to come just because it's necessary and they're not going to have a choice is they're going to have to get over their inhibition about talking to China.
- [55:57]
Prof. G
So is that sort of the solution is get in a room with Xi Jinping and become partners in tackling this together? I mean, it sounds, sounds like kind of simplistic, but maybe that actually is the way to do it. The alternative would be, you know, force their hand in some way, create some sort of policy where you say, no, you're not going to get any chips or you're not going to get this or you're not going to get that. Your view is we just need to talk with them and have more of a relationship.
- [56:27]
Sebastian Mallaby
It's a bit more complicated than that. I mean, I Think you can talk and also put pressure on them at the same time. I mean, going back to that Cold War analogy, there was a vicious competition between the Soviet Union and the United States at the same time as there was collaboration over proliferation. And so I think there will be competition. And by the way, there are ideas around strengthening the chip export controls. And I'm not against that. There is one theory of the case. Economists sometimes talk about corner solutions. You can either have a fully pegged currency or a fully floating one. But if you go for some mushy middle ground where it's kind of semi pegged, then hedge fund speculators are going to see that you're not really determined to defend that and they're going to eat you for lunch, breakfast and dinner. So it's the same thing with this AI policy. There are corner solutions. You could either give up the export controls or be willing to give them up and go talk to them and say, okay, we know you didn't like that. As a show of our sincerity in wanting to work with you, we're going to offer to loosen those controls, but in return we want you to collaborate on fixing this non proliferation risk. Right. That would be one corner solution. Or the other corner solution is you don't say that. To the contrary, you tighten up the chip export controls. There's this massive loophole right now whereby if you're a Chinese model builder, get this, you can train on Nvidia chips, the most advanced versions, all day long, because the cloud compute that you access is in Malaysia. Malaysia or some other offshore place which is fully allowed to import Nvidia chips. The most recent sort. Right. This is a crazy loophole. You're telling the Chinese they can't use Nvidia chips, but then you're letting them just use a data center kind of across the border. It's nuts that that loophole exists. Right? So the other corner solution is get serious about the policy that you've enunciated and cut off the loophole and cut off the destination and put China in a position where it's so weak, it's kind begging for collaboration. Now I'm agnostic. I'm like, we need to collaborate. I'm flexible on how we get there. I think there's different theories.
- [58:40]
Prof. G
Just going back to Trump's changing of his positioning. It used to be we're not going to regulate, we're not going to have any oversight because we believe that if we do that then it stifles innovation and we want markets to do their Thing and AI labs to run free, uninhibited, et cetera. Then Mythos happens. Anthropics model that was a real concern for cybersecurity. And then Trump changes his tune and issues this executive order, which you believe in this, I think is fairly so, that that actually is like stringent. They do take it seriously. Why do you think that happened? What was it about Mythos? Was it maybe something to do with China? Why did they do this thing that ultimately did amount to a 180 on AI policy?
- [59:33]
Sebastian Mallaby
Simply because Mythos was so powerful, it was very threatening. I mean, the prospect that you could take this model and find code vulnerabilities in every single entity on the Internet and then hack it, that's curtains for the financial system. So that's why they took it seriously.
- [59:51]
Prof. G
Yeah, fair enough.
- [59:53]
Sebastian Mallaby
I mean, I think throughout this topic, throughout this topic, the logic of the technology is going to force governments to do things which six months earlier they said they would never, ever do. And that's happened with domestic regulation already in the U.S. i believe it's going to happen with international collaboration.
- [60:11]
Prof. G
I've already started to see pushback from. I mean, Silicon Valley spent a long time not being friends with Washington, and then in the last couple years they became very close friends with people in Washington. And I would assume that this is going to be, I don't know, this is going to cause a rift again because a lot of the technologists said that what we want is government to have no involvement in AI, in artificial intelligence capabilities. And Trump said, sounds good, I'm with you. And now he's not. I'm not really sure what that means for the relationship between Silicon Valley and Washington, but I assume, I don't know if you have any insight into this. I assume it's not going to be great.
- [60:57]
Sebastian Mallaby
Well, look, I mean, you've got this sort of Putin in the oligarchs sort of story. You've got endless examples of authoritarian governments with big business titans and where does the power lie and how stable is that relationship? And the answer is it tends not to be stable. Point one and point two, the government wins because they have the monopoly on coercion. And so I think Silicon Valley is figuring that out and they realize that the government is too powerful to ignore. I mean, Dariel made tried to say to the government, you shouldn't use these tools for certain things like mass domestic surveillance. The government said, get lost. We're going to call you a supply chain risk, and we're not going to be dictated to you. I mean, who won that fight? Clearly, the government fought one.
- [61:47]
Prof. G
It has been fascinating, what watching Trump use the full power of that coercion, even this week when he decided to step into the proceedings of the World cup and he got exactly what he wanted and the US got their player back. Just as we start to wrap up here, you have studied a lot of the characters in AI you wrote your book about Demis Hassavis, founder of Google DeepMind, kind of the OpenAI. Before OpenAI, you studied a lot of these characters. From your research, from writing that book, what did you learn about the people in AI and what has that kind of told you about what might ultimately happen next and who might ultimately win the AI race?
- [62:30]
Sebastian Mallaby
You've got Sam Altman, who is essentially a commercial opportunist, who wants to win commercially, or at least survive commercially, and his drive is to be a big shot. And he thought of running for governor of California at one point and being a political big shot. But then he decided that building AI was like a bigger big shot. And he wants to just put his imprint on it. He's not obviously a PhD scientist. He doesn't have even a first degree because he dropped out of Stanford to do other stuff. It's not to say he isn't anything other than massively smart, but he isn't a deep scientist. Then you've got people like dari Amadei and Demisisabis, who are PhD scientists who come at this from that perspective, who want to use AI to advance deep science. That's their deepest motivation, motivation. And I believe it's very deep with both of them. And I believe that's the reason why they are number one and number two in this race. It's good for recruiting the best scientists. It's also good for holding together and leading a fundamentally scientific enterprise like building artificial general intelligence. And the point where this came home to me is when I was talking to Denis one day about his motivation for building AI AI, and he started to say, listen, when I'm reading scientific papers at 2 o' clock in the morning, Sebastian, I see reality staring at me in the face, calling at me, saying, I'm here to be discovered. And if I had artificial general intelligence, I could discover the fundamental rules that explain the fabric of reality. It would be like understanding all of nature. Nature which presumably may have been created by some kind of divine intelligence. And so in this sense, my quest for AGI is kind of like my way of getting closer to what I might call God.
- [64:20]
Prof. G
Sebastian Mallory is the Paul A. Volker, Senior Fellow for International Economics, the Council on Foreign Relations, a two time Pulitzer Prize finalist. He is the author of six books, including More Money Than God and the Power Law, which have become investment classics. His latest book is the Infinity machine, Demis Hasawas, DeepMind, and the quest for Superintelligence. He also co hosts a weekly CFR podcast, the Spillover, which examines the ripple effects of global events across policy, geopolitics, economics, technology and financial markets. Sebastian, thank you so much for your time.
- [64:50]
Sebastian Mallaby
Thank you so much. Nice to talk to you.
- [64:54]
Prof. G
This episode was produced by Claire Miller and Alison Weiss and engineered by Benjamin Spencer. Our video editor is Jorge Carti. Our research team is dan Shalon, Kristen O' Donoghue and Mia Silverio. Jake McPherson is our social producer, Drew Burrows is our Technical Director and Catherine Dillon is our Executive Producer. Thank you for listening to Profg Markets from Prof. G Media. If you liked what you heard, give us a follow and join us for a fresh take on markets on Monday.
- [65:30]
Shopify Announcer
Happy
- [65:34]
Prof. G
and kind
- [65:36]
Sebastian Mallaby
reunion
- [65:41]
Prof. G
as the water.
- [65:57]
Ryan Reynolds
Hi, Ryan Reynolds here for Mint Mobile. Are you looking for a beach read this summer? May I suggest your big wireless build? It's got suspense, mystery, a slightly flat emotional arc, and a shocking twist where you realize you've been overpaying the entire time. Fortunately, though, Mint's story is better. Every plan $15 a month, even unlimited. That's it. Happy ending, zero tears. Give it a try@mintmobile.com Switch upfront payment
- [66:20]
Shopify Announcer
of $45 for three months, $90 for six months, or $180 for a 12 month plan. Required $15 per month equivalent taxes and fees Extra initial plan term only greater than 50 gigabytes. Me slow when network is busy. See terms.