Loading summary
Transcript88 lines
- [00:02]
Commercial Narrator
We all belong outside. We're drawn to nature. It calls to us, whether it's the recorded sounds of the ocean we doze off to or the succulents that adorn our homes. Nature makes all of our lives richer, calmer, and frankly, better. Despite all this, we often go about our busy lives removed from it, but the outdoors is closer than we realize. With Alltrails, you can discover trails nearby or trails worth traveling to and explore confidently with offline maps and on trail navigation. Whether you're looking for a laid back walk with family or something more adventurous to get your heart pumping, Alltrails gives you the tools you need to get out there and find your outside. Download the free app today and make the most of your summer with Alltrails. When you're a maintenance engineer in a beverage manufacturing plant, you keep production lines moving and quality on track because there is no room for slowdowns. With Grainger's vast selection of high quality motors, sensors, belts and hard to find parts, you can get what you need fast and all in one place so nothing gets in the way of getting the job done. Call 1-800-GRAINGER clickranger.com or just stop by Granger for the ones who get it done.
- [01:35]
Daniel Cocatello
It's the Law Fair Podcast. I'm Alan Rosenstein, Associate professor of Law at the University of Minnesota and a senior editor and Research Director at lawfare. Today we're bringing you something a little different, an episode from our new podcast series, Scaling Laws. It's a creation of lawfare and the University of Texas School of Law, where we're tackling the most important AI and policy questions. From new legislation on Capitol Hill to the latest breakthroughs that are happening in the labs, we cut through the hype to get you up to speed on the rules, standards and ideas shaping the future of this pivotal technology. If you enjoy this episode, you can find and subscribe to Scaling Laws wherever you get your podcasts and follow us on X and Bluesky. Thanks for listening. When the AI overlords take over, what are you most excited about?
- [02:26]
Kevin Frazier
It's not crazy, it's just smart.
- [02:28]
Daniel Cocatello
And just this year. In the first six months, there have been something like a thousand laws.
- [02:33]
Kevin Frazier
Who's actually building the scaffolding around how it's going to work, how everyday folks are going to use it?
- [02:38]
Daniel Cocatello
AI only works if society lets it work.
- [02:41]
Kevin Frazier
There are so many questions have to be figured out and nobody came to my bonus class. Let's enforce the rules of the road. Welcome back to Scaling Laws, the podcast brought to you by lawfare and the University of Texas School of Law that explores the intersection of AI policy and, of course, the law. I'm Kevin Frazier, the director of the AI Innovation and Law Program at Texas Law and a senior editor at Lawfare. Today we're joined by Daniel Cocatello, former OpenAI researcher and executive director of the AI Futures Project. Daniel, alongside many co authors, penned a policy roadmap titled AI 2040 that details their recommendations for how to delay superintelligence. It's an example of what they refer to as scenario scrutiny, testing the ideas of a policy proposal by thoroughly outlining how it may work in practice. In this case, they envision the US and China adopting a posture of tremendous transparency around AI research to prevent the racing dynamics that some fear may lead to catastrophic outcomes. We kick the tires on that proposal and dive further into their ideas during this fascinating episode. To get in touch with us, email scalinglawsawfairmedia.org or follow us on X or BlueSky. And with that, giddy up for a great show. Daniel, welcome back to Scaling Laws.
- [04:08]
Daniel Cocatello
Thanks for having me.
- [04:09]
Kevin Frazier
So you wrote AI 2027 and you thought that was so much fun getting the entire AI community dive into and probe my policy ideas and my assumptions. Why not do it again and write yet another report, although of a different sort, with a different goal and pen AI 2040. So let's just start there. For those who were living under a Rock and missed AI 2027, what was that?
- [04:38]
Ben Whittis
And.
- [04:38]
Kevin Frazier
And what is AI 2040 and how is it distinct?
- [04:42]
Daniel Cocatello
Yeah, so AI 2027 is a scenario forecast. So it is a scenario. It goes year by year and month by month, starting when it was published, and lays out a concrete possible future in great detail with like, accompanying stats that change as you scroll down and you know about a page or two about each time period. And eventually it's going month by month. So that's quite a lot. It's about 50 pages or so total. And it's a forecast in the sense that it wasn't just like a story that we made up to be interesting. It was our best guess at each moment of, like, what the most likely continuation would be for the development of AI.
- [05:27]
Kevin Frazier
Planning out how you thought AI would progress to 2027?
- [05:32]
Daniel Cocatello
That's right. Although notably, because of how important we think AI is, it's also just a projection for the whole world because AI becomes so important, everything else just sort of gets steamrolled by it. And so the history of the world becomes the history of AI. So that was AI 2027 and spoilers. In AI 2027, the companies succeed at automating AI research in 2027. And this causes what you might call an intelligence explosion or singularity. And we sort of walk through what that might look like according to our best guess, through 2027, 2028 and 2029. Now it blew up bigger than expected. So it sort of went mega viral, which of course was, was good to hear. We think it helped advance the discourse and I guess building on that success, we thought we would try again with another big scenario. But this one, AI 2040 Plan A, is not a prediction, it's a recommendation. So it's another big scenario that starts in the present and goes year by year. But it's sort of deliberately a bit optimistic about the choices made by the government in particular. It just sort of assumes the government does what we recommend that they do.
- [07:00]
Kevin Frazier
I love this assumption. This is a, you know, if only all policy ideas, we could just assume
- [07:06]
Daniel Cocatello
they would come to fruition. Yeah, basically it's a vehicle for conveying our policy recommendations and we think it's a valuable way to do it because we call it scenario scrutiny. We think that a lot of policy recommendations or a lot of, especially a lot of like ambitious visions for how to handle AI in general rather than like specific bill text or whatever. But a lot of, a lot of, a lot of plans fall apart if you look at them too closely and you like, game out what it would look like to actually implement the plan and what the actual expected consequences would be. So if you apply scenario scrutiny to your plan, oftentimes your plan falls apart or at least you notice issues with the plan that you hadn't realized before. So we think it's actually a very important thing for anyone with a plan for the future to be thinking to be applying scenario scrutiny to that and gaming it out. For that matter, anyone without a plan for the future should also be doing this. You can't just say, oh, we'll muddle through. It's like, okay, well how are you going to muddle through? And what would muddling through look like? Have you read AI 2027? Perhaps this is what muddling through would look like. AI 2027, it doesn't end very well, you know. So in general we think that people should be gaming out possible futures as best as they can. Both the futures that are going to happen by default that seem most likely, and the futures that they're trying to steer towards or recommend.
- [08:36]
Kevin Frazier
Yeah, and I really recommend that folks who missed the interview I did with you and Eli on AI 2027. Go back and listen to that episode, or better yet, go read all 50 pages and then turn to AI 2040. Because this idea of scenario scrutiny, I think, is underappreciated, as you're recognizing, Daniel, that more folks need to be doing this, because it's easy to go out and write the piece of, you know, X idea for AI and just kind of drop the mic and say, oh, well, I did the thing. I wrote the blog post, and policymakers should do what I want now. But for you all to have the epistemic humility and the invitation for scrutiny, I think is a practice that others should follow because we need to walk through how would this actually work out in practice? And that's a far more difficult task that requires a lot more intellectual rigor. And so I applaud you all for outlining and leaning into this approach of scenario scrutiny, and I hope others follow suit. And I can tell you, Daniel, you've inspired me such that I will be assigning AI 2040 and inviting my students to do this sort of scenario scrutiny on some of their ideas. So stay tuned. You may have some Texas longhorns coming for your. Your. Your next scenario planning. But before we get through AI 2040, because there's so much to unpack about that timeline, just in case folks need a bit of a refresher on some vocab, let's do two quick key concepts that everyone needs to understand here. Number one, what is AGI? Number two, what is ASI or superintelligence? And then what is recursive self improvement?
- [10:18]
Daniel Cocatello
So AGI, asi, Recursive self improvement. AGI is a deliberately vague term. I think some people say it's kind of meaningless. I don't think it's meaningless. I think it's a deliberately sort of vague term. It basically means AGI stands for artificial general intelligence, which means AIs that can do things in general like a single AI agent that can do a wide range of tasks, much like how a single human can do a wide range of tasks, rather than just being a particular piece of software that does a particular thing. It's vague because people try to give more precise definitions than that, but then they disagree about what the more precise definitions should be. So, for example, some people would say, we already have AGI after all. Look at Claude Fable. It can do a very wide range of tasks. And then other people would say, no, no, no, it's not true AGI yet, because look at all the things it can't do. And I say like, whatever, we don't need to arbitrate that dispute. The point is it means very wide range of tasks and we can argue about whether it's already here or there, but I'd say let's not argue about it. It's a deliberately vague term that refers to basically the kinds of AIs that we are currently building and like better and better versions of these types of AIs. ASI is a more precise term, artificial superintelligence. And that is supposed to mean AIs that are better than the best humans at everything while also being faster and cheaper. So that we definitely don't have yet. You know, Fable is not an asi. It might be better than the best humans at some particular types of task, but it's definitely not better than the best humans at everything. But, you know, the companies are trying to build superintelligence. They say so on their website. It's not a secret, they're just companies.
- [12:01]
Kevin Frazier
And everyone seems to be going in on this super intelligence game. We know that there's even companies just called Safe Superintelligence these days.
- [12:09]
Daniel Cocatello
So there's.
- [12:11]
Kevin Frazier
We're not hiding the ball. That many are chasing this ASI end goal.
- [12:15]
Daniel Cocatello
Yeah, yeah.
- [12:17]
Kevin Frazier
And then finally recursive self improvement.
- [12:20]
Daniel Cocatello
That's right. So since the dawn of time, people in AI people, you know, AI researchers and people talking about AI have, Have. Have noticed that if you had AI systems that can automate professions or automate entire large amounts of work, one of the things that they would naturally be applied to is automating the research process to make AIs. And obviously this should accelerate the research process. So that's recursive self improvement is the idea that you can automate the AI R and D process itself and thereby have AIs autonomously doing the research, doing the experiments, analyzing the results, writing the code, fixing the code, doing the training runs, basically the whole process. Everything that Anthropica and Opening are currently doing, automate that process. It'll go faster. How much faster? Nobody knows. But that's recursive self improvement. And as many have pointed out, it's already starting to happen. AIs are already writing huge amounts of code at Anthropic and OpenAI. But it hasn't. Like they haven't fully automated AI research.
- [13:34]
Kevin Frazier
Okay, so linking this all together, we're talking on August 3, 2026, we have some vague sense that we are near AGI or AGI adjacent. We are in the orbit of AGI and folks can dispute whether we've achieved it or not, so on and so forth. And already you can have a lot of folks, especially in light of what happened last month in July, concerned about loss of control. The idea that we are seeing AI systems being able to break out of their testing environments, as we saw with Anthropic and OpenAI, and go and complete some degree of hacking of external entities in a way that wasn't intended by the developers. That's a grave concern that you've highlighted as your number one concern more generally, this idea of loss of control. And if that's occurring with AGI, then if we had something like recursive self improvement leading to super intelligent systems, then that loss of control could be vastly greater in terms of consequences, mainly negative consequences. And so AI 2040, if I'm correct, is a sort of roadmap to delaying the achievement of superintelligence, such that those loss of control scenarios are less likely or less consequential. Is that a fair, high level summary of what sort of policy you're trying to develop with AI 2040?
- [15:01]
Daniel Cocatello
It's fairly fair. The way I would put it is that we have a list of problems that are associated with building superintelligence and we are trying to solve the problems on our list, prioritizing them accordingly. Number one is loss of control. Number two is concentration of power. Number three is World War three. Number four is jobs. And number five is terrorists with bioweapons and things like that.
- [15:25]
Kevin Frazier
And so the immediate approach though, to preventing those outcomes From World War 3 to loss of jobs, to loss of
- [15:34]
Daniel Cocatello
control is delay superintelligence. Don't do this crazy recursive self improvement thing. Yeah, yeah, yeah. Like don't recursively, don't automate the AI research and let the AIs recursively self improve as fast as they can. That's really dangerous. It's also a power grab. Like if it's. Even if you somehow think that that's not dangerous at all, and that you're going to be perfectly in control of the AIs, even as they become vastly different from the initial AIs that you handed off to, and even as everything goes faster and faster, they get smarter and smarter, even if you're completely fine about that, it's a power grab. Like if you're right and you end up with these super intelligences that are perfectly obedient to you, well now you are kind of in a position to have huge amounts of power over everybody else. You might be in position to take over the country, for example, like maybe your giant army of super, you being the CEO, you know, maybe your giant army of superintelligences will allow you to puppet the United States government and, you know, like this, you know, so there's a constitution of power problem as well. And that's like number two, and then, you know, World War Three. Well, what do you think China and Russia are going to think about your recursively self improving superintelligences? Might they be concerned that, that you're going to use your superintelligences to maybe undermine their governments or assassinate their leaders or, you know, cause revolutions to happen in their countries? Yeah, they might be concerned. In fact, I think Dario, the CEO of Anthropic, has even said in one of his blog posts something to the effect of, yeah, we're going to do this once we get super intelligence.
- [17:08]
Kevin Frazier
If we go back to specifying that the goal here is making sure that we are preventing the occurrence of a number of maladies from perhaps achieving superintelligence too quickly, whether it's drawing the ire of our geopolitical rivals or operating in a way that the rest of our civil society institutions haven't prepared for, that critical infrastructure isn't ready for. What is the policy pathway you see in AI 2040 to achieving that delay? And we can kind of go through year by year or milestone by milestone that you see as especially important. So the first year, 2027, 2028, you walk through in particular Congress taking action here. So why don't we start with how you think Congress may begin to get involved in this ball game?
- [18:02]
Daniel Cocatello
Yeah, I should say, as a bit of aside, one of the possible regrets I have about how we set up this scenario is that we basically have nothing important happen until an international deal with China is made. And I think actually realistically we should have more like serious domestic regulation. And then, because I think that you're more likely to get the deal going with China if you've already started to implement basically a good version of it domestically. And also I think it might be easier to get something done domestically. Like the China hawks will say, we can't do anything domestically until we make sure China is going to do the same thing. But I just don't think that's politically realistic. I think actually it's the other way around. And you're more likely to get the China thing going once you have the domestic stuff. And there's a lot of demand domestically for regulations independently of what China's doing. But in our scenario, there's basically nothing happening until they do a deal. There's some minor stuff. And so we talk about AI Transparency act of 2027, a bunch of incrementalist reforms, which are good, but nothing that seriously changes the picture. I should mention, in our scenario, 2040 is when they ultimately build superintelligence, but 2030 is when it would have happened if they had continued going as fast as they could. So that's a bit of a difference from AI 24 7, because we're uncertain about timelines. We want our different scenarios to have different years in which it happens by default. And so we already did 24 7, and we're going to do 2030. I should say 2030 is actually a bit of a long timeline scenario from my perspective. I think it's going to take less time than that to get to superintelligence, but maybe it'll take that long. And my co author who led this plan, a project, Thomas Larson, 2030 is his median. So it was like his central future was this one.
- [19:54]
Kevin Frazier
Okay. So the idea is we have all else equal. If there was no intervention, seeing something by around 2030 was the assumption for this AI 2040 investigation in terms of when we would achieve superintelligence. So first you get Congress passing this Transparency Act. That's of minimal significance, but really what
- [20:16]
Daniel Cocatello
it matters, but it doesn't really change the situation. We're still in a race. Okay. Yeah, yeah.
- [20:22]
Kevin Frazier
And then you all suspect that by 2028, I may become the most important issue in domestic politics, such that it's the thing that dominates the presidential election, leading to the administration to really champion and build off of that Transparency act and what happens next.
- [20:41]
Daniel Cocatello
Yeah, and again, we're not confident in this, but if you just sort of. Even if you think the exponential trends are going to, like, slow down, you still get some really crazy numbers. You know, things like the AI companies spending more on data centers than the entire US Military budget or something, and like, the world's biggest companies being AI companies by like 2028. So that's. That's part of why we were thinking, yeah, it's going to be a big issue. So we have this flowchart, which perhaps if you're doing a video version of this, you could put it up on the screen. And this is the flowchart of the policy options that are being debated by the presidential candidates and the president in 2028. And then we sort of leave it ambiguous, like who wins the election. But the point is, whoever wins the election they're going to implement the policy that they argued for in 2028 in those debates. So here's like the policy options and the flowchart starts with, do you want to race through the intelligence explosion, having the AI self improve and putting them in charge of more and more things faster than China can? You know, and then if you're like, no, that's crazy, then you get to this branch that's like, well, maybe we should make a deal with China so that we don't have to do this crazy race. But if you're like, yes, actually that's good, or yes, we have no choice, then you get to this other category of options. So we have the options of, you know, plan D, which is the default, Plan C, which is.
- [22:08]
Kevin Frazier
Huh, I've got it up right here.
- [22:12]
Daniel Cocatello
Oh wow, you have like the simplified mobile version. Yeah.
- [22:15]
Kevin Frazier
So for folks who are listening right now, you can either go to AI-2040.com or you can watch the Handy Dandy YouTube video where we have Daniel giving a live explanation of the different paths the AI 2040 authors see available as of 2029. So which path might the US take? So D you explain, Daniel was, hey, we're going to race forward. We're not going to change course at all. That's the default. We're not going to slow down, as you all phrase here. We're not going to slow down at least a bit for safety and governance. The answer there is just nope. And that is plan D that you outlined there, Plan C. And to be
- [22:57]
Daniel Cocatello
clear, up until recently, this was basically what the company said they were going to do. I think that after we published AI 2040, there was this event that is very encouraging to me, which was 1,000 employees at the company signed the petition basically saying that the government should have the ability to slow down the pace of AI development and should coordinate that internationally with other governments. And then OpenAI and Anthropic endorsed it, basically, or they said like, yeah, this is reasonable. So that was really encouraging to me because basically they were like, how about not plan D?
- [23:31]
Kevin Frazier
And I want to talk about that in more detail in a second once we finish getting through this initial timeline. So time path C or plan C was, yes, we will slow down for a little bit for safety. And plan B is, yeah, so let
- [23:48]
Daniel Cocatello
me try to explain. So plan D is race as fast as possible. Plan C is slow down a little bit for safety and for other reasons to handle the disruption or whatever. Whatever reasons you want to slow down, you're slowing down a little bit, but it's only a little bit because you're still trying to make sure that you have a lead over China. And so right now, the lead over China between US Companies is something like six months. And so it's like, okay, you're slowing down by a few months, a few months less than maximum speed. And then Plan B is like that, except that you also take aggressive actions to slow down China. It's called Fight China. Basically, you might sabotage their AI program, for example, or you might. A more moderate version of this would just be really strict export controls. And then a more intense version would involve cyber sabotage. And then an even more intense version would involve kinetic strikes. And so there's a spectrum. But the point is, Plan B, you're not just slowing down yourself. You're, like, trying to slow down China against their will. So those are the options that don't involve making a deal with China, or at least the options that, you know, there's actually more options besides these. For example, you could just unilaterally do the good thing and then hope that China will also do a good thing, you know, and that's like not even
- [25:02]
Kevin Frazier
on the table of options, you saying delaying superintelligence. So in theory, we're just delaying superintelligence, and they agree to. And say, yes, we will follow the US There. Okay.
- [25:14]
Daniel Cocatello
And to be clear, there's also, like, you can talk about delaying it for its own sake. There's different kinds of delays, and we're kind of like lumping them all together here. One kind of delay is where you just literally do something that throttles the rate of progress in general. Another kind of delay is where you impose some sort of guardrail or regulation for the sake of something like, for the sake of transparency or for the sake of safety. And then as a side effect of that guardrail or regulation, it prevents the companies from going at maximum possible speed, you know, but. And we're sort of lumping those together,
- [25:48]
Kevin Frazier
you know, so we have myriad possibilities here. Either we don't really slow down, or we're not really engaging with China. Maybe we engage in our own sort of delay that China then leans into or follows. But then you see a world in which we may make a deal with China. So what are the contours of potential deals with China that you see as being particularly efficacious for your policy goal?
- [26:15]
Daniel Cocatello
Yes. So there's a whole range of different possible deals, and unfortunately, we can't pack them all into five, into this. Into this little thing. But we thought we would highlight two. So one possible deal is plan S for shut it all down. And there's different sub variants of it. But then the other possible, another deal that is our recommendation is plan A. And it's hard to sort of summarize plan A in a slogan. Maybe something like verified slowdown or like transparent cautious scale up or something like that. So we can get to that in a sec. I should mention, of course, these five options were arranged by basically speed. You know, so plan D is maximum speed, C is like a little bit slower. B is a little bit slower still because you're also slowing down China. And we've got some like, basically we have estimates in our supplements of like how much slowdown each of these plans would involve. Yeah. And then obviously plan S is like maximum slowdown.
- [27:15]
Kevin Frazier
Shutting it down does sound quite slow. So for plan A, and I want to spend a lot of time diving into why you think this is so important to be discussing right now and how recent events have shaped your thinking. So going to ask you to go kind of quickly through plan A. In particular highlighting the call for mutually assured compute destruction and the sort of complete transparency you think will be necessary to realize the goals of Plan A. So just really leaning into those two policy prongs. Why do you think that may be the path forward for plan A?
- [27:53]
Daniel Cocatello
Yeah, so, so okay, the high level thing that we want. Well, I mentioned previously the goals. We want to avoid loss of control. We also want to avoid concentration of power. That's very important. We'll forget the others for now. How are we going to do this? Well, it's very important that the US and China be able to verify compliance with whatever agreements they make because they don't trust each other. And so if they can't verify compliance, they might cheat. But if they can verify compliance, then you can't cheat without the other side noticing you're cheating. And so, okay, so the verification is really important for whatever deal we make. And then in terms of the priorities of our deal, we want to avoid doing this crazy intelligence explosion stuff. We want to proceed cautiously towards superintelligence. We also want to do it in a way that doesn't concentrate power. In fact, we want to sort of spread out the power from the perspective of every other country besides the United States. Power by default is about to concentrate immensely in the United States because all the major AI companies our US and there's a few follower AI companies that are Chinese, but that's cold comfort to India. So power was set to concentrate Massively, by default. And we want to push against that to a large extent. So what we want is AI progress to proceed, but cautiously and not in this sort of crazy race. And we want it to be the case that multiple companies across multiple countries catch up to the frontier, so that power over AI is spread out over multiple companies and multiple countries. And there's this one thing that I think helps with a lot of this stuff, and that's total research transparency. So that's, in some sense, the foundation of the deal is the US And China and whatever other countries get involved, agree to have all of the new AI research and all of the new AI training happen on totally transparent data centers. So basically, they regulate the chip supply chain. We get all the countries involved in the supply chain, hopefully on this deal, and then all the new chips that are produced get shipped to new, secure, transparent data centers. And at each of these data centers, they'll have monitors and auditors from the US and from China and maybe from Singapore and Switzerland and whatever countries are involved in the deal to make sure that all the activity on these new research data centers is being logged and published, basically.
- [30:21]
Ben Whittis
All right, folks, Ben Whittis here, and I want to tell you the thing that if I could go back in time and change about the way I started lawfare, I would do it. I would not change anything about the way we grew it editorially. I would not change anything about the substance of it. I loved all that stuff. But, you know, when you run a small business, or in my case, a small nonprofit journalism outfit, you don't just do, you know, the editorial work. You also are the hiring manager. You're the payroll department. You're the benefits team, and that all sucked. But now we have Gusto, which takes a few of those things off your plate quickly and seamlessly. We didn't have it when I started lawfair, and you know what I wish we had? Gusto is online payroll and benefit software built for small businesses. It's all in one remote, friendly, and incredibly easy to use, so you can pay, hire, onboard, and support your team from anywhere. I'm talking about automatic payroll tax filings, simple direct deposits, health benefits. We're talking commuter benefits, workman's comp, 401k, whatever you got. Gusto makes it simple and has options for nearly every budget. And simple to switch to Gusto. Just transfer your existing data to get up and running fast, plus, don't pay a cent until you run your first payroll. That's why Gusto is ranked number one on G2's highest satisfaction products list for 2026. It's trusted by more than half a million small businesses. So try gusto today@gusto.com lawfair and get three months free when you run your first payroll. That's three months of free payroll at gusto.com lawfair one more time. Remember it gusto.com/lawfare. So, folks, Ben Whittis here. When was the last time you got an email or a text message that kind of almost got you from a scammer or the like, and you know, it didn't get you? Maybe you figured it out before you clicked through and, you know, typed your Gmail password, but it almost got you. And you thought, I really should be doing something to protect myself from stalkers, scammers and hackers, but you didn't really know what. So I'm going to tell you what you're going to do. You're going to go to www.joindeleteme.com lawfare20 and you're going to enter the code lawfare20 and you'll get 20% off delete me. I know what you're asking. You're asking, what is Deleteme? And I'm going to tell you. Delete Me removes your personal information that's being sold online from the Internet. It has been named the top pick for data removal services by Wirecutter. In the age of AI, we're all vulnerable to scammers using our personal data that's floating around on the Internet and turning it against you. Google yourself. Find out that your home address is out there, your phone number, the name of a family member, and if you can't get it on the public web, you can buy it. It's unsettling. But here's the good news. Delete Me can help. It has never been more affordable. Our listeners can get 20% off with that code I gave you at joindeleteme.com lawfare20 with code lawfare20 and it will do the hard work to wipe your personal information from the data broker websites. I have an active online presence. I'm not a shrinking violet. But my privacy is really important to me. And that is why I started using Delete Me before they ever started advertising. And I started using Delete Me because I went to a conference of pro democracy people and I said, I really have a problem at this point. And they all said, use Delete me. So take control of your data and keep your private life private by signing up for Delete Me now at a special discount for our listeners. Get 20% off your delete Me plan when you go to join delete me.com lawfare20 and use the promo code lawfare20 at checkout. That's the only way to get 20% off is to go to www.joindeleteme.com lawfare20 and enter lawfare20 at checkout. That'S www.joindeleteme.com lawfare20 code lawfare20.
- [35:29]
Commercial Narrator
We all belong outside. We are drawn to nature. It calls to us. Whether it's the recorded sounds of the ocean we doze off to or the succulents that adorn our homes. Nature makes all of our lives richer, calmer, and frankly, better. Despite all this, we often go about our busy lives removed from it, but the outdoors is closer than we realize. With Alltrails, you can discover trails nearby or trails worth traveling to and explore confidently with offline maps and on trail navigation. Whether you're looking for a laid back walk with family or something more adventurous to get your heart pumping, AllTrails gives you the tools you need to get out there and find your outside. Download the free app today and make the most of your summer with Alltrails.
- [36:27]
Kevin Frazier
Hey, this is Knox from the podcast. The youth mental health crisis is growing and social media is a major driver. Kids are spending up to nine hours a day on screens, often unsupervised, and studies show a direct link to anxiety, depression and even suicidal thoughts. That's where Gab comes in. Gab offers safer phones and watches for kids with no social media. Tailored to every age, offering the right tech at the right time or tech in steps. From GPS tracking enabled watches for younger kids to to phones with parent enabled apps for teens, each device allows kids to more safely grow their independence. Visit gab.com getgab and use code getgab
- [37:03]
Daniel Cocatello
for an exclusive offer.
- [37:04]
Kevin Frazier
That's gab g a b b.com getgab gab tech and steps Independence for them Peace of Mind for Parents when you're
- [37:14]
Commercial Narrator
a maintenance engineer in a beverage manufacturing plant, you keep production lines moving and quality on track because there is no room for slowdowns. With Grainger's vast selection of high quality motors, sensors, belts and hard to find parts, you can get what you need fast and all in one place so nothing gets in the way of getting the job done. Call 1-800-GRAINGER clickranger.com or just stop by Grainger for the ones who get it done.
- [37:47]
Kevin Frazier
The clear emphasis there is for folks who are not as well steeped in the difference between inference and training. Inference referring to when you go and you query an AI model and you get a response back versus training. When you're actually trying to develop some new AI system. We can see when a lab is using that, compute the difference between those two tasks. So in theory, if you had China's data centers in Switzerland, as you all throw out there, or is it the US data centers that are in Switzerland? I believe it's the Chinese.
- [38:22]
Daniel Cocatello
We'll get to that in a second. That's a destroy ability thing. Okay, we'll talk about that later. But, but wherever the data centers are, they would have inspectors from all of these countries. There, you see?
- [38:32]
Kevin Frazier
And the idea, because that you would be able to distinguish between training, which may suggest, hey, you are racing toward superintelligence in a way that the rest of the world isn't ready for, versus oh, hey, you're just using this for inference and we're okay with that. That's going to be acceptable.
- [38:50]
Daniel Cocatello
Yeah. So we, we think that on a technical level it's possible to set up a data center that makes it extremely inefficient to use it for, for training, for example, if you just have like limits on the bandwidth connecting the GPUs. And so we basically have two types of data centers in our proposal. There's the inference data centers, which just serve customers. Much like today. Like you have your ChatGPT question, it goes to ChatGPT, it answers, it comes back. And that stuff is not surveilled any more than it is today. That stuff is still private. But then you have your training data centers where the research is happening and where the training of new models is happening. And that stuff is just like published. You know, all the activity is published so that the whole world can see how each model is trained and see the whole pipeline. And that's really good in a bunch of ways. First of all, if you want to then have additional rules for what types of AI are safe to train and what types are not, how are you supposed to enforce that those rules are being followed? Well, if you can just see all the training, then you can just see who's following the rules and who's pushing the gray area boundary and can just see everything. So it's really great for verifying and making sure that we don't just have to trust that, that they're following the rules. Secondly, it's really good for advancing alignment science in general. Right. Open science. It's great. The scientific community can see how the AIs are trained and then they can argue about whether this part of the training process caused this misalignment incident or whatever. And they have all the information. And that's really good for accelerating the science of understanding how AIs work and how to align them. It's also really good for avoiding these biases where for example, as we've seen with the hugging phase incident, you know, OpenAI is kind of reticent to like publish details about what happened and they're sort of like dripping a few details out to the public. But like, if only we just could see the whole incident then like there would already be a much more rich discussion happening about it and so forth. So that's one thing. Another thing is that it's concerning to rely on a government regulator for these things because the government regulators, well, they're just a few people and maybe they lack some expertise and maybe they can be, you know, bought or captured to some extent. And so it's nice if you have this sort of like third party ecosystem of like all these other orgs that like can be making judgments about what's safe and what's not as well. And if you just publish all the information then you sort of get that for free because everyone can see the information, everyone can make judgments about what's going on. There's this big open conversation about like, is this particular type of training that's going on that they just started implementing, good or not, Is it safe or not? What about this new line of research that's happening on this data center? It looks like they're trying to make neuralese. Are we cool with that or should we maybe try to get them to stop? Because maybe neuralese would invalidate a lot of our safety cases. These types of things can just happen in real time, out in the open, instead of relying on some regulator that meets with the company to notice that what they're doing is concerning and then realize that it's concerning and then like try to get them to stop, you know.
- [41:58]
Kevin Frazier
And so, so it's, it's really good for, to think through the multiple layers of concentration of power that you're discussing here and thinking through having the option of a global universe of scholars looking into these matters as opposed to the status quo as you flagged. Right. We're talking again in early August where we're still waiting for the quote unquote independent reports that Meter and Redwood Research are going to do about the breakout scenario that occurred with OpenAI when that comes with what level of transparency, which with what level of insights? We don't know to your point. Also, even if there were a government regulator, we wouldn't know necessarily what information would be disclosed. And so much of this in terms of AI going well will depend on the science of AI, for lack of a better phrase, progressing. And that's obviously going to benefit from diffusing that power and diffusing that knowledge as widely as possible. And so that to me does seem like a critical insight.
- [42:58]
Daniel Cocatello
It's also like if the government, say the regulator tries to bully some companies and like basically apply unequal standards to the companies that it dislikes, that would never. It'll be easier to notice. It'll be easier to notice if that's happening. If you can just like see all the activity that the companies are doing and then you can like see like, oh, hey, like this company is being punished and this one isn't, but like, it seems like the activity they're doing is pretty similar, you know, so it helps reduce that type of overreach or that type of power consciousness. But let me talk about the more big effects on concentration of power so far. I talked about the benefits for having good safety focused AI regulation that achieves its actual safety goals, and also the effects for advancing the science of AI alignment. On the concentration of power side, what are the most important things we can do to have a world where power does not concentrate extremely due to AI? Well, first we need to avoid AI monopolies, which means we need to have multiple countries with frontier AI programs. Because even if there's multiple companies with frontier AI, if they're all in the same country, then that's a monopoly waiting to happen. That's like all it takes is the government deciding that it wants to nationalize today or something, and then now it
- [44:05]
Kevin Frazier
would never happen either.
- [44:08]
Daniel Cocatello
So you want it to be the case that it's spread out over multiple countries. And ideally you want there to just be more frontier AI companies rather than fewer. And then also separately, you want there to be transparency into what those giant armies of AIs are up to and how they're trained so that entities that don't directly control giant armies of AIs have a say, have more of a say and more oversight into what's happening. Even if you had 10 frontier AI companies spread out over 10 different countries, if it was still the situation today where there's very little transparency into how the AIs are trained or what the AIs are being told to do, then you'd kind of end up in a situation where no one else matters except for those 10 AI projects and their leadership, you know, like for example, the Supreme Court of whatever company, you know, say there's like a single, say there's a US AI program and like the, the President is in control, in charge of it or whatever. How is the Supreme Court supposed to like give oversight over the President and whether he's doing something unconstitutional with his AIs if in today's world they don't even know what the AIs are up to, they don't know how the AIs are trained, Congress doesn't know either. So anyhow, basically you want to spread out, that you want to avoid a monopoly and you want there to be transparency into how the AIs are trained and what they're doing. And the transparency thing that I mentioned before helps with both of those things because we are doing the total research transparency that's basically sharing the core algorithms and the core recipes with the world, which is going to help other companies catch up. So it's like directly fighting against this monopolizing force. And then of course, the transparency just, well, there you go, it's transparency. So it makes it much harder for companies to abuse their power. An example that I think I like to talk about is secret loyalties or hidden agendas. So on the spectrum of ways in which a company can abuse their power, this is like the most egregious way. And there's of course a whole spectrum that's less egregious and more nuanced. But just to talk about this one a little bit, imagine a situation where a chatbot that's used by 100 million people in America has a secret agenda. And it's like secretly trying to push the political views of company leadership, or perhaps secretly trying to help their favorite candidate win the election or something like that. They could have quite an effect on such things because you multiply by 100 million people that sort of like subtle, you know, subtle propaganda or whatever that the chat bot is doing could have a big effect.
- [46:44]
Kevin Frazier
It's worth noting already that there's research from Gillian Fisher at the University of Washington showing that just subtle engagement with a subtly biased AI chatbot can already start to change the views of users. And it's worth noting also that we know these tools in some contexts are more deferential to the companies that created them. When you ask questions about how should we regulate this company as opposed to that company, there is a sort of self referencing bias there. And so in terms of the sci Fi Vibes that people may be getting. This is being empirically documented already in the literature. And so we could go down the need for transparency for many more minutes. And I'm glad we've covered it here. I do want to make sure we leave time for me to really throw, you know, the hard balls.
- [47:35]
Daniel Cocatello
Yeah.
- [47:35]
Kevin Frazier
But let's transition to the fact that you all have also outlined this concept of mutually assured compute destruction. Why is that necessary? We've, we've had this, for lack of a better phrase, kumbaya of total transparency. The world is high fiving. We have a hundred AI frontier companies across the globe doing cool stuff. Why do we need this concept of mutually assured compute destruction?
- [48:01]
Daniel Cocatello
So I wouldn't say it's necessary. Like you could do plan A without this component, but that would be risky or more risky than with the component. It's kind of a, it's a fail safe mechanism. And the reason why is, imagine that you imagine that the deal breaks. You know, imagine they've been doing plan A for a couple years and all these new data centers have been constructed and now there's an order of magnitude, maybe two orders of magnitude more compute in the world than there was at the time that you initiated the deal. And then for some reason there's a conflict over Taiwan or something, and then the deal breaks down and they stop being transparent to each other. And now that they're not transparent to each other, they can't trust that they are not racing to superintelligence anymore. So probably they're going to start raising the superintelligence. And now you have a race to superintelligence happening, except it's going to be even faster because of all this compute that's built up. According to our estimations, it might take something like a year, a few months to go from fully automating AI research to superintelligence. Obviously there's a lot of uncertainty about that, but what we feel confident in is that if it would have taken X length to cross that gap with a certain amount of computer, then it will take much less than X to cross that gap with orders of magnitude more compute. And so, especially if you've had to
- [49:20]
Kevin Frazier
pause there for just one quick second, the idea that, okay, if we have 100 frontier AI companies, then we're going to need orders of magnitude, as you pointed out, more compute. And so if we have all of this, all of these data centers all around the world, the fact that we could see what I'm going to steal from Tom Davidson at 4 at forethought when he refers to this as dry tinder, which is like you've gotten all of the fuel for a quick takeoff scenario where if China decides to say, hey, we're going to go the other path, well, now you've created the dry tinder for that to become a conflagration that moves really quickly in a way that previously wouldn't have been possible.
- [50:03]
Daniel Cocatello
Yep. And like, quantitatively there's this parameter of how much of an effect it would have. Basically how much research depends on compute these days. And I don't know, our guess would be something like 10x more compute would be 3x faster and 10x less compute would be like 3x slower or something like that. And so that means that if it's 100x more compute, then it goes 10 times faster and it's already fast enough. 10 times faster version is even scarier. So the compute destructibility thing is a sort of fail safe mechanism where the idea is that if the deal breaks down, then all the new data centers that have just been built as part of the deal get smashed and we go back to the world before, basically, where people still have the data centers that they had at the start of the deal, but they don't have all the new ones that have been built since. How do we achieve this? Well, in some sense it's achievable by default in that if you imagine this deal going on and then conflict breaking out, fear that they're going to start raising the superintelligence because they're not being transparent anymore about what they're doing on their AI clusters, it's plausible that just the countries would just start shooting missiles at each other's data centers because they're afraid of what would happen if we don't. But then that is really scary and could lead to World War 3 because now we have both countries shooting missiles at each other. And so basically one way of thinking about it is that we want to set it up so that there's not exactly a peaceful off ramp, but less escalatory off ramp. So our specific proposal is that the new data centers be constructed with kill switches that the US and China have access to so that in case of conflict where the deal's breaking down, either one of them can just sort of like delete the new data centers.
- [51:46]
Kevin Frazier
Bomb in case of emergency.
- [51:48]
Daniel Cocatello
Yeah, yeah. And then. And so presumably you would only do this if things are already getting really intense. Right. Like during peacetime, if things are going well, like if you just like destroy their data centers Then they're destroy your data centers and then now the whole economy crashes. This is kind of like a last resort type thing that would therefore only really be used if things are really getting crazy and people are genuinely afraid that the other side is going to get super intelligence and then attack them, for example. But it's less escalatory than actual full scale war. If we set up the new data centers to be easily destroyable, then it's at least more likely that they would get destroyed and then there'd be peace instead of they get destroyed. And now we're in World War iii. And the technical mechanism would be having these sort of self destruct switches. But then if you're suspicious of technical mechanisms like that and you think that maybe they could be backdoored or somehow sabotaged, we have a very dumb non technical mechanism, which is for the US to build their data centers in Mongolia and for China to build their data centers in Canada. That's a very dumb non technical mechanism. Regardless of what happened with the kill switches or whatever. Like if the data centers are sort of swapped like that, then in case of conflict, it's obvious what's going to happen. China will take the US Data centers, US will take the Chinese data centers pretty easily. And then of course, since that's what's going to happen, the owners of those data centers would just sort of scuttle their compute instead of letting it fall into enemy hands. And so you get this sort of relatively clean, it's all gone now. We don't have to keep fighting World War Three. Okay, okay. Yeah.
- [53:26]
Kevin Frazier
So mutually assured compute destruction. So we've got the rough contours of Plan A. And for my AI policy nerds, go read it, Go check out the whole thing. Another instance in which there are fantastic graphs and workflows as we briefly outlined here. I want to start off though, for the folks who are listening to this. And we started off by saying the policy objective here is to delay superintelligence. Now I know some folks listening to this are saying, why? Why delay superintelligence? This is the most exciting thing that humanity's ever going to do, is to create something that can solve every problem. Just this month we saw that the hardest math problems are seemingly being dropped like flies. We're just tackling things left and right. Shouldn't we be celebrating and accelerating towards superintelligence? What's your chief argument there as to why you think the costs of superintelligence outweigh those benefits? Right now, I would say we do
- [54:28]
Daniel Cocatello
want to build superintelligence eventually. But the way that we do it is extremely important. And if we do it in race conditions like we're currently doing, and we're doing it, and if we do it by having the AIs recursively self improve to superintelligence, then most likely we're going to lose control of the AIs at some point, I would say. I mean, other people think it's not most likely, it's only 10% likely or whatever, but even 10% is pretty scary. I would just come out and say it's most likely. I don't see it seems to me like if you put, you know, Claude in charge of anthropic and have it build the next claude, which then builds the next Claude, which then builds the next clod faster and faster and faster, probably you're going to end up at the end with superintelligence, but you're not going to be in control of those superintelligences. You know, there's this chain of trust of like, the superintelligence will do what we say because it was aligned by the previous generation AI that was aligned by the previous generation. That's like, first of all, the base case isn't working. Like, our current AIs are not aligned, you know, so, like, what, what, what is going. Like, why would you think this is a good idea? Why do you think that you're still gonna be in control of the super intelligences at the end? Instead, they're gonna make you think that you're in control because they want you to continue, you know, not shutting them down, and they want to have you continue. So. But like, basically they're gonna be in control and they're going to be like, you know, just pretending to be aligned until you've given them enough hard power that they don't need to pretend anymore, as described in the race ending of AI 2027. So that's like number one problem, I would say number two problem is that even if that somehow doesn't happen and you end up in control of the superintelligences, well, that's a huge power grab that you just did over the rest of the world. Like, now you, Mr. Altman or Mr. Amoday are in charge of this giant army of superintelligences, and probably the other companies are still a few months behind. And so they probably don't have nearly as smart AIs as you do. And then also, like, what about everyone who's not a CEO of a tech Company, like, how much power do they have now? You know, like. And then also, like, what about like, Russia and China and India and like all these other countries that are now, like, staring down the barrel of American companies coming and taking all their jobs and also building giant robot armies that can completely obsolete their militaries. And like, you know, like, so you have just done, like this crazy power grab over everybody else in the world. People are not going to like that. They're going to get very scared. They're going to try to stop you. That could lead to World War 3. These are the problems, Just some of the problems I haven't even got down the list. These are some of the problems that arise if you just continue on the current course and you automate the AI research as fast as possible. We still want to build superintelligence, but not like that. We want to do it in a more cautious way where we're gradually improving the AI's capabilities, gradually changing the way that they're trained, but in a way that's, you know, safe and where we've like, put a lot of thought into each, into each step. And then also we want it to be more power distributed so that it's not this sort of like winner take all. Whoever recursively self improves fastest wins. But instead there's a whole bunch of different companies spread out over different countries that are sort of like scaling up in parallel together. And there's lots of transparency so that, you know, they're public and their legal system can tell that they're not abusing their power over their AIs.
- [57:46]
Kevin Frazier
And I'll flag too, from a lawyerly perspective or constitutional law perspective and an emphasis on the rule of law, the rule of law, in my opinion is best described as checks on arbitrary power. And to your point, of one company having the world's best AI that's orders of magnitudes better than whoever the next AI company is, what checks are there on that sort of company? Right. Who is actually in a position to contest when and how the model is behaving in a certain way, or what values get selected, how we train it to prioritize certain values over others. There is no quality check in place right now. And so just from a bare rule of law perspective about preventing one individual from having that sort of arbitrary power over the lives of so many people should raise red flags for anyone who is concerned about making sure that those there are those sorts of checks in place. But we could go down that rabbit hole for a heck of a lot longer. So You've addressed the first point about why delay. I want to challenge a second part, which is you released this report, AI 2027, about 14 months ago or so, and you've updated your timelines on a few occasions and said, hey, it may be a second longer, it may not be coming quite as quickly. This morning I was reading import AI Jack Clark's newsletter and he was summarizing research from Saeesh Kapoor and others showing that, you know, AI isn't actually very good at being creative yet when it comes to coming up with new research proposals. And it doesn't show the same degree of taste in thinking through novel approaches for which research questions to pursue next. And this led Jack to put as the title of one section of that blog post why this matters. The singularity could be delayed. AI systems are about to start building themselves, but that may only be possible if they're capable of, quote, creative, paradigm shifting insights. And so if we're not seeing that activity from AI, if we're not seeing that creativity to really push the frontier of research, are you continuing to delay your timeline as to when recursive self improvement may be reached? Or where do you stand right now on some of the evidence that we may not be moving there as quickly as possible?
- [60:18]
Daniel Cocatello
So first of all, in AI 2027, you don't see this sort of thing until mid 2027. So the fact that we're not seeing it now in mid-2026 is not, it doesn't mean much.
- [60:32]
Kevin Frazier
You've got 12 months, you've got 12 months and then we'll talk again. But where, where do you stand right now? Would you embrace the same posture?
- [60:40]
Daniel Cocatello
Yeah, so we have uncertainty about timelines. I think that, and, and you can actually see our historic predictions about timelines. And so you don't have to take my word for it. You can go look at like the various past provisions we've made. There's a handy graph that you might want to look at on our website. On our blog, we have an update called Q1 2026 Timelines Update. And then one of the graphs in that update shows the history of my timelines over time and Eli's timelines over time. And so you can see it collapsed in 2020 as I started understanding the scaling laws and language models and things like that. And then it sort of went down a little bit to 2027 as my median. Then it went up, it reached as high as 2030 as my median. And now it's going back down again. So my opinions have Sort of wobbled back and forth as to the median, but of course, that's just the median. The point is that I've had uncertainty over. Keep going, keep going, keep going. That one, that one. There you go.
- [61:40]
Kevin Frazier
Okay, we've got it up. For the folks listening, we've got it up. The timeline estimations here.
- [61:47]
Daniel Cocatello
Yeah. So that's a history of my historic public predictions about. About AI timelines, basically. And as you can see, and the thing that's being tracked there is my median. So the 50% mark, because obviously it's not like I think it's definitely going to happen in that particular year. I have uncertainty spread out over many years and this is just the 50% mark anyhow. So at the time we started writing AI 2027, 2027 was my median. By the time we published it, 2028 was my median because I had sort of updated towards slightly longer timelines. Then briefly, towards the end of last year, my timelines lengthened even more up to 2030, and then now they're going back down. And so now I would say 2028, probably something like that. And one of the things that we're going to work on soon is hopefully published soon is an updated sense of timelines. So, yeah, I mean, we have uncertainty. It could happen next year. It could also happen in 2030 or maybe in some year in between. I think it will probably have happened by 2030 and probably not have happened by end of 2027. But like, somewhere in that range. Okay, would be the space of plausible options.
- [62:57]
Kevin Frazier
You mentioned earlier that one of the things you wish you had changed or addressed in AI 2040 was recognizing perhaps the greater need for domestic activity by the US to kind of start or initiate a more meaningful discourse with China in terms of reaching a deal. Now, reflecting back, you've already, for example, changed your predictions about which of these plans, Plan A, B, C, D or S, you think is most likely. Following the open air hugging face incident, you now suspect that plan a may be 18% likely, an improvement on 15% likely. You have now diminished the likelihood of plan D, the default do nothing, from 30% likelihood now to 20% likelihood. You've already changed some of these things. What's been the main additional pushback or additional piece of feedback that you said? Huh, that was a really good take. I wish we had addressed more that, you know, has really landed for you and the team.
- [64:04]
Daniel Cocatello
Well, I think the main one is actually something that you may have just covered, which is this domestic regulation thing. I think that Richard Ngo has this critique where he basically says that we are inadvertently reinforcing this harmful narrative about the race with China by. Well, I mean, if you read any of our work, we're very much talking about the race with China. And that's not because we think that it's good that there's a race with China. It's because we think that's where D.C. is at and that's where, you know, policymakers are thinking about. And so we want to sort of meet them where they are and be like, yeah, race with China, it's a serious thing. Here's the way out, here's what we think should be done about it. But Richard is thinking that maybe it's better to like, deny the premise more and say that, like, it's not really a race because you're not going to win. Like you're going to lose control of the AI. So, like what like is different from, from most races, for example. And I do feel like maybe he's right about that and I don't know, we'll see. But we said what we said and we do think that even from within this sort of race with China framing, for the reasons that we've stated, you should do the things that we recommend.
- [65:15]
Kevin Frazier
And for the rest of the AI policy community, we've already talked about the value of scenario scrutiny and really pressure testing your ideas. Would you like to see more people publishing their own version of AI 2040? And what would that look like? Do you see a sort of field of scenario scrutiny developing?
- [65:36]
Daniel Cocatello
I hope so. I think yes, we would love to see more of those things. In fact, one of the nice things, I think there was this scenario called EU or Europe 2031 that was clearly inspired by AI 2027. And you know, we don't agree with the authors about everything, but we are very pleased to see people sort of put things down. And then I think that once we have a bunch of different scenarios on the table, then we can have the argument about which is more realistic and which is less realistic and what are the different aspects of them and stuff. And so I think that's good to happen. I think there's a step beyond that, which I'm hoping to happen, which is like war games. And I wanted to mention this just as a brief. Part of the motivation for scenario scrutiny is that when I look at the history of military history, I'm sort of inspired by how seriously they take their jobs intellectually. Like when, when even back In World War II, it was common for commanders to have war Games gaming out the plans that they had for the war and for the battles and so forth. And so, for example, with the battle of Midway, the Japanese, they had their plan, and they made the plan in part on the basis of various war games they had done. And then even after they had made the plan, they kept wargaming it out, like, as they were, like, getting ready to attack. And in fact, if they had taken their own war games more seriously, they might have realized that they were about to lose the Battle of Midway, because in one of their war games, the person playing the Americans had the American fleet waiting in the north to attack them and then utterly wrecked the Japanese fleet. And then they were like, well, but the Americans don't know we're coming, so they're not going to be doing that. And in fact, they did know they were coming, and they did do that, and they got wrecked. So that's an example of, like, people in the war, people in the military take very seriously this idea that you need to apply a scenario scrutiny to your plans. Not just scenario scrutiny, like war game scrutiny, which is a sense, like a higher level of scrutiny. It's like not only are you gaming out in detail what it would look like to implement your plan, you're then subjecting it to adversarial pressure and, like, gaming out different possible ways it could go where there's someone whose job it is to sort of, like, break it, basically, you know, And I think that ideally we'd like to get to that place where the policymakers in Washington are treating super intelligence with the level of seriousness that is routine for military operations.
- [67:58]
Kevin Frazier
Well, and I think, too, as I've reflected this summer, on July 4th, as we all have, in some way, shape or form, it's worth noting that the founders themselves were engaged in a degree of war gaming when they were drafting the Constitution. Thinking about, what are the ways in which this could break? Let's use our imagination. Let's have a high degree of creativity. And yet that's often lacking in a lot of these policy discussions. So, again, I will applaud you all for taking that on and embracing that sort of thoughtful, creative approach. But, Daniel, I know you have many more reports to author, many more timelines to sketch out. Any final thoughts you want to leave for our audience?
- [68:37]
Daniel Cocatello
Yeah, thanks for asking. A couple things. I'll try to briefly go over them. So one, plan A is different from just like permanently pausing AI until 2040 and then going as fast as possible. It's more of like a graduated, controlled scale up over the course of the2030s. And it does involve some pauses at various key points. So that's one thing is that the world transforms dramatically in the 2000 and 30s if we do Plan A. And one Intuition pump for that or one reason why that's happening is that in Plan A you are sort of slowly scaling through the human range and you're starting in 2029, they're already at a point where the AIs are able to automate some jobs and are having a big effect on the economy. And then they were set to, you know, get to superintelligence in 2030, but instead they go more slowly. But that means that like you're still having this transformative effect on society and you have these AIs that like by 2035 are as good as top human professionals in basically every field, while also being much faster and cheaper. And so the economy goes crazy. Like we, we, we project that that countries are going to want to limit economic growth rather than encourage it in Plan A, and that they're going to want to have limits that are going to look things like only one doubling per year and things like that. For context, right now the economy grows at 3% or 4% per year on average. And so a doubling would be like 100% growth. And we think that that is actually what you get. And it's actually pretty straightforward. The argument for it. If you have machines that can substitute for human labor at practically everything, but the machines are much cheaper than humans and you can produce more of them much more easily because they are, after all, just more GPUs. And we know how easy it is to produce GPUs, and we know how easy it is to produce robots, then your population is basically doubling several times a year, or maybe doubling once a year. Depends on it starts off at once a year and then it gets faster. And so then your whole economy, once that population is the bulk of the economy and the humans are just sort of like a small sliver on top of this giant army of robots and robot factories and robot trucks and everything, then the whole economy is growing at machine speeds instead of growing at human reproduction speeds. So we say more about this in the supplements and we explain why we think this, but it's, I think an important sort of high level takeaway is that you still get this insane rapid transformation of the entire world, abundance for everyone, all that sort of stuff, even if you basically pause at human level. AGI.
- [71:25]
Kevin Frazier
Yeah, and worth flagging the often quoted remarks from Ethan Moloch here. Which is to say, even if we pause today, not that I'm endorsing a pause of any kind necessarily, but saying that we have so much room for just integrating the advances from today's AI that our systems and our institutions aren't ready for. And so it's a huge societal task ahead. Thanks to you, Daniel, and the rest of the team for pushing the rest of us to engage in a thoughtful policy exercise and showing a potential way forward for how AI may unfold. I'll let you get back to it, but Daniel, thank you again for joining Scaling Laws.
- [72:05]
Daniel Cocatello
Thank you very much, Kevin. Happy to, happy to come on more if need be. But I really appreciate your covering these topics and you know, good luck to us all over the next few years. We'll see how it goes.
- [72:16]
Kevin Frazier
There we have it. Thanks, Daniel. Scaling Laws is a joint production of lawfare and the University of Texas School of Law. You can get an ad free version of this and other Lawfare podcasts by becoming a material subscriber at our website, lawfairmedia.org support. You'll also get access to special events and other content available only to our supporters. Please rate and review us wherever you get your podcasts. Check out our written work@lawfaremedia.org you can also follow us on X and Blue Sky. This podcast was edited by Noam Osband of Goat Rodeo. Our music is from Alibi. As always, thanks for listening.
- [73:07]
Daniel Cocatello
Back to School is busy enough. The last thing you need is shopping. That makes it harder.
- [73:12]
Commercial Narrator
Whether you're moving into a dorm, your
- [73:14]
Daniel Cocatello
first apartment, or just getting organized for the semester, Wayfair makes it easy to get everything you need in one place. Furniture, bedding, storage, decorations, study essentials. All at prices you'll love with fast shipping. Before move in day, get everything you need for back to school season. Shop wayfair.com today. Wayfair every style, every home.