← The Lawfare Podcast
Loading summary
Transcript59 lines
- [00:01]
Kevin Frazier
The following podcast contains advertising to access an ad free version of the Lawfare Podcast. Become a material supporter of lawfare@patreon.com lawfare that's patreon.com Lawfair also check out Lawfare's other podcast offerings, Rational Security Chatter, Lawfare, no Bull and the Aftermath.
- [00:32]
Daniel Cocatello
A Real Lemonade Pet Insurance Review by Madison H.
- [00:35]
Ei Lifland
You know, I thought it was a little ridiculous to get pet insurance, but.
- [00:38]
Daniel Cocatello
I really didn't want to not take.
- [00:40]
Ei Lifland
My little kitten to the vet because of money. In the last eight months, I've taken him in six times because of ear infections and saved literally hundreds of dollars. You may even need to get surgery, and I couldn't consider it without Lemonade's help. I recommend Lemonade to all my friends constantly, and now I'm recommending it to you.
- [00:58]
Hannah Berner
Ch it out@lemonade.com pet if you're anything like us, you love attention. And my favorite way to get all eyes on me is with Next level shiny, glossy hair.
- [01:10]
Ei Lifland
Which is why we're so excited to tell y'all about the new Lamellar Gloss collection from the girlies at Tresemme.
- [01:16]
Hannah Berner
And Gigglers. We've got you too, because Tresemme partnered with us to bring you 1-800-gloss, a special bonus episode of Giggly Squad, where Hannah and I give advice on all things hair and giving gloss. Check out the episode and grab the Lamellar Gloss collection today because I'm officially declaring this spring gloss season.
- [01:39]
Thomas Larson
What we hypothesize is that if you have this super what we call a superhuman coder, which is like, you know, an AI system that is as good as the best human coder, except much faster and cheaper as well, that this would kind of like, in various ways, improve the research productivity by a significant amount.
- [01:56]
Ei Lifland
Kevin it's the Lawfare podcast. I'm Kevin Frazier, the AI Innovation and Law Fellow at Texas Law and a contributing editor at Lawfare, joined by Daniel Cocatello, former OpenAI researcher and executive director of the AI Futures Project, and Ei Lifland, an AI Futures Project researcher.
- [02:16]
Daniel Cocatello
If we do get something like superintelligence, it's probably going to look crazy. There's a lot to think about and a lot's going to happen really fast. And not enough people are talking about this and not enough people are thinking about it, and a very small set of people are like, thinking about it specifically using the medium of actual concrete stories.
- [02:37]
Ei Lifland
Today we're talking about a report that Daniel and eli Co authored, AI 2027. It's a hypothetical narrative exploring how AI may evolve in the coming years. Its bold predictions warrant a close read and, of course, a thorough podcast. What if you could peer just two years into the future and catch a glimpse of a world shaped by superhuman AI? What if that future was bleaker than many hope? What if we change policies, alter AI development, make AI a key issue for the general public? What if that future was more utopian? What would you do to have those answers? Well, Daniel, Eli and a few other co authors set out to describe in Vivid detail what AI development may bring come 2027 and beyond. In their newly released report, AI 2027, they offer a bold year by year narrative of how AI might evolve and upend our social, political and economic systems by the end of the decade. Relying on extensive research, tabletop exercises, and their own forecasting acumen, Daniel and Eli paint quite the picture. Powerful new AI systems shaping geopolitics, accelerating scientific discovery, and forcing world leaders to grapple with questions of alignment, oversight and control. Some have praised the report's creativity and clarity. Others have raised an eyebrow or two at its speculative leaps. But few can deny that it struck a nerve. That's why I'm so glad they joined the Lawfare podcast today. Daniel, I just finished watching White Lotus and I was exposed to way too many spoilers. So some folks may be upset, but I'm going to ask you to just go straight to the chase, Daniel. What future can we expect? What is the best case scenario for AI? What is the worst case scenario that you all paint in your AI 2027 report?
- [04:33]
Daniel Cocatello
So AI 2027 is a scenario forecast. It is just one of many possible futures. It just is the one that seems most likely to us. But it's still, you know, the real future will probably be quite different in many ways from AI 2027. We did our best shot at trying to guess things, but obviously we're going to be wrong in a bunch of ways we still think it's valuable to try. If we had infinite time, we would have portrayed a whole spread of different possibilities representing a whole bunch of different ways things could go. Because we had limited resources, we just chose to pick two possibilities branching off from one branch point. So in our story, we get superhuman encoder autonomous agents in early 2027, and then that speeds up the overall AI research process enough to get complete automation of the AI research cycle. So what you might call artificial general intelligence, for AI research at least, maybe not for other stuff, but at least for AI research, it's able to do everything by mid-2027. And then we have a sort of branch point where there are some concerning warning signs that maybe the AIs aren't actually aligned. Maybe the goals that they're supposed to have, they haven't quite stuck in one of the branches. They sort of apply more scrutiny, slow down a little bit, fix the underlying problems, and then proceed. And the other branch, they do some sort of surface level patch that makes the problem seem to go away and doesn't require a sort of serious overhaul of what they're doing. And that allows them to go faster, but means that they end up with misaligned systems that are smarter than them running their data centers and all of that. And then in both of our endings, the year 2028 is very exciting. In both endings, there's this army of super intelligent AIs on the data centers over the course of 2028 that's working with the president and the company to be aggressively deployed into the military and the economy to make tons of money to. To build all sorts of new weapons, build all sorts of new factories to beat China, right? And meanwhile, of course, over in China, a similar thing is happening with our own AI systems. In both endings, we depict there ultimately being a deal between the US and China. Rather than a war, it could very easily go to war, but we depict a deal happening instead in both endings. And then later, after this sort of amazing robotic transformation has gone on for a while, the outcome is either really, really, really good for most humans or really, really, really bad, depending on who controls the AIs, basically. And in the ending where the AIs were misaligned, nobody controlled them. They had goals that were different from what their human owners wanted. And this only becomes truly apparent when it's too late and they've been put in charge of everything. And then in the other ending, where they managed to solve the technical alignment issues and figure out exactly how to put the goals that they want into the AIs, then the oversight committee of the project ends up in control because they're the ones who get to choose. They're the ones who control the AIs, basically. And the oversight committee is a merger of. It involves people from both the corporate world, the CEOs, and also the government, the president in particular.
- [07:54]
Ei Lifland
So we have this race scenario, we have this slowdown scenario. And what I want listeners to realize is you can read this report, you can listen to this report, you can check out great graphics throughout the entire report. This is quite the multimedia experience. Honestly. I did listen to part of it on 1.25, because we scheduled this so quickly and during runs I was running from super coding AI, running from the Chinese, running from misaligned AI. It was a great experience. So highly recommend folks dive into this.
- [08:30]
Daniel Cocatello
The best way to read it is on the website.
- [08:32]
Ei Lifland
I think reading on the website I do recommend because you get to check out these really cool graphics develop as you're reading it. Admittedly I did just have to shove it into some of my workouts. So I do apologize. I promise I will get the full experience. But Eli, I gotta know. You two and your co authors are some of the smartest AI researchers. Tons of experience, lots of technical knowledge. Why do this? Why come up with this story as. As Daniel used the term, the story of how AI might develop. What were your goals in developing this story and sharing it so broadly?
- [09:12]
Thomas Larson
Yeah, in terms of our goals, I guess sort of like, you know, we believe that we're not sure, but we believe this sort of transformation to a world where, you know, superhuman and eventually, you know, very superhuman AIs could come within three years, five years, if not maybe 10 years, 20 years, et cetera. And you know, we think that this is something that's very hard to predict. So as Daniel said, you know, we're not going to get everything right. But we think it's very important to think through very carefully. You know, it's very easy to kind of like have a very vague, high level story of what might happen. But you know, when you drill into the details, you, you realize some things are wrong. It also seems helpful just to communicate as a tool of communication. You know, kind of like doing the scenario allowed us to even figure out ourselves what we thought would happen because, you know, we didn't know going into the, into the story we hadn't thought in this little detail. And we also hope that it serves as a good communication tool, right? So others can look at our scenario and be like, you know, I disagree with that. You know, I disagree with this part because of this reason. And then, you know, we can listen to their arguments and potentially change our mind if we agree. And they can also lay out their alternative scenarios. And then over time we can kind of compare the scenarios and see how things are going and see which direction it's heading. So I think a big part of just the motivation for this is because first of all, it just helped us ourselves figure out what's going on. And we believe that kind of society is not paying nearly enough attention to this possibility. And kind of writing it out in this detailed way can help us move towards a situation where we can really see a bunch of different concrete ways that things could play out from a wide variety of viewpoints and kind of move forward from there.
- [10:55]
Daniel Cocatello
It's important context. I think a lot of people might be coming to AI 2027 relatively fresh, not having thought much about superintelligence. But that's not the case for the authors of this. We have been thinking about these things for years. And it's not just us. Lots of people have been thinking about these things for years, especially lots of people at these companies. So OpenAI Anthropic and Google DeepMind were literally founded on the idea that they were going to build super intelligent AGI that's better than humans at everything, and that this would massively transform the world and that this could lead to amazing utopia for everybody, or that it could lead to human extinction. These ideas were there at the founding of these companies, and they're part of the motivation for why so many people joined these companies. And within these companies, this conversation has been ongoing for years. And then similarly, outside the companies, there's a small but growing literature on the topic. There's people trying to forecast timelines until AGI. There's people trying to forecast takeoff speeds, which is sort of like, what will the rate of growth look like in various metrics once we have powerful AI systems that are able to automate all or most of the work? There's a growing literature on this topic. But one thing that seemed to be missing from the literature to me was a concrete story of how it's all supposed to come together, right? There's lots of stuff you can go read about, like, here's my estimate for how long it will be, here's my probability distribution for the arrival time of artificial general intelligence, and here's my definitions of what artificial general intelligence is. And here is, like, some essay about why I think we should wake up the US government and get the government involved. And here's someone else's essay about why they think that's bad. And, like, there's all this discourse. But one thing that seemed to be notably lacking was a sort of like, so how is this all supposed to go? What's the actual picture? How does it all come together? So we set out to write that. I think of it as a complement to all that discourse rather than a substitute. I'm not saying you should do scenarios and not do that other stuff. You should totally be doing the other stuff. You could be extrapolating trends Making bets, forecasts about things, all that stuff is great. But then it helps to have these sort of concrete scenarios to sort of focus the discussion and also sort of like stress test your ideas. Because I think a very common experience that many people have had, including myself, is realizing that the sort of vague sense of how they thought things were going to go doesn't actually. It's not even internally consistent. And if you try to write it out in detail, you realize that your own vague sense was actually just incoherent. Right. So Eli and I have run these scenario exercises occasionally over the last year or two where we get a bunch of people in a room and say, you're going to spend the next couple hours writing up like a scenario that represents how you think the next couple decades are going to go in the history of AI and people have found them helpful. For example, multiple different people have tried this and then realized that their timelines have to shorten because they wrote out the advancements that they expect in the near term and then wrote. And then 15 years later we get super intelligence or something, and then they're like, wait a minute, given all the things that have happened up to here, it just shouldn't take that long. So people find them useful, we found them useful. And we're hoping that it will inspire lots of other people to start asking the right questions and thinking more seriously about these topics and hopefully writing alternative scenarios.
- [14:13]
Ei Lifland
Yeah, I gotta say, one of my favorite parts that you all include in the report is this openness for, hey, if you disagree, give us your scenario. Right? This sort of invitation for fan fiction, for lack of a better phrase, I'll come up with a better term when, when I'm feeling more creative. But this fan fiction of, okay, hey, great, if you think we're going to branch off in a different direction, tell us when, tell us why, and let's keep this conversation going. And to your credit, there are so many folks and this isn't necessarily a product of their own doing, but we've reduced a lot of the conversation about the future of AI to what's your P. Doom? Which doesn't do anything for anyone of just saying, this is my probability that I will result in some extinction level event for humanity. But this sort of exercise really forces everyone to say, okay, what are the tangible steps that may occur in the near future that could alter the course of human history or at least the course of AI development? And Eli, I guess one thing that stood out to me was this focus, as Daniel pointed out, on an AGI for super coding, basically a super coder. And why that's so important for the future of AI development. So, you know, I'm just a humble law professor, so if you could describe to me why is it so important for an AGI researcher, a super coder, to develop with respect to the overall takeoff path that we can see for AI's future?
- [15:46]
Thomas Larson
Yeah, I guess let's kind of start by talking a bit about the sort of, like, AI R and D process. So we're going to simplify it a bit here, but kind of at a high level, there's a few phases. The first phase is you choose what experiment you want to run. You have a hypothesis that you want to test, you have a direction, you want to see how well it works. And then you code up this experiment. And so this is a big part of the coding is just kind of like, okay, you have this experiment idea. Now I'm going to actually code this up. And then you run the experiment and you see what the results are. And. And then you have this kind of cycle of making different adjustments to the code, then rerunning, and you're monitoring the experiments, et cetera. And so at a high level, we can think of there as being kind of two types of skills that are involved here. One is experiment prioritization, or we sometimes call it research taste, which is deciding which experiments to run and how to interpret the results of experiments. And then the other kind of skill is implementing the experiments. And so this is the coding basically, that you need to do to test out. Maybe you have some hypothesis like, what if we try to alter the structure of the neural network a bit in this way, and then you kind of need to test it out. And so what we hypothesize is that if you have this, what we call a superhuman coder, which is like an AI system that is as good as the best human coder, except much faster and cheaper as well, that this would kind of like, in various ways, improve the research productivity by a significant amount. One example is that there might be just experiments that they're very valuable to run, but they currently take a long time to code. And so because of that, right now they don't get run. Or if they do get run, you have to invest really a lot of time into actually coding up the experiment to test how it goes. But once you have the superhuman coder, you're able to unlock these new experiments for much cheaper that are very valuable. And maybe a bit more context here is that the way to think about this is that there's two kind of inputs here. There's the labor from the researchers, for example, who are doing the coding, and then there's the compute, basically the sort of amount of computing power that is needed to run these experiments. And so basically what we do in our forecast is we think about different ways that having these superhuman coders would allow you to more efficiently use the compute in experiments, to sort of like run more valuable experiments and get better results. And yeah, the one I gave was just like, sort of like one example, but just, you know, generally, for example, like, there are various other things that the AIs like, maybe they're like better at writing code which doesn't have bugs. You know, like, maybe there's oftentimes right now, like, you know, you run an experiment and then you realize you weren't actually testing what you thought you were testing. And, you know, if you had these superhuman coders who were like pouring over the code before it sort of the experiment even started, and also monitoring it constantly, this is something that could allow you to sort of reduce the amount of the bugs that you have. And so we list various ways that having the coder could kind of Speed up the AI R&D process. And then we see how long it'll take to go from coding to the next step of a fully superhuman AI researcher. And then we kind of go on, go on from there. And correspondingly, with each milestone, we estimate how much it would Speed up the AI R&D process. And then we estimate what that would sort of imply for how long it took to get from that milestone to the next one. And one more thing to note here about the coding is that the AI companies are already focusing a lot on coding to varying degrees, but there's already a lot of focus there. So that's something that is not fully even a prediction, but just an observation that AI companies are really putting a lot of effort into making their AIs better at coding already.
- [19:35]
Dr. Patrick McGrath
This episode of the Lafair podcast is brought to you by Wildgreen. Wildgreen is the first bake from frozen subscription box for artisanal breads, pastries and pastas. Wildgrain's boxes are fully customizable to your tastes and dietary restrictions. In addition to their classic variety box, they recently launched a new gluten free box and a plant based box that is 100% vegan. Best of all, they take the hassle out of baking since all items bake from frozen in 25 minutes or less with no mess or cleanup. I'VE been really impressed by how easy Wildgreen makes the whole experience, everything from online ordering to tracking deliveries, to unpacking and storing, and of course, the baking itself. Are you ready to bring all your favorite carbs right to your doorstep? Be sure to check out Wildgrain so you can begin building your own box of artisanal breads, pastas and pastries. For a limited time, Wildgrain is offering our listeners $30 off the first box plus free croissant in every box when you go to wildgrain.com lawfair to start your subscription. You heard me. Free croissant in every box and $30 off your first box when you go to wildgrain.com that's wildgrain.com lawfair or you can use promo code lawfair at checkout.
- [20:58]
Kevin Frazier
Did you know? Fast Growing Trees is the biggest online nursery in the United States with thousands of different plants and more than 2 million happy customers. They have all the plants your yard needs like fruit trees, privacy trees, flowering trees, shrubs, and so much more. More Whatever plants you're interested in, Fast Growing Trees has you covered. Find the perfect fit for your climate and space. Fast Growing Trees makes it easy to get your dream yard. Order online and get plants delivered directly to your door in just a few days without ever leaving home. Their alive and thrive guarantee ensures your plants arrive happy and healthy. Plus, get support from trained plant experts on call to help you plan your garden. I've been redoing my cabin in the woods recently and gutting a house is pretty expensive. Instead of paying thousands more for expensive landscaping, I have a honey crisp apple tree. Coming from Fast Growing trees to plant, I couldn't be more excited about the shade and frankly the apples and the cost savings is really hard to beat. You don't have to drive around to nurseries and big garden centers and choose plants from a limited assortment. Fast Growing Trees makes it easy to order online and your plants are delivered to your door in just a few days. Whether you're looking to add privacy, shade or natural beauty or apples to your yard, Fast Growing Trees has in house experts ready to help you make the right selection. With plant experts on call to offer growing and care advice, you can talk to a plant expert about your soil type, landscape design, how to take care of your plants and everything else you need. No green thumb required. This spring they have the best deals for your yard, up to half off on selected plants and other deals and listeners to our show get 15% off their first purchase when using the code Lawfair at checkout that's an additional 15% off at fast growing trees.com using the code lawfair at checkout fast growing trees.com code lawfare now's the perfect time to plant. Use Lawfare to save today. Offer is valid for a limited time. Terms and conditions may apply. Work management platforms endless onboarding it bottlenecks.
- [23:43]
Ei Lifland
Admin requests but what if things were different? Monday.com is different.
- [23:51]
Kevin Frazier
No lengthy onboarding Beautiful reports in minutes.
- [23:55]
Ei Lifland
Custom workflows you can build on your own Easy to use prompt Free AI huh?
- [24:00]
Kevin Frazier
Turns out you can love a work management platform.
- [24:03]
Ei Lifland
Monday.com the first work platform you'll love to use.
- [24:08]
Hannah Berner
If you're anything like us, you love attention. And my favorite way to get all eyes on me is with next level shiny glossy hair.
- [24:15]
Ei Lifland
Which is why we're so excited to tell y'all about the new Lamellar Gloss.
- [24:19]
Hannah Berner
Collection from the girlies at Tresemme and Gigglers. We've got you too, because Tresemme partnered with us to bring you 1-800-gloss, a special bonus episode of Giggly Squad, where Hannah and I give advice on all things hair and giving gloss. Check out the episode and grab the Lamellar Gloss collection today because I'm officially declaring this spring Gloss season.
- [24:40]
Scott Alexander
What if I told you that right now, millions of people are living with a debilitating condition that's so misunderstood, many of them don't even know that they have it. That condition is Obsessive compulsive disorder, or OCD. I'm Dr. Patrick McGrath, the chief clinical officer of NOCD, and in the 25 years I've been treating OCD, I've met so many people who are suffering from the condition in silence, unaware of just what it was. OCD can create overwhelming anxiety and fear around what you value most, make you question your identity, beliefs and morals, and drive you to perform mentally and physically draining compulsions or rituals. Over my career, I've seen just how devastating OCD can be when it's left untreated. But help is available. That's where NOCD comes in. NOCD is the world's largest virtual therapy provider for obsessive Compulsive Disorder. Our licensed therapists are trained in exposure and Response Prevention Therapy, a specialized treatment proven to be incredibly effective for OCD. So visit nocd.com to schedule a free 15 minute call with our team. That's nocd.com.
- [25:42]
Ei Lifland
Yeah, this idea of just a constantly improving AI system, it's. It's pretty obvious even for someone like me with Less technical experience to see why. If you were an AI lab, you would want to bet as much as you can on that strategy because you're going to jump ahead of your competitors if you get there first. And I think just to ground this report a little bit further, I'll go over a quick couple quick high points. So we focus here on Open Brain, a fictional AI company set in the United States that builds a powerful AI system known as Agent 1. China then, quote, wakes up and doubles down in a big way on its own AI efforts, stockpiling and concentrating compute, which we all know is a core ingredient of AI development. On the domestic front, we see AI lead to some layoffs. Some folks begin to get upset about AI's integration into the economy. Then great plot twist. China steals one of the advanced AI models from Open Brain. What does the US President do? Naturally responds with a cyber attack. The US DoD and Open Brain then reach an extensive contract with one another. We see continual tech breakthroughs and developments, a few setbacks from time to time, but then ultimately we see that self improving AI is attained as soon as June 2027, which leads to a huge upswing in AI progress. So, Daniel, with this just initial set of circumstances that we could see develop in the next two years, can you walk through your degree of uncertainty with respect to some of these developments? Obviously, like you said, if you had all the time in the world, you would have outlined a whole bunch of alternative scenarios. You're no stranger to coming up with some really astute, really accurate AI predictions. What gives you a sense that these scenarios in particular may be on the table or are worthy of study by folks who are thinking about AI governance right now?
- [27:52]
Daniel Cocatello
The way that we like to think about it is by breaking down into these milestones of capability. So we sort of chopped it up into superhuman coder, superhuman AI researcher, which is sort of AGI, but for AI research at least, it can do the whole thing. And then beyond that, we have a couple other levels beyond that. Eventually you get to superintelligence. So we chopped it up into these levels. And each of these levels is a sort of separate intellectual question of like, how long do we have until some company builds a system that qualifies as a superhuman coder? And then separately, how long do we have from that point until some company has a system that qualifies as a superhuman AI researcher? And then separately from that point, et cetera. Right. So to answer your question, I sort of have to sort of talk about all those things separately, but I won't bore you with all of that, you should go read the actual thing. But I'll try to give a lightning summary for the superhuman coder milestone there. It's like, well, that's the first, that's the closest milestone, and it's the thing we have the most evidence for to try to guess at. We're still not confident in it, but almost around the same time that we published AI 2027, some benchmarks came out that we like quite a lot, such as those produced by meter. And then also OpenAI has a paper replication benchmark. Basically companies and nonprofits and so forth are trying to make, are starting to make these benchmarks that are not like traditional benchmarks which are basically multiple choice questions, but are instead like you plug the AI into some set of GPUs and you give it Internet access and you tell it you have eight hours to make progress on this engineering problem. And then it just interacts with the GPUs and runs experiments on them and basically does the whole loop by itself. So they're starting to get benchmarks that are with those sorts of relatively challenging tasks. And those benchmarks I think are like starting to get roughly in the vicinity of what we would actually want to be measuring for the superhuman coder milestone. It's still not quite there yet. The first system that crushes these benchmarks will still not be a superhuman coder, probably because they're only eight hour long tasks and they're relatively well scoped. But it's moving pretty fast in terms of performance on these benchmarks. Every couple months, the AIs are getting new state of the art performance on these benchmarks. And if you extrapolate the trends, it looks like in a year or two they'll be crushing these benchmarks. And so they'll be routinely able to do reliably 8 hour long coding tasks fully autonomously. So it's that sort of evidence that we point to, and you can read about it on the website, that's the sort of evidence that we point to for why we think the superhuman coder milestone could arrive as early as 2027. Our actual credence distribution, of course, is still pretty spread out. So we think maybe it's going to go a lot faster than we think could happen by the end of the year, but probably not, you know, maybe it'll take longer than we think and it'll be, you know, 2029, 2030, 2031 before we get the superhuman coder. But I think my 50% mark is like early 2028. And I think Eli has like a different 50% mark, but not that different or something. Yeah, so that's all our expression of uncertainty about the first milestone. And then for the second milestone it's like, well, we also have uncertainty about that. And you can read about this on the website. We have our takeoff speeds research page that talks about how we quantified our uncertainty there. So you can go read about that way. But there's just a lot of uncertainty. But that's where it is.
- [31:22]
Ei Lifland
Well, and Eli, to give us a sense of this actual process, because I think folks will be fascinated to learn who is actually writing this report. Who were you all consulting? There is a ton about international relations going on here. There's obviously big technical questions, there's legal questions, there's political questions. Who was involved beyond the two of you? And how did you all try to build additional expertise into this process to try to make the narrative as compelling and accurate as possible?
- [31:54]
Thomas Larson
Yeah, so in terms of who is involved besides Daniel and me. So there's one other person, Thomas Larson, who is kind of a full time contributor to the report. He has experience, both the technical AI safety and some experience with AI policy. And then Scott Alexander mostly helped rewrite the report, but he also has thought a lot about AI over the last many years. He has a lot of experience with that as well. And then one other person, Romeo Dean was a contributor and in Romeo's case, he wrote the kind of compute and security supplements. So for example, he was thinking about the questions of would China be able to steal these, these weights and why. So we started actually for background, we started writing the scenario actually 15 months ago in January 2024. And I think Romeo, you know, he focused a lot on the compute and security experts. And I think over the course of the report he talked to a lot, a lot of the best, you know, experts in both of these fields and kind of became an expert himself, I would say, in many ways. And then in kind of the other aspects, you know, we kind of did our best, especially I'd say on the technical level. You know, we sort of, we sent out the draft to, you know, hundreds of people for feedback. We had multiple iterations of drafts.
- [33:04]
Daniel Cocatello
We probably, we got feedback from more than 100 people.
- [33:07]
Thomas Larson
Yeah, I think over 100 people commented on around 100 people comment on the last draft. And Definitely like over 100 people commented on at least one of our drafts. A lot of them were kind of coming from a technical background. Yeah, but we also, you know, had some, some People with policy. We tried to get some feedback. You know, we kind of. We had this sort of like, we weren't sure kind of how we wanted to write, for example, the slowdown ending, what the geopolitical relations should be like in that. So we sent out this document to some people who have some expertise in that to sort of inform that. And then the other thing I'd say is, yeah, so we also did a few kind of sessions with various experts where we would just kind of like, you know, have kind of a whiteboarding session about a particular aspect of the scenario. I mean, the other thing is these war games. So, you know, these kind of like tabletop exercises. We. We've run about 30 of them now, and, you know, they're with varying groups of people, you know, with varying levels of expertise and types of expertise. But we did cover, you know, a decent amount. So, you know, we had some people who have, you know, been in the government who are playing the US Government, for example. You know, some experts, you know, some technical experts were like, often playing, you know, the AIs, you know, some, some, some. We have a player that plays the AIs, and we have a player that plays, you know, the leading company. And anyway, so, you know, we kind of tried to sort of like, do as best as we could in terms of using, using those and using the expertise that came out of that to inform the scenario as well.
- [34:27]
Ei Lifland
Well, I'm. I'm thinking just as a side note, we should probably release a AI 2027 board game of sorts. You just. War game everywhere. I want to walk into an arcade bar and find this tabletop exercise, but we'll, we'll scheme that out later. Daniel, you've already shared this in a number of different forums, including getting coverage in the New York Times. Part of that write up was from Ali Fahardy, the chief executive of the Allen Institute for AI, and he had this to say about the report. Quote, I'm all for projections and forecasts, but this forecast doesn't seem to be grounded in scientific evidence or the reality of how things are evolving in AI. End quote. Not exactly a ringing endorsement. What's your response to Ali? You just walked through hundreds of comments, lots of roundtables. Why do you think he reached this conclusion that it wasn't grounded in scientific evidence? And what perhaps would you say he was missing in reaching that conclusion?
- [35:24]
Daniel Cocatello
Yeah, I mean, I'm glad he said he's all for scenario projections, I would say. Can you point to anything else that's remotely as good I think the answer is no. This is a very sort of under supplied thing that we're doing and that's why we're doing it. We want to see other people write their own counter scenarios and say why they think it's going to go like this instead of like this. And here's the reasons for it. That's like part of the hope that we have for this. You know, you can see our thing on the website, you can see our reasoning and our research behind it on the website. If you have objections to it, you can talk about them, you can message us. We actually have a bounty. We're going to be giving out small prizes, monetary prizes, a couple thousand dollars for the objections and bugs reports that we find most compelling. And we're also going to be giving out prizes to the alternative scenarios that people write that seem. Seem good to us. So, you know, we'll see.
- [36:24]
Ei Lifland
Yeah, well, hopefully he takes you up on that offer and writes his own alternative history.
- [36:29]
Daniel Cocatello
Yeah, again, I think it's important to mention that, like, these companies are trying to build super intelligence. It says so on their website. Like the CEOs talk about it. Like, who knows what the future is going to look like. But if we do get something like superintelligence, it's probably going to look crazy. There's a lot to think about and a lot's going to happen really fast. And not enough people are talking about this and not enough people are thinking about it. And a very small set of people are like, thinking about it specifically using the medium of actual concrete stories. And we're hoping to change that.
- [37:04]
Ei Lifland
And what I appreciate about your analysis as well is one of the key factors you're tracing throughout this story is what is the public's concern, what is the public's level of concern about AI, and by extension, what is the level of public awareness about this issue. And there was just a Pew poll that came out a few days ago showing a huge disparity in expert evaluations of AI and whether, whether it's going to be for good or for ill, and then the general public's own evaluation of AI and they're wildly different. And so I think having these more accessible stories and approaches that are admittedly, there are parts of the report where I had to, you know, rewind on my podcast, go back, do that mile again and listen back to it, but far more accessible than a lot of technical reports. So, Eli, can you share a little bit more about the reception you've had so far? Are you getting calls from your Old high school buddies saying, just read AI 2027. Can't wait to write my own fan fiction. What's the reception been like so far?
- [38:07]
Thomas Larson
Yeah, I think it's been overall great. Better than my expectations. Yeah. In terms of like old friends reaching out. I've had a few.
- [38:15]
Daniel Cocatello
When Eli says better than his expectations, that really means something. He made forecasts quantitatively beforehand about various metrics.
- [38:23]
Ei Lifland
Forecast on forecast. It's just forecast to the nth degree.
- [38:26]
Thomas Larson
This is great. Yeah, yeah, that is true. It did surpass my forecasts, I think in terms of like the Twitter views, it was like, you know, maybe 70, 75th percentile or something like that, which is pretty good. And I think overall, yeah, I've had, you know, a few, a few friends reach out. Generally pretty positive. I mean, unsurprisingly from friends, of course. But I think in terms of the reception of what I've seen, you know, I think the Twitter, for example, the discourse on Twitter has been overall, overall pretty good. We've been, we've been hearing as well, you know, good things from. It seems like, you know, it's generating a lot of discussion in, in lots of different places, like in, you know, for example, in the. Within the AI companies, within some, you know, government or government adjacent, you know, think tanks etc. And yeah, so overall I'm quite, I'm quite happy. You know, obviously not every comment has been positive, but that's to be expected and I think I've been quite hardened. One other thing I'll say is that I've been heartened to see, you know, people who, even if they strongly disagree with this scenario, some people have been saying, you know, they found it very valuable, you know, for example, Dean Ball who, you know, disagrees with us about sort of like at least the takeoff speeds, I think, and maybe also the timelines and some other things that he still find it very, very valuable to read through.
- [39:42]
Ei Lifland
Yeah, well. And Daniel, I'm coming to think of you of sorts as a Nate Silver of AI, you know, putting your reputation on the line, making some bold predictions. Is this going to become a habit from you? Are we going to see every two years a new AI 2029 and AI 2031? Is this something we should expect to be a recurring product?
- [40:05]
Daniel Cocatello
Plausibly. So we are going to have a team retreat in a few weeks and decide like what we're going to do next and we have a lot of exciting options. For example, we might turn the tabletop exercise into more of a thing as you were hoping for and Making it like our main product and making it really good instead of this sideshow. We did the tabletop exercise basically for ourselves to give us ideas for how to write the story, but it's turned out to be way more popular than we expected. And anyhow, so one idea is to lean into that. Another idea is to keep doing more of these things on a regular cadence, like once a year or something like that, because the evidence is going to keep rolling in, the arguments are going to keep becoming better and more advanced and more nuanced. And so we, we're going to be updating our beliefs and our sense of how the future is going to go is probably going to change substantially every year. And so we should keep writing about it. And, you know, so, yeah, there's tons of ideas for what we could be doing, but that's one of them.
- [40:57]
Ei Lifland
That's excellent. Well, before I let you all go, I'll turn to each of you for a sort of rapid, rapid question here. What's one thing you really want folks to take away from this report? Is it a sort of we have agency over how AI is going to go? We can see this slowdown or we can see this race where, as Daniel mentioned, we could have really, really bad outcomes or really, really good outcomes. Is it general awareness? What's one thing you hope people take away from this?
- [41:27]
Thomas Larson
Yeah, for sure. I mean, so obviously it depends on different people have kind of like different things they can do in terms of action. I think the main thing that I am excited about is people kind of like having more of a sense that, wow, this could, something like this crazy could actually happen. You know, maybe on this time scale, maybe on a longer time scale, who maybe even on a sort of time scale, as Daniel mentioned, But just understanding that exactly how important this topic is and really getting that on a gut level is I think, something that I think is important. And I hope that basically it spurs, you know, some direct action, but also, you know, some reaction of, wow, you know, there aren't, there really isn't enough going into this for its importance. You know, like, there should be more things like this scenario. You know, government should be investing much more in like, understanding what's going on, more like emergency preparedness. Daniel has, you know, written up some proposals with Dean, Dean Ball about transparency. So, like, you know, the public and the government has a better understanding of what's going on inside these AGI companies. So I think that's something that I'm hoping will at least be one reaction is people thinking, wow, this is really important. We need to invest more in various ways in kind of understanding the likelihood of this and what to do about it.
- [42:41]
Ei Lifland
Excellent. Daniel, how about yourself?
- [42:44]
Daniel Cocatello
I think it's important to mention that there isn't any one particular thing that we were hoping by its nature, this is a sort of comprehensive holistic scenario about how we think things might go and what people take away from it is going to be different for different people. Some people might come away from it being like, oh, wow. Yeah, I can totally see how superhuman AGI could happen. Actually, previously I thought that that was just complete sci fi, but now I can see a sort of like, step by step pathway to it. Wow. Other people might be like, oh, yeah, I already thought that. Obviously, like, I work at, you know, OpenAI, but I hadn't really taken into account the, like, power grab risk before. Like, I hadn't really realized, like, even if, like, alignment is not really an issue and we can easily control the AIs, there's this important political question of who gets to control the AIs. And like, currently it's to be like a CEO or something. So that's another example of something that someone might take away from it. Other people might have already been concerned about that AI, concentration of power, et cetera. But then they read the section on the alignment stuff and they're like, oh, yeah, these things are neural nets. I guess I feel dumb for not noting that before, but wow. Yeah, neural nets means we can't actually necessarily see what they're thinking. That means maybe they're not actually controlled. So it's going to be different for different people. And. And that's part of what's exciting about this, is that we're getting feedback from all these people rolling in, being like, oh, this footnote was really interesting, or, oh, I disagree with this part here. It's really sort of this broad push and it's going to be different for different people. There's not any one particular thing that we're targeting.
- [44:12]
Ei Lifland
Well, folks, we're going to have to leave it there. Be sure to check out your local library for a new section called AI Phi. I'm just kidding. That's my own coinage for this new category of AI development, but really exciting AI 2027 report. Daniel, Eli, thank you so much for joining. We'll have to leave it there until next time.
- [44:31]
Daniel Cocatello
Thanks, Kevin.
- [44:32]
Thomas Larson
Thank you.
- [44:36]
Ei Lifland
The lawfare podcast is produced in cooperation with the Brookings Institution. You can get ad free versions of this and other Lawfare podcasts by becoming a Lawfare material supporter at our website LawFairMedia. You'll also get access to special events and other content available only to our supporters. Please rate and review us wherever you get your podcasts. Look for our other podcasts, including Rational Security, Allies, the Aftermath, and Escalation. Our latest Lawfare Presents podcast series about the war in Ukraine. Check out our written work@lawfairmedia.org the podcast is edited by Jen Pacha. Our theme song is from Alibi Music. As always, thank you for listening.
- [45:28]
Hannah Berner
If you're anything like us, you love attention. And my favorite way to get all eyes on me is with next level shiny glossy hair.
- [45:36]
Ei Lifland
Which is why we're so excited to tell y'all about the new Lamellar Gloss.
- [45:40]
Hannah Berner
Collection from the girlies at Tresemme and Gigglers. We've got you too, because Tresemme partnered with us to bring you 1-800-gloss, a special bonus episode of Giggly Squad, where Hannah and I give advice on all things hair and giving gloss. Check out the episode and grab the Lamellar Gloss collection today because I'm officially declaring this spring gloss season.