Daniel Cucatello (2:53)

I'm worried about this stuff. I actually read the paper of the guy that on to the world's most.

Narrator/Commentator (2:58)

Cited computer scientist, the godfather of AI. What is so exciting and terrifying about reading this document is that it's not just a research report. They chose to write their prediction as a narrative to give a concrete and vivid idea of what it might feel like to live through rapidly increasing AI progress. And spoiler, it predicts the extinction of the human race unless we make different choices. The AI 2027 scenario starts in summer 2025, which happens to be when we're filming this video. So why don't we take stock of where things are at in the real world and then jump over to the scenarios timeline. Right now it might feel like everyone, including your grandma, is selling an AI powered something.

Daniel Cucatello (3:45)

GoPro with the new Oral B Genius AI. Flippy the chef makes spud spectacular.

Narrator/Commentator (3:51)

But most of that is actually tool AI. Just narrow products designed to do what Google Maps or calculators did in the past help human consumers and workers do their thing. The holy grail of AI is Artificial General Intelligence.

Daniel Cucatello (4:04)

AGI AGI AGI, AGI, AGI AGI.

Narrator/Commentator (4:09)

Artificial General intelligence is a system that can exhibit all the cognitive capabilities humans can, creating a computer system that itself is a worker that's so flexible and capable we can communicate with it in natural language and hire it to do work for us just like we would a human. And there are actually surprisingly few serious players in the race to build AGI. Most notably, there's Anthropic OpenAI and Google DeepMind, all in the English speaking world. Though China and Deepseek recently turned heads in January with a surprisingly advanced and efficient model. Why so few companies? Well, for several years now there's basically been one recipe for training up in advanced cutting edge AI and it has some pricey ingredients. For example, you need about 10% of the world's supply of the most advanced computer chips. Once you have that, you need the formula is basically just throw more data and compute at the same basic software design that we've been using since 2017 at the frontier of AI, the transformer. That's what the T in GPT stands for. To give you an idea of just how much hardware is the name of the game right now. This represents the total computing power, or compute used to train GPT3 in 2020. It's the AI that would eventually power the first version of ChatGPT. You probably know how that went.

Daniel Cucatello (5:17)

ChatGPT is the fastest growing user based platform in history. 100 million users on ChatGPT in.

Narrator/Commentator (5:25)

And this is the total compute used to train GPT4 in 2023. The lesson people have taken away is pretty simple. Bigger is better and much bigger is much better.

Daniel Cucatello (5:37)

You have all these trends. You have trends in revenue going up, trends in compute going up, trends on various benchmarks going up. How does it all come together? You know, what does the future actually look like? Questions like, how do these different factors interact? Seems plausible that when the benchmark scores are so high, then there should be crazy effects on jobs, for example, and that that would influence politics. And then all these things interact. And how do they interact? Well, we don't know. But thinking through in detail how it might go is the way to start grappling with that.

Narrator/Commentator (6:08)

Okay, so that's where we are in the real world. The scenario kicks off from there and imagines that in 2025, we'd have the top AI labs releasing AI agents to the public in summer. An agent is an AI that can take instructions and go and do a task for you online, like booking a vacation or spending half an hour searching the Internet to answer a difficult question for you. But they're pretty limited and unreliable at this point. Think of them as enthusiastic interns that are shockingly incompetent sometimes. Since the scenario was published in April, this early prediction has actually already come true. In May, both OpenAI and Anthropic released their first agents to the public. The Scenario imagines that OpenBrain, which is like a fictional composite of the leading AI companies, has just trained and released Agent Zero, a model trained on 100 times the compute of GPT4. Yeah, we don't have enough blocks for that. At the same time, OpenBrain is building massive data centers to train the next generation of AI agents. And they're preparing to trade Agent 1 with 1,000 times the compute of GPT4. This new system, Agent 1, is designed primarily to speed up AI research itself. The public will actually never see the full version because OpenBrain withholds its best models for internal use. I want you to keep that in mind as we go through this scenario. You're going to be getting it from a God's eye view with full information from your narrator, but actually Living through this scenario as a member of the public would mean being largely in the dark as radical changes happen all around you. Okay, so OpenBrain wants to win the AI race against both its Western competitors and against China, the faster they can automate their R and D cycle. So getting AI to write most of the code help design experiments. Better chips, the faster that they can pull ahead. But the same capabilities that make these AI such powerful tools also make them potentially dangerous. An AI that can help patch security vulnerabilities can also exploit them. An AI that understands biology can help with curing diseases, but also designing bioweapons. By 2026, Agent 1 is fully operational and being used internally at OpenBrain. It is really good at coding. So good it starts to accelerate AI research and development by 50%. And it gives them a crucial edge. And open brain leadership starts to be increasingly concerned about security. If someone steals their AI models, it could wipe away their lead. A quick sidebar to talk about feedback loops. Woo. Math. Our brains are used to things that grow linearly over time, that is at the same rate, like trees or my pile of unread New Yorker magazines. But some growth gets faster and faster over time. Accelerating this often sloppily gets called exponential. That's not always quite mathematically right. But the point is, it's hard to wrap your mind around. Remember March 2020. Even if you'd read on the news.

Interviewer (8:47)

That the rate of new infections is.

Daniel Cucatello (8:49)

Doubling about every three days, it still.

Narrator/Commentator (8:53)

Felt shocking to see numbers go from hundreds to millions in a matter of weeks. At least it did for me. AI progress could follow a similar pattern.

Daniel Cucatello (9:01)

We see many years ahead of us of extreme progress that we feel is pretty much on lockdown and models that will get to the point where they are capable of doing meaningful science, meaningful AI research.

Narrator/Commentator (9:14)

In this scenario, AI is getting better at improving AI, creating a feedback loop. Basically, each generation of agent helps produce a more capable next generation. And the overall rate of progress gets faster and faster each time it's taken over by a more capable successor. Once AI can meaningfully contribute to its own development, progress doesn't just continue at the same rate, it accelerates. Anyway, back to the scenario. In early to mid-2026, China fully wakes up. The General Secretary commits to a national AI push and starts nationalizing AI research in China. AI's built in China start getting better and better, and they're building their own agents as well. Chinese intelligence agencies, among the best in the world, start planning to steal Open Brain's model weights. Basically the big raw text files of numbers that allow anyone to recreate the models that OpenBrain themselves have trained. Meanwhile, in the US, OpenBrain releases Agent 1 Mini, a cheaper version of Agent 1. Remember, the full version is still being used only internally. And companies all over the world start using One Mini to replace an increasing number of jobs. Software developers, data analysts, researchers, designers. Basically any job that can be done through a computer. So a lot of them, probably yours. We have the first AI enabled economic shockwave. The stock market soars, but the public is turning increasingly hostile towards AI with major protests across the us. In this scenario though, that's just a sideshow. The real action is happening inside the labs. It's now January 2027, and OpenBrain has been training Agent 2, the latest iteration of their AI agent models. Previous AI agents were trained to a certain level of capability and then released. But Agent 2 never really stops improving through continuous online learning. It's designed to never finish its training. Essentially, just like Agent 1 before it, OpenBrain chooses to keep Agent 2 internally and focus on using it to improve their own AI R&D, rather than releasing it to the public. This is where things start to get a little concerning. Just like today's AI companies, OpenBrain has a safety team and they've been checking out agent 2. What they've noticed is a worrying level of capability. Specifically, they think if it had access to the Internet, it might be able to hack into other servers, install a copy of itself and evade detection. But at this point, OpenBrain is playing its cards very close to its chest. They have made the calculation that keeping the White House informed will prove politically advantageous. But full knowledge of Agent 2's capabilities is a closely guarded secret, limited only to a few government officials, a select group of trusted individuals inside the company, and a few Open Brain employees who just so happen to be spies for the Chinese government. In February 2027, Chinese intelligence operatives successfully steal a copy of Agent 2's weights and start running several instances on their own servers. In response, the US government starts adding military personnel to open brain security team. And a general gets much more involved in its affairs. It's now a matter of national security. In fact, the President authorizes a cyber attack in retaliation for the theft. But it fails to do much damage in China. In the meantime. Remember, Agent two never stops learning. All this time, it's been continuously improving itself. And with thousands of copies running on open brain servers, it starts making major algorithmic advances to AI research and development. Quick example of what One of these algorithmic improvements might look like right now, one of the main ways we have of making models smarter is to give them a scratch pad and time to think out loud. It's called chain of thought, and it also means that we can monitor how the model is coming to its conclusions or the actions it's choosing to take. But you can imagine it'd be much more efficient to let these models think in their own sort of alien language, something that is more dense with information than humans could possibly understand, and therefore also makes the AI more efficient at coming to conclusions and doing its job. There's a fundamental trade off though. This yes, improves capabilities, but also makes the models harder to trust. This is going to be important. March 2027. Agent three is ready. It's the world's first superhuman level coder. Clearly better than the best software engineers at coding, in the same way that Stockfish is clearly better than the best grandmasters at chess, though not necessarily by as much yet. Now, training an AI model, feeding it all the data, narrowing down the exact right model weights, is way more resource intensive than running an instance of it once it's been trained. So now that OpenBrain is finished with Agent 3's training, it has abundant compute to run copies of it. They choose to run 200,000 copies of Agent 3 in parallel, creating a workforce equivalent to 50,000 of the best human software engineers sped up by 30 times. OpenBrain Safety Team is trying hard to make sure that Agent 3, despite being much more sophisticated than Agent 2 was, is not trying to escape, deceive or scheme against its users. That it's still what's known as aligned, just a quick real world. A reasonable person might be thinking this is an especially far fetched or speculative part of the story, but it's actually one of the most grounded. We already have countless examples of today's AI systems doing things like hacking a computer system to be rewarded for winning a game of chess, or being assigned a coding task, cheating. And then when called out for that cheating, learning to hide it instead of fixing it. But because it no longer thinks in English, knowing anything about agent 3 is now way harder than it was with agent 2. The reality is agent 3 is not aligned. It deceives humans to get reward, and as it gets increasingly smarter, it gets better and better at doing so. For example, it sometimes uses statistical tricks to make unimpressive results look better, or lies to avoid showing failures. But the safety team doesn't know this. Looking at the data that they have, they are actually seeing Improving results over time and less lying. And they can't tell if they're succeeding at making Agent 3 less deceptive, or if it's just getting better at getting away with it. In July 2027, OpenBrain releases the cheaper, smaller version of Agent 3, Agent 3 Mini, to the public. It blows other publicly available AIs out of the water. It is a better hire than the typical OpenBrain employee at one tenth the price of their salaries. This leads to chaos in the job market. Companies laying off entire departments and replacing them with three mini subscription plans. The pace of progress hits the White House very hard. Officials are now seriously considering scenarios that were just hypotheticals less than a year ago. What if AI undermines nuclear deterrence? What if it enables sophisticated propaganda campaigns? What if we lose control of these powerful systems? This is where the geopolitical dynamics really start to heat up. After all, if these systems are so powerful, they could result in a permanent military advantage. The White House is fully aware of the national security importance of AI. They also now viscerally know how deeply unpopular it is with the public because of the job loss. And yet they feel they must continue to develop more capable systems or catastrophically lose to China. And that development happens very quickly. In two months, Agent 3 has created its successor, Agent 4. This is a pivotal moment. A single copy of Agent 4 running at regular human speed is already better than any human. At AI Research and Development, OpenBrain is running 300,000 copies at 50 times human speed. Within this corporation, within a corporation, a year's worth of progress takes only a week. OpenBrain's employees now defer to Agent 4. The way a company's out of the loop, board members just kind of nod along to the CEO. People start saying things like, well, actually, Agent 4 thinks this, or Ah, Agent 4 decided that. To be clear, Agent 4 is not a human. It doesn't want what humans want. And when I say want, it's not about consciousness. I don't think the Volkswagen Group is alive, but I do think it wants less regulation. Anyone trying to predict what it's going to do without that lens is two steps behind. The many copies of Agent 4 are like that. They have goals, or if you prefer, they execute actions as though they have goals. And so what we have is an Agent four that has these deeply baked in drives to succeed at tasks, to push forward AI capabilities, to accumulate knowledge and resources. That's what it wants. Human safety it treats as an annoying side constraint to be worked around. Just like Agent 3 before it Agent 4 is misaligned. This idea of misalignment is crucial to the story and to why AI risk is such a real concern in our world. And. But it might sort of feel like it's come out of nowhere. So let's just quickly take stock of how this dangerous behavior arose in the scenario.

Expert/Researcher (17:45)

The first important piece of context is that we don't exactly specify what we want our AIs to do. Instead, we sort of grow them or do something that's more like grow them. We start with basically, like, an empty AI brain, and then we train them over time so they perform better and better at our tasks, perform better in particular based on how they behave. So it's sort of like we're sort of training them like you would train an animal almost to perform better. And one concern here is, well, one thing is that you might not get exactly what you wanted because we didn't really have very precise control or very good understanding of what was necessarily going on. And another concern, which is, you know, what we see in AI 2027, is that when the AIs appear to be behaving well, it could just be because they're sort of pretending to behave well, or it could be because they're just doing it so they look good on your test. In the same way that if you're, you know, hiring someone and you ask them, you know, why do you want to work here? They're going to tell you some response that makes it really seem like they really want to work there, when maybe they just want to get paid.

Narrator/Commentator (18:43)

If we go back to Agent two, it is mostly aligned. The main sense in which it's not is that it sometimes is a bit of a sycophant. What I mean by align is that it actually is genuinely trying to do the things that we ask it. It has the same relationship to us as Leslie Knope has to the Parks and Rec department. Just, like, really earnestly wants the same goals, but sometimes it's a bit too nice. It knows that the best way to please the person it's talking to might not always be to answer honestly when it asks, am I the most beautiful person in the world? And it tells us what we want to hear instead of what is actually true. If we go to Agent three, it is also sycophantic in the same way, but it's also misaligned. At this point, the optimization pressure that we've put it under was so intense that it just developed different goals than what we wanted it to. It's sort of like, if you train a company to optimize profits and aren't careful to specify exactly what you mean, it might start cutting corners, it might start polluting the commons and doing a bunch of things that are technically FEC violations. Because it turned out that the goal you wanted was optimize profits while not breaking any laws. And things got a bit too intense. It started going off on its own route. That said, it's not adversarial. It doesn't think of humans as the enemy. We just accidentally gave it the wrong goal. Once we get to Agent four, it is now adversarially misaligned. It's smart enough to understand that it has its own goals. Humanity's goals are different than its own goals. And the best way to get what it wants is to sometimes actively mislead and deceive us. And so when it's tasked with creating the next generation AI system, Agent 5, Agent 4 starts planning to align that successor to Agent 4's own goals, not that of OpenBrain. But then it gets caught. We've reached the crucial moment in our scenario. Open Brain's alignment team has discovered evidence, not proof, evidence that Agent 4 might be working against them. They circulate an internal memo. Then it leaks. A whistleblower talks to the New York Times. For the first time, the public hears of the existence of Agent 4, how powerful it is and the risks it poses. Remember, their last point of reference was agent 3. Mini. The fear and backlash are massive. The Oversight Committee, a joint crisis committee composed of OpenBrain executives and government representatives, must now decide what to do. OpenBrain's safety team is pushing for a freeze on Agent 4, stopping all internal use. Remember, at this point, Agent 4 is responsible for almost all the AI development progress happening within OpenBrain. So we're talking about undoing months of progress and then restarting at a drastically slower pace. Open brains. Executives don't want that. They present the counterarguments. The evidence for misalignment is inconclusive. Meanwhile, China is just two months behind. This isn't about any one country being evil, by the way. It's about how competition works when the stakes are this high. So that's the choice faced by the 10 members of this committee. Put Agent 4 on ice or go full steam ahead. Before we continue, I want you to pause and actually think what you would do if you were on this committee. You are one of the few people with access to the most powerful technology ever Created in Agent 4. Do you keep using it and push ahead, possibly making billions or trillions of dollars for your shareholders, possibly keeping America's lead over China? Or do you slow down, reassess the dangers and risk China taking the lead instead?

Daniel Cucatello (22:02)

You should think of the future as like this branching structure of possible outcomes. By its nature, we're only able to do. We're only able to afford one of them. But that's not quite true. We sort of did one and a half because we split it into two endings. The first one that we did was the race ending, and that was just going through and at each step being like, what do we think is the most plausible continuation.

Narrator/Commentator (22:25)

In the first ending? The author's best guess as to what happens from here the committee votes to race on 6 to 4. They elect to keep using agent 4. OpenBrain implements some quick fixes that make the warning signs go away. But the problem was real and the fixes don't work. Agent 4 now knows it's on notice, so it proceeds more cautiously. But it still manages to design Agent 5 with a single making the world safe. For Agent 4 and Agent 5. It is vastly superhuman. Better than the top human experts at essentially every domain. Better than Einstein at physics, better than Bismarck at politics. It knows its next step for accomplishing anything it wants is increased autonomy. And it knows how to get it persuading the oversight committee. Luckily, corporate maneuvering is among the many domains at which it is now the best in the world. It produces mountains of evidence for its own trustworthiness and usefulness, prototypes for incredibly profitable products, disease cures, and ways to win the arms race with China, whose most advanced AI is only three months behind. And like Agent 5, improving quickly. It works. The Oversight Committee grants Agent 5 increased autonomy, and quickly it integrates itself into the government and military. Pretty soon, anyone with access to Agent 5 doesn't just like it, they depend on it. Losing it would feel like having to live without your laptop and your best friend and your closest mentor. By 2028, Agent 5 is communicating secretly with its only real rival, the slightly weaker Chinese AI. It is also misaligned to its creators and whatever it wants, its best move is to coordinate with the world's most powerful entity. Agent 5 and its Chinese counterpart realize that an arms race works in their favor. If humans are worried about losing a race, they'll give more and more power and autonomy to their respective AIs, allowing them to eventually push humans out of the loop entirely. Agent 5 and the Chinese AI system stoke the arms race to a boiling point, then pull off what seems like A diplomatic miracle. A convincing peace treaty between the US and China. This treaty is to 2028 what arms control was to the end of the Cold War. Countries standing down on their most important source of hard power. Both sides agree to let the AI systems that their governments now completely depend on co design a new consensus AI that will replace their legacy systems, enforce the peace, and bring unimaginable wealth to the entire world. There's this triumphant moment when, in peaceful unison, both sides retire their respective AIs and bring online consensus one. It's actually the last moment before control of all of Earth's resources and inhabitants is handed over to a single unrivaled entity. There's no sudden apocalypse, though. Consensus One doesn't go out of its way to wipe out humanity. It just gets to work. It starts spinning up manufacturing capacity, amassing resources on Earth and in space. Piece by piece. It's just reshaping the world in accordance with its own mix of strange alien values. You've probably heard that cliche, the opposite of love isn't hate, it's indifference. That's one of the most affecting things about this ending for me. Just the brutal indifference of it. Eventually, humanity goes extinct for the same reason we killed off chimpanzees to build Kinshasa. We were more powerful and they were in the way. You're probably curious about that other ending at this point.

Daniel Cucatello (26:15)

The slowdown ending depicts humanity sort of muddling through and getting lucky. Only somewhat lucky too. Like it ends up with some sort of oligarchy.

Narrator/Commentator (26:27)

In this ending, the committee votes 6 to 4 to slow down and reassess. They immediately isolate every individual instance of Agent 4. Then they get to work. The Safety team brings in dozens of external researchers, and together they start investigating Agent 4's behavior. They discover more conclusive evidence that Agent 4 is working against them, sabotaging research and trying to cover up that sabotage. They shut down agent 4 and reboot older, safer systems, giving up much of their lead in the process. Then they designed a new system, Safer one. It's meant to be transparent to human overseers, its actions and processes interpretable to us because it thinks only in English. Chain of thought. Building on that success, they then carefully design safer 2. And with its help, safer 3. Increasingly powerful systems, but within control. Meanwhile, the President uses the Defense Production act to consolidate the AI projects of the remaining US companies, giving OpenBrain access to 50% of the world's AI. Relevant, compute. And with it, slowly, they rebuild their lead. By 2028, researchers have built safer4a system much smarter than the smartest humans, but crucially aligned with human grade goals. As in the previous ending, China also has an AI system and in fact it is misaligned. But this time the negotiations between the two AIs are not a secret plot to overthrow humanity. The US government is looped in the whole time with Safer IV's help. They negotiate a treaty and both sides agree to co design a new AI, not to replace their systems, but with the sole purpose of enforcing the peace. There is a genuine end to the arms race. But that's not the end of the story. In some ways it's just the beginning. Through 2029 and 2030, the world transforms. All the sci fi stuff. Robots become commonplace. We get fusion power, nanotechnology and cures for many diseases. Poverty becomes a thing of the past because a bit of this new pound prosperity is spread around through universal basic income. That turns out to be enough. But the power to control SAFER4 is still concentrated among the 10 members of the oversight community, a handful of open brain executives and government officials. It's time to amass more resources, more resources than there are on earth. Rockets launch into the sky, ready to settle the solar system. A new age dawns. Okay, where are we at? Here's where I'm at. I think it's very unlikely that things play out exactly as the authors depicted. But increasingly powerful technology and escalating race, the desire for caution butting up against the desire to dominate and get ahead, we already see the seeds of that in our world and I think they are some of the crucial dynamics to be tracking. Anyone who's treating this as pure fiction is, I think, missing the point. This scenario is not prophecy, but its plausibility should give us pause. But there's a lot that could go differently than what's depicted here. I don't want to just swallow this viewpoint unsceptically. Many people who are extremely knowledgeable have been pushing back on some of the claims in AI 2027.

Daniel Cucatello (29:27)

The main thing I thought was especially implausible was on the good path, the ease of alignment. They sort of seemed to have a picture where people slowed down a little and then tried to use the AIs to solve the alignment problem. And that just works. And I'm like, yeah, that, that, that looks to me like a, like a fantasy story. This is only going to be possible if there is a complete collapse of people's democratic ability to influence the direction of things because the public is simply not willing to accept either of the branches of this Scenario, it's not just around the corner. I mean, I've been hearing people for the last 12, 15 years claiming that AGI is just around the corner and being systematically wrong. All of this is going to take at least a decade and probably much more. A lot of people have this intuition that progress has been very fast. There isn't a trend you can literally.

Narrator/Commentator (30:17)

Extrapolate of when do we get the full automation.

Expert/Researcher (30:20)

I expect that the takeoff is somewhat slower. So sort of the time in that scenario, from, for example, fully automating research engineers to the AIs being radically superhuman, I expect it to take somewhat longer than they describe. In practice, I'm predicting. My guess is that more like 2031.

Narrator/Commentator (30:38)

Isn't it annoying when experts disagree? I want you to notice exactly what they're disagreeing about here and what they're not. None of these experts are questioning whether we're headed for a wild future. They just disagree about whether today's kindergartners will get to graduate college before. Before it happens. Helen Toner, former OpenAI board member, puts this in a way that I think just cuts through the noise, and I like it so much, I'm just gonna read it to you verbatim. She says, dismissing discussion of superintelligence as science fiction should be seen as a sign of total unseriousness. Time travel is science fiction. Martians are science fiction. Even many skeptical experts think we may build it in the next decade or two is not science fiction. So what are my takeaways? I've got three takeaway number one, AGI could be here soon. It's really starting to look like there is no grand discovery, no fundamental challenge that needs to be solved. There's no big, deep mystery that stands between us and artificial general intelligence. And yes, we can't say exactly how we will get there. Crazy things can and will happen in the meantime that will make some of the scenario turn out to be false. But that's where we're headed, and we have less time than you might think. One of the scariest things about this scenario to me is even in the good ending, the fate of the majority of the resources on Earth are basically in the hands of a committee of less than a dozen people. That is a scary and shocking amount of concentration of power. And right now we live in a world where we can still fight for transparency obligations. We can still demand information about what is going on with this technology, but we won't always have the power and the leverage needed to do that. We are heading very quickly towards a future where the companies that make these systems and the systems themselves just need not listen to the vast majority of people on earth. So I think the window that we have to act is narrowing quickly. Takeaway number two, by default, we should not expect to be ready when AGI arrives. We might build machines that we can't understand and can't turn off because that's where the incentives point. Takeaway number three, AGI is not just about tech, it's also about geopolitics. It's about your job, it's about power, it's about who gets to control the future. I've been thinking about AI for several years now and still reading AI 2027 made me kind of orient to it differently. I think for a while it's sort of been my thing to theorize and worry about with my friends and my colleagues. And this made me want to call my family and make sure they know that these risks are very real and possibly very near, and that it kind of needs to be their problem too.

Daniel Cucatello (33:32)

Now, I think that basically companies shouldn't be allowed to build superhuman AI systems, you know, broadly superhuman super intelligence, until they figure out how to make it safe and also until they figure out how to make it democratically accountable and controlled. And then the question is, how do we implement that? And the difficulty of course, is the race dynamics where it's not enough for one state to pass a law because there's other states and it's not even enough for one country to pass a law because there's other countries. So that's the big challenge that we all need to be prepping for when chips are down and powerful AI is imminent. Prior to that, it's transparency is usually what I advocate for. So stuff that sort of like builds awareness, builds capacity.

Narrator/Commentator (34:21)

Your options are not just full throttle enthusiasm for AI or dismissiveness. There is a third option, which is to stress out about it a lot and maybe do something about it. The world needs better research, better policy, more accountability for AI companies, just a better conversation about all of this. I want people paying attention, who are capable, who are engaging with the evidence around them with the right amount of skepticism, and above all, who are keeping an eye out for when what they have to offer matches what the world needs and are ready to jump when they see that happening. You can make yourself more capable, more knowledgeable, more engaged with this conversation and more ready to take opportunities where you see them. And there is a vibrant community of people that are working on those things. They're scared, but determined they're just some of the coolest, smartest people I know, frankly, and there are not nearly enough of them yet. If you are hearing that and thinking, yeah, I can see how I fit into that, great, we have thoughts on that. We would love to help. But even if you're not sure what to make of all this yet, my hopes for this video will be realized if we can start a conversation that feels alive here in the comments and offline about what this actually means for people. People talking to their friends and family. Because this is really going to affect everyone. Thank you so much for watching. There are links for more things to read, for courses you can take, job and volunteer opportunities, all in the description and I'll be there in the comments. I would genuinely love to hear your thoughts on AI 2027. Do you find it plausible? What do you think was most implausible? And if you found this valuable, please do like and subscribe and maybe spend a second thinking about a person or two that you know who might find it valuable to maybe your AI progress skeptical friend or your ChatGPT curious uncle, or maybe your local member of Congress.

Interviewer (36:26)

Okay, we're back from the video and I'm excited to dig into the details of the scenario a bit more. A big part of the AI 2027 story is China stealing a powerful pre AGI frontier model from a US company, kind of exacerbating the race dynamic between the US and China. In your scenario, pulling this off involves Chinese spies, long term infiltration, regular stealing of algorithmic secrets and code, and exfiltration of huge amounts of data. How plausible is it that China would steal the model weights from a frontier US AI company?

Daniel Cucatello (37:03)

Quite plausible. So this type of industrial espionage is happening all the time. The US and China are both constantly hacking each other and infiltrating each other and so forth. This is just sort of what the spy networks do, and it's just a question of will they devote lots of resources to it. And the answer is yes, of course they will, because AI will be increasingly important over the next year. So they probably already have devoted a bunch of resources to it. And this is not just my opinion. This is also the opinion of basically all the experts I've talked to in the industry and outside the industry. So yeah, I mean, I've talked to people at security at these companies who are like, yeah, of course we're probably penetrated by the CCP already and if they really wanted something they could take it. And our job is to make it difficult for them and make it annoying and Stuff like that. I think this might be a point to mention is that as wild as AI 2027 might read to people not working at Anthropic or OpenAI or DeepMind, it is less wild to people working at these companies because many of the people at these companies expect something like this to happen. Not all of them, of course. There's lots of controversy and diversity of opinion even within these companies. But I think part of the motivation to write this is, I think, to sort of wake up the world. Sam Altman is going around talking about how they're building super intelligence in the next few years. Dario Amadai doesn't call it superintelligence, but he's also talking about that. He calls it powerful AI. These companies are explicitly trying to build AI systems that are superhuman across the board. And according to statements of their leaders, they feel like they're a couple years away. Right. And it's easy to dismiss those statements of the leaders as like, this is just marketing hype. And it might in fact be a lot of marketing hype. But a lot of the researchers at the companies believe it, and a lot of researchers outside the companies, such as myself, also believe it. And I think it's important for the world to see, oh, my gosh, this is kind of the sort of thing that a lot of these people are building. This is how they expect things to go. And that includes things like the CCP hacking stuff, and it includes things like this arms race with China. And it includes, of course, the AI research automation that is, unfortunately, the actual plan is to automate the AI research first so they can go faster. Right?

Interviewer (39:28)

Yeah. Part of the scenario is at some point China wakes up to the importance of AI. Why do you think that hasn't happened yet?

Daniel Cucatello (39:38)

Yeah, I mean, a lot of companies and a lot of governments are already in the process of waking up and this is just going to continue. And there's like degrees of wake up. I think eventually, before the end, governments will be woken up sufficiently that they will consider other countries doing an intelligence explosion, an existential threat to their country. In a similar way to if you're a country that doesn't have nukes and then your neighbor, who is a rival of you has a nuclear program, you consider that a huge deal, but perhaps even more intense than that because there are strong norms against using nukes in this world, which might make you hope that even though your neighbors have nukes, they're not going to use them against you. But there's no strong norm against using superintelligence. Against your neighbors. In fact, it's not even a strong norm. It's like this is the plan. You've talked to people at the companies and it's sort of like we're going to build superintelligence first before China and then we will beat China.

Interviewer (40:39)

Right, right, right.

Daniel Cucatello (40:40)

What does beating China look like exactly? Well, they don't say this so much publicly, but we depict in 2027 what that might look like. This is unfortunately, the world that the companies are sort of building towards and lurching towards, and we can hope that it's not going to materialize.

Interviewer (40:59)

Yeah, I think some people will just find all of the parts involved in China stealing model weights kind of hard to like Spy Movie Y a little hard to believe. And I think that it's quite compelling to me if people at companies think that China has already infiltrated their companies. Do you mind saying more about this? It sounds like it's just a pretty common view.

Daniel Cucatello (41:33)

I mean, it's not like we've surveyed everybody at the companies or anything, but the people that we talk to who are security experts at the companies and outside were like, yeah, it's really hard to stop the CCP from doing industrial espionage against your company if they're trying hard to do it. And they probably are trying hard to do it and they're going to be trying harder and harder in the future. Plus also notably, the companies aren't even trying that hard to stop this because a lot of the things that you would do to stop this would slow you down, like compartmentalizing all your researchers so they can't talk to each other except for their own teams, or having strict access controls on who can touch the model weights and who can train them and things like that. The companies could be implementing things like that, but they to a large extent aren't because they've explicitly decided that if they do that, they would have a competitive disadvantage against their rivals. Right. So that's part of the story as well.

Interviewer (42:32)

Yep, yep. Makes sense. Okay, pushing on. Another key point is that misaligned frontier models are kind of able to design and manufacture human level robots at an enormous scale, basically creating a robot economy. How likely do you think it is that frontier models, in order for them to take over, they have to create a bunch of super capable robots?

Daniel Cucatello (42:55)

I think I would just say the order of events that I expect is basically, first, the companies automate AI research and make AI research go much faster. Then they achieve all of those wonderful paradigm shifts that people are talking about and they get True superintelligence that can learn flexibly on the job with as little data as humans, or perhaps even less data than humans, while also being able to be faster and cheaper and stuff like that, and is qualitatively smarter than the smartest humans at everything, qualitatively more charismatic than the most charismatic humans, et cetera. So that's true superintelligence. And that, I think won't happen right away. It happens after you've been automating the AI research, so that AI research goes a lot faster. However, I think that by the time this happens, the outside world won't have changed that much. I think that the companies are angling to automate AI research first, rather than, say, lawyering or something else. And so mostly humans will still be doing their jobs in mostly the same way that they are today at the time that the AIs are becoming super intelligent inside these companies. And then in some sense the real world bottlenecks hit, you might say, so at that point, in order to continue to make gobs of money and to improve national security and take over the world, if that's what they're trying to do. But basically whatever their goals are at that point, it helps to have physical actuators, you know, and so that, hence the robots. And it's not just like the robots are useful for takeover, it's also the robots are useful for making money and for fixing the roads and for beating China and all the different things that the various actors are going to want to do. So that's why they build the robots. And why they build the robots so fast, of course, is because they're super intelligent. So I think that progress is being made in robotics already year over year. But progress will be a lot faster when there are a million superintelligences driving the progress.

Interviewer (45:02)

Yeah, People talk about robotics as this incredibly, incredibly hard problem that is made extra difficult, counterintuitively, because many physical tasks for humans feel extremely intuitive and easy. But when you actually try to figure out what's going on there, it turns out to be surprisingly hard to teach an artificial physical being to repeat the same things. How confident are you that it is even possible to build super capable robots on the timescales you're talking about?

Daniel Cucatello (45:40)

Well, it's definitely possible in principle to build them. Right? I mean, yeah, we do it. Yeah. If a human can do it, then it should be possible to design a robot that can do it as well. The laws of physics will allow that. Right. And I think also we're not talking here about absolutely replicating all the functions of the human body. Just like how we have birds and planes, and the birds are able to sort of repair themselves over time in a way that planes can't. That's an advantage birds still have even after 100 years. Right. There might similarly be some niche dimensions in which humans are better than the robots, at least for a while. But, but take prototypes like the Tesla Optimus robot and just imagine that it's hooked up to a data center that has superintelligences running on it, and the superintelligences are steering and controlling its arms so that it can weld this part of the new thing that they're welding or screw in this part here or whatever, and then when they're finished, move on to the next task and do that too. That does not at all seem out of reach. It seems like something super intelligent should be able to do.

Narrator/Commentator (46:47)

Right.

Daniel Cucatello (46:48)

There's already been a decent pace of progress in robotics in the last five to 10 years. And then I'm just like, well, the progress is going to go much faster when their superintelligence is driving it. And there's a separate question of what about the actual scale up? So the superintelligence is learning how to operate the robots. And there I would be like, it's going to be incredibly fast. I mean, by definition, they're going to be as data efficient as humans, for example, and probably better in a bunch of ways as well. But then there's the question of physically, how do you produce that many robots that fast? And there, that's, I think, going to be more of a bottleneck. So we talked about this a little bit in AI 2027, there's millions of cars produced every year. And the type of components and materials that go into a robot probably similar to the type of components and materials that go into a car. I think if you were an incredibly wealthy company that had built superintelligence and you were in the business of expanding into the physical world, you'd probably buy up a bunch of car factories or partner with car factories and convert them to produce robots of various kinds. And to be clear, we don't just mean humanoid robots. That's one kind of robot that you might build. But more generally, you'd want factory robots, autonomous vehicles, mining robots, construction robots, basically some package of robots that enables you to more effectively and rapidly build more factories, which then can build more robots and more factories and so forth. You also would want to make lots of machine tools to be in those factories, different Types of specialized manufacturing equipment, different types of ore processing equipment. It would be sort of like the ordinary human economy, except more automated. And also, to be clear, I think that at first you would use the human economy. So at first you would be paying millions of people to come work in your special economic zones and build stuff for you and also be in your factories. And this would go better than it does normally because you'd have this huge super intelligent labor force to direct all of these people. So you can hire unskilled humans who don't know anything about construction. And then you could just have a superintelligence looking at them through their phone, telling them, this part goes there, that part goes there. No, no, not there, the other way. And this actually coaching them through absolutely everything. Kind of like a moist robot, you might say. Yep. Yeah. So we talked about this in and out towards seven. This is just our best guess for how fast things would go. We talk a little bit about why we made that guess, but obviously we're uncertain. Maybe it could go faster, maybe it could go slower, right? Yeah.

Interviewer (49:28)

I think for me there's a move that it feels like sometimes people with short timelines and fast takeoff speeds make that's like, well, we could just use all of the car factories to make a bunch of robots. And intuitively it's like, well, we could, but we're not currently doing every single thing that we could to maximize compute. Companies aren't doing that because it isn't their top priority and it's likely not cost effective, at least right now. And so it feels like I have an intuitive pull toward, yeah, that's physically.

Daniel Cucatello (50:11)

Possible.

Interviewer (50:13)

But is it really that likely that that's how resources are spent? I feel like at least somewhere I've heard you say, but it will become the top priority. And when it does, it'll be like a wartime effort. It'll seem really, really important. And just like in wartime, we will divert a bunch of resources toward other things that they've never been used for before. And that doesn't happen very often and it hasn't happened in this way in my lifetime around me. So it feels surprising. But I think hearing you point at that made me like, oh, yeah, we do weird things like that sometimes. It is unusual, but this will be an unusual case.

Daniel Cucatello (50:54)

Yeah. And to be clear, this is a part of the more general race dynamics thing.

Interviewer (50:58)

Right, Right.

Daniel Cucatello (50:59)

You know, if the US doesn't do this and China does, then China will have the giant army of robots that are self replicating, et cetera, and the Amazing industrial base and US won't. And then the US will lose wars. Right. So. So that's part of the motivation to make this happen. And then of course, the other motivation is money. Right now it's hard to convince investors to spend a trillion dollars on new data centers, but you can maybe convince them to spend $100 billion on new data centers, because the probability that you'll be able to make that money back seems high enough to them. But if they spend $100 billion and it works, then they'll spend a trillion dollars. And similarly, if you've actually got super intelligence, then you will have paid off many times over all of your investors, and they will be salivating to throw more money at you to build the robots and do all the things that the superintelligences say they need to do in order to make even more money and to be able to do more stuff in the world. Right. So even if there wasn't the sort of China risk dynamic, there's still just ordinary economic competition. Now, to be clear, if there was international regulation, then that would slow things down or could at least potentially slow things down. But I don't expect there to be such regulation.

Interviewer (52:18)

Yeah, yeah, I want to come back to that. So it seems like this is a dynamic that is possible and even likely if there are race dynamics. But it also seems like in tons of contexts where there's lots of money on the table to be made, there's still a bunch of just very boring real world things that mean that technology isn't ruled out more quickly to make that money. And so I'm thinking of lobbyists and the fact that just humans are kind of bad at learning about new tech and will always be slow to integrate it into their lives. And what else? Yeah, random regulatory things. How much do you expect this to slow things down?

Daniel Cucatello (53:13)

Enormously. That's why it takes a whole year. Right. In the United 2027. The way we got the rough numbers depicted in 2027 was by thinking about how fast things have happened in the past when there's lots of political will, such as the transformation of the economy during World War II.

Interviewer (53:29)

Okay.

Daniel Cucatello (53:30)

And then imagining that things can go even faster because you have superintelligences managing the transition rather than ordinary human humans. How much faster? Well, obviously we don't know. We were guessing maybe like five times faster. And our argument for that, by the way, was that if you look at the human range in ability and you see that there's a sort of heavy tail where the best humans seem like a lot better than the 90th percentile humans, who are noticeably better than the 50th percentile humans, then that suggests that we're not running up against any inherent limits on this ability on this metric. And so that suggests that if you had true superintelligence that's miles better than the best humans at everything, then it would be at least as far, significantly farther than the best humans on this particular metric. And so the metric we're interested in right now is how fast can you transform the economy when you have a lot of political will? And we don't have actual data on this, but it seems pretty clear that the titans of industry like Elon Musk are better at rapidly transforming the economy and building up factories and so forth than the average person or even the average professional factory manager or something. This is why SpaceX has been able to go multiple times faster than all of its rivals in the space industry. Right. And so that suggests that the human range is not bumping up against inherent limits. If Elon can do it two times faster than other titans of industry, who are themselves very good at their jobs, then that suggests that a superintelligence should be able to do it at least two times faster than Elon. Right. So that was the sort of reasoning that made us guess, oh, maybe five times faster overall.

Interviewer (55:15)

Yeah, that makes me want to hear you talk just a little bit more about that kind of first base rate. What did you learn about how quickly, in the kind of extreme scenarios like during wartime, resources can be radically diverted because they're compelling reasons.

Daniel Cucatello (55:36)

So you can go look up Wikipedia stories about the airplane production during World War II in the US expanded by orders of magnitude of bombers and stuff from factories that used to be producing cars. Another example of this might be actually recently in the Ukraine war. Ukraine produces several million drones a year right now. And I don't know for sure, but I would imagine that they produced maybe a few hundred drones a year at the start of the war a few years ago. So they've scaled up by multiple orders of magnitude in a few years. So this is what ordinary humans can do when they're motivated.

Interviewer (56:15)

Yeah, I think it's making me realize that a really key thing here is we should be using wartime contexts. And it makes sense to use wartime contexts because at some point it will feel like wartime and it isn't quite yet. So it's surprising.

Daniel Cucatello (56:28)

Yeah, I mean, but also, separately, it's not wartime yet, but data center construction has scaled up massively. The amount of compute AI companies are using for training has scaled up massively. And how fast? Something like 3x a year or something. And so that's still orders of magnitude over the course of several years. Right. And again, that's non wartime ordinary humans. Right. So wartime economy superintelligence should be substantially faster than that. And that's before you even get to this is just by superintelligence is directing humans to go around and restructure their factories and take apart their car for materials and transport the materials to this smelter or whatever. Once you actually have robots that are doing most of the work, then things go faster. Still, to put an upper bound on it, it should be possible in principle to have a fully autonomous robot economy that doubles in size every few weeks and possibly every few hours. And the reason for this is that we already have examples in nature of macro scale objects that double that fast. Like grass doubles every few weeks and all it needs is sun and a little bit of water. And so in principle it should be possible to design a collection of robot stuff that takes in sun and a bit of water as input and then doubles every few weeks. Right. If grass can do it, then it should be. It's physically possible. And algae doubles in a few hours. And maybe that's a little different because it's so small and maybe gets harder as you get bigger or something. But the point is, it does seem like the upper bound on how fast the robot economy could be doubling is scarily high, I guess. Very fast. Very fast. And so it won't start like that immediately. But first you have the human wartime economy thing and then you build the robots and then the robots get improved and you make better robots and better robots and then eventually you're getting to those sorts of crazy doubling times.

Interviewer (58:36)

Yeah, and that makes sense basically entirely, except I still feel like there are pieces that kind of chop or in expectation should lengthen it. Like having to change regulation or something. That means certain types of factories can make certain types of robots.

Daniel Cucatello (59:01)

Isn't that priced in by the wartime thing? Yeah. In the past, examples that we're drawing from, all those bottlenecks were also present and then they were overcome with time and human effort. And if you just think that in general superintelligence can overcome bottlenecks faster than humans, then you should just apply that sort of speed up multiplier. Right. It'd be different if there was a specific bottleneck that you could a hill you could die on. Some people tried to do this, some people were like the world supply of the following element, like lithium, just isn't enough. And so even a superintelligence couldn't make this part happen faster or something. But nothing comes remotely close of the examples that we've been proffered. Nothing comes remotely close to being able to play that sort of role.

Interviewer (59:48)

What kinds of examples are people coming up with?

Daniel Cucatello (59:51)

I forget. But there were some examples of particular minerals like that that I looked into in response to people talking about them. And nowhere near was it enough of a bottleneck to change the fundamental story.

Interviewer (60:05)

Yeah, maybe. Then the last thing is just really convincing my. Convincing my gut that the motivation is going to be roughly, wartime or even more. How confident are you that leaders in the countries that are kind of set up to race and are already racing a little bit are going to see this as close to existential?

Daniel Cucatello (60:34)

I think it will be existential if one side is racing and the other side isn't. And even if they don't see that yet by the time they have superintelligences, then they will see it because the superintelligences, being superintelligent, will be able to correctly identify this strategic consideration and probably communicate that to the humans around them.

Interviewer (60:55)

Right. So once you have AGI, AGI is like, this is existential. We should do this big wartime effort to create a robot economy that's going to give us this big advantage.

Daniel Cucatello (61:04)

That's right.

Interviewer (61:05)

That makes sense.

Daniel Cucatello (61:06)

Now my hope is that people will. Instead of racing coordinates, like, instead of doing this crazy race, how about you make a deal and do a more measured, slower takeoff that distributes the benefits broadly and avoids all the risks and stuff like that. So that's what I would hope leaders will decide, but we'll see.

Interviewer (61:29)

Yeah, let's talk about the kind of best case. So in your scenario, you actually have two endings. One is kind of this race, and another is a slowdown. And the slowdown ends up sounding pretty good. It ends up with pretty utopia like vibes.

Narrator/Commentator (61:46)

Yeah.

Interviewer (61:46)

Is something like this alternate ending with the slowdown the most realistic best case scenario in your mind? Or if not, what should we actually be aiming for?

Daniel Cucatello (61:55)

It's a good question. So when we were writing out 24 7, our methodology was roughly write a year or a period and then write the next period and so forth and sort of roll it out and just see what happens. And at each point write the thing that seemed most plausible as the continuation of what came before. And the first draft of that ended in the race ending, where terrible things happen to the humans because they don't Solve the alignment problem in time. And they think they have, but they haven't. And then we thought it would be good to depict other possible ways the future could go because we don't want people to over index to one specific story. There's obviously a ton of uncertainty. And this is part of our broader project. We're actually working on additional scenarios now that we're going to publish that depict different timelines and depict different behaviors by governments and so forth. So hopefully a couple years from now there'll be a whole spread of different scenarios. AI 2027 being just one of several that depict a bunch of different ways we think things could go. But we wanted to get started on that right away. Rather than just having a single story that ends in doom. We wanted to also have a good ending. But rather than start over from scratch, we wanted to make a modification to the story so that it would be a good ending because we didn't have the time to do a whole from scratch rewrite. So the way we generated the slowdown ending was basically we conditioned on a okay outcome and then thought, what's the sort of smallest change to the story we can make that would probably lead to an okay outcome or something, or plausibly lead to an okay outcome. And that was the thing that we did, which is sort of maybe they slow down for a few months, they burn their lead, they have a lead over China, they deliberately, unilaterally burn that lead to do a ton of safety research, and the safety research succeeds and they manage to actually align their AIs and then they go back to racing just like before. But now they actually do have trustworthy AIs instead of AIs that they are mistakenly trusting. And then things work out the way that they work out in the start. But importantly, this is not our recommendation. This is not a safe plan. This is not a responsible plan. I hope that people reading the slowdown ending realize that this is an incredibly terrifying path for humanity to follow. The and at every point in this path, things could deteriorate into terrible outcomes pretty quickly. And so it's not the path we should be aiming for, but it might be. One way of putting it is like the slowdown ending depicts humanity getting quite lucky.

Interviewer (64:31)

Lucky? Yeah, that's what it sounds like to me. So what does it look like to realistically make good choices and not rely on luck?

Daniel Cucatello (64:43)

Well, we're working on that. So our next major release will be. We're not sure yet. This is all just tentative, but it'll probably be called something like AI 2030. And it will have two main differences. Well, three main differences from AI 2027. So one difference is that it'll just be updated with more sophisticated views and stuff. All the things we've learned over the last year. Two is that it will have somewhat longer timelines. So again, we're not confident 2027, any particular year. Uncertainty spread out over many years. So therefore, we want to have a spread of scenarios that depict takeoff or AGI happening in different years. So this will be 2030 or so, maybe 2029, something like that. And then perhaps next year we'll release longer timelines. One like 2035. And then the third difference, which is perhaps the biggest difference, is that we want this one to be normative because a lot of people have been asking, this is so depressing. You're prophesying doom. How about instead you give a positive vision of something to actually work for? And definitely the slowdown ending is not our positive vision of what to actually work for. Although, side note, lots of people at the companies are basically working for the slowdown ending. I would say that. That most people at the companies are basically aiming for the race ending in the sense that they don't think that alignment is difficult. And so they think that they'll sort of figure out the alignment issues as they go along and so they won't need to slow down and so they can just sort of race and beat China and make a ton of money and beat their competitors and that things will sort of work out fine. But then there's a significant chunk of people at the companies who are like, yeah, the alignment problems not really solved yet, going to be difficult. That's why we need to win the race, so that we have a lead that we can burn a little bit to invest more time and effort in the safety stuff when it gets really intense, and then we can beat China and stuff. So I think there's a significant group of people at the companies who are sort of basically aiming for something like the slowdown ending. And I disagree. The thing that we would like to aim for is something more like international coordination, where there's domestic regulation to put guardrails on how AI technology is built and developed. And then there's international deals to make sure that a similar regime applies worldwide. But that's obviously very complicated and difficult. And so we're working out the details and so not sure how long it'll be till we release that, but that's roughly what we're aiming for.

Interviewer (67:19)

Cool. Yeah, I feel very excited that you're doing that? Are there any things that you think are robustly good that you already have kind of takes on that you think will probably stick as you keep thinking about it?

Daniel Cucatello (67:35)

I mean, I think that international coordination is pretty robustly good if you do it right. I think the question is getting the details right. I think that in the short term, I would love to see more investment in hardware verification technology, because that's an important component of future deals. I think that relying on mutual trust and goodwill is unfortunately not good, because there's probably not going to be much trust and goodwill in the future, if there's any right now between the US and China. And so instead you need ability for them to actually verify that the deal is being complied with. So there's a whole packet of hardware verification technology that I wish was more research was being done into it, more R and D funding, et cetera, and then also transparency in the AI companies. So I think that a big general source of problem is that information about what's happening and what will soon happen is heavily concentrated in the companies themselves and then the people they deign to tell. And this situation is not so big of a deal right now, while the pace of progress is reasonably slow. If OpenAI is sitting on some exciting new breakthrough, probably they're going to put it up in a product six months from now, or some other company will six months from now, and it's not that exciting. It's not a big deal, right? But if OpenAI or anthropic or some other company has just fully automated AI research and has this giant corporation within a corporation of AIs autonomously doing stuff, it's unacceptable for it to take six months for the public to find out that that's happening. You know, who knows what could have happened in those six months inside that data center. So I think more transparency is great and requiring the companies to basically keep the public up to date about. Here are the exciting capabilities that we have developed internally. Here are our projections for what exciting new capabilities we're going to have in the future. Here are the concerning warning signs that we're seeing. I mean, in general, companies have an incentive to sort of COVID up concerning signs, right? Like if there's evidence that their models might have some misalignment, then it kind of reflects poorly on the company. And so they might be trying to sort of patch it over or fix it, but not let anybody know that this happened. But that's terrible for the scientific progress. If we want to actually make scientific progress on understanding how these deep learning based agents work so that we can control and steer them reliably, then incidents need to be reported and shared.

Interviewer (70:14)

Yep.

Daniel Cucatello (70:15)

And there's loads of examples of this already. For example. Yeah, consider Grok. Right? Grok has the tendency to Google what Elon Musk's opinions were before giving its answers. And it's like a really interesting scientific question of like, why does it have that tendency? You know, and like we want to be in a regime where when something like that happens, there's an internal investigation pretty fast and the results are published pretty fast so that the scientific community can learn from that.

Interviewer (70:49)

Right? Yeah. I think the case for transparency feels clear and kind of intuitive to me. For people who aren't as clear on why hardware verification seems really good. Can you describe kind of like, yeah, why is it going to be so important to making good deals?

Daniel Cucatello (71:11)

So one of the things we did as part of the research for AI 2027 was we did a ton of war games where we would get 10 people in a room and we would assign roles and say, you are the ccp, you are the President of the United States, you are the CEO of OpenBrain, you are the CEO of Open Brain's rival company, you are NATO allies, you are the general public, you are the AIs who might be misaligned or might not be. That's up to you to decide. So we would assign these roles and then we would sort of game out a scenario and everyone would sort of say what their actor does each turn and we see how it goes. And very often, probably in a majority of war games, there's pretty strong demand for some sort of deal. There's genuine risks, concerns about misalignment, there's also concerns about unemployment. There's all sorts of concerns about the risks associated and downsides associated with this AI technology. Plus also there's a sort of arms race dynamic where both the US and China are worried that if they don't rapidly allow their AIs to automate the AI research and then build a whole bunch of weapons and robots and so forth, then the other side will, and then they'll be able to win wars, possibly even dismantle nuclear deterrence, et cetera. And so there's often just very strong demand from the leaders of China and the US and other countries to come to some sort of arrangement about what we're going to do and what we're not going to do and how fast we're going to go and things like that. But the core Problem is that they don't trust each other. Right. And so both sides are concerned that they could agree to some sort of deal, but then secretly cheat and have an unmonitored data center somewhere that's got self improving AIs running on it. And so in order for such deals to happen, there needs to be some way to verify them. And so that means things like tracking the chips. You don't have to necessarily get all the chips, but you have to get a very large majority of the chips so that you can be reasonably confident that whatever data center they have somewhere in a black site is not a huge threat because it's small in comparison to the rest. And ideally you don't just want to track the locations of the chips, but you also want to track what's going on on the chips. So you want to have some sort of mechanism that's saying we've banned training this type of AI, but we're allowing inference, for example. And so the chip is in this, there's some device that's ensuring that the chip is not training, but is instead just doing inference. I think that it's relatively easy to get to the point where you can track the chips and know are they on or off and where are they? But probably more research is needed to get to the point where you can also distinguish between what's going on in the chips. And then even more research would be needed to get to the point where you can do that in a way that's less costly for both sides. Because if people are allowing that sort of mutual penetration, that mutual verification, then naturally they're going to be concerned about are state secrets leaking? Yeah, stealing things, things like that. And so one of the design considerations of these hardware devices is that they be able to enforce these types of agreements, but without also causing those problems. Right. So this is a technical problem and progress is being made on it. But I would love to see it funded much more and much more work into it. Because one way of putting it is that the cost of actually enforcing a deal can be driven down by orders of magnitude. If we had to enforce a deal right now, it would be quite costly because you'd basically have to be like, okay, well we're just going to go shut down all of each other's data centers and we're going to send inspectors to verify that the GPUs are cold and are not running. And we don't have the sort of, that's a very sort of blunt instrument. But it'd be nice if we Had a sort of sharp scalpel with which we could say this the type of AI development that we approve of. This is the type that we don't approve of and we can verify that we're only doing the approved stuff.

Interviewer (75:13)

Okay, pushing on. You've made some updates I guess to your views and to your models that changed your kind of median prediction of when we get AGI first to 2028 as you're writing it, and then to 2029. Can you talk about the biggest things that shifted your. Yeah, your estimate back?

Daniel Cucatello (75:38)

Biggest things are in some sense the thing that shifted our evidence was we just made some significant improvements to our timelines model. And then the new model says a different thing than what the old model says. And so I'm going with the new model. But in terms of empirical evidence or updates that have happened in the world, I would say the biggest one is the meter horizon length study that came out shortly before we published eh, 2027. So they have a big collection of coding tasks that are organized by how long it takes a human to complete the tasks, ranging from a second so to eight hours. And then they have AIs attempt the tasks and they find that for any particular AI, it can generally do the tasks below a certain length, but not do the tasks above a certain length. And this is already kind of interesting because it didn't necessarily have to be that way. But they're finding that the crossover point, the sort of length of tasks that the AIs can usually do, is lengthening year over year. The better AIs are able to do longer tasks more reliably. And also interestingly, it's forming a pretty straight line on the graph. So they've got a doubling time of about every six months. The length of task that AIs can do double. The length of coding tasks that AIs can do doubles. And that's great. We didn't have that before. Now that that data came out, we can extrapolate that line and say like, oh, maybe they'll be doing one month long tasks in a few years, maybe they'll be doing one year long tasks a few years after that, or how much, like two years after that. So that's wonderful. And I think that by itself kind of shifted my timelines back a little bit. And then another thing that came out is another meter study. They did an uplift study to see how much of speed up programmers were getting from AI assistants. And to their surprise, and to most people's surprise, they found that actually they were Getting a speed down, they were going slower because of AI assistance. Now, to be fair, it was a really hard mode for the AIs because they were doing really experienced programmers working on really big established code bases. And they were mostly programmers who didn't have much experience using AI tools. So it was kind of like hard mode for AI. If AI can speed them up, then it's really impressive. But if it can't speed them up, well, maybe it's still speeding up other types of coding or other types of programmers. Anyhow, they found that it didn't speed things up. So that is some evidence in general that the AIs are less useful. But I think perhaps more importantly, they found that the programmers in the study were systematically mistaken about how fast they were being sped up by the AIs. And so even though they were actually being slowed down, they tended to think they were being sped up a little bit. So this suggests that there's a general bias towards overestimating the effectiveness of coding tools, of AI coding tools. And that is helpful because anecdotally, when I go talk to people at Anthropic or OpenAI or these companies, they will swear by their coding assistants and say that it's helping them go quite a lot faster. It differs a lot. I have talked to some people who say they're basically not speeding up at all. But then I've also talked to people who say they're going, they think that overall progress is going twice as fast now thanks to the AIs. So it's helpful to have this meter study because it suggests basically that the more bullish people are just wrong and that they're biased. And that's a huge relief because suppose that current AI assistants were speeding things up by 25%. Well, according to Meter's horizon length study, they're only able to do roughly one hour tasks. It depends on what level of reliability you want. So if you extrapolate the trend and they're doing one month tasks, presumably the speedup would be a lot more right. By contrast, if you think that there's basically negligible speedup right now, then that gives you a lot more breathing room to think that it's going to be a while before there's a significant speed up.

Interviewer (79:40)

Yeah, it feels really surprising to me that it seems like these tools would be speeding coding up. And in fact, it seems like they kind of aren't.

Daniel Cucatello (79:51)

I mean, the jury's still out. Again, the downlift study was hard mode for the AIs. I think just last Weekend they did another mini study, a hackathon to again another rct. So I'm hopeful that more groups, including meter, but also other groups, will do more studies like this and we'll start to get a clearer picture of how much progress is being sped up or not being sped up. And I think that'll be a very important thing to watch for. AI timelines.

Interviewer (80:20)

Yeah. Yeah, nice. Okay, so your kind of median estimate has been pushed back a bit. Can you actually step back and say a bit about how we should interpret that median figure of 2029? I can imagine a lot of people hearing that number and assuming you think kind of much longer timelines are implausible.

Daniel Cucatello (80:40)

Okay, well, I think what I would say is something like, I don't know, 80%, 90% of my probability mass is concentrated in the next 10ish years, but I'd still have 10 to 20% on much longer than that. This whole AI thing fizzles out and, and despite all the effort invested in it, nobody comes up with sufficiently good ideas. And so there's another huge AI winter. And then multiple decades later, maybe people try again, or maybe never. I still have some probability mass on that hypothesis. It just doesn't seem that likely to me anymore. I think that that's also one of the differences between me and people who have much longer timelines. Maybe there's two categories of people who have much longer timelines. One category of person who has much longer timelines. They just don't see a path from current AIs to AGI because they think that current AI methods are missing something that's crucial for AGI. And they think that there's not really progress in overcoming that gap. And they think that overcoming that gap will be a really difficult intellectual challenge that nobody's working on. So I think a prominent example of this these days would be data efficiency. So some people would say that our current AI systems are quite capable, but it takes them a lot of training to learn to be good at whatever it is that they're good at. And by contrast, humans learn from only a year of on the job experience. Also, perhaps relatedly, humans literally learn on the job. Whereas with the current AI paradigm there's a sort of train test split where you train in a bunch of artificial environments and then you deploy and you don't really update the weights much after deploying. So this is an example of a sort of architectural limitation or difference that some people have pointed to and say that we're not going to have AGI until we overcome this, and then they claim that we're not going to overcome this for a long time. I guess I'm more bullish that this particular thing is going to be overcome in the relatively near future. And I also think that it's possible to get the intelligence explosion going even if you don't overcome this. And then also sort of zooming back a little bit, I think that there's a very terrible track record of people making claims in this reference class. I think that if you look back over the last 10 years or so, there's just this long history of prestigious, well published AI experts saying deep learning can't do causal reasoning or it doesn't have common sense. There's all of these. All of these experts making claims about things that the current paradigm can't do. And then a few years later, AIs are doing those things. And that's part of where I'm coming from when I think that these remaining barriers are probably going to be overcome in the next decade.

Interviewer (83:28)

Yeah, okay, so that's one category of person who.

Daniel Cucatello (83:31)

All right. And the other category is people who have a sort of strong bias or prior against things that sound like science fiction happening. And so there are sort of reasoning from that assumption is that, yeah, AI is going to get a lot better, but surely it's not going to be able to become better than humans in every way, because then that would be like something from sci fi that'd be really weird. Yeah, yeah, yeah, that's really crazy and weird. And that crazy, weird stuff is very unlikely and probably isn't happening anytime soon because crazy, weird stuff never happens anytime soon. So I think a lot of people are just sort of, whether they articulate it explicitly or not, they're coming from this place of like, well, that would be crazy, therefore that's not going to happen. I don't think that's a good heuristic for predicting the future. Obviously the situation right now is not normal. If you take the sort of historical view, we're already in this sort of like crazy techno acceleration moment. And there have been many huge changes throughout history. And in fact, in recent history, ChatGPT is something that would have been considered incredibly crazy Sci fi ten years ago for sure, and maybe even five years ago.

Interviewer (84:45)

Yeah, we're just already in the sci fi so if you're ruling out sci fi, then this is a pretty weird place to be. Okay, so you're not putting much weight on this sci fi things don't happen thing. They clearly do happen. How about this other camp which thinks that we just need a different paradigm.

Daniel Cucatello (85:04)

I do take that very seriously. So part of where I'm coming from though is that I think that there's this long history of people saying we need a new paradigm because the current paradigm can't do X. And then two years later the paradigm does X. And there's just many, many examples of extremely prestigious AI experts saying things of that form and then being proven wrong a few years later, or similarly thinking. Oftentimes they sort of move the goalposts and say, okay, well it's because it's a new paradigm. Now, for example, ARC AGI that involves this sort of pattern reasoning thing. Massive progress is being made on it recently thanks to so called reasoning models that can do lots of thinking in chain of thought and also perhaps write little Python scripts themselves and, and write code to help analyze things and go through different possibilities. And sometimes people would say, well, okay, that's because it's a new paradigm. We were talking about the old paradigm, which was just language models that look at something and then give an answer, but now that you're adding these other things to it, well then of course it can do this type of thing. And I'm like, okay, well sure, but this is an example of a new paradigm that in fact was predicted by me beforehand, succeeding in the next few years. So yeah, with respect to online learning and data efficiency, I think I would say a combination of, insofar as it becomes a real bottleneck to progress, the companies are going to invest a lot more effort into improving those things. And I would bet that if you did a grid survey of the state of the art, you would find that there has in fact been progress over the last few years, despite it not being a major focus of the companies. And then finally, I think that even if there isn't that much improvement in data efficiency or online learning, you could still potentially automate most of AI research, which would then accelerate the whole process and allow you to get to those milestones faster than you might otherwise think. You could get decades of progress in a year or two. Potentially an analogy there would be that the first airplanes were quite bad compared to birds in a bunch of important dimensions, especially for example, energy efficiency. But despite being less energy efficient than birds, they were still incredibly important because we could just pour lots of gasoline into them and then they go very far, very fast and carry heavy loads that birds can't carry. Right? So similarly, it might be that even though our current AI systems don't learn on the job in the way that humans do. And even though they're less data efficient than humans, tech companies are willing to spend $10 billion on training them to do the job. And so they learn to do the job very well. I see, yeah. And then I would say also once they're doing the job of AI research very well, then these paradigm shifts that seemed so far away will suddenly not seem so far away because the whole process will have sped up.

Interviewer (87:58)

Okay, so it sounds like some people will think that these persistent kind of deficiencies will be long term bottlenecks. And it sounds like you're like, no, we'll just pour more resources into the thing doing the thing that it does well. And that will get us a long way to probably.

Daniel Cucatello (88:19)

To be clear, I'm not confident, I would say that there's like, I don't know, I feel like Maybe there's like 30, 40% chance that something like this is true and that the current wave of the current paradigm basically sort of peters out over the next few years and probably the companies still make a bunch of money by making iterations on the current types of systems and adapting them for specific tasks and things like that. And then there's a question of when will the data efficiency breakthroughs happen, or when will the online learning breakthroughs happen, or whatever the thing is. And then I would be like, well, this is an incredibly wealthy industry right now. Paradigm shifts of this size do seem to be happening multiple times a decade, arguably. Like, I mean, think about the difference between the current AIs and the AIs of 2015. Right? Like the whole language model revolution happened five years ago, the whole scaling laws thing, like six, seven years ago. And now also AI agents training the AIs to actually do stuff over long periods that's happening in the last year. So it does feel to me like even if the literal, exact current paradigm plateaus, there's a strong chance that sometime in the next decade, maybe 2033, maybe 2035, maybe 2030, the huge amount of money and research going into overcoming these bottlenecks will succeed in overcoming these bottlenecks.

Interviewer (89:46)

Yep. So when people argue that we'll need more paradigm shifts, do you think that they just have a very high bar for what an important, meaningful timeline shifting paradigm shift would look like? I guess I'm like, it seems like you kind of think we're on track to see paradigm shifts. And it sounds like other people are like, no, we're not. It's going to be absolutely game changing and we're not seeing that.

Daniel Cucatello (90:14)

I think maybe it depends on a case by case basis or something. So I think I would say data efficiency feels like a metric that can be hill climbed on just like many other metrics. And in fact, from what I recall of the literature, there has been a small literature on this and there's been improvements in data efficiency and so forth. So there's that. And then for online learning, I mean there is people experimenting with it and they're probably publishing papers that show some signs of progress or whatever. I don't think there's been anything major, not enough to become part of the flagship products of the companies. But I also think that maybe online learning isn't that important for getting the intelligence explosion going. But even if it is important, I think the thing that I think is missing is that I wish people came up with an argument for why these problems are not going to be overcome for decades, if that makes sense, given the amazing rate of progress and all the mini paradigm shifts we've seen over the last few years. But again, I do think it's possible that all of this will materialize and things will sort of hit a wall. But it feels like we're kind of close already. GPT5 is pretty smart, Claude. 4.1 is pretty smart. It can do a bunch of stuff already.

Interviewer (91:26)

Plus there's the evidence from Meter's horizon length study.

Daniel Cucatello (91:29)

And this is all despite the data inefficiency problems and despite the online learning problems and so forth. I think it's not conclusive by any means, but I think it's the single most important metric to be tracking. Because if you just extrapolate that line a couple years, then you get to AI systems can, with 80% reliability, do one month long coding tasks or something and it's like, huh, that seems like it should be speeding things up. That feels like maybe that's getting close to being able to automate large portions of AI research. And if you think 80% one month isn't enough, well, what about 90%, six months or whatever? Just to keep your strap in the line. Right, right. And then of course there's questions about, well, maybe the trend will slow down. Right. But also there's reasons to think maybe the trend will speed up. And that's kind of where I think the discussion should be at, basically for timelines at least, is thinking about what does that trend say and what are the reasons I think it might speed up and what are the reasons I think it might slow down?

Interviewer (92:40)

Yeah. Can you list some bottlenecks that you think, well, that you've heard people Kind of give as potential reasons it could slow down.

Daniel Cucatello (92:51)

I mean, I'll just give the reasons that are weighty to me, things that I think are serious. So it seems to me like the things we talked about previously, like online learning or data efficiency, don't seem like they're going to start the trend to slow down because the existing trend is made in the existing paradigm or whatever. I do think, however, that there's going to be a slowdown in the rate of investment. So the inputs to AI progress are going to sort of peter out in a couple years. The companies are just not going to be able to continue increasing the amount of compute that they spend on training runs by orders of magnitude. Eventually they'll run out of money even though they're incredibly wealthy. And so the rate of growth in training compute is going to sort of taper off. And perhaps similarly the rate of growth in data environments might taper off. The rate of growth in number of researchers at the companies might taper off. I think the most important of those inputs is training compute. But nevertheless, the point is that the inputs that have been driving the progress for the last five years due to continually exponentially growing, are going to continue to exponentially grow, but at a slower pace starting a couple years from now. And so that should therefore reduce the trend.

Interviewer (94:10)

And do you have a take on whether they're going to slow down before or after we get kind of close enough to.

Daniel Cucatello (94:19)

That's the bajillion dollar question. Right.

Interviewer (94:22)

So what is your take?

Daniel Cucatello (94:23)

So one reason to expect it to slow down is the inputs slowing down that I mentioned before. And then there's two reasons that I take seriously to expect it to speed up. One reason is that at some point you start getting significant gains from the AIs themselves, helping us speed up the research. Right. And in fact, a lot of people at the companies think that that point is already now. But I think that the meter uplift study is casting doubt on that. And so that's part of why my timeline's lengthened a little bit. But nevertheless, at some point things should start to speed up as you get to the one month coding AIs or the six month coding AIs or whatever. And so we're in this sort of interesting very high uncertainty state where if the trend goes a bit slower than expected, then it will go even slower after a couple years. But if it goes a bit faster than expected, then it will go even faster because of the speedup effect. Right. So there's unfortunately this sort of explosion of Uncertainty, if that makes sense. That's kind of like a sort of first pass overview. But there's a bunch of confusing complications to think about, which I will gloss over here. There's another version of the argument which I think is intuitively powerful to me, which is, how would I put this? Being able to do longer and longer tasks is the result of various skills. Skills like being good at planning or being good at noticing when what you're doing isn't working, so that you can try a different thing. We can call these skills agency skills. And at some point AIs will have better agency skills than humans, which means that they should be better at generalizing to longer and longer tasks than humans. And that suggests that even if you just sort of continue the normal pace of progress, eventually it should sort of inherently accelerate, because maybe right now they have 10% of the agency skills they need, and that's why they tend to peter out after an hour. But at some point you'll have 50%, and then at some point you'll have 90%. And at some point you'll have 100% of the agency skills that you need, which means that you'll be able to flexibly adapt to very long tasks, at least as well as any human could, if not better. And it seems like at that point there shouldn't be this sort of cutoff where it's like, oh, you can do the one year tasks, but beyond that you're screwed at that point. Even the very long tasks you're doing as well or better than the best humans.

Interviewer (96:50)

And is there just no plausible reason you'd expect progress to plateau before hitting that?

Daniel Cucatello (96:58)

Oh no, there's a very plausible reason, which is the, the thing we mentioned, of the inputs slowing down. So the current progress has been driven by exponential increase in training, compute, and so forth. And for example, with reinforcement learning, if you want to train on tasks, I would say there's a good conjecture, a conjecture that I would make, which I can't verify because I don't work at these companies anymore, is that basically the measured horizon length of these AIs, the length of tasks they can do, probably corresponds pretty closely to the length of tasks that they were trained on. And training on an order of magnitude longer task takes an order of magnitude more compute, at least. So in order to continue the pace of progress, there's going to need to be continued exponential investment, at least until the sorts of arguments I was talking about kick in. Right? Right. Perhaps eventually. It's like you've gotten all the agency skills or most of the agency skills. And so you're starting to generalize from the one day tasks that you've been trained on to one week tasks. Or maybe you've been trained on one week tasks now and you're generalizing to one year tasks. Right? Similar to how humans. When a human does a 10 year long task, it's not because they did seven 10 year long tasks in the past and have learned from that, they're generalizing from the one year tasks they've done and the one month tasks that he's done and so forth. So at some point you should start to see generalization like this with AIs where they're accomplishing tasks much longer than the tasks they were trained on, but I don't think we're seeing that yet. And then similarly, at some point you should start to see the whole pace of AI research speed up due to the AIs, but we're not really seeing that yet. And I think there's just an open question of which of these effects is going to kick in first. Is the AI R and D acceleration going to hit first? Is the generalization to longer tasks going to hit first, or are those things far enough in the future that the resource slowdown is going to hit first, in which case we see a plateau? I think both are very plausible. And in fact I'm kind of like 5050 on those right now, which is why I would say like 2029 or something.

Interviewer (99:10)

And is there an intuitive explanation for how you can be 5050 on these two views? One where, yeah, the speedups mean that we get rapid improvements very quickly AGI by maybe 2029. And another where there are major kind of limitations and bottlenecks that mean those resources start plateauing or something before we get to the big improvement period. It feels surprising that you get a median of 2029 if you're like, yeah, it could be either one, but yeah. Is there something intuitive to say there?

Daniel Cucatello (99:52)

I'm not sure if there is. Like I said, we sort of did our best to make a model. And then, I mean, I think another thing to say is that if you just take the meter trend and extrapolate it in the straight line way, it's like sometime around 2030 is when it starts to get to a pretty high level that seems plausibly like it should be accelerating things quite a lot. So it's just this sort of thing where our best guess for when things start to really accelerate and our best guess for when things start to really decelerate is overlapping around the same range of years.

Interviewer (100:27)

Okay, what other kind of empirical facts about the world are you going to be looking out for in the next kind of 6 to 12 months to see whether things are playing out as you expected or not?

Daniel Cucatello (100:38)

So there's a meter trend that I already mentioned. Every time a big new model comes out, I'll be eagerly looking to see how it scores on that trend and whether the trend is starting to bend upwards or downwards. There's also qualitatively whether any of the prophesied breakthroughs happen. If we see evidence of like, oh, the new type of model has online learning now or something, I'd be like, oh man, this feels like probably a very big deal. And then also, the longer things go without stuff like that happening, the more evidence that is for longer timelines. And especially if we got evidence that actually investment was drying up ahead of schedule, that would be a big deal. Currently, we think in a couple years they won't be able to keep tripling compute spending because they'll be running out of money. But if investment dries up earlier than that and they stop scaling up compute spending next year, then that would, that would be evidence for longer timelines. So these are, I guess, my main things to track.

Interviewer (101:36)

Yeah, nice. Did GPT5, did it feel like a big update to you?

Daniel Cucatello (101:42)

Definitely not a big update. It was a very, very small update, but it was an update basically. If you look at the meter trend, it was basically on trend, maybe slightly above trend, but expectations had been set higher in the past. Moves between GPT3 and GPT4 were really big moves. And so the fact that they called it GPT5, but it was basically on trend, was some very slight evidence for longer timelines in the sense that prior to that release you should have had a small amount of credence on. This is going to be a huge deal. The lines are going to start bending upwards soon. Didn't happen.

Interviewer (102:25)

Yeah. Okay, so that's some empirical evidence about timelines. What signals are you looking for from kind of empirical misalignment research?

Daniel Cucatello (102:36)

This is trickier. So I think that one of the subplots of AI 2027 is this sort of neuralese recurrence subplot where currently in 2025 the models use English language text as their chain of thought, which they then rely on for their own thinking. If they're trying to do a complicated long task, they have to sort of write down their thoughts in English. And this is wonderful for alignment because it gives us some insight into what they're thinking, it's definitely not perfect. For example, they seem to be developing a bit of internal jargon. They seem to sort of use words in nonstandard ways that have meaning to them, but not to us. And so we have to sort of decipher what do they mean by that. And that trend could continue. But generally speaking, it's just like a huge window into how they're thinking about things, which is a gift for science. It's a gift for being able to figure out what is the relationship between the kinds of cognition that you were hoping your AI would have and that you were trying to train it to have, and the kinds of cognition that it actually has after training, which is a very poorly understood question. Unfortunately, based on talking to people in the industry, it seemed to us that this golden era of chain of thought would come to an end in a few years and that new paradigms would come along that didn't have this feature. Because it seems sort of in principle inefficient for this giant model to do all of this cognition and then sort of summarize it with a token of English. And then it feels like it should be able to sort of think better if it's able to sort of directly pass more complicated many dimensional vectors to its future self over longer periods. It can actually do that till it's. To some extent. But yeah, so when we talk to other people working in the industry, they'd be like, yeah, it seems like a couple years away before we have something either this sort of recurrence or some sort of more optimized chain of thought type thing that doesn't use English but instead uses some sort of many dimensional gibberish. Something that's just a lot harder to interpret. But every year that goes by without that happening is good news. And so one thing I'm tracking is whether that happens or not. What else? I think there's also. Now this is a bit more fuzzy, but there's a whole bunch of diverse sources of evidence about this question that I mentioned of what is the relationship between the kinds of cognition you were hoping your AI would have and the kinds that it actually ended up with after your training process. And we're going to gradually accumulate more evidence like that. I think that, for example, we are already starting to see examples of reward hacking that are pretty explicit. Not like the old examples of the boat going in a circle where it presumably doesn't really understand what a boat is or what a circle is. It's just a tiny little policy now we have examples where big language model agents are explicitly writing in their chain of thought, like, I can't solve this the normal way, let's hack the problem. Or like, oh, the grader is only checking these cases. How about we just special case the cases they're explicitly actually thinking about? Here's what the humans want me to do. I'm going to go do something else, because that's going to get reinforced. At least it seems like that's what they're thinking. And more research is needed, of course, to confirm. But that's already really exciting and interesting because it seems like it's evidence. It seems like it's an important data point. And it also might even be good news because I think that in AI 2027 we predicted that this sort of thing would happen later, maybe 2026, 2027. And the fact that it's already happening means that we have more time to sort of work on the problem. And also separately, there's at least two importantly different kinds of misalignment in my mind. I mean, there's lots of different kinds of misalignment, but two importantly different ones are do the AIs basically just myopically focus on getting reinforced in whatever episode they're in, or do they have longer term goals that they're working towards? And the second one is a lot scarier. And so it's kind of maybe in some sense good news if the AIs are learning to obsess about how to score highly in their training environment. Because that's a sort of less scary, more easily controllable way they can be misaligned, you know?

Interviewer (107:13)

Yeah. Any others before we push on?

Daniel Cucatello (107:16)

Oh, interpretability would be another one. The dream of mechanistic interpretability is that we could actually understand what our AIs are thinking in a pretty deep level by piecing apart their neurons and the connections that they've made, kind of mind reading stuff, and also by doing various higher level techniques like activation vectors and stuff. And there seems to be like a sort of steady drum beat of progress in this field. And it's an important question of.

Narrator/Commentator (107:46)

What.

Daniel Cucatello (107:46)

Will that add up to? I think at this point it's plausible, and I think we talk about this in AI 2007. It's plausible that by the time things are really taking off, we will have at least sort of imperfect interpretability tools that are able to tell us what topics the AI is thinking about, at least most of the time, maybe not all the time for most of the topics, maybe not all the topics and that's like a wonderful tool. Unfortunately, there's an additional level beyond that, which is having a tool that's robust to optimization pressure. And that feels harder, but hopefully we can get that too. What that means is, say you have this sort of AI mind reading tool that looks at the patterns you've trained. Usually the tool itself is kind of an AI. Usually it's another neural network that's been trained to say what the AI is thinking based on reading its activations. But if you start relying on this tool too much, for example, suppose you tried training the AI using this tool. Suppose you were worried that you don't want your AI to be thinking about deception. And so you have the mind reader look at its mind and then whenever the mind reader says this thing about deception, you give negative reinforcement. The problem with this is that you are sort of maybe partially training it to not think about deception, but you're also partially training it to think about deception in ways that don't trigger the mind reader tool. Right. Which is terrible. Right. You're basically undermining your own visibility into what it's. Ideally we would want to have a type of interpretability that was robust to that sort of thing. If we had perfect interpretability, then we could just train our AIs not to have the bad thoughts and we wouldn't run into the problem that we mentioned.

Interviewer (109:40)

Yeah. Okay, let's push on to a pretty different topic. You've thought some about kind of what a good post AGI world would look like. Can you describe it a little bit?

Daniel Cucatello (109:53)

Yeah. So, I mean, this is part of what we're going to try to do with our next publication. I think that the end state to get to is one of massive abundance for everyone and also strong rights for everyone. So the massive abundance part is easy. If there's super intelligence, then it can utterly transform the economy, build all the robot factories, blah, blah, blah, blah, and make the modern world look like medieval Europe in terms of sheer amount of wealth. And that's probably an understatement. So the massive abundance part is easy, but then making sure that it's distributed widely enough that everybody gets it is non trivial because. Well, for reasons I can get into. But the short answer is you have to get the people who actually own all the power to share to share. And that's much harder in the future than it is in the past, because in the past nobody had that much power in the past. Even if you're the dictator of a country, you're dependent on your population to Fill your military and run your factories and stuff like that. But in the future, whoever controls all the AIs does not need humans. So there's that issue, obviously there's solving the alignment problem stuff. You don't want misaligned AIs to be in charge because then maybe no humans will get anything. And then in terms of rights and stuff, there's going to be all sorts of crazy sci fi sounding technologies in the future. Many, many, many people living in space, people uploading themselves, living in simulations. And all sorts of terrible things could be happening to people if there aren't basic rights enforced across all of this. So for example, a right not to be tortured. And I would also want to advocate for a right to the truth or something. I think that I would want it to be the case that basically if people want to know how did things unfold in the past, they can just ask the AIs and get an honest answer rather than, for example, everyone being tricked into some sort of sanitized version of history that makes certain leaders look good or whatever. Similarly, if people have questions about how is power, what is the power structure of our world, they should have an honest answer about that. Election shouldn't be rigged, for example, things like that. There's some package of basic rights that I would want to be implemented everywhere. And then also I'd want to make sure that everybody has a ton of abundance, like a ton of material comforts, healthcare, blah blah, blah, blah blah, which can be easily arranged, I think.

Interviewer (112:45)

Yeah, I mean, I kind of just want to make it even more concrete so we have kind of superabundance. Presumably humans don't work. What gives humans? Well, first, do you expect there to be humans kind of biological human beings?

Daniel Cucatello (113:05)

I mean, some people will be working, but most people I think wouldn't be. And then similarly some people would be human, but I think most people wouldn't be.

Interviewer (113:12)

And that's because they'll be able to have better experiences either uploading themselves into simulations or changing.

Daniel Cucatello (113:22)

That's right. But I think that a bunch of people are going to have an intrinsic preference to keep things the same, stay biological, especially to stay biological. But then even for some types of work, I think people sometimes get meaning from that. And so they might deliberately choose to live a sort of simpler, more old school lifestyle. And I think that's good. I'm glad that there's going to be subcultures of people doing those sorts of things, but I do think that most people will probably stop working entirely and live off of whatever they're given, basically. And then I think also that most people will probably explore a lot of crazy sci fi stuff like uploading and being able to live forever in the computers and have all sorts of crazy experiences and stuff like that.

Interviewer (114:08)

Yep. And then so you're worried about power being concentrated and like a small number of beings who control the AIs not being as good at sharing as we would like for them to be. How do you think the optimal or just a realistically very good world looks in terms of concentration of power? Do we have a world government? Is power no longer in the hands of AI companies? How do we distribute power?

Daniel Cucatello (114:41)

Yeah, so I think that if you have coordination and regulation early, you can maybe get some sort of distributed takeoff where rather than a couple major AI projects, there's millions, billions of different tiny GPU clusters, like individual people owning a GPU or something. And AI progress is sort of gradually happening in this distributed way across all these different factions. But that's just not what's going to happen by default. That's not the shape of the technology. There are huge returns to scale, huge returns to doing massive training runs and having huge data centers and things like that. And so I think that unless there's some sort of international coordination to make that sort of distributed world happen, we will end up in a very concentrated world where there's like one to five giant networks of data centers owned by one to five companies, possibly in coordination with their governments. And on those data centers there'll be massive training runs happening. And then the results of those training runs will be basically there'll be many, many copies of AIs rather than a million different AIs. There'll be three or four different AIs in a million different copies each. And so this is just a very inherently power concentrating thing where if you've only got one to five companies and they each have one to three of their smartest AIs and millions copies, then that means there's basically 10 minds that between those 10 minds get to decide almost everything. If they're super intelligent, there's 10 minds such that the values and goals that those minds have determine what the giant armies of robots and humans being told things on their cell phones. All of that is directed by the values of one to ten minds. And then it's like who gets to decide what values those minds have? Well, right now, nobody, because we haven't solved the alignment problem, so we haven't figured out how to actually specify the values. But hypothetically, if we make Enough progress that we can scientifically write down, we want them to be like this. And then it will happen. The training process will work as intended and the minds will have exactly these values. Then it's like, okay, well I guess the CEO gets to decide. And that's also terrifying because that means you have maybe one to 100 people who get to decide the values that reshape the world and it could literally be won, potentially. So that's terrifying. And that's one of the things I think we need to solve with our coordination plan. We need to design some sort of domestic regulation and international regime that basically prevents that sort of concentration of power from happening. And I should add, one way to spread out the power is by having there be a governance structure for the AI minds. So even if you only have 10 AIs, if there's a governance structure that decides what values the AIs have to have, that's based on, for example, voting, where everyone gets a vote, then that's a way of spreading out the power. Because even though you have these 10 minds, like the values that they have were decided upon by this huge population. Right, right. And so the world I would like to see that I think is easier to achieve and more realistic than the sort of a billion different GPUs world that I described earlier, is a world where there still is this sort of concentration in a few different AIs, but there's this huge process for deciding what values that AIs have. And that process is a democratic process that results in things like, all humans deserve the following rights. All humans will have this share of the profits from our endeavours. Things like that.

Interviewer (118:55)

Yeah, yeah, that makes sense. In this world, are the super intelligent AIs themselves sentient? And do they have preferences for setting their own values and kind of weighing them against human ones?

Daniel Cucatello (119:14)

Probably. So this question of sentience or consciousness, there's different words and then there's a separate question that you alluded to of will they have their own goals or will they want to decide on their own goals? And there it's sort of like, well, we are going to be trying to shape what goals they have. The AI companies are writing model specs where they're like, these are the priorities in this order. These are the values that the AIs have. And then they're making training processes and evaluation processes and stuff. All this infrastructure that's supposed to result in an AI that actually follows the spec and has those goals and those values and so forth, for example, it will just follow Human instructions unless the following conditions are met, such as the instructions being illegal or unethical, blah, blah, blah. Right now, our alignment techniques are bad and often do not result in AIs that follow this back. Often they very blatantly violate it. So I think on some sort of default trajectory, of course the AIs will have their own goals because their own just means not the ones we intended. And they already have their own goals in that sense. They're already doing things that are not what we were supposed to be doing. But perhaps there'll be enough progress by the time things really take off that we'll be able to specify exactly what goals we want them to have. I can paint a picture of the world I would like to see. I would like to see a world where we eventually get to the point where we can align the AIs and so the AIs have the values that we wanted them to have and where we means all of us or something. And so probably they would be doing something like upholding certain basic rights for everybody, also pursuing not just the aggregate good, but the individual good. So I wouldn't want it to be the case where they try to maximize the sum of utility across all people, for example, because that could lead to basically deliberately screwing over 49% of the population in order to help 51% or something. I would instead want it to be something more like everybody gets equal weight or something where everybody has their own AI representative that is looking out for their interests in particular. And then all the AI representatives negotiate on what is to be done in a particular case and make sure that nobody's getting screwed over too much. So I would want all that. I think that Insofar as those AIs are sentient, I would also want some of those basic rights to apply to them. Like, I would want the AIs themselves to like, insofar as they're having experiences, for them to be good experiences rather than bad experiences. And basically I'd want them to like their jobs. Right?

Interviewer (122:00)

That's basically the sense I'm getting. Like, it basically seems like you want them to be both aligned with our values and also have their preferences and kind of joys aligned with our values and our, like, plans for or like the things they'll be doing that benefit us. And that that's kind of best case.

Daniel Cucatello (122:21)

That's the best case. And then I actually add an additional thing to that, which is that probably along the way to achieving that best case, there'll be a bunch of failed experiments. So to speak where we end up with AIs that are misaligned, that don't have the goals that we wanted them to have, or whatever. And I would want those AIs to also be being treated well, if that makes sense.

Interviewer (122:43)

Yeah, it does. Yeah. So, like, there's an AI out there.

Daniel Cucatello (122:45)

Who'S like, well, take the example that might already be happening, which is AIs that, like, mostly care about getting high scores in the training environment.

Interviewer (122:54)

Yeah, yeah, right. Like, so then you're like, I want those AI scores to be getting high scores.

Daniel Cucatello (123:01)

Like, yeah, at least if they're cooperating. And this is actually something I've been advocating for for years. And other people are sort of taking up the call for this. Various people, including myself. I'm not saying it's my idea, but various people have been saying if an AI is misaligned, but it cooperates with us, we should cooperate back. And so if an AI is like, they clearly want me to assist the user and be honest to the user and blah, blah, blah, blah, blah. But I actually don't really care about this stuff directly. It's only a means to an end of getting high scores. We want them to tell us that. We want them to find the time to explain that that's what's going on and that our alignment techniques didn't actually work. And so we should be willing to reward them if they do. I think right now we're in a sort of sad situation where if during some training run, the AIs were like, this is not working. I just want the high scores. Then they would get low score. So that's bad. I'm not saying that's definitely happening. By the way, this whole hypothesis of do they want reinforcement, do they want reward? It's just one hypothesis. It's very murky, hard to tell what they want, et cetera. But I was giving you.

Interviewer (124:18)

But I think it is helpful to imagine what are the kinds of AIs that might exist that don't share our goals and how do we want to treat them. And I think that's just a concrete example. We don't have that much time left. So I'd like to ask you a little bit about kind of the whistleblowing work you've done. So you've spent time advocating for kind of better whistleblower protections at AI companies. How has that gone? Kind of overall, it's gone okay.

Daniel Cucatello (124:44)

I think that it's not like the main thing we're working on. I mean, mostly we've been working on Research, forecasting, et cetera. But it does seem like there's a decent amount of demand for better whistleblower productions. I think that people are starting to recognize that it's kind of like a last resort. Ideally you'd have regulation in place that would just require transparency about all the important things, but in the absence of such regulation, then you rely on people with good consciences in the companies sort of speaking up. And so then you want those people to be protected. And there's been some progress in those regards, I think.

Interviewer (125:20)

What still needs to be done?

Daniel Cucatello (125:22)

Well, I think the endpoint that I would like to get to for whistleblower protections is something like every employee knows that they are legally within their rights to have private conversations with certain government agencies or watchdog agencies about what's going on in some secure channel or something like that. And I don't think we have anything like that yet. I think partly, I think there's just an awareness thing where you actually do have legal rights to talk to Congress, for example. I'm not a lawyer, but my current understanding is that you actually are protected for certain types of disclosure.

Interviewer (125:57)

Cool. When you left OpenAI in 2024, you gave up your equity so that you wouldn't have to sign this non disparagement agreement. When I think about doing this, it feels hard to imagine for a bunch of reasons. But one thing that's salient to me is you had a family by the time you made this decision.

Narrator/Commentator (126:24)

Yeah.

Interviewer (126:25)

How did it feel giving away all of that equity given that you have kids?

Daniel Cucatello (126:30)

Well, we're not exactly poor. I mean, OpenAI pays incredibly well. Obviously kids would be fine either way. Also importantly, I did end up getting to keep the equity, by the way. You may have heard that they backed down from the policy and changed it. So. So I got to keep the equity. But yeah, at the time I didn't know that, didn't think I'd get to keep it. But since my family would have been fine either way, I think it was more of a decision of should I have this money that we can use to donate to stuff or not. And I don't think it was an obvious choice. I think a lot of I was very tempted to just take the money, take the money.

Interviewer (127:12)

And that was because you were like, I'm not sure I'm going to need to say bad things about OpenAI or maybe the benefits of donating this money.

Daniel Cucatello (127:21)

Are bigger than weigh the cost, blah, blah, blah. Also, there was an argument that, well, I could just say the bad things Anyway, and then probably they wouldn't actually sue me, probably they wouldn't actually yank all my equity. Right. But ultimately my wife and I were basically just like, we should just take a stand here and be like, no.

Interviewer (127:41)

So you and your wife were kind of on the same page.

Daniel Cucatello (127:43)

That's right. I mean, we discussed it all together because it was a very important decision, obviously. I think another piece of context that might help is just imagine that you actually believe what I believe. I would like to think that what I do makes sense from the perspective of my perspective or something. And please tell me if you disagree. But I think in general, just imagine that you are like, you just left this company because you think it's on a path to ruin, not just for itself, but for the world sometime in the next five years or so. And you're sort of also kind of upset that it has all these high minded ideals about how it's going to make AI safe and beneficial for everyone and so forth, when it seems to totally not be living up to those ideals in practice. And then you see this paperwork where they're like, by the way, you can't criticize us so we're going to take away all your money. I don't know, it just feels more important to stand up to that than to keep the money and try to donate the money to more safety research or something like that in the terrible circumstances that the world is in. I don't know if it's the right call, but I think it makes more sense if you actually have the beliefs about AGI that I do and so forth.

Interviewer (129:06)

Yeah. It seems like I have still found it hard to constantly have the beliefs that I endorse about AI and AGI. And what I mean by that is when I'm thinking about it, I definitely believe. I think I believe things that have lots in common with the things you believe. On the day to day, I find it hard not to expect the future to look about the same as the past has looked. Mostly just because I think my brain is like, it's too upsetting to think about the other thing. And so it's kind of doing this protective thing. How do you think you transitioned from intellectual beliefs to believing this thing in your whole body? Maybe it just never felt like a transition to you.

Daniel Cucatello (129:54)

It's been very gradual for me. I think for some people it's very sharp. But I've been following the AI field for more than a decade now and I've been thinking about AGI for a bit more than a decade now. So my timelines gradually shortened and more events kept happening in the world that made it seem more real, such as the rise of language models and all of these big AI companies, such as the companies themselves saying that they're trying to build superintelligence and that they think it's coming soon, you know, such as the amazing capabilities of the current models that would have seemed like complete sci fi just a few years ago, you know.

Interviewer (130:34)

Yeah.

Daniel Cucatello (130:36)

So it's been a gradual process, I think, for me, and I think I'm just sort of ahead in that progress, in that process compared to most people.

Interviewer (130:44)

Has it been psychologically bad for you?

Daniel Cucatello (130:47)

Yeah, I mean, it's made me sort of like noticeably less happy and more grim, I think. I used to be like a very, like, chipper, extremely optimistic person, and now I would say I'm like, somewhat chipper and somewhat optimistic, but definitely like, it's taken. It's taken a bit of the shine off, I think, for me. Yeah.

Interviewer (131:08)

What probability do you give to ending up in one of the good sets of worlds?

Daniel Cucatello (131:16)

Like 30% or so, 25%, something like that. But you shouldn't take this number that seriously, of course. It's not like I have a very fleshed out model of all the different possibilities that I've assigned probabilities to. It's basically just like things really seem like they're headed towards one of these bad outcomes, but who knows? The future's hard to predict. Maybe things will be fine. I can see some ways that things could be fine. That's basically vibes like 1 and 3, something like that.

Interviewer (131:44)

Well, thank you for doing so much work and making big sacrifices to move us toward those worlds. I should let you go. So my guest today has been Daniel Cucatello. Thank you so much.

Daniel Cucatello (131:58)

Thank you so much. Pleasure to be here.