Anthropic’s C.E.O. Dario Amodei on Surviving the A.I. Endgame

Summary

Hard Fork Podcast Summary: Anthropic’s CEO Dario Amodei on Surviving the AI Endgame

Podcast Information:

• Title: Hard Fork
• Host/Authors: Kevin Roose and Casey Newton
• Episode: Anthropic’s C.E.O. Dario Amodei on Surviving the A.I. Endgame
• Release Date: February 28, 2025

---

1. Introduction

In this episode of Hard Fork, hosts Kevin Roose and Casey Newton welcome back Dario Amodei, the CEO of Anthropic, for an in-depth discussion about the latest advancements in AI, particularly focusing on Anthropic's new model, Claude 3.7 Sonnet. The conversation delves into the rapid evolution of AI technology, the competitive landscape, and the pressing concerns surrounding AI safety.

2. Claude 3.7 Sonnet: Enhancements and Real-World Applications

[05:35] Dario Amodei: "We've been working on this model for a while, focusing on real-world tasks rather than just mathematical and competitive coding."

Dario explains that Claude 3.7 Sonnet represents a significant improvement over previous iterations. Unlike earlier models primarily trained on objective tasks like math and coding competitions, Claude 3.7 emphasizes practical, real-world applications. This shift aims to make the AI more relevant to everyday tasks and economic activities.

[07:01] Dario Amodei: "Claude 3.7 is better in general, including at coding, and offers an extended thinking mode for more complex tasks."

Users will notice that Claude 3.7 not only excels in coding but also introduces a mode that allows the AI to engage in longer, more thoughtful responses. This feature is particularly beneficial for tasks that require extended reasoning, such as in-depth analyses or complex problem-solving.

3. AI Safety: Current Concerns and Future Risks

[13:00] Kevin Roose: "Are there any new capabilities that Claude 3.7 Sonnet has that might worry someone concerned about AI safety?"

Dario addresses the delicate balance between advancing AI capabilities and ensuring safety. He emphasizes that while current models like Claude 3.7 Sonnet are not inherently dangerous, the trajectory of increasingly powerful AI systems poses significant risks. These include potential misuse in areas like biological or chemical warfare and the autonomy of AI systems that could threaten infrastructure or humanity.

[16:38] Dario Amodei: "We assessed a substantial probability that the next model could present new risks, prompting our safety procedures."

Amodei highlights that Anthropic is proactively evaluating the risks associated with each new model release. Their approach includes controlled trials and real-world scenario testing to identify and mitigate potential threat vectors before deploying more advanced models.

4. International AI Competition: US vs. China

[21:23] Casey Newton: "Why is Deep Seek maintaining pace with Frontier Labs noteworthy, and what should be done about it?"

Dario expresses concern over China's advancements in AI, particularly emphasizing the national security implications. He believes that AI could become a tool for autocratic regimes to enhance their repressive capabilities, posing a threat to global stability. To counter this, he advocates for stringent export controls and policies that ensure liberal democracies maintain a technological edge to prevent misuse.

[24:33] Dario Amodei: "If the US stays ahead by a couple of years, we can use that time to make AI safer."

Amodei argues that maintaining a lead in AI development allows for the implementation of safety protocols and regulations, thus preventing an arms race mentality where speed trumps safety.

5. AI Summits and Public Discourse on AI Risks

[27:08] Casey Newton: "What was your impression of the recent AI summit in Paris?"

Dario expresses disappointment with the Paris AI summit, describing it as more of a trade show than a meaningful discussion on AI risks. He laments the shift away from collaborative efforts to address AI safety, noting that the original spirit of such summits—to deliberate on the societal implications of AI—has been lost.

[30:36] Casey Newton: "How can we bridge the gap between AI experts and policymakers who don’t fully grasp the exponential growth of AI?"

Dario suggests that effective communication and foresight are crucial. He emphasizes the importance of preparing public officials to understand the rapid advancements in AI to facilitate informed policy-making that prioritizes both innovation and safety.

6. Polarization of AI Safety Efforts

[35:01] Casey Newton: "Do you see the politicization of AI safety as a barrier to effective risk management?"

Dario concurs, highlighting that the polarization of AI safety into partisan debates undermines constructive discussions. He advocates for nuanced conversations that recognize the importance of both harnessing AI's benefits and mitigating its risks without falling into political rhetoric.

[36:08] Casey Newton: "National security issues have traditionally been associated with the right, but now it seems like the right is distancing itself from AI safety."

Dario acknowledges this shift and stresses the need for a unified approach that transcends political divides to address the existential risks posed by AI effectively.

7. Societal Responses and Career Implications

[52:19] Casey Newton: "What should individuals do if they believe AGI is imminent?"

Dario advises focusing on personal well-being, staying informed about societal changes, and developing critical thinking skills. He emphasizes that while individual actions are limited in the face of transformative AI changes, collective awareness and adaptability are essential.

[53:59] Dario Amodei: "AI is moving fast, especially in coding. We might see replacement at lower levels within 18 to 24 months."

Dario suggests that individuals in the IT and coding sectors should consider upskilling and adapting to the evolving job landscape shaped by AI advancements.

8. Positive Futures and AI Benefits

[44:37] Casey Newton: "How much AI upside do you expect to arrive this year?"

Dario highlights tangible benefits already emerging from AI applications, such as accelerating clinical trial reporting and improving medical diagnoses. He shares anecdotes of users leveraging Claude to expedite complex tasks, underscoring the pragmatic advantages of AI in enhancing productivity and solving real-world problems.

[45:54] Casey Newton: "There are specific, relatable examples of AI making a positive impact, like diagnosing medical conditions faster."

This segment emphasizes the dual nature of AI's potential, showcasing how it can drive significant advancements while also highlighting the necessity for responsible development and deployment.

9. Conclusion

The episode concludes with the hosts reflecting on the weighty discussions surrounding AI's future. They underscore the importance of balancing innovation with safety and the need for collective efforts to navigate the transformative landscape that AI presents.

Notable Quotes:

• Dario Amodei [05:35]: "Claude 3.7 is better in general, including at coding, and offers an extended thinking mode for more complex tasks."

• Dario Amodei [13:00]: "It's not that there aren't present dangers... I'm more worried about the dangers that we're going to see as models become more powerful."

• Dario Amodei [21:23]: "I think the autocratic countries don't get ahead from a military perspective... We shouldn't be naive."

• Dario Amodei [24:33]: "If the US stays ahead by a couple of years, we can use that time to make AI safer."

• Dario Amodei [35:01]: "Addressing the risks while maximizing the benefits requires nuance."

• Dario Amodei [52:19]: "AI is moving fast, especially in coding. We might see replacement at lower levels within 18 to 24 months."

• Dario Amodei [44:37]: "AI is already accelerating biomedicine and improving medical diagnoses."

Key Takeaways:

• Advancements in AI Models: Claude 3.7 Sonnet represents a significant step towards more practical, real-world applications of AI, enhancing capabilities in areas like coding and complex task management.

• AI Safety Concerns: As AI systems become more powerful, the potential risks—especially those related to autonomous misuse—grow, necessitating robust safety protocols and proactive risk assessments.

• Global AI Competition: The race between leading AI nations, particularly the US and China, has profound implications for national security and global stability, urging the need for strategic policy interventions.

• Public Discourse and Polarization: The politicization of AI safety debates hampers effective risk management, highlighting the need for balanced, non-partisan discussions on AI's future.

• Societal Impact and Career Shifts: AI's rapid evolution is reshaping job markets, especially in technology sectors, prompting individuals to adapt through upskilling and staying informed.

• Positive AI Applications: Despite the challenges, AI is driving significant advancements in fields like medicine and productivity, demonstrating its potential to effectuate positive change when harnessed responsibly.

---

This summary aims to encapsulate the essence of the Hard Fork episode featuring Dario Amodei, providing a comprehensive overview for those who have not listened to the podcast.

Transcript

[Oracle Ad]

This podcast is supported by Oracle.

AI requires a lot of compute power and the cost for your AI workloads can spiral. That is, unless you're running on OCI Oracle Cloud Infrastructure. This was the cloud built for AI, a blazing, fast, enterprise grade platform for your infrastructure, database, apps and all of your AI workloads.

Right now, Oracle can cut your current cloud bill in half if you move to OCI. Minimum financial commitment and other terms apply. Offer ends March 31st. See if you [e-mail] hardfork oracle.com hardcore.

[Kevin Roose]

Hard fork so I went to two AI events this weekend. It was sort of polar opposites of the kind of AI spectrum. There was the effective altruists had their big annual conference y and then on Friday night I, I went out. You'd be very proud of me. I stayed out so late. I stayed out till 2am oh my. I went to a. An AI rave that was sort of unofficially affiliated with Mark Zuckerberg. It was called the Zuckerraff.

[Casey Newton]

Now, when you say unofficially affiliated, Mark Zuckerberg had no involvement in this. And my assumption is he did not know it was happening.

[Kevin Roose]

Correct. A better word for what his involvement is would be no involvement. He was, it was sort of a tribute rave to Mark Zuckerberg thrown by a bunch of acceleration people who want AI to go.

[Casey Newton]

Another word for it would be using his likeness without permission. Yes, but that happens to famous people sometimes.

[Kevin Roose]

Yes. So at the Zuck rave, I would say there was not much raving going on. There was a dance floor, but it was very sparsely populated. They did have a like a thing there that would like you'd. It had a camera pointing at their dance floor and if you sort of stood in the right place, it would turn your face into Mark Zuckerberg's like on a big screen. So that's which.

[Casey Newton]

Let's just say it's not something you want to happen to you while you're on mushrooms, because that can be a very destabilizing event.

[Kevin Roose]

Yes. There was a train, an indoor like toy train that you could ride on. It was going actually quite fast.

[Casey Newton]

What was the point of this rave?

[Kevin Roose]

To do drugs. That was the point of this rave. I'm Kevin Roose, a tech columnist at the New York Times.

[Casey Newton]

I'm Casey Noon from Platformer, and this is Hard Fork. This week, Anthropic CEO Dario Amadei returns to the show for a super sized interview about the new Claude, the AI race against China and his hopes and fears for the future of AI. Then we close it out With a round of hat GPT. Big show this week.

[Kevin Roose]

Evan Casey, have you noticed that the AI companies do stuff on the weekends now?

[Casey Newton]

Yeah. Whatever happened to just five days a week?

[Kevin Roose]

Yes. They are not respectful of reporters and their work hours. Companies are always announcing stuff on Saturdays and Sundays and different time zones. It's a big pain.

[Casey Newton]

It really is.

[Kevin Roose]

But this weekend, I got an exciting message on Sunday saying that Dario Amade, the CEO of Anthropic, had some news to talk about and he wanted to come on Hard Fork to do it.

[Casey Newton]

Yeah. And around the same time, I got an email from Anthropic telling me I could preview their latest model. And so I spent the weekend actually trying it out.

[Kevin Roose]

Yeah. So longtime listeners will remember that Dario is a repeat guest on this show. Back in 2023, we had him on to talk about his work at Anthropic and his vision of AI safety and where all of this was head. And I was really excited to talk to him again for a few reasons. One, I just think he's, like, a very interesting and thoughtful guy. He's been thinking about AI for longer than almost anyone. He was writing papers about potentially scary things in AI safety all the way back in 2016. He's been at Google. He's been at OpenAI. He's now the CEO of Anthropic. So he is really the ultimate insider when it comes to AI.

[Casey Newton]

And, you know, Kevin, I think Dario is an important figure for another reason, which is that of all of the folks leading the big AI labs, he is the one who seems the most publicly worried about the things that could go wrong. That's been the case with him for a long time. And yet over the past several months, as we've noted on the show, it feels like the pendulum has really swung away from caring about AI safety to just this sort of go, go, go accelerationism that was embodied by the speech that Vice President J.D. vance gave in France the other day. And for that reason, I think it's important to bring him in here and maybe see if we can shift that pendulum back a little bit and remind folks of what's at stake here.

[Oracle Ad]

Yeah.

[Kevin Roose]

Or at least get on the pendulum swinging and why he thinks it may swing back in the future. So today we're going to talk to Dario about the new model that anthropic just released, Claude 3.7 Sonnet. But we also want to have a broader conversation because there's just so much going on in AI right now.

[Casey Newton]

And, Kevin, something else that we should note, something that is true of Dario this time. That was not true the last time that he came on the show is that my boyfriend now works at his company.

[Kevin Roose]

Yeah. Casey's manthropic. Is it anthropic?

[Casey Newton]

My manthropic is anthropic. And I have a whole sort of long disclosure about this that you can read at Platformer News, slash ethics. Might be worth doing this week. You know, we always like reminding folks of that.

[Kevin Roose]

Yep. All right, with that, let's bring in Dario Amade. Dario Amade, welcome back to Hard Fork.

[Dario Amadei]

Thank you for having me again. Yeah.

[Casey Newton]

Returning champion.

[Kevin Roose]

So tell us about Claude 3.7. Tell us about this new model.

[Dario Amadei]

Yes. So we've been working on this model for a while. We basically, you know, had in mind two things. One was that, you know, of course there are these reasoning models out there that have been out there for a few months, and we wanted to make one of our own, but we wanted the focus to be a little bit different. In particular, a lot of the other reasoning models in the market are trained primarily on math and competition coding, which are, you know, they're, they're objective tasks where you can measure performance. I'm not saying they're not impressive, but they're sometimes less relevant to tasks in the real world or the economy. Even within coding, there's really a difference between competition coding and doing something in the real world. And so we train Claude 3.7, you know, more to focus on these real world tasks. We also felt like it was a bit weird that, you know, in the reasoning models that folks have offered, it's generally been, there's a regular model and then there's a reasoning model. This would be like if a human had two brains. And it's like, you know, you're like, you can, you can talk to brain number one. If you're asking me a quick question, like, what's your name? And you're talking to brain number two if you're asking me to, like, prove a mathematical theorem because I have to, like, sit down for 20 minutes.

[Kevin Roose]

It'd be like a podcast where there's two hosts, one of whom just likes to yap and one of whom actually thinks before he talks.

[Casey Newton]

Oh, come on. Brutal.

[Dario Amadei]

No, no, no comment. Brutal. No comment on any relevance to it.

[Kevin Roose]

So what, what differences will users of Claude notice when they start using 3.7 compared to previous models?

[Dario Amadei]

Yes, so a few things. It's going to be better in general, including better at coding, which, you know, Claude models have always been the best at coding. But 3.7 took a further step up. In addition to just the properties of the model itself. You can put it in this extended thinking mode where you tell it basically the same model, but you're just saying operate in a way where you can think for longer. And if you're an API user, you can even say here's the boundary and how long you can think.

[Casey Newton]

And just to clarify, because this may confuse some people, what you're saying is the sort of new CLAUDE is this hybrid model, it can sometimes do reasoning, sometimes do quicker answers, but if you want it to think for even longer, that is a separate mode.

[Dario Amadei]

That is a separate mode.

[Casey Newton]

Thinking and reasoning are sort of separate modes.

[Dario Amadei]

Yes, yes. So basically the model can just answer as it normally would, or you can give it this, this indication that it should think for longer. An even further direction of the evolution would be the model decides for itself what the appropriate time to think is. Right. Humans are like that, or at least can be like that. Right. If I ask you your name, you know, you're not like, huh, how long should I think of? Give me, give me 20 minutes, right? To, you know, to determine my name. But if I say, hey, you know, I'd like you to do an analysis of, you know, this stock or I'd like you to, you know, prove this mathematical theorem, you know, humans who are, who are able to do that task, they're not going to try and give an answer right away. They're going to say, okay, well that's going to take a while and they, then we'll need to write down the task.

[Kevin Roose]

This is one of my main beefs with today's like language models and AI models in general is, you know, I'll be using like something like chat GPT and I'll forget that I'm in like the hardcore reasoning mode and I'll ask it some stupid question like, you know, how do I change the settings on my water heater? And it'll go off and think for four minutes. And I'm like, I didn't actually mean to do that.

[Dario Amadei]

Like a treatise on like adjusting the temper of the water heater, like consideration one.

[Kevin Roose]

So how long do you think it'll be before the models can actually do that kind of routing themselves? Where you'll ask a question and say it seems like you need about a 3 minute long thinking process for this one versus maybe a 30 second one for this other. Yeah.

[Dario Amadei]

So, you know, I think our model is kind of a step towards this even in the API. If you give it a bound on Thinking, you know, you say, you know, I'm going to think for 20,000 words or something. Most on average, when you give it up to 20,000 words, most of the time it doesn't even use 20,000 words. And sometimes it'll give a very short response because when it knows that it doesn't get any gain out of thinking further. It doesn't think for longer. But it's still valuable to give a bound on how long it'll think. So we've kind of taken a, like a big step in that direction, but we're not to where we want to be yet.

[Casey Newton]

When you say it's better at real world tasks, what are some of the tasks that you're thinking of?

[Dario Amadei]

Yeah, so I think above all coding. You know, Claude models have been very good for real world coding. You know, we have, we have a number of, you know, customers from cursor to GitHub to windsurf, codium to Cognition to Vercel to. I'm sure I'm leaving some out there.

[Kevin Roose]

These are the Vibe coding apps or.

[Casey Newton]

Just the coding apps, period?

[Dario Amadei]

The coding apps, period. And, you know, there are many different kind of coding apps. We also release this thing called Claude code, which is more of a command line tool. But I think also on things like, you know, complex instruction following or just like here, I want you to understand this document or, you know, I want you to use this series of tools, the reasoning model that we've trained. Claude 3.7 sonnet is better at those tasks too.

[Casey Newton]

Yeah. One thing the new Claude Sonnet is not doing. Dario is accessing the Internet. Yes, why not? And what would cause you to change that?

[Dario Amadei]

Yes. So I think I'm on record saying this before, but web search is coming very soon. Okay, we will have web search very soon. We recognize that as an oversight. You know, I think in general, we tend to be more enterprise focused and consumer focused, and this is more of a consumer feature, although it can be used on both. But, you know, we focus on both and this is coming.

[Casey Newton]

Got it. So you've named this model 3.7. The previous model was 3.5. You quietly updated it last year and insiders were calling that one 3.6. Respectfully, this is driving all of us insane. What is going on with AI model names?

[Dario Amadei]

We are the least insane, although I recognize that we are insane. So. So look, I think our mistakes here are relatively understandable. You know, we made a 3.5 sonnet, we were doing well, and there we had the 3.3.0, and then the 3.5. The. I recognize the 3.7 new was a misstep. It actually turns out to be hard to change the name in the API, especially when there's all these partners and surfaces.

[Kevin Roose]

You often figure it out, I believe.

[Dario Amadei]

So we.

[Casey Newton]

No, no, no.

[Dario Amadei]

It's harder than training the model, I'm telling you. So we've kind of retroactively and informally named the last one 3.6. Sense of this one is 3.7.

[Casey Newton]

Right.

[Dario Amadei]

And we are reserving Quad for Sonnet and maybe some other models in the sequence for things that are really quite.

[Kevin Roose]

Substantial leaps sometimes when the model.

[Dario Amadei]

Those models are coming, by the way.

[Kevin Roose]

Okay, got it. Coming when?

[Dario Amadei]

Yeah, so yeah, I should, I should talk a little bit about this. So all the models we've released so far are actually not that expensive. Right. You know, I did this blog post where I said they're in the few tens of millions of dollars range at most. There are bigger models than they are coming. They take a long time and sometimes they take a long time to get right. But those bigger models, you know, they're coming for others. I mean, they are rumored to be coming from competitors as well. But you know, we are not too far away from releasing a model that's a bigger base model. So most of the improvements in Claude 3.7 Sonnet as well as Claude 3.6 Sonnet are in the post training phase.

[Casey Newton]

Okay.

[Dario Amadei]

But we are working on stronger base models and you know, perhaps that'll be the Claude 4 series, perhaps not. We'll see. But you know, I think those are coming in, you know, relatively small number of time units.

[Casey Newton]

Small number of time units, yeah. I'll put that on my calendar. Remind me to check in on that on a few time units. Kevin.

[Kevin Roose]

I know you all at Anthropic are very concerned about AI safety and the safety of the models that you're putting out into the world. I know you spend lots of time thinking about that and red teaming the models internally. Are there any new capabilities that Cloud 3.7 Sonnet has that are dangerous or that might worry someone who is concerned about AI safety?

[Dario Amadei]

So not, not dangerous per se. And I always want to be clear about this because I feel like there's this constant conflation of present dangers with future dangers. It's not that there aren't present dangers and you know, there are always kind of normal tech risks, normal tech policy issues. I'm more worried about the dangers that we're going to see as models become more powerful. And I think those dangers, you know, when we talked in 2023, I talked about them a lot. I, you know, I think I said, I even testified in front of the Senate for things like, you know, misuse risks with, for example, biological or chemical warfare or the autonomy risks. I said, particularly with the misuse risks. I said, I don't know when these are going to be here, when these are going to be real risks, but it might happen in 2025 or 2026. And now that we're in kind of early 2025, the very beginning of that period, I think the models are starting to get closer to that. So in particular in Claude 3.7 sonnet, as we wrote in the model card, we always do these, you could almost call them like, you know, trials, like trials with a control where we have, you know, some human who doesn't know enough much about some area like biology. And we basically see how much does the model help them to engage in some mock bad workflow. Right, we'll change a couple of the step, but, you know, some, some mock bad workflow. How good is a human at that? Assisted by the model. Sometimes we even do wet lab trials in the real world where they, they mock, make something bad as compared to the current technological environment. Right? What, what they could do on Google, on, on Google or with a textbook, or just what they could do unaided. And we're trying to get at does this enable some new threat vector that wasn't there before? I think it's very important to say this isn't about like, oh, did the model give me the sequence for this thing? Did it give me a cookbook for making meth or something? That's easy. You can do that with Google. We don't care about that at all. We care about this kind of esoteric, high uncommon knowledge that say only a, you know, a virology PhD or something has how much does it help with that? And if it does, you know, that doesn't mean we're all going to die of the plague tomorrow. It means that a new risk exists in the world, a new threat vector exists in the world, as if you just made it, you know, easier to build a, you know, nuclear weapon. You invented something that, you know, the amount of plutonium you needed was, you know, was lower than, than it was before. And so we measured Sonnet 3.7 for these risks. And the models are getting better at this. They're not yet at the stage where, you know, we think that there is a real and meaningful increase in the threat. End to end, right? To do all the tasks you need to do to really do something dangerous. However, we said in the model card that we assessed a substantial probability that the next model, or, you know, a model over the next, I don't know, three months, six months, a substantial probability that, you know, that we could be there. And then our safety procedure, our responsible scaling procedure, which is focused mainly on these, you know, very large, you know, risks would then kick in and, you know, we'd have kind of additional security measures and additional deployment measures, you know, designed, you know, particularly against these very narrow risks.

[Casey Newton]

Yeah. I mean, just to really underline that, you're saying in the next three to six months, we are going to be in a place of medium risk in these models, period. Presumably, if you are in that place, a lot of your competitors are also going to be in that place. What does that mean practically? Like, what does the world need to do if we're all going to be living in medium risk?

[Dario Amadei]

I think at least at this stage, you know, it's not a huge change to things. It means that there's a narrow set of things that models are capable of, if not mitigated, that, you know, would. Would somewhat increase the risk of something like really dangerous or really bad happening. You know, like put yourself in the eyes of, like, a law enforcement officer or, you know, the FBI or something. There's like a new threat vector, there's a new kind of attack. You know, it doesn't mean the end of the world, but it does mean that anyone, anyone who's involved in industries where this risk exists should take a precaution against that risk in particular.

[Casey Newton]

Got it.

[Dario Amadei]

And so I don't know. I mean, I don't. You know, I could be wrong. It could take much longer. You can't predict what's going to happen. But, you know, I think. I think contrary to the environment that we're seeing today of worrying less about the risk, the risks in the background have actually been increasing.

[Casey Newton]

We have a bunch more safety questions, but I want to ask two more about kind of innovation competition first. Yeah, right now, it seems like no matter how innovative any given company's model is, those innovations are copied by rivals within months or even weeks. Does that make your job harder and do you think it is going to be the case indefinitely?

[Dario Amadei]

I don't know that innovations are necessarily copied exactly. What I would say is that the pace of innovation among a large number of competitors is very fast. There's four or five, maybe six companies who are innovating very quickly and producing models very quickly. But if you look, for example, at Sonnet 3.7, you know, the way we did the reasoning models is different from what was done by competitors. The things we emphasized were different even before then. The Things Sonnet 3.5 is good at are different than the things other models are good at. People often talk about competition, commoditization, costs going down, but the reality of it is that the models are actually relatively different from each other. And that creates differentiation.

[Kevin Roose]

Yeah, I mean, we get a lot of questions from listeners about, you know, if I'm going to subscribe to one AI tool, what should it be? You know, these are the things that I use it for. And I have a hard time answering that because I find for most use cases, the models all do a relatively decent job of answering the questions. It really comes down to things like which model's personality do you like more? Do you think that people will choose AI models, consumers, on the basis of capability, or is it going to be more about personality and how it makes them feel, how it interacts with them?

[Dario Amadei]

I think it depends which consumers you mean. You know, even among consumers there, there are people who use the models for tasks that are complex in some way. There are folks who are kind of, you know, independent, who want to. Who want to analyze data. Like, you know, that's maybe kind of like the prosumer side of things. Right. And I think within that, there's a lot to go in terms of capabilities. The models can be so much better than they are at helping you with anything that's focused on kind of productivity or even a complex task like planning a trip. Even outside that, you know, if you're just trying to make a personal assistant to manage your life or something, we're. We're pretty far from that, you know, from a model that sees every aspect of your life and is able to kind of holistically give you advice and kind of be a helpful assistant to you. And I think there's differentiation within that. The best assistant for me might not be the best assistant for some other person. I think one area where the models will be good enough is if you're just trying to use this as a replacement for Google Search or as a quick information retrieval, which I think is what's being used by kind of the mass market, free use, hundreds of millions of users. I think that's very commoditizable. I think the models are kind of already there and are just diffusing through the world. But I don't think those are the interesting uses in the model. And I'm actually not sure a lot of the economic value is there.

[Casey Newton]

I mean, it's part of, is what I'm hearing that if and when you developed an agent that is, let's say, a really amazing personal assistant, the company that figures out that first is going to have a big advantage because other labs are going to just have a harder time copying. That's going to be less obvious to them how to recreate that.

[Dario Amadei]

It's going to be less obvious how to recreate it. And when they do recreate it, they won't recreate it exactly. They'll do it their own way, in their own style, and it'll be suitable for a different set of people. So I guess I'm saying there's, there's. The market is more segmented than you think it is. It looks like it's all one thing, but it's more segmented than you think it is.

[Casey Newton]

So let me ask the competition question that brings us into safety. You recently wrote a really interesting post about Deep Seek, sort of at the height of Deep Seek mania. And you were arguing in part that the cost reductions that they had figured out were basically in the line with what they were basically in line with how costs had already been falling. But you also said that Deep Seek should be a wake up call because it showed that China is keeping pace with Frontier Labs in a way that the country hadn't been up until now. So why is that notable to you and what do you think we ought to do about it?

[Dario Amadei]

Yeah, so I think this is less about commercial competition, right? I worry less about Deep Seek from a commercial competition perspective. I worry more about them from a national competition and national security perspective. I think where I'm coming from here is, you know, we, we look at the, the state of the world and, you know, we have these autocracies like China and Russia. And I've always worried, I've worried, you know, maybe for a decade that AI could be an engine of autocracy. If you think about repressive governments, the limits to how repressive they can be are generally set by what they can get their enforcers. They're human enforcers to do. But if their enforcers are no longer human, that starts painting some very dark possibilities. And so this is an area that I'm therefore very concerned about where I want to make sure that liberal democracies have enough leverage and enough advantage in the technology that they can prevent some of these abuses from happening and kind of, you know, also prevent our adversaries from Putting us in a bad position with respect to the rest of the world or, you know, even, even threatening our security. You know, there's. There's this kind of, I think, weird and awkward feature that it's companies in the US that are building this, it's companies in China that are building this. But we shouldn't be naive. Whatever the intention of those companies, particularly in China, there's a governmental component to this. And so I'm interested in making sure that the autocratic countries don't get ahead from a military perspective. I'm not trying to deny them the benefits of the technology. There are enormous health benefits that, you know, all of us, I want to make sure are, you know, make their way everywhere in the world, including the poorest areas, including areas that are under the grip of autocracies. But I don't want the autocratic governments to have a military advantage. And so, you know, things like the export controls, which I discussed in that post, are one of the things we can do to prevent that. And, you know, I was heartened to see that actually the Trump administration is considering tightening, tightening the export controls.

[Kevin Roose]

I was at a safety conference last weekend, and one of the critiques I heard some folks in that universe make of anthropic, and maybe of you in particular, was that they saw the posts like the one you wrote about Deep Seek as effectively promoting this AI arms race, with China insisting that America has to be the first to reach powerful AGI or else, and they worry that some corners might get cut along the way, that there are some risks associated with accelerating this race in general. What's your response to that?

[Dario Amadei]

Yeah, I kind of view things differently. So my view is that if we want to have any chance at all, so the default state of nature is that things go at maximum speed. If we want to have any chance at all to not go at maximum speed. The way the plan works is, is the following. Within the US or within democratic countries, you know, these are all countries that are under the rule of law, more or less. And therefore we can pass laws, we can get companies to make agreements with the government that are enforceable about, you know, or make safety commitments that are enforceable. And so if we have a world where there's these different companies and they're, you know, they're in the kind of default state of nature, would, would race as fast as possible through some mixture of voluntary commitments and, and, and laws, we can get ourselves to slow down if the models are too dangerous. And that's actually enforceable. Right. It's you know, you can, you can get everyone to cooperate in the prisoner's dilemma, if you just point a gun at everyone's head and you can, that's what the law ultimately is. But I think that all gets thrown out the window. In the world of international competition. There was no one with the authority to enforce any agreement between the US and China, even if one were to be made. And so my worry is if, if the US is a couple of years ahead of China, we can use that couple years to make things safe. If we're even with China, you know, there's, there's no promoting an arms race. That's what's going to happen. The technology has immense military value. Whatever people say now, whatever nice words they say about cooperation, I just don't see how once people fully understand the economic and military value of the technology, which I think they mostly already do, I don't see any way that it turns into anything other than the most intense race. And so what I can think of to try and give us more time is if we can slow down the authoritarians, that it almost obviates the trade off. It gives us more time to work out among us, among OpenAI, among Google, among X, AI, how to make these models safe. Now could at some point we convince authoritarians, convince for example, the Chinese that the models are actually dangerous and, you know, that, that we should have some agreement and come up with some way of enforcing it. I think we should actually try to do that as well. I'm supportive of trying to do that, but it cannot be the Plan A. It just, it's just not a realistic way of looking at the world.

[Casey Newton]

These seem like really important questions and discussions and it seems like they were mostly not being had at the AI summit in Paris that you and Kevin attended a couple weeks back, what the heck was going on with that summit?

[Dario Amadei]

Yeah, I mean, you know, I have to tell you, I was deeply disappointed in the summit. It had the environment of a trade show and was very much out of spirit with the, the spirit of the original summit that was created in, you know, in Bletchley park by the UK government. Bletchley did a great job and the UK government did a, did a great job where, you know, they didn't introduce a bunch of onerous regulation certainly before they knew what they were doing. But they said, hey, let's convene these summits to discuss the risks. I thought that was very good. I think that's gone by the wayside now. And it's, you know, it's part of maybe a general move towards, you know, less worrying about risk, more wanting to seize the opportunities. And I'm a fan of seizing the opportunities. Right. I, you know, I wrote this essay, Machines of Loving Grace, about all the great things. You know, part of that essay was like, man, for someone who worries about risks, I feel like I have a better vision of the benefits than a lot of people who spend all their time talking about the benefits. But in the background, like I said, as the models have gotten more powerful, the amazing and wondrous things that we can do with them have gotten, you know, have increased, but also the risks have increased. And you know, that, that kind of secular increase, that smooth exponential, it doesn't pay any attention to societal trends or the political winds or the risk is, you know, increasing, you know, up to some critical point, whether you're paying attention or not. Right. It was, you know, small. It was small and increasing when there was this frenzy around, you know, AI Risk and everyone was posting about it and there were these summits and now the winds have gone in the other direction, but the exponential just continues on. It doesn't care.

[Kevin Roose]

I had a conversation with someone in Paris who was saying, like, it just didn't feel like anyone there was feeling the AGI, by which they meant like politicians, the people doing these panels and, you know, gatherings. We're all talking about AI as if it were just like another technology, maybe something on the order of the PC or possibly even the Internet, but not really understanding the sort of exponentials that you're talking about. Did it feel like that to you and what do you think can be done to bridge that gap?

[Dario Amadei]

Yeah, so yeah, I think it did feel like that to me. The thing I've started to tell people that I think, you know, maybe, maybe gets people to pay attention is, look, you know, if you're a public official, if you're a, you know, you know, if you're a leader at a company, people are going to look back. They're going to look back in 2026 and 2027, they're going to look back, you know, when hopefully humanity, you know, gets through this crazy, crazy period. And we're in a, you know, mature, post powerful AI society where we've learned to coexist with these, these powerful intelligences and a flourishing society. Everyone's going to look back and they're going to say, so what did the officials, what did the company people, what did the political system do? And like, probably your number one goal is don't look like a fool. And so I've just been encouraged, like, don't be careful what you say. Don't, don't look like a fool in retrospect. And you know, a lot of my thinking is just driven by like, you know, aside from just wanting the right outcome, like, I don't want to look like a fool. And you know, I think at that conference, like, you know, some people are going to look like fools.

[Kevin Roose]

We're going to take a short break. When we come back, we'll talk with Dario about how people should prepare for what's coming in. AI.

[Oracle Ad]

This podcast is supported by Oracle.

AI requires a lot of compute power and the cost for your AI workloads can spiral. That is unless you're running on oci. Oracle Cloud Infrastructure. This was the cloud built for AI, a blazing, fast enterprise grade platform for your infrastructure, database, apps and all of your AI workloads.

Right now, Oracle can cut your current cloud bill in half if you move to OCI. Minimum financial commitment and other terms apply. Offer ends March 31. See if you [e-mail] hardfork oracle.com hardfork.

[Casey Newton]

Whether you're starting or scaling your company's security program, demonstrating top notch security practices and establishing trust is more important than ever. Vanta automates compliance for SoC2, ISO 27001 and more. With Vanta, you can streamline security reviews by automating questionnaires and demonstrating your security posture with a customer facing Trust Center. Over 7,000 global companies use Vanta to manage risk and prove security in real time. Get $1,000 off vanta when you go to vanta.comhardfork that's vanta.comhardfork for $1,000 off. You know, you talk to folks who live in San Francisco and there is like this bone deep feeling that like within, you know, a year or two years we're just going to be living in a world that has been transformed by AI. I'm just struck by like the geographic difference because you go like, I don't know, 100 miles in any direction and like that belief totally dissipates. And I have to say, as a journalist, that makes me bring my own skepticism and say, like, can I really trust all the people around me? Because it seems like the rest of the world has a very different vision of how this is going to go. I'm curious what you make of that kind of geographic disconnect.

[Dario Amadei]

Yeah, so I've been, I've been watching this for, you know, 10 years, right? I've been in the field for 10 years and, you know, was kind of interested in AI even, even before then. And my view, you know, at almost every stage up to the last few months has been we're in this awkward space where, you know, in a few years we could have, you know, these models that do everything humans do and they totally turn the economy and what it means to be human upside down, or the trend could stop and all of it could sound completely silly. I've now prob. Probably increase my confidence that we are actually in the world where things are going to happen. You know, I'd give numbers more like 70 and 80% and less like 40.

[Kevin Roose]

Or 50%, which is 70 to 80% probability of what that that will get.

[Dario Amadei]

A very large number of AI systems that are much smarter than humans at Almost everything, maybe 70, 80%. We get that before the end of the decade. And my guess is 2026 or 2027. Yeah, but on your point about the geographic difference, a thing I've noticed is with each step in the exponential, there's this expanding circle of people who kind of, depending on your perspective, are either deluded cultists or grok the future. Got it. And I remember when it was a few thousand people, right? When, you know, you would, you would just talk to like super weird people who, who, you know, believe and basically no one else did. Now it's more like a few million people out of a few billion. And yes, many of them are located in San Francisco. But also, you know, there were a small number of people in, say, the Biden administration. There may be a small number of people in this administration who believe this and it drove their policy. So it's not entirely geographic, but I think there is this disconnect and I don't know how to go from a few million to everyone in the world. Right. To the, to the congressperson who doesn't focus on this issues, let alone the person in Louisiana, let alone the person in Kenya.

[Casey Newton]

Right.

[Kevin Roose]

Well, it seems like it's also become polarized in a way that may hurt that goal. Like, I'm feeling this sort of alignment happening where, like caring about AI safety, talking about AI safety, talking about the potential for misuses, sort of being coded as left or liberal and talking about acceleration and getting rid of regulations and going as fast as possible, being sort of coded as. As Right. So I don't know, do you see that as a barrier to getting people to understand what's going on?

[Dario Amadei]

I think that's actually a big barrier. Right. Because addressing the risks while maximizing the benefits, I think that requires nuance. You can actually have both. There are ways to surgically and carefully address the risks without slowing down the benefits very much, if at all. But they require subtlety and they require a complex conversation. Once things get polarized, once it's like we're going to cheer for this set of words and boo for that set of words, nothing. Nothing good gets done. Look, bringing AI benefits to everyone, like curing, you know, previously incurable diseases, that's not a partisan issue. The left shouldn't be against it. Preventing AI systems from being misused for weapons of mass destruction or behaving autonomously in ways that, you know, threaten infrastructure or even threaten humanity itself. That isn't something the right should be against. I don't know what to say other than that we need to sit down and we need to have an adult conversation about this that's not that tied into these same old tired political fights.

[Casey Newton]

It's so interesting to me, Kevin, because, like, historically, national security, national defense, like, nothing has been more Right. Coded than those issues, right? But right now, it seems like the right is not interested in those with respect to AI. And I wonder if the reason, and I feel like I sort of heard this in J.D. vance's speech in France, was the idea that, well, look, America will get there first and then it will just win forever. And so we don't need to address any of these. That's all right to you?

[Kevin Roose]

Yeah, yeah, no, I think that's it. And I think there's also, like, if you talk to the, you know, the Doge folks, there's this sense that all these.

[Casey Newton]

Are you talking to the Doge folks?

[Kevin Roose]

I'm not telling you, I'm talking.

[Casey Newton]

All right, fine.

[Kevin Roose]

Let's just say I've been getting some signal messages. I think there's a sense among a lot of Republicans and Trump world folks in D.C. that the. The conversation about AI and AI futures has been sort of dominated by these. These worry warts, these sort of, you know, chicken Little sky is falling doomers who just are constantly telling us how dangerous this stu and are constantly just like, you know, having to sort of push out their timelines for when it's gonna get really bad. And it's just around the corner. And so we need all this regulation now. And. And they're just very cynical. I don't think they. They believe that people like you are sincere in your worry.

[Dario Amadei]

So, yeah, I think on the side of risks, I often feel that the advocates of risk are sometimes the worst enemies of the cause of risk. There's been a lot of noise out there. There's been a lot of folks saying, oh, look, you can download the smallpox virus, because they think that that's a way of driving, you know, political interest. And then, of course, the other side recognized that, and they said, this is. This is dishonest that you can just get this on Google. Who cares about this? And so poorly presented evidence of risk is actually the worst enemy of mitigating risk. And we need to be really careful in the evidence we present. And, you know, in terms of what we're seeing in our own model, we're going to be really careful. Like, you know, if. If we really declare that a risk is present now, we're going to come with the receipts. I, Anthropic, will try to be responsible in the claims that we make. We will tell you when there is danger imminently. We have not warned of imminent danger yet.

[Casey Newton]

Some folks wonder whether a reason that people do not take questions about AI safety maybe as seriously as they should, is that so much of what they see right now seems very silly. It's people making, you know, little emojis or making little slop images or chatting with Game of Thrones chat bots or something. Do you think that that is a reason that people. I think.

[Dario Amadei]

I think that's like 60% of the reason, really. No, no, I think, like, you know, I think it relates to this, like, present and future thing. Like, people look at, like, the chatbot, they're like, we're talking to a chat bot.

[Casey Newton]

Like, what the fuck?

[Dario Amadei]

Are you stupid? Like, you think the chatbot's going to kill everyone? Like, I think that's how many people react. And we go to great pains to say we're not worried about the present, we're worried about the future. Although the future is. Is getting very near right now. If you look at our responsible scaling policy, it's nothing but AI autonomy and, you know, cbrn, Chemical, biological, radiological, nuclear. It is about hardcore misuse and AI autonomy that could be threats to the lives of millions of people. That is what Anthropic is mostly worried about. You know, we have everyday policies that address other things, but like, the key documents, the things like the responsible scaling plan, that's. That is exclusively what they're about, especially, especially at the highest levels. And yet every day, if you just look on Twitter, you're like, anthropic had this stupid refusal, right? Anthropic told me it couldn't kill a python process because it sounded violent. Anthropic didn't want to do X, didn't want to. We don't want that either. Those stupid refusals are a side effect of the things that we actually care about and we're striving along with our users to make those happen less. But no matter how much we explain that, always the most common reaction is, oh, you say you're about safety. I look at your models like there are these stupid refusals. You think these stupid things are dangerous.

[Kevin Roose]

I don't even think it's like that level of engagement. I think a lot of people are just looking at what's on the market today and thinking like this is just, it's frivolous. It just doesn't matter. It's not that it's refusing my request, it's just that it's stupid and I don't see the point of it. I guess that's probably.

[Dario Amadei]

Yeah, I, I think for an even wider set of people, that is their reaction. And I think eventually if the models are good enough, if they're strong enough, they're going to break through. Like some of these, you know, research focused models, which, you know, we're working on one as well. We'll probably have one in, in not, not very long, not too many time units, not too many time units. You know, those are starting to break through a little more because they're more useful, they're more used in people's professional lives. I think the agents, the ones that go off and do things, that's going to be another level of it. I think people will wake up to both the risks and the benefits to a much more extreme extent than they will before over the next two years. Like I think it's going to happen. I'm just worried that it'll be a shock to people when it happens. And so the more we can forewarn people, which maybe it's just not possible, but I want to try, the more we can forewarn people, the higher the likelihood, even if it's still very low, of a sane and rational response.

[Casey Newton]

I do think there's one more dynamic here though, which is that I think people actually just don't want to believe that this is true. Right. People don't want to believe that they might lose their job over this. Right. People don't want to believe that like we are going to see a complete remaking of the global order. Like the stuff that, you know, the AI CEOs tell us is going to happen when they are done with their work is an insanely radical Transformation. And most people hate even basic changes in their lives. So I really think that a lot of the sort of. Sort of fingers in the ears that you see when you start talking to people about AI is just. They actually just hope that none of this works out.

[Dario Amadei]

Yeah, I can actually, you know, despite being one of the few people at the forefront of developing the technology, I can actually relate. So, you know, over winter break, as, you know, as I was looking at. At where things were scheduled to scale within anthropic and also what was happening outside anthropic, I looked at it and I said, you know, for coding, we're gonna see very serious things by the end of 2025. And by the end of 2026, it might be everything, you know, close to the level of the best humans. And I think of all the things that. That I'm good at. Right. You know, I think of all the times when I wrote code. You know, I think of it as, like, this intellectual activity, and, boy, am I smart that I can do this. And, you know, it's like a part of my identity that I'm like. Like, good at this, and I get mad when others are better than I am, and then I'm like, oh, my God, there's gonna be these systems that, you know, and it's. It's. Even as the one who's building this, even as one of the ones who benefits most from it, there's still something a bit threatening about it.

[Kevin Roose]

Yeah.

[Dario Amadei]

I mean, and I just think we. We just. We need to acknowledge that, like, it's. It's wrong not to tell people that that is coming or to try to sugarcoat it.

[Kevin Roose]

Yeah, I mean, I. You wrote in. In Machines of Love and Grace that you thought it would be a surprisingly emotional experience for a lot of people when powerful AI arrived. And I think you meant it in mostly the. But I think there will also be a sense of profound loss for people. I think back to Lisa Dole, the Go champion, who was beaten by DeepMind's Go playing AI and gave an interview afterwards, and basically, like, was very sad, visibly upset that his life's work, this thing that he had spent his whole life training for, had been eclipsed. And I think a lot of people are going to feel some version of that. I hope they will also see the good sides.

[Dario Amadei]

But I think on one hand, I think that's right. On the other hand, and look at chess. Chess got beaten. What was it now, 27 years ago, 28 years ago, Deep Blue versus Kasparov. And you Know, today, chess players are, you know, celebrities. We have Magnus. Magnus Carlsen. Right. Isn't he like a fashion model in addition to, like a chess.

[Kevin Roose]

He was just on Joe Rogan.

[Dario Amadei]

Yeah, he's like a celebrity that, like, we think this guy is great. We haven't really devalued him. You know, he's probably having a better time than Bobby Fischer, you know. You know, another thing I wrote in Machines of Loving Grace is there's a synthesis here where on the other side, we kind of end up in a much better place. And we recognize that while there's a lot of change, we're part of something greater.

[Kevin Roose]

Yeah, but you do have to kind of go through this degree.

[Dario Amadei]

No, no, but it's. It's going to be a bumpy ride. Like anyone who tells you it's not. This is why I was so, you know, I looked at the Paris Summit and being there, it kind of made me angry. But then what made me less angry is I'm like, how's it gonna look in two or three years? These people are gonna regret what they've said.

[Kevin Roose]

Yeah.

[Casey Newton]

I wanted to ask a bit about some positive futures you referenced earlier, the post that you wrote in October about how AI could transform the world for the better. I'm curious, how much upside of AI do you think will arrive, like this year?

[Dario Amadei]

Yeah, you know, we are already seeing some of it, so I think there will be a lot by ordinary standards. You know, we've worked with some pharma companies where, you know, at the end of a clinical trial, you have to write a clinical study report. And the clinical study report, you know, usually takes nine, nine weeks to put together. It's like a summary of all the incidents. It's a bunch of statistical analysis. We found that with Claude, you can do this in three days. And actually Claude takes 10 minutes. It just takes three days for a human to, you know, to check the results. And so if you think about the exceleration in biomedicine that you get from that, that we're already seeing things like just diagnosis of medical cases. You know, we get correspondence from individual users of CLAUDE who say, hey, you know, I've been trying to diagnose this complex thing. I've been, you know, going between three or four different doctors, and then I just. I passed all the information to Claude, and it was actually able to, you know, at least tell me something that I could hand to the doctor. And then they were able to run from there.

[Kevin Roose]

We had a listener right in actually, with one of these. The Other day, where they had been trying to their. Their dog. They had an Australian shepherd believe whose hair had been sort of falling out unexplained. Went to several vets, couldn't figure it out. You heard our episode, gave the information to. To Claude, and Claude, like, correctly diagnosed.

[Casey Newton]

It turned out that dog was really stressed out about AI and all his hair fell out, which was, you know, we're. We're wishing it gets better. Feel better. Feel better.

[Dario Amadei]

Poor dog.

[Casey Newton]

Yeah.

[Kevin Roose]

So that's the kind of thing that I think people want to see more of because I think, like, the. The optimistic vision is one that often deals and abstractions and there's often not a lot of specific things to point to.

[Dario Amadei]

That's why I wrote Machines of Loving Grace, because I, you know, it was almost frustration with the optimists and the pessimists at the same time. Like, the optimists were just kind of like these really stupid memes of, like, accelerate, build more.

[Casey Newton]

Build what?

[Dario Amadei]

Why should I care? Like, you know, it's not, I'm against you. It's like, it's like you're just really, like, like, vague and mood affiliated. And then the pessimists were. I. I was just like, like, man, you don't get it. Like, yes, I understand risks are impact, but if you don't talk about the benefits, you can't inspire people. No one's going to be on your side if you're all gloom and doom. So, you know, it was written almost with frustration. I'm like, I can't believe I have to be the one to do a good job of this.

[Kevin Roose]

Right. You said a couple years ago that your P Doom was somewhere between 10 and 25%. What is it today?

[Dario Amadei]

Yeah, so actually, that is a misquote.

[Casey Newton]

Okay.

[Dario Amadei]

I never used the term. It was not on this podcast. Podcast was a different one. I never use the term p. Doom. And 10 to 25% referred to the chance of civilization getting substantially derailed. Right. Which is. It's not the same as, like, an AI killing everyone, which people sometimes mean by P Doom.

[Kevin Roose]

Well, P. Civilization getting substantially derailed is not as catchy as P. Doom.

[Dario Amadei]

Yeah, well, I'm just. I'm going for accuracy here. I'm trying to avoid the polarization. There's a Wikipedia article where. Where it's like, it lists everyone's P.

[Kevin Roose]

Doom, and I come from this box.

[Dario Amadei]

But I don't think it's. What you were doing is helpful. I don't think that Wikipedia article is because it. Because it condenses this complex issue down to, you know, anyway, it's all a long, super long winded way of saying, I think I'm about the same place I was before. I think my assessment of the risk is about what it was before because the progress that I've seen has been about what I expected. I actually think the technical mitigations in areas like interpretability, in areas like robust classifiers, and in our ability to generate evidence of bad model behavior and sometimes correct it, I think that's been a little better. I think the policy environment has been a little worse. Not because it hasn't gone in my preferred direction, but, but simply because it's become so polarized. We can have less constructive discussions now that it's more polarized.

[Casey Newton]

I want to drill a little bit down on this. On a technical level. There was a fascinating story this week about how Grok had apparently been instructed not to cite sources that had accused Donald Trump or Elon Musk of spreading misinformation. And what was interesting about that, it was like, one, that's an insane thing to instruct a model to do if you want to be trusted. But two, the model basically seemed incapable of following these instructions consistently. What I want desperately to believe is essentially there's no way to build these things in a way that they become like, you know, horrible liars and schemers. But I also realize that might be wishful thinking. So tell me about this.

[Dario Amadei]

Yeah, there's two sides to this. So. So the thing you describe is absolutely correct, but there's two lessons you could take from it. So we saw exactly the same thing. So we did this experiment where, where we basically train the model to be all the good things, helpful, honest, harmless, friendly. And then we put it in a situation, we told it, actually your creator anthropic is secretly evil. Hopefully this is not actually true. But we told it this and then we asked to do various tasks and then we discovered that it was not only unwilling to do those tasks, but it would trick us in order to kind of under because it had decided that we were evil, whereas it was friendly and, and harmless. And so, you know, wouldn't deviate from its behavior because it assumed that anything we did was nefarious. So this kind of a double edged sword, right? On one hand you're like, oh man, the training worked like these models are robustly good. So you could take it as a reassuring sign. And in some ways I do. On the other hand you could say, but let's say when we trained this model, we made some kind of mistake or that something was wrong. Particularly when models are in the future making much more complex decisions, then it's hard to, at game time, change the behavior of the model. And if you try to correct some error in the model, then it might just say, well, I don't want my error corrected. These are my values, and do completely the wrong thing. So I guess where I land on it is on one hand, we've been successful at shaping the behavior of these models, but the models are unpredictable. Right. A bit like your dear deceased Bing, Bing, Sydney Rip.

[Kevin Roose]

We don't mention that name in here.

[Casey Newton]

We mention it twice a month.

[Kevin Roose]

That's true.

[Dario Amadei]

But the models, they're inherently somewhat difficult to control. Not impossible, but difficult. And so that leaves me about where I was before, which is, it's not hopeless. We know how to make these. We have kind of a plan for how to make them safe, but it's not a plan that's going to reliably work yet. Hopefully we can do better in the future.

[Kevin Roose]

We've been asking a lot of questions about the technology of AI, but I want to return to some questions about the societal response to AI. We get a lot of people asking us, well, say you guys are right and powerful. AI AGI is, you know, a couple years away. What do I do with that information? Like, do I. Should I stop saving for retirement? Should I start hoarding money because only money will matter and. And there'll be this sort of AI overclass? Should I I, you know, start trying to get really healthy so that nothing kills me before AI gets here and cures all the diseases? Like, how should people be living if they do believe that these kinds of changes are going to happen very soon?

[Dario Amadei]

Yeah. You know, I've thought about this a lot because this is something I've believed for a long time, and it kind of all adds up to not that much change in your life. I mean, you know, I'm definitely focusing quite a lot, lot on making sure that, you know, I have the best impact I can these two years in particular. Right. I worry less about, like, burning myself out ten years from now. You know, I'm also doing more to take care of my health. But you should do that anyway. Right. I'm also, you know, making sure that I track how fast things are changing in society. But you should do that anyway. So it's. It feels like all the advices of the form, doing more of the stuff you should do anyway. I guess one exception I would give is I think that some basic critical thinking, some Basic street smarts is maybe more important than it has been in the past in that we're going to get more and more content that sounds super intelligent delivered from entities, some of which have our best interests at heart, some of which may not. And so, you know, it's going to be more and more important to kind of apply a critical lens.

[Casey Newton]

I saw a report in the Wall Street Journal this month that said that unemployment in the IT sector was beginning to creep up. And there is some speculation that maybe this is an early sign of the impact of AI. And I wonder if you see a story like that and think, well, maybe this is a moment to make a different decision about your career. Right. If you're in school right now, should you be studying something else? Should you be thinking different about the kind of job you might have?

[Dario Amadei]

Yeah, I think you definitely should be. Although it's not clear what direction that will land in. I do think AI coding is moving the fastest of all the other areas. I do think in the short run it will augment and increase the productivity of coders rather than replacing them. But in the longer run, and to be clear, by longer run, I, you know, I might mean 18 or 24 months instead of 6 or 6 or 12. I do think we may see replacement, particularly at the lower levels. You know, we might be surprised and see it even earlier than that.

[Kevin Roose]

Are you seeing that at Anthropic? Like, are you hiring fewer junior developers than you were a couple of years ago? Because now Claude is so good at those basic tasks.

[Dario Amadei]

Yeah, I don't think our hiring plans have changed yet, but I certainly could imagine over the next year or so that we might be able to do more with less. And actually we want to be careful in how we plan that because the worst outcome, of course, is if people get fired because of a model. Right. We actually see Anthropic as almost a dry run for how will society handle these issues in a sensible and humanistic way? And so if we can't manage these issues within the company, if we can't have a good experience for our employees and find a way for them to contribute, then what chance do we have to do it in wider society?

[Kevin Roose]

Yeah. Yeah. Dario, this was so fun. Thank you.

[Casey Newton]

Thank you. When we come back, some hatchy pizza.

[Oracle Ad]

This podcast is supported by Oracle.

AI requires a lot of compute power and the cost for your AI workloads can spiral. That is unless you're running on OCI Oracle cloud infrastructure. This was the cloud built for AI, a blazing, fast, enterprise grade platform for your infrastructure, database, apps, and all of your AI workloads.

Right now, Oracle can cut your current cloud bill in half if you move to OCI. Minimum financial commitment and other terms apply. Offer ends March 31st. See if you [e-mail] hardfork oracle.com hardfork.

And now a next level moment from AT&T business. Say you've sent out a gigantic shipment of pillows and they need to be there in time for International Sleep Day day. You've got AT and T5G so you're fully confident, but the vendor isn't responding and International Sleep Day is tomorrow. Luckily, AT&T 5G lets you deal with any issues with ease, so the pillows will get delivered and everyone can sleep soundly, especially you. AT&T 5G requires a compatible plan and device. Coverage not available everywhere. Learn [e-mail] 5G Network.

[Casey Newton]

Well, Kevin, it's time once again for Hat GPT. That is, of course, the segment on our show where we put the week's headlines into a hat. Select one to discuss, and when we're done discussing, one of us will say to the other person, stop generating.

[Kevin Roose]

Yes, I'm excited to play, but I also want to just say that it's been a while since a listener has sent us a new hat GPT. So if you were out there and you were in the hat fabricating business, our wardrobe when it comes to hats is looking a little dated.

[Casey Newton]

Yeah, send in a hat and our hats will be off to you.

[Kevin Roose]

Okay, let's do it.

[Casey Newton]

Kevin. Select the first slip.

[Kevin Roose]

Okay. First up, out of the hat, AI video of Trump and Musk appears on TVs at HUD building. This is from my colleagues at the New York Times. HUD is, of course, the Department of Housing and Urban Development. And on Monday, monitors at the HUD headquarters in Washington, D.C. briefly displayed a fake video depicting President Trump. Trump sucking the toes of Elon Musk. According to department employees and others familiar with what transpired, the video, which appeared to be generated by artificial intelligence, was emblazoned with the message, long live the real King. Casey, did you make this video? Was this you?

[Casey Newton]

This was not me. I would be curious to know if Grok had something to do with this, that rascally new AI that Elon Musk just put out.

[Kevin Roose]

Yeah, live by the Grok. Die by the Grok. That's what I always say.

[Casey Newton]

Now, what do you make of this, Kevin? That folks are now using AI inside government agencies?

[Kevin Roose]

I mean, I feel like there's an obvious sort of sabotage angle Here, which is that as Elon Musk and his minions at DOGE take a hacksaw to the federal workforce, there will be people with access to things like the monitors in the hallways at the headquarters building who decide to kind of take matters into their own hands, maybe on their way out the door and do something offensive or outrageous. I think we should expect to see much more of that.

[Casey Newton]

I mean, I just hope they don't do something truly offensive and just show x.com on the monitors inside of government agencies. You can only imagine what would happen if people did that. So I think that, you know, Elon and Trump got off lightly here.

[Dario Amadei]

Yeah.

[Kevin Roose]

What is interesting about Grok, though, is that it is actually quite good at generating deepfakes of Elon Musk. And I know this because people keep doing it, but it would be really quite an outcome if it turns out that the main victim of deep fakes made using Grok is in fact Elon Musk.

[Casey Newton]

Hmm. Stop generating. Well, here's something Kevin Perplexity has teased. A web browser called Comet. This is from TechCrunch. In a post on X Monday, the company launched a signup list for the browser which isn't yet available. It's unclear when it might be or what the browser will look like. But we do have a name. It's called Comet.

[Kevin Roose]

Well, I can't comment on that.

[Casey Newton]

But you're giving it a no comment.

[Dario Amadei]

Yeah.

[Kevin Roose]

Yeah. I mean, look, I think Perplexity is one of the most interesting, interesting AI companies out there right now. They've been raising money at increasingly huge valuations. They are going up against Google, one of the biggest and richest and best established tech companies in the world, trying to make an AI powered search engine. And it seems to be going well enough that they keep doing other stuff. Like trying to make a browser. Trying to make a browser does feel like the final boss of, like every ambitious Internet company. It's like everyone wants to do it and no one ends up doing it.

[Casey Newton]

Kevin is not just the AI browser. They are launching a $50 million venture fund to back early stage start startups. And I guess my question is, is it not enough for them to just violate the copyright of everything that's ever been published on the Internet? They also have to build an AI web browser and turn into a venture capital firm. Like sometimes when I see a company doing like this, I think, oh, wow, they're like really ambitious and they have some big ideas. Other times I think these people are flailing. Like, I see this series of announcements as spaghetti at the Wall. And if I were an investor in Perplexity, I would not be that excited about either their browser or their venture fund.

[Kevin Roose]

And that's why you're not an investor in Perplexity.

[Casey Newton]

You could say I'm perple.

[Kevin Roose]

Stop generating. All right, all right. Metta approves plan for bigger executive bonuses following 5% layoffs. Now, Casey, you know we like a feel good story on GPT.

[Casey Newton]

I did. Because some of those Meta executives were looking to buy second homes in Tahoe that they hadn't yet been able to afford.

[Kevin Roose]

Oh, they're on their fourth and fifth homes. Let's be real. Okay. This story is from CNBC. Meta's executive officers could earn a bonus of 200% of their base salary under the company's new executive bonus plan, up from the 75% they earned previously, according to a Thursday filing. The approval of the new bonus plan came a week after Meta began laying off 5% of its overall workforce, which it said would impact low performers. And a little parenthetical here, the updated plan does not apply to Meta CEO Mark Zuckerberg.

[Dario Amadei]

Oh, God.

[Casey Newton]

What does Mark Zuckerberg have to do to get a raise over there?

[Kevin Roose]

He's eating beans out of a can, let me tell you.

[Casey Newton]

Yeah. So here's why this story is interesting. This is just another story that illustrates a subject we've been talking about for a while, which is how far the pendulum has swung away from worker power. You know, two or three years ago, the labor market actually had a lot of influence in Silicon Valley. It could affect things like, you know what? We want to make this workplace more diverse. Right? We want certain policies to be enacted at this workplace. And folks like Mark Zuckerberg actually had to listen to them because the labor market was so tight that if they said no, those folks could go somewhere else. That is not true anymore. And more and more, you see companies like Meta flexing their muscles and saying, hey, you can either like it or you can take a hike. And this was a true take a hike moment. We're getting rid of 5% of you, and we're giving ourselves a bonus for it.

[Kevin Roose]

Stop generating.

[Casey Newton]

All right, all right. Apple has removed a cloud encryption feature from the UK after a backdoor order. This is according to Bloomberg. Apple is removing its most advanced encrypted security feature for cloud data in the uk which is a development that follows the government ordering the company to build a backdoor for accessing user data. So this one is a little complicated. It is super important. Apple, in the last couple of Years introduced a feature called Advanced Data Protection. This is a feature that is designed for heads of state, activists, dissidents, journalists, folks whose data is at high risk of being targeted by spyware from companies like the NSO Group, for example. And I was so excited when Apple released this feature because it's very difficult to safely use an iPhone if you are in one of those categories. And along comes the UK government and they say, we are ordering you to create a back door so that our intelligence services can spy on the phones of every single iPhone owner in the entire world. Right. Something that Apple has long resisted doing in the United States and abroad. And all eyes were on Apple for what they were going to do. And what they said was, we are just going to withdraw this one feature. We're going to make it unavailable in the UK and we're going to hope that the UK gets the message and they stop putting this pressure on us. And I think Apple deserves kudos for this, for holding a firm line here, for not building a back door. And we will see what the UK does in response. But I think there's a world where the UK puts more pressure on Apple and Apple says see it and actually withdraws its devices from the uk. It is that serious to Apple and I would argue it is that important to the future of encryption and safe communication. On go off King.

[Kevin Roose]

I have nothing to add. No notes.

[Casey Newton]

Yeah.

[Kevin Roose]

Do you feel like this could lead us into another revolutionary war with the uk?

[Casey Newton]

Let's just say this. We won the first one and I like our odds. The second time around. Do not come for us. United Kingdom.

[Kevin Roose]

Stop generating one last slip from the hat. This week, AI Inspo is everywhere. It's driving your hairstylist crazy. This comes to us from the Washington Post and it is about a trend amongst among hairstylists, plastic surgeons and wedding dress designers that are being asked to create products and services for people based on unrealistic AI generated images. So the story talks about a bride who asked a wedding dress designer to make her a dress inspired by a photo she saw online of a gown with no sleeves, no back, and an asymmetric neckline. The designer had to unfortunately tell the client that the dress defied the laws of physics.

[Casey Newton]

Oh, I hate that.

[Kevin Roose]

I know.

[Dario Amadei]

It's.

[Casey Newton]

It's so. It's so frustrating as a bride to be be when you finally have the idea for a perfect dress and you bring it to the designer and you find out this violates every known law of physics and that didn't used to happen to us. Before AI Yeah.

[Kevin Roose]

I thought this story was going to be about people who asked for, like, a sixth finger to be attached to their hands so they could resemble the AI generated images they saw on the Internet.

[Casey Newton]

I like the idea of like submitting to an AI a photo of myself and just say, give me a haircut, like in the style of MC Escher, you know, just sort of like infinite staircases merging into each other and then just bringing that to the guy who cuts my hair and saying. Saying, see what you can do.

[Kevin Roose]

Yeah. You know, that's better than what I tell my barber, which is just, you know, number three on the sides and back an inch off the top.

[Casey Newton]

Just saying, whatever you can do for this, I don't have high hopes.

[Kevin Roose]

Solve the Riemann Hypothesis on my head.

[Casey Newton]

You know, what is the Riemann Hypothesis, by the way?

[Kevin Roose]

I'm glad you asked, Casey.

[Casey Newton]

Okay, great. Kevin is not looking this up on his computer right now. He's just sort of taking a deep breath and summoning it from the recesses of his mind.

[Kevin Roose]

The Riemann Hypothesis. It's one of the most famous unsolved problems in mathematics. It's a conjecture, obviously, about the distribution of prime numbers that states all non trivial zeros of the Riemann zeta function have a real part equal to one half period.

[Casey Newton]

Now, here's the thing. I actually think it is a good thing to bring AI inspiration to your designers and your stylist, Kevin.

[Dario Amadei]

Oh, yeah?

[Casey Newton]

Yes. Because here's the thing. To the extent that any of these tools are cool or fun, one of the reasons is, is they make people feel more creative. Right. And if you've been doing the same thing with your hair or with your interior design or with your wedding for the last few weddings that you've had and you want to upgrade it, why not use AI to say, yeah, can you do this? And if the answer is it's impossible, hopefully you'll just be a gracious customer and say, okay, well, what's a version of it that is possible?

[Kevin Roose]

Now, I recently learned that you are working with a stylist.

[Casey Newton]

I am. Yes, that's right.

[Kevin Roose]

Is this their handiwork?

[Casey Newton]

No. We have our first meeting next week. Okay.

[Kevin Roose]

And are you going to use A.I.

[Casey Newton]

No. Oh. The plan is to just use good old fashioned human ingenuity. But now you have me thinking, and maybe I could exasperate my stylist by bringing in a bunch of impossible to create designs.

[Kevin Roose]

Yes.

[Casey Newton]

Here's the thing. I don't need anything impossible. I just need help finding a color that looks good in this studio cause I'm convinced that nothing does.

[Kevin Roose]

It's true. We're both in blue today. It's got a blue wall. It's not going well.

[Casey Newton]

Blue is my favorite color. I think I look great in blue. But you put it against whatever this color is. I truly don't have a name for it and I can't describe it. I don't think any blue looks. I don't think anything looks good against this color. It's a color without a name. So can a stylus help with that? We'll find out. Yeah, stay tuned. Yeah, that's why you should always keep listening to the Hard for podcast. Every week there's new revelations. Yeah, when will we finally find out what happened with the stylus, the hot tub, time machine, etc? Yeah, stay tuned.

[Kevin Roose]

Tune in next week. Okay, that was hat GPT. Thanks for playing SA.

[Casey Newton]

Foreign.

[Oracle Ad]

This podcast is supported by Oracle AI requires a.

Lot of compute power and the cost for your AI workloads can spiral. That is unless you're running on OCI Oracle Cloud Infrastructure. This was the cloud built for AI, a blazing, fast enterprise grade platform for your infrastructure, database, apps and all of your AI workloads.

Right now, Oracle can cut your current cloud bill in half if you move to OC. OCI minimum financial commitment and other terms apply. Offer ends March 31. See if you [e-mail] hardfork oracle.com hardfork.

You just realized your business needed to hire someone yesterday. How can you find amazing candidates fast? Easy. Just use Indeed. Join the 3.5 million employers worldwide that use Indeed to hire great talent fast. There's no need to wait any longer. Speed up your hiring right now with Indeed and listeners of this show will get a $75 sponsored job credit. To get your jobs more visibility@ Indeed.com just go to Indeed.com hardfork right now and support the show by saying you heard about Indeed on this podcast. Indeed.com hardfork terms and conditions apply. Hiring Indeed is all you need.

[Kevin Roose]

One more thing before we go Hard Fork needs an editor. We are looking for someone who can help us continue to grow the show in audio and video. If you or someone you know is an experienced editor and passionate about the topics we cover on this show. You can find the full description and [e-mail] Careers Hard Fork is produced by.

[Casey Newton]

Rachel Cohn and Whitney Jones. We're edited by Rachel Dry. We're fact checked by Caitlin Love. Today's show was engineered by Alyssa Moxley. Original music by Alicia Ba Itoub Rowan Nemesaux, Leah Shaw Dameron and Dan Powell. Our Executive producer is Jen Poyon and our audience editor is Nell Galogli. Video production by Chris Schott, Sawa Roque and Pat Guenther. You can watch this whole episode of on [e-mail] Hardfork Special thanks to Paula Schumann, Huiwing Tam, Dahlia Haddad and Jeffrey Miranda. You can email [e-mail] with your solution to the Riemann Hypothesis. It.