How AI safety took a backseat to military money

Summary

Decoder with Nilay Patel (Guest Host: Hayden Field)

Episode: How AI safety took a backseat to military money

Date: September 25, 2025

---

Overview

In this episode of Decoder, guest host Hayden Field (senior AI reporter at The Verge) speaks with Heidi Klaff, Chief AI Scientist at the AI Now Institute and former OpenAI systems safety engineer. Together, they explore the dramatic pivot among leading AI companies from championing ethics and safety to aggressively pursuing military contracts. They discuss the relaxation of bans on military AI applications by companies like OpenAI and Anthropic, the motivations and risks behind this shift, and what it means for AI safety both technically and politically—especially as these models become entangled with high-risk and high-stakes defense operations.

---

Key Discussion Points & Insights

1. Rapid Shift in AI Company Policies on Military Use

(04:16–07:27)

• OpenAI and Anthropic drop military use bans: Both had long-standing bans on military and warfare applications, but rolled them back in early 2024 in anticipation of major Defense Department contracts.
• "Clean slate" approach: Companies have not publicly addressed their prior warnings about LLMs being unsafe for military use, instead reframing their missions to align national security and AI safety.

“It was almost like a clean slate was being created where they behaved as if this was always aligned with their mission.”
—Heidi Klaff (06:11)

• Financial and strategic incentives: The pivot is aimed at securing government subsidies and lucrative contracts amid the unprofitable reality of current generative AI business models.
• National security narrative: Companies adopt the rhetoric of a US–China arms race to justify rapid AI deployment while sidestepping critical safety and security scrutiny.

“Now their reliance on this narrative of a US China AI arms race ... allows them to sort of avoid safety and security scrutiny within military applications.”
—Heidi Klaff (07:15)

---

2. Risks in Military Adoption of Commercial Foundation Models

(07:27–10:00)

• Senator Warren’s concerns about XAI’s DoD contract: Raised the issue that XAI, and similar companies, lack adequate safety audits before deploying to military contexts.
• Expanded risk surface: Use of commercial models in military settings expands attack vectors and threatens national security due to inadequate vetting and traceability.
• Data security: Military applications risk exposing sensitive personal or private data used to train these models, which could be compromised or manipulated by adversaries.

“Commercial models are unvetted. They don’t have a supply chain that follows the typical military supply chain, and they can be compromised in a lot of ways.”
—Heidi Klaff (08:17)

---

3. Military Contracts as a Financial Lifeline and Regulatory Workaround

(10:00–15:25)

• Financial imperative: Pre-profit AI companies chase military contracts for revenue and perceived legitimacy.
• Bypassing standards: Traditional military procurement standards—rigorous, multi-year, safety-focused—are not met by commercial generative AI models.
• Laxer evaluation: Newer deals push military systems built on commercial models that are much less robust or secure than previous defense technologies.

“The thing with AI systems ... is they do not meet the sort of very basic threshold that’s typically expected for a military system.”
—Heidi Klaff (11:35)

• Explanation of military procurement: Normally requires air-gapped, traceable supply chains and extensive testing. Recent shortcuts for AI highlight an erosion of these standards.

---

4. Are “Gov” AI Products More Secure? Not really.

(18:54–22:50)

• Air-gapping and fine-tuning aren’t enough: While government models may be somewhat more isolated, their foundational risks remain due to training on large, public datasets.
• “Sleeper agents” and prompt attacks: Models could be compromised at the training stage, with adversarial data poisoning leading to unpredictable responses later.

“A lot of research has shown that … you can implement what’s called a sleeper agent, which is given a specific prompt or a command. It will then behave in a harmful way … based on something that was implemented in the training data.”
—Heidi Klaff (20:32)

• The myth of control: Once the military takes delivery, neither the vendor’s terms of service nor their ethical promises reliably determine how the technology is used.

---

5. Ethical Dilemmas and Loss of Vendor Control

(22:50–25:52)

• Post-sale, no oversight: Vendors lose all control, and militaries are not bound by vendors' terms of service or intended use policies.
• Ambiguity of “direct harm”: Even non-lethal tools (e.g., information sorting) can assist targeting and surveillance, blurring lines about what constitutes “harm.”
• Procurement is not commercial contracting: Unlike familiar software sales, military procurement involves deep integration with state power and confidential, unaccountable use cases.

“Militaries do not follow terms of service ... this is not how military procurement works, period.”
—Heidi Klaff (24:24)

---

6. The CBRN (Chemical, Biological, Radiological, and Nuclear) Hype

(25:52–28:49)

• Little evidence for CBRN-capable AI—so far: No proof that current models can autonomously develop or deploy such weapons, though training on sensitive data could enable future dangers.
• The real concern: Military use of AI with low accuracy in high-stakes domains (e.g., nuclear command and control) is deeply irresponsible.

“If you’re thinking about nuclear command and control … it certainly shouldn’t be AI systems, because ... these systems are highly flawed and they’re always going to have inaccuracy.”
—Heidi Klaff (27:19)

• Projection vs. reality: Companies and regulators focus on hypothetical existential threats, sometimes at the expense of addressing immediate operational risks and misuse.

---

7. Redefining “AI Safety”—A Convenient Erosion of Standards

(30:34–41:32)

• Technical safety defined historically: In traditional domains, “safety” means protecting humans and the environment from harm—even in worst-case scenarios.
• “Safety revisionism” by AI labs: The meaning of safety is redrawn to mean “alignment,” “existential risk,” or vague generalities—departing from the established, technical, sector-specific understanding.

“[AI companies] use the same safety terminology ... but instead redefine those safety techniques with washdown alternatives that actually accelerate the deployment of inaccurate AI in high risk scenarios.”
—Heidi Klaff (32:00)

• Delegitimizing democratic control: By letting companies dictate risk thresholds and definitions, society loses time-tested, democratically determined standards for what is “safe enough.”
• General vs. specific safety: Military and safety-critical domains require tailored, highly reliable solutions—not generalized or “one-size-fits-all” safety frameworks favored by big AI vendors.

---

8. Why Current AI Safety Efforts Fall Short

(39:38–43:56)

• Risk assessment frameworks co-opted and misapplied: Originally designed as a step toward better regulation, such frameworks are now misused as catch-all justifications for broad AI deployment.
• Excessive focus on unproven hypothetical “existential” risks diverts attention from ongoing, measurable harms.
• Regulation at risk: If regulation focuses on future hypotheticals rather than current dangers, then “you might as well not have regulation at all.”

“If you’re not able to mitigate for the lack of safety and security of AI models today, then you have no chance of mitigating again these hypothetical risks that people like to bring up.”
—Heidi Klaff (43:07)

---

Notable Quotes & Memorable Moments

• On the abrupt pivot to military work:
  “It was almost like a clean slate was being created … they behaved as if this was always aligned with their mission.”
  —Heidi Klaff (06:11)

• On compromised military-grade AI:
  “Commercial models are already compromised from the day they’re built because they’re based on public data.”
  —Heidi Klaff (21:18)

• On vendor control after sale to military:
  “Militaries do not follow terms of service … this is not how military procurement works, period.”
  —Heidi Klaff (24:24)

• On the myth of existential risk dominating real regulation:
  “If regulation focuses on future hypotheticals rather than current dangers, then ‘you might as well not have regulation at all.’”
  —Heidi Klaff (42:30)

• On safety standards in traditional critical infrastructure vs. AI:
  “It even goes up to 99.999% ... if you allow zero risk, we’re never going to build anything ... with technological systems, there is some risk, but you have to mitigate for when systems fail.”
  —Heidi Klaff (36:14–36:19)

---

Important Timestamps for Segments

• [04:16–07:27] — Industry shift to military applications
• [07:27–10:00] — XAI and Security Concerns of Military AI Contracting
• [10:00–15:25] — Military procurement, standards, and commercial AI limitations
• [18:54–22:50] — Are “government” AI models more secure?
• [22:50–25:52] — Vendor ethical statements vs. military oversight reality
• [25:52–28:49] — The hype and reality of CBRN weapon risks
• [30:34–41:32] — Redefinition of “AI safety,” loss of democratic risk threshold setting
• [39:38–43:56] — Misapplication and inadequacy of current risk frameworks; real vs. hypothetical risk

---

Tone and Takeaways

The conversation is clear, well-informed, and critical. Heidi Klaff challenges prevailing narratives from both AI companies and governments, emphasizing the seriousness of deploying untested AI models in defense while pointing out how the meaning of “safety” has been quietly revised to suit profit and policy expediency. The episode ends on a note of caution: true safety must be measured, verifiable, and rooted in established engineering practices—not redefined on the fly for financial or political motives.

---

Transcript

[ServiceNow Representative]

AI is only as powerful as the platform it's built into. That's why it's no surprise that more than 85% of the Fortune 500 use the ServiceNow AI platform, while other platforms duct tape tools together. ServiceNow seamlessly unifies people, data workflows, and AI connecting every corner of your business. And with AI agents working together autonomously, anyone in any department can focus on the work that matters Most. Learn how ServiceNow puts AI to work for [e-mail] Most AI coding tools generate sloppy code that doesn't understand your setup. Warp is different. Warp understands your machine stack and code base. It's built for the entire software lifecycle from prompt to production with the powers of a terminal and the interactivity of an ide. Warp gives you a tight feedback loop with agents so you can prompt, review, edit and ship production ready code trusted by over 600,000 developers, including 56% of the Fortune 500. Try Warp Free or Unlock Pro for just $5 at warp.dev decoder.

[Thumbtack/Lowe's Advertiser]

Avoiding your unfinished home projects because you're not sure where to start? Thumbtack knows homes, so you don't have to don't know the difference between matte paint, finish and satin or what that clunking sound from your dryer is. With Thumbtack, you don't have to be a home pro, you just have to hire one. You can hire top rated pros, see price estimates and read reviews all on the app Download today.

[Hayden Field]

Hey there and welcome to Decoder. I'm Hayden Field, senior AI reporter at the Verge and your Thursday episode guest host. I have another couple of shows for you while Nilai is out on parental leave and we're going to be spending more time diving into some of the unforeseen consequences of the generative AI. Boom. Today I'm talking with Heidi Klaff, who is chief AI scientist at the AI Now Institute and one of the industry's leading experts in the safety of AI within autonomous weapons systems. Heidi has actually worked with OpenAI in the past. From late 2020 to mid 2021, she was a senior systems safety engineer for the company during a critical time when it was developing safety and risk ass frameworks for the company's Codex coding tool. But now the same companies that have in the past seemed to champion safety and ethics in their mission statements are now actively selling and developing new technology for military applications. In 2024, OpenAI removed a ban on military and warfare use cases from its terms of service. Since then, the company has signed a deal with autonomous weapons maker Anduril, and this past June 2 signed a $200 million Department of Defense contract. And OpenAI isn't alone. Anthropic, which has a reputation as one of the most safety oriented AI labs, has partnered with Palantir to allow its models to be used for US Defense and intelligence purposes. And it also landed its own $200 million DoD contract. And big tech players like Amazon, Google, and Microsoft, who have long worked with the government, are now pushing AI products for defense and intelligence, despite growing outcry from critics and employee activist groups. So I wanted to have Heidi on the show to walk me through this major shift in the AI industry, what's motivating it, and why she thinks some of the leading AI companies are being far too cavalier about deploying generative AI in high risk scenarios. I also wanted to know what this push to deploy military grade AI means for bad actors who might want to use AI systems to develop chemical, biological, radiological and nuclear weapons, a risk the AI companies themselves say they're increasingly worried about. Okay, here's Heidi Klaff on AI in the military. Here we go. Heidi Klaff, Chief AI scientist at the AI NOW Institute. Welcome to Decoder.

[Heidi Klaff]

Thank you for having me.

[Hayden Field]

First up, I wanted to talk about how AI companies have moved their goalposts a lot with regard to what they're okay with with and what their mission statements allow regarding work with the US Military and other militaries. So do you remember the whole controversy over Google removing the phrase don't be evil from its code of conduct?

[Heidi Klaff]

Yep, absolutely.

[Hayden Field]

It reminds me a bit of something more recent, which is how OpenAI and Anthropic both used to have certain bans on military use of their products and then they relaxed them. OpenAI walked its ban back in January 2024 when it began to work with the DoD on AI tools. And the month before that it partnered with Anduril and for Anthropic, partnered with Palantir in 2024 to offer Claude to intelligence and defense agencies with the US government. So I wanted to ask what you were thinking when you saw these announcements like made month after month. What did you make of that kind of parade of changes that were happening over a couple of months?

[Heidi Klaff]

Well, to many, including myself, the timing didn't seem like it was a pure coincidence of when OpenAI removed the span in January 2024. If you consider, for example, that Israel at that time was ramping up its mass targeting campaign in G that we now know is being supported by Microsoft's cloud services that offer AI and in this case, OpenAI models as extensions of their IT and cloud infrastructure. And so this rollback was really a signifier on where AI companies were heading, what they were interested in deploying their technologies, despite OpenAI being well aware of the risks that their models posed in defense and safety critical settings, which is something that I actually worked on with them and in one of our papers together when we were looking at evaluation of models like Codex, and interestingly enough, with the announcement of these collaborations that you mention, and that includes Meta Anthropic and OpenAI all announcing US national security work that aligns them with defense contractors like Palantir, NJL and Lockheed, these AI companies never address their previous statements, understands that LLMs or foundation models are unsafe and insufficient for defense use. So it was almost like a clean slate was being created where they behaved as if this was always aligned with their mission.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

For example, they started making claims that US national security is synonymous with safety under this pretense of an AI arms race with China. And this push for this AI adoption seems quite convenient when you're considering the current unprofitable reality of AI and how expensive it is, where it seems like they're trying to sort of de risk their portfolio through government subsidies and military contracts. So now we have this complete pivot from banning military uses, all this talking about their main mission being building systems that benefit all humanity, to now their reliance on this narrative of a US China AI arms race to drive policy initiatives that not only boost the use of AI, but allows them to sort of avoid safety and security scrutiny within military applications.

[Hayden Field]

That makes total sense. And that reminds me of how, you know, I think last week Senator Warren sent a letter about Xai's own contract with the DoD and expressing concern that the company hadn't done the same level of safety audits as other companies before receiving that type of contract, that they weren't ready. She was worried about how the company would use data it had access to as part of, you know, its government partnerships. What did you make of that letter? And kind of Xai's, I guess it seemed grandfathered in kind of approach to this DoD contract announcement.

[Heidi Klaff]

I think the way that I see it is it's part of the Trend of the US DoD not recognizing that there's a national security risk with the use of commercial foundation models in the military, because they do significantly expand the attack vectors of military systems and defense infrastructures that they interface with. Because commercial models are unvetted. They don't have a supply chain that follows the typical military supply chain, and they can be compromised in a lot of ways. So to me, sort of the contract with XAI is another risk that's added onto that. That sort of stems from the issues that all commercial models have.

[Interviewer/Moderator]

Right?

[Heidi Klaff]

And obviously depending on sort of the platform that these models are trained on and the personal data that these models are trained on, they come with a lot of risks and capabilities that allows them to promote or even deploy surveillance systems because they're able to use data that other companies may not have. For example, XAI has a huge amount of data, could be from not just public posts, but also private messages of their users. And, and what does that mean for that to be used in military applications?

[Interviewer/Moderator]

Right.

[Heidi Klaff]

There's a legitimate concern here, but there's also this larger concern that these systems are also unsafe and trained on data that can have been compromised by an adversary, including China.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And that sways the way that the AI system behaves. So there's like risks on both sides here, right? From both sides, the data that is meant to be protected.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And also the type of data that could have been compromised and then thus change the behavior of an AI system that's being used in something like very sensitive military operations.

[Hayden Field]

So something that comes up a lot in my reporting and in my conversations with other people, even my pitching, my editor, is that a lot of these companies are pre profit. I mean, maybe all of them. And we're seeing a bunch of them unveil government products, enterprise products, and that seems to be where a lot of the money lies. So, you know, obviously OpenAI, Anthropic and Xi have all unveiled government products designed for U.S. defense and intelligence agencies to use. They also all received, you know, those government contracts from the DoD. So with companies that burn through cash at super high rates, do we think that the government play is about seeing cold hard cash come in finally or staying above regulatory pressure or both? I'd love to hear your thoughts on that.

[Heidi Klaff]

It's definitely both.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

Because as you mentioned, these companies are pre profit and there's a really big pot of money in the military industrial complex. There simply is, and I think that's well known. But also there is the aspect that these companies would not traditionally pass any of the testing and evaluation required for military procurement. And here's the thing that a lot of people don't know is that defense and military procurement is actually some of the most strict, prior to this AI era. I mean, to sort of evaluate these systems, they have some of the most strict standards, and I think a lot of people assume that's not the case. But often our safety critical systems, if you're talking about like energy infrastructure, you know, and so on and so forth, is derived from those defense standards because of how robust they are. The thing with AI systems, and we're talking about generative AI systems, because AI systems have been used in the military for decades at this point. But these foundation models, or large language models, whatever it is that you want to call them, they do not meet the sort of very basic threshold that it's typically expected for a military system. Right. And so there is this kind of issue now that they want this pot of money. You know, as I mentioned before, they're trying to de risk their portfolios through military contracts, but they have this issue where safety as defined by defense and safety critical system is too stringent for their systems to meet just by their nature.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

They're highly inaccurate systems. And when you're looking at, I can't really get into defense systems, but I'm going to talk a little bit about safety critical system. If you're looking at a nuclear power plant, for example, you're looking at safety of 99%. That's the minimum and often the accuracy of AI systems is 60%. If I'm being optimistic about specific types, there's an enormous gap here to make AI systems as they exist for foundation models be able to satisfy the strict testing and evaluation measures often required by military procurement.

[Hayden Field]

For the listeners, let's just define military procurement really quick.

[Heidi Klaff]

What is that military procurement? It's. Well, it really depends, right. There's a huge amount of processes that exist for different types and the process is often strict depending on how critical the technology is going to be used. Like for example, if it's going to be used for lethal operation versus bureaucratic operations. Very different types of procurement. If you, if we are looking at a sort of more general idea, typically the government puts out a specific ask for the type of systems that they're looking for. And people submit to that, you know, procurement ask. Ultimately, these systems often have to go through what we call a testing and evaluation process for them to even be considered, you know, even before they sign the contract.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

This process is quite stringent in that it has their specific thresholds on how accurate these systems need to be and how secure they need to be. Often the security thresholds, it's extremely, extremely high.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

They have to be air gap. The supply chain has to be completely traceable. They have to know who coded the system, who developed the system, if there's any sort of backdoors that can be compromised, so on and so forth. And once sort of a system goes through that procurement, that safety and security procurement process, the DoD often just takes complete control of that technology.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

Like this is now something that they possess and are in complete control of using in whichever way that they see fit.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And so this is often why procurement for the military takes. Can even take many years.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

This is not a process that is meant to take a couple of weeks or even several months. Often this is quite a rigorous process. So this is very different from signing a commercial contract.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

Where you have traditional terms of service. The nation state, not just, you know, the US Cud often are the ones that get to define the terms of how they want to use the technology. And typically people abide by it because people want that pot of money.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

It's very lucrative to even be considered up for procurement because it means that you're sort of be on call for them for potential other technologies. But even to get your foot in the door can take years. So it's a very different type of assessments than I think, what people expect for commercial contracts.

[Hayden Field]

We need to take a quick break. We'll be right back.

[Strawberry Me Advertiser]

Let's be honest. Are you happy with your job? Like, really happy? The unfortunate fact is that a huge number of people can't say yes to that. Far too many of us are stuck in a job we've outgrown or one we never wanted in the first place. But still we stick it out and we give reasons, like, what if the next move is even worse? I've already put years into this place, and maybe the most common one. Isn't everyone kind of miserable at work? But there's a difference between reasons for staying and excuses for not leaving. It's time to get unstuck. It's time for Strawberry Me. They match you with a certified career coach who helps you go from where you are to where you actually want to be. Your coach helps you get clear on your goals, create a plan, build your confidence, and keeps you accountable along the way. So don't leave your career to chance. Take action and own your future. With a professional coach in your corner. Go to Strawberry Me Unstuck to claim a special offer. That's Strawberry Me Unstuck.

[Hayden Field]

Fox Creative.

[Alnylam/Ziply Fiber Advertiser]

This is advertiser content from Alnylam. Living with disease can be draining. Managing new symptoms and medications, Scheduling endless appointments and tests. Losing the flexibility and independence you're used to. The drip drip, drip of Disease takes its toll On a genetic level, disease is like a leaky tap. Our genes instruct our cells to produce proteins that the body needs to function. But sometimes genes instruct cells to produce unwanted proteins or too much of a protein, which can cause or contribute to disease. Most conventional medicines treat disease by targeting the symptoms, like mopping up the puddle rather than tightening the leaky task. But an innovative class of medicines pioneered by Alnilam Pharmaceuticals targets disease at the source. With RNA interference, we can disrupt the production of unwanted proteins to silence the drip, drip, drip of disease. This innovative approach to treating disease is already helping thousands of people around the world live amplified lives. And it's just the beginning. Learn more about RNAi therapeutics and Alnylam [e-mail] Alnylam silence disease, amplify Life.

[Thumbtack/Lowe's Advertiser]

It's Pro Savings Days at Lowe's get up to 35% off select major appliances and save an additional $1,000 when you buy four select LG major appliances plus get a free Dewalt 20 volt max 5amp hour battery when you buy a select Dewalt 20 volt max tool. Get the job done for less At Lowe's we help you Save valid through 926. Selection varies by location while supplies last. See associate or lowe's.com for more details and qualifying items.

[Hayden Field]

We're back with Heidi Claf of the AI Now Institute. Before the break, Heidi was breaking down the standard military procurement process and why it feels like AI systems don't meet the rigorous standards we might expect when it comes to being used for for high risk operations. Now I want to ask Heidi about the specific AI products being sold to the US Military and whether they're really much more secure than the commercial models on the market today. I also wanted to ask the models these companies use for government products like their government designed products like Claudegov OpenAI's government product, Xai's government product by design have looser guardrails for government use and they're trained to better analyze classified information. And although these types of models allegedly underwent the same type of safety testing as these companies, other models I'm using Anthropic here as an example. They have certain specifications for national security work like they have a greater understanding of intelligence and defense documents and they refuse less when they are asked to engage with classified information that's being fed into them. So in your eyes, how does the development work? How secure really are they and what are the implications here?

[Heidi Klaff]

So I wouldn't say they're much more secure. They may be more secure in that they're more air gapped. So, for example, you can take a commercial model, right, and you can fine tune it on sort of sensitive military data, and then that model then becomes accessible to the military. But that still misses out on some of the biggest risks of that commercial model is that it was trained on data sets that were publicly available. And so a lot of research has shown that not only can you poison web data that these models are trained on, but you can implement what's called like a sleeper agent, which is given a specific prompt or a command. It will then behave in a sort of a harmful way that sort of the operator of that system did not intend based on something that was implemented in the training data or something that, you know, the model was trained on. And so we see this all the time with like prompt injections. But this can happen on a sort of deeper level with what we call sort of web poisoning attacks, which can then be used to implement these sleeper agents, as we call them. And so this is in the commercial supply chain, right? The only way that these models are trained is to be trained on sort of mass amounts of data that are publicly available. So they're already compromised.

[Interviewer/Moderator]

Right?

[Heidi Klaff]

These models are also fine tuned through methods like reinforcement learning, human feedback, which unfortunately uses basically sweatshops of people in developing nations that are paid nothing to then make these models behave in a specific way. And you can imagine a military operation, right, where a foreign adversary is able to sort of have a covert operation in which they run one of these data labeling and data fine tuning shops essentially, and are sort of aware that they might eventually be used to be fine tuned for military application and implement backdoors or sleeper agents which trigger a specific behavior based on a specific command. And because of that, that's what makes them so unsafe. So sure, you might be able to fine tune it on specific data that isn't then released publicly, which might remove some vectors of attack. But ultimately, at the end of the day, commercial models are already compromised. So when you're saying that they are more secure, I mean they're more secure in the traditional security way in that you air gap the system so you kind of limit the control of people who have access to it. And so thus people who can probe it and get information out of it. But it doesn't remove the fact that commercial models are already compromised from the day that they're built because they're based on public data.

[Hayden Field]

I wanted to see if you agree with this. Take I'm about to tell you. So I once interviewed Meg Mitchell from Hugging Face and she said that for these types of military contracts, even if you have in your mission statement like, you know, anthropic and OpenAI do, that your tech can't be used to directly harm others. The problem is that you don't have control in the end over how your tech is actually being used with the military. If you do have any control, you definitely don't have control in the longer term once you already shared that with the military organization, especially without having security clearance and knowing really how it's being used down the line. She also was talking about what's considered direct harm. You know what, if you're summarizing social media posts that then lead to making a list of enemy combatants or potential people of interest that have a certain view on a topic on X, for example. I wanted to see if you agree with that in terms of, you know, these companies often have in their mission statements, oh, don't worry, even though we're working with the military on this, this and this, we know for a fact that our tech isn't being used to directly harm people. But yeah, how can they really know that? Can they?

[Heidi Klaff]

I completely agree with that statement. And I think something that often people miss out on is that militaries do not follow terms of service. They might do that if they're buying like a Microsoft Office suite, right, for their bureaucratic purposes. But when it comes to military procurement, the companies do not have control and they know that over how these systems are being used. And they actually have no say in terms of the terms of service as well. These things get often determined by international law and also by the nation state itself.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

They have the power here. And so when people tell me, oh, but wouldn't that break the terms of service? And it's like, this is not how military procurement works, period.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And we've also seen examples of, as I mentioned before, Microsoft working directly with militaries to implement some of these systems. So I would say that in a lot of cases, they are well aware of, of how their systems are being used to some extent.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

We don't know all the details, but governments put out procurement documents of the type of data they want, how they want to use it and how they want to store it, because then companies can offer services like what their AI can do with that.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

So I do think that it's not the case that they are just selling something commercially like they do to everyone else. And then they hope the Military abides by it. It's a much more involved process to do military procurement that often requires testing and eval, and the companies typically have to be in the know about the technical details to see if they can offer support for that. Again, as I use a Microsoft case as one of the most recent examples of, of that being the case with their work with the idf. So I think it's easy for them to point to terms of service. But as someone who has worked on procurement before, this is not a commercial contract that these companies are signing.

[Hayden Field]

Okay, so now I want to get into the CBRN side of things. Although, as we've talked about, you know, it may not be as big of a concern as AI companies are making it out to be. It is a big concern for the public, probably because of that marketing and just because it's a scary idea. So let's be real here. Obviously, smarter AI that can do anything for you isn't always good, especially when people want to use it to do bad things like creating chemical, biological, radiological, and nuclear weapons. So top AI companies say they're increasingly worried about the risk of that. Of course, they're not maybe worried enough to stop building. But I want to get into again how big of a risk this is. Let's just go into more detail there.

[Heidi Klaff]

We have not seen any proof of CBRN capabilities right now, but those capabilities could come to fruition if we start training very, very sensitive nuclear data, for example, nuclear technologies on these models. And I actually believe that the risk that comes with that is very different from what most people are thinking about. Most people are thinking about that the AI is somehow going to develop weapons by itself.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

Or it will give access to adversarial actors to do so. But even if it's just a military who has access to a model that has been trained on CBRN data, what that means is that they are likely going to use it for those purposes within the military. And that's extremely dangerous. Like, if you're thinking about nuclear command and control, right, who gets to essentially make the decisions about nuclear weapons deployment? And it certainly shouldn't be AI systems, because regardless of the data distribution, these systems are highly flawed and they're always going to have inaccuracy. As I mentioned before, often when we're looking at military systems that deploy AI, they can have as low of an accuracy rate as like 20%. And if you're being really optimistic, maybe 60 to 80%.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

These are the levels of accuracy that you're looking at with these types of system. And so then to train a model on CBRN data and then to then attempt to use it for those tasks is extremely dangerous when you're looking at those accuracy rates.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

For me, the concern that I have is that they will then think that these models are reliable because we trained them on that set of data and thus we can then use them and the military and in defense operations to dictate decisions about those types of systems and where they should be used and when. And I think, you know, that's a very different type of risk than most people are thinking about. As before, this idea that like they're somehow going to gain CBRN capabilities by themselves. Very hypothetical and not really like tied to the reality that we're in right now with AI systems. But if we're taking AI systems today and we train them on sensitive military data, I have a concern of how those systems are going to be used.

[Hayden Field]

We need to take another quick break. We'll be right back. Mint is still $15 a month for premium wireless. And if you haven't made the switch yet, here are 15 reasons why you should. One, it's $15 a month. Two, seriously, it's $15 a month. Three, no big contracts.

[Heidi Klaff]

Four, I use it.

[Hayden Field]

Five, my mom uses it. Are you, are you playing me off? That's what's happening, right? Okay, give it a try.

[Heidi Klaff]

@Mintmobile.Com Switch upfront payment of $45 for a three month plan. $15 per month equivalent required. New customer offer first three months only, then full price plan options available, taxes and fees extra. See mintmobile.com Ford BlueCruise Hands Free highway driving takes the work out of being behind the wheel, allowing you to relax and reconnect while also staying in control. Enjoy the drive in Bluecruise enabled vehicles like the F150 Explorer and Mustang Mach E. Available feature on equipped vehicles. Terms apply. Does not replace safe driving. See Ford.com BlueCruise for more details.

[Thumbtack/Lowe's Advertiser]

Race the rudders. Raise the sails. Raise the sails. Captain, an unidentified ship is approaching.

[Heidi Klaff]

Over.

[Thumbtack/Lowe's Advertiser]

Roger, wait. Is that an enterprise sales solution? Reach sales professionals, not professional sailors. With LinkedIn ads, you can target the right people by industry, job title and more. Start converting your B2B audience today. Spend $250 on your first campaign and get a free $250 credit for the next one. Get started today at LinkedIn.com campaign terms and conditions apply.

[Hayden Field]

We're back with chief AI scientist Heidi Klaff discussing the ways in which AI companies are pushing into defense contracting before the break, we were talking about how real the risk is that AI systems might be used to develop nuclear or biological weapons. But now I want to zoom out and talk to Heidi about the broader field of AI safety and how she thinks it's changed since she worked with OpenAI years ago. Let's shift and talk about AI safety for a bit. So you helped establish and pioneer the field of AI safety engineering. What's the technical meaning of safety like with your background? And how has the AI safety world changed that meaning? Or how has it become more colloquial now?

[Heidi Klaff]

So if we take a step back and not think about what the AI companies have been telling us, what safety means for the past four years, or even more than that at this point, safety has historically meant, especially in the context of safety critical systems ensuring no harm to humans or the environment. So if you're thinking about aviation or nuclear power plants, for example, you want to ensure that when your systems fail, and systems do fail, that humans are not harmed, that there's no death and that there's no environmental catastrophe. It's quite a simple definition. Now, what is happening in terms of what safety now means is very different, and it has been redefined by AI companies as of late. So I believe AI labs engage in what I call safety revisionism, where they use the same safety terminology that are often used for regulating and assurance defense and safety critical systems, but instead redefine those safety techniques with washdown alternatives that actually accelerate the deployment of inaccurate AI in high risk scenarios like defense or nuclear. So, for example, AI companies often reduce the term safety to now mean alignment or existential risks. Now, this is pretty distinct, right, from the definition of safety that I just gave you because alignment focuses on human preference and that makes us question, well, alignment with whom and which humans and whose preferences. And the existential risks that are also emphasized, like cbrn, are hypothetical, like I mentioned, and are often used as sort of a pretense for an AI arms race to ignore other risks and safety thresholds like surveillance systems, right? So in allowing AI companies to do this, right, which a lot of governments have, they've sort of ceded that control of defining what safety is to them, puts them in a position to define what a risk threshold is or what what actually safe enough means. And the entire idea of risk thresholds, because I imagine a lot of people might not know this, is to provide sort of a metric or a measure of the level of risk exposure that our society collectively agreed to take. And this often shapes how we determine the safety of Technological systems, including nuclear plants. And typically this is done through a democratic process that we have established over decades. In other high stakes scenarios, like all of our thresholds for other safety critical systems have come from sort of democratically determined idea of what society thinks safety is. So in allowing AI companies to sort of co opt these traditional safety terms, we've sort of given them permission to not only decide what counts as safe enough, which again breaks these democratic norms that we've had, but it also lowers and undermines existing safety threshold that would have otherwise regulated AI use in things like defense. So ironically, you know, this I, this is kind of their way of how they bypass some of the safety measures that I've talked about earlier in that they're looking for this pot of money right from the military. But the safety thresholds for defense are extremely high. So what do you do? Well, you redefine what safety means and you say it's different for AI. You say because our systems are so different, they're at a scale we've never seen before, we cannot abide by these safety rules, which is definitely not accurate. I think a lot of our existing safety critical standards hold for AI systems. And ironically, this hollowing out of safety, although being sold as crucial to win the AI arms race, we can't be regulated. We have to beat China is accelerating AI adoption at the cost of more unsafe and insecure systems, which may be exactly what disadvantages the US military and our technological capabilities against China. If we're sort of letting inaccurate and easily compromised systems be deployed in our front lines because it's profitable for these AI companies.

[Hayden Field]

Let's talk a little bit about earlier. You mentioned safety of 99%, for example, at a nuclear power plant. What does that mean in context? We talked just now about the meaning of safety in that regard. But what would a safety of 99% entail? It's that there's a 99% chance that it won't harm people or the environment or, you know, how does, what does that mean in practice?

[Heidi Klaff]

Yeah, I mean it's a lot more technical than that, but basically, right, we have these thresholds of these systems have to be accurate and be able to perform. So these are typically what we call reliability and availability measures of the system. So they can only fail often. Even 99 is like 1 of, it's like the lowest threshold for a nuclear plant. It even goes up to 99.99%.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And so obviously if we allow zero risk, we're never going to build anything.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

Like I think that's very important to remember is that with every technological system that there is some sort of risk, but you have to mitigate for the when those systems fail. So this idea that our safety critical systems have this like 99.99% reliability means that they're meant to operate basically well, 99 to 99.999% of the time, depending on the kind of system that you're looking at.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And the safety criticality. And then when that system fails, we then have to have mitigations in place.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And there will be risks with that. And typically these mitigations are based on, like I said, these thresholds of how many people could be harmed. So in the case of like airplanes, I think that's a very simple example, a catastrophic incident is considered if everyone on a commercial airplane dies. So typically that number is like 300 people get or die. That's the threshold for aviation as being the most catastrophic thing that could happen. So safety is actually very specific to the use cases. What we mean by 99.99% reliability often relates to these systems failing. But if you're looking at how to actually mitigate for those risks and what the threshold is, that depends on every single field because kind of the impact of the system will vary. An airplane crashing is very different from a nuclear plant crashing. So nuclear plant doesn't have this idea that 300 people dying is the worst case scenario. In fact, it's much more than that and also has to do with environmental nuclear disaster.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And so this is why this idea of AI and the safety that they push forward is problematic, because they want us to adopt this idea of universal or general safety that has to do with like, as they call it, alignment. And it's this misguided idea that there exists like a universal safety solution that would make all general functionality of all LLMs safe.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And this is also one of the ways that procurement is changing in the military, where you're looking at companies like Scale AI, they are putting forward these types of general frameworks, but there is no standard safety approach to generic systems in any domain. In fact, this would contradict established safety practices that require sort of a well defined use case to map risks against. And so often what we're seeing now happen is that companies like Scale AI, they say we're going to build a risk assessment framework for AI systems because existing ones simply don't work. I'm being sarcastic. That's not the case. And then the way that they define safety again is through the safety revisionism they call it something else. It ends up being about something else and completely disconnected from actually being accurate for military operation. It ends up being again, these high level ideas of safety that we're seeing them push, whether it's about cbrn, it's like, right, but can it do the thing that we're asking the AI systems to do? You actually never see an assessment of that in a lot of these frameworks. And so this is sort of why this idea of safety becomes very confusing because it has diverted so much from how we've traditionally used it to assess like nuclear plants or airplanes and, and so on.

[Hayden Field]

And you led the safety evaluation of Codex at OpenAI. So what was that like and would it be a different process, do you think, if you were leading that work.

[Heidi Klaff]

Now for Codex, the idea was to introduce something like a risk assessment for AI, which is not what people were doing before prior. There was like a lot of benchmarking.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And these benchmarks didn't really consider the risks that the AI system poses with having specific capabilities. So the idea was to really try to investigate that and use some techniques inspired from safety critical fields. It was not meant to be a replacement for assessments for safety critical systems.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And I think that's a really, really big distinction. And in terms of like, what would I be doing now? It was my choice to not continue working with OpenAI because it became very clear to me again, this idea that they're pushing of general safety just does not align with how safety actually should be assured in sort of the real world.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

And so this idea of existential risk CBRN alignment to me was like, no, but these are not the current harms that we're going to see if we're going to deploy AI systems in these safety critical situations. And if we are going to deploy them in safety critical situations, situations like defense, we have to assess the system as we always have for every other system. I thought introducing something like risk assessments would be helpful to the field because then people could understand the risks that come from using the systems. But what it ended up, unfortunately evolving to and being used by many labs is that these risk assessments are now sort of being used like the end all, be all of all assessments of AI being used in all systems everywhere. And that I regret very much. But that was never sort of the intention to begin with. When we set out this work, was.

[Hayden Field]

There one thing or a couple different examples of what made you kind of decide not to continue? Do you remember anything specific?

[Heidi Klaff]

Yeah, I think it is the existential risk this concern that AI will somehow become self aware or have these capabilities that lead to nuclear proliferation. And as someone who has worked on sort of risk assessments now for about a decade, you have to have real data to back up your claims. And so when you are then using risk assessment frameworks to try to substantiate hypothetical claims that there are no proof for, you're not doing science. And to me, I'm willing to be convinced that perhaps AI models could have CBRN capabilities in the future. I am not opposed to that idea, but they don't have them now. And so for all of us to put our safety and regulation efforts, and that includes by the US Government and the UK governments, to be about hypothetical risk that can't be measured, right. That can't be quantified or qualified, and our entire regulatory system then becomes about risks that we have yet to see. You might as well not have regulation at all.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

So I think a lot of people talk to me about, well, what do you think of this framework and what do you think of that framework? I'm like, to me, practically speaking, as someone who has done risk assessment, it is equivalent of having no regulation because we're actually not addressing the risks of the harms that AI is posing and we're in fact focusing on hypothetical risks. And there's this idea that I've heard before, well, what if those risks come true and you're unprepared? And the way that I see it is that if you're not prepared for today's risks and you're not building the frameworks for that, you're not going to be prepared for future risk because these frameworks and risk assessments built on top of each other. So if you're not able to mitigate for the lack of safety and security of AI models today, then you have no chance of mitigating again these hypothetical risks that people like to bring up.

[Interviewer/Moderator]

Right.

[Heidi Klaff]

Because that is kind of one of the core concepts of safety is the smallest catastrophe, not the smallest catastrophe, like the smallest hazard can cascade into a large catastrophe. So if you're not able to address the very, the things that are considered, you know, they consider this stuff inconsequential, then you're never going to be able to prepare for these, you know, much more large scale events that they talk about in that, that's very much like a standard safety perspective to have the.

[Hayden Field]

Snowball effect in practice. Well, thank you so much, Heidi. This is incredibly helpful. And you know, your perspective is so unique. So I'm really glad we were able to, you know, talk about this and have the audience kind of weigh in and comments and stuff. I think this is, you know, something that's not talked about enough. So I'm really glad we were able to talk. And thanks for making the time and moving your schedule around.

[Heidi Klaff]

Thank you for having me.

[Hayden Field]

I'd like to thank Heidi for taking the time to speak with me, and thank you for tuning in. I hope you enjoyed this episode. If you'd like to let us know what you thought about this show or what else you'd like us to cover, drop us a line. You can email us at Decoder at the Verge. We really do read every email or hit me up directly on X Bluesky or Threads. I'm Aydenfield on all platforms. Decoder also has a TikTok and it and now also a YouTube channel. Check those out at DecoderPod. They're a blast. If you like Decoder, please share it with your friends and subscribe wherever you get your podcasts. Decoder is a production of the Verge and it's part of the Vox Media Podcast Network. Our producers are Kate Cox and Nick Stadt. Our editor is Ursa Wright. The Decoder music is by Breakmaster Cylinder. See you next time.

[Thumbtack/Lowe's Advertiser]

For a limited time at McDonald's, get a Big Mac Extra Value meal for $8. That means two all beef patties, special sauce, lett lettuce, cheese, pickles, onions on a sesame seed bun and medium fries and a drink.

[Heidi Klaff]

We may need to change that jingle.

[Hayden Field]

Prices and participation may vary.

[Alnylam/Ziply Fiber Advertiser]

When the Moore family ditched cable Internet and switched to Siddly Fiber, they got so much more. Mr. Moore got more upload speed for next level gaming and livestreaming to the masses with reliable service. Mrs. Moore is no longer her family's IT guru, leaving her more time to stream games into overtime.

[Heidi Klaff]

Let's go.

[Alnylam/Ziply Fiber Advertiser]

And young Mason Moore got more done quickly uploading HD product demos and video conferencing without FreeSync.

[Heidi Klaff]

The numbers look good.

[Interviewer/Moderator]

Brad.

[Heidi Klaff]

You're on mute.

[Alnylam/Ziply Fiber Advertiser]

Switch from cable Internet to ziply Fiber and get more of what you love for $65 less per month than cable at ziplyfiver. Com.