Iran and AI on the battlefield

Summary

---

Front Burner: "Iran and AI on the Battlefield"

Date: March 6, 2026
Host: Jayme Poisson (B)
Guest: Heidi Klaff (C), Chief AI Scientist at AI NOW Institute, AI safety and autonomous weapons expert, ex-OpenAI

---

Episode Overview

In this episode, Front Burner explores the rapid escalation of AI integration into modern warfare, with a focus on the recent US and Israeli military operations against Iran. Host Jayme Poisson and guest Heidi Klaff discuss how large language models (LLMs) like Anthropic’s Claude have become central to military decision-making, the confusion these systems create around accountability, and the broader implications for state power, ethics, and international law. The conversation ranges from specific incidents (like a disputed airstrike on a girls’ school in Iran) to the global race to deploy AI in military and surveillance contexts, highlighting the ethical and existential risks of this new era.

---

Key Discussion Points and Insights

1. How AI Is Used in the Iran Conflict

• AI as Decision Support

  • Anthropic’s Claude acts as a decision support system, aggregating and analyzing data sources including satellite images, social media, and intercepted communications to recommend military targets and prioritize strikes.

    “Claude is currently being used as...decision support systems...looking at satellite images, social media feeds, intercepted phone communications...then it uses them to make recommendations, including target recommendations, prioritizing them and even providing coordinates for those targets.” (C, 02:13)

  • Claude works in tandem with Palantir’s Maven Smart system, combining vast classified and open-source datasets for analysis.

• Obscuring Accountability

  • Use of AI muddies the chain of responsibility for military actions.

    “AI is actually being used to evade accountability...it makes it difficult to distinguish if some attacks were in fact deliberate due to intelligence failures or due to the lack of AI accuracy...” (C, 03:53)

  • Ambiguity exists over whether civilian casualties result from human error, outdated data, or AI misjudgments.

    “It's very difficult for us to say if this was due to an AI mistake or if this was deliberate or not. And that's exactly why a lot of militaries are using AI. It obscures that.” (C, 03:53)

---

2. Evolution and Accuracy of Military AI

• Progression of AI in Warfare

  • AI use in military contexts is not new, but LLMs/GPT models are a recent development—previous AIs were more task-specific, trained on controlled data, and more accurate.
  • LLMs, originally civilian tech, are now minimally adapted for targeting but remain error-prone.

    “Military purpose built AI...tends to be very task specific...LLMs...are very general purpose...LLMs have an incredibly low accuracy rate...you're looking at something like 25 to 50% accuracy, and yet they're still being deployed.” (C, 06:04)

• Comparison to Old Attacks

  • AI-driven warfare is likened to high-tech carpet bombing, lacking precision, and blurring legal norms.

    “It's almost just a high tech version of carpet bombing. And ...AI is being used to evade accountability.” (C, 10:42)

---

3. Real-World Examples: Gaza, Lavender, and Gospel

• Previous Israeli systems (Lavender, Gospel) did not use LLMs but similar AI targeting approaches; more recent adoption of GPT-4 and Google Gemini in targeting confirmed after October 7, 2025.
• All target-identifying AIs share the purpose of maximizing targets, often justifying mass human impact.

  "They're all looking to generate as many targets as possible...they all ingest similar types of information...both types of AI really have accuracy problems here." (C, 09:29)

---

4. Rubber Stamping and Automation Bias

• "Human in the loop" is eroded by automation bias—humans tend to trust AI outputs without verifying, creating de facto autonomous weapon operations

  “Humans often trust the recommendations of algorithms without corroborating with other sources or checking...so it does in the end of the day...end up being rubber stamping.” (C, 12:01)

• The gap between “decision support” and “autonomous weapons” is problematic and largely a formality.

---

5. Nightmare Scenarios and AI Expansion

• AI deployment in nuclear command and control (e.g., Scale AI’s recent contracts) is highlighted as a peak risk.

  “I think we are in the nightmare scenario when it comes to military use...giving these models access to nuclear weapons…we are already in such a terrible worst case scenario…” (C, 13:27)

• The normalization of LLMs in critical infrastructure is seen as reckless and dangerous.

---

6. Defining Autonomous Weapon Systems

• Many forms exist: from landmines to smart drones; frontier AI (LLMs) aims to automate entire targeting and deployment processes, removing humans entirely.

  "There's many different types of autonomous weapon systems…so it's to just eliminate the human from that loop and just to automate that process." (C, 16:29-17:57)

---

7. Corporate-Government Tensions: Anthropic, OpenAI, and US DoD

• Anthropic’s Red Lines and Pentagon Demands

  • Anthropic resisted removing guardrails on domestic surveillance and fully autonomous weapons.

    "We are okay with all use cases...except for two...domestic mass surveillance...and fully autonomous weapons.” (D, 18:35)

  • Military preference is clear: they seek flexibility and minimal restrictions, partly to maintain plausible deniability.

    "They very clearly want to use these technologies as an alibi for whatever actions they want to carry out…AI really helps you evade accountability...” (C, 19:16)

• Surveillance and Data Dual-Use

  • LLMs enable analysis of bulk civilian data, raising domestic surveillance risks.

    "These models are what we call dual use. So they can be used for civilian purposes and...for military purposes." (C, 20:24)

• International Legal and Regulatory Voids

  • Decisions often rest on private corporate choices and US DoD interpretations, rather than international law or transparent oversight.
  • Even with supposed legal compliance, international norms and institutions like the UN or ICRC have not weighed in meaningfully yet.

    “We shouldn't be at the whims of a private corporation red lines on whether or not this dangerous technology should or should it be deployed..." (C, 21:58)

• Supply Chain Concerns

  • US government labeled Anthropic a “supply chain risk,” leveraging its influence; Klaff argues the risk is inherent to all LLMs, given their open training and potential for backdoors.

    "I actually do believe that all LLMs are a supply chain threat to national security and defense...trained on the open Internet and publicly available information..." (C, 24:18) "You only need about 250 malicious documents to produce a backdoor into one of these models..." (C, 25:37)

• OpenAI’s Position and "Safety Theater"

  • OpenAI’s response to military interest is described as “safety theater”—public claims of guardrails are not operationally feasible, and monitoring real military use is not possible.

    “The safety guardrails…are not operationally feasible...you can't monitor after that what that individual does..." (C, 27:09) "I would call it safety Theater, Safety co option…" (C, 28:36)

  • Notable for dropping stated military use bans in 2024.

    “Like military use. That was eliminated in 2024 when it was initially part of their charter or their terms of service…” (C, 28:36)

---

8. Why AI Companies Court Military Contracts

• Military contracts provide both a critical revenue stream for expensive-to-train models and strategic entrenchment—embedding companies in essential state infrastructures.

  “When you have the military, that's a really big money pot…embeds you within military and safety critical infrastructure, which means that you're then too big to fail.” (C, 29:47)

---

9. AI as a Tool of State Power

• General-purpose AI enables unprecedented surveillance, centralized data collection, and decision automation by the state, with little transparency or recourse.

  “It is the perfect tool to concentrate power because you're collecting essentially all data possible on humans, on our behavior...” (C, 31:11)

• A dangerous “black box” logic now mediates critical decisions in migration, judicial systems, and war.

  “Now it feels like no one can question that power. This is an unfortunate side effect of something like these very large general purpose models.” (C, 31:11)

---

Notable Quotes & Memorable Moments

• On Accountability:

  “When you're using these types of systems, it makes it difficult to distinguish if some attacks were in fact deliberate due to intelligence failures or due to the lack of AI accuracy...AI is actually being used to evade accountability.” (C, 03:53)

• On AI Accuracy:

  “LLMs have an incredibly low accuracy rate...you're looking at something like 25 to 50% accuracy, and yet they're still being deployed.” (C, 06:04)

• On Human Oversight:

  “Humans often trust the recommendations of algorithms without corroborating with other sources...so it does in the end...end up being rubber stamping.” (C, 12:01)

• On Nuclear Risks:

  “I think we are in the nightmare scenario when it comes to military use...we are already in such a terrible worst case scenario.” (C, 13:27)

• On Legal Responsibility:

  “[AI] decisions shouldn’t be at the whims of a private corporation’s red lines on whether or not this dangerous technology should or shouldn’t be deployed...there is international law.” (C, 21:58)

• On LLMs as National Security Threats:

  “I actually do believe that all LLMs are a supply chain threat to national security...this is about the nature of LLMs themselves...very different from purpose military built models.” (C, 24:18)

• On OpenAI’s Safety Claims:

  “The safety guardrails…are not operationally feasible...it could just very well be taken to then select and engage a target without any further oversight.” (C, 27:09)

• On AI as State Power:

  “It is the perfect tool to concentrate power because you're collecting...all data possible on humans, on our behavior...there’s no accountability because these models are black boxes.” (C, 31:11)

---

Key Timestamps

• 00:42 – Episode premise: AI on the battlefield, US/Israeli operations, civilian school strike.
• 02:13 – Explanation of Claude's role in targeting.
• 03:07 – How Palantir and Claude integrate for military analysis.
• 03:53 – AI and the crisis of military accountability.
• 08:04 – LLMs' accuracy compared to earlier military AIs.
• 09:29 – Gaza/Israel: evolution from earlier AI systems to LLMs.
• 10:42 – Declining precision and legality in modern, AI-driven warfare.
• 12:01 – Automation bias and the myth of “human in the loop.”
• 13:27 – Fears about nuclear command and loss of oversight.
• 16:21 – Defining and unpacking “autonomous weapon systems.”
• 18:35 – Anthropic’s boundary conditions vs. DoD preferences (quote from Pentagon official, D).
• 20:24 – The dual-use nature of LLMs and the surveillance implications.
• 24:18 – Anthropic labeled a supply chain risk and the inherent threat of LLMs.
• 27:09 – OpenAI’s deal with DoD, and the futility of proclaimed safety guardrails.
• 29:47 – The financial and strategic motivations for AI firms courting military clients.
• 31:11 – AI as the new tool of centralized, unaccountable state power.

---

Conclusion

This episode delivers a sobering look at how rapidly AI is transforming military operations—not with greater precision and safety, but with more confusion, less accountability, and the erosion of basic legal and ethical standards. From decision support to autonomous targeting and surveillance, LLMs are enabling states and militaries to centralize power and evade responsibility, with little oversight from either international law or the corporations developing the technology. The conversation between Poisson and Klaff paints a picture not of progress, but of regression, toward a world where life-and-death decisions are made inside a black-box algorithm, with no one ultimately to answer for the consequences.

---

Transcript

[A]

Hey, this is Adam Grant, host of ted's podcast Rethinking with Adam Grant. Have you heard of Bill? It's the intelligent finance platform that uses AI to help you avoid costly errors and optimize cash flow. In fact, Bill reports that over 90 of the top 100 US accounting firms trust them to manage, move and maximize money. Proven by over a trillion dollars in secure transactions. Eliminate the friction and start scaling with the proven choice. Visit bill.com proven to talk with an expert about automating your business finances and and get a $250 gift card as a thank you. That's bill.com proven terms and conditions apply. See offer page for details. This is a CBC podcast.

[B]

Hi everyone, I'm Jamie Poisson. For decades we have been hearing about the possibility of AI driven warfare. Now it's here. Anthropic's AI platform, Claude has reportedly been central to America and Israel's war on Iran. Within the first 12 hours, there were more than 900 strikes, including one that killed the Supreme Leader, Ayatollah Khamenei. An airstrike on a girls elementary school killed at least 165 people, mostly students. Today we're talking about AI military capabilities, how companies like Anthropic and OpenAI have become or are on their way to becoming deeply enmeshed in the military. And what happens when these companies and governments start building systems that help decide who lives and who dies in a war. I'm joined today by Heidi Klaff. She is the chief AI scientist at the AI NOW Institute and an expert on AI safety within defense and national security, including in autonomous weapons systems. And she previously worked at OpenAI. Heidi, hi. Thanks so much for making the time.

[C]

Thank you for having me.

[B]

It is really great to have you. So let's start with what we know about how AI is being used in the war with Iran right now. What role is it playing?

[C]

So CLAUDE is currently being used as what we call decision support systems, or which means that it brings together a lot of different types of data that has been gathered. So you're looking at satellite images, social media feeds, intercepted phone communications, and then it uses them to make recommendations, including target recommendations, prioritizing them and even providing coordinates for those targets. They're also likely used in other phases of the kill chain. So that includes things like intelligence gathering, surveillance, and even collateral damage assessment. And as sort of have been confirmed by recent reports.

[B]

And this is all happening as I understand it, because of this pairing with the military's Maven Smart system, which is built by the company Palantir, which is mining all this data, including classified data, and then it's been meshed with Claude. And can you just like, describe to me what's going on there?

[C]

Yeah. So Palantir is a data analysis company and they work a lot on intelligence and surveillance. And back in 2024, anthropic and palantir had agreement where essentially Anthropic's model, that being Claude, would then power sort of this recommendation engine, which takes all of the enormous amounts of data that Palantir has sort of analyzed, collected, and sorted through, and then feed it to an AI to then make some sort of decision.

[B]

And so I mentioned that strike on the girls school in Iran. The Pentagon is denying targeting civilians. Israel says that there was no IDF operations in the area. But is it possible that AI was used in some form here?

[C]

I think that's actually a very good question because it brings the point forward that AI is actually being used to evade accountability. And that's because when you're using these types of systems, it makes it difficult to distinguish if some attacks were in fact deliberate due to intelligence failures or due to the lack of AI accuracy or the parameters that were even set for these systems in terms of how many casualties they were willing to accept. So it actually muddies the accountability altogether while also obscuring what the LLM is actually doing here. And so, you know, it's very difficult for us to say if this was due to an AI mistake or if this was deliberate or not. And that's exactly why a lot of militaries are using AI. It obscures that.

[B]

Right, right. Because I had been reading that, like, the school was at one time a base for the irgc. Right. And so is it possible that that information was just scraped from some kind of system and then analyzed using these tools?

[C]

It is a possibility, but it's impossible for us to know. And also, you know, when we're talking about these systems as well, a lot of the times they provide these recommendations and it could be on outdated intelligence. So it could have been a combination of both intelligence failures and also the model really not understanding that that data was, you know, no longer relevant. And that's exactly why these models shouldn't be used for targeting, because they have a very difficult time making these types of distinctions where a human might be able to. It also very well could be that this was deliberate or not the fault of the AI.

[B]

Right.

[C]

But as I said, it's very hard to know.

[B]

Claude, through this partnership with Palantir, was Also used by the Pentagon during the US raid to capture Venezuelan President Nicolas Maduro, according to the Wall Street Journal, though there is not a lot of details on the precise role there either. And just like what other examples of military AI use do we know about in recent years in the US or elsewhere, really?

[C]

So there's a lot of different types of AI, and I just like to always make it clear that we've had AI be used in the military, including within the U.S. maybe even since the 1960s, as far back as that. That's when a lot of research started on using these types of systems. But those are very different types of AI than what's being deployed today, and that being large language models. Right. Even Maven initially used very different types of AI before large language models came in. And then they sort of had the partnership with Anthropic. And the thing about these different types of AI is that military purpose built AI is what we call sort of the former category tends to be very task specific. It's trained on specific sets of data and very specific sets of tasks. LLMs, on the other hand, or generative AI are very general purpose and then they're kind of like fine tuned to be repurposed for military purposes. But the thing to remember is that military purpose built AI or task specific AI tends to be more accurate than general purpose models like LLMs. LLMs have an incredibly low accuracy rate. You're looking at a lot of hallucinations, you're looking at something like 25 to 50% accuracy, and yet they're still being deployed. And that's not to say that other types of AI weren't that accurate either, but at least they were. You know, instead of us going towards a direction of maybe stepping away or understanding a little bit more how to responsibly deploy AI in the military, instead we went to the complete opposite direction. We're deploying something that's even worse than what we had before. So I would say if we're talking about different types of conflicts, yes, we have seen AI. There's been a lot of investigations on drone strikes in Afghanistan and Syria and Iraq even. But that's very different from what we're seeing today, where we're essentially having a system really make a final decision on, on everything, and they're just simply not capable of doing that.

[B]

Right, and just to be super clear here about what you're saying is it's the same kind of system as these chatbots that we're using? Yes, right. Like OpenAI's ChatGPT that regularly hallucinates and gets stuff wrong. Exactly, yeah.

[C]

What they actually do is they take these everyday models that you and I have been using and, and they input new data on top of it and essentially you basically have the same baseline model with a bit more, you know, context on military targeting, for example. But it's still the same type of hallucinations, it's still the same lack of accuracy. So these are an improved model, as you know, they would like us to believe.

[B]

Foreign We've talked on the show before about this really explosive reporting from the Guardian and 972 magazine that revealed through intelligence sources how Israel used AI at one point to identify like 37, 000 potential human targets based on their apparent links to Hamas during the war in Gaza. This AI system was called Lavender. The reporters interviewed people who said they would take the information and drop a dumb bomb on a target, often killing an entire household of people. And is this what we're talking about here? Was this generative AI?

[C]

So actually Gospel and Lavender did not rely on large language models, but they used different other types of AI to essentially have the same outcome. But after that we did receive confirmation that both GPT4 and Google Gemini, which are language models, they were then eventually used to generate and validate targets. And you know, we started seeing hints of that when we saw the cloud contracts with Google and Microsoft that were ramped up after October 7th. So even though they had Lavender and Gospel prior to that, as soon as they saw the opportunity to use large language models, they went ahead, sought out these tools to deploy them very quickly. And regardless of the type, the AI algorithms are all looking to do the same thing, right? They're looking to generate as many targets as possible and they all ingest similar types of information and then they give recommendations, so they're used towards the same purposes. And as I mentioned, both types of AI really have accuracy problems here.

[B]

And just big picture, how would you say the use of these LLMs is changing modern warfare? What we're seeing today?

[C]

I'd actually argue that it isn't changing it. I think when you keep in mind the abysmal accuracy rates and the speed of LLMs, it's almost just a high tech version of carpet bombing. And what we're actually seeing is that AI is being used to evade accountability. The very use of these systems, like with the school, will make it very difficult to say, okay, whose fault was this? And so, you know, when you're looking at these types of issues together, the evasion of accountability and the fact that these systems aren't accurate, we're just going towards, you know, a world where we're no longer following the traditional legal rules of war. And I think that's very, very concerning. So to me, we know we're just seeing us move away from that accountability, from understanding the type of targets that we have, from us understanding the decisions that are even being made. Because with AI we don't know why some of the decisions, decisions are being made. So it's not, to me, it's not an advancement, but rather a regression.

[B]

Right, right. Like I've just seen legal experts talk about how they worry that what this will ultimately result in is rubber stamping, automatic strikes with no meaningful human oversight. Do you say that? That's a pretty good summary, yes.

[C]

And I think it's really important to remember here, especially, you know, if we're looking at, and specifically Anthropic, because they had this fallout with the Department of War. A lot of people are talking about, well, Anthropic has a strong red line because they're talking about they don't want their models to be deployed for autonomous weapons systems and they don't want less human oversight. I don't consider that a moral high ground because actually in practice the difference between decision support systems, which is how they're currently being used, and AI driven autonomous weapon system is this human in the loop that you're talking about. And that's impacted by what in our field we call automation bias. And automation bias is this idea based on decades of research showing that humans often trust the recommendations of algorithms without corroborating with other sources or checking if that recommendation is correct or not. So it does in the end of the day, you know, end up being grubber stamping. And then again, when you combine that with the knowledge that these things aren't very accurate, you, you're looking at the normalization of this technology that shouldn't be used for targeting. And I think that has huge implications of the worst to come. And we already have a very good example of that already, which is what happened in Gaza where these types of AI systems were also used by the idf.

[B]

Right, right. In Gaza. I know a lot of people called RESA as well. Just for people listening, what's your worst fear here? Do you have a nightmare scenario that, that keeps you up at night?

[C]

Yeah, I think we are in the nightmare scenario when it comes to military use. I think something I'm really worried about is also giving these models access to nuclear weapons. And when we're Talking about things like autonomous weapon systems. And then we're talking about, you know, we saw scale AI, which is a different AI company, recently land a contract on nuclear command and control. It really makes you question, what's the next step? Here we are already in such a terrible worst case scenario in terms of giving these models ability to target with very little human oversight. And now we're seeing just the expansion of that. The fact that we're even having the conversation about autonomous weapon systems was scary for me that no one took a step back and say, wait, hold on, they shouldn't even be used for decision support systems. And here we are talking about autonomous weapons systems and now there's nuclear command and control. And I think, you know, we're actually deploying these systems in a very, very, very reckless way. And it seems like there's no stopping. And you know, a lot of people are hoping that Anthropic is a hero here and they'll draw the red line, but at the end of the day, they're back on the negotiating table with the Pentagon as of today. And at the end of the day, Daryu Amade, who's the CEO of Anthropic, came out and said, oh no, I'm not opposed to autonomous weapons and systems. We just, we just don't think they're reliable enough. And we wanted to work on developing them, you know, so they don't have those red lines. So if the governments aren't doing and the companies aren't doing it, this puts us in a very, very bad position of what's to come next.

[A]

Hey, this is Adam Grant, host of ted's podcast, Rethinking with Adam Grant. Have you heard of Bill? It's the intelligent finance platform that uses AI to help you avoid costly errors and optimize cash flow. In fact, Bill reports that over 90 of the top 100 US accounting firms trust them to manage, move and maximize money. Proven by over a trillion dollars in secure transactions. Eliminate the friction and start scaling with the proven choice. Visit bill.com proven to talk with an expert about automating your business finances. And get a $250 gift card as the thank you. That's bill.com proven terms and conditions apply. See offer page for details.

[C]

With VRBoCare, help is always ready before, during and after your stay. We've planned for the plot twists, so support is always available because a great trip starts with peace of mind.

[B]

I want to dig into this whole anthropic OpenAI kind of fight, I guess, with you just in one Minute. But first I wonder if I could ask you, like, what I think might an incredibly obvious question, but also I think might be helpful, like, what would an autonomous weapon system do?

[C]

This is actually a really good question because I think a lot of people think of killer drones. That's the first thing that they think of.

[B]

And they, I mean, I am thinking of killer drones.

[C]

Exactly. There's many different types of autonomous weapon systems. In fact, you have organizations, international organization that consider minds, autonomous weapon systems. Right. They could look like many things. When you're talking about drones, that's a very different type of AI. You need an AI with specific skills like object recognition. These are not general purpose models. And then based on that and you know, some data that's fed into them, they then sort of lock in and target something. That's very different from how we're using frontier AI models which run on these enormous cloud systems in just a huge amount of information and based on that, give recommendations. And if we're thinking about a targeting recommendation, you can then use any type, type of weapon to strike that target. Right? So it could be a drone. You could send a drone off after that target has been identified. It could be a missile if it's an infrastructure target. So there's many different levels of autonomous weapon systems. And I think most people really have focused on this like, you know, killer drone, drone swarms. But that's actually not where frontier AI or cloud is being used. They're using on a much bigger scale where they're given many different types of targets. But typically a human then takes that recommendation and decide, right, Like I'm going to send a missile after that. Right. Or I'm going to and a drone there. But the idea is then to eliminate that human that when we're talking about frontier AI, to eliminate all that together. So whatever recommendation is given, the model itself can even decide the type of weapon that's being used. We saw scale AI sort of give demos on that using large language models. Like it's a different company that also had a lot of contracts with the Department of War. But, you know, it's like, what would be the best type of weapon to use for this recommendation? And so the idea is to just eliminate the human from that loop and just to automate that process.

[B]

Got it. And just make it so that maybe a human, just as we were talking before, kind of rubber stamps it. Yeah, exactly. Okay, so given that, I mean, given this request that the Pentagon had of Anthropic, that they wanted the company to remove restrictions on Things like autonomous weapons and also large scale surveillance.

[D]

And so we have said to the Department of War that we are okay with all use cases. Basically 98 or 99% of the use cases they want to do, except for two that we're concerned about. One is domestic mass surveillance there. We're worried that things may become possible with AI that weren't possible before. Case number two is fully autonomous weapons. This is the idea of making weapons that fire without any human involvement.

[B]

I just wonder if you could tell me a little bit more about what that told you about what the Pentagon ultimately wants to do here, what direction they want to go in with AI.

[C]

I think they very clearly want to use these technologies as an alibi for whatever actions they want to carry out. Like I said, when we go back to the point that AI really helps you evade accountability, it makes people question, was this deliberate? Did they do this on purpose? And it also makes it very easy to say things like, well, the AI determined it, so it must be true, right? There's, I think, a group of people who view AI decisions as being more objective than humans. And that's not the case in any. In any way. And so it's much easier to hide behind AI decision and not have to justify them. So I think when they want no guardrails whatsoever, it generally shows that they want to be able to use these systems in whichever capacity they feel like.

[B]

On the surveillance stuff, the Atlantic reported that Anthropic was told the Department of Defense, I guess the Department of War now wanted to use Claude to analyze bulk data collected from Americans. So everything I think you type into a chatbot, right, all your search, your credit card history, and just what could be done by the military with that level of information.

[C]

So I think it's really important to note that these models are what we call dual use. So they can be used for civilian purposes and they could be used for military purposes. And when you essentially start thinking about how they already started using these models for intelligence gathering, right? And intelligence sort of analysis, you start to think, right, this can also be used on, you know, the citizens of that country themselves because it's the same feature. So if you collect data from different sources so that things like data brokers, location data, Internet habits, and that's not just the things you're using the model for. That's really commercial data that you can buy from anyone, from data brokers, as we call them, and then you train or input these specific Data points into LLMs. You can then draw inferences on whom these Individuals are and track them accordingly. And again, as I mentioned, it might not be accurate, but you might be making decisions off of that, which again, is really, really problematic.

[B]

The US Undersecretary for Defense who is negotiating the deal with Anthropic was defending the position and he said, well, at one point he said, quote, at some

[D]

level, you have to trust your military to do the right thing. And if anything, we're the biggest, you know, organization in the world with the most rules of any organization in the world.

[C]

And.

[D]

But we do have to be prepared for the future.

[B]

He also made the argument that what Anthropic was afraid of or what people were afraid of was already barred by the law. And just. Does he have a point there?

[C]

I guess I would say that the Department of War conducts its own legal reviews and compliance with international law, and they likely see that Anthropic is overstepping its role in these determinations, including their legal judgment or whether or not to deploy these systems. However, from an international law perspective, there's going to be different interpretations of what the lawfulness and predictability of these AI based autonomous weapons systems that differs from how the current Department of War currently chooses to interpret them. So the international community output is necessary here. And we haven't seen any consultations either from orgs like the UN or the ICRC on these types of decisions, especially when you have Dario raising concerns that AI is too unreliable. And that would support an interpretation, actually, that LLM based autonomous weapon systems are not in line with international law. And either way, we shouldn't be at the whims of a private corporation red lines on whether or not this dangerous technology should or should it be deployed. But I think it's at the end of day also important to remember that there is international law. Right. And it's not just is it private corporation versus a Department of War. Right. There's already, you know, restrictions on what types of autonomous weapon systems can be used. And currently, when we're looking and when Dario admits himself that these models are reliable, it says a lot about sort of their legal status, if that makes sense.

[B]

Yeah, Though I guess some people might be listening to this right now and, and feeling like maybe they don't have a lot of confidence in international law at the moment. Just state that obvious fact. Look, like I, I take your point that, you know, you don't want to make Anthropic out here to be the hero and that CEO to be the hero, because they are back at the table now negotiating with the Pentagon. But just I Wanted to ask you about the threat to declare them a supply chain risk.

[C]

Yes.

[B]

And then also obviously that coming to fruition. Right. This label has never been used against the US company as I understand it, and would bar the, the company from any work with the government. And just like how big of a deal is that for a company like Anthropic? I mean it's quite a lever for the government to pull.

[C]

Definitely, definitely. Because ultimately this is the first frontier AI company that's been working with the military and they're embedded with them. So to suddenly use that label clearly doesn't align with what's already been happening. But on the point of supply chain, I actually do believe that all LLMs are a supply chain threat to national security and defense. And a lot of people internally in the military also believe the same things. And you know, this isn't about the companies themselves being foreign adversary in any way, which they're American companies, they're clearly not. But this is about the nature of LLMs themselves that I think is really important for people to understand is that they're trained on the open Internet and publicly available information. Again, very different from purpose military built models, models where every data point and software decision is traceable. So as a result of these being built on an unprotected supply chain, there's a lot of new and undetectable attack vectors which include things like poisoning web training data sets or building backdoors into these models which actually may intentionally or inadvertently lead to the subversion of their behavior, which includes in military applications. And this isn't something that can be patched. And in fact a study by Anthropic themselves showed that you only need about 250 malicious documents to produce a backdoor into one of these models. And so when you consider that with the news that China and Russia have been compromising online information to influence the answers of large language models, it's likely that these backdoors already exist. So I think it's very difficult for me to speak on the Departments of War decision of why they would choose to say this about Anthropic, but I actually do think large language models as a whole do have supply chain issues.

[B]

I'm reticent to like repeat what people have just heard themselves, but I just, I do think it's really important to just put a, a line underneath this that essentially what you're saying here is that it's not Anthropic that is the threat to national security, it's the data that all this stuff is trained on and that an enemy of the United States you mentioned China or Russia can mess around with the data and it can affect the outcome.

[C]

Exactly.

[B]

What did you make of OpenAI's decision to kind of swoop in and make a deal with the Department of War after this falling out with Anthropic? Is that deal materially different from what Anthropic had wanted to do? You worked at OpenAI. I would be very curious to hear your reaction to this.

[C]

Yeah, I think it was very clear to me that it was not the same deal, although they claimed that it was. It really wasn't. Especially when you're familiar with military contract and these like open ended words that are being used. Right. And so it's very different language from what Anthropic was fighting for. And initially they came out as saying, oh, we just managed to get the same deal. And then it ultimately came out that that wasn't the case. Right. And so it seems that they threw Anthropic under the bus while also trying to sort of benefit from the same pr. And also in their announcement because they talk about a safety stack. And this is what I work and I work on safety. I would say the safety guardrails that OpenAI is referring to are not operationally feasible because when you have a model and it gives you an output, you can't monitor after that what that individual does without. They can't double check that that's being overseen by a human. It could just very well be taken to then select and engage a target without any further oversight, making it an autonomous weapon. Even if this is deployed on the cloud. Because this is an operational matter, not a technical one. Whether or not you put a human in the loop.

[B]

Right. I think Sam Altman has like basically admitted that. Right. Like that they don't. They won't be able to actually know what the Department of War was doing with their technology.

[C]

Exactly. So, so when they say we have the safety stack is going to be on the cloud and that's going to somehow limit it, I think that's very, very much misleading.

[B]

How would you describe how a company like OpenAI approaches safety?

[C]

I would call it safety Theater, Safety co option, because ultimately a lot of these AI companies like to use the same terms that safety engineers use and reference. My background is safety engineering, which is we work on safety critical systems like nuclear plants, aviation, autonomous vehicles, where if they fail, human lives are at risk. They take a lot of that same terminology and essentially kind of safety wash everything with it, every sort of mission that they have they eventually rolled back on, like military use. That was eliminated in 2024 when it was initially part of their charter or their terms of service. We have a lot of safety incidents that occur all the time with people using their models. There's no guardrails really. And so I would say for a lot of people, you know, it's. They believe, they often believe the message in like, oh, we care and so on, so forth, but they can write as much as they want. If you just look at the actions of these companies, these AI models are causing a huge amount of harm.

[B]

Contracts with militaries are very enticing for AI companies. Right. For, for a couple of different reasons. And could you just lay those out for me?

[C]

Well, these AI models are incredibly expensive to train and it's actually likely the case that they're not even breaking even on being able to generate the revenue needed to keep producing these big models, training them and deploying them. So when you have the military, that's a really big money pot for you to be able to make that money back. And it also embeds you within military and safety critical infrastructure, which means that you're then too big to fail. So for them, you know, when there's a huge amount of backlash right now against the uses of AI, this is a very secure position for them to have monetarily and also in practice because, you know, as we've seen with the reporting of claude, the entire military infrastructure now relies on CLAUDE for this type of analysis and it's very hard for them to disconnect from it. So although they wanted a ban, they were like, oh, we're going to try to phase it out. And they still use the interod, right. It just shows the type of dependence and this dependence is very, very strategic for them.

[B]

I want to kind of go back to state power. You know, we've been talking about how much power these private companies have in influencing how wars are fought. But when you look at the pace of AI development, when you see the military interest in this, the money being spent on it, is one of the real main worries here that what we are witnessing right now is AI becoming a central tool of state power.

[C]

Absolutely. It is the perfect tool to concentrate power because you're collecting essentially all data possible on humans, on our behavior, on anything really. It even goes beyond humans. Right. You're also collecting data on states, on everything, and then allows them to use those same very systems, are able to collect that huge amounts of information, be trained on it, to then make any sort of decision and it becomes very centralized to them and there's no accountability because these models are black boxes. No one now can question, why did you make that decision? Like, we're seeing these models be used for immigration rates by ice. We're seeing youth in the judicial systems and military, and yet there are black boxes. And now it feels like no one can question that power. This is an unfortunate side effect of something like these very large general purpose models.

[B]

Okay. Seems like a good place for us to end. Heidi, thank you.

[C]

Thank you so much. Foreign.

[B]

That is all for today. Front Burner was produced this week by Joytha Shankupta, Shannon Higgins, Matthew Amha, Lauren Donnelly and Mackenzie Cameron. Our intern is riley Cunningham. Our YouTube producer is John Lee. Our music is by Joseph Chavison. Our senior producers are Imogen Burchard and Elaine Chow Out. Our executive producer is Nick McKay Blokos. And I'm Jamie Poisson. Thanks so much for listening.

[A]

For more cbc podcasts, go to cbc ca podcasts.