C (3:07)

Yeah. So Palantir is a data analysis company and they work a lot on intelligence and surveillance. And back in 2024, anthropic and palantir had agreement where essentially Anthropic's model, that being Claude, would then power sort of this recommendation engine, which takes all of the enormous amounts of data that Palantir has sort of analyzed, collected, and sorted through, and then feed it to an AI to then make some sort of decision.

B (3:34)

And so I mentioned that strike on the girls school in Iran. The Pentagon is denying targeting civilians. Israel says that there was no IDF operations in the area. But is it possible that AI was used in some form here?

C (3:53)

I think that's actually a very good question because it brings the point forward that AI is actually being used to evade accountability. And that's because when you're using these types of systems, it makes it difficult to distinguish if some attacks were in fact deliberate due to intelligence failures or due to the lack of AI accuracy or the parameters that were even set for these systems in terms of how many casualties they were willing to accept. So it actually muddies the accountability altogether while also obscuring what the LLM is actually doing here. And so, you know, it's very difficult for us to say if this was due to an AI mistake or if this was deliberate or not. And that's exactly why a lot of militaries are using AI. It obscures that.

B (4:39)

Right, right. Because I had been reading that, like, the school was at one time a base for the irgc. Right. And so is it possible that that information was just scraped from some kind of system and then analyzed using these tools?

C (4:55)

It is a possibility, but it's impossible for us to know. And also, you know, when we're talking about these systems as well, a lot of the times they provide these recommendations and it could be on outdated intelligence. So it could have been a combination of both intelligence failures and also the model really not understanding that that data was, you know, no longer relevant. And that's exactly why these models shouldn't be used for targeting, because they have a very difficult time making these types of distinctions where a human might be able to. It also very well could be that this was deliberate or not the fault of the AI.

B (5:27)

Right.

C (5:27)

But as I said, it's very hard to know.

B (5:40)

Claude, through this partnership with Palantir, was Also used by the Pentagon during the US raid to capture Venezuelan President Nicolas Maduro, according to the Wall Street Journal, though there is not a lot of details on the precise role there either. And just like what other examples of military AI use do we know about in recent years in the US or elsewhere, really?

C (6:04)

So there's a lot of different types of AI, and I just like to always make it clear that we've had AI be used in the military, including within the U.S. maybe even since the 1960s, as far back as that. That's when a lot of research started on using these types of systems. But those are very different types of AI than what's being deployed today, and that being large language models. Right. Even Maven initially used very different types of AI before large language models came in. And then they sort of had the partnership with Anthropic. And the thing about these different types of AI is that military purpose built AI is what we call sort of the former category tends to be very task specific. It's trained on specific sets of data and very specific sets of tasks. LLMs, on the other hand, or generative AI are very general purpose and then they're kind of like fine tuned to be repurposed for military purposes. But the thing to remember is that military purpose built AI or task specific AI tends to be more accurate than general purpose models like LLMs. LLMs have an incredibly low accuracy rate. You're looking at a lot of hallucinations, you're looking at something like 25 to 50% accuracy, and yet they're still being deployed. And that's not to say that other types of AI weren't that accurate either, but at least they were. You know, instead of us going towards a direction of maybe stepping away or understanding a little bit more how to responsibly deploy AI in the military, instead we went to the complete opposite direction. We're deploying something that's even worse than what we had before. So I would say if we're talking about different types of conflicts, yes, we have seen AI. There's been a lot of investigations on drone strikes in Afghanistan and Syria and Iraq even. But that's very different from what we're seeing today, where we're essentially having a system really make a final decision on, on everything, and they're just simply not capable of doing that.

B (8:04)

Right, and just to be super clear here about what you're saying is it's the same kind of system as these chatbots that we're using? Yes, right. Like OpenAI's ChatGPT that regularly hallucinates and gets stuff wrong. Exactly, yeah.

C (8:20)

What they actually do is they take these everyday models that you and I have been using and, and they input new data on top of it and essentially you basically have the same baseline model with a bit more, you know, context on military targeting, for example. But it's still the same type of hallucinations, it's still the same lack of accuracy. So these are an improved model, as you know, they would like us to believe.

B (8:50)

Foreign We've talked on the show before about this really explosive reporting from the Guardian and 972 magazine that revealed through intelligence sources how Israel used AI at one point to identify like 37, 000 potential human targets based on their apparent links to Hamas during the war in Gaza. This AI system was called Lavender. The reporters interviewed people who said they would take the information and drop a dumb bomb on a target, often killing an entire household of people. And is this what we're talking about here? Was this generative AI?

C (9:29)

So actually Gospel and Lavender did not rely on large language models, but they used different other types of AI to essentially have the same outcome. But after that we did receive confirmation that both GPT4 and Google Gemini, which are language models, they were then eventually used to generate and validate targets. And you know, we started seeing hints of that when we saw the cloud contracts with Google and Microsoft that were ramped up after October 7th. So even though they had Lavender and Gospel prior to that, as soon as they saw the opportunity to use large language models, they went ahead, sought out these tools to deploy them very quickly. And regardless of the type, the AI algorithms are all looking to do the same thing, right? They're looking to generate as many targets as possible and they all ingest similar types of information and then they give recommendations, so they're used towards the same purposes. And as I mentioned, both types of AI really have accuracy problems here.

B (10:31)

And just big picture, how would you say the use of these LLMs is changing modern warfare? What we're seeing today?

C (10:42)

I'd actually argue that it isn't changing it. I think when you keep in mind the abysmal accuracy rates and the speed of LLMs, it's almost just a high tech version of carpet bombing. And what we're actually seeing is that AI is being used to evade accountability. The very use of these systems, like with the school, will make it very difficult to say, okay, whose fault was this? And so, you know, when you're looking at these types of issues together, the evasion of accountability and the fact that these systems aren't accurate, we're just going towards, you know, a world where we're no longer following the traditional legal rules of war. And I think that's very, very concerning. So to me, we know we're just seeing us move away from that accountability, from understanding the type of targets that we have, from us understanding the decisions that are even being made. Because with AI we don't know why some of the decisions, decisions are being made. So it's not, to me, it's not an advancement, but rather a regression.

B (11:46)

Right, right. Like I've just seen legal experts talk about how they worry that what this will ultimately result in is rubber stamping, automatic strikes with no meaningful human oversight. Do you say that? That's a pretty good summary, yes.

C (12:01)

And I think it's really important to remember here, especially, you know, if we're looking at, and specifically Anthropic, because they had this fallout with the Department of War. A lot of people are talking about, well, Anthropic has a strong red line because they're talking about they don't want their models to be deployed for autonomous weapons systems and they don't want less human oversight. I don't consider that a moral high ground because actually in practice the difference between decision support systems, which is how they're currently being used, and AI driven autonomous weapon system is this human in the loop that you're talking about. And that's impacted by what in our field we call automation bias. And automation bias is this idea based on decades of research showing that humans often trust the recommendations of algorithms without corroborating with other sources or checking if that recommendation is correct or not. So it does in the end of the day, you know, end up being grubber stamping. And then again, when you combine that with the knowledge that these things aren't very accurate, you, you're looking at the normalization of this technology that shouldn't be used for targeting. And I think that has huge implications of the worst to come. And we already have a very good example of that already, which is what happened in Gaza where these types of AI systems were also used by the idf.

B (13:16)

Right, right. In Gaza. I know a lot of people called RESA as well. Just for people listening, what's your worst fear here? Do you have a nightmare scenario that, that keeps you up at night?

C (13:27)

Yeah, I think we are in the nightmare scenario when it comes to military use. I think something I'm really worried about is also giving these models access to nuclear weapons. And when we're Talking about things like autonomous weapon systems. And then we're talking about, you know, we saw scale AI, which is a different AI company, recently land a contract on nuclear command and control. It really makes you question, what's the next step? Here we are already in such a terrible worst case scenario in terms of giving these models ability to target with very little human oversight. And now we're seeing just the expansion of that. The fact that we're even having the conversation about autonomous weapon systems was scary for me that no one took a step back and say, wait, hold on, they shouldn't even be used for decision support systems. And here we are talking about autonomous weapons systems and now there's nuclear command and control. And I think, you know, we're actually deploying these systems in a very, very, very reckless way. And it seems like there's no stopping. And you know, a lot of people are hoping that Anthropic is a hero here and they'll draw the red line, but at the end of the day, they're back on the negotiating table with the Pentagon as of today. And at the end of the day, Daryu Amade, who's the CEO of Anthropic, came out and said, oh no, I'm not opposed to autonomous weapons and systems. We just, we just don't think they're reliable enough. And we wanted to work on developing them, you know, so they don't have those red lines. So if the governments aren't doing and the companies aren't doing it, this puts us in a very, very bad position of what's to come next.

A (15:12)

Hey, this is Adam Grant, host of ted's podcast, Rethinking with Adam Grant. Have you heard of Bill? It's the intelligent finance platform that uses AI to help you avoid costly errors and optimize cash flow. In fact, Bill reports that over 90 of the top 100 US accounting firms trust them to manage, move and maximize money. Proven by over a trillion dollars in secure transactions. Eliminate the friction and start scaling with the proven choice. Visit bill.com proven to talk with an expert about automating your business finances. And get a $250 gift card as the thank you. That's bill.com proven terms and conditions apply. See offer page for details.

C (15:49)

With VRBoCare, help is always ready before, during and after your stay. We've planned for the plot twists, so support is always available because a great trip starts with peace of mind.

B (16:03)

I want to dig into this whole anthropic OpenAI kind of fight, I guess, with you just in one Minute. But first I wonder if I could ask you, like, what I think might an incredibly obvious question, but also I think might be helpful, like, what would an autonomous weapon system do?

C (16:21)

This is actually a really good question because I think a lot of people think of killer drones. That's the first thing that they think of.

B (16:27)

And they, I mean, I am thinking of killer drones.

C (16:29)

Exactly. There's many different types of autonomous weapon systems. In fact, you have organizations, international organization that consider minds, autonomous weapon systems. Right. They could look like many things. When you're talking about drones, that's a very different type of AI. You need an AI with specific skills like object recognition. These are not general purpose models. And then based on that and you know, some data that's fed into them, they then sort of lock in and target something. That's very different from how we're using frontier AI models which run on these enormous cloud systems in just a huge amount of information and based on that, give recommendations. And if we're thinking about a targeting recommendation, you can then use any type, type of weapon to strike that target. Right? So it could be a drone. You could send a drone off after that target has been identified. It could be a missile if it's an infrastructure target. So there's many different levels of autonomous weapon systems. And I think most people really have focused on this like, you know, killer drone, drone swarms. But that's actually not where frontier AI or cloud is being used. They're using on a much bigger scale where they're given many different types of targets. But typically a human then takes that recommendation and decide, right, Like I'm going to send a missile after that. Right. Or I'm going to and a drone there. But the idea is then to eliminate that human that when we're talking about frontier AI, to eliminate all that together. So whatever recommendation is given, the model itself can even decide the type of weapon that's being used. We saw scale AI sort of give demos on that using large language models. Like it's a different company that also had a lot of contracts with the Department of War. But, you know, it's like, what would be the best type of weapon to use for this recommendation? And so the idea is to just eliminate the human from that loop and just to automate that process.

B (18:14)

Got it. And just make it so that maybe a human, just as we were talking before, kind of rubber stamps it. Yeah, exactly. Okay, so given that, I mean, given this request that the Pentagon had of Anthropic, that they wanted the company to remove restrictions on Things like autonomous weapons and also large scale surveillance.

D (18:35)

And so we have said to the Department of War that we are okay with all use cases. Basically 98 or 99% of the use cases they want to do, except for two that we're concerned about. One is domestic mass surveillance there. We're worried that things may become possible with AI that weren't possible before. Case number two is fully autonomous weapons. This is the idea of making weapons that fire without any human involvement.

B (19:05)

I just wonder if you could tell me a little bit more about what that told you about what the Pentagon ultimately wants to do here, what direction they want to go in with AI.

C (19:16)

I think they very clearly want to use these technologies as an alibi for whatever actions they want to carry out. Like I said, when we go back to the point that AI really helps you evade accountability, it makes people question, was this deliberate? Did they do this on purpose? And it also makes it very easy to say things like, well, the AI determined it, so it must be true, right? There's, I think, a group of people who view AI decisions as being more objective than humans. And that's not the case in any. In any way. And so it's much easier to hide behind AI decision and not have to justify them. So I think when they want no guardrails whatsoever, it generally shows that they want to be able to use these systems in whichever capacity they feel like.

B (19:59)

On the surveillance stuff, the Atlantic reported that Anthropic was told the Department of Defense, I guess the Department of War now wanted to use Claude to analyze bulk data collected from Americans. So everything I think you type into a chatbot, right, all your search, your credit card history, and just what could be done by the military with that level of information.

C (20:24)

So I think it's really important to note that these models are what we call dual use. So they can be used for civilian purposes and they could be used for military purposes. And when you essentially start thinking about how they already started using these models for intelligence gathering, right? And intelligence sort of analysis, you start to think, right, this can also be used on, you know, the citizens of that country themselves because it's the same feature. So if you collect data from different sources so that things like data brokers, location data, Internet habits, and that's not just the things you're using the model for. That's really commercial data that you can buy from anyone, from data brokers, as we call them, and then you train or input these specific Data points into LLMs. You can then draw inferences on whom these Individuals are and track them accordingly. And again, as I mentioned, it might not be accurate, but you might be making decisions off of that, which again, is really, really problematic.

B (21:22)

The US Undersecretary for Defense who is negotiating the deal with Anthropic was defending the position and he said, well, at one point he said, quote, at some

D (21:32)

level, you have to trust your military to do the right thing. And if anything, we're the biggest, you know, organization in the world with the most rules of any organization in the world.

C (21:45)

And.

D (21:45)

But we do have to be prepared for the future.

B (21:48)

He also made the argument that what Anthropic was afraid of or what people were afraid of was already barred by the law. And just. Does he have a point there?

C (21:58)

I guess I would say that the Department of War conducts its own legal reviews and compliance with international law, and they likely see that Anthropic is overstepping its role in these determinations, including their legal judgment or whether or not to deploy these systems. However, from an international law perspective, there's going to be different interpretations of what the lawfulness and predictability of these AI based autonomous weapons systems that differs from how the current Department of War currently chooses to interpret them. So the international community output is necessary here. And we haven't seen any consultations either from orgs like the UN or the ICRC on these types of decisions, especially when you have Dario raising concerns that AI is too unreliable. And that would support an interpretation, actually, that LLM based autonomous weapon systems are not in line with international law. And either way, we shouldn't be at the whims of a private corporation red lines on whether or not this dangerous technology should or should it be deployed. But I think it's at the end of day also important to remember that there is international law. Right. And it's not just is it private corporation versus a Department of War. Right. There's already, you know, restrictions on what types of autonomous weapon systems can be used. And currently, when we're looking and when Dario admits himself that these models are reliable, it says a lot about sort of their legal status, if that makes sense.

B (23:28)

Yeah, Though I guess some people might be listening to this right now and, and feeling like maybe they don't have a lot of confidence in international law at the moment. Just state that obvious fact. Look, like I, I take your point that, you know, you don't want to make Anthropic out here to be the hero and that CEO to be the hero, because they are back at the table now negotiating with the Pentagon. But just I Wanted to ask you about the threat to declare them a supply chain risk.

C (23:57)

Yes.

B (23:58)

And then also obviously that coming to fruition. Right. This label has never been used against the US company as I understand it, and would bar the, the company from any work with the government. And just like how big of a deal is that for a company like Anthropic? I mean it's quite a lever for the government to pull.

C (24:18)

Definitely, definitely. Because ultimately this is the first frontier AI company that's been working with the military and they're embedded with them. So to suddenly use that label clearly doesn't align with what's already been happening. But on the point of supply chain, I actually do believe that all LLMs are a supply chain threat to national security and defense. And a lot of people internally in the military also believe the same things. And you know, this isn't about the companies themselves being foreign adversary in any way, which they're American companies, they're clearly not. But this is about the nature of LLMs themselves that I think is really important for people to understand is that they're trained on the open Internet and publicly available information. Again, very different from purpose military built models, models where every data point and software decision is traceable. So as a result of these being built on an unprotected supply chain, there's a lot of new and undetectable attack vectors which include things like poisoning web training data sets or building backdoors into these models which actually may intentionally or inadvertently lead to the subversion of their behavior, which includes in military applications. And this isn't something that can be patched. And in fact a study by Anthropic themselves showed that you only need about 250 malicious documents to produce a backdoor into one of these models. And so when you consider that with the news that China and Russia have been compromising online information to influence the answers of large language models, it's likely that these backdoors already exist. So I think it's very difficult for me to speak on the Departments of War decision of why they would choose to say this about Anthropic, but I actually do think large language models as a whole do have supply chain issues.

B (26:07)

I'm reticent to like repeat what people have just heard themselves, but I just, I do think it's really important to just put a, a line underneath this that essentially what you're saying here is that it's not Anthropic that is the threat to national security, it's the data that all this stuff is trained on and that an enemy of the United States you mentioned China or Russia can mess around with the data and it can affect the outcome.

C (26:33)

Exactly.

B (26:47)

What did you make of OpenAI's decision to kind of swoop in and make a deal with the Department of War after this falling out with Anthropic? Is that deal materially different from what Anthropic had wanted to do? You worked at OpenAI. I would be very curious to hear your reaction to this.

C (27:09)

Yeah, I think it was very clear to me that it was not the same deal, although they claimed that it was. It really wasn't. Especially when you're familiar with military contract and these like open ended words that are being used. Right. And so it's very different language from what Anthropic was fighting for. And initially they came out as saying, oh, we just managed to get the same deal. And then it ultimately came out that that wasn't the case. Right. And so it seems that they threw Anthropic under the bus while also trying to sort of benefit from the same pr. And also in their announcement because they talk about a safety stack. And this is what I work and I work on safety. I would say the safety guardrails that OpenAI is referring to are not operationally feasible because when you have a model and it gives you an output, you can't monitor after that what that individual does without. They can't double check that that's being overseen by a human. It could just very well be taken to then select and engage a target without any further oversight, making it an autonomous weapon. Even if this is deployed on the cloud. Because this is an operational matter, not a technical one. Whether or not you put a human in the loop.

B (28:14)

Right. I think Sam Altman has like basically admitted that. Right. Like that they don't. They won't be able to actually know what the Department of War was doing with their technology.

C (28:23)

Exactly. So, so when they say we have the safety stack is going to be on the cloud and that's going to somehow limit it, I think that's very, very much misleading.

B (28:31)

How would you describe how a company like OpenAI approaches safety?

C (28:36)

I would call it safety Theater, Safety co option, because ultimately a lot of these AI companies like to use the same terms that safety engineers use and reference. My background is safety engineering, which is we work on safety critical systems like nuclear plants, aviation, autonomous vehicles, where if they fail, human lives are at risk. They take a lot of that same terminology and essentially kind of safety wash everything with it, every sort of mission that they have they eventually rolled back on, like military use. That was eliminated in 2024 when it was initially part of their charter or their terms of service. We have a lot of safety incidents that occur all the time with people using their models. There's no guardrails really. And so I would say for a lot of people, you know, it's. They believe, they often believe the message in like, oh, we care and so on, so forth, but they can write as much as they want. If you just look at the actions of these companies, these AI models are causing a huge amount of harm.

B (29:38)

Contracts with militaries are very enticing for AI companies. Right. For, for a couple of different reasons. And could you just lay those out for me?

C (29:47)

Well, these AI models are incredibly expensive to train and it's actually likely the case that they're not even breaking even on being able to generate the revenue needed to keep producing these big models, training them and deploying them. So when you have the military, that's a really big money pot for you to be able to make that money back. And it also embeds you within military and safety critical infrastructure, which means that you're then too big to fail. So for them, you know, when there's a huge amount of backlash right now against the uses of AI, this is a very secure position for them to have monetarily and also in practice because, you know, as we've seen with the reporting of claude, the entire military infrastructure now relies on CLAUDE for this type of analysis and it's very hard for them to disconnect from it. So although they wanted a ban, they were like, oh, we're going to try to phase it out. And they still use the interod, right. It just shows the type of dependence and this dependence is very, very strategic for them.

B (30:42)

I want to kind of go back to state power. You know, we've been talking about how much power these private companies have in influencing how wars are fought. But when you look at the pace of AI development, when you see the military interest in this, the money being spent on it, is one of the real main worries here that what we are witnessing right now is AI becoming a central tool of state power.

C (31:11)

Absolutely. It is the perfect tool to concentrate power because you're collecting essentially all data possible on humans, on our behavior, on anything really. It even goes beyond humans. Right. You're also collecting data on states, on everything, and then allows them to use those same very systems, are able to collect that huge amounts of information, be trained on it, to then make any sort of decision and it becomes very centralized to them and there's no accountability because these models are black boxes. No one now can question, why did you make that decision? Like, we're seeing these models be used for immigration rates by ice. We're seeing youth in the judicial systems and military, and yet there are black boxes. And now it feels like no one can question that power. This is an unfortunate side effect of something like these very large general purpose models.

B (31:58)

Okay. Seems like a good place for us to end. Heidi, thank you.

C (32:03)

Thank you so much. Foreign.

B (32:13)

That is all for today. Front Burner was produced this week by Joytha Shankupta, Shannon Higgins, Matthew Amha, Lauren Donnelly and Mackenzie Cameron. Our intern is riley Cunningham. Our YouTube producer is John Lee. Our music is by Joseph Chavison. Our senior producers are Imogen Burchard and Elaine Chow Out. Our executive producer is Nick McKay Blokos. And I'm Jamie Poisson. Thanks so much for listening.

A (32:44)

For more cbc podcasts, go to cbc ca podcasts.