The Pentagon’s AI war machine

Summary

Podcast Summary: The Pentagon’s AI War Machine

The Gray Area with Sean Illing (Vox), April 20, 2026
Guest: Katrina Manson, National Security Reporter at Bloomberg, author of Project Maven
Host: Sean Illing

---

Episode Overview

This episode tackles the profound transformation of warfare due to artificial intelligence, focusing on the evolution and impact of Project Maven, the Pentagon's program to integrate AI into military operations. Sean Illing and Katrina Manson explore the historical origins, technological advances, ethical dilemmas, the Google rebellion, the capabilities of current AI warfare tools, and the debates raging between AI companies (like Anthropic) and the U.S. government over red lines and control.

---

Key Discussion Points and Insights

1. What Is Project Maven?

[03:23 – 06:05]

• Origins: Began in 2017 within the Pentagon to develop AI tools—specifically, computer vision to analyze drone footage in counterterrorism operations.
• Motivation: Fears of U.S. technological lag behind commercial sector advancements, especially with the dawn of automated and AI-driven systems in tech industries.
• Leadership: Conceived under Deputy Defense Secretary Bob Work; Colonel Drew Cukor (Marine) was a visionary leader and chief operator.
• Purpose Expansion: While initial language said it was for non-offensive purposes like analyzing ISIS footage, internal vision included targeting from the outset.

“He [Cukor] was very clear from the beginning that AI could put people out of work... For him, AI was a way of getting information to those on the front lines.” – Katrina Manson [06:09]

---

2. Early Challenges and Implementation

[07:48 – 12:28]

• First Deployments: Early Maven algorithms were messy, unreliable, and frustrating to field analysts, leading to poor adoption.
• Improvements: Negative feedback drove iteration. Success stories (e.g., detecting Marines through smoke in Afghanistan to prevent friendly fire) built trust.
• Human in the Loop: Early systems required human operators at crucial decision points; true autonomy was (and officially remains) a red line—but definitions are blurry.

“If the algorithm failed to identify the same object frame to frame, it would flash. So the operators were having real difficulty even looking at it... People would just turn it off.” – Katrina Manson [08:04]

“At what point am I no longer asking the analytical questions that are required for target engagement?” – Katrina Manson [10:53]

---

3. Intentions, Targeting, and the Google Revolt

[12:28 – 16:30]

• Original Charter vs. Actual Ambition: While memo language was narrow, leadership always envisioned offensive capabilities and targeting.
• Transparency Issues: Google employees protested when they discovered participation, leading to public outcry and Google pulling out.
• Ethical Reasoning: For some leaders, AI was justified by the promise of reducing civilian and friendly casualties.

“He always had targeting in mind from the outset… he was very clear with me that he always had targeting in mind. And he knew... he was going to be bulldozing through a part of [intelligence]." – Katrina Manson [12:46]

“The problem with war is the humans. They're materially corrupt, inefficient, and they get tired.” – (citing Drew Cukor) [15:10]

---

4. AI at Scale: Transformation of Military Targeting

[23:03 – 25:14]

• Maven Smart System: Built with Palantir; integrates computer vision and large language models (LLMs) like Claude (Anthropic) for battlefield management.
• Volume and Speed: AI-augmented systems can process and build thousands of targeting packages per day; numbers reported up to 5,000 in 24 hours.
• Implications: Humans retain legal checks, but the sheer scale raises fears about accountability and error rates.

“With the help of AI, Maven Smart system can now get through a thousand targets a day... They could now get to 5,000 targets in a day if they wanted to.” – Katrina Manson [24:16]

---

5. Toward Autonomous Weapons and Swarm Warfare

[25:14 – 29:46]

• Autonomy Aspirations: U.S. aims to develop fully autonomous drone swarms (e.g., “Hellscape” scenario for Taiwan defense). Still in developmental stages—main problem is interconnectivity and reliability.
• Bureaucratic Shifts: Language has shifted; the Pentagon now openly pursues "autonomous warfare."
• Private Sector Involvement: Major companies (SpaceX, XAI, OpenAI, Palantir) are competing to build voice-commanded drone swarms.

“The idea was to take some of those algorithms that Maven had produced, train them on data from the Indo Pacific… have that AI on the drone automatically detect the target and then be able to have the drone go and take the target out.” – Katrina Manson [25:14]

---

6. Accepting AI Risk: Black Boxes and Mistakes

[29:46 – 36:45]

• Testing and Drift: AI is inherently risky—vulnerable to errors, ‘drift,’ and hallucination. Accepting deployment means accepting some inevitable mistakes.
• Scenarios and Civilian Harm: Lower risk at sea, higher in urban environments. The risk calculation plays a role in where the U.S. chooses to deploy autonomous systems.
• Recent Controversies: Reports of civilian casualties (e.g., school bombing in Iran) raise questions about AI’s role and the reliability of targeting and cross-referencing civilian structures.

“If you are going to put it in a system where you can't see behind the hood, you're going to be relying on something that has inbuilt risk.” – Katrina Manson [30:14]

“If that school turns out to be on a military intelligence database when it should have been on the restricted target list, no AI can fix that.” – Katrina Manson [32:53]

---

7. Public-Private Struggle: Anthropic vs. Pentagon

[36:45 – 40:42]

• Anthropic’s Red Lines: Refused to support fully autonomous weapons or mass domestic surveillance. Relationship with Pentagon broke down, despite leaning in far beyond Google.
• Political and Practical Fallout: Pentagon wary of ceding control or facing private companies that dictate policy.
• The Manhattan Project Analogy: If AI is as transformative as nuclear weapons, should private tech firms have such central, profitable roles?

“By the time you have a frontier AI company that is the first to put its model on classified cloud, you have a company that's leaning in in a way that is not reminiscent of Google...” – Katrina Manson [37:09]

---

8. Ethics, Control, and the Future

[40:42 – 45:34]

• Host’s Concerns: Sean Illing expresses discomfort with making war easier and delegating lethal power to unaccountable technology and institutions.
• Transparency and Morality: Manson notes that even original Maven leaders oppose full autonomy with LLMs.
• Domestic Use and Surveillance Potential: Tools/tech could migrate from battlefield to law enforcement, raising fears about mass surveillance and civil liberties.

“My personal view is that war and killing people should be very hard and very costly and anything that makes it easier and faster and cheaper... I think that just makes killing people easier.” – Sean Illing [41:43]

---

9. Lack of Oversight and Societal Change

[46:42 – 49:47]

• Policy Vacuum: There is little to no meaningful regulation in the U.S.; Congress has yet to impose limits, in contrast with Europe.
• Societal Implications: The host warns that as war becomes increasingly technological, the essential human element may disappear, fundamentally altering the nature of conflict.

“You can only change the character of war so much before you change the nature of it entirely.” – Sean Illing [49:04]

---

Notable Quotes & Memorable Moments

• On Cukor’s View:

  “The problem with war is the humans. They're materially corrupt, inefficient, and they get tired.” – Sean Illing, quoting Drew Cukor [15:10]

• On Data and Surveillance:

  “The volume of data points available on any given individual, not with traditional intelligence, just with commercially available information from your phone or your route... all of these things.” – Katrina Manson [43:10]

• On Policy and Oversight:

  “Congress has spent a long time looking at AI, but there's no regulation.” – Katrina Manson [48:10]

• On Autonomy’s Direction:

  “Autonomy might be a hope, but it's also inevitable...look at the progress in the last 12, 18 months alone.” – Sean Illing [27:57]
  Response: “I'm always wary of the word inevitable. As a history student, I was taught nothing is ever inevitable.” – Katrina Manson [28:15]

---

Important Timestamps

• 03:23 – What is Project Maven?
• 06:09 – The vision and internal resistance
• 08:04 – Early technical challenges and successes
• 10:53 – "Human in the loop" discussion & shifting policy
• 12:46 – True ambitions—targeting and Google protestors’ concerns
• 15:10 – Quoting Drew Cukor’s critique of ‘the human problem’
• 23:03 – Scaling up: Maven Smart System and AI targeting packages
• 24:16 – Capacity: Processing up to 5,000 targets a day
• 25:14 – Drones, autonomy, and the future of swarming warfare
• 29:46 – Acceptable risk and the realities of black box AI
• 32:53 – Civilian casualties and the dangers of flawed data
• 37:09 – Anthropic’s conflict with the Pentagon
• 41:43 – Sean Illing expresses personal discomfort and risks
• 48:10 – Policy and absence of regulation
• 49:04 – The changing fundamental nature of war

---

Summary Tone

The conversation is nuanced, probing, and at times skeptical—rooted in deep reporting and first-person military experience. Both Manson and Illing maintain a tone of concern about the societal and ethical implications, even as they describe the astonishing technical and bureaucratic advances made in AI-driven warfare.

---

Conclusion

This episode presents a detailed, candid look at how Project Maven started as a Pentagon side project and evolved into a central pillar of American military transformation—aided by private tech giants in a messy, ethically fraught space. With targeting cycles that can process thousands of decisions a day, the stakes for errors and abuses are higher than ever. Regulatory gaps, the migration of these technologies to policing and surveillance, and ongoing disputes with AI firms make clear that the United States and its citizens have only begun to grapple with the realities of autonomous warfare.

Transcript

[Katrina Manson]

This episode is brought to you by Choiceology, an original podcast from Charles Schwab.

[Sean Illing]

Hosted by Katie Milkman, an award winning behavioral scientist and author of the best selling book how to Change. Choiceology is a show about the psychology

[Katrina Manson]

and economics behind people's decisions. Hear true stories from Nobel laureates, historians, authors, athletes and more about why people do the things they do and how

[Sean Illing]

to make better ones to help avoid costly mistakes. Listen to [e-mail] podcast or wherever you listen. Recommendations can be great. Maybe someone recommended this podcast and here you are. But home projects are a little different. If the podcast isn't your thing, you might lose a few minutes from your day. But if you hire your cousin's neighbor to mount your tv, you might end up with a lopsided screen and wall damage. I know a guy isn't a good strategy for your home. That's why thumbtack works so well. It matches you with top rated local pros and with photos, reviews and credentials all in one convenient place for your next home project. Try Thumbtack. Hire the Right Pro today. Back in 2017, there was a fringe experiment at the Pentagon called Project Maven. Most people knew nothing about it you probably still haven't heard of was basically a software project to help analysts sort through drone footage. But today these systems are all over the military, fusing massive streams of data, identifying targets at a scale that would have been totally impossible a decade ago. And they're being integrated with some of the most advanced AI models ever built. This right here is an example of how our society is simply not prepared for the AI revolution. We have this technology most of us don't fully understand, and it's changing very quickly how power can be exercised and how little friction stands in the way. This, I believe, is a monumental challenge for our democratic system. I'm Sean Illing and this is the Gray area. Today's guest is Katrina Manson. She's a national security reporter at Bloomberg who has just written a book about all of this. It's called Project Maven and it's the best piece of reporting I have seen on how this small AI program became central to to the Pentagon's push to fold AI into warfare and potentially much more. We talk about this history, why Google employees revolted when they discovered their company was involved, and what this all means now as companies like Anthropic and OpenAI negotiate how far their AI systems can be used by the American government. Katrina Manson, welcome to the show.

[Katrina Manson]

Thanks, thanks for having me.

[Sean Illing]

Let's just assume that Everyone listening and watching has never heard of Project Maven knows nothing about it. What is it?

[Katrina Manson]

It was an effort that began in 2017 for the Pentagon to develop an AI tool, develop a way of using AI on the battlefield so that AI could AI computer vision. A specific kind of AI could look at drone feed, video footage. This was the feed that the US was taking in the counterterrorism wars, the gwat global war on terror, apply AI to look at it better. That was ostensibly what it was, but it was also part of a much bigger effort to really bring AI to the battlefield and speed up the way the Pentagon was thinking about the future of war. The US had been in was still in Afghanistan, Iraq and Somalia, Yemen. There was a lot still going on and people were beginning to think, what about China? What if the US has to square up to China? And there was a group of people under the first Trump administration who said the US was behind. Despite having the biggest defense budget in the world, bar none, the US started to feel that it was using too unsophisticated tools and needed to catch up with what the commercial sector was doing. Bringing AI, trying out driverless cars. Could the US move to automated warfare in such a way? That folded in AI. So Project Maven was birthed by the deputy Defense secretary at the time, Bob Work. So it had serious buy in from the top leadership. But what the people who really tried to forward this always felt was that they were going to be up against it. They felt that they risked putting the intelligence side of the shop out of a job. If they could bring AI into operations that would essentially be cutting out or going around or somehow undermining the intelligence folks, or they thought they would, the intelligence folks would feel that way. And the colonel who was he wasn't the director. The director was a two star Air Force general, three star Air Force general named Jack Shanahan. But the colonel who led it as chief sort of the day to day operator, the doer who also had a lot of the vision was called Drew Cukor. He was a Marine colonel.

[Sean Illing]

What's his deal? What's his role in this history?

[Katrina Manson]

He was the chief of Project Maven and really got it going for the first five, six years. He was also one of the visionaries behind even pursuing it. And he worked tirelessly to try and bring AI warfare to life. He wanted to get AI out into the battlefield in as safe a way as possible, but to test it in as real life like scenarios as possible. And he came from this background of being Marine for Years before he started doing Project Maven, he'd been sent into Afghanistan in October 2001 and had lived through seeing Marines not have sufficient information to keep them safe. And he told me that he had carte blanche to fire anyone who got in his way, which immediately got me thinking, why would anyone be in your way? Surely you want to do the same thing? But the answer was no, they didn't. And he was very clear from the beginning that AI could put people out of work. It would test people's resolve, their mettle, their way of thinking about how war was done. And he came at that from this position of, I think it's fair to say, always believing that the operator had been unfairly, insufficiently supported. So the people who were on the front lines needed to get more information. And for him, AI was a way of getting information to those on the front lines.

[Sean Illing]

In the early deployments of Maven, if you were an analyst or a drone pilot or a targeter sitting at a screen in a Pentagon or Nevada or some forward operating base, how did Maven change your job in the really early days?

[Katrina Manson]

So if we say before 2020, the first two or three years, it was actually just a mess. It was such a mess that the people involved in Mabon would say, these algorithms don't work. So the first users they worked with were in Somalia. At the very end of 2017 into 2018, they used it. The system was so annoying, they stopped using it. They then sent someone to try and encourage them. So so much of what this Project Maven team, which was quite a small knit team of mostly Marines, but not just Marines, was trying to do was just to get someone to even try it. So it wasn't in all those places that you just listed. They did really well with Special Operations Command. The more forward leaning tech part of the military, you can have more personal relationships with the commanders and the operators. Some of them already had relationships with them in former deployments. So they drew on and it was simply to identify something. If you think about drone video footage, there are multiple frames in a second of footage, and the algorithm went to work on each frame. So if the algorithm failed to identify the same object frame to frame, it would flash. So the operators were having real difficulty even looking at it. And if the sensitivity was set very low, everything was being identified. So in some of those early experiments, there could be, you know, dozens, hundreds of boxes all flashing at the same time, and people would just turn it off. So they started to improve on that with the, with the, with the Very negative feedback that they got. One of the breakthroughs that I was told about is probably the 2018, 2019 timeframe. And there were some Marines who'd been booby trapped in a raid against a compound in Afghanistan. And so they were getting fired at from multiple places. And the wall had exploded. And the AI helped perceive through the smoke. The drone feed was picking it up. But the AI spotted through the smoke the individual Marines much more quickly than the human eye would have done. And of course, one of the problems the US has faced is it's called friendly fire. But not identifying their own people. And so being able to count out the Marines with AI very quickly in a specific kind of scenario like that made people start to believe in the potential of AI.

[Sean Illing]

But certainly at the time, there was still clearly a human in the loop. Right. These humans were making all the judgments. These were not autonomously directed. These systems weren't launching drone strikes of their own volition. Right. There were humans at various checkpoints involved.

[Katrina Manson]

Yes. And that language of human in the loop is really interesting because you have a lot of military commanders then. And since saying, we will always have a human in the loop, it's not actually technically the policy of the Defense Department or Department of War, as they call themselves now. The first directive on autonomy came in, I think, in 2012, and then it was updated in 23. And the update that was given in 23 says appropriate levels of. Appropriate levels of human judgment over the use of force. So it implies something a little closer to supervision rather than each who decides what's appropriate? Such a good question. And some of the reporting that's come out on the big fight between Anthropic and the Pentagon has focused in on the word appropriate and said, in the end, we're just quibbling over the word appropriate. But from the anthropic perspective, what is appropriate could determine whether a human is involved or not. At the time in 2018, 2019, 2020, there's no sense at all that an algorithm is making these decisions. But of course, people are beginning to wonder if I start relying on algorithms, if I start trusting the outputs without being able to check myself, at what point am I no longer asking the analytical questions that are required for target engagement? And again, they weren't at that stage at that point, but there was certainly, I think, always concern, discomfort from some of the people being asked to use it. And so it needed repetition and practice workflows, all that kind of thing.

[Sean Illing]

Initially, as you said, this was about they were using the software to sort through drone footage. But was the idea, the plan always to develop the tech, scale it up and deploy it across the entire military and defense department? Was that a clear vision from the jump?

[Katrina Manson]

It really depends who you ask. The memo itself that started Project Maven just talked about drone footage in the fight against ISIS and potentially extending to other defense intelligence purposes. When I did the research, a really important question for me was to establish two things. One was, tell me how you thought about targeting from the get go in terms of this project. Because when we haven't got there yet, but eventually Google protesters become very concerned about discovering that their company was working on this. So that's partly an issue of transparency, but they were concerned that they could be involved in the business of war. And Google said at the time, it's only for non offensive purposes. So I really wanted to check was that true? Was it always intended for non offensive purposes? Was Google correctly describing the project? And Drew Cukor told me he always had targeting in mind from the outset. That language is not in the memo. Others told me Drew Cukor would wince if you said this was a, a targeting project. But when I actually managed to speak to the man himself and when I went back and read his papers, his thesis, he believed in this idea of white dots that you could look at a map. Essentially he wrote this before we even had Google Maps. But imagine just looking at Google Maps, clicking with your cursor and being able to pick up the precise coordinate from your cursor and send a weapon to it. So he has his own, he was a Marine intelligence officer, his own papers describing this very idea. That then when he also suggests the idea for Project Maven and leads it, he was very clear with me that he always had targeting in mind. And he knew that he would be going up against eons of intelligence practice where there are specific programs used to take a coordinate, there are specific ways of checking elevation, there are specific things to do for geo rectification. It's obviously very, very comm. Complex system. Plus then there are processes of no strike list. But that system was the system that he wanted to not blow apart. But he knew he was going to be bulldozing through a part of it.

[Sean Illing]

He, Colonel Cukor, he's a very interesting character in the book and in this history really. And you know, you quote him in the book saying that the, the problem with war is the humans. They're materially corrupt, inefficient, and they get tired. You know, I, I'm familiar with this type of military officer, you know, Very often when they like rail against the bureaucracy and that sort of thing, they're really protesting all those pesky rules of engagement that make it harder to kill people. And you know, I served with people like this and I'm not saying their villains are bad people at all. I just think sometimes well intentioned people inside the war machine have a very hard time appreciating the importance of guardrails. And in their defense, it's not their job to do that. Right. Their job is to prosecute wars. But to me, that quote suggests what they're really looking for is easier ways. Obviously they want to save lives, right? Particularly their own troops. But they're looking for ways to make it easier and more efficient to kill. And that's a very dangerous game to play.

[Katrina Manson]

Deep. I mean, he is a very interesting person. I think he's very aware of that. Reid. He comes to the way he presents himself certainly is that he comes from a very moral place about the consideration of war. So yes, he does say those things, that humans are the problem with war. And he can sound cold in that sense, but he never suggested the rules of engagement should be diminished. And the main reason he puts forward in his conversations to commercial entities for why they should come on board with the Pentagon was always, we could save civilians this way. We could make sure we don't hurt our own. So I think for him, he had been sent to Afghanistan in 2001, the month after 9 11, and was one of those first targeting officers, intelligence officers, who was having to suggest targets. And of course, very quickly US military personnel were being hit by improvised explosive devices. And he was having to put together the packages, where should we go? Who should be hit? Who was the enemy? And was frustrated that there was so little information to protect U.S. personnel. And has also talked about other moments where he was frustrated the US Couldn't intervene in support of civilians because they didn't have the information. So the way he's always framed it, he's done the first part of what you said. It's almost brutal or brusque certainly. But he's never suggested he wanted to kill more people. He doesn't use rhetoric like that. Other people do. And other people on the team said to me anonymously, great, let's get AI now we can kill people all the time. So there are definitely people who considered that AI would help speed up and scale killing of the people perceived as America's enemies. He himself has a slightly different filter on it.

[Sean Illing]

Support for the show comes from Bombus. If your sock drawer could use a little love. You can upgrade to Bombus. They designed their socks with a keen eye for detail, offering everything from dress socks to sports socks. The latter is made with a cushioned sweat wicking design that stops them from sliding down your foot, which I hate while you stay active. This spring I wear my Bombus socks pretty much every day and I especially wear them when I'm running, which I do all the time and we're now entering the summer down here on the Gulf coast, which is essentially a giant sunny sauna. And the only socks I wear at this point are are my Bombas socks. Everything else gets too hot or too gross, but when I take off my shoes when I get back from a long run, my Bomba socks are still dry, still cool, as are my feet. I cannot recommend them enough. They really are great. They have more socks too with breathable soft, high quality basics including underwear and T shirts. You can go to bombas.com gray area and use code gray area for 20% off your first purchased that's B O M b-s.com gray area code gray area checkout. Support for the show comes from Shopify. Every thriving business starts with a series of what if questions. There's what if nobody likes me? But also what if they really really do? There's only one way to find out and you can make it happen with help from Shopify. They say millions of businesses around the world rely on Shopify for e commerce. From businesses just getting started to household name brands, it can help you with everything from payment processing to analytics to website design. You can choose from hundreds of templates to create a great looking website. Their email and marketing tools can help you get your name out there and stay connected with customers. And if you ever need help, Shopify's 247 award winning customer support has your back. You can turn those what ifs into a thriving business with Shopify today. You can sign up for your $1 per month trial period today at shopify.com Vox you can go to shopify.com Vox that's shopify.com Vox. Support for the show comes from Deleteme. Deleteme makes it easy, quick and safe to remove your personal data online at a time when surveillance and data breaches are common enough to to make everyone vulnerable. The reality is that we're all susceptible to having our private information stolen, public figures and private citizens alike. Deleteme can help protect you and your family's personal privacy or the privacy of your business from doxxing attacks. Before sensitive Info can be exploited. Our colleague Claire White recently tried DeleteMe.

[Katrina Manson]

DeleteMe is a tool that anyone online should have in their back pocket. Deleteme has saved me not only hours of removing my data, but from online, but has saved me hours of worrying about who has their hands on my data and what they're doing with it.

[Sean Illing]

Take control of your data and keep your private life private by signing up for Delete Me now at a special discount for our listeners. Get 20% off your Delete Me plan when you go to JoinDeleteMe.com Vox and use promo code VOX at checkout. The only way to get 20% off is, is to go to JoinDeleteMe.com Vox and enter code Vox at checkout. That's JoinDeleteMe.com Vox code Vox. What does the chain of decision making look like? How much do we really know about that?

[Katrina Manson]

So CENTCOM has told me that they're using a variety of AI tools. I've separately reported that that includes Maven Smart System, which is the system that Palantir helped develop for the algorithms to feed into. So it's almost like the digital display that you'd have in a headquarters or maybe on a a handheld device so that you can look at the battlefield digitally and then more than 150 different data feeds feed into it and you can crunch through that using AI. So they've got the computer vision, but they've also now got large language models, specifically claude, which is the anthropic model that is cleared to work on classified cloud. Then the US fights its wardrobe wars on classified networks. So I was told in 20. Last year, in summer I went to visit NGA, which is the combat support agency that supports the Defense Department but is also a member of the intelligence community, National Geospatial Intelligence Agency. So last summer they told me that with the help of AI, Maven Smart system can now get through a thousand targets a day.

[Sean Illing]

A thousand.

[Katrina Manson]

In the first 24 hours of the US operations in Iran, they went through a thousand targets. And with the help of LLMs, really using that to speed up the processes, the kind of admin processes involved in building a targeting package, getting permission for it still from a human, still from a commander, still with legal review. But sped up, they told me one person official told me they could now get to 5,000 targets in a day if they wanted to.

[Sean Illing]

That's a lot.

[Katrina Manson]

Yeah.

[Sean Illing]

So take something like drones. Obviously drones are such a big part of modern warfare. We're using them. Everybody seems to be using Them are humans still piloting our drones, or are these mostly autonomously controlled now, even if there still is somewhere on the back end a human in the loop, green lighting strikes.

[Katrina Manson]

Ukraine has a lot of drones and Russia has a lot of drones, but the US is not producing that many. The US is desperately trying to now take those lessons on board and produce and compete them against each other. They are almost entirely not autonomous. So autonomy is the hope, and under the Biden administration, especially for something like Hellscape, which is the Indo Pacific Command's idea of how they could defend Taiwan from an invasion by China if China decided to do that. And the admiral there, Admiral Paparo, talks about using autonomous weapons to buy him a month. So just make it impossible for China to take Taiwan and then send in the larger U.S. platforms. So under the Biden administration, I think in 22 or 3, they launched something called Replicator, which is to bring in cheap, their word, attritable, basically, you don't need to use it again, drones, and those are meant to be autonomous. And so they've been trying to develop the software. They've been competing with different companies to do that. And through the course of my reporting, I discovered that the idea was to take some of those algorithms that Maven had produced, train them on data from the Indo Pacific, really at that boat level. So boat drone cameras, aerial drone cameras, infrared, anything that might be looking at a Chinese vessel, capture those pictures, train the algorithms, sit them on the drone. Now, instead of having it on a digital platform at headquarters level, and have that AI on the drone automatically detect the target and then be able to have the drone go and take the target out. It was very tough going, those experiments. Even before the Trump administration got in, they were making progress. They also wanted to do something very ambitious, which was to link up drones in the sky, drones on the water and drones under the water into one big autonomous swarming mesh. It boggles the mind. The enabled part of it wasn't working, so they had the best data stores, I'm told. So the algorithms were potentially the best, but they couldn't integrate the algorithms onto the platform. And so much of AI isn't the specific piece of tech itself. It's, can you make all these platforms talk to each other? Can you make an operator believe in this platform? Can you workflow it? Can you start operating as one continuous ecosystem? And the answer is not without a huge amount of practice and trial and error and maybe just no, but it's

[Sean Illing]

just a matter of time. Autonomy might be a hope, but it's also inevitable. It's just a question of the tech getting there and it's moving in one direction. That's where this is going. Maybe not tomorrow, maybe not next week, but look at the progress in the last 12, 18 months alone. Right? I mean, that's where this is going.

[Katrina Manson]

I'm always wary of the word inevitable. As a history student, I was taught nothing is ever a good reporter.

[Sean Illing]

And I am not a reporter in any sense of the the term. So fair enough.

[Katrina Manson]

My opinion in support of your point. Almost in support of your point. The Trump administration came in, tore up Replicator a little bit, changed the name, but just a repackaging of the name. Dog, the Defense Autonomous Warfare Group. So autonomy is in the name and is warfare. So all of those concerns where the Pentagon was too nervous to say we want to put AI autonomy and death together because everyone was outraged about it back in 2018, the language now is so much more permissible. The Pentagon is simply saying it. The fight now with Anthropic is over not just autonomy, but fully autonomous weapon systems. And they're trying again. So they have this new project that I've reported recently on. It was launched in January. It's a hundred million dollar prize challenge. Same ideas of Maven, to a certain extent of competing the companies against each other. And SpaceX and Xai are one of the contenders. Palantir I've reported as a contender. OpenAI is named as the second on two other contenders, a couple of others I think I've reported. And they're all trying to make voice controlled autonomous drone swarming tech. So you could have an operator on a beach, let's say, saying move left and the drones would move left and you have to hope they could identify the target.

[Sean Illing]

That's wild. So there is a quote in the book that I really wanted to mention and it's from Jane Pinellas, I hope I'm saying her name correctly. She was in charge of testing Maven in those early days and she said, and now I'm quoting, if the US military wanted to use AI enabled systems, it had to become more accepting of risk. Based on the people you talk to, what is the level of acceptable risk she has in mind there?

[Katrina Manson]

I don't have a number for it, but I think it's about this. I think it means they know that AI is a black box technology that can go wrong and that it needs vetting. But at a certain extent, if you are going to put it in a system where you can't see behind the hood, you're going to be relying on something that has inbuilt risk. We know about hallucinations bias. She spoke extensively about algorithmic drift, this tendency for an algorithm to get worse over time. So she wanted of course to hold standards high, but she wanted to understand how AI will fail. The risk element is often put to me this way. If you use AI in an urban environment, there's a huge chance that you could be getting civilians. If you're using AI in a war at sea, in a China scenario, that box of operations you're going to have already there won't be civilians walking around because it's at sea. The commercial boats will long ago have thought I'm not going to go in that area or it's banned. And so all you really have are targets at sea. And the risk then for the US becomes are they going to shoot their own targets by accident and are they definitely shooting at Chinese military vessels who are legal targets under the law of war. But it's the idea that if you go wrong with AI at sea, you're just getting water and so they might not be as accurate, even though the claim for AI is often accuracy. But if it does go wrong, the risk of harming civilians is much lower.

[Sean Illing]

Are we watching that in real time? So on the first day of this conflict in Iran, American weapons bombed a school in Iran that killed lots of people, lots of children. And that was on the first day of the conflict. And based on the reporting of the Washington Post at least and maybe others by now, these AI systems maven, powered by Claude, was involved in identifying hundreds of targets before that conflict started. And presumably many of those targets were the ones that we hit on that first day. Do you know anything about that? Do we know if that was in fact an AI identified target that a human in the loop failed to realize that it was based on, I believe like decade old intel.

[Katrina Manson]

Bunch of caveats. First which is that the US says it's investigating and they haven't said they did it themselves.

[Sean Illing]

Fair enough.

[Katrina Manson]

The reports that are out in outlets, not, not my own, have suggested the US did it. This is, there's no suggestion yet that AI is involved, no confirmed suggestion. This is what I would say. The the US builds its targeting lists based on a stored data. If something is a valid target or not, it's kept in a list. What the AI can do is identify a specific object, a specific threat or something moving. Often if AI is pulling on an existing targeting list, if that school turns out to be on a military intelligence database, when it should have been on the restricted target list, no AI can fix that. So a key question is, was that school on a targeting list by mistake? Was that target list kept updated when the school peeled away from being, you know, was an IRGC facility and then suddenly it had a bunch of kids in it? Did they update the targets? Could they have been using AI to check against open source information? If the school was listed on Google Maps, what on earth is the point of AI if you're not checking that? As the US military becomes better at checking open source information and that lesson was really learnt in the US support to Ukraine. They were drawing on social media feeds, they were pulling Twitter posts so that maven smart system could analyze it for is there a yellow flag tied to this bench? Does that mean this town supports Ukraine? Does this mean this town actually supports, you know, or has a Russian presence? Has something just exploded over there? If you can pull from social media and use that to inform your understanding of the battlefield, can you pull from Google Maps? Now my understanding is any system, even if it's open source, needs to be an authorized system on US kind of networks. So where is the gap, if there is one, on being able to pull open source information and cross check? It should be extremely easy for AI to cross check. If there's a girl school, it should happen before there's a blink. But the question is we just don't know yet. And they may choose to put out a public report, journalists may have to sue for it. You know, that information will come out. But we know from previous errors that the Beijing embassy attack is a really good1 From 1999, I think the US hit. The Chinese embassy in Europe and it was 2 or 300 yards off from the target they were meant to hit and they didn't have it labeled right now with AI, all of that should be much easier. There is an argument to be made that sort your systems out. But if someone doesn't care sufficiently about protecting civilian lives or someone isn't forcing AI into the bits of the system that will protect people as opposed to speed up the death cycle or the kill chain, all of that becomes a really big problem. And if AI has been involved in any way in this hit, of course it's, well any which way, it's not just a tragedy, it's a very powerful mistake.

[Sean Illing]

What do you make of the very public fight between anthropic and the government? Anthropic, from what I understand, set a couple of red lines. No mass domestic surveillance and no fully autonomous Weapons without human oversight. Those were their red lines. And apparently they could not come to an agreement with the Defense Department. Just what do you make of that and the consequences?

[Katrina Manson]

I think by the time you have a frontier AI company that is the first to put its model on classified cloud, you have a company that's leaning in in a way that is not reminiscent of Google Back in 2017, 2018, Anthropic was on the very systems where there are lethal operations and clearly comfortable with that. If you read Derrio's two big essays Dario is the CEO of, he has these two big essays that he wrote making his case for why his company should be involved in national security. You're grappling with that thing that everyone in AI who's worried about existentialism or whatever it is about whether AI comes to kill us all or not takes over. He's grappled with that too. And he has found peace with the idea that you can do national security work and still be, quote, unquote, the good guy. He talks about a real fear of robot swarms. And his position, which I think emerged in greater clarity only partway through this fight, is not even that he's against fully autonomous weapon systems, is that he's against fully autonomous weapon systems now. And it raises questions about what was actually under discussion. Was there a system that he was being asked to put AI onto that he didn't want to, or was he just worried that they weren't doing the testing and evaluation? Right, because Anthropic did submit, I reported to this hundred million dollar price challenge to create voice control drone swarming tech. So that's leaning really far forward for a company that's concerned about autonomy. They were prepared to take part in the creation of lethal autonomy, or parts of it. There's clearly a political dimension because the president himself called them left wing nutjobs. There is clearly, I think you have to take the Pentagon at its word. They're genuinely worried that a company could dictate policy to them, or they're certainly genuinely annoyed at that prospect. And the castigation of the company as a supply chain risk, then taking it to court, then having Microsoft file the amicus brief shows that once again, the ability for the Pentagon to get to the tools that it thinks it needs is somehow at risk. Even when it has a company that was leaning really far into it. And whether they can get UP XAI and OpenAI onto classified cloud and intermitment smart system in a way that works as well as Claude in those six months of Transition time while the US is using Claude in Iran. A really unexpected turn of events, I think, to have it collapse so spectacularly, that relationship, just at the point that the US decides to test it the most it's ever been tested. It really is genuinely surprising.

[Sean Illing]

It's not my original analogy, but I did hear someone say that allowing a handful of private companies to control AI is kind of like leaving Amazon, Google, OpenEye, whoever in charge of the Manhattan Project, and then also allowing them to control and profit from the bombs. Of course, like, in order for that to land, you have to accept the premise that AI is as revolutionary and transformative attack and as powerful as nuclear weapons. But if you do accept that premise, and I'm certainly open to, is startling.

[Katrina Manson]

So what is your moral position on it, given your kind of own national security background on.

[Sean Illing]

Well, I served in the military. I wouldn't say I have a national security background beyond just having been one soldier. But what is my position on what exactly?

[Katrina Manson]

On the morality of AI in these national security uses when you're talking about the moral position of the companies?

[Sean Illing]

I'm extremely uncomfortable with it. Extremely uncomfortable with it. I understand the utility, I understand all the potential applications and I can see the case for all the lives it might save and all the good it might do. But the tail risks really alarm me. And my personal view is that war and killing people should be very hard and very costly and anything that makes it easier and faster and cheaper and less expensive in terms of human life, when you can just pull, pull a lever or push a button, I think that just makes killing people easier. And I just. It makes me very uncomfortable. And then that's just even beyond all that. I'm not entirely sure that AI is a technology that we are going to be able to, to control. A lot of these conversations presume that we'll be able to control what these systems do and don't do. And I'm not sure that's the case, which scares me even more. So, you know, I, I don't know how clear a position that is, but I'm just. That's really all I can say right now, that I'm very uncomfortable with it, Very uncomfortable with it. And I don't, I don't really trust anybody to make the, with that much power to make those sorts of decisions. I don't know. What do you think about that?

[Katrina Manson]

I think, you know, the director of Project Maven, who is no longer Jack Shanahan, has spoken up during this crisis of the anthropic Pentagon fault line not in favor of AI, even though he was the director of Project Maven to say no LLM should be anywhere near an autonomous weapon at this stage. And you know, the other red line that Anthropic raised was mass domestic surveillance. Now, I don't quite know what they think the Pentagon had in mind because the Pentagon's position is, you know, we have rules around that and we follow them. But the volume of data points available on any given individual, not with traditional intelligence, just with commercially available information from your phone or your route that your vehicle takes or your shopping habits or, you know, all of these things. And that discussion where you had people like Elon Musk signing on to letters saying we shouldn't have technologists involved in AI because of the risk of we shouldn't be creating new weapons of war. And it's his company that signed on to make these voice controlled drone, autonomous drone swarming tech. The changing comfort levels about what technologists are prepared to accept is a sort of massive tribute to the Pentagon's ability to change people's minds. Starting a business can seem like a daunting task unless you have a partner like Shopify. They have the tools you need to start and grow your business. From designing a website to marketing, to selling and beyond. Shopify can help with everything you need. There's a reason millions of companies like Mattel, Heinz and Allbirds continue to trust and use them. With Shopify on your side, turn your big business idea into sign up for your $1 per month [e-mail] SpecialOffer Fox

[Sean Illing]

News is now streaming live on Fox One. When it matters most, turn to the voices you trust. We go beyond the headlines, bringing you the stories you won't hear anywhere else. Live coverage, sharp analysis, real perspective at home or on the go. Stay connected when it counts. Stream Fox News on Fox 1 download today.

[Katrina Manson]

Your next chapter in healthcare starts at Carrington College's School of Nursing in Portland. Join us for our open house on Tuesday, January 13th from 4 to 7pm you'll tour our campus, see live demos, meet instructors and learn about our associate degree in nursing program that prepares you to become a registered nurse. Take the first step toward your nursing career. Save your spot now at Carrington. Edu Events. For information on program outcomes, visit carrington. Edu Sci.

[Sean Illing]

Foreign. I just happen to think that we are not even close to really stepping back and wrestling with how profoundly all of this tech is going to change our society and our institutions. Did you get the sense that there were like serious discussions going on about about how these tools might migrate from the battlefield to American cities, how they might end up in the hands of police departments across the country, using it for. For surveillance and God knows what else is that.

[Katrina Manson]

Yeah, obviously that idea is exactly what has animated so many of the protesters of Project Maven and campaigners against the development of these AI tools. And I do think it's interesting that the twin things that Anthropic has raised is not just fully autonomous weapons against presumably an enemy, but also domestic surveillance. And it is because the overlay of data and the knitting up of systems presents such a potentially powerful tool. And that will come down to policy choice and law, because the technology is now possible. It's still hard, but the data points are out there. You just need to suck them up.

[Sean Illing]

Do you think policymakers in D.C. are taking this seriously? Are they paying enough attention to even care?

[Katrina Manson]

I was put to be. I mean, Congress has spent a long time looking at AI, but there's no regulation. And one of the things that the Trump administration has really focused on is setting AI free. So the way the Europeans are regulating on data, specifically, nevermind, also AI, but on data, the US is taking a different approach. And that's to do with the champions would say the innovative US spirit, that entrepreneurial ability to go fast and make things and maybe break things as well. So I think in this debate between Anthropic and over the fault line between Anthropic and the Pentagon, several of the expert voices have said, where's, where's Congress in this? And it stopped short of regulation at the moment.

[Sean Illing]

I don't know, Katrine. I mean, I think at some point war ceases to be a human activity. It will still impose enormous human cost, but the actual war fighting will just be a technological affair for the most part. And that's a very different world, you know, and you can only change the character of war so much before you change the nature of it entirely. And I think that's where we, where we are. I appreciate you writing this. It's so important. It's so important and it's so well reported. I feel like I understand the world, this world, better than I did before I opened it. So thank you for writing it and thank you for, for coming on the show.

[Katrina Manson]

Thank you. Thanks for the discussion.

[Sean Illing]

Once again, the. The book is called Project Maven. If people want to check out your. Your other. Your reporting for Bloomberg or any of your other work, where can they go?

[Katrina Manson]

Bloomberg, yeah, just my name and Bloomberg will do it. But Bloomberg's plenty.

[Sean Illing]

All right. I hope you enjoyed this episode. I know I did. It's a very important piece of reporting and I hope you learned and got as much out out of it as I did. But as always, we want to know what you think. So drop us a line at the gray [e-mail] or you can leave us a message on our new voicemail line at 1-800-214-5749. Please also rate Review subscribe to the podcast it helps us grow our show and that's great. This episode was produced by Thor Neuer and Beth Morrissey, who also runs the show, engineered by Shannon Mahoney. Fact Checked by Melissa Hirsch and Alex Overington wrote our theme music. Our executive producer is Miranda Kennedy. The Gray Area comes out on Mondays and Fridays. Find it wherever you listen to podcasts. If you watch podcasts while you listen, you can do that too. Go to YouTube.com Vox for video versions of the Gray Area. The show is part of Vox. Support Vox's journalism by joining our membership program today. Go to Vox.commembers to sign up and if you decide to sign up because of this show, let us know. Support for the Gray Area comes from Mint Mobile. Generally speaking, most of us like our money. Unfortunately, big wireless companies also like your money, and it seems like they're always coming up with new and innovative ways to get it. Mint Mobile says they exist to make sure you're not overpaying on your wireless bill. They offer a premium wireless plan for just 15 bucks a month. All of their plans come with high speed 5G data and unlimited limited talk and text. Plus you can keep your same phone and the existing number. It only takes a few minutes to activate your ESIM and you can start saving immediately. You can ditch your overpriced wireless plan and get three months of premium wireless service from Mint mobile for just 15 bucks a month. If you like your money, Mint Mobile is for you. Shop [e-mail] gray area that's mintmobile.com gray area upfront payment of $45 for 3 month 5GB plan required equivalent to $15 a month new customer offer for first 3 months only, then full price plan options available, taxes and fees extra. See Mint Mobile for details. Hi, this is Alex Goldmark from NPR's Planet Money. We're really excited for you to listen to the Planet Money Audiobook. It's a smart, fun guide to how economics affects every facet of your life life written and read by the hosts of the podcast. Come explore the hidden world of economics. The Planet Money Audiobook is out now on Spotify or wherever audiobooks are sold.