Loading summary
Transcript46 lines
- [00:01]
Carvana Advertiser
This message comes from Carvana, who makes buying and financing your next car easy. Thousands of vehicles, terms up front and 100% online. Even get it delivered to your door. Buy your car the easy way with Carvana. Delivery fee may apply.
- [00:17]
Katrina Manson
They wanted to get a jump start on AI and the aim of this was autonomy, to take humans off the battlefield and deliver overwhelming US Power.
- [00:31]
Mary Louise Kelly
The problem with war has always been the humans. We humans are inefficient. We get tired, we get killed. That's the view of a Marine Corps colonel named Drew Cukor who arrived at the conclusion that humans do better when machines help us and that AI will completely change, maybe already is changing the way America fights wars. I'm Mary Louise Kelly. This is Sources and Methods from npr. The story of that Marine Corps colonel is at the heart of a new book about the Pentagon's campaign to incorporate AI into combat, a campaign known as Project Maven. Project Maven is also the title of the book. The author is Katrina Manson, and Katrina is our guest for today's special episode. As always, we'll be back here again Thursday with our regular episode to talk through the week's biggest national security news. Katrina, welcome to Sources and Methods.
- [01:31]
Katrina Manson
Thanks so much.
- [01:32]
Mary Louise Kelly
So we are going to get to newsy stuff. I want to ask you about how the Pentagon is using AI today, like in the Iran war and so forth. But start just by introducing us to the human who, as I noted, is at the heart of your story. Tell me about Drew Cukor like the first time you met him.
- [01:52]
Katrina Manson
Oh, the first time I met him, I was already a year into trying to meet him. And by then he was a retired Marine colonel, but he'd run Project Maven for five years. And everyone who I spoke to about Project Maven said, you need to meet this person in order to tell Project Maven's story. So I was in the middle of trying to do this and I finally got to meet him after work. By then, he was working in a bank in New York trying to deliver AI for finance, having tried to deliver AI for war. And he invited me to the workplace cafe. He offered me a water. He took nothing for himself. And we sat opposite each other in a booth. And I realized that here I was in front of this former intelligence officer thinking I was interviewing him. And of course, he was the one interviewing me.
- [02:50]
Mary Louise Kelly
Trying to figure out whether he wanted to share any of his story with you.
- [02:53]
Katrina Manson
Exactly. Yeah.
- [02:55]
Mary Louise Kelly
So his background, when he was active duty, he deployed to Afghanistan not long after 9 11. He was at the forefront of trying to apply technology to the war. But as you describe it, he got very frustrated very fast in Afghanistan. Explain.
- [03:15]
Katrina Manson
He was instantly fed up. Even on the helicopter in Takandahar, the seat next to him was meant to be for a human, but instead, someone put the computer that he needed to use, a big, bulky box, if you can imagine, back in those days. And that was how he did intelligence analysis to support US Operations in Afghanistan. And he was using programs that didn't work, whether they were produced by the military, whether they were Microsoft Office, Excel, PowerPoint, and none of that data was live. He couldn't draw on the underlying data to even keep track of where, by this stage, improvised explosive devices were buried or going off to even try to find patterns that might save US Operators outside the wire.
- [04:08]
Mary Louise Kelly
Yeah, so they had all this intel, they had all these data points, but it sounds like as US Troops were cycling in and out, every six months, there'd be new arrivals and they were having to relearn all the patterns that they already had learned about the enemy.
- [04:21]
Katrina Manson
I spoke to one person at the Pentagon who worked for Drew Cukor and had also served in Afghanistan. And of that time, he recalled to me that they never knew if they would be met by guns or tea because of exactly this problem of changeover, and no one was keeping the records.
- [04:38]
Mary Louise Kelly
So, fast forward in the story. Project Maven actually gets stood up in 2017. Why? What was happening then that this got greenlit?
- [04:50]
Katrina Manson
By then, the US is deep into its forever wars, which are meant to be winding down in Afghanistan and Iraq, but they're also fighting isis. And at this time, several people at the very senior most ranks of the intelligence and defense communities are also looking towards a potential future conflict with China. And the Deputy Defense Secretary at the time, Bob Work, is really focused on what he considers the China challenge, which is essentially fighting what was then called a near peer adversary. I think many would argue China is now a peer adversary and needing to lean into, in their view, modern tech, cutting edge tech. Seeing that the commercial world in the US was now relying on AI, increasingly bringing together what was then known as big data, and finding out that the Pentagon really was behind, in their view, and they wanted to develop much more sophisticated weapons in the same way almost that the US had tried to get a jump start on the nuclear bomb, they wanted to get a jumpstart on AI. And the aim of this was autonomy, to take humans off the battlefield and deliver overwhelming U.S. power.
- [06:04]
Mary Louise Kelly
So you just used the phrase that they wanted to lean into cutting edge tech. I'M trying to cast my mind back to 2017 and where AI was, and it certainly would not count as cutting edge tech today. There must have been early disasters, early triumphs, as they're trying to figure this out. Because it occurs to me, if you're trying to figure out how do you get humans off the battlefield and test AI on the battlefield, the only way to do that is test AI on the battlefield.
- [06:32]
Katrina Manson
Well, even that was a controversial decision. But Drew Cukor really pushed for this. He called it field to learn. They tried to do it in safe ways. So they weren't immediately running algorithms into operations, but they were running it over operations at forward deployed centers and trying to get the operators on board. But as you say, they encountered a number of problems. They were using computer vision algorithms and they really were cutting edge. But these were algorithms that had been trained initially on things as prosaic, I suppose, as human as wedding cakes. So the algorithms, initially the models could recognize wedding cake, tears, bridal veils, a groom's suit. And this technology was repurposed to start recognizing things on the battlefield to help analysts know what was there so the US operators could also find out who was friendly, who was faux, and ultimately what to consider targeting. And these algorithms were not working in the early days. They would miss mistake. Trees for people, rocks for buildings. One time I report in the book that a cloud was identified as a school bus. It was flickering all the time, so the analysts often didn't like it. And even Drew Cukor himself, who was this big evangelist for AI, said that AI was just a bag of potato chips to other people, meaning that it simply wasn't good enough. But he argued that it would get better and he wanted to build the systems, the operating systems, the digital interface, and really the trust and almost muscle memory of operators to try to lean into new tech.
- [08:12]
Mary Louise Kelly
Were there consequences to some of those early. It sounds like huge errors, like mistaking a, what did you say, a cloud for a bus.
- [08:20]
Katrina Manson
I think the consequences there were fury and a lack of take up. So operators just stopped using it. And then they had to rethink. And they sent out people who were very skilled as drone analysts to try and encourage them to say, look, AI could help. One of the first breakthroughs they had was AI did detect someone hiding quicker than a human did. On another occasion in Afghanistan, the AI detected a farmer walking across a field who the US was about to target. And a human simply hadn't seen them. They had been able to cool off the strike in time, but it had taken the human something like 40 seconds to notice there was a farmer there. The AI had spotted that farmer very quickly and sometimes was able to Marines in the fray of battle quickly enough to count out those Marines, say they were safe, and then call in a missile against the enemy targets. So they did start seeing results with some algorithms.
- [09:20]
Mary Louise Kelly
Am I right in imagining that one of the biggest hurdles that Drew Cukor and his team must have faced would have been the Pentagon bureaucracy itself? How eager were they to embrace all this new tech?
- [09:33]
Katrina Manson
That is certainly the way it's been put to me. Drew Cukor and many of his team he recruited from Marine reservists. They were people he knew whose careers he had followed. And also, as he put it to me, they were cheap. He could call them up from reserve duty. And they very much acted as if they were an insurgency inside the Pentagon. Cukor talks about even the shops in the Pentagon selling statins, because the people who work there are of an aging population. He brought in a very aggressive and young crowd who really acted as if they were on the front lines, as if they were Marines in battle. And many of them had seen battle and were still furious with the experiences that they'd had in the front lines where things had gone wrong, where they hadn't been sufficiently supported by tech. And so he said, why would I be any different in the Pentagon than I am in Kandahar? To me? And so this kind of aggression that he brought got them in hot water. He and his team often fell out. They also fell out with Pentagon bureaucrats, and they did struggle to encourage, such as the Air Force to adopt AI and even Indo Pacific Command, which is exactly the command that they had their sights on, given this was meant to be about deterring China. But they made their way, really, through friendship, connections. One of the people on the team, Colin Carroll, had a friend he'd served with in Somalia. Through those connections, he said, let us try out this tech. Others began to form those very human relationships in order to start experimenting with the robotics.
- [11:06]
Mary Louise Kelly
Huh. You're making me wonder if it was a coincidence that Project Maven was led by a Marine.
- [11:15]
Katrina Manson
I did hear that even Eric Schmidt, who was the chairman of Google. And really, Google's a very interesting case because Google became extremely split and fraught over Project Maven. But he was always a cheerleader. Once he was past that role, he continued to advise the Pentagon, really, on how to lean forward into tech. He was a great champion of theirs. He had seen how behind, in his view, the Pentagon was, and being quite abrasive in his comments. But when he heard that Project Maven was going to be staffed by Marines, I'm told he said they're going to get it done. And the pissed off Marines, as he referred to them, were a very small team, underpowered. And some of the stories I heard are almost comedic because they really were not informed about what AI was, but they set about it nonetheless and following that idea of commander's intent and just get it done.
- [12:10]
Mary Louise Kelly
So we've touched on some of the bumbles and stumbles and growing pains of the technology. I want to ask about one of it seems like the big ethical question, the biggest of multiple ethics questions at the center of this, which is we're talking about using machines to make life and death decisions whether to target and kill humans. How did the Project Maven team approach that?
- [12:38]
Katrina Manson
They just wanted to move forward.
- [12:40]
Mary Louise Kelly
But was that controversial within their ranks?
- [12:42]
Katrina Manson
No, they just wanted to push. And in fact, people told me that they didn't even spend their time considering ethics. Other parts of the Pentagon did. They were all move, move, move. Where they did realize eventually that they needed to develop some level of analysis was in testing, evaluation and development. In the initial stages they were just comparing algorithms on screen to see which was better at identify by eye. Eventually they developed data. The statistical approaches that outsiders said weren't very good. They revised them. But it was very clear to me that they had very little truck with that ethical debate at the time because they were just trying to get the very basics of the technology to work. And there was no consideration at that stage that machines would be making decisions. And if you look at even the Pentagon policy today, they are not at the point of allowing machines to make decisions, although they're trying to get there. I do think there was a huge ethical dimension to this when Google workers protested and said we don't want to be part of this technology because one day it could be part of a machine led decision to kill. But also we simply don't want to be involved in the business of war. And I did come across some former mavenites, as they called themselves, who were very, I guess the word is gung ho or cavalier. Some even in their job interview said they wanted to use AI to reduce the non American population. Another said that they thought that with AI they'd be able to kill people all the time. These were of course light hearted comments, but I think that it gives you insight into the kind of attitude they brought with them to their work.
- [14:25]
Mary Louise Kelly
Well, I mean people listening may well be thinking about the showdown, the war right now between the Pentagon and Anthropic, which is all about this question. Anthropic, the AI giant, is suing the Pentagon. The CEO says their ethical line he's not going to cross. Among them, fully autonomous AI controlled weapons, domestic mass surveillance.
- [14:47]
Katrina Manson
What's so interesting about this case is Anthropic has been really leaning in to national security work. Its LLM Claude is on Maven Smart system. It is being used right now in support of our Iran operations. I've been able to report, and they are on classified networks and we're on those classified networks before any of their competitors. So Darius Amadei, the chief executive of Anthropic, has leaned into this national security work and brought with him the support of the company. This red line that he is drawing, that the Pentagon obviously rejects, is about fully autonomous weapon systems. Now, what I discovered in the course of reporting this book is just how far the Pentagon is leaning into trying to invent these. They have a couple of systems that they're trying to put algorithms, AI onto drones so that they can support automatic target recognition, which really would be AI identifying and potentially selecting and allowing a weapon to fire at a target. And so this red line has become terribly urgent, not only because of Anthropic, but also because this really is now the cutting edge of where the Pentagon wants to take this technology.
- [16:05]
Mary Louise Kelly
Time for a break. Coming up, I'll ask Katrina Manson how she went about reporting on something as secretive, secret as Project Maven. And we'll talk about how the Pentagon is currently using AI in the war on Iran. You're listening to Sources and Methods from npr.
- [16:22]
Carvana Advertiser
This message comes from Carvana, who makes buying and financing your next car easy. Thousands of vehicles, terms upfront and 100% online, even get it delivered to your door. Buy your car the easy way. With Carvana, delivery fee may apply.
- [16:38]
Pop Culture Happy Hour Promoter
Do you love pop culture? Hate some of it, too. You're in good company. Pull up a metaphorical chair to Pop Culture Happy Hour, the podcast that breaks down the best and some of the most questionable moments in pop culture. We'll tell you what's great, what's interesting, and break it all down with debates that'll have you yelling at your speakers, but in a good way. Listen to NPR's Pop Culture Happy Hour by finding us wherever you get your podcasts.
- [17:03]
Mary Louise Kelly
Katrina Manson, a question or two about how you went about reporting this book. Because Project Maven happened, I gather, mostly in secret, behind closed doors. They were not inviting reporters into those meetings. They weren't releasing minutes of their meetings, what was the hardest part of figuring out what they were up to?
- [17:22]
Katrina Manson
They started to become more open in 2022, I think, because of the value they saw of Project Maven in support of. Of Ukraine. But after the Google walkout in 2018, it actually became impossible to foyer Project Maven to lodge a Freedom of Information act request. And so for years, Project Maven operated in a kind of blackout. And through my reporting, I found out it was very deliberate. People were asked not even to put on their LinkedIn that they had worked on Project Maven. They weren't supposed to talk about it. And I did discover that Project Maven went far further than just computer vision alone. It was looking at satellite imagery, it was looking at text. It was looking at edge, which means putting AI on weapon systems, potentially at the edge on the battlefield. So, yeah, it was tough. And I think in those circumstances, you do your best as a reporter to find people who are relying on their recollections. You ask them to find documents, calendars, sketches. I was even able to find cartoons of the time that people were working on Project Maven.
- [18:34]
Mary Louise Kelly
What do you mean cartoon? Like. Like doodles in their notebooks?
- [18:38]
Katrina Manson
Yeah, doodles. I mean, I think the workplace culture was also part of the fascinating tale. And so people at the time were sort of trolling Drew Cukor and his very hard work ethic, and he also drove other people to work hard. And so they were talking about that. But there were even clues there that they were looking for better data. And I would discover that some of the data had been poisoned or sabotaged even by their own workers, who were fed up of labeling data because it's very monotonous work. So I was always looking, looking for clues to really thread together different elements. And, of course, corroborative documents, photographs, all of those things really, really help. And then in time, I was able to secure the support of 18th Airborne Corps to go down and visit, and they showed me an unclassified demonstration of Maven smart system that really brought the system to life. But, of course, operations themselves were much harder to get information about, and eventually I managed.
- [19:41]
Mary Louise Kelly
So, Maven smart System. That is where this has landed. I want to bring us up to how the Defense Department is using AI today. You've talked about. It was used to share targeting information with Ukraine near the start of the war, 2022, that it was used in 2024 in strikes against Syria and Iraq, the Houthis in Yemen. What do we know about the current war in Iran?
- [20:05]
Katrina Manson
I Think it's very interesting that CENTCOM has been prepared to take time out during these operations to make public that they are using AI tools. The commander of Central Command on some of his video broadcasts has said he's using AI and that that is helping bring down processing time from days and hours, sometimes to seconds. The spokesperson of Central Command has also told me they're using a variety of AI tools to generate points of interest. Now, points of interest is sort of military speak for everything before a decision to target. So the line they're drawing there is that AI is not deciding what to shoot at, but they are using AI to develop targets, including location, elevation, description. And most recently a senior defense official even explained that the system maven smart system can develop courses of action and work through something called target work, workbench, all of which is about developing not only a target, but also the weapon you would pair with it and what order you might shoot it in.
- [21:17]
Mary Louise Kelly
We have been reminded during this war in Iran that targeting, whether human or AI generated is only as good as the maps that we're working from. Do we know if AI was involved in the attack on the girls school in Iran where so many were killed?
- [21:34]
Katrina Manson
They haven't said. And I think a key question will be what was that underlying data? And this really gets to the heart of AI as well, because even if AI is not to blame, if you are speeding up your ability to call on errant data, you are speeding up your rate of potential mistake making. The scale of this war has been described as double that of the shock and awe campaign in Iraq from 2003. In the first 24 hours alone, Centcom went through 1,000 targets. They're now beyond 7,000 targets. And as one commander put it to me, eventually it stops being about targets and whether you have sufficient munitions to put to them. So what has happened in this girls school, like former mistakes that the US military has made, for example, the attack on the Beijing embassy in Belgrade really matters when it comes down to record keeping. And I think many people I spoke to within the military who had concerns about reliance on AI, about over relying on AI, about atrocities that could continue in the age of AI, really do make the point that data and accountability, revealing what has gone wrong and examining where safeguards may not be present, that should be is very key. Others have said to me that if you put AI in the right place, you can help that. Should AI be drawing on Google Maps? If Google Maps can tell you, hey, there's a school there. So if they are going to Take this direction. The role and placement of AI becomes key, as well as the data that you're feeding it.
- [23:14]
Mary Louise Kelly
This brings me to ask about a line in your book that caught my eye. You write, AI remains a narrow, faulty tool with considerable limits to its usefulness and reliability that the US military is still discovering. Limits like what?
- [23:32]
Katrina Manson
There's widespread knowledge within the Pentagon that AI can make mistakes. We all know that AI can hallucinate. It can be prone to bias. It also has this thing called algorithmic drift. Over time, algorithms tend to become less right. And in addition, you have this idea of escalation, particularly as the US starts to thread in LLMs to its systems, relying on LLMs to help with reasoning and speed up processes. When you ask a chatbot or even an agent to carry out a task. Research has shown, and some of the advisors to the Pentagon have highlighted this research to me, that chatbots can be escalatory. They can tend to agree with you.
- [24:16]
Mary Louise Kelly
You're reminding me of that old the, what is it, the 1980s Matthew Broderick movie War Games.
- [24:21]
Katrina Manson
Right, right, exactly. And one official I interview did say it's not. It's, you know, we're not building the Whopper. But actually, if you are asking questions, shall I take this move? Is this a sensible move? Are we in line with the laws of war? Will this result in a good hit? You have to be very careful about the way in which you ask that question. And I do report in the book that they have thought about this or some quarters of the Pentagon have, and they're trying to add guardrails into the prompt. So when you prompt an LLM underneath the hood, it red teams that LLM, it tries to say, are you going to escalate? Check that you don't. And so the claim was made to me that you can actually rein in that capacity for error rather well. I think that needs to be continually tested. And the extent to which this administration is prepared to accelerate AI and also consider the policy implications and just the technical realities of AI is still something that's rolling out.
- [25:25]
Mary Louise Kelly
Katrina Manson is a Bloomberg reporter who covers tech and national security. Her book is Project Maven. A Marine Colonel, His Team and the dawn of AI Warfare. Katrina Manson, thank you.
- [25:39]
Katrina Manson
Thanks.
- [25:42]
Mary Louise Kelly
And before we go, a plug for our Thursday episode when we'll have some special news to share about an upcoming event where we hope to answer your questions about the national security world and events in Iran. Right now, as always, if you have questions about our work or anything you hear on the show, you can email us. We're at sourcesandmethods@NPR.org. i'm Mary Louise Kelly. Thanks for listening to sources and methods from npr.