Loading summary
Transcript38 lines
- [00:01]
A
Your film is now ready to be shown. Good morning.
- [00:13]
B
I'm Justin Hendricks, editor of Tech Policy Press. We publish news, analysis and perspectives on issues at the intersection of tech and democracy. Questions around applications of artificial intelligence and warfare have been building for decades, but they have become increasingly urgent in recent months. We've seen AI systems deployed in conflicts such as in Ukraine, in Gaza, and in recent strikes on Iran. We've seen the Pentagon push to make AI central to American defense strategy. And we've seen the tensions that creates, including most recently, a very public conflict with one AI firm anthropic over where the lines should be drawn around autonomous weapons systems. For the Trump administration, supremacy in AI isn't just a military goal. It's a primary strategic aim, the foundation from which all other forms of power are seen to flow. And yet, even as those ambitions accelerate, we are living through a moment that puts questions of power and judgment in the starkest possible terms. As I record this, the world is still processing the fact that the President of the United States posted on social media that a whole civilization will die tonight, never to be brought back again. A threat against civilian targets in Iran that legal experts described as promising war crimes. When that is the context in which these tools are being built and deployed, the stakes feel almost impossible to overstate. What does AI do to the judgment of those who conduct wars? What happens when friction around the use of lethal force is removed? What do the wars of the future actually look like? And will the people that execute them be subject to democratic checks and balances? My guest today has spent years reporting on such questions which are the subject of her new book.
- [01:48]
C
I'm Katrina Manson. I'm a reporter and I'm also author of the book Project A Marine His Team and the dawn of AI Warfare, which has just come out.
- [01:58]
A
Katrina, I'm so pleased to speak to you. And I want to start with this character, this person that you put in the title, this Marine colonel who is Drew Cukor. And when did you realize he would be a central character in your book?
- [02:13]
C
He is the Marine colonel at the time who was chief of Project Maven. He led the real the doing of it. So there was someone above him who was the director of Project Maven, who's been a public figure, General Jack Shanahan. But Drew Cu? Cor was the colonel who really drove the day to day activity and as I found out, also had a lot of division underlying Project Maven even before it got going and then leading to some of the key platforms that Project Maven developed, including Maven Smart System, which is being used today in every US Combatant command and in support of US military operations against Iran. I started writing about Project Maven and where it got to in about 2023. At that time I went down to the 18th Airborne Corps and was given an unclassified demonstration of Maven Smart System. And after that magazine article came out, I really was encouraged that to tell the story of Project Maven, I had to speak to Drew Kukul and had been seeking to speak to him for some time and eventually over the course of that year, managed to to meet him and learn more about him.
- [03:27]
A
You say in your first meeting your main task was to meet his stare.
- [03:31]
C
Yes, he's been described to me in various ways. For me, he was always extremely soft spoken and searching, I think, and very keen to discuss things. I describe him like a medium altitude drone. Any question, however tough I put to him, he was always able to answer with context first, kind of circle closer to the detail. But he never shirked a question. And we spoke every Friday for a. For a year. But he did make a decision about whether to speak with me. It took a long time for him to decide to go on record and to participate in a book. I think it's an interesting decision because of course I'm a reporter, I'm not a think tank. I don't share the US military position. I look at everything in as unbiased and balanced a way as I possibly can. It was really important for me to get to the what and the how of Project Maven. It's obviously a very famous infamous for some effort that the US military undertook to try and put AI at the heart of how America makes war. But there were lots of different claims for what it was really trying to do and what it did do. How successfully on their own terms it was achieved, how controversially it was seen by campaigners who were really ranged against it. And what I felt was that the debate was happening without concrete examples. What actually was AI delivering on the battlefield? Is AI all hype? Could it be used? Could it be trained? Could it be made reliable? Was the US military sufficiently cognizant of all these worries about unpredictability, black box automation bias? And so having this chance to explore it from the military operators perspective, and certainly the people backing Project Maven for me, presented a critical window of opportunity to try and really show what had they been trying to do and had
- [05:30]
A
they done it before we dive into some of those bigger questions, those bigger issues, the way that AI got baked in to the United States military the way that Kukor and the rest were able to effectively take the project forward. I want to ask you just a few other kind of framing questions about, about him and how this all got started. I mean, you call him a leading historical figure in a war that hasn't happened yet. I, I wanted to ask you, you know, is that war happening now?
- [06:01]
C
I've been thinking about that. I don't know. I mean, the war I meant to, when I wrote it was some kind of putative World War three, seeing a conflict between US and China. But of course, military history never rolls out in the way people expect. I think it's certainly fair to say that the US is using Maven in combat at a scale that it never has done before right now. So you have Central Command publicly saying they've hit more than, I think it's now 12,000 targets since the campaign started. They hit a thousand targets within the first 24 hours. Now, that doesn't have to be because of AI, but they've also taken time out, both the commander and the spokesperson in an interview with me, to say they are using a variety of AI tools they haven't named Maven Smart System. But I have separate reporting that it is Maven Smart System, which is using not only computer vision, which is the main focus I look at in the book, but also large language models and in this case, of course, Claude, the model that has become so controversial within the Pentagon. So both of those are being used. And CENTCOM themselves have said that AI is helping to speed up processes that used to take days and hours down, sometimes to as little as seconds. So they themselves are saying it. The war I had in mind for the future really involves both sides using AI at speeds faster than what Pentagon officials told me would always come faster than humans can think. So there may be more wars to come, whether it does or doesn't come, whether it's deterred. And of course, for many of the advocates of AI, they claim that AI will help deter a war and if they have to fight that war, win it. So it's not happening at that very futuristic scale that was envisaged, but it's certainly happening at a never before seen scale.
- [07:59]
A
I want to ask you about another thing that occurred to me just in reading this sort of prehistory of Maven and some of the motivations of the individuals involved. I'm going to get the sense that one of the motivations here really is born out of failure. You know, born out of the failures of the wars in Iraq and Afghanistan and some of the conviction that the individuals pursuing these systems have. That's where it comes from on some level. Just sort of profound sense of being upset with bureaucracy, being upset with the way things are done, being upset with constraints.
- [08:34]
C
Yeah, I think certainly the book really explores the long tail of the forever wars and how let down many military operators felt. In Drew Cukor's case, he was an intelligence officer, so his aim was to always bring better intelligence to operations. And for him, he's also fighting a slightly different battle that isn't just to do with AI. It's about the distinction, the separation between the intelligence side of things and the operations side of things. And much of his fight is to go around intelligence and deliver AI straight to operations. And that's what Maven Smart System is. It's a sort of intelligence infused platform used by operators. There are other intelligence platforms that rely on AI, but it doesn't have such an integral role on the operations side. And for me, I think I could see this pattern that the US could bring enormous firepower to bear, but it couldn't always put it in the right place. That was down to often a number of mistakes. But also technology that was too sophisticated in some cases for the moment that was required. And then of course, with improvised explosive devices, which is the main way that Americans themselves were dying and being maimed in those wars. Just record keeping and bringing that record keeping to the fore. That is the thing that so frustrated and infuriated Drew Cukor. He couldn't establish that kind of information on a regular turn. He was using analog records. He was using old fashioned programs that he just didn't think were suitable for a modern war. And the US had very sophisticated weapons, but it didn't have sophisticated intelligence processes.
- [10:23]
A
You have this anecdote that I love this moment where they're trying to use a computer server, I think, in Kandahar and kind of recognize that at the time it's really only good as a heater.
- [10:34]
C
Yes. Drew Cukor and a young Marine lug the computer around. It even takes up the space of a seat on a helicopter going into Kandahar. Kukor is very, I suppose, droll about this. And for him, the tools were completely useless. Others have told me, you know, I think he said this, Microsoft is where data goes to die. And yeah, the room had all the windows blown in because of the fighting. And the heater kept him warm.
- [11:05]
A
One of the things that this individual's doing is also bringing in other people who want to shake things up and who have very aggressive tactics. One of those characters is Alex Karp, the CEO of Palantir. Talk about how Palantir comes in early on.
- [11:21]
C
Palantir actually isn't involved in Project Maven early on, at the very beginning. True. Cucor is desperate to get Google. He sees Google as the, as the global A team. He wants DeepMind. If he doesn't get DeepMind, which he doesn't because he can't even get a response from them. He told me he wanted Google Brain. If he doesn't get Google Brain, he didn't get traction from them either. He goes to Google Cloud, which is looking for customers. It's very early on. And they start to provide what Kukor hopes will be a user interface, a kind of Google Earth for war, that can map coordinates on a digital display, using AI to find coordinates and infuse that intelligence with extra information and also to produce the algorithms, the actual computer vision algorithms that will look over drone video feedback deeds and identify what is there classifying it or even sub classifying it. Now, Google leaves in spectacular fashion after the Project Maven protests, which is a kind of continuing fault line between Silicon Valley and the Pentagon, and I think really in US civil military relations writ large when that happens. So Google decides not to renew its contract. Google workers protest against any involvement in the business of war. Kukor picks up the phone to Alex Karp, or, well, to Palantir, and says to Aki Jane, who's one of the early engineers at Palantir, it's been a minute, can I come see you? And he flies to Palantir and he makes his pitch. And it really is for Palantir to become the user interface. So somewhere where all intelligence feeds can go into a digital display, the information can be crunched and then displayed in one sort of easy single pane of glass for a military operator to look at. And it's not even an easy sell for him at that stage. Palantir doesn't want just to make a fancy user interface. I discover in the, in the book they want to do the data analysis. And Kukor's used Palantir before to solve some of the problems that he saw in Afghanistan. He convinces them. And then over the series of six months, Palantir, pitted against some other companies, develop a pilot for what becomes Maven smart system. I think it's clear from my reporting and probably fair to say that Kukour always expected Palantir to win. He saw Palantir as the best. Some of the others dropped out Some people claimed he gave special favor to Palantir. It's harder to sort of analyze that after the fact. But I did certainly establish that he spent Friday nights at Palantir for hours at a time. Some of the people working with him would constantly miss their trains. Cuco's are very a hard working, pressured individual who wanted results and looked at software development, tech development for war as if he were still fighting a war. And so he was working on a timetable that many people who worked in and around him couldn't fathom, but some could, and some who sort of hated him at the time even look back and consider it one of the best times of their life. Just they were working so hard for something that ultimately they believed in. Palantir starts to develop this interface and it's not a straightforward process because Palantir is already really controversial within the department. In 2016, Palantir used the army to get a foothold over its data analytic platforms. And then even as they're developing Maven smart system, sometimes the Palantir font is being used, sometimes Palantir branding is on the documents. And so there's this effort from even Palantir supporters within the Pentagon to say, please don't make such a big deal of yourselves. This is a Pentagon project. So you do see that continuing fault line. And then I think the money that Palantir starts making really doesn't kick in for a while. I'm told at the beginning they weren't making at all much money for this, but it was a kind of a foothold, a greater foothold into the department.
- [15:18]
A
Your book does suggest that Cukor had a relationship to Palantir going well back. Well back to 2009, before even the Google involvement.
- [15:27]
C
Yes. So he hears about Palantir early on and he goes to see them. I think it is 2009. He's introduced to them through another Marine junior to him and that that Marine actually sends Palantir some of Kukor's papers. So at the same time that Kukor is thinking about how to really crunch data and extract the most from it, Palantir has a platform that's doing something similar and they are brought together and Palantir had a very interesting effort to try and spread itself throughout the Defense Department. Rather than working with the very top brass, they wanted to work with the people on the front lines and secure support that way. Often hired people from those roles as advisors or full time. And so the Palantir revolving door, and every defence contractor has a revolving door, but their revolving Door looked a little different from some of the sort of traditional big defence contractors like Lockheed Martin. They tend to be hiring from the middle ranks. Cukul never ends up working for Palantir, but there is this long controversy that he is accused at some point of, of favoring Palantir and enabling Palantir to gain more contracts within the department. He contests those very heavily. And my understanding is nothing was ever uphold. He continues to have a relationship with Palantir. He never went to work for them in his now post government job, which isn't actually focused on defense, it's focused on finance. So his perspective, I think, is that he always just really believed in the tech and that they would deliver. He also very interestingly counsels Palantir on how to improve their standing within the Defense Department. And there's this scene I relate in the book which is essentially saying, look, you're perceived as extremely arrogant. You need to tone it down. Yes, we think you're great, but just change the tone a little bit. And Aki Jane goes on what he called a listening tour and what others have framed to me as an apology tour. And slowly that relationship starts to improve. But even today, Palantir is divisive within the Defense Department.
- [17:32]
A
You tell various stories about the moments where this technology starts to get actually deployed in the field and what some of the first learnings were. I want to talk about Somalia in particular. You have a chapter on Somalia and this I think is, you know, interesting just for the choice of the place, you know, where they decided to kind of take a stand, make a go of it, see if AI could be deployed in a useful way, but also where you start to see the first glimmers of the kind of vision being realized. There's this story you tell about the algorithm spotting a person hiding in the bushes that none of the analysts had seen.
- [18:10]
C
Somalia was chosen partly because if it went wrong, it wouldn't ruffle so many feathers, and also because at the beginning, Project Maven wasn't making any headway to the extent that they wanted within the services. And so in the face of a lot of pushback, they relied on people they already knew. And someone on Project Maven had a relationship with a commander in Somalia. And they got the algorithms out to a place I report in the book called Baladogli. And those first algorithms don't work well. They are not very able to integrate with the legacy systems. This is a time before cloud. So if they were rolling Project Maven out now, it would be much easier. But they were Doing it in a pre cloud environment, having to get algorithms onto systems that need an enormous number of cybersecurity safeguards. Also just a lot of internal process checks. They were, I think, in some of their own words, cutting corners wherever they could. Kukul would say, if it breaks the law, don't do it, but if it breaks policy, he. He didn't mind so much. So they were pushing their way in. And initially the algorithms would flicker because what they were doing was assessing each frame of video footage, which, of course each second is made up of multiple stills, and the AI may or may not detect the object on each still. And so it would come on and off. It was very distracting to the operators. It was also picking up everything. And so the analysts didn't like it, and they very quickly stopped using it. They then send someone who understands the work of drone screen analysis to help, who effectively becomes free labor and starts working among them, and then encourages them to start trying to use the algorithms again and to start fine tuning the way it's used. So should it be a box that can reveal what is in front of the operator, or should it be a blob which risks obscuring what might be on their shoulder if it's a weapon or something else. Obviously a very important distinction to be able to make. And they do have this final breakthrough moment where even the person who was supposed to be selling Project Maven thought it was never going to work. And an algorithm for the first time spots someone hiding in the bushes that no human had yet spotted. And in a tense live situation, that could mean the difference between life and death for the US Military operators and others. So, yeah, that was one of the first breakthroughs where even the quite skeptical supporters of Maven began to change their mind and they began to get some traction, but it was still extremely shonky for years after that.
- [20:36]
A
Well, I assume that it's the utility of these systems that ends up overwhelming any external pressure, including, for instance, the backlash against Google over its contract, et cetera. Inside the Pentagon, folks are, I suppose, now very much all in. This is a, you know, key part of the sort of strategic plank of Pete Heth's Department of War. When do you think that that, I don't know, shift occurred in full? You know, now you've got Silicon Valley firms that are essentially bought in. There's much less resistance in Silicon Valley to working with the Department of Defense. When do you think that that really turned?
- [21:16]
C
I think there are multiple turning points. One would be the Ukraine war, which we can come back to. But that is really a moment where US development of Maven was sort of put through its paces and it still wasn't good enough. But the US was able to see and argue internally that Maven smart system was really, really helpful. The other is that a lot of people who were involved in Project Maven have returned to the Pentagon under the second Trump administration, really pushing for the adoption of AI. And then the public debate over the use of AI. Of course, whatever public debate there's been, it's never really focused on the specifics because the specifics have never been made public before. But even as recently as 2025, Google itself drops its objections to working on weapons of war. Ever since the LLM explosion. OpenAI now x obviously anthropic, trying to work with the Department of Defense and the Department of Defense really rushing to try and turn LLMs into a new part of this effort to bring AI to war. And Drew Cukor had always argued that LLMs transformer models would be the future of where AI could help, not only for computer vision. And Project Maven, I learned, was much more than computer vision. It was also analyzing, translating text from captured enemy materials. They were also trying to do edge even at the beginning, meaning putting AI on drones or maybe even missiles. All the things that campaigners worried about that wasn't made public at the time were well founded concerns that this effort would do exactly that. The public temperature has changed. The protests went away. There are now a new generation of protests about the way AI may be used for domestic purposes, more for a DHS ICE component. But the effort to prevent AI going to war, I think those protests have diminished. Just as if we look at the company's own positions on working on this, those positions have changed. And what many of the companies did was say, if you want to work on government work, work in this department, in this part of our company. If that's not for you, you have other parts of the company you can work at. So there was this sort of accommodation of what those concerns looked like, which has satisfied some and not others.
- [23:47]
A
You mentioned Ukraine. And I want to just ask about another technology that seems like, you know, in parallel to AI, seems so important to this book, which is drones, which of course have transformed the situation in Ukraine. And now we're seeing the same thing happen, of course, in the Middle east and beyond. But how do, how do you think about this relationship between AI and drones?
- [24:11]
C
I think I'll go back to something. Deputy Defence Secretary at the time, Bob Work saw he was The Deputy Defense Secretary at the time Project Maven started. And when we spoke about his aims, he explained that AI for him was always about the pursuit of autonomy. And he described autonomy in two ways. You could have autonomy at rest and autonomy on the move. Autonomy at rest was essentially what Maven Smart System has become. A system to identify targets, pair it with weapons that could ultimately become automated. It's not entirely automated. There's several stages where humans are still involved, but conceptually that could become automated. And the second was autonomy on the move. This would be machines, robots being the first across the front lines. And in the case of drones, of course, not only in the air, which we're more familiar with from Ukraine, but also on water and underwater as well, and trying to create an autonomous software system architecture so that drones in the air, on the surface of the water, and underneath it can all somehow speak to each other and identify targets and pursue them in some way. That really is the ultimate vision of some behind Project Maven. Of course, at the time, people said publicly, maven has nothing to do with targeting. This is not the way we're intending to use AI. But I think I'm able to trace quite robustly that that was the intention. And today, as we look at the arrival of drones at such scale, of course, Ukraine is not the first war that has both sides have used drones. That happened in 2020. But in the Ukrainian case, you have 4 million drones being produced by the Ukrainians used in some way in a single year. And the US simply not being able to keep up with that. It changing the conversation about what war looks like. Rapid developments. No one I'd spoken to had foreseen that drones would fly tethered to a fiber optic cable. That just an extraordinary thing in order to get around jamming the point of autonomy, particularly in a scenario where the US may choose to defend Taiwan against a potential and certainly not inevitable Chinese invasion of Taiwan is the fear that drones will be jammed. And so drones need to be able to move autonomously without human intervention and select a target and pursue it autonomously as well. That's really hard tech. That effort really did start under the Biden administration in response to watching the number of drones that were being used in Ukraine against Ukraine. And there was a program named Replicator which sought to develop cheap drones, one way attack drones that could be used to defend against China in this scenario. And I learned throughout the course of reporting the book that there were efforts to integrate the algorithms that Project Maven produced and make them specifically good at Identifying Chinese vessels. So that involved all the lessons that Project MAVEN had undertaken. Collecting a huge amount, a huge new amount of data of Chinese vessels, training the algorithms, and then seeing if they could develop good enough algorithms and then integrate them onto the drone platforms that were made by commercial vendors. That process didn't go very well, but I understand that that store of data is still the strongest store. And there were video demonstrations given to even the chief, the chairman of the Joint Chiefs of Staff, showing that maven algorithms could identify Chinese destroyers. Some of the problems they've encountered is that if there's a. If you imagine a. A ship drone, a surface drone, quite a small boat taking imagery, if a spray of ocean, ocean spray got on the lens, that could, on occasion, interrupt the ability of the algorithm to keep tracking a certain vessel, then there were big problems getting the algorithms onto the actual commercial vendors platforms. And the commercial vendors themselves didn't think they needed MAVEN algorithms. They had their own computer vision, although my understanding is not fed by such a huge amount of data. So the accuracy of any of these algorithms begins to really be something you might question. It was always put to me that with AI taking a little bit more risk would be part of the scenario, that the US military understands that, and that pursuing an algorithmic war at sea where if you miss, you hit water, versus pursuing an algorithmic war in a city where if you miss, you hit civilians, were really different scenarios, and that in a wartime scenario, the US would likely be prepared to take a higher level of risk.
- [29:04]
A
You say early in the book, that biggest moral and practical question, who or what gets to decide to take a human life? Who bears that cost? That question, you know, reverberates throughout. You've kind of just got at it here. You know, one of the things that kind of comes through from the book as well is the extent to which a lot of this is about interface design and about prompting people to make decisions, what types of choices people have in front of them. You talk about the gamification of these systems, the extent to which they are maybe leading to certain outcomes just based on purely the interface design. I don't know. How do you think about that at this stage? You know, are humans more or less still making, you know, mortal decisions, or have we crossed that line?
- [29:53]
C
I think the US Military would certainly argue, yes, humans are making those decisions. But there's a constant problem within the way the US Military leaders talk about this. They say there will always be a human in the loop. That isn't U.S. defense Department policy. U.S. defense Department policy is there should be appropriate levels of human judgment over the use of force. That's something really different. That implies something more like supervision. And when I spent time with the people trying to create Maven smart system, it was very clear that in their own decision making system, humans were present in six points on what is called a decision making loop. You could substitute that for or something towards a kill chain, but a cycle that could lead to a lethal action. With the help of Maven smart system, computers are replacing humans at three of those places where a human is involved and a human is becoming supervisory in one of those places. So even according to that standard, which is based on reporting I did in 23:24, humans play less of a role in decision making now. They are making still the final decision to hit. The commander still bears that responsibility. And we will have to see what happens in the cases where there are mistakes, how that is investigated, what level of audibility there is. It's been put to me that these systems could add to transparency and accountability. Because every piece of data is tagged, everything can be followed through. And headquarters now has a much better sense of what is happening on the front lines than ever before. Those in headquarters can watch operators moving around. You can even put a beacon on a person. So being able to have that overview could, depending on how it's used, add to accountability. Essentially, that's not so much to do with AI as data integration, cloud spreading, that proliferation of information or surveillance. The extent to which it speeds up decisions that aren't adequately vetted will require a huge amount of transparency from the military to share how long they are taking for each decision, how many different information sources they are using to corroborate and vet each target, how they are even thinking through targeting. One of the main claims that Drew Cukor made to me about how AI could help with war wasn't even just for targeting. He said it would help with the questions, with the preparation to make sure the US military is ready. I thought it was very instructive that former Defense Secretary Jim Mattis recently came out and said that targetry is no substitute for strategy. And if you look at the US war aims, what the US itself has said about what it's trying to achieve in Iran, despite the huge number of targets struck, they have not yet achieved what they themselves set out to achieve. And so where you put AI in that system and where of course you ask the big questions, which may be human or data assisted, what is the point of this operation? Will we achieve it more quickly? Thanks to AI, or will our over reliance on AI mean that we fail to look at the medium and long term ramifications? That still really is down to critical analysis. And AI can help with that potentially, if you ask it the question and if you go through the vetting and checks and balances. But if you're not putting sufficient energy and thought into that vetting checks and balances, you may not get to a better solution at all.
- [33:41]
A
So it's almost like, you know, you may see, you know, as one of your chapters puts it, tens of thousands of targets, but you don't really see the battlefield. You don't really see the sort of strategic questions that need to be answered in order to determine what's the right step forward.
- [33:57]
C
You could. It depends on how you bring together all the data. And in Drew Cukor's vision you would. The whole point was to try and pierce the fog of war. But if the pursuit of targets is considered sufficient to overcome something that the US looked at in the case of Ukraine, for example, the will to fight something the US got really wrong in its analysis of what was happening in Afghanistan before the withdrawal. Those mass scale intelligence failures combined with the US obviously has repeatedly said how proud it is that it was able to judge in advance that Russia was going to invade Ukraine and encountered great difficulty in trying to convince European allies that Russia really was intending to invade Ukraine. It just depends, but in no way. And I think even advocates of AI say it shouldn't be seen as a magic solution. And I think again and again in the book I see examples where it's only as good as the question asked of it. And even then it could fail if it's been fed faulty data or used beyond its capacity.
- [35:05]
A
I want to ask about one detail. You point out that one of the things that Maven is drawing in is social media information. You talk in particular about Ukraine and the idea that Maven would get a ping every time a member of the public would post a missile explosion to their phone. TikTok showing up with geotags. You know, the extent that social media is getting hoovered into this thing, sentiment analysis is being done. Not hard to imagine, you know, some group of people being deemed enemies or combatants merely from the sentiment of their TikTok posts. Don't know if that's happening, but from what you've collected here, makes you wonder exactly how decisions are made. But I don't know how do you think about that piece of it, this sort of like mass surveillance aspect of it, how that changes the way we do warfare.
- [35:56]
C
So two things. The way that was used in Ukraine was to try and establish whether local sentiment in Ukraine was switching to support Russia or in any way losing the will to fight against Russia. And so that was helping the US decide what, where they should best put their support to Ukraine and where Ukraine would be best positioned to move limited troops and munitions to fight hardest. So a very different scenario from the one you just sketched out. In terms of picking out targets. It was really to inform where the US might best support Ukraine or Ukraine pursue its own decisions. What I do think it shows is the ability of anything to, to become a data point and how those data points are collected, the knowledge that the public needs to have of how much. We've all been told for years now, your phone is essentially a weapon in your pocket. The extent to which ad tech is used by the military machine, the extent to which any of this will be used to harness the information of civilians, of course becomes really important. Now the US military is not conducting operations against US citizens. So the question for me becomes to what extent are there similar platforms operated by domestic agencies? And certainly I explore in the book this idea of imperial boomerang, that the technologies a country invents to pursue so called enemies abroad or perceived enemies abroad eventually comes back to be used against a country's own citizenry. We know because DHS puts out a very interesting list of the AI tools it's using that many of the ICE enforcement officers are using AI tools. There's been reporting elsewhere. It's not my reporting that Palantir products are being used, that AI is looking at information. For me the big question is what information? Which data sets are being brought together? To what extent do people understand when they sign those licenses, where they may end up? To what extent do the companies themselves bear any responsibility for where those data sets end up? Looking at the way the US military is developing, its relationship with ad tech brokers will certainly be a question of mine moving forward. And it is just extraordinary that when you have satellites, drones, phones and the Internet all combined, that you can create a very comprehensive picture of almost anywhere in the world.
- [38:41]
A
I have to ask you about Anthropic and the Pentagon, of course you've covered. It came after the book was published. I'm sure you've done probably hundreds of media interviews on that at this point, but have we learned anything from, from this event?
- [38:54]
C
I think it's a very significant dispute to have the government take this unprecedented action and to try and pass. To what extent is This a real philosophical debate. To what extent is it about the inability of technology to have boundaries and to what extent this is a political fight? Obviously the president's called the company left wing nutjobs. When you have that language, it makes it difficult to try and understand. Is this a partisan political fight or is this really about what Anthropic says it's about, which is having red lines on the use of LLMs in fully autonomous weapon systems and for domestic mass surveillance, both of which the Pentagon has pushed back very strongly on to say we don't do mass domestic surveillance and we have a policy guiding us on autonomy and the development of autonomous weapon systems, which, which is the one I described earlier. The court case is still rolling, so we'll see who wins or not. I think for me it is really interesting that Anthropic leaned in so early to classified work. The classified networks are where the US military fights its wars. And the extent to which Anthropic had full vision over the way its LLMs were being used or could be used in a future war is a big question for me. Even if you have auditability at the time. What I've learned through Project Maven is that no one has access to the full use of its tools in a classified setting. So to some extent you are asking the US military to be good custodians of very advanced tech and never going to be able to know exactly the way it was used. That was Anthropic leaning in very early on with this technology and now trying to pull back. The US is trying to develop fully autonomous weapon systems. I report in the book that some systems already exist and even this year the second Trump administration launched a prize contest for. It's a prize challenge involving the AI Frontier Labs to create voice controlled drone swarming tech. So if you imagine someone being out, let's say on a beach, saying move left and then the drones being able to move left, it's something like that. And it would involve voice translation using LLMs. Anthropic I've reported, put in for that but was not selected. So they were even comfortable that that level of the pursuit of autonomy could work for them. They had some research goals that they associated with that. So Research and Development program XAI is part of that. OpenAI is supporting two bids I've reported. Palantir is part of that. So you start to see those who are most forward leaning in data and AI really trying to develop these weapons. Anthropic pulling back may not slow down that pursuit.
- [41:59]
A
The book's penultimate chapter is called the Winchester House. Why did you choose that as the title? And what do you. What do you think it communicates? What are you trying to get across?
- [42:11]
C
Well, I spoke at length for the book with Drew Cukor's wife, Kirsten Cukor, and she had been recommended to me by people who knew her. And I did discover that throughout the time of Project Maven, of course they were talking about the significance of what they were undertaking. Drew Cukor would be coming home saying people are accusing him of trying to build Skynet. And she is a really interesting and important person for me, trying to understand what is it like for a family so close to a new, highly controversial technology that is not well understood, claims over which are this could end the world and humanity. How did people think about that in their private lives? And Kirsten Cukor discusses that with me. She is no fan of war. She has made her peace with what her husband does because she argues that if anyone is going to deliver war, it should be done ethically. She believes that her husband is ethical, but she does relate to me that she half jokes to him, one day will I become like the lady in the Winchester house? And she's talking about the Winchester rifle. And at the time when this was in mass supply, it was an effort to. It was a self reloading rifle that could speed up the pace and scale of war and in theory, deliver greater accuracy again. So for her, it was already a metaphor for the claims being made over AI. And she, the widow of the Winchester family, ends up making a home with all her millions that she keeps building and rebuilding and rebuilding. And the kind of folklore around it is that she's haunted by the spirits of those killed by the Winchester rifle. And so Kirsten Cukor muses, will I be haunted by the spirits of like, dead Russians? And she is grappling with what the legacy of AI tech will be. And I think for me, one of the constant claims made for tech is that tech will make war nicer, better, will win more easily. And if you go back even as early as 1899, there's a Polish banker named Ivan Bloch who wrote this treatise that's translated in English as Is war now Impossible? And he was looking at whether the mass manufacture of weapons, this question, this claim made for that, that war would be so gruesome and killing would be so quick that no one would ever dare go to war again. And he argued that was wrong, that war would not become shorter thanks to better weapons, it would become longer. There wouldn't be decisive victories, there would be stalemate. And he, he almost predicts the trench warfare of World War I. Not quite, but he's almost there. And I've spent a long time, many years just mulling his position and trying to understand that those who claim tech will improve war, what that really looks like and the ways in which, since technology is coming, whether people want it or not, what those checks and balances and those understandings of where technology can fail really look like. There's a very, if you'll permit me, an example, early example I turn to in the book of US forces in Afghanistan in 2001 and a strike is called in using a machine that runs out of battery and when it turns back on, the coordinates reset to the location of where the machine is instead of the coordinates of the enemy and the missile strikes the US operators themselves. And in so many of those moments, there is a big gap between human understanding and technology. And if you're going to use powerful technology, you need operators to be trained on it. You need to know where it fails. You need to understand so much more than the best case scenario. You need to really work through all those worst case scenarios. And too often those worst case scenarios are learned in the field.
- [46:19]
A
Well, if it's not clear to my listeners just from this interview, these are urgent issues and this book is must reading. It's called Project A Marine His Team in the dawn of AI Warfare by Katrina Manson. Katrina, thank you very much for speaking to me.
- [46:36]
C
Thanks. Thanks for having me.
- [46:49]
A
That's it for this episode.
- [46:50]
B
I hope you'll send your feedback. You can write to me at JustInEchPolicy Press. Thanks to Katrina Manson, thanks to my co founder Brian Jones, and thank you for listening.
- [47:07]
C
Tech policy press.