Loading summary
Transcript36 lines
- [00:00]
A
Welcome to the Tech Pill, a podcast that looks at how technology is reshaping our lives every day and exploring the different ways that governments and companies use tech to increase their power. My name is Gus Hossain and I'm the Executive Director at Privacy International.
- [00:14]
B
And I'm Caitlin and I'm PI's campaigns coordinator. Hi. This week we're joined by Professor Peter Asara, chair and co founder of International Committee for Robot Arms Control and chair of the Stop Killer Robots campaign, which works to ensure human control and the use of force and calls for new international law on autonomy and weapon systems. And by Frank, Sleeper Arms trade project leader, pax, an organization working on building inclusive peace, protecting civilians from the violence of war, and ending armed conflict to discuss military use of AI. You may have seen the recent controversy in the news about Anthropic's battle with the U.S. department of Defense, now called the Department of War. And that's why we've brought Peter and Frank here to talk about it. So should we start with Peter, do you mind explaining what on earth is going on with the Dow and Anthropic?
- [01:07]
C
Technically still the Department of Defense, but if they want to identify as the Department of War, that's fine. So it's been a sort of a strange saga, and it seems to be a mix of debates about the future of technology, but also kind of a battle of egos and a bit of a. A media publicity slash intimidation sort of maneuver from the Pentagon. And as far as I've been able to piece together from journalistic reports, it appears that for some time, Anthropic has been the only large language model chatbot used by the Department of Defense on their classified data systems. And that's been for a year and a half or two years, I think. And that is utilized as a feature within the Palantir Maven system, which is a kind of all source data gathering, analytics type of system, depending on uses for all sorts of intelligence planning and operational planning. And it was reported somewhere in that they had used AI in the Maduro raid in Venezuela. And it seems that an Anthropic employee asked a Palantir employee how it was used. And this sort of set off a series of events. And it got to the Under Secretary of Defense, Emil Michael, who's Under Secretary for Research and Engineering and is basically in charge of all of the Pentagon contracts for technology. And he felt that this questioning about how it was used was an attempt by Anthropic to somehow control how it was being used or tell the Department of Defense, how to use it, and took offense at that at some level. And this opened some discussions. And part of the contract and part of Anthropic's business practice is that they have these two ethical principles, that their systems will not be used for autonomous weapons, which is an issue we care very much about, and also that it will not be used for domestic mass surveillance of US citizens, which is in some ways a kind of arbitrary distinction, but under US law is a significant one. And it's also important to note that the founders of Anthropic were formerly the founders of OpenAI, or one of them was a founder of OpenAI. The other one was their chief policy officer at OpenAI, which was founded one to be an open sort of software model for AI, but also based on ethical principles to defend humanity from a possible super intelligence that would be unlocked through AI. And left the company to start Anthropic because the company was moving away from those ethical principles a few years ago. So these are very important principles to the founders and CEO of modi. So they were very concerned about how the Pentagon was using this. The Pentagon was then became insistent that we can use it for any lawful purpose. And we want you to sign a new agreement that you, you know, will not hold us to your ethical principles or something like that. And. And they refuse to back down from their ethical principles. And that sort of resulted in a very kind of public sort of battle between the two and a sort of battle of wills. And then it was more threats that if they wouldn't do this, not only would they lose the contract, which is a significant but not a major business realize. So the way these contracts are structured, they have a sort of a minimum and a maximum. And so this is a 2 to $200 million contract. So I think they were actually only paid $2 million, but the way contracts work, they wouldn't have to negotiate a new contract or seek additional appropriations unless it goes over 200 million. But if you look at Anthropic, their annual revenue is like 18 billion. So a $2 million contract isn't going to, you know, break the company in any sense, but it's, you know, it's significant business. So. But the Pentagon kind of raised the stakes by then saying, well, we're going to label you a supply chain risk. And this is a fairly new kind of designation that emerged in recent times mostly, I think, over the introduction of Chinese chips and other electrical components into things like drones and satellites and concerns that they could have Trojan horses sort of built into the Hardware things, things like that. And it's only ever been used against foreign country companies, and actually only against foreign companies that are significantly controlled by their governments and those are adversarial governments. So it's only been used really against Chinese and Russian companies previously. So the idea of using it against a US Company is, is quite novel. And then it's also potentially has much more larger implications, which is that no Pentagon contractor can use any software provided by Anthropic. And I think it falsely claimed because it's not even in the statute that they're trying to invoke. But I think Hegseth has said that it would result in all government agencies having to stop using Anthropic, which is not the way the thing works. That's not how the line.
- [07:14]
A
Oh, okay, I didn't know that. Okay. But that. And that's a much larger number.
- [07:18]
C
Yeah. And that would be a lot more business. And then also you have to sort of think, well in terms of your kind of strategic competition. Anthropic is received as a sort of better and more reliable large language model. And also why it was able to certify to be used on classified systems earlier. There are then now contracts with open AI, which was signed at the same time that they were sort of trying to negotiate this deal. But there was a previously signed contract with xai, which is Elon Musk's AI company, to also be operational on the classified systems. But that's going to take each of those companies six months to a year or two to achieve all of the kind of security protocols that they need to be operational there. So even though they're claiming that it's a national security risk that Anthropic's system is being used, they're also planning, depending on this plan, to continue using it for the next year regardless, or six months, I think, to a year to phase it out. So it must not be that much of a national security risk. And that's pretty much what the legal case that now Anthropic has sort of sued the government to over this for. And they've also received then a sort of amicus brief from Microsoft supporting that lawsuit for a number of different reasons, but basically saying that how can there be a security risk if you're continuing to use it and actually relying on it for a lot of the operations in Iran, which I think we'll get into later in this podcast.
- [09:07]
A
Yeah, if I could just riff for a second what I find fascinating about how this story has unfolded just from a slightly nerdy perspective is that usually when a story has a single angle to it, it's easy to follow. But this story had two angles, which was the, the, the mass surveillance angle and the autonomous weapons angle. And what's fascinating about the coverage and the discussions is that they have covered, like it's become two huge discussion points. Should the Department of Defense be conducting mass surveillance? What is lawful mass surveillance? And finally, some scrutiny going on there that opened up questions as to what data was it? And then all of a sudden, because of leaked memos, we now understand that the Department of Defense, slash Department of War is buying. Buying vast amounts of data from data brokers. And so it's not like they're conducting communication surveillance. They're just buying this data from third parties. And that was part of the disagreement. And then on the other side around autonomous weapons, the coverage has been so rich and like. I've seen the term kill chain in my daily news, reading more in the last two months than I have in the last five years. And it's extraordinary that these, both these issues have managed to stay front and center to this really fascinating case.
- [10:36]
B
You talked about, like, moral red lines and. But is anthropics kind of objection ethical, or is there objection around the capacity of what they believe that Claude and their large language model is able to do? Because I've seen different coverage saying either we don't think it's capable of doing this versus we think it's unethical that it do this. And those are obviously two different conversations. One is potentially arguably overcomeable and one is not.
- [11:09]
C
Yeah, I mean, I, I think it would probably vary depending on which specific person you're. You're talking to within the company, even. And that's one of the actual problems that I have with ethical principles as a means of governing technology is they're subject to different kinds of interpretation. Legal structures are much stronger in that respect. And while they are subject to interpretation, you also have a system of courts and law to adjudicate those sorts of disagreements. And that's not really the case with ethical principles. But I, I think they definitely believe these ethical principles and believe that they are ethical. It seems that the criteria by which they measure are somewhat different or the basis for the principle. Right. So what I've heard, and I think this is what you're getting to, is that they believe at this point in time the AI is not sufficiently capable to be, you know, responsible for making autonomous targeting decisions. I've argued that ethically they are incapable of it because of the nature of their being machines and software, however sophisticated and sort of technically capable they may be, they're not legal or moral agents. And so they are never going to be responsible legal, moral agents. Agents. Even though they can effectively make decisions, those decisions are not. Have no sort of basis and responsibility. So they think, though, potentially that these systems will be capable at some point in the future, but they're just not reliable enough right now. So I think, you know, probably some people or most of the people believe that whether or not they would be capable or what the criteria would be to evaluate whether, you know, we should let them do it because they're really good at it or something. We ultimately have been arguing for human control and meaningful human control over all of these things, and that that should be codified in law. And that keeps it also to being a kind of a human enterprise, right? Like once you pass off this kind of responsibility to machines, like machines can decide to go to war or escalate conflicts and make all kinds of decisions that, you know, what. What is the criteria for? That's a good decision. It's an effective military strike. But is that achieving your political goal? Is that achieving, you know, a de. Escalation or a resolution to a conflict at some point in the future? Probably not.
- [13:47]
D
Right.
- [13:47]
C
And these systems aren't capable of understanding any of that. They don't really have any sort of desire or goals of their own, which is actually kind of. I think the underlying fear from a lot of the people who work in this space, OpenAI and Anthropic and other AI companies is that, you know, they're building some kind of godlike super intelligence that we need to sort of actually defend ourselves against at a certain level. Right. And so giving them the sort of the buttons to all the weapons and control over all the weapon systems is also sort of a scary moral proposition for them because they're, you know, they fear that. I'm not particularly afraid of that in the near future. Maybe it's possible in the long term. And right now they're still worried that these systems aren't good enough to even reliably control a single strike, much less take over the world or something. So there's a lot of sort of hype around the fear, but there's also genuine fears that we've had for quite a long time at the campaign.
- [14:52]
B
I mean, going beyond Anthropic and this particular kind of discussion, Frank, you've been looking a lot, I think, at all of the kind of broader companies and systems that have been making their inroads into military contracts and military weapons. Like how what, what, what have you been seeing? I guess in the broader, the broader space?
- [15:22]
D
I think what, what, what you can see is that a lot of the tech companies, and I would say especially the legacy tech companies, the IBMs, Intel, Hewlett Packard, AMD, they've been working with the Pentagon for decades, but mostly on a much smaller scale for mostly rather specialist purposes. For example supercomputers doing the work of simulating nuclear tests. And so those companies have been working on, on these side of sort of contracts for a long time. And as we know with especially Project Maven and all the, the discussions and controversies that burst out in, in 2018 around that, that was sort of the start with the tech companies and then mostly the newer tech companies, the Googles, the Microsoft and the development of their ethical principles. And that sort of generated much more reluctance to get involved in military contracts. And it was a bit quiet for some time. But at around the same time of Project Maven, there was also a big contract pending for offering cloud services to the Pentagon. And that contract was first awarded, but then all sorts of, of legal battles and then it was repurposed and in, in late 2000, what was it, 2022, it was awarded to Microsoft, Oracle, Amazon and, and Google like they all got it. They, they, it was a combined contract, nine, nine billion dollars. And, and I think that was one of the most significant contracts until then for the tech companies. And it's not just offering cloud services. It was dedicated. It's called the Project Joint War Fighting Cloud Capability. So the link with the war is clear there. And I think that most of all Amazon never had any problems with the military use of their products. Oracle also not, I think for Microsoft and Google it's been different also because of internal resistance. There was much more backlash there in the whole discussions about responsible use of, of AI. But then I think with the war in Ukraine, with the invasion of, of Ukraine by Russia and increased military budgets and the, the, the emergence of, of the large language models, all of that has, has contributed to, and last but not least, the big US Budget military budget has contributed to the tech companies becoming much more open and eager to fight for these military contracts. And I think also at the same time you see partly public perceptions of these sort of things changing, like we live in a different world. It's also the sort of rhetoric that Microsoft and anthropic use like we need to use AI to strengthen democracies and to make sure that the bad countries will not become better in AI than we are. And so I think there you've seen quite a significant shift over the past, let's say, four or five years.
- [20:03]
A
That timeline you, you, you paint is spot on. And just to, to riff off your point about how these companies are now talking about defense of, of democracies. Palantir doesn't talk about the defense of democracies, they talk about the defense of the West. And it's a very, it's a very different mindset and it's all you can. And what I found fascinating about the timeline is while these contracts were negotiated originally 2018 and 2021, it was in 2022. We had the large language model moment. And the reason we paid attention to that was because we saw that the UK government equivalent of the National Security Agency being GCHQ, they had negotiated a contract with Amazon in 2021. And at the time, in our imaginations, that was just, oh, for stories, storage of vast amounts of data and a little bit of processing. But by 2022 and 2023, all of a sudden, Amazon's offerings change just a little bit and they probably offer large language model processing and cloud compute, which is a completely different offering than the contract would have been in 2021, when arguably there might have been some oversight into that decision. And so I think similarly, oh, we'll use CLAUDE in our government processing of data in the Department of Defense. Then all of a sudden it's like, oh, but we can use CLAUDE for more things now. And it's that constant shifting in what the capacities are of these systems, and yet these contracts remain the same. It's, it's, it's just fascinating.
- [21:39]
D
Yeah. And maybe also going back a bit to, to what, what you previously discussed, yes, Anthropic doesn't want their technologies to be used for internal surveillance and for lethal autonomous weapons. But the very simple fact that their large language models are being used for targeting Venezuela and for the whole war in Iran, now that apparently is less of a concern. And if you relate that to that discussion of the west, that should be able to lead in the AI revolution, I think that also nuances a bit the public image, maybe that Entropic now has gotten because of its resistance, which I think is very good against the Pentagon. But at the same time, they have not resisted, as far as I know, the use of their technology in these illegal wars.
- [22:43]
B
Which actually brings me to, and I don't know if it's shout, if it's not fair to ask this, because it literally happened Today, But I saw that someone else submitted a new amicus to the supply chain case which said essentially it was on neither particular side, but it said essentially that anthropic red lines are whatever, but there is existing international law, war crimes are an existing thing. And whether or not it's used in autonomous weapons, Anthropic can't get round and the Department of Defence can't get round the use of these AI technologies in potential war crimes and the existing criminal, kind of international criminal situation. That might be a horrific description, by the way, of what the amicus actually said.
- [23:34]
A
But no, I read it. It said pretty much that, okay, who was. Was from Tech Justice League and Access now and one other organization.
- [23:50]
B
But I suppose my question from that is, like, how are LLMs currently being used and what is the substantive difference between how they're currently being used for targeting decisions versus autonomous decision making or autonomous weaponry? Like, what is that gap that Anthropic doesn't want to be bridged?
- [24:15]
C
Yeah, I mean, again, it's difficult to know exactly where they're drawing lines. And this was even an issue when the 2018 Project Maven, they added an ethical principle about not using or not developing AI for weapons systems within Google. And we all asked, well, what about Project Maven? Is that a weapons system? And it's, you know, a system that was designed to identify basically to process UAV drone footage coming out of Iraq and Afghanistan looking for, you know, things of interest. Right. Which is essentially target profiling and development. And that's, I think, kind of how and what we're seeing the, the use of These chatbots for LLMs within, within these defense systems. And we've seen a lot of reporting from Israel in Gaza over the use of lavender and Gospel and Where's Daddy? And these different. Each system is a little bit different. It's designed for a specific kind of data set and this specific function, but essentially to identify members of Hamas, to identify their locations and patterns of movement, and then to use that information to develop targeting lists for where to find them. The Where's Daddy is to try to find out where they sleep at night and target them there. And all of that sort of being integrated into the kill chain that we're hearing so much about to speed up this process, automate this process, allow them to use the mass surveillance data that they've collected. So while I appreciate that there's still this two, you know, sides of this issue, mass surveillance and autonomous weapons, they're clearly being integrated in these systems, and it's Almost hard to draw a line around them because you're using, you know, security camera footage, satellite footage, phone call data, potentially, you know, health, business, tax records, whatever, to try to identify who people are, what are their system of integration. And so are they a member of an enemy organization? And then use that information to also, the same body of information to also identify their location and drop a bomb on them. Right. And that sort of going to a human at a certain point who says, is this an acceptable target or, you know, for human approval. And even when I spoke to the UN the first time in 2014 on this issue, I called this meaningless human control, where you just have a system that produces a target for you. The light comes on and you have a choice to press the button or not press the button. But you don't really have the opportunity to delve into the context and the situation and whether that killing is justified, whether the evidence supports that action, what the implications of that are militarily, strategically, politically. You just don't know. You're just like, well, the computer says this and I'm going to approve that. And what we heard about Lavender was, you know, all the. The only check humans were making was, is the person who's been identified male or female? Because Hamas doesn't have female soldiers and Israel does. And so if it was female, they would reject those. But any male target was accepted. They knew, also going in just on test data that they used before the conflict, like 10% error rate. So, you know, 10, 10% of the targets, and that was prepared, clean data for testing. So what the actual error rates might have been. And then you get into questions of proportionality and collateral damage. And when you're doing, again, this number of strikes, strikes, because you're, you're automating the process in order to generate more and more targets. And that means less and less ability for individual humans to evaluate each target and determine its, its validity. But it also means more and more bombing, which means even if you have small ratios of civilian casualties, by having more targets, you still have a lot more civilian casualties. And I think that's one of the real sort of worries about this, is the intensification of warfare, both in terms of speed and in terms of destruction, all with this sort of plausible deniability of, well, we have, you know, selected these targets and these are. Computer says these are lawful targets. So, you know, are. We're covered from a war crimes, you know, tribunal perspective, which I don't think they are, but it's enough for them to effectuate that, and then it's very hard to prosecute those as crimes.
- [28:56]
A
Anyway, yeah, if I could segue between the two of you on that, because I remember a podcast done with not by us, but by Lex Friedman with Mark Andreessen back in 2023. Mark Andreessen being Andreessen Horowitz. He's the creator of, of the first web browsers and now investment investor extraordinaire and AI fanboy beyond belief. And in 2023, he's, he's arguing that he would want to see drones, only automated drones, doing, doing killing. He says humans make mistakes all the time and it's so much better to have an automated weapon rather than a human. And this is the guy who for the last three years has essentially been helping to build this industry that is now the new defense tech industry. And going back to something you were saying before, Frank, about how is the entry of these new tech companies like the big tech companies, and it feels like what we saw with this moment with Anthropic and say with Microsoft back in September when it came to Israel and the Ministry of Defense, is that it was almost like a naivete like these, it's these young children thinking all these toys can only ever be used for good. And of course the data is clean and of course we'd only kill bad people. And of course they're only going to use our systems for the reasons in the contract. But I wonder if, if the older companies you were mentioning, like the IBMs and IBM famously that played a role in the Holocaust, do these older companies, are they a little bit more wise to, to all of these? And is that maybe why they don't have ethical principles, because they're not trying to sell themselves as such?
- [30:53]
D
I think it's also because they're, they're offering different technology, different work. And it's mostly the new technologies that play a role here with the Palantir and its database analysis, which of course also Oracle traditionally very much was, and I think is slowly adapting and trying to catch up, but basically having lost a lot of territory in the military domain, mainly Palantir. And so it's, I think that's actually also an important aspect that a lot of the companies now really in, in the news and, and in the business are basically companies that didn't exist 25 years ago. And I mean, if you look at anthropic, just, just four or five years old, open AI, 10 years old, then Google Alphabet maybe being one of the older ones, what is it, 25, 30 years old and Amazon as well, but having also shifted enormously from being firstly and foremost consumer companies, offering a marketplace, offering a web browser and having changed so much into that military work that for very long, I think also from a commercial, commercial perspective wasn't so interesting for them. And yeah, I think that that is a big shift. And also if you see how the interdependencies are between these companies with Microsoft, of course a lot involved in anthropic and OpenAI and Nvidia, which out of, I mean almost nowhere, I mean It's a company 30 years old, but how it has become the world's biggest company in terms of market capitalization, a company that most of us had never heard of 10 years ago and is now central again to these large language model companies that work together and make very much use of each other and, and invest in each other. And I think that's also a very new dynamic that does not apply so much to the, the legacy tech companies.
- [33:27]
C
I mean, it's also interesting that, you know, there was that movement kind of in the late teens pre Covid to, to have AI ethics principles and AI ethics teams and all of those were dissolved over the last few years. And I think most recently 2025, January, we saw, you know, Microsoft and Google get rid of their remaining AI ethics principles. And whether that was in anticipation of the new administration or just the shifts around AI and it's, you know, many, many applications, it's both concerning that. They, you know, they were concerned about it at one point and then they sort of abandoned it and the Pentagon did the same thing. So there were AI ethics principles for the military use of AI, there was AI safety principles for government use more generally. Some of that was executive order, some of that was internal policy. But through executive order, Trump abolished all of that on the first few days of his administration.
- [34:28]
B
Do you think that the AI ethics principles, principles that were there were meaningful in the sense that like were they there so that they could be seen to be there or were they there because they really were thinking about them and were concerned about the ethical principles involved?
- [34:46]
C
I think there was a fair bit of ethics washing is what we're calling it now, avoiding regulation, real regulation by adopting these sort of self governed principles. I think it was also, as we talked earlier, the, you know, appeasing dissent from within the companies, from their programmers and engineers who really didn't, you know, want to be involved in those sorts of things and trying to say no, no, it's all going to be okay. But then also to avoid any kind of government regulation of these new technologies. And I think after, you know, you sort of have the post chat GPT3 moment when the world sort of figures out, oh, this is what AI is and how it's going to work, it wasn't a huge technical leap. Right. It was just an interface leap that made it accessible to a huge swath of the population very, very rapidly. But it changed our perception of it and dramatically. And I think after that, the implications for many kinds of applications, the vast amount of money that was going to be made off of it, has just been accelerating really since then. But that's the moment they decided to sort of back away from a lot of the ethics as a sort of, well, that's going to hold it back somehow. And what we need is this unconstrained development and some sort of gesturing at least at a sort of arms race with China or with other competitors who are going to also be trying to build AI and we have to kind of capture market share and in the domain of the military, you know, capture this sort of new technology that's going to give us some kind of huge military advantage. And I think both of those sorts of fears and arms races are more about hype than, than reality. But there is certainly a lot of money being made in all of this.
- [36:45]
A
It's. And the. You, you hinted at it right there as well, Peter. But what also happened in that very short window of time, say 2003, 2023 to now, is that the tech company started laying off tech staff. It was, you know, you can almost see these ethical principles as recruitment advertising to say, hey, we're the good guys, come work for us. And then they realized they had enough people. And then with the change of the world and the talk of the rise of the west or defense of the west, they thought that they didn't need to recruit people based on ethical behavior anymore and that this was a new era.
- [37:25]
C
Yeah. And this was identified by the, the Defense Technology Agent Unit. So when Eric Schmidt was CEO of Google, he started talking to the Pentagon and they had all this trouble with Maven and things. And when he left, he became a sort of advisor and really sort of gave himself the personal mission of bridging the world of Silicon Valley and the Pentagon. And at that time there was this huge shortage of tech labor and specifically in the AI space. And there was these astronomical salaries that you could get for working in AI, which is still the case in some of these small companies. And there's a kind of proliferation of AI related Jobs like prompt engineering and things like that. But as you say, like programming jobs are drying up because now Claude can program as well as most entry level programmers. So there is that part of it. But yeah, there was a recruitment issue and that was actually what they identified, you know, in 2018-2020 era as the big impediment to developing military AI because the engineers didn't want to work on it. And that's actually, I think maybe behind the scenes that we're not hearing about right now is, you know, is Claude, and Claude code developing being used to develop autonomous weapons? Because I would think that would violate their principles, but nobody's really discussing that. And of course the military itself doesn't do development, but they invest in these companies that are doing development. And are those companies. I'm sure those companies are using various AI coding schemes. So I think that's a whole nother angle to worry about.
- [39:08]
B
Can I ask, kind of, and this might be a bit of an entry level question, but to both of you, like, if you're these very new companies that are getting involved in the military space, obviously the military space is one that is inherently extremely high risk. Whether you're a soldier, whether you're a civilian, whether you're, you know, within 100 miles of a war zone. It's an incredibly high risk kind of place to be in a situation to be in. And these companies are incredibly new. And it might be, I'm quite naive, but when it comes to the new companies versus the kind of old traditional military equipment companies, are they not like just military risks or are they not kind of. If you, if you're a soldier who's, who's, you know, in one of these war zones, would you not feel a bit not great about, you know, a company that's five years old that's suddenly being deployed in a war zone? I just, it's that aspect of it, and maybe I'm wrong that there are kind of higher levels of process and safeguarding and review and consideration for failure and all these other things in the traditional companies. But you know, court code is quite good, but it's not that good. And like most people interact with LLMs now and they know they're, you know, seemingly surface level quite good, but they're not that good. I don't know. And maybe that's partly because they're more complicated technology than, you know, a gun, but there's something there that freaks me out, I think.
- [41:00]
D
I mean, we, we still don't know to what extent Claude or any other large language model has been responsible or involved in that bombing of that girls school in, in Iran. But obviously I think anybody and, and especially soldiers should be very wary of, of the newness of these technologies and the risk that they bring in inaccuracies in all sorts of biases built in the systems. So I think, and that's also what we saw with the enforced introduction of AI in the Pentagon with these posters of, of hack set saying I want you to use AI little bit silly, I can as well. But also what you hear, a lot of the people at the Pentagon and probably as well in the field don't really trust the technology as it is now. So I think that is the risk indeed. And I think that's also what you hear a lot for example, about Palantir. There's a lot of bragging and that's bringing a lot of investments. But it's still, I think not completely clear what the benefits are that a company as Palantir is really bringing. And especially if you look at the consequences on the ground and beyond Iran, looking at Gaza and how all these West Daddy Gospel lavender have contributed to the mass destruction and the genocide in Gaza, I think that should definitely give reason to a serious pause to how we introduce these decision support systems in, in new, in, in our current warfare.
- [43:15]
A
But what we need is, is, is that moment of investigation like in, in my PhD, I remember the, the in the research phase. One of the articles that book chapters that really just opened my eye to this entire domain was the case study of the M16 being used in Vietnam War. And you know, the creation of that guy gun was a, it was a political artifact. It was forcing together old manufacturers and new bullets. And in the end it just didn't work. And the only people who knew it didn't work were the soldiers who couldn't, who were having misfires all the time or it wasn't working in the field. And it was only after it all, when Congress did its investigation did all of this finally come out. And it's just like why do we have to wait until then? But that was the 1970s and that was after the Vietnam War. Are we going to have this kind of moment at some point to look at what happened in Gaza to actually understand the role of the decision support systems and the data going into it? I somehow doubt it. And so, but that's not, I don't want to end us on a negative note because one of the reasons I was most excited that you both joined was because I know that you're both pushing for actual, tangible change. So, like Frank, you said you just were at negotiations last week, and Peter, you and I first met at the UN in New York where you're part of or Stop Killer Robots is pushing convention change. And I was wondering if you could, on the back of this conversation, if you could say what it is that you're seeing where you think there's an opportunity for real change, for if it's not going to be a reckoning, at least a change going forward.
- [45:01]
C
Yeah, I would just say in terms of the campaign, and we've been working since, you know, 2013 to get an international treaty, but it's really ultimately about a norm. And the norm is meaningful human control of military force, lethal force. And, and it's, you know, I think it's. It's deep, but it's also broad and it's also clear. And it functions a lot like other legal principles that you have to, you know, teach to soldiers a basic training.
- [45:30]
A
Right.
- [45:30]
C
Like you don't execute prisoners of war and you don't kill civilians. And like, it's just a clear rule that if you're using force that there's meaningful human control. How you actually interpret that in specific cases can get very complicated potentially. But as long as we have a globally shared norm and we put that in the law, I think that's a really good step. I think it gets a lot harder to come up with a norm that is that clear. When it comes to mass surveillance, you know, data collection can be used for lots of different things. Some of those could be good things, right? Many of them could be good things, but they can also be used for, you know, authoritarian regimes to be very impressive. They can be used to target and kill people who dissent. They can be used in a variety of ways that we can't even imagine.
- [46:20]
D
Right.
- [46:20]
C
And then you start to look at other ways in which these sorts of technologies can, you know, come up with new chemical weapons designs or, you know, lower the barriers to entry and costs for fringe extremist groups to develop weapons and weapons of mass destruction. You look at, you know, propaganda and the crisis of epistemology and, and how do we control information or understand, like, what is valid information and what agents and what computer systems have the ability to sway public opinion and transform votes and democracies and all of these sorts of threats that are posed by this one technology, which is sort of so many different things. But I think there are ways to draw lines in there, and I think a lot of it derives from, you know, private data and securing our data from these kinds of systems and understanding how these systems work is part of that. And the risks that are being posed to civilians is a big part of that. But we also, I think, more importantly, have to organize as civil society and to do it globally. Because if we get it in just one country or even the whole eu, right, that's not enough to change what's happening in Silicon Valley or what's happening in China. And right now those are, you know, the leading sort of exporters in this technology on a global level. But, you know, that could also shift. We really need a kind of global cooperation and understanding and civil society to be engaged on this.
- [47:55]
B
I think that's a really nice place to leave it. So thank you both so much for your time and for joining us. Been an absolute pleasure to talk to both of you and we really appreciate your time and unfortunately, I imagine this will keep coming up and so we will probably ask you back in no short order.
- [48:12]
C
It's been a pleasure and I'm happy to come back.
- [48:14]
B
Thanks for listening. You can sign up to be the first to learn more about our work at pvcy.org podsignup and we'll include some links to relevant articles and information in the description, wherever you're listening or on our website at the pvcy.org techpill don't forget to rate and subscribe to the podcast on whichever platform you use. Music courtesy of Sepia.