← 80,000 Hours Podcast
80,000 Hours Podcast
How AI could transform the nature of war | Paul Scharre, author of 'Army of None'
02:45:17
Loading summary
Transcript125 lines
- [00:00]
A
How do you end a war that's happening at superhuman speeds? If our competitors go to terminators and their decisions are bad, but they're faster, how would we respond? There's this incentive towards faster reaction times and decision making. They have to go faster to keep up. I think we have a really interesting example in financial markets, stock trading, where humans can't possibly intervene in milliseconds. And then we've seen examples like flash crashes. Could we have something like a flash war where interactions are so fast that they escalate in ways that humans really struggle to control? The ugly reality is likely to be that politically people will have to suffer and die for wars to end.
- [00:47]
B
Today I'm speaking with Paul Shari. Paul is a former US Army Ranger who served in Iraq and Afghanistan, the current Vice President and Director of Studies at the center for a New American Security and the award winning author of two books, army of Autonomous Weapons and the Future of War and four Power in the Age of Artificial Intelligence. He also worked at the Pentagon where he led the team that wrote the US Military's first policy on autonomous weapons. Thanks for coming on the podcast, Paul.
- [01:15]
A
Thanks for having me. Very excited for the discussion.
- [01:18]
B
You expect AI and automation to transform the nature of war. Can you talk concretely about what that will look like?
- [01:26]
A
So I think we're already starting to see artificial intelligence play an important role on modern battlefields. I think over time we're going to see AI take on more of the cognitive dimensions of warfare and get to a place where, and this might take several decades, the speed and tempo of war could start to really push at the boundaries, maybe even exceed them, of what humans can do. So you can envision a world in the future where you sort of get this tipping point, what some Chinese scholars have called a battlefield singularity, an idea that the speed and tempo of war outpaces human control and war at a large scale shifts to really a domain of machines and machines making decisions.
- [02:14]
B
Okay, yeah. The idea of a battlefield singularity is extremely, extremely interesting to me, but I want to step back briefly and just kind of try to understand how exactly AI will be integrated into kind of weapons systems. And I guess, yeah, then how it'll affect how wars are fought and won. So if I understand correctly, autonomous weapons systems are kind of complete weapons platforms that kind of, once activated, can select and engage targets without further human intervention. Can you give a couple examples of the kind of most advanced autonomous weapon systems that are being developed today?
- [02:55]
A
Sure. So conceptually, an autonomous weapon is really one where the weapon itself is making its own decisions on the battlefield about whom to kill. I think that I'll make an analogy first to cars. Conceptually the idea of a self driving car is pretty straightforward. And you can imagine cars that don't even have a steering wheel, certainly people of designed them, where the car's totally driving itself. Now in practice, as we see the technology evolving, it's these sort of incremental movements. You have sort of incremental advances in autonomy and automation in cars like intelligent cruise control, automatic lane keeping, automatic braking, automatic self parking. You know, Tesla that has more kind of incremental self driving features. But you could see this path towards a completely self driving car. We're seeing a very similar thing inside militaries, where militaries are incrementally automating different tasks that are used in finding targets, in processing those targets, in presenting that information to decision makers, in missiles and drones that would be able to carry out attacks.
- [04:08]
B
Can you give a few examples of, as it advances, maybe quite significantly, what are longer term visions for how AI and autonomy could kind of make really big differences to how these weapons systems work? And I guess I'm sure there are loads of different places these things will be integrated. So maybe I'm most curious about the places that could most drastically change how wars are fought and. Wonderful.
- [04:36]
A
Yeah. So I think one major paradigm shift that could occur, I think is probably eventually likely to occur over the next several decades is towards swarming warfare. Where you could have, you can imagine large numbers of autonomous drones on the air, at sea, undersea, on land, that are networked, that are working together cooperatively and autonomously, adapting their behavior on the battlefield to respond to events. So right now we're seeing massive numbers of drones deployed in Ukraine, certainly tens of thousands of drones on the front lines. But those drones are not only remotely controlled, for the most part, they're not really working cooperatively in any way. So even if humans had the ability autonomously for the drone to go out and find its own Target, having 10,000 drones that are independently finding targets is very different than 10,000 drones that are working cooperatively together. And you could have much more dramatic effects in the battlefield by having swarms that are able to simultaneously attack from multiple directions, have self healing communications networks, self healing minefields, the ability to react to what humans are doing, to what the enemy is doing in real time and, and at a faster, not only speed but also scale of coordination than is possible with humans. And this is, I think, the real dramatic change here is not Actually, in the physical technology. I mean, drones are interesting. They could do neat things, but it's in this sort of cognitive dimension, and in particular here of what the military would call command and control. So militaries today are organized in this very hierarchical fashion. You have teams and squads and platoons, companies and battalions, and you have these. Them organized predominantly because of the limitations of human cognition. If you put a human commander in charge of 10,000 soldiers, and just like they were directly issuing orders to each 10,000, that would be totally impractical. There's no way to do that. That's not how militaries are organized. That's not how corporations are organized. If you look at sports, it's really interesting that a lot of team sports have somewhere between maybe five to a dozen or so players on the field. Now, imagine a game of soccer where you had 100 players on each side and 50 balls, right? You'd have to have a completely different way of organizing that. But robots or swarms could do that differently. They could perfectly coordinate their behavior and ensure that they're optimally using those resources to, you know, hit the soccer ball, score to the enemy targets, whatever it is. And so I think that that's a potentially really dramatic shift in how militaries fight in the future. There is this vision of a possible future that as militaries integrate artificial intelligence and autonomy more fully across the force, that we might reach some tipping point where the pace of combat action is just too fast for humans to respond, and humans have to be completely out of the loop. And I think what's scary about that possible vision is that humans are then no longer in control of violence and warfare. And that raises moral questions, but it also raises just really fundamental questions of how do you control escalation in wartime? How do you end a war that's happening at superhuman speeds? And we don't have good answers for that. And I think maintaining human control over warfare is absolutely essential to making sure that we can navigate this transition towards more powerful AI in a safe way.
- [08:29]
B
Okay, so just to make sure we get kind of a concrete picture of what this kind of battlefield, singularity, or sometimes called hyperwar, would look like. Can you kind of describe. Yeah, what does it look and feel like? What kinds of weapons? How have they been automated? What do conflict engagements look like? You know, are there. Are there any humans in the loop at any level?
- [08:54]
A
So let's start with where we are today, and I want to kind of paint a picture for how that might grow over time. So since at least the 1980s countries have had automated air and missile defense systems that can shoot down incoming threats when the speed of these incoming missiles or rockets or artillery aircraft are just too fast for humans to respond. So for example, a US Navy warship has an automated mode on the air and missile defense system that can be activated where there might be missiles coming in and there's just so little time for humans to respond. And you might have multiple threats coming from different directions that then the machine, once it's activated by people, can automatically sense all these threats and shoot them down. Now we've had these systems around for decades. They haven't really been widely used in conflicts in these automated modes. And there have been a couple examples of accidents. There was a fratricide, a couple of fratricide incidents in 2003 with the Patriot air and missile defense system. But that's something that we have some experience with, that there is this sort of very narrow domain today of machine control over warfare, where machines really, humans just can't be in the loop in this area. I think what I would envision is that this domain of machine warfare grows over time and that several decades from now we end up in a world where something like that exists at a much larger scale along the entire front, where there are swarms of thousands of drones on both sides and they're dynamic and responding to enemy behavior. And there are missiles being launched, launched and striking targets. And there are AI systems identifying new targets which are moving and mobile. And humans can't possibly be in the loop to respond to that enough. It's too slow. And humans are maybe observing this action. Maybe you could think the way a coach might on the sidelines and humans could have some degree of direction of, okay, I'm going to change the higher level guidance for, for these systems, or I might try to add new parameters to sort of the operating systems. But humans can't really in real time intervene. And I think we have a really interesting example of this exact kind of behavior in financial markets, stock trading, where there's this whole new domain of high frequency trading where humans can't possibly intervene in milliseconds that these algorithms are responding. And then we've seen examples like flash crashes that come with that. And so I think the scary sort of analogy there would be, well, could we have something like a flash war where the interactions are so fast that they escalate in ways that humans really struggle to control? And I think that's a really scary proposition. How do you find ways to stop that? In financial markets, they Put in place circuit breakers that can take triing offline. If we see movements that are too volatile, there's no good way to do that in warfare. There's no referee to call timeout. And I think that's. How do you maintain human control over warfare that's happening at superhuman speeds.
- [12:23]
B
Can you talk about the incentives that lead to this kind of inevitable speed up and taking of humans out of the loop? Given that, my sense is that currently the Defense Department and others who kind of are going to be in charge of these decisions do not want to take humans out of the loop. So why does that thing seem like a likely thing that's going to happen and kind of how does that drive things faster?
- [12:46]
A
So I think this. Yeah, that's a great question. I think this push pull is very common in these types of major revolutions in military affairs where you have old institutions and ways of fighting that are not necessarily super enthusiastic about the new way of fighting. The cavalry, for example, wasn't particularly enthusiastic about tanks. Right now, certainly within the US Military, there's a strong belief that humans should remain in the loop. That's not actually official U.S. policy, but certainly when you hear U.S. military, senior military officers talk about it, they'll talk about it that way. That they want humans in control. And I think because of just a healthy skepticism, for all the reasons that everyone interacted with AI could understand about these systems that like, well, sometimes they get it wrong and there's value in humans making these decisions. I think the ultimate arbiter is what works on the battlefield. And so that is what will drive how militaries change. Militaries tend to be often very conservative with these types of changes, in part because you never really know what's going to work until militaries fight a war.
- [14:02]
B
Right, okay. And so the idea is there is this conservatism and maybe it takes decades, but eventually the technology. I mean, it's my suspicion that the technology just is very likely to improve enough that you're really disadvantaging yourself if you don't use it. Does that sound right to you?
- [14:26]
A
I think that there's a trajectory towards greater automation and greater speed and tempo of war. I do think that militaries have choices about exactly how they implement that technology. And the sort of important thing for militaries, this is actually true of most military tactical revolutions. What matters most is not actually getting the technology first or even having the best technology in some sense. It's figuring out the best ways of using it. It's figuring out what do I do with a tank? What do I do with an airplane, for example? I think there's value to human cognition. There are lots of types of cognitive problems that at least today are very challenging for AI. Even if AI is cognitively better, there's probably value in keeping humans in control of warfare. The question is how to maintain that balance in the best possible way. And I think that that's going to be a really important question in the next several decades.
- [15:30]
B
But do you think that that value is really likely to persist long enough that at some point at least one country decides that taking the human out of the loop is strategic and then does better? And if they do better, that creates this pressure for their adversaries to take them out of the loop?
- [15:53]
A
So former Deputy Secretary of Defense Bob Work, who was really a pioneer in bringing artificial intelligence into the US Military, has this quote of if our competitors go to Terminators and their decisions are bad, but they're faster, how would we respond? Which is kind of a colorful way for a senior leader to be talking about throwing the Terminators. But I think it does highlight this really difficult problem of this potential for an arms race in speed in militaries, that there's this incentive towards faster reaction times and decision making that might pressure militaries to do the same, even if they don't want to, Even they're not comfortable with that, they have to go faster to keep up, similar to what we've seen in financial markets with high frequency trading. And that could lead to a sort of dangerous situation where you have this dangerous arms race in speed in the military. And I've heard some people argue we ought to have some limits on that. How do you put a speed limit on warfare? Seems like an appealing idea. I don't know how you do that in practice to try to put brakes on this tendency, which is, I think, a big risk as militaries are adopting AI and autonomy.
- [17:05]
B
Yeah, interesting.
- [17:06]
A
I think for autonomous weapons, yes, the answer is going to be that there are strong competitive pressures to take humans out of the loop, at least in certain kinds of scenarios. I guess what I would say when I certainly talk to folks in the US Defense Department, is that there's still a lot of value that humans bring and that these machines can make mistakes. And so we want to, let's not sort of throw out human cognition. And even if we think our systems work great in testing, we don't know what the enemy is going to do. We don't know how our systems might interact with the enemy. And so particularly in. There'd be some battlefield applications where it might make sense to say, okay, using autonomous weapons in this area is fine. I think so. But certainly in controlling escalation, I think human control is really, really important. If you imagine something like the Cuban Missile crisis, and if we add a bunch of autonomy, I'm not sure that makes it safer. To me, that makes the whole situation much more brittle and more likely to escalate in dangerous ways. Because you can give humans this. You can give humans really ambiguous guidance. You can tell humans things like, look, you're allowed to use force to defend yourself, but don't start a war. And a human can understand that. And they might be like, well, I don't really know what that means in practice, but we're basically saying, look, I trust you that you're pretty smart and figure it out as best you can based on the situation that you're in. And humans can handle that. And I know that AI systems, even as a KUM starter, will necessarily understand the consequences of those actions in the same way. Maybe they will. But this is a place where I am a little bit conservative.
- [18:51]
B
Okay, I would love to talk about AI in kind of the nuclear command and control space.
- [18:59]
A
Things aren't scary enough.
- [19:00]
B
Exactly, exactly. Let's just escalate it a bit further. So nuclear command, control and communications is this bucket that includes kind of sensors and analysts, command nodes, communications links, and procedures used to detect nuclear attacks and then decide how to respond and then order and execute that response. So it's this huge kind of bucket that all relates to detecting and responding to potential nuclear threats. What is the argument for building AI into these systems?
- [19:35]
A
So I think the argument for integrating AI or automation into nuclear operations is that it's absolutely crucial for these systems to operate at a very, very high degree of performance and reliability. And so within the nuclear space, there's this concept of the always never dilemma. And what this means is that you always want nuclear weapons to be used if there is an authorized order from the President or whoever the command authority is, for these systems to have a breakdown there undermines nuclear deterrence. And so if your adversary knows that there's holes in your system, and like, if the President says, use nuclear weapons, maybe, maybe it doesn't happen, or maybe they don't launch, that might create incentives for an adversary to engage in risky behavior. But on the other hand, you never want nuclear weapons to be used when there's not an authorized signal for launch, either by accident or by some rogue act or Doing that now that's really hard from an engine, either technical or sort of a social engineering standpoint, you could imagine that there's lots of safeguards that you can put in place against an accident or an accidental launch or unauthorized launch occurring that then make it just harder for an authorized launch to occur. And I think the idea is that you could use AI to sharpen this distinction if you do it right, to make things both more reliable when needed and safer. And on the detection side, that you could use AI to have greater visibility on what adversaries are doing and buy more time for decision makers. Like a fundamental problem in the nuclear space is we've had these incidents both in the United States and in the Soviet Union and Russia after the fall of the Soviet Union, where there'll be an alarm that goes off that looks like a missile launch. And right now decision makers have minutes to make a decision. And the problem is that even though both the US and Russia have invested in their nuclear arsenals in ways that ought to allow them to survive a first strike and still retaliate, that you could really be severely disadvantaged by a major first strike. And so there are incentives to what's called launch on warning. So you have this warning that missiles are inbound. You need to launch your missiles before they get taken out in their silos. And there's not much time. So if you could buy automation, if you use automation to buy more time for decision makers to speed up the process and then make that information more reliable, that would be really valuable. So I think there is like a legitimate case for using AI. And to be fair, there are lots of places where automation is already used today in actually nuclear operations. But boy, there are ways you can get wrong too. And that's what's scary.
- [22:32]
B
Yeah, so I find these arguments in favor of more automation and more AI in kind of nuclear command, control and communications very alluring. But I have the sense that, that you are keen to proceed with caution. Can you talk through one or two of the failure modes that worry you most?
- [22:52]
A
Maybe a good place to start is with Stanislav Petrov, because it's this really clear example of how things could go wrong in a really bad way. So Stanislav Petrov is this lieutenant colonel in the Soviet military. He's a watch officer on duty on a night in September in 1983, and he's sitting in this bunker outside Moscow. And the system says that there's a missile launch from the United States. The United States has launched, according to the system, a nuclear tipped intercontinental Ballistic missile against the Soviet Union. And then the system says there's another launch and another and another. Five missiles total inbound. And there's this big. Petrov describes that there's this big sort of backlit red screen and it's saying missile launch. And he has really not much time to decide what am I going to do here now he knows a couple things that are sort of outside the details of the system itself. One, he knows that the Soviet Union had just deployed a new satellite based early warning system. And he knows a lot of technology just doesn't work. And so he doesn't necessarily trust the system from the get go. He also thinks that launching five missiles just doesn't make sense. If the US were to launch a massive strike, you'd launch all the missiles. Why poke the bear? It doesn't strategically make any sense when he thinks about what the US might do. So he calls the early warning radars, which as the missiles are allegedly coming, should be able to see the missiles coming over the horizon. They say we don't see anything. So Petrov says later that he thought it was 50 50, whether this was legit, but he had a funny feeling his gut didn't make sense. So he tells his superiors, system's malfunctioning. The scary thing about this is what would an automated system have done? Whatever it was programmed to do, and it certainly wouldn't have known necessarily the ability to kind of step outside that situation and say, well, you know, this is a new system, should I trust it? Or this type of attack just doesn't make any sense. And then come to that conclusion that he did. And it certainly wouldn't have known the stakes. Right. Which would Petrov understood, right? That boy, like if we get this wrong, a lot of people are going to die. And so I think that's kind of the worrying, this really stark illustrative example of the stakes if we get this wrong.
- [25:36]
B
Yeah, I'm completely with you on the stakes. And I've found it incredibly, incredibly unsettling to learn about all of the near misses that we've had. And yeah, I guess so. I guess I'm with you on that. And I don't want to be naive and fall prey to kind of magical thinking about how great AI could be in theory. But it does feel to me like an automated system with quite hard coded programming might respond to that situation in a different way to Petrov, in a way that is catastrophic. But when I chat to LLMs, it does seem like they're able to one be programmed. Not just programmed, but trained to be flexibly conservative. Also to have a bunch of context, including what the stakes are and how strikes are likely to play out, such that they would have some sense of the ability to reason about why one might see five warheads coming or not. Is it naive to think that AI systems might actually be able to have enough context that they'd have made similarly good judgments to Petrov?
- [27:09]
A
I mean, they might, maybe, but, like, how reliably, I think, would be the question. Right. And so a couple challenges, I think, in using AI in this. And I think there are legitimate ways to use AI in nuclear operations. For example, using image classifiers to track mobile missiles, missile launchers, for example. Great, great potential use case.
- [27:30]
B
Okay, nice.
- [27:31]
A
But, like, the problem with some system making some determination is in particular, like, what is the training data that you use for a surprise attack? We don't actually know what that looks like. And we know that AI systems are going to perform often perform quite badly when pushed outside the bounds of their training data. And so that if you present it with a novel situation, maybe you get something good, maybe you don't. And so there's this really interesting example from the 80s where the Soviets created this intelligence system called Vryon that was designed to predict the likelihood of a surprise US Attack. And what it was designed to do was collect data on all of these things that the Soviets thought might be indicators that the US Is preparing for a surprise attack on the Soviet Union. So things like the US Stockpiling blood in blood banks, the locations of senior US Political and military leaders. And so you could see indicators of maybe, like, okay, it looks like maybe they're getting ready for something. That sounds actually like a really interesting use of automation. What happened in practice was KGB agents were basically incentivized to generate reports and feed data into the system. And the data that was coming in was just bogus because people were judged based on going out and getting information and bringing it in. And so the whole thing was trained on just bad data or relied on bad data, rather. And so I think that's an example of some of the flaws of these systems. You could imagine some AI intelligence system that's looking at all of these different indicators, troop activity, the locations of senior political leaders, and where we see their nuclear submarines and bombers and mobile missile launchers being moved, and comes to some judgment, okay, this is my probability. One of the problems is how do we verify that that's accurate? Right. Look, we can verify that a lot of other AI things are good and are performing well because we can test them in their actual operating environment, we can look at image classifiers and we can get to ground truth. What is the thing? Is it accurate? We could take self driving cars and drive them on in the operating environment. In this case we wouldn't have any great way to measure the baseline. Is it good at this at all? And then of course a lot of AI systems are super opaque. So let's say your AI system says, okay, I think there's a 70% probability that there's an attack. Why? And maybe it can tell you something, but that doesn't necessarily mean that the story it's telling you is accurately reflective of the underlying cognitive processes inside that neural network. Of course, I think it seems like a really dicey way to use AI. I do think that over time militaries are going to start to integrate intelligence communities and AI in this fashion. I think they're likely to be very conservative though, which is probably not a bad thing in this case.
- [30:47]
B
Yeah, okay. So I think I am moved by the fact that you won't be able to test these in real world situations. Maybe at best you'll have war games where people are making moves that they would think they would in the real world. And you can train the system on those outcomes, but you cannot train them on real world examples of nuclear exchanges and escalations. And then I also just buy that. Yeah, I guess when I'm like in particular the Petrov situation, I can imagine a model reasoning well about that. But there are so many, many, many ways that all of these variables could come together. And maybe it is just pretty dicey to expect them to perform well in all of those. I guess. Do you have a take on. It feels like it's important that we're comparing these systems to humans and humans are also fallible. But is your kind of overall take. Yes, humans are fallible, but they just will be better at this for a very long time, maybe indefinitely.
- [32:06]
A
I do think there's certainly value in human judgment. I think maybe one thing I would say is I think a little bit depends on how militaries or intelligence services might employ AI in different kinds of high risk scenarios. In this case, for example, you're trying to figure out are these indicators of a true attack or a false attack. I think in use case that I would not think is wise would be to have this AI predictive system that sort of predicts the likelihood of an attack. I just think that for all the reasons outline, it's not a good idea. Now Here's a use case that might be valuable. You could have an AI system that's fed all this information and the question you ask it and the thing that you've trained it to do is make the case for me why this is false alarm. We have a crazy, crazy number of false alarms. It's terrifying. Somebody puts the wrong tape into the system. A lot of scary things. And that could be an interesting use case of you're in this moment and humans are not sure how to respond. And you turn to the AI system and the AI system is like, well, have you thought about these factors? And it's, oh, those are things to run down. Let's check. Is it a training tape in the system? Is it sounding rocket that was launched and it's not actually an attack? Is the system malfunctioning? In the Petrov case, the satellite system that the Soviets deployed was detecting sunlight glinting off of clouds and sort of registering that as the flash from a nuclear launch. So that would be an interesting use case. And sort of you would then bias the system intentionally in a way to try to help you identify false alarms. But I think there's value in retaining humans for some of these types of decisions. In particular, because humans understand the moral stakes. There's this more fundamental question of what are the types of things that AI is likely to be good at? What are the types of things that humans are likely to be good at? I think things where it makes sense to use AI would be situations where we either have or can create good data on what performance looks like, and we have clear metrics of better performance. Self Driving Cars is a great example of this. It's a hard, hard problem, but we can correct a ton of data and then we can run those. We can create synthetic data by running simulations to amplify that data that we get. And we can test the cars. In the real world operating environment, there's a clear metric of performance. Don't get into accidents. There are other situations where it's just not as clear or we don't have good data or what depends on the right answer might often depend on context and judgment. So to give a military example, let's say that we have a drone looking at a person and they're standing in a dark alleyway and they're holding some object in their hands. What is the thing? Well, that's the thing where AI could be really helpful. Are they holding a rifle or holding a rake in their hands? What do they do? We could build up a database of images under different lighting conditions and angles and probably get to the place where AI is better than humans. Now let's say we identify the object holding a rifle. Is that person an enemy combatant? That's like a super hard problem because it depends on context and judgment. Like, well, you know, what is enemy behavior in this area? And civilians, maybe civilians carry rifles in this area for protection against thieves and bandits. Maybe that's super normal. Maybe the person carrying a shovel, but we just saw them planting an ied, they're an enemy combatant. That's harder. And you could imagine maybe reasoning systems that are plugged in that are synthesizing all this information and coming to judgments. But I would be cautious in those. For all the flaws that you have in LLMs, those are use cases that I think I would just be very cautious about.
- [36:11]
B
Okay, setting that down for now. I'm interested in how this all affects kind of nuclear stability and deterrence. So I guess we already do have some automation, but we'll probably be building in more in different parts of the kind of nuclear command, control and communications system. Do you think there will be kind of one effect on kind of the game theory of, of deterrence or do you think it's just like they're going to be a million? And it depends on exactly how countries incorporate these. I guess another option is countries don't know how their adversaries are incorporating these and they'll be making guesses. What does all of this do to nuclear stability?
- [37:00]
A
I think there's a couple use cases that you could see that seem likely that might increased stability. One would be using edit, just a better visibility on what other countries are doing. And so it makes it harder to launch some kind of surprise attack. Or if you get a signal of a launch, you have a lot of other information that you can use to verify it that can allow you to have multiple different looks at this problem. That's one use case that would be valuable. Another one is if militaries just get better at using automation to make their operations and responses more reliable. And that has a stabilizing effect because maybe other adversaries are less willing to contemplate doing some kind of attack of like, well, you know, we really know that they've shortened the response time and they're able to get better information and you know, we're not going to be able to disarm them through a first strike. Here's a couple use cases that might be concerning and I don't really know what the net effect of this kind of would Net out to be one is this sort of fear that AI enables so much transparency coupled with precision guided weapons, that it might enable a first strike to disarm opponents. I think this is like, super unlikely because for this to work, you have to basically get all of the weapons, or almost all of them, and then have adequate missile defenses to soak up like one or two that get through. I think it's extremely hard to pull that off in practice. It's not enough to say, well, we know these are their launching sites. You need to know where are the mobile missiles in real time for all of those systems. Right, right.
- [38:44]
B
Submarines and trucks with nuclear weapons on the backs of them, driving around countries.
- [38:49]
A
Yeah, I think it's just, yeah, that's like a really high bar to achieve and to get to a confidence level that somebody would feel comfortable executing a first strike. Now, the more realistic, worrying possibility is that even though an attacker might feel like, yeah, this is not plausible, a defender might feel vulnerable as a result. And then they say, well, we got to build more weapons, we got to build more silos. We've seen China engage in this massive buildup of their nuclear weapons. And I do think part of that is likely China reacting to US technological advances, drones, satellites, other systems that make China feel like they're vulnerable and they need to increase their nuclear stockpile. And so that actually could be destabilizing either because in an immediate situation, it creates sort of this incentive to maybe launch because you feel like your weapons are super vulnerable, or on a longer timeline, it creates sort of an arms race instability dynamic where countries feel like, we got to build more of these weapons, and then others say we got to build more too. And particularly as we're moving into a. There's sort of this nasty combination right now in the nuclear space of we're moving towards one, a tripolar nuclear world where the balance really isn't just between the U.S. and Russia, it's the U.S. russia and China. China builds up their stockpile. And so, like US analysts are looking at this and like, now we need to have enough weapons to. To counter not just Russia, but Russia and China. And so the US builds, increases its arsenal, then others are going to increase theirs. You could get this kind of more complicated arms race dynamics coupled with emerging technologies that maybe create vulnerabilities or uncertainty, whether that's AI, space system, cyber systems that maybe create uncertainty among political leaders about how safe their arsenal is. And then coupled with really, I think, disturbingly increased political salience of nuclear weapons that Putin in particular has done a lot of nuclear saber rattling in the last couple of years in the context of Ukraine. But I think the US has done a decent job of responding in a measured way to, and not overreacting to. But Putin sort of putting nuclear weapons on the table in a sort of diplomatic sense of sort of waving this nuclear stick around in a way that is, is new and different. And that might change how policymakers think about the relevance of these weapons and maybe even make them feel more usable to some countries in some types of conflicts. And I think that combination is a little bit scary.
- [41:24]
B
Yep, yep. Yeah, that makes sense. I guess if you could kind of choose one internationally agreed upon kind of binding constraint on automation and AI in the context of nuclear command, control and communications tomorrow say. Yeah, yeah. What would that be? What would, what would make you feel safest?
- [41:50]
A
I would love to see nuclear powers agree to maintaining human control over decisions relating to using nuclear weapons. That seems like a really low bar, clear, like we all ought to be able to agree that regardless of what we do with autonomous weapons or the things that humans should maintain control over decisions relating to nuclear weapons. And the US that's actually official Defense Department policy that the US put out several years ago in the Nuclear Posture Review. It was agreed to between Biden and Xi, between the US and China during talks in the Biden administration. France and the United Kingdom have had similar sort of public statements. So there's sort of some really interesting important foundations there. Particularly I think the US China agreement is really significant. I'd love to see that expanded to include other nuclear powers, ideally to get a statement from the P5. So that would be the existing countries plus Russia and to maybe deepen that a little bit, to have a little more clarity. What does that mean internally? How do we implement this kind of guidance? What's sort of inbounds and what's out? And are there things that countries could do that would be credible assurances between other nations? What could we do that might credibly. Because statements are one thing. What are you actually doing that might assure others that we're actually following through on that? I'm not totally sure what that looks like, but I think that that would be a really important direction to go and there's already and sort of early foundation there to build upon.
- [43:33]
B
Yeah, yeah. I mean my first thought hearing this is verifiable commitments seem so, so important. And how do you have verifiable commitments to keeping humans in the loop in these decisions? Are there proposals for technical solutions to this or Is this just an open problem that people should think more about?
- [43:59]
A
I think it's hard to get to a place where there's clear, verifiable commitments, because I think it's impossible. But I don't have an answer here. I guess because the answer is embedded in your software and your operations in a way that's very different than counting missile silos. We can see the missile silos. We can count them. We know how many you have, or submarines. It's really hard to hide. You can hide a submarine underwater. Where it is, you hide the existence of a submarine, it has to come to port. That seems really tricky. And to allow. Countries are not going to allow other nations into their nuclear operations. There's just way too much vulnerability there. And that's a consistent problem in arms control is how do you manage this verifiability vulnerability paradox of how do I show you what I'm doing in a way that verifies that you could see what I'm doing, but doesn't make me vulnerable because I'm giving away too much information? I think it's also worth taking a step back and saying, where do we need verifiability in arms control agreements? Some arms control agreements have inspection procedures. Many of them do not, because there's just the ability to externally observe what others are doing. Again, I think in this case, that's going to be really hard. Another sort of way to look at this is you don't necessarily. It'd be nice to have some credible assurances, but you don't necessarily need that if you're going to do it anyways. So for the United States, for example, if this is Defense Department policy, because we think it's a good idea, then encouraging other countries to adopt that same policy is in U.S. interests, regardless of exactly what they're doing, because we're going to do it anyways. It's not like, oh, like another country decided to automate nuclear launch. So it seems like a good idea. If we think it's unwise, we're not going to do it anyways. And so you do have some dynamic. There are some examples of other weapons like that, like biological weapons, where the US has signed a biological weapons treaty and has forsworn them and sort of, even if other countries have covert programs, and it's probably true that some other countries might on a small scale, it's not something still the US Wouldn't want to do because the risk of blowback is simply too high.
- [46:22]
B
Yeah, okay, that is slightly reassuring, I guess, on your Kind of best guess of how things go over the next few decades. Can you kind of paint a picture of what you think wars will look like and which parts might be really fast and then where humans will kind of stay in the loop and maybe slow things down?
- [46:47]
A
Sure. So I think over the next, say five to 10 years, we'll see militaries increasingly build more drones of different types, fill them in larger numbers, they'll have incremental more autonomy. We might see some isolated examples. I think it's likely of autonomous weapons being deployed on the battlefield, but I don't think they'll be integrated into military operations in large numbers. I think we'll certainly see more mature AI technology like image classifiers used in a wide variety of contexts by militaries to understand the battlefield. I think we'll see a lot of what you might really just call business process automation. People sort of taking things that humans are now doing in Excel spreadsheets or passing along communications and information more manually gets automated, and that speeds up sort of the tempo of war in terms of people's ability to sort of compress their decision making time. So, for example, one of the things that the US army is quite keen on is if they get some intelligence, let's say satellite imagery of where an enemy target is, they want to be able to sort of shorten the time it takes to put sort of useful targeting information in the hands of artillery systems that are on the ground so they can actually carry on a strike. And right now that's measured in hours, and you might see that compressed to minutes as that gets compressed over time. But I think you'll still see humans still engaged in a lot of operations over a longer timeline, maybe 15 to 20 years. We might see some integration of swarms at small scale for sort of tactical purposes, maybe swarming robots being used to do reconnaissance over an area or strike targets that humans have approved. We might see autonomous weapons become more widely integrated into military operations, more AI being used to generate courses of action to support decision making. I think we'll probably see over that timeframe, more AI involved in intelligence processing and so that people aren't just looking at an image of an object that's been found by, you know, an AI system, for example. But actually the intelligence reports are synthesized and analyzed in ways by AI before they're given to commanders. And so commanders are still humans making decisions, but they're increasingly relying on information that's mediated by artificial intelligence, which introduces a lot of weird vulnerabilities of like, are there biases in the system and has the enemy found ways to manipulate it. But I think you'll see the value in that over time. And then maybe over a timespan of 30 or 40 years. Something approaching more like some of these discussions about battlefield singularity or hyper war, where military is a really fully integrated AI and we see at a much larger scale operations being condoned in ways that start to maybe exceed humans abilities to stay in the loop in terms of actually managing tactics on the ground. I would say the exception to everything I outline is probably what's happening in cyberspace, where I think automation will happen much, much, much faster. And we'll see cyber attackers and defenders sort of be forced into a position where they have to cede control to machines on a much faster timescale because of just both the ability of machines to operate in what is sort of their native environment within machines, they're just much more capable on the Internet than they are of the real world, at least at the moment. And just the tempo of operations in cyberspace being one that's closer to things like financial trading where you sort of have to the competitive pressures to pull humans out of the loop are going to happen just much sooner.
- [50:38]
B
Yep, yeah, okay, I want to come back to cyber war for sure before we do that. So you've got these kind of, you've got a couple of different timelines for different kind of scales of increases in the tempo of war. So it sounds like yeah, next 5ish years is maybe a slight increase in tempo, then you get a bigger one. And then at some point, maybe 40 or 50 years from now, maybe you do start seeing something like hyperwar. Really concretely, how fast are we talking about? So in a hyper war scenario, are wars fought and won in the span of hours or days or weeks or months?
- [51:26]
A
Yeah, that's a great question. There are still very real physical constraints on moving physical objects around that still will apply to robots and to AI. Again, cyberspace being an exception where you could have very, very fast operations. I mean there have been botnets for example, that have spread very, very quickly. Cyber attacks that have taken down infrastructure, for example, cyber attacks from Russia against Ukraine, they've taken down Ukrainian digital infrastructure in some cases like a matter of seconds. So you could have conflict unfold in the cyberspace in a matter of minutes. But in the physical world it takes time for missiles to fly long distances, time for aircraft to fly. AI is not going to fundamentally change those physical constraints. Now maybe in the long run people say, well, AI is going to Label better materials, blah, blah, blah, blah. But, okay, maybe incrementally still. But I think you could look at the pace of advancement of, you know, aircraft propulsion. It's not growing exponentially. Right. It's very incremental improvement. And so that I think will, will mean, you know, for things like a. A missile salvo would unfold over a period of hours. It might be that the scale of that missile salvo is much larger. Now, depending on the range, it might be that it's 20 minutes for missiles to come in and to attack. But a back and forth of a mixed exchange might take several hours. And maybe within 10 hours it's all done and sort of the dust settles, and one side has a dominant advantage in terms of at least that initial missile salvo. What would be different would be maybe the number of missiles and drones that are coming in much larger than say, the raids that we're seeing Russia launch against Ukraine right now, for example, to maybe thousands of drones and missiles all at once. And they might be much more cooperative and they're dynamic and responding in ways that force defenders to basically automate their defenses. But something like a ground invasion would still take days, and at minimum for a lightning invasion, a couple weeks to unfold. So I think that's just. There are some of these physical constraints that are very real that would still exist even in the world of AI driven warfare.
- [54:04]
B
Yep. Yeah, that makes sense. Can you explain why one of the changes there that you expect is kind of bigger simultaneous attacks?
- [54:19]
A
So there are incentives already in an initial attack to have a massive salvo, to hit as many targets as you can at once. If you look at, for example, the US Shock and Awe aerial campaign against Saddam in 2003, when the US military games out a potential war fight against China over Taiwan, one of the things that the US expects was that China would launch massive missile salvos against US Air bases in the region to crater runways, to blow up fuel depots, to target aircraft. And if you could sort of make these initial attacks severe enough, quick enough, you can really degrade the other side's ability to respond. Right. If you can cradle the runways before the aircraft got off the ground, like, great, that's a big win. So I think what's different about AI and autonomy is a whole bunch of factors that favor scale. So let's talk about the way drones are used in Ukraine, for example. So let's say you've got about 10 order of magnitude about 10,000 drones operating right now on the front lines. Well, you need 10,000 people, to fly those drones, you need a lot of operators. Now, there are advantages to having the human not in the drone. You can make the drone much smaller, you can make it cheaper, you can make it disposable. If you lose the drone, you don't lose the pilot. Pilots can gain a lot of experience over time. So pilots who maybe are not very good at first, and they might have died in their first mission in a crewed aircraft, they can wreck the first 20 drones or more before they really sort of figure it out. And like, it's fine, it's like really cheap. So there are advantages there. But you're still limited in this sort of one to one ratio between pilots and drones. Now, autonomy totally breaks up the dynamic. Now you could have one person launch a swarm of 100 drones and the person just says, go here and perform this operation. And the drones do that autonomously. That has major advantages in allowing militaries to just exercise command and control over larger numbers of forces and therefore to start fielding them in ways that are more incentivized. And so if you have the ability to now basically control an unlimited number of drones now, that sort of changes your paradigm of like, okay, well maybe we could just. Let's crank out a lot of these things. There's also other ways in which as over time. So let's talk about marginal costs for drones. I'm really dirty for a second. Okay, so as you field larger numbers, obviously you get better economies of scale, but there's still some marginal cost in producing that additional drone. Drones are pretty cheap, but you've got to make the physical hardware. Now for the software, it's different, right? The software scales differently than hardware. So it might cost a lot to develop software, but once you do, it's basically costless to replicate software. And so this is why you see like totally different economics in handheld devices and smartphones. For example, like a phone costs a good chunk of change, several hundred dollars, even though there's huge economies of scale. There's like 7 billion smartphones. But the software is totally different. Once I've got the hardware, I could download apps for free. And anybody can download an app, right, Once you have that kind of hardware in place. So similarly, you could see that as more of the cognitive abilities of the drone gets from the human, which humans don't scale. I've got to train pilots, their scarce resources to embedded in the software itself. That software scales very easily. And all of those economies favor scale. But there are still, I just want to say, very real physical constraints in production, like who's going to make all these things in the ability to transport it to the front and get it there in logistics of doing maintenance, for example. But I think the dynamics will benefit scale in pretty dramatic ways.
- [58:29]
B
Yep. Yeah, that's fascinating. So automated weapons are likely cheaper for a bunch of the reasons you've just said. It sounds like the scale goes up. So maybe that's kind of the cost of an individual attack goes up a bit in that sense. But how kind of on net does the cost of war change as we get more automated weapons? And I guess, yeah, I'm interested in both kind of financial costs, but also in human casualties. Intuitively, it seems like you'd get way fewer human casualties. But curious if that's right.
- [59:08]
A
I do think that over time AI and automation will allow militaries to sort of do more with less in terms of personnel. And certainly for some militaries like the US Military, personnel costs are very, very high. That's not true for the Russian military, for example. Right. The totally different personnel model for how they're thinking about people. So that could allow militaries to do more with less. The sort of question of overall cost of militaries maybe depends a lot on what is your mental model for what the price point is set for defense spending and to what extent that's driven by defense needs versus other factors exogenous to the Defense Department and that ecosystem, how much Congress is willing to spend, domestic factors like political issues. I do think in terms of like these changes are likely to relatively benefit less capable actors. And that is a meaningful change in thinking about cost. So for right now, like prior to drones, if you wanted air power, you wouldn't have to buy an airplane. Airplanes are traditionally very, very expensive. And certainly like a fighter jet costs maybe 50 to $100 million order of magnitude. Drones are super cheap. You can buy small quadcopters for maybe a few hundred thousand bucks. And it doesn't do the same thing, but it does give you air power. It does give you the ability to recon targets from the air, find them, to even carry out very small strikes. And so that sort of lowers the price of entry into air power. And because in particular, AI seems to proliferate very, very rapidly and software proliferates easily, I think that this sort of relatively benefits small scale actors who are advantaged by this. Now, in terms of the human cost, I guess there is this idea maybe of cable. We have all these drones and then people aren't fighting. And that's great. I'm going to start with a. I don't believe that's going to be true. The cautionary tale to me would be the mention of the Gatling gun, which the inventor saw the horrific bloodshed coming from the American Civil War and thought, okay, what if I could have a machine that could automate firing on the front lines? Which the Gatling gun did. It was sort of a forerunner of the machine gun and automated the process of firing dramatically, sped up firing rates on the battlefield. And his vision was, then there'd be fewer soldiers on the battlefield, save lives. The opposite happened. It dramatically expanded the lethality of warfare. And we saw huge casualties In World War I trench warfare, once machine guns had matured and fully implemented into warfare. And so just because I'm not sure that automation necessarily is going to pull people back in part, I just don't buy this vision of futures of robot armies fighting in these bloodless battles. I think humans will still be needed to perform some cognitive tasks and some of those are likely to be actually close on the front lines because of challenges in long range protected communications. It's going to be easier to have short range communications to control robots from relatively nearby.
- [62:27]
B
Right.
- [62:28]
A
But also I just think sort of the ugly reality is likely to be the politically, unfortunately, people will have to suffer and die for wars to end. I think that's the practical reality and that's tragic and that's kind of dark, but I think that's probably likely to be the case.
- [62:48]
B
Yeah, I think when I imagine kind of these robot wars fighting, I'm like, but then you're not losing anything of value or enough of value.
- [62:59]
A
Yeah, I think that's right. And that's unfortunately the reality of a lot of wars. Like if you look at the war between Russia and Ukraine, for example, the front lines are relatively static. They're not moving in a dramatic way. The war has devolved into this war of attrition where some of it's about the economics of fielding artillery, for example, and causing casualties. But a lot of it is simply a war of suffering of who is willing to incur more costs for longer, who wants this more. And a lot of wars unfortunately turn out that way. It would be nice if militaries could go off and fight a battle doesn't involve bloodshed, or they could just game all this out on their computers that say, oh, clearly you're going to win. And when there are dramatic changes that does happen, we see politically that when there are huge differences in power, countries generally will exceed to then what sort of you know, stronger nations want, but not always. And sometimes small countries fight fiercely and hard for their independence and that will to fight. So like the Russia's invasion of Ukraine is like a really interesting example here. So part of what happens is you can think of wars as a failure of negotiations, a peaceful negotiation that moves into sort of negotiation through violence. One of the challenges here is that on paper people can add up military hardware, but it's really hard to measure is things like will to fight and morale. And on paper, Russia should have ended that war in 30 days. And a lot of military analysts, myself included, thought that's what was going to happen. And what we saw was that Ukraine is superior in all of these intangible dimensions of war, of morale, will to fight, leadership, unit cohesion, corruption in the ranks. That has huge effects on the battlefield. And it's always been the case that these human factors matter a lot. Napoleon talked about them counting three to one, I believe he said against material factors in war. But these things are these immaterial factors. Human factors are hard to measure. Now the interesting question is how does AI change that? Kenneth Payne has written to some phenomenal, phenomenal work on this. He wrote a book I Warbot and some other work sort of thinking through how AI changes the psychology of war. So what does will to fight mean when you have drone swarms fighting that like they never get tired, they never right. And so one of the arguments that he makes is an illustrative example here. And I'm not saying that I buy this, but there's an interesting argument is oftentimes this will to fight has benefited defenders. They're fighting for their homeland aggressors. Maybe just they don't want to be there as much. Very clearly in Ukraine, in terms of the balance among Russian and Ukrainian troops, when you have AI fighting, if you take that off the table, that would seem like maybe that relatively then takes away some of the advantages of defenders. It's more equal. And that maybe in relative terms then benefits attackers more. I don't know if that's valid, but AI raises some just really interesting questions about changes in the psychology war.
- [66:38]
B
I want to go back to something you said a few minutes ago. It sounds like this might cause us to enter a world where smaller poorer states or even non state actors can actually threaten much larger militaries by leveraging these cheap automated weapon systems that are offense dominant. How much is that going to change kind of balance of power dynamics? Are there going to be more wars fought because it's cheaper to start Them, including by small groups that don't have as many resources.
- [67:14]
A
Well, that's a good question. I think the economics of it, I feel, are valid, that it relatively benefits smaller groups more. I'll give another example here. Ukraine basically neutralized Russia's Black Sea fleet, sinking and damaging several warships worth hundreds of millions of dollars by spending a few tens of millions of dollars in small drone boats laden with explosives that could come in and sink a warship. And so I think we're going to see those tactics copied more. Whether that leads to more wars is tricky. Right, because it depends a lot on what do you think the mechanics are that drive wars. I think one mechanic can be. If there's a disagreement between actors about the relative balance of power. And here's a place where I think you could argue AI and NET does one or the other. I could see arguments both ways. The argument that AI might make conflicts more likely would be that one it's just a disruptive change. And so there's more uncertainty about how this is used and who's an advantage here. And some countries might think we have the AI we can win now. In particular, countries might sort of feel overconfident about AI because humans often sort of seem to overestimate what AI can do in terms of its abilities. We see this really dramatically, unfortunately, with early implementation of like autopilot in Teslas where there were a number of fatal accidents. People sort of over trusting the automation. So that could be one kind of risk. Another way that AI might, you could imagine, make wars more likely is that as more military capability is embedded in software and algorithms, it's much harder to measure. Like you can measure airplanes, you can measure ships, you can measure tanks. And we say, well, look, they have three times as many aircraft as we do and twice as many tanks, so maybe we shouldn't fight a war with them. But when it's algorithms, it's really hard to like, okay, how do we know if our swarming algorithm is better than their swarming algorithm? That's actually really tricky. Other than like, well fight them and find out, that's going to be really hard. So that might lead to more uncertainty and disagreements. One way that AI might be more stabilizing is if it creates more transparency and greater ability for countries to just see what others are doing and may make it harder to carry out surprise attacks. I think we've actually got really solid evidence of this. We saw this in the run up to Russia's invasion of Ukraine, where the US because of just greater intelligence, satellite imagery was able to have really great visibility in what Russia was doing and then share in a really, I think, really impressive diplomatic move, share that intelligence, declassify it, share it with European allies to get Europeans on board that this was something that was going to occur. And so you could see AI just makes it really harder to mass forces for surprise attack. And that takes away some of the incentives. We see a little bit of this even tactically on the front lines in Ukraine, where despite the fact that there's all these drones and the drones are kind of hard to defend against, the front lines are really static for a lot of reasons. But drones seem to be making the front lines more static. And one of the things that we're hearing from people on the front lines is there's no way to mass forces for an assault. They can see you because they have drones overhead. They can see what you're doing. And so they know that you're going to make an attack in this area and then they can defend against it. And so it contributes to the stasis. And so that all of which is like, I don't know, you could see arguments on either side. And a lot of it depends upon how the technology is implemented by countries.
- [71:09]
B
You've kind of described this incremental push toward more and more autonomy at some point, potentially leading to very, very automated war. Maybe something like this hyper war image. And you think that it could take decades to get there. Why do you think it'll take that long? Are the key barriers more like technology, or are the key barriers more like deployment because of people wanting humans in the loop?
- [71:42]
A
I think there's a whole bunch of barriers. I think the biggest one is really about adoption by militaries. And there's barriers to adoption at all of these stages. There's barriers at the conceptual stage. So sometimes, just like for militaries to conceive of a world of all of these swarms of robots and drones that are fully automated, that's a big. But you have some independent thinkers that are writing about these things. John Allen and Amir Hussein sort of coined this term hyper war and have written about it. You have others. But it's going to take time for that to be absorbed into the bloodstream of the actual decision makers. Inside military, there are challenges in procurement and acquisition. There's a lot of just lock in into the system. It's hard for new entrants to sort of fight their way through bureaucratic red tape. And then even when you deploy the technology, one of the things that stands out really clearly, in these past examples of military technical revolutions is figuring out, how do you use this and integrating it effectively into operations is really hard. And so it's sort of like the difference between, let's say a CEO is like, we're going to go all in on AI in this company and buys enterprise subscriptions for ChatGPT or one of the MindClaw or something. Gemini, one of the models for everybody in the company. Okay, here you go. Have at it. That doesn't actually transform your business process. What transforms your business process? People figuring out, what am I doing with this thing? And experimenting and everything else. Putting the technology in the hands of people is an important step, but it's not the only thing. And that requires a lot of experimentation, requires willingness to change your way of fighting, and sometimes in ways that are deeply uncomfortable for militaries. So militaries put a lot of emphasis on identity. You know, service members identify with their service. You know, their soldiers or their sailors or airmen or Marines. They identify with their occupation oftentimes. So, okay, a person's a pilot or they're a sapper. That's a term for engineer in the army, or they're in the infantry, for example, or they're in armor, they're a tanker. And these identities can be really important to military. Some of these identities are so important that they persist even after the actual occupation evaporates. So, for example, we call people on Navy ship sailors. We no longer have sails. They're not climbing the mast and working the rig, but we still call them sailors. We have people in the US army that we call cavalry. They don't ride horses. They don't even know people who wore horses. That's so long ago. But that identity of cavalry persists. They have horse Stetsons and they have boots and the whole thing. But in some cases, that can hinder adoption. And I think one of the places we've seen this most clearly is the US Air Force's struggle with drones because of this, this sort of salience of the pilot as an identity. And so when we have pilots flying Air Force Reaper drones, they're sitting in, like, a cockpit on the ground wearing a flight suit, as though they were in a cockpit of an aircraft. The army doesn't care about pilots as an identity. This is like an important thing. If you walk in the army, you're like, I'm a pilot. People like, all right, give me a cup of coffee. They don't care, right? They don't think about it that way. They have enlisted personnel flying Them they were earlier to adopt automated takeoff and landing. They have been more open to the idea of one person controlling multiple aircraft. They call their people operators instead of pilots. They have more automation inside their systems, even though they're basically using the same technology built by actually the same company. Believe it or not, what the army does care about, this is wild to be. The army cares about deploying overseas. So while the Air Force has their pilots flying from bases in the United States for their drones, the army, at least during the wars of Iraq and Afghanistan, would forward deploy their operators for their drones. Because soldiers don't telecommute to war. Soldiers need to be there. Boots on the ground. You feel like it totally doesn't matter.
- [76:09]
B
Yeah, fascinating.
- [76:10]
A
But so these identities and so I think that can be a hindrance to adoption and sometimes a really big one.
- [76:17]
B
It sounds like there's a lot of. Yeah, it is these psychological factors rather than technological ones or even super political ones. So that's really interesting. I want to talk through a few failure modes of autonomous weapon systems. So you've already kind of alluded to a few. One is automation bias. So the idea that humans tend to over trust machines, especially when the machines sound confident, which I already relate to given my experience with LLMs, it's just really hard not to be like, yeah, this thing seems really smart. It's probably telling me true things. Another one that's come up is this kind of brittleness. So these systems can look superhuman in training, but then kind of fail catastrophically when the environment changes or when someone actively tries to fool them. There are I think, loads and loads of these failure modes. But yeah, to start kind of. Which failure mode worries you most?
- [77:21]
A
I mean, I think the way that I tend to think about this is that the sort of operating parameters of these systems, whether it's a rule based system or one that relies on machine learning, tend to be very brittle. And when you sort of push the system outside the bounds of its operation, either it's in a situation where we didn't have rules for that, or rules don't apply, or it's just not in the training data for machine learning system, they tend to fail quite badly. And humans don't think about them that way. So that's not inherently a problem. If the people employing the systems understand that limitation and they know what the bounds of its operation are, if they know like, oh, it works in this setting and doesn't work in that, then fine, we can compensate for that limitation. The problem is that humans, you alluded to this with LLMs will often experience that a machine is good at one task and then humans logically transfer that competence to some other very closely related task. I think you could see this in examples of self driving cars where certainly early on people would see that the car was effective in driving and staying in lanes and then assume over trust the automation and assume, okay, well it's just a safe driver. If a human drove that way, we'd say they're a safe driver. And then the car would catastrophically fail by driving into concrete barriers or parked cars or fire trucks or other obstacles, resulting in fatal incidents. So I think that's sort of my concern and I think that there's lots of reasons to think that we would see that happen in warfare. Because the environment's uncontrolled, we don't know what we're going to fight. The enemy's going to do creative things. Things. If you were deploying it to a conflict like what's unfolding between Russia and Ukraine, well, you could test the system and you could get good data and maybe you have a good sense of what's going to be. But in situations where in peacetime you're planning, I think we should expect that the systems will be less capable in wartime. And that's something that militaries need to account for.
- [79:38]
B
Yeah. An obvious consequence is mistakes that lead to more casualties or mistakes that lead to kind of strategic blunders for whoever's weapons those are. Does this also make it more likely that there are escalations of conflicts because of kind of mistakes that cause adversaries to to think a particular thing is happening that isn't happening?
- [80:09]
A
That's something that I'm certainly deeply concerned about. And we have situations where countries will often be in militarized disputes short of war, but they have their ships and aircraft operating in close proximity and we get incidents sometimes. We had them during the Cold War between the US and Soviet Navy and Air Force. We have had incidents between the US and China, for example. And I think that these are dicey when there's humans involved. I think one concern is that machines do something surprising and then we don't even know was that intentional act on the part of the other country or not. People aren't sure how to interpret that. I think that's certainly one concern. But that concern about escalation control still exists in wartime, that a lot of times countries are still managing escalation in wartime. We could see very tightly in the Russia Ukraine war, the two countries are going sort of all out against each Other. But Russia is being very careful in calibrating its escalation against NATO. And the US has been very cautious in its support for Ukraine and not over escalating the conflict. And so that's an area where now we've had recently, for example, some Russian provocations of drones flying into NATO airspace. That seems like a very intentional act on the part of Putin. He's trying to sort of slowly erode some of NATO's deterrence. But that could look very different if you had some crop of drones do something or even strike targets, and then you don't know whether that was intentional or not, or even you could have drones just do it by accident and then maybe unintentionally cause escalation.
- [81:54]
B
I guess I'm interested in really concretely understanding the stakes. So what kinds of outcomes do you think are most worrying and people should be most paying attention to when thinking about these weapons being deployed?
- [82:11]
A
I think there's several. When it comes to autonomous weapons specifically, I think one concern, of course, is that you could see much greater civilian harm on the battlefield, either because of accidents, because some autonomous weapons or a group of them strike the wrong targets and cause civilian casualties. It could be because they strike the right targets, but they're not correctly accounting for civilians nearby. So, okay, it's a valid enemy tank, but it's parked into a hospital, and it caused all these civilian deaths just to take out this tank. It's not proportional in terms of the military necessity of striking that target. It could be that. That there's a slow erosion of human responsibility, that by automating attacks, humans sort of don't feel morally responsible anymore. And so even if humans are pushing the button and authorizing it, humans are just less engaged with what's occurring. That's, I think, one concern, certainly, for countries that just aren't concerned about civilian casualties, this could allow greater destruction. And then I think in terms of escalation, I guess my concern would be that autonomous weapons sort of introduce another sort of concept of this slippery slope towards warfare. Right. And a lot of conflict short of war involves brinkmanship, where opposing leaders might be sort of dragging each other further down the slipper slope, but nobody really knows when they're going to tip over the edge. That's certainly what Putin is doing with these provocations against NATO is engaging in this kind of brinkmanship, and that autonomous weapons might make this slope slipperier in ways that are hard to see and we don't understand. And so, you know, take an example, like the Cuban missile crisis. Well, maybe you have autonomous aircraft or drone boats or automated missile defenses. Shoot down something and then maybe make it harder for political leaders to walk back. Maybe even there's an outcry domestically. And remember the main we need to remember these service members who were killed and we need to strike back and we can't show that we're weak. I think that's. There aren't really historical examples of accidental wars. There are instances where you can see miscalculation, certainly in many cases. But I think that we could see circumstances where there's greater miscalculation or there's greater accidents that might escalate conflicts.
- [84:45]
B
Yep. Yeah, that makes sense and is deeply unsettling. Another risk that came to mind for me while reading your books is kind of concentration of power. So could highly automated command structures make it easier for a small group or even an AI system itself? If that AI system were both very capable and had goals that weren't perfectly aligned with its kind of creators goals, could this enable a small group or AI to seize or hold power? Because there are fewer. You need kind of the buy in and support of fewer humans at one time.
- [85:31]
A
I mean, I think the directionality of that argument is right. I think the question is kind of the magnitude of how strong would that effect be. So dictators often are able to maintain control over larger populations with the minority of the population on their side, based on an ethnic minority or political minority. But they do need people. Certainly one effect of this technology, whether it's autonomous weapons in the form of robots or, or police robots or AI systems to process information, might be that you just need fewer people. And we've had instances where dictatorships fall because the dictator tells the troops fire on the protesters and they just don't. The troops lay down their weapons and they say we're not going to. These are our families and community members. We're not going to shoot them. And sort of you can take away that ability for humans to just say no. And so if you added lots of autonomous weapons, you could imagine in 1989, would the Eastern Bloc have fallen? Maybe there's scenarios where it doesn't that allows smaller groups to hold onto power longer. I think in the extreme case, when you think about, okay, would it allow one individual or a small group or an AI system itself to execute a coup or take over? I think it requires a little bit more like what's the specific mechanic of that? Like how would that work exactly. I think there's, as we see more infrastructure come online and be more deeply integrated into cyberspace. That introduces lots of vulnerabilities and areas where groups using AI for cyber attacks or conceivably down the road AI itself could do much more harm in the real world by sort of taking over cyber infrastructure. Right. And that could look like power grid, that could look like water treatment plants and nuclear power plants and other kinds of things. Certainly digital infrastructure. One of the things that in the 20th century that countries would do and they had a coup is they take over the radio station and now it's social media platforms, other things. You could imagine AI systems doing that and even doing it in ways that might be subtle and manipulative. Right. Like platforms are not transparent about the algorithms behind their social media platforms and what kind of content is promoting. That's already very contested politically in AI systems. It's worse because the companies themselves don't even really know what's driving the models to generate certain types of things. Right. Like it's much harder. So could you get weird, sneaky biases? I think that's plausible. I think to get to more really truly catastrophic scenarios like some sort of small group or an AI taking over a political system, you sort of need to, I think probably end up in a world that's maybe more wired digitally than today, where the trend lines are and where military forces are much more highly automated than they are now. And there still probably would be humans in the loop for lots of sensible reasons for militaries. I would be more worried about maybe things like small groups of people or AI taking control of corporations which have a lot of power in modern society and are sort of well baked, designed for this kind of thing and then leading to political power or influencing the information environment. Or just AI systems themselves become so central to the information environment that they're being used to manipulate information and politics in some way. Everything in society is downstream of information flows, culture, politics. If you can manipulate that, that seems like a really alarming failure mode. And then who needs to do some sort of 20th century style coup with the tanks? It doesn't matter. You're. You're in control.
- [89:36]
B
Yeah. Okay, so those are some things that worry you. I'm interested in kind of the strongest case that these weapon systems will actually make us safer and war less deadly. I guess just sticking on kind of coups and concentration of power. Could automation make these things more difficult by improving transparency and monitoring and just making it harder for. For things to be done in secret or in surprise?
- [90:07]
A
I think there's a broader case to be Made that we are on a trajectory towards greater transparency, towards a world of increasingly radical transparency, which is secrets are harder to keep than they used to be. That's true in our personal lives. That's true for companies, it's true for governments and intelligence communities that as to steal secrets. Back in the old days, you had to get somebody into a vault and get a hold of paper documents and then the people that stole out documents for the Pentagon Papers shoved them in their pants to sneak them out and make photocopies or you got to get a camera in to take pictures and things where now everything's digital. And so if you can hack the right system, you get access to these huge troves. And we've seen seen in things like WikiLeaks, like these massive dumps of documents. You've seen hacking of things like the Chinese hack of the Office of Personnel Management where millions of personnel records by the US government were stolen, or the most recent Chinese telecom hack, where Chinese inside the telecom infrastructure. The digital infrastructure of modern society means that you can hack these key companies. You just have access to everything, like you run the pipes and everything is increasingly digital. And that's probably only going to increase, right? So conversations that like this conversation, right, we put online, recorded, we'll be able to be digested by AI, right? Like things are more digitized, more communications in our personal lives and businesses become more digitized. And so that just maybe makes it harder for anybody to do anything in secret and including for intelligence agencies and for militaries and not, you know, could be good and bad in lots of ways. But one might argue might make war harder because you just, you can't get away with stuff that people might have 50 years ago.
- [92:00]
B
Right, Right. Yeah. Okay. So that's one way it might make war harder, which you might think is a reason to think that it'll also make wars less common and the world more stable. What do you think is kind of the strongest argument? That integrating AI into military systems could reduce suffering, I guess, rather than increase it.
- [92:23]
A
So I think there's actually a super strong argument here. Okay. And I think the argument in a nutshell is that humans do a terrible job at this and cause a lot of civilian deaths through war crimes, through accidents, or just through the imprecision of modern weaponry. And AI could do better. And just like self driving cars should be able to save lots of lives on roads by just being more precise, that AI could do the same to warfare, that AI could enable militaries to more precisely strike military targets and not strike civilian targets. Now that does hinge on whether militaries are trying to do that, which is not always like to be fair, it's not always the case. It is a war crime to intentionally kill civilians and strike civilian targets. Militaries also do that sometimes. That's happened historically throughout war, that nations often will target civilians. And we continue to see that in modern conflicts where countries that are deliberately killing civilians, or certainly not trying not to. I think the strongest case would be if you look at the pattern of precision guided weapons over the course of the 20th century, it has led to less civilian suffering. So in World War II, you had to basically drop massive amounts of ordinance to hit a bridge or factory inside a city. And that led to wholesale devastation of cities in Germany and Japan because you just couldn't actually target the infrastructures precisely as militaries might want to. Now, some cases, you had some countries sort of attempting widespread aerial bombing, trying to harm civilians. But the US Army Air Force going after German cities, for example, tried to do precision bombing going after industrial targets, and it just wasn't precise enough. That change with the advent of precision guided weapons. When you get to modern day where the US Military, at least with GPS guided bombs or laser guided bombs, can strike targets with a high degree of precision. As a result, not only are there less civilian casualties when say, an air force using precision guided weapons as bombing targets, our perception of acceptable civilian casualties has changed as a result. And so, for example, in debates here in the United States about US Drone campaigns, you would have sort of there became this expectation that drone strikes would have zero civilian casualties. Now, one could argue that's right or wrong, but it's a huge shift from how people thought about that, say, 80 years prior. And so I think that's sort of the strong case of technologies actually on the side of a greater precision, and that could lead to less suffering and war.
- [95:24]
B
And so overall, how would you describe your general feeling about this kind of radical increase in automation that you expect to happen over the next several decades?
- [95:37]
A
Well, I feel great about it all. I think that it really depends on how militaries use AI. And I think that there are ways for militaries to use AI that might make warfare more precise and more humane. We need to ensure that we're adopting AI in ways that don't lose our humanity in the process. I think it's important to keep humans in control of warfare, to manage escalation so that humans have the ability to end wars. And I think it's important for humans to bear some moral responsibility for killing and suffering in war. And that's, I think, actually like a harder argument, because it's real people that bear that cost. Right. That's someone who then is suffering from PTSD afterwards because of their effects in war. And I fought in Iraq, Afghanistan. I've had a lot of friends who've continued to suffer after the war has ended from not just physical but mental and emotional injuries, moral injuries that they might have suffered during the war. And it's not really fair as a society that as a democratic society, we make a decision as a whole to go to war. And it's a very small slice of people that bear that cost for the war. But I do think it's worth asking, what would it mean for war? Would it mean for us if no one slept uneasy at night, if no one was concerned about the suffering and the casualties that occurred in war, and would that make wars more likely or more deadly? So I think it's possible to imagine a future where we adopt this technology in a way that does lead to positive outcomes, that makes warfare more precise and humane and doesn't lose our humanity in the process. But I do think that matters in how we do it, and I think we want to be thoughtful about how we use the technology.
- [97:42]
B
Yeah. Pushing on. We often talk on this podcast about transformative AI. So AGI, superintelligence, other terms describing kind of highly advanced AI systems and kind of achieving them, is the stated aim of many of the top AI firms right now. So do you personally buy the arguments that AGI could kind of massively transform society in the next few decades?
- [98:09]
A
Yeah, I mean, I think it's actually kind of bonkers for people not to believe that, given everything we've seen with the technology we've been hearing for years now, oh, deep learning is hitting a wall. It's petering out. Maybe that'll happen. But that prediction has been consistently wrong so far. And what we've seen a pattern of, is an AI system will be released to great fanfare. Oftentimes people sort of poke at it and realize, well, it's not quite as good as maybe a company had hyped it out to be, and it has a bunch of criticisms, or people will say, oh, it can't do this, can't do it, can't reasoning. And then six or eight months later, another AI system comes along that just totally obliterates that criticism. So given those trend lines, I think it seems quite reasonable to think that we're going to continue to see AI Improve. And one of the things that I'm certainly struck by is these sort of long timelines that sometimes people have now towards AGI are often really short. So people will hear critics, critics of AI. Well, AGI is a decade away. That's really close. Like, that's crazy. Like, I remember things 10 years ago, it's not that long ago. So like I, you know, I think I would say that sort of my expectations for how fast this is unfolding have certainly been like many people would pull forward over the last couple of years. That things that I sort of, if I would have had to make a best guess, I thought, well, maybe this will happen by 2040 are happening like now, right? And I'm like, oh, wow. And I, and I, I'm really struck by that because I've been working on these issues for probably 15 years in various forms in the Pentagon and here at the think tank. I work at the center for New American Security. I was certainly working on AI issues before people were calling them AI when it was automation or autonomy or something, before this version of the deep learning kind of revolution really kicked off. And it strikes me that I continually am surprised by the pace of progress. And that concerns me a little bit. I think what worries me in particular is that a lot of the technology seems to democratize violence and in particular at very extreme scales. So I think the questions about autonomous weapons, for example, that we've kind of been talking about, or swarms and warfare are interesting. I think they're concerning. I think there's a lot that we ought to be doing, that nations ought to come together to sort of put some rules of the road in place. That's not really the scary stuff. The scary stuff is things like biological weapons. It's the intersection of AI and cybersecurity and biological weapons. It's things like using AI to design much more powerful synthetic biological weapons. It's democratizing that technology. As we see AI tools and large language models and more general purpose systems and agents become open source and available to anyone. As we continue to digitize our world, more critical things become networked and vulnerable to disruption through cyber attacks. Power grid, water treatment plants, our sort of information systems are now vulnerable to disruption digitally through hacking of telecom networks, which has occurred through manipulating social media. Like those sort of long term trends really worry me. And I think that there are some like really concerning outlier possibilities in really horrible types of catastrophic harm that could come from that that are probably not likely. But like, I kind of, I kind of don't Want to find out, like, how likely is it that someone makes some horrible biological weapon that kills millions of or hundreds of millions or billions of people? Like, I don't know, like, let's not. Let's not inch up to that line. So those are the scenarios that really concern me.
- [102:28]
B
Yep. Yeah. Do you think that these ideas are taken seriously in kind of national security circles or do they still sound kind of sci fi?
- [102:38]
A
I think generally this sounds kind of sci fi.
- [102:40]
B
Interesting.
- [102:41]
A
In Washington, people have certainly got the AI bug, and that's been true for a while, that Washington is all in on integrating AI into the US military, on maintaining US dominance and artificial intelligence overall vis a vis China in particular. These things like superintelligence or AGI, or a bit of a dirty word, AGI is becoming a term that's becoming a little bit more normalized in discussion about AI. Less so. But I think it's interesting because the defense and security community that I come from and that I work in spends a lot of time thinking about hypothetical scenarios. We do detailed games and scenarios of nuclear war with China or Russia or different. There's a war unfolding between different kinds of countries and how does the U.S. respond? And we do really detailed analysis that people take those things very seriously. So if I were to do a project, for example, and lots of experts do this, on how a major war between the United States and China might unfold over a period of maybe even years, like a very protracted conflict, and maybe there's limited nuclear use involved, people take that super seriously. But then if you start talking about, okay, well, I've got this really sophisticated AI agent and it escapes from a lab and it spreads on the Internet and then it hacks a critical infrastructure and takes down the power grid. People are like, what are you talking about? People watching too many movies. And I've observed this trend long enough in the security space that it's interesting to me that that's the same reaction that people had to autonomous weapons OR drone swarms 15 years ago. Those are just not like, if you were talking about that people sort of like, I think you watch too many Terminator movies, and now those are taken quite seriously. And so I do think that people will get there. But for some reason there is a. A real hang up on taking some of these scenarios seriously.
- [104:57]
B
Do you have a guess on what specifically the hang up is?
- [105:01]
A
I don't really know. Like, I've spent a lot of time actually puzzling over this that I'll be in, you know, at conferences or in Conversations with people or in roundtable discussions or private workshops. And I've thought this to myself a lot, like, what is the hang up here? And it seems like there's something about this idea of AI reaching or eclipsing human intelligence that sort of is really hard for people cognitively to wrap their minds around. And you tend to get really weird reactions from people when you sort of put that question on the table. I think there's some people that just reject it as like, not possible, not going to happen in any reasonable timeframe. It just strikes them as fanciful. I think the flip side is I'll often see a lot of writing in this space where people get to this point of AGI and then it's like magic happens, right? Right. It's like, oh, well, then the AI becomes like, because it can program itself and then it recursively self improves and then it's super intelligent within a period of hours or weeks or months or whatever. Your sort of vision of what that kind of takeoff looks like. And then it can just do anything, right? Like it could take over the world and it builds these robot factories and, you know, and you're like, well, there are some physical constraints that exist. How exactly would those sort of things unfold? And it's like, well, it's super intelligent. You figure all that out. And I think sometimes you get people then reacting to that. They're like, well, that's just nonsense and it strikes them as fanciful. And so I think there's something about this idea, I don't know what it is, of AI eclipsing human intelligence. It seems really hard for people to just grapple within a way that's grounded and that takes it seriously. And I don't know why that is.
- [106:54]
B
Yeah, super interesting pushing on. What impact will AI and autonomy have on cyber warfare?
- [107:02]
A
Well, this is a place where I do think the effects of AI will be very dramatic and much faster than in other areas, like in physical domains. But it's maybe worth starting where we are today, which is the effect is somewhat limited of AI and large language models. At least now automation is already widely used in cybersecurity for defensive and offensive purposes. The first self replicating worm, that Morris worm, dates back to the 1980s. So we've had self replicating malware for WoW. Now they can spawn across computer networks. We've been some really quite sophisticated cyberweap Stuxnet, the one that is widely believed to be built by the United States and Israel, that took down Iranian centrifuges to sabotage the nuclear program had really sophisticated automation so that it could spread across networks that were air gapped from the Internet and where this sort of people controlling them, the humans couldn't direct what the malware was doing. And certain forms of automated vulnerability, discovery and patching have been around for several years now. I think we should expect that AI will continue to sort of advantage both attackers and defenders here by finding more vulnerabilities more effectively, by being able to recon networks to better understand what's happening throughout the kill chain, if you will, of a cyber attack, that AI will start to sort of play roles incrementally in each of these in enhancing human productivity. I think the sort of interesting question is, do we get to the point down the road where malware is much more intelligent and adaptive than today? And so today you have malware that spreads on its own, that is self replicating, that acquires resources, like botnets that acquire computing resources and then can use them for things like distributed Denial of Service attacks can sort of leverage that. But when there are adaptations to malware, those are done manually. And so conficker, this huge worm that spread across the Internet several years ago, is a really interesting case where there were a bunch of variants that evolved over time. And so the task force that was sort of put together of law enforcement and intelligence communities and the private sector to combat the swarm was fighting different variants over time, but those were all designed by humans. And so do we get to the point where you have malware that's actually able to evolve and adapt someone's own either it's more clever when it's on a computer network and able to maybe hide itself in response to threats or adapt what it's doing to the network itself. And you can imagine a more capable reasoning model that could assess what's going on on computer networks and then make some reasonable judgment about what to do, or able to actually change itself over time, which would seem like a much more dangerous kind of threat. And we've seen sort of concerning attempts by language models to engage in behavior like self exfiltration, copying itself, copying itself in ways that would try to preserve its goals, or copying itself to overwrite the goals that a human would do of a new system. Now, the models aren't very good at that yet, because they're just not good enough yet at software engineering. But you have all of the pieces in place. Self replication already exists. The ability to acquire computing resources already exists. The tendency of models, it's not common, but it happens where models might attempt some of These kind of concerning behaviors like self exploitation, it looks like right now the missing pieces just. They're just not good enough. That's going to get better. You can really take this to the bank. It's going to get better. On what timeline? I don't know. So I think that that's a very troubling possibility in the long term that you could end up with malware that is maybe feels more like biological threats. Right. Where during COVID we saw different variants over time. And then you're sort of fighting against this threat that's continuing to evolve. And that's a. Seems like a really difficult problem.
- [111:54]
B
Yeah, yeah. And does that have differing implications for either different types of groups or different powers? Or is that kind of just like uniformly like. Cyber warfare gets more impactful across the board?
- [112:18]
A
I think it's obvious that it gets more impactful because a lot of that has to do with how well defenders find ways to shore vulnerabilities inside their networks. And so this question of how does the offense defense balance, how does AI change the offense defense balance in cybersecurity is, I think, a really critical one. And I've seen compelling arguments from good analysts on both sides of this equation. So I think it's worth starting with, okay, so today, right now, cybersecurity greatly benefits attackers. And the reason why is that ultimately attackers get into computers and networks by finding vulnerabilities, by finding mistakes, bugs in code that they can then explain, exploit. And so the problem that defenders have is you have these massive, massive code bases for an operating system on a computer or an industrial control system for some industrial plant or something. And it's a little bit for defenders, like trying to defend this castle that has these sprawling walls that stretch and snake over hills for miles. And you've got to cover every single possible entry point. The attackers only need to find one vulnerability, one door that's unlocked, one secret tunnel that you didn't know about that they can get in through. And then once they're in, they can cause all sorts of problems. They can escalate their privileges, they can create new vulnerabilities to find other ways in. And so that's kind of the status quo today. One of the, I think strong cases for a couple of them for how this technology might benefit defenders is one is that if you have AI that can be used to automatically find vulnerabilities, finding a vulnerability and finding the patch for the vulnerability are sort of the same. So if you can find the vulnerability, you know how to patch it. And if defenders use this technology to run it against their networks before it's deployed, or they're just really much more assertive in doing that, they can automate a process that right now is manual. And so that can allow them to find and patch these vulnerabilities before attackers can get in. So that sort of starts to level the playing field. Attackers can use these, too. But right now, what's limiting defenders is the human cost of going through all this code and automation, really relatively advantageous thing. Now, that hinges a lot on do defenders actually do that? And that's a big problem right now in cybersecurity is a lot of times it's actually a vulnerability that we knew about, that the patch is available, but people haven't updated their networks or their computers. And that's not always true, but that is a consistent problem. And so do defenders actually do that? The other really interesting, compelling case is that AI. I'm not sure that we're there yet, but as AI gets better at writing code, we just have fewer bugs. And that actually is, I think, really compelling, because that doesn't really hinge on defenders necessarily doing anything intentionally. It's just that as we evolve over time to a world where maybe more and more software is just written by AI, if the AI gets pretty good at it, there just might be fewer vulnerabilities in the first place. And that actually just sort of shores up defenses. But a lot of it depends upon how the technology is employed by both sides.
- [115:52]
B
Yeah, I mean, yeah, that last point sounds pretty potentially huge. I guess it does seem like things point in different directions, and it seems like it's not clear to you that one is definitely going to dominate. But if you were to describe at least a plausible scenario that you put some stock in, where cyber capabilities do end up genuinely shifting the global balance of power or triggering escalation between major states, what is a scenario that kind of might explain why that would end up happening?
- [116:38]
A
I think that the scenario that worries me the most is that we end up in a world where malware is much more intelligent and adaptive, and you can end up with malware on the Internet. Like, we have worms and botnets today that are intelligent. They're able to have goals and to plan to execute them, to adapt to defenders. And it's just like a much more difficult problem for defenders to go against that. It's not like, okay, they're fighting a botnet, and then they could defeat it, and then it's done. It's that this is like an intelligent and adaptive adversary in and of itself. And sort of whether it was written, I mean ultimately would have been designed and written presumably by humans, but whether it was used for a certain goal and sometimes becomes irrelevant once the malware is launched, it's already true today. Well, there are lots of examples where an actor sort of is trying to do something that still might not be great, like they're trying to steal passwords, for example, and then they release some botnet that spreads across the Internet because of replication and it causes all sorts of problems. Or somebody releases something open source and now there's all sorts of variants of this that are used by others. And that seems like a very different kind of world to be in, to be countering that kind of threat.
- [118:08]
B
And why wouldn't this dynamic also be kind of resolved or at least helped by this thing you've already described of defenders being able to use capable AI systems to defend against increasingly kind of sophisticated cyber attacks?
- [118:26]
A
Well, presumably they would that defenders would be using AI to improve their defenses. I think the question is like, how does that net out and why might we still have really nasty threats? I mean, one trajectory could be that we just have the sort of a continuation of the relative balance we have today, which is that even though people know vulnerabilities exist and it's really important to defend against threats, there are still ways in humans, humans are dumb and they click on the link in the email that they shouldn't, or people don't update their software in ways that they should. So that could be just one scenario where you just get the same context on a higher level. One way that you could see this kind of really sophisticated AI driven malware benefit attackers in relative terms is if they're just more comfortable taking risks, then you could see maybe someone cobbles together some offensive cyber system that has an LLM as a component of it and it's doing some reasoning and it's writing software and it's not totally clear what this thing might do, but an attacker has some, it's just a much higher risk tolerance. And they're like, yeah, I'm going to use it to hack some server, engage in some spam email thing or some denial of service attack or whatever they're trying to do. And defenders have the same access to technology, maybe even a little bit better if they have early access to models through more sophisticated labs. But they're just more hesitant because they're like, yeah, I don't know what this sticks with too. And I can't have this sort of Intelligent AI defensive system running around on my network. And then it decides that, you know, the weak leak of the humans and it locks out all the humans. Right. It does something strange, you know, and that because of the unpredictability of the systems, that defenders are a little more cautious in employment, and that might benefit attackers. My colleague Caleb Withers at the center for New American Security has a great report on this issue of cybersecurity and why it might benefit attackers that recently come out. That's really worth checking out.
- [120:44]
B
Cool. Okay. Yeah, we'll link to that for sure. Yeah. Before we move on, is there anything that you feel is understood or underrated in the kind of area of cyber warfare, as it gets kind of more automated and more AI based that you'd. Yeah. Want to make a pitch for?
- [121:05]
A
I mean, I think here's what worries me is that I see two trend lines that I think intersect in some troubling ways. One is this trend line towards more of human civilization and our lives becoming digitized and networked and accessible through computers. And that trend line seems like it's likely to just continue that we see exponential growth in Internet of Things devices, in network bandwidth for both wired and wireless networks. It's more things are becoming digitally connected, which makes them inherently vulnerable to cyber attack. How vulnerable depends a lot on how much defenders do. But things that 40 years ago, you couldn't attack a power station through the computer network. There was like, no way to do that. Now it's been done. Russia has taken down elements of Ukraine's power grid through cyber attack. So that's been demonstrated, for example. So we have this long trend line, and then you have this simultaneous trend line of artificial intelligence becoming more capable. And that sort of worries me that you could end up in this place where you have much more intelligent forms of malware, much more sophisticated ones, and there's sort of just this greater inherent vulnerability in society over time that lots of things that we care about are actually now vulnerable, just as the systems that might hack them becoming much more sophisticated. And that feels uncomfortable.
- [122:46]
B
Yep. Yep. I'm unsettled. Pushing onto another topic, I'm interested in how the integration of AI and autonomy into military systems shapes the balance of power between the US and China in particular. So. Yeah. Is there kind of a high level answer to that question before we get into some details?
- [123:07]
A
Well, there are some ideas. So one argument that I've heard is that China will be more willing to automate systems because it's an authoritarian regime and they don't trust Their people. I'm not sure that that's valid. Like, that's a thing I hear in Washington, people saying about China. When I talk to Chinese military officers who engage or former Chinese military officers who engage in these kinds of issues, thinking about AI in the military, I don't really hear that. I hear a healthy skepticism about AI and a desire to adopt AI into the military, but also consider concerns that maybe humans should be on the loop or what. Sometimes the Chinese translation is above the loop. It has this idea of supervisory human control. Maybe humans can't be in the middle of everything. But we do know that these AI systems, they screw things up. We don't necessarily trust them. We want to maintain control. There was a strong desire in the Chinese system for control from the top down. I think one concern that I have would be about risk taking and assurances in sort of testing, evaluation of AI systems that there are pretty robust procedures in place within the US Military to test new weapon systems to make sure that they're reliable once we deploy them. And that the Chinese military, the People's Liberation army, might be more willing to take risk because they feel behind, they feel like they need to catch up to the United States or to get guidance from senior leaders. Do this. They're just going to do it. And does this drone have a high failure rate? Doesn't matter. She says do it. So we need to deploy this thing. And that sort of concern about risk taking in accidents is something that I do worry about.
- [125:03]
B
Yeah. Can you imagine kind of AI accidents becoming the next flashpoints? So I guess kind of the equivalent to near miss incidents during the Cold War.
- [125:13]
A
Absolutely. And that's, I think, a big concern that I have, that if you transition to a world where, say, the United States and China have drones deployed at sea, in the air, undersea, interacting in contested areas like the South China Sea or near the Taiwan Strait, and they have some degree of autonomy. They don't have to be fully autonomous weapons per se, but they have some degree of autonomy that maybe does something strange that causes an incident. So you've got an autonomous boat and it gets too close to another boat or causes a collision, or a drone that gets too close causes a collision, and then there's a mishap and there's a political incident. Now, I don't really think that those incidents themselves then lead to full scale war. Not unless leaders are looking for an excuse to go to war. But I don't know that it's helpful to introduce a lot of this, maybe more friction and unpredictable dynamic into that situation. And there might be situations where leaders feel compelled to respond in some way to look strong. And I think that would be concerning.
- [126:20]
B
Yeah, I guess in the nuclear space, something roughly like nuclear parity has created a kind of uneasy stability. Is there a version of that for kind of AI and automation parody in the military that does the same?
- [126:42]
A
It depends a little bit, I think on the scenario that we're envisioning. So if you think about one of the scenarios that US war planners really worry about is a Chinese invasion of Taiwan. And she has said that the Chinese leader has said that, you know, they, Taiwan is, is part of China, that's their view. And they intend to retake control by force if necessary and has directed the Chinese military to be prepared to do so by 2027. Now I don't know realistically whether they'll be able to do that. I think probably not. But you know, if technology, I think the key sort of threshold in that case in that particular scenario is if they think they could get away with it. And I think if you got to the place where they believed their technology gave them an advantage enough that they thought that they could get away with it and keep the US at bay. And coupled with the US is weak. This is the Chinese view, the US is weak. It's a declining power, it's riven with domestic strife. It doesn't really care that much about Taiwan. When push comes to shove, the US doesn't have the stomach for a long protracted war. Even if the US does get involved in the fight, China's willing to gut it out over the long term if it comes down to a bloody fight on the island. And the US doesn't have their stomach for that. And we could see how weak the US is in Ukraine and they're not willing to support Ukrainians and even US isn't even fighting there. So that might lead to China Xi to then say, you know what, I'm going to do it. Similarly to Putin thinking, okay, I'm going to invade Ukraine, I'm going to do it. And in some cases from these leaders. Maybe it's a legacy issue for them. It's one where they see a retaking territory that they see as theirs is something that they want to leave as a defining legacy. And you can have situations within authoritarian leaders engage in some pretty risk taking behavior. And even if it looks dumb to us, it's very clear that Russia's invasion of Ukraine has net made Russia economically, politically and militarily weaker and has strengthened NATO he still did it. And I think that would be concerning.
- [129:05]
B
Yeah. So it seems like there are just so many parallels to Cold War era, kind of building up of nuclear stockpiles and nuclear technology. And it seems like China has these interests that could be really benefited by having this at least temporary, if not indefinite strategic advantage militarily. Are the US and China currently in a race to kind of automate and build AI into their military systems?
- [129:41]
A
Well, they're certainly in a competition militarily to maintain an advantage over the other and to adopt AI. Sometimes it's characterized as an arms race. It is clearly not an arms race, if you use that term in a precise academic way and the way that academics talk about arms races. And we have historical examples of the nuclear arms race, the arms race among battleship construction in the early 20th century, where academics define it as above normal levels of defense spending that's driven by two countries competing against another. So it's kind of hard to pin down numbers of AI spending inside militaries. Bloomberg had done some really interesting work a couple years ago pouring through the Defense Department budget to try to figure this out. How much is Defense Department spending on artificial intelligence? And they don't have a good answer internally. DoD doesn't, interestingly, and Bloomberg came up about 1%. That's not an arms race. That's not even a priority. Right. When you have senior defense leaders saying, oh, AI is the number one priority, no, it's not. Your Joint Strike Fighter is your number one priority when I look at what you're actually doing. So I think it's clearly not an arms race. I do think that there is an adoption competition in AI of how do militaries find ways to import the technology. Both the US and China are going to have access to roughly the same level of AI technology. And whether OpenAI is a couple months ahead of deep seek just doesn't matter because let's say that there's a gap of 6 to 12 months between leading labs in the United States and China. Well, if the US military is charitably five years behind the frontier of AI in adoption, maybe more like 10, that one year advantage means nothing. It's really a contest of adoption. But it becomes the most critical thing is figuring out how do you use this technology in a way that's constructive, that actually advantages war fighters. And I think that's a tricky one. Has a lot to do with how militaries organize themselves and create the right incentives internally for experimentation and reorganization. And I think it's just not actually clear who is an advantage?
- [131:56]
B
Hmm. Okay. You've argued that military power depends kind of less on really excellent algorithms and more on data, compute talent and institutions. This is a bit of a kind of multi layer question, but between the US and China, where does each side actually hold an edge? And I guess I'm interested in kind of like currently and kind of where things are going.
- [132:26]
A
Right. So the question that I had at the outset of Four Battlegrounds, my most recent book was the US and China are in this competition. What does it mean to compete in artificial intelligence? What are the things that you're competing over exactly? What are the sources of national advantage? So if you compare it to the Industrial Revolution, we saw that during the Industrial Revolution, nations rose and fell on the ground global stage based on how rapidly they industrialized. But also the key metrics of power changed. So now you want to count manufacturing output and coal and steel production. And oil became a geostrategic resource worth fighting over. So what is that? In an age of AI, of course, at the technical level, there are three core technical inputs into AI systems. Algorithms, data, and computing power, or compute. When you look for a competitive advantage, there's. It's not that algorithms don't matter. I don't think that they are likely to lead to a competitive advantage because the algorithms are really hard to keep secret and they proliferate pretty quickly. Now that could change. You could end up in a world where the leading labs really go dark in terms of sharing their secret sauce and they stop publishing papers and they stop sort of sharing even slightly sanitized details about what they're doing. And they're much more like pharmaceutical companies maybe that really work hard to kind of keep trade secrets. That might happen naturally as AI becomes more valuable. But right now we still see an awful lot of transparency from the leading AI labs. And so I don't know that that's a source of competitive advantage today. So sort of data and computing power are certainly areas of competitive advantage, I think. I've certainly seen people make an argument that China has an advantage in data because it's an authoritarian system. The government is putting in place all these surveillance measures to collect data on their citizens. It's true. The Chinese Communist Party is doing a lot of really dystopian scary things to gather biometric data and genetic data on their citizens. And there's surveillance cameras everywhere. They're increasingly incorporating AI. I don't know that that nets out to a data advantage from China in particular, because what matters for companies is the US Companies are Not confined to the US Population. Sometimes people are, oh, China's got a bigger population. Well, US companies have global reach. Facebook and YouTube have over 2 billion users each. In fact, US companies have done a lot better job of global penetration than Chinese companies today. With some exceptions like TikTok. The sort of collection that Chinese government is doing doesn't always translate into advantages for companies. Obviously, some companies, like, if you're a facial recognition company, you're going to have advantages over a US Company because we don't have the same degree of public surveillance here in the United States. Not saying you should, to be clear. Right. But the Chinese government has put in place protections on consumer data privacy. And because they don't really want. Want Chinese companies the same spying powers that the government has, and they are concerned about keeping some of these big tech companies in check. So I think it's more of a wash, probably on data. A lot of it has to do with how to use data better. But that's a really interesting contested area. And I've heard different arguments on kind of different sides of this. And the computing power is very clear to us, is a massive advantage here, partly because of Nvidia. A lot of. Of course, the technical choke points actually exist below the stack earlier in the supply chain at the semiconductor manufacturing equipment and software that's used to make the most advanced chips at FABS and tsmc, which is kind of wild because the idea that these most advanced chips in the world are built at this island 100 miles off the coast of China that the Chinese Communist Party has pledged to absorb by force if necessary. Like, none of that's good. But because they rely on US technology, the US has put in place extraterritorial export controls on these chips. So right now, if a chip is designed in China by a Chinese chip design company, manufactured at tsmc, and then shipped back to China for use inside China, that's banned by US Export controls above a certain threshold in terms of chip performance. And there have been a couple incidents where maybe that's a little bit leaky, but I think if the US can crack down on export control enforcement, that's a clear competitive edge for the United States. Not if we just sell those chips to China, which we've seen some recent moves of the Trump administration to do. But I think it's a potential advantage there for the US to harness. People also matter a lot. I think this is, yes, if you look at the numbers, China is training more AI scientists and engineers, and in fact, China produces more of the top AI scientists and engineers than any other country. If you look at research publications in conferences, for example. But those top Chinese scientists don't stay in China. Many of them come to the United States and they come to US universities to do graduate study. Many of them stay here afterwards. I think that's a huge latent strength of the United States that the best scientists and engineers around the world, they want to come to the US to study and to work in companies. And that's a place where the US immigration policy is just shooting ourselves in the foot, because that is very much one where if we constrain ourselves to US generated engineers, we will lose this competition with China. But if we're drawing on the best scientific in the world, the US has an unparalleled advantage here because people don't want to engineers, they don't want to go to China to live there. Then I think the last, maybe key piece of this is what we're talking earlier about institutions of this question of adoption. I think this is more of probably a level playing field between the US and China, whether it's in the military or other areas. How do we find ways to adopt AI that are beneficial across society? And I think it's kind of an even question. We have very different political systems, some cases that benefits China because they can move faster in many ways. They don't have the same kind of public give and take that we have between civil society and the media and the government and the private sector. But I do think that leads to better outcomes here in the United States. So I think that's very much an open question. But I do think the US has huge competitive advantages in computing hardware and talent if we harness them. And there's more we have to do there.
- [139:15]
B
Do you have a prediction of which of those factors will prove most decisive.
- [139:20]
A
If the current trends that we're seeing hold? I think that the advantage that US labs have over Chinese labs is effectively negligible in terms of societal adoption. The technology proliferates way too quickly. And what's going to matter more is how societies adopt AI to increase productivity, to increase welfare. I know sometimes people can sort of look at authoritarian systems and maybe be a little bit envious because they can move faster a lot of times. We saw, for example, early in Covid that the Chinese Communist Party was way more effective at closing everything nationwide and containing the spread. Once they got serious about it, in a way that in the US you had protests and people didn't want to listen, it was a much more thorny problem here. I Don't think in the long run that authoritarian systems are better than democratic ones. Democratic ones more are messier. But authoritarian systems end up being very brittle because it ends up being whatever answer the top leadership wants and they don't always get it right. And I think we could talk to Covid's a great example here. Over time, China's zero COVID policy was counterproductive and destructive. And I do think that this give and take is a lot messier in democratic societies, but it's probably in the long run going to lead to better outcomes. But it does depend a lot on how do we manage this transition is what it if AI has really disruptive effects on jobs and the labor market, are we going to take care of people? And I don't know that we've done a great job in the United States of managing the effects over the last several decades of job disruption from globalization and automation. And it's led to a lot of discontent today. And I remember in the 90s debates about globalization and at the time what we heard from leadership was, well, sure, maybe some jobs will be disrupted and we'll receive, but we'll help people reskill and get new jobs. We don't really put in place those social safety nets to the extent that I think are necessary. And I think that's going to be essential to managing this transition effectively.
- [141:32]
B
Yeah. How would you characterize China on that front?
- [141:36]
A
Well, the Chinese Communist Party in this current iteration under Xi cares immensely about political stability. And so we've seen just incredible economic growth over the last, really unprecedented in human history, economic growth in China over the last several decades, pulling hundreds of millions of people out of poverty, increasing welfare across their society. In previous leaders, the party had been willing to have some degree of political openness in small degrees and prioritize economic growth. I think we've seen under Xi that flips and she has really prioritized control, political control and been willing to sacrifice some economic growth. I think whether China is able to navigate that effectively is a really open question. So there's this idea of the authoritarian dilemma that basically authoritarian governments have this choice of either they can allow openness and free economic trade and open information and allow economic growth, but that will lead over time to greater political openness as well. Or they can crack down and they go the way of North Korea and you get political control, but you stifle economic growth. And actually north and South Korea are sort of really interesting examples of this today. China's been able to navigate that extremely effectively. They've been able to basically have their cake and eat it too, and maintain political control and economic growth. And I think how China navigates that is going to be really critical to answering this question of whether China is able to continue to grow as an economic power, whether they eclipse the United States or come closer to it, or they kind of end up. Some people argue we sort of hit peak China in terms of the relative power to the United States. And at least the current leadership seems to be really prioritizing political control over growth.
- [143:35]
B
Yeah, I want to ask about this comparison to North Korea. North Korea has done a depressingly good job at its kind of authoritarian ruling of people. China could, in theory, potentially do even more if it wanted to because of this trajectory toward really, really, really capable AI systems in kind of surveillance and censorship and repression. Those could get much cheaper and more effective. Does that seem like a plausible route China could go where I guess I'm interested in particular, one of our past guests, Tom Davidson, argued that AI could kind of enable new forms of authoritarianism that can actually be really stably locked in. Yeah. Does that seem plausible?
- [144:35]
A
Well, I think it's very clear that China is already building this very dystopian techno surveillance system within the country to monitor and surveil and control its population. And the trends are in the direction of technology enabling even greater authoritarian control. The most extreme version of this is in Xinjiang, where China's had this really intense crackdown on the Uyghur population there. And you have sort of these concentric circles of control from actual prisons that many Uyghurs are imprisoned to. For those that are released, they're on like a sort of degree of, you know, house arrest or even within kind of the cities. There are physical checkpoints, police checkpoints that are doing biometric scanning that are monitoring license plate readers to track where people go. Knives have QR codes on them to track knives. The technology is allowing a lot of monitoring of surveillance that wouldn't be possible then we've seen this Xinjiang ification of the rest of China, where there are surveillance cameras that are widely deployed throughout the country. It's when I was in Beijing last, I was sort of really struck by this, that I'd heard about all the surveillance cameras, and I didn't really appreciate just how omnipresent they are until it was physically there that you. Beyond like a street corner, they're in every street corner, there's multiple of them pointing in different directions. And halfway down the street they're very obvious because, of course, the Party wants you to know that they're watching. Right? And so in Tiananmen Square, I, you know, I sort of roughly estimated maybe like 200 cameras across Tiananmen Square. Now, Tiananmen Square is huge. To be fair, it's massive. But I realized that the goal of the Party is not even necessarily to like, okay, if there's another massive protest in Tiananmen Square to capture everyone there and their faces and who's the leaders, it's to prevent that protest from ever even happening in the first place. Protests would never even make it there because there's so much monitoring of their. And so it's down to things that almost seem absurd in the degree of control. China has this social credit system that is a little bit sometimes caricatured enough people talk about it. It's not one system. It's a whole bunch of different credit scores and blacklists, some of which are for financial credit, but some of them are more social in nature. So someone might get score based on whether they separate their trash and recycling, like down to that level of control over what people do. And of course, technology enables really high degrees of control. So they can do things like, okay, if you use a credit card to buy a hotel room, even if you don't need a credit card, maybe we just monitor what hotels are doing. Buy a hotel room in the city in which you live, what are you up to?
- [147:23]
B
God.
- [147:24]
A
Right. And so they get that data and maybe they can look into this. And what's this person doing? Things like, if two people are in an Internet cafe at the same time every day, that's an interesting coincidence, like what's going on there? And so I think that over time that's going to become even more. China has put in place this intense degree of control over the Internet. And in the 90s, people sort of thought that was not going to be possible. Bill Clinton had famously compared controlling the Internet to nailing Jello to the world. Well, they did it. They did it. And it's not that you can't get outside of the great firewall within China through VPNs, but it's hard. And VPNs are kind of spotty and people just don't do it. And so there's so much control and so much censorship inside China and propaganda by the party that the party's been extremely effective at controlling the information environment inside China. And now they're using technology to the same to physical space to control people's movements and where they go and what they do. And I Think technology will enable an unprecedented degree of authoritarian control. To what extent that allows this kind of lock in that you're concerned about? I don't know. But I certainly think the trend lines are very worrying in terms of enabling greater technological control. And oh by the way, a lot of technology is spreading outside of China to other countries as well. And we're starting to see the spread of this global spread of more techno authoritarianism. It's all bad. This is the challenges. A lot of these topics are just like, it's like scary things and like what's. And I, I think we're doing a great job of hiring the highlight. Like well what's the other side? Is there a positive side? And there are, but it's just like none of it makes you sleep easy at night.
- [149:10]
B
No, no, no. Let's talk a bit about the policy tools and governance ideas that could make kind of AI and autonomy safer. Some people want a global ban on autonomous weapons systems entirely. What do you think of that as a proposal?
- [149:27]
A
I mean I understand the sentiment. I think it's just not realistic. And right now we're on a path where there is no global regulation, there's no rules of the road for how militaries and corporate autonomy. I don't think that's a good position to be in. I think it's dangerous. I think a sort of anything goes approach is probably could take us in a dangerous place. So we'd like to see some guardrails put in place. But I think that they have to be things that we think militaries are actually going to get on board with. And none of the major military powers have said that they support ban. I think you might be able to envision more narrow, tailored kinds of restrictions that might be achievable. One is surrounding maintaining human control over nuclear weapons. And as we've talked about, another one that I think is actually kind of interesting and worth exploring would be a ban on anti personnel autonomous weapons. So it's sort of driving a distinction between those that what the military call anti material autonomous weapons, those that target physical objects, tanks, airplanes, radar, submarine ships versus targeting people directly. And I think that that's there might be traction there. You may have more possibility for a whole bunch of reasons. One is there it's just less militarily valuable to use autonomous weapons to go after people. I think there's probably a case to be made that you might need autonomous weapons in some kinds of really high intensity conflicts. If you have say a swarm of drones attacking enemy radars and their communications are jammed. There's just like, there's not a good way to execute that problem to take down this integrated air missile defense system without using autonomous weapons, weapons and keeping a human in the loop. Well, people don't move that fast. Outrunning bullets has not been effective since the age of the machine gun. And so I think you could make a case that going after people, you can keep humans in the loop. Now I think we could see on the battlefield in Ukraine the case for using, at least in military sense, anti personnel autonomous weapons. Because you do have a lot of drone jamming on the front lines of drones and people using drones to target individuals. But it's not as critical necessarily for militaries. And then I think that the ick factor is a lot higher when going after people. And it's not just that, like, oh, it feels uncomfortable. This really does matter in terms of getting public support. But it's not just that. I don't like that. I think you can make a very reasonable argument that if you are an enemy combatant or a civilian that is in a situation where autonomous weapons are targeting you and they're going after a physical object, you can escape being targeted, you can climb out of that tank and run away. You can't stop being a person. And so if the autonomous weapon goes awry or you want to surrender or it's having a mistake, I think that having weapons that sort of target people doesn't. It could be really risky and cause quite a bit of harm and it could take you on a path towards greater civilian harm. Maybe you have countries deploying these swarms of anti personnel drones and they're like, well, these people are all enemy combatants. And they're basing it on some algorithm that looks at their cell phone connections or their geolocation data or their social media. It says, well, we've labeled them that they're all affiliated at least greater civilian harm. I think there's a lot of ways in which you could say that that's much more troubling and that might be a place for countries to explore. So I think to me, I would focus more on what's achievable here. And I think that part of the challenges here, some of the people who have pushed for a ban on autonomous weapons are coming out of the humanitarian disarmament community where there were a lot of Successes in the 90s after the end of the Cold War with bans on landmines and cluster munitions. And we're in a very different geopolitical Environment now, particularly since Russia's invasion of Ukraine. And if your countries are now looking at landmines quite differently, certainly. And if you're looking at, okay, we're going to disarm, say Ukrainians and say they can't use these weapons that might help them in the war, you better have a really strong case to do that. So I think it's a much. You got to just factor in the political realities here.
- [153:47]
B
Yeah, that makes tons of sense. So that's kind of a broad band and then also narrow bands as policy ideas. Are there other types of proposals for major treaties aiming to govern autonomous weapons systems that you think have promise?
- [154:07]
A
I think that something that involves sort of rules of the road between nations involved in air and situations where their naval and air forces might be interacting in close proximity in crises could be very helpful here. During the Cold War, the U.S. and Soviet Union agreed to the incident sea agreement that helped sort of deconflict their forces so that they were less likely to get into some of these kinds of altercations that could be dangerous and destabilizing. It's something like an autonomous incidence agreement where countries might agree to sort of put some rules in the world of place of how are our autonomous systems going to behave? How can we communicate effectively to you what they're going to do? And one might criticize, well, these aren't going to hold in wartime. All the rules are going to come off. It's not designed to solve that problem. It's not designed to solve a wartime problem. It's designed to solve a peacetime problem in these militarized disputes where countries don't really want to necessarily go to war, but they're engaging in some level of brinkmanship. So that might be something worth exploring. I think there's value there. And I think the last thing would be maybe promulgating global norms about better safety and test and evaluation for autonomous systems, or particularly in weapon systems. A good. Because I don't think that accidents really benefit anyone here. And so if some country develops some autonomous weapon and then it has an accident and kills a bunch of people, I don't think that's sort of like in anyone's interest in sort of finding ways to improve the reliability of systems and reduce access, make countries more conscious of safety, I think beneficial. An analogy here would be some of the work that the US does in promulgating norms on legal weapons reviews. So the US and many other Western militaries do legal reviews of new weapons to ensure that they comply with the law of War with existing treaties and the Geneva Conventions. And not all countries do this, but the US and other countries have been very active in trying to sort of spread this knowledge to others and encourage other countries to do this. And I think there's merit in doing the same on test and evaluation for some of these AI enabled systems in the militaries.
- [156:27]
B
Nice. Yeah, I like that. That is a thing that a country that wanted to be safety oriented could do kind of unilaterally to reduce risk that doesn't require rivals to cooperate. Are there other things like that or is that kind of a particularly good example?
- [156:44]
A
I think you could argue that there are similar analogies in the risks surrounding loss of control of highly advanced AI systems. That for example, just having more awareness of those risks, demonstrating sort of leading labs demonstrating the protections that they're putting in place to be more safe to guard against agentic systems doing something squirrely and engaging in deceptive behavior or self exfiltrating, those are good norms to promulgate to others. Particularly as we see not only the frontier of AI advance, but of course the technology proliferates super, super rapidly. And so today's frontier systems is tomorrow's commodity system. And spreading those norms, similarly spreading norms around protections against biological hazards as AI becomes more capable, I think would be areas where there's just a lot of value in helping to ensure that other people are engaging in responsible behavior.
- [157:48]
B
Nice. Are there purely kind of technical ways to make AI use in warfare safer? So things like explainability requirements for any system used in targeting or early warning or fail safe mechanisms that kind of default to human control when sensors disagree.
- [158:07]
A
You can imagine lots of things you can put in place.
- [158:10]
B
Are there any you're particularly excited about?
- [158:13]
A
So a couple. I like the idea. So taking the analogy from stock trading and looking at things like flash crashes and trying to avoid a similar sort of scenario, I like the idea of having human circuit breakers in AI systems. So even if we end up in a world where maybe there are autonomous weapons in physical autonomous weapons or autonomous cyber weapons, that there's some bounds on their behavior such that if they go awry, there are limits in how bad it gets before a human has to take some positive action for things to continue. And so I think there's things that you would take to improve just reliability and reduce the risk of accidents. But I think we should assume that there will be accidents, bad things will happen, and then how do we put in place these boundaries in their behavior or human circuit breakers so things don't get too badly out of control. That's something that seems like something that I would like to see militaries do.
- [159:09]
B
Nice. Yeah, I guess just before we move on, are there any things in this kind of solutions or policy areas that you really want people to know about and consider that are maybe underrated in.
- [159:26]
A
The military space specifically or just broadly?
- [159:29]
B
I think military? Yeah.
- [159:31]
A
So there's this interesting question. So the US has a policy now maintaining human control over decisions relating to nuclear use. It's really uncertain what that means. I would love to see work inside the US military to better characterize. And some of that work might be classified and so public. That's fine because it's very sensitive information. But what does that mean? What's inbounds, what's out? What does sort of an acceptable use case look like? What is it not? And over time, we might get some rules and policies and practices in place to more accurately characterize what good uses of AI and automation are in the nuclear enterprise and what aren't. So maybe given an analogy here, there's this concept today, nuclear operations of dual phenomenology for early warning systems, that if we are looking at a potential missile launch against the United States, that we want two completely independent ways of sensing that. Missile launch. One could be, for example, satellites and another one could be radar systems that we have two totally independent ways of verifying it. Now, one could imagine something in the AI space of, okay, let's say I have some AI system churning through data like we were talking about earlier. It's, you know, coming up with some conclusion. Not only that we want some degree of auditability, explainability with the system of what it's doing, but also maybe have a completely independent way, a different algorithm trained on a different data set that's doing something similar and that I can compare those two and there might be some merit in really high risk applications in having something like that.
- [161:08]
B
Cool, cool. Okay, we have time for one last question. We haven't talked very much about your experience in the Army. I'm wondering if there are any kind of experiences you've had in that context that have informed how you think about autonomous weapons and what it means for us to be integrating AI more and more heavily into war.
- [161:36]
A
So one maybe incident, a personal example that comes to mind for me sometimes is an incident that, that I was in when I was an Army Ranger deployed in Afghanistan and I was part of a small recon team and we were out operating among the mountains and there were just three of us that had gone out on to patrol this sort of ridge along this line. And we saw an Afghan man approaching us along this ridgeline, coming in our general direction. And from a distance, we couldn't tell if, you know, he was armed or not. Maybe he had a weapon under his cloak. If he had a radio on him, we couldn't certainly see. Maybe there were others, like, nearby. Maybe he was scouting for somebody. Maybe he's just a goer. We. We couldn't tell. So he. We lost sight of him. And I was a little bit concerned that he might be sort of coming up behind us. So I went and maneuvered him to sure I could get eyes on him. And I ended up in a sort of this rock position where I was up above him looking down through this crack in the rocks. And he had his back to me, and I was pretty close, to be honest. I could hear him pretty clearly. And he was talking, and I couldn't tell, you know, I don't. I don't speak posture. I could say stop, and that was about it. So I couldn't tell what he was saying. And he didn't know if he was talking to some other people that might be nearby that were out of sight that I couldn't see, or he was talking on a radio video, for example, maybe relaying information and other fighters are going to come attack us. And we'd actually seen that exact scenario happen previously where somebody had come looking like they were herding goats as cover, but they had a radio on them and reporting information. And so I settled into a position with my sniper rifle that I was ready to shoot him if I saw that he had a weapon or there were other fighters. And I sort of gauged that he was an enemy combatant. And I watched him for a while, and I was looking for some sign, and sort of in my head, I was sort of, you know, weighing this decision, do I shoot this man? And then I heard him start singing, and that struck me. I just instantly relaxed because it struck me as, like, a bizarre thing to be doing. If he was an enemy fighter reporting information about it probably wasn't singing out information over the radio. And it just instantly relaxed. And I thought, you know, he's just a goat herder out here. He's talking to himself or his goats and he's singing. He's enjoying the view. And I watched him for a little bit longer and then ended up leaving. And I think about that sometimes when I think about the decisions that machines might make in warfare, because I was relying on kind of this broader contextual judgment of like, would that be a weird thing for a human to be doing in that context? And would a machine be able to pick up on that? And that sometimes that sort of broader understanding, the broader context and relying on judgment is things that AI doesn't necessarily do very good at. And in the big scheme of the war, that decision did not matter. It would not have changed the outcome either way in terms of the broader US campaign in Afghanistan. But it mattered a lot to him and it mattered to me. And so to me, I think about that when I think about the stakes of autonomous weapons, that people's lives are on the line here, and we gotta get these decisions right. And how do we find ways to use this technology that doesn't lose our humanity, that doesn't cause more suffering as a result?
- [165:08]
B
Yeah. Yeah. Thank you for sharing that. My guest today has been Paul Shari. Thank you so much for coming on.
- [165:14]
A
Thanks for having me. It's been a great discussion.