← The Lawfare Podcast
The Lawfare Podcast
Lawfare Daily: The Double Black Box: Ashley Deeks on National Security AI
00:55:44
Loading summary
Transcript90 lines
- [00:01]
Lawfare Host
The following podcast contains advertising to access an ad free version of the Lawfare Podcast. Become a material supporter of lawfare@patreon.com lawfare that's patreon.com Lawfair also check out Lawfare's other podcast offerings, Rational Security Chatter, Lawfare, no Bull, and the Aftermath.
- [00:34]
Alan Rosenstein
Hey, it's Adam Grant from Work Life, a podcast from TED. This episode is brought to you by freshworks. Freshworks believes that complexity is the enemy of efficiency. So stop wrestling with bloated, expensive service software that takes forever to implement an update where ROI is someday, not today. You've been overcharged and underserved for way too long. Uncomplicate with Fresh Service for IT and Fresh Desk for customer support. And with Freshworks AI assisted service software, you'll work smarter, not harder. Freshworks uncomplicates. Learn more@freshworks.com.
- [01:13]
Ashley Deeks
Out here, we feel things the sore calves that lead to epic.
- [01:17]
Alan Rosenstein
Views, the cool waterfall mist during a hot hike, and the breeze that hits just right at the summit.
- [01:23]
Ashley Deeks
But hey, don't just listen to us. Experience it for yourself.
- [01:27]
Alan Rosenstein
Alltrails makes it easy to discover the.
- [01:29]
Ashley Deeks
Best of the outdoors, with more than.
- [01:31]
Alan Rosenstein
450,000 trails around the world, points of.
- [01:34]
Ashley Deeks
Interest along the trail, and offline maps for always on navigation.
- [01:38]
Alan Rosenstein
Download the free app today and find.
- [01:39]
Ashley Deeks
Your next outdoor adventure. If you're in the Justice Department and you're working on a case that involves classified information about some nuclear secrets, you will be cleared for that type of information, but not about some counterterrorism activity that's happening around the world. So it's absolutely right that different agencies and different actors inside those agencies experience the national security black box quite differently.
- [02:11]
Alan Rosenstein
It's the lawfare podcast. I'm Alan Rosenstein, associate professor of law at the University of Minnesota and a senior editor at Lawfare. I'm joined by Ashley Deeks, The Class of 1948 professor of Scholarly research and law at the University of Virginia.
- [02:25]
Ashley Deeks
It just goes to argue that it will be very hard, based on my current understanding of how these systems are built, to get us to a place where we could use an autonomous system confidently or confident that it complied with the laws of armed conflict. But we should never say never.
- [02:46]
Alan Rosenstein
Today we're discussing her new book, the Double Black Box, which tackles the challenge of democratic accountability when opaque artificial intelligence systems are used inside the already secret world of national security. So before we get into talking about the Double Black Box and what it is, I want to first just start by laying the groundwork for the sorts of AI national security systems we're talking about. The book touches on several high risk scenarios, lethal autonomous weapons, AI driven cyber operations, all of which could escalate, intentionally or otherwise. And so, of all the potential applications of AI in the national security space, which one? Or you can pick two or three if you don't have a favorite one. Which one worries you the most and why?
- [03:33]
Ashley Deeks
So I think of high risk national security AI as primarily AI tools that can inflict lethal harm on people or lead to their detention. I tend to think of those as among the most significant. But of course, there are also concerns in, for example, the cyber context, where the initial activity is not lethal or does not result in detention, but but does create effectively a flash crash that can lead to conflict, which ultimately would produce casualties, but does not itself create those casualties. So those are some of the concepts I have in mind. As your question suggests, though, there are lots of places in which I think the national security community is going to ultimately adopt AI that are not just in those buckets. So the intelligence community using it for all sorts of analysis for counterintelligence activities, homeland security using it, and so on. So a lot of the oxygen in the room does get sucked up with the lethal autonomous weapons systems conversation, and that's an important one. But I wanted to think a little bit more broadly about other sets of tools in the national security space.
- [04:58]
Alan Rosenstein
You used, as one of your case studies the idea of AI enabled cyber tools. And I'm curious why you chose that. Is it because that's where the systems are most developed, or because it's just an example? You can imagine the autonomous nature of them being most effective? I'm just curious why you chose that one as your case study.
- [05:16]
Ashley Deeks
I guess I'd say maybe two reasons. The first is based on my research, it did seem as though that was the place where we might see the earliest adoption of basically autonomous AI tools out in the wild, maybe used defensively, maybe used offensively. And so that seemed like a realistic place to start to do a case study. A second reason is that I think it's a pretty useful way to demonstrate how Congress can quickly fall out of the picture, even in a place in which we think under the Constitution, Congress should have a significant role in deciding when to resort to force. I thought that some of the things that I think would play out in a cyber autonomy context would show how quickly Congress could lose control of its role.
- [06:16]
Alan Rosenstein
One point you make, and one of the assumptions that you are explicit about in the book, is that what's Sometimes called artificial general intelligence, which is this idea that perhaps one day AI systems will be in. Definitions differ, but kind of a rough definition is that an AI system will be as good as kind of an above average human at all tasks, or at least all tasks that don't involve opposable thumbs. Though perhaps with robotics we'll get there pretty quickly. And you make an assumption that AGI is not, not imminent. And so I'm curious both why you make that assumption and then also what happens if that assumption is incorrect. Because at the very least, my reading of, or at least my following of industry suggests that it's honestly a 5050 of whether AGI is coming in the next few years. And even some of the more skeptical folks think that within 10 years we'll have something pretty close to it, at least functionally.
- [07:13]
Ashley Deeks
Mm. Yeah, it's a good question. And it's one that our friend Jack Goldsmith also asked me when I had him read an earlier draft. So, first of all, I'm not a superforecaster and I'm not an employee of a high tech company working on AI, so I didn't want to get too far out in front on trying to predict what a world looks like in which either the US or China or another country achieves AI, as your question suggests. There are really smart people that come out on both sides of this debate. So let's assume that I shouldn't have assumed that. Let's assume that we do achieve AGI. I guess the question is, is that still a double black box problem? Are there still things in this book that I think would help inform the problem and maybe some of the solutions? And I think the answer to that is probably yes. I want to do more thinking about it, but to me it still sounds like there are parts of the use of AGI in national security that produce a similar and worse, I would say, a more magnified problem. If we're talking about, for example, full autonomy in armed conflict, with the humans kind of doing very little to execute the strategies, the tactics and so on. Maybe you also are talking about government systems that are very capable of accurately forecasting foreign policy developments, systems that are engaging in extensive and very competent collection and analysis of intelligence, it might become clear quickly to the public that some government has achieved AGI in some settings. Right. That would make it less of a national security black box. But they probably won't know precisely what those systems can do, assuming that the government wants to keep part of that classified and all of the AI black box piece of things Those issues worsen. So I think, right, we still have a situation in which Congress will be ill suited, at least on current trends, to understand the tech. They're too under resourced, too politically fragmented to easily regulate what has emerged. The executive agencies that are controlling the AGI are hugely empowered at the expense of non technology agencies. Courts are really out of the loop and we can talk about this more. But I do think some of my prescriptions would still be pretty highly relevant, at least as I'm envisioning AGI where you have allies who can still play an important checking role. Just because you have an AGI system doesn't mean you should be using it everywhere. I do think Congress would need to pull up its boots and impose framework regulations. I think corporate whistleblowers who are inside the companies that are helping develop this AGI could help flag abuses and so on. And the general public, having learned that we've entered this AGI world, will need to vote with their feet and their voices. So I think a magnified problem. There may be other things though, that I haven't fully thought through that would change it and change my analysis in more significant ways, for what it's worth.
- [10:36]
Alan Rosenstein
And the reason that part of the book jumped out at me is because, if anything, I would think that the closer we get to AGI and then artificial superintelligence, which comes later, the more relevant your analysis becomes on both of those dimensions.
- [10:53]
Ashley Deeks
Yeah, just so I think one reason why I originally made that assumption is because I start out by trying to recognize that there are pressures on two sides here, right? There are pressures in wanting to make sure that our government is acting in a coherent, lawful way. And there are pressures as other countries are developing these systems for us to do more and to do more faster. And so I do feel like we're kind of at an equilibrium now in kind of a narrow AI world, but that those pressures will potentially that balance will change if one of our adversaries achieves AGI. So that was, I think, why I built that assumption in without trying to kind of preordain how the diagnoses and the solutions would attach to an AGI world.
- [11:45]
Alan Rosenstein
And I want to come back to that question of external pressure from an adversary, which in almost any scenario would be China later in this conversation. But we've done a bunch of table setting on the technology here. I just want to then now go to your analysis of it. And to start then with the central metaphor of the book, the double black box. So you've mentioned it a little Bit already, But just to be very explicit, what are the two boxes? Why are they black, and why is one inside the other?
- [12:15]
Ashley Deeks
Okay, so the first black box is the national security black box. I think, as many listeners will know, the government today, our government and other democracies, make a lot of their national security decisions in secret. They do that because they have to use sensitive intelligence, sensitive technologies, and our governments have to keep those tools secret from adversaries, which also means they have to keep it secret from its own public. Right. And so this makes it more difficult to oversee the executive than it does in areas where the government is not operating behind the veil of classification. So I think it's fair to say that our national security agencies largely operate inside of a national security black box, where there are lots of things happening, and it is hard for those outside the box to have a sense of what's happening, why it's happening, who's doing what.
- [13:14]
Alan Rosenstein
And also, I mean, just to kind of deepen that further, it's hard to see from outside the box, but it's also hard to see within the box as well. I mean, we were both in our government lives, national security lawyers or adjacent lawyers, and I was in that box, and I had no idea what was happening at the proverbial two feet in front of me. It was very dark. Is that a fair description? I want to see how much I can squeeze out of this metaphor.
- [13:39]
Ashley Deeks
Yeah. So I do think, and I say in the book, that people will experience the national security black box. We'll stick with that for a sec. At different levels of opacity. And so they will also experience the overall double black box in different levels as well. But you're right. So there are some people inside the government who are super users who have access to anything they want to see because of their levels of seniority or the type of ability they have or the assignments they have or so on. Not many people in the government have that. It's a kind of need to know basis. So if you're in the Justice Department and you're working on a case that involves classified information about some nuclear secrets, you will be cleared for that type of information, but not about some counterterrorism activity that's happening around the world. So it's absolutely right that different agencies and different actors inside those agencies experience the national security black box quite differently.
- [14:45]
Alan Rosenstein
Okay, so that's box number one.
- [14:47]
Ashley Deeks
That's box number one. And I will just say in passing, the reason why people have written about government secrecy for years is because the Fact of the government doing things in secret challenges what I call and others have called our public law values. In other words, it is harder for us to tell when the government is operating in secret whether it is doing things lawfully, whether it is being effective and efficient about what it's doing. It is harder to require officials to justify their decisions because there are fewer people to challenge them. And it is sometimes harder to hold them accountable for the decisions they've made, partly because the public may not know about it.
- [15:33]
Alan Rosenstein
Well, let's stay on those public law values actually, for a second. So those are all we might call sort of procedural values. And we're both lawyers, which is another way of saying we're weird robots who are obsessed with procedure. I feel like that's the kind of good definition for humans that have been turned into procedure loving robots.
- [15:51]
Ashley Deeks
Except I don't teach civil procedure.
- [15:53]
Alan Rosenstein
Well, I think we're all procedure teachers, whether we teach civil procedure or not. Why focus on procedural values like this? Why not substantive values?
- [16:04]
Ashley Deeks
Yeah. So I am trying to write a book that has some traction in reality.
- [16:11]
Alan Rosenstein
That'S so refreshing from an academic.
- [16:15]
Ashley Deeks
And I may have failed, but I tried and I took the view that it was more likely in the era of political polarization that we're in today to find common ground among people who differ about what the government should be doing, maybe agreement around how they should be doing it. So I like to think that these values are relatively neutral in that whether you are on the left, right, center of the political spectrum, you want our government to be acting lawfully, you want it to be acting efficiently, and you want to be able to hold your officials accountable for the choices that they've made. I felt like you could achieve more consensus around that than around a debate about whether the government should develop a facial recognition software tool that it was going to use against some group overseas who may be freedom fighters, maybe terrorists, or whether we should be interfering in foreign government elections using AI tools that the substantive fights are hard and real. But maybe we could achieve more consensus if we focused on these, as you say, more procedural values.
- [17:35]
Alan Rosenstein
So just to make the point explicit, we may not all agree on whether we should build Skynet, but we should all agree that if we're going to build Skynet, it should be through some reasonably democratic, reasonably transparent process in which the people doing it have some idea of what they're doing.
- [17:49]
Ashley Deeks
Correct. That's fair.
- [17:51]
Alan Rosenstein
Excellent. Okay, so box one. Okay, let's talk about box two.
- [17:55]
Ashley Deeks
Okay, so box two is the AI black box. And again I did not come up with this term. I think it has been around for a while, especially as the machine learning and AI tools have become more and more sophisticated. The idea is that they are basically using neural nets to develop predictions, identify things, make recommendations. And they're doing it in a way that is hard for even the computer scientists who have designed the systems to understand how it is that the systems are reaching their conclusions. So there's data in, there's training in, there's something comes out the other side and almost no one understands. It's almost impossible to understand what has happened in the middle. So that is a kind of paradigmatic black box. So inside the national security black box, we are dropping a series of AI black boxes that are going to inform, advise and operationalize national security activity.
- [19:02]
Lawfare Host
Right now the headlines are chock full of data breaches and regulatory rollbacks, making us all vulnerable. But you can do something about it. DeleteMe is here to make it easy, quick and safe to remove your personal data online. We all want an easier way to deal with data breaches and I'm telling you you should get Deleteme. The fact is we're all at risk. How many times have you gotten an email or a letter saying your data has been stolen? It's unsettling. But the good news is Delete Me can help. It's not just a one time service. You give them the information you want to remove from the public Internet. And Delete Me is always working for you, constantly monitoring and removing the personal information you don't want out there. I am somebody with an active public presence. I antagonize people, I bother people. But privacy is important to me. I don't want people knowing things about my private life, about my family, about all sorts of things where they can kind of mess with me. And I keep hearing about these data breaches in the news. A recent Coinbase data breach, for example. A hacker obtained names, addresses, phone numbers and the last four digits of users Social Security numbers plus masked bank account numbers. They vacuum it up, they create a searchable profile and they sell it to whoever wants it. Thankfully, DeleteMe can help you preserve it from those sites and keep your information from falling into the wrong hands. So take control of your data and keep your private life private. By signing up for Delete Me now at a special discount for our listeners, you get 20% off your delete Me plan when you go to join deleteme.com lawfair20 and use the promo code lawfair20 at checkout. The only way to get 20% off is to go to joindeleteme.com lawfair20 and enter the promo code lawfare20 at checkout. That's joindeleteme.com LawFair20 code lawfare20.
- [21:28]
Kevin Frazier
You know how we all have that one friend we go to with all our financial questions? Well, meet Experian, your bff, as in Big Financial Friend. I'm telling you, Experian is gonna be your Go to app when it comes to taking control of your finances. Experian not only has a bunch of ways to help you save, but lets you check your FICO score, matches you with credit cards and a bunch more things you'd expect from a big financial friend. Just download the Experian app and get started for free. Trust me, you'll be up to speed with this BFF in no time. Based on FICO Score 8 model offers an approval not guaranteed Terms apply Offers not available in all states. See experian.com for details.
- [22:06]
Alan Rosenstein
Ever feel like you're carrying something heavy.
- [22:08]
Ashley Deeks
And don't know where to put it down?
- [22:09]
Alan Rosenstein
Or wonder what on earth you're supposed to do when you just can't seem to cope? I'm Hesu Jo, a licensed therapist with years of experience providing individual and family therapy, and I've teamed up with BetterHelp to create mind if We Talk? A podcast to demystify what therapy's really about. In each episode, you'll hear guests talk about struggles we all face, like living.
- [22:28]
Ashley Deeks
With grief or managing anger.
- [22:30]
Alan Rosenstein
Then we break it all down with a fellow mental health professional to give you actionable tips you can apply to your own life. Follow and listen to Mind if We Talk on Apple Podcasts, Spotify, Amazon Music, or wherever you get your podcasts. And don't forget, your happiness matters.
- [22:46]
Kevin Frazier
Ready to level up your everyday Quince makes premium essentials without the premium price tag. From quality clothing and stylish accessories to travel staples and high end home goods, Quince has it all. And by partnering directly with top artisans and ethical factories, Quince delivers high quality at half the cost of similar brands. Shop elevated essentials without the markup at quints. Go to quinte.com LevelUp for free shipping and 365 day returns. Quince.com LevelUp.
- [23:19]
Alan Rosenstein
So I want to stay on this point for a bit because I think it's important to kind of understand maybe some at a technical level what's going on. So here's my understanding. Correct me if I'm wrong. The way that these machine learning Systems in particular operate, and machine learning being one subtype of artificial intelligence. It's that you have a bunch of basically parameters that are taking input and turning them into output. But the way this is trained, you can't look at any particular parameter, any particular quote unquote, neuron inside and say, that's the neuron that lights up when the machine thinks that this target in the drone footage is a terrorist. And so I can look at the after action and say, well, that neuron lit up. So therefore I can say this is why the machine behaved that way. Rather the information is sort of diffuse around that. Right, Yep. So that is, and that has been one of the central challenges in machine learning. There is, and you do talk about this in the book, a subfield of machine learning pursued by some labs more than others. I think Anthropic in particular has been really focused on this, on what's called interpretability or explainability. And this is the idea of figuring out through a bunch of clever computer science and mathematical and statistical techniques why a machine acted the way it did. So is your read of this that it's just never going to be enough? Or it might work, but it might not work. I'm curious how much of your account hinges on explainability not being solved. It'll probably never be solved, but reasonably well dealt with in the next few years.
- [24:59]
Ashley Deeks
It is, I would say, an important piece of the book, but I think there are enough other points in play that even if we improve, if we make the AI black box somewhat less opaque, that's all for the good. But I think a lot of the concerns described in the book do still survive, as I understand it. I mean, you're absolutely right that explainable AI is one helpful way to try to narrow the size of the. The AI black box that's inside this double black box. People have been working on this for years and years and have taken a range of approaches. My understanding is the more you make a system explainable, the less effective it is. And so there's some tension between wanting to improve its explainability or interpret interpretability and improving its capacity. And my sense, again, not as a computer scientist, is that as we are trying to shift to Meta's super intelligent systems, it's going to be even harder to really produce quality, explainable AI. The other reason why I think it's going to persist as a problem for a while is the system can produce explanations that may just simply be made up. They may kind of make intuitive sense to a human, but it's still Very hard for us to tell if that's the real basis, the real reason why the system recommended X instead of Y or thought it was a cat instead.
- [26:40]
Alan Rosenstein
Of a dog, which is true for what it's worth of humans as well. Right? I mean, the amount of explanations and ad hoc rationalizations that are false, and perhaps even worse, not just false, but are unknowingly false to the person making them, is also quite, quite large.
- [26:56]
Ashley Deeks
Yeah, it's true. It's true. I mean, this is always, this is a sort of persistent question of, you know, are computer systems actually less transparent than humans? But, you know, you can bring the Secretary of the treasury up in front of Congress and ask a bunch of questions and ask a bunch more questions if you feel like the answers aren't reasonable and try to kind of triangulate to a truth in a way that it might be harder to do with Skynet system in the witness box. Maybe, Maybe.
- [27:29]
Alan Rosenstein
Okay, so we have the matryoshka doll of boxes here. So let's talk about why that's a problem for the public law values that you've articulated. And so I want to kind of go through the different traditional checks that we have in the national security space, Congress, the courts within the executive branch, the private actors and international actors, and just go through and sort of have you identify what is the problem and then, you know, what, what are your, some of your proposals? I think one thing, one of the things I loved about the book is how practical it is, like how many specific proposals there are. And one thing that you emphasize, and I think rightly so, is that there's not one silver bullet. There's not going to be some, you know, as the Internet likes to tell us, one, one weird trick that's going to solve all of this. It's going to kind of be like all interesting problems cobble together from a bunch of different dimensions. So, so let's start with, let's start with Congress. Okay, so, so maybe one way of asking this is why do you start with Congress? Why is Congress the key here? I mean, I, you know, I think that you don't have to be professional politics watcher to have not a lot of faith in Congress's ability to do anything these days, let alone check another branch of government. But why historically has it been so important and even reasonably effective sometimes in national security oversight? And how is it going to do that or how can it continue to do something like that in this double black box age?
- [28:51]
Ashley Deeks
So I start with Congress because I do think that they are the or it Congress is a they not an it that they are the most traditional powerful historical check on the Executive in the national security space. They have the power of the purse, they have the power to legislate, they have the power to investigate, convene hearings, hold up presidential nominees and so on. So when they are operating at full speed, they can be pretty effective. And we can think back to the 1970s, where in the wake of massive problems with what the Executive had been doing in secret, Congress is able to convene the Church Committee, produce a credible report of 1000 pages plus. House did the same thing. They produced the Foreign Intelligence Surveillance act, the War Powers Resolution, starting to get after the COVID action problems. So maybe this is a part of the book that is less realistic than I would like, but they are the kind of traditional, most powerful counterweight to what the Executive is doing in secret. The four public law values again that I focus on are legality, competence, accountability, and requirement of justifying your action. And so if you think about a couple of the key framework statutes that Congress has enacted where it worries that the Executive is making important key decisions, high stakes foreign policy and national security decisions, they have developed the Foreign Intelligence Surveillance act and a covert action statute. Right. The concern is that you would have actors inside the Executive branch authorizing covert actions that end up taking the US Into a conflict or just producing really bad outcomes, maybe without the knowledge of the President and so on. So they've enacted a statute that says, no, that's not how we're going to do it, President has to make a finding, and so on. So it struck me that maybe a potential way to deal with the double black box here on Congress's side is to use those statutes as a kind of model to require the President himself or herself to sign off on very high risk uses of national security AI acting on recommendations from a range of national security agencies and their lawyers, which is kind of like how the President signs off on covert actions today. The statute could require the President to notify Congress of those approvals and keep Congress currently and fully involved, for example, so that at least makes sure that there is another set of actors that can check the legality competence and create some accountability for the choices being made to deploy these systems.
- [31:51]
Alan Rosenstein
And I guess it's going to be a theme as we go sort of from institution to institution, because I could ask this, I think, for the courts as well and for the inspectors general and various lawyers and offices of goodness. Right? As I think just a great phrase that's sometimes Used talk about kind of intra executive branch watchdogs. This all seems helpful in piercing the outer black box, in piercing the national security black box, but not the AI black box itself. Right? Because again, assuming that these things remain unexplainable, which is why I think this, this, it is an interesting empirical question. Sure, Congress can have the President sign off, Congress can have drag such and such in front of the Select Committee on Intelligence and yell at them about what Skynet is doing, but at the end of the day, if the answer is, well, we're not entirely sure we think it does okay, its aerator is pretty low. No, I can't tell you why it authorized the Hellfire missile in that instance versus another instance. Is there anything in this oversight regimes that can get at the inner black box, not just the outer black box? And maybe the answer is no, and that's just a trade off.
- [33:07]
Ashley Deeks
I would say probably not directly, but I can think of a couple of ways indirectly it could be relevant. So if Congress regularly asked the actors who were coming in front of it to describe to them a particular high risk use of national security AI, okay, well, what was the system's explanation for why it chose this target as opposed to that target? The officials then have an ex ante incentive to make sure that their systems produce some form of explanation so they can report to their congressional overseers because they know those are the kinds of questions that Congress is going to ask. The other thing is it's sort of a blunter tool. But if Congress convenes a hearing and the assistant secretary of whoever comes up to explain it and the person has to say yes, we've used the system several times. Sometimes it's done a really important, it's made a very accurate recommendation. We were able to find this missile site in the mountains and a couple of times it made a mistake and we bombed a village, I don't think DoD would necessarily do that. But just hypothetically, Congress can say you have to stop using this system until it reaches a particular level of accuracy or competence or confidence. And we're going to legislate to turn off the power of the purse until you fix that. So it's prodding those who are most able to look inside the AI black box to do better. But as we already talked about, it may be ultimately impossible to do that. And you can only use sort of proxies to figure out the internal workings of the system.
- [34:58]
Alan Rosenstein
So I actually think this is a good opportunity to ask my China question. And the China question is this. Let's stick with Congress for a second because they are of course the sort of the most political and I don't mean that in a pejorative way, just the most ambiently political for obvious reasons of the branches. Do you think there will ever be realistic pushback or oversight of these AI tools in the context of what feels to me sometimes to be the only source of bipartisan consensus in Washington, which is that we're in a second Cold War or however you want to call it with China, that one of the battlefields in that Cold War is going to be AI and that the metaphorical occasional village that we accidentally bomb. It's a bummer. Obviously we'd rather not, but certainly we're not going to do anything to take the foot off the gas pedal as long as we're in this China AI race.
- [36:04]
Ashley Deeks
Right. So I agree with you that if there's one thing that Washington can basically get behind its consensus against China. And Kristin Eichensior and I wrote a piece called Frictionless Government where we kind of take the China example as a launching point for the piece. You might well be right. I talk about a range of different actors and bodies who could provide some pushback and checks who would not be captured by that consensus. But I do think there are. In a scenario in which China is leaping ahead on AI tools, including national security AI tools, Congress may not choose to act. I do talk about a non traditional Czech of foreign allies who we often need for things and who can often see behind our sort of veil of secrecy we can sometimes see behind theirs. I'm thinking about NATO allies, Aukus Five Eyes.
- [37:11]
Alan Rosenstein
So it's because we're cooperating, not because they're spying on us though.
- [37:13]
Ashley Deeks
No, no, I'm sorry. Voluntarily saying we share a military alliance. You need to understand how our systems work. We want to understand how yours work. And I can imagine even where there's a lot of consensus in Washington about needing to push forward as fast as possible to combat China, that some of our allies might be cautious about that, including for legal reasons. And their caution can potentially infuse some caution in us or make us think hard about some of the choices we're making if we don't want to lose their cooperation in places X and y. One other thing to add here, and I think this is true, maybe it's too Pollyanna ish, but other people, I think, like Michelle Flournoy and Avril Haines have made this point, is there may be a real kind of political and economic advantage to the United States in making sure that our systems do comply with public law values that we will then have the kind of gold standard system that our own military is comfortable using. Right. They don't want to use systems that turn on them, for example. And foreign countries may in a choice between do I want to buy China's aggressive, maybe less tested, maybe less verified AI or the US killer AI with.
- [38:42]
Alan Rosenstein
Socialist characteristics is how I think about it.
- [38:45]
Ashley Deeks
Splittest characteristics.
- [38:47]
Alan Rosenstein
Nice.
- [38:47]
Ashley Deeks
So that's good that maybe there is a longer term advantage there if we can keep our eye on that ball.
- [38:54]
Alan Rosenstein
Let's talk about the courts for a little bit. Obviously the courts play a role in overseeing national security, especially the intelligence context. You have the Foreign Intelligence Surveillance court. At the same time, though, I'm reminded of this great quip that Justice Kagan made in Forget if It was the NetChoice argument, or maybe it was the year before in the Gonzalez v. Google argument, that we're not the nine greatest experts on the Internet. And so I do wonder, given already how difficult it is for the courts for various institutional reasons, to really police the executive, how much harder is it when you're asking a bunch of generalist lawyers to start peering again not just into the first black box of national security, but into that, that central black box of the AI systems themselves?
- [39:47]
Ashley Deeks
Yeah, I don't put a lot of weight on or hope in the courts having a significant role here, I guess with maybe three minor caveats. So the first is I wrote a piece a while ago that suggested that courts may help drive actors towards explainable AI as they start to face cases, maybe involving experts, expert witnesses, or where the government's done something, maybe in the Social Security Administration using an algorithm. And the court says, well, why did the system make that recommendation? The person says, I don't know. Court says, that's not good enough. So it could be that by asking certain questions and demanding certain evidence in court, the judges might drive the sort of computer science world towards greater explainability in these pockets of cases that come up. Second is the FISC naturally lives behind inside the national security black box. They conceivably will confront cases in which the FBI or the CIA comes seeking probable cause. Justice Department would do it, of course, directly, but maybe the intel community indirectly. And saying we've used AI tools, we think this is probable cause. And the Fisk says say more. So just kind of testing that at a relatively. I don't think you have to be a really high tech person to ask questions in that space. And I think the Fisk judges are pretty sophisticated about surveillance at this point. And then the third question I have in my mind is, are we going to see some public cases involving non national security AI that produce judgments that shed some light on how the government decides that it should operate behind the veil of secrecy in a kind of parallel situation? So just to be more specific, maybe there's something comes out about facial recognition in a public case in federal court and the holding is such that it gives people inside the government a little bit of pause to say, well, we've kind of been doing something different from that. Maybe we should, maybe we have some legal obligations under the fourth Amendment that weren't what we thought they were. So a little bit of slop over potentially in those sort of public cases causing the government to rethink what it's doing in its, in its national security work.
- [42:25]
Kevin Frazier
AI policy debates these days move faster.
- [42:28]
Ashley Deeks
Than D.C. tourists chasing shade in the middle of July.
- [42:32]
Kevin Frazier
If you're struggling to keep up with.
- [42:33]
Ashley Deeks
AI law and regulation, lawfare and the University of Texas School of Law have your back with a new podcast, Scaling Laws.
- [42:41]
Alan Rosenstein
In the show, we dig into the questions that are keeping folks like Sam Altman awake that are driving legislative policy and steering emerging tech law. I'm Alan Rosenstein, lawfare's research director and a law professor at the University of Minnesota.
- [42:53]
Kevin Frazier
And I'm Kevin Frazier, the AI Innovation.
- [42:55]
Ashley Deeks
And Law fellow at Texas Law and a senior editor at lawfare.
- [42:59]
Alan Rosenstein
We've lined up guests you won't want to miss, so find us on Apple, Spotify, YouTube, wherever you get your podcasts. Subscribe and don't miss. What's next? Let's go into the executive branch itself. I think one of the, one of my favorite parts of your book, and maybe I'm biased because I was one of these people, I mean, obviously very junior, but this was at least my experience was that the government, the executive branch itself, has all sorts of internal checks. And obviously you can debate about how effective they are and not effective. And obviously we're living in an interesting constitutional political moment. But speaking generally, obviously the internal self regulation is very important. Now within that you describe a potential power shift away from lawyers and toward the engineers who build these systems. And so I'm curious how this kind of code is law, to quote the great tech Internet scholar Lessig, how the sort of code is law reality changes the role of the government lawyer. What does the government lawyer need? Are we all going to have to get master's in computer science, or does the government lawyer just become less, less important? In this world where so much relies on how the technology actually works and who is building that technology.
- [44:20]
Ashley Deeks
So I think this is a little bit of a black box. Maybe this is our third black box.
- [44:24]
Alan Rosenstein
Oh, my God.
- [44:24]
Ashley Deeks
I know.
- [44:26]
Alan Rosenstein
It becomes a black hole at some point, I think once you hit any more black boxes.
- [44:33]
Ashley Deeks
Dark matter. Dark matter. So just to second your point, that there are of course, a lot of checks inside the executive branch that are not always appreciated from the outside. And that do, I think, really, really important work. And some of that is interagency lawyers group. Some of it is just the tug of different agencies sharing the same broader goal and a lot of different views as to how to get there. Each of their agencies thinks there's a better way to do it. And the idea of lawyers kind of falling a step behind as these systems, as we shift more and more towards relying on AI for all sorts of things, some of which are national security decision making and some of which are just kind of lower level inputs into it, it's hard for me to fully know where the lawyers can and should be inserting themselves into this process. Ideally it would be right at the front end, right? It would be somebody saying, look, we think we're going to develop X kind of system for the State Department that will be heavily infused with AI. And we want to have, before we actually start building the system, we want to sit down with the lawyer and understand the basic law that attaches to this kind of scenario. What kinds of outputs would be most useful in helping the policymakers and the lawyers get where they want to go and so on. I'm skeptical that that is how it works right now, especially because a lot of these tools are being acquired from private companies. But I do think that we kind of have to get ourselves to that space as lawyers if we want to maintain some relevance and not just be on cleanup duty of saying like, oh, well, we've already acquired it, we've already started to use it. Now we see there are these problems. How can we clean that up? It would be far preferable, I think, to do it on the front end and just at a more macro level. I do think there's going to be a power shift towards the agencies that are heavily using these tools and away from the agencies that use them a lot less. I think of that means more power to DOD and CIA, less power to Treasury, Justice, State, but that's a hypothesis. I don't have direct evidence of that.
- [47:03]
Alan Rosenstein
Just to jump off the last point, why are you making that cut? Right. I'm actually just curious just for a second to explore Your intuition that DoD and CIA probably add NSA to that is going to be on the sort of pro AI side. But treasury does a ton of analytical work. You know, State, State can do a lot of, you know, can make use of a lot of AI in analyzing, you know, foreign relations and open source intelligence. I mean, to me the, the nature of AI, especially as we get closer to AGI, is that it can do any cognitive task. And it's not obvious to me that, you know, the tasks that DoD does are more or less cognitive than the tasks that DOJ does. Now there may be cultural differences and maybe that's what you're getting at, but it just jumped out at me that the way you bucketed those agencies was not sort of intuitive to me.
- [47:54]
Ashley Deeks
Yep. So I guess I'm thinking of at first cut which agencies are going to use, develop faster, the kind of higher risk AI. And that I think was the CIA, dod, NSA bucket.
- [48:10]
Alan Rosenstein
The ones that blow stuff up.
- [48:12]
Ashley Deeks
Yeah. And do other. Right. Other intelligence related activity. Although of course treasury and State and justice all have intel capacities as well. But I think also the cultural point, if we took a slice right now to see how far along DOD is on thinking about AI, how to purchase it, when to use it, what the rules of the road are, I think DoD is probably further down that road than state. DoD has issued a policy on this. IC issued a series of questions on this that its users and developers should be asking five years ago. I don't think that State and Treasury and justice are thinking as aggressively, actively about this. But I totally agree with you that there can be really important uses for, for all of these agencies and I would urge them to pursue them so that in part it's starting to infuse their culture more so that they can kind of keep up with their counterparts in other agencies.
- [49:14]
Alan Rosenstein
Let's stick from the perspective of a lawyer for a second. And maybe this is just sort of professional vanity, but I still think it's, it's helpful because I can imagine a world in which I'm a lawyer and I'm very worried that the folks in my building are using AI that I can't understand.
- [49:27]
Ashley Deeks
Fair.
- [49:28]
Alan Rosenstein
I can also imagine a world in which I'm a lawyer and I am relieved that people are using AI because at the end of the day what I want is legality. I want systems that follow whatever rules that I have come up with or that I believe are operative and it's not obvious to me. And here we Briefly touched on our conversation about, well, humans are black boxes too. It's not obvious to me that AI is necessarily more inscrutable than humans are. I mean, at the very least I can train an AI system in a much more direct way than I can a human system. And when you look at the history of predictions that AI can't do such and such because it's too difficult and amorphous, those predictions have generally not fared well as we've continued development. Just to give an example, we have self driving cars. Now obviously there are lots of technical impediments, but they work. You can go to San Francisco and drive a Waymo and they are at least an order of magnitude safer than human drivers. It seems to me that if you just amassed enough data, and that data could be, for example, laws of war targeting data, you could train an AI model to be at least as and potentially better and more importantly improving in a way that a human targeter might not be. So I can imagine if I was a lawyer, not just preferring an AI system rather than Bob right, who I can train as much as I want, but God knows what he's thinking inside his head. And I might also push it a step further, be worried about over cautious use of AI. Just again, to go back to the self driving car analogy, one might be worried that we focus so much on one bad Waymo accident that we delay the rollout and then a bunch of people die because the alternative to Waymo isn't not driving, it's humans driving and humans are horrible drivers. So I just kind of curious love for you to respond to that thought.
- [51:27]
Ashley Deeks
So the argument that you've just made I think is in large part the argument that the US government makes for why we don't need a new treaty to regulate lethal autonomous weapon systems there. I think the argument is that there is existing law on the books, laws of armed conflict, some key principles, distinction, proportionality, precautions, and it could well be in the short to medium term that the AI systems will be more effective in complying with the laws than, and produce fewer civilian casualties than humans would not subject to being tired, not subject to having seen their friend die, and so on. So I mean, I think the US government's argument here is a reasonable one, that we don't know whether the systems could in fact get to that point. And I think what you're saying is there's reason to think maybe they can. I have thought a bit about this idea of coding the law of Armed Conflict notice. Of course this is just A slice of the relevant law that we would want to think about. Right. There are lots and lots of constitutional, international, statutory laws that would be relevant to the kinds of programs we've been talking about. But let's just take the laws of armed conflict. Someone named Lisa Shea and a couple of her colleagues, she, I think is or was a professor at West Point, did this experiment where she got three groups of coders to code what's a very highly determinant law, so a speed limit. She took 52 computer programmers, put them into three groups, and then asked them to encode a speed limit and determine violations based on real world driving data. She told one group to implement the letter of the law, one group to implement the intent of the law, and a third group, I think, gave very specifically crafted specifications on which to base their program. And the three groups produced wildly different numbers of tickets on the same group of drivers. And the whole point is to show how many decisions have to be made during this process that you wouldn't necessarily think of. So it depended whether the groups decided to treat repeat offenses within some number of miles as one violation or two, or whether the duration of the violation mattered. So three seconds of speeding versus three minutes of speeding, did they both get tickets? Did only one, whether to take into account weather conditions. I think it just goes to argue that it will be very hard, based on my current understanding of how these systems are built, to get us to a place where we could, you know, use an autonomous system confidently or confident that it complied with the laws of armed conflict. But we, we should never say never on this. Right? And, and I think the way people have thought about this particular problem is it might be that there are different. You can sort of geofence the use of these things. And so you would feel more confident that there weren't civilians in the space where the system was operating, for example. But if the larger point is that there may be ways to bring these systems closer to code them in ways that bring them closer to public law values, then I agree with that and I think that would be great. And I think most lawyers would and should agree with that as a goal.
- [54:57]
Alan Rosenstein
So I want to finish by talking about the international dimension to all of this, because I think one of my favorite parts of the book comes at the end when you talk about how to think about international cooperation and competition in this domain. And you note that often the discussion immediately jumps to some analogy based on nuclear non proliferation treaties. And you know, if we're really worried about killer robots, maybe, you know, we should treat them like we do nukes. And obviously nuclear non proliferation isn't perfect, but it's done pretty well. But you argue that that's a, that's a bad analogy and that a better analogy is how we deal with cyber weapons and cyber threats, which is to say not great. And so you just unpack, if you would, why the nuclear analogy doesn't work, why the cyber analogy works better and kind of what your, what your outlook is for the possibility of cooperation. And, and again, I will just again cite the, the looming China effect, which I just think is always such an important political reality here.
- [56:06]
Ashley Deeks
Yeah, well, and thanks to you, you and I wrote a paper for you and you were the lawfare editor of it. That ended up becoming this part of the chapter. So a belated thanks to you for your inputs on that. So the idea is basically once we recognize this challenge of the double black box, you might immediately wonder, well, are there things we could do on the international plane to shrink the size of the double black box, that is to take certain uses of AI, certain systems off the table ex ante, which would make our domestic box smaller. And you're right. So I think there is some work to be done on the international plane. But I think it's important not to oversell how successful those discussions are going to be in kind of limiting where states go with this. And it was striking when I started thinking about this that the key analogy, as you said, was to nuclear weapons. And so the argument goes, if the Soviet Union and the United States were able to reach a number of agreements about restricting the use of nuclear weapons, size, location verification, then surely those types of states, Russia, US, maybe China, et cetera, should be able to come together and regulate what seems to be a system or series of systems that could pose just as serious a risk to the world. And for a range of reasons that analogy seemed misplaced, partly because nuclear weapons are not dual use systems. Largely they are hard to make, built by governments, they are things you can count AI, not that at all, sort of the opposite of that. And that the better analogy it seems is like cyber. And indeed at some point cyber and AI may overlay each other in terms of these tools. So I think we've seen pretty modest and pretty non linear progress in how we have gone about trying to develop cyber norms internationally. We've used a couple of different buckets of tools and I think we will probably see the same use of those buckets in the AI space. In fact, I think we've already started to see Some of those same uses of the buckets. So first, I think there is some level of broad multilateral agreement that states should apply existing international law to these tools. That may mean laws of armed conflict, it may mean. You said bellum, it may mean human rights law. Debates linger about how exactly to apply international out of these tools, but there's a kind of broad conceptual agreement. I think there's also been an effort to develop in both the cyber setting and in the AI setting, some somewhat new, pretty vague, non binding norms among a wide group of states. For cyber, it's in the UN Group of government experts. For AI, it's in the Convention on Certain Conventional Weapons forum. I think in both cases we'll see these work in minilateral coalitions. By that I mean groups like NATO or the Five Eyes to develop more specific norms to be more concrete about how we should engage in testing and verification and senior level approval and so on. But that will happen in private. I think we'll see states make unilateral statements of policy. We have seen a lot of that in cyber and I think we'll see some of that in AI as well. We might see sanctions on bad AI actors. We've seen sanctions on bad cyber actors. I can't imagine there'd be a reason not to do that on misuses of AI systems too. And then finally we've used criminal law on the margins in cyber and maybe we would see that in misuses of AI AI too. But I think your final comment before turning this over to me was about China. And I think at bottom, what's driving this inability to really develop a robust kind of series of regulations on this is just a deep mistrust at this point of really important players in this space, right? So the P5 and Israel and Iran and North Korea, these other players that have a lot of these tools or looking to build these tools are not in a place right now to sit down and have a serious conversation and trust each other that they will comply.
- [60:48]
Alan Rosenstein
I think that's a good place to leave it. Ashley, congrats on a really excellent and timely book and thanks for coming on the podcast to talk about it.
- [60:55]
Ashley Deeks
Thanks so much for having me. Enjoyed the conversation.
- [61:01]
Alan Rosenstein
The Lawfare podcast is produced in cooperation with the Brookings Institution. You can get ad free versions of this and other Lawfare podcasts by becoming a Lawfare material supporter at our website lawfaremedia.org support. You'll also get access to special events and other content available only to our supporters. Please rate and review us wherever you get your podcasts. Look out for our other podcasts including Rational Security, Allies, the Aftermath and Escalation. Our latest Lawfare Presents podcast series about the War Ukraine. Check out our written work@lawfairmedia.org this podcast is edited by Jen Patya. Our theme song is from Alibi Music. As always, thanks for listening.
- [61:46]
Kevin Frazier
You know how we all have that one friend we go to with all our financial questions? Well, meet Experian, your bff. As in Big Financial Friend. I'm telling you, Experian is gonna be your go to app when it comes to taking control of your finances. Experian not only has a bunch of ways to help you save, but lets you check your FICO score. Matches you with credit cards and a bunch more things you'd expect from a big financial friend. Just download the Experian app and get started for free. Trust me, you'll be up to speed with this BFF in no time. Based on FICO Score 8 model offers an approval not guaranteed Terms apply. Offers not available in all states. See experian.com for details.