Loading summary
Transcript55 lines
- [00:00]
A
Mike Horowitz of Penn University, formerly with Biden's dod. We didn't get enough on Monday on autonomous weapons systems. This whole Iran war thing got in the way. So we both thought it would behoove the audience to do a little bit of a 101 on what these things are, how they kill people, and just how autonomous the world is in 2026 and perhaps beyond. Like take it away. So Mike, how would you characterize where the fear lies in the well meaning researcher or head of an AI lab who thinks their technology used for certain types of autonomy would be a bad direction to go go and maybe contrast that with how this stuff is used today in Ukraine and Iran.
- [00:59]
B
I think that the, the average sort of, maybe Silicon Valley AI safety researcher, AI safety researcher who's worried about autonomous war bots is probably worried about AI making the decision essentially about who lives and who dies and is, you know, thinks that that's a, like, that's some dystopia that they don't necessarily want, that they don't necessarily want any part of. And so get worried about the incorporation of AI into the like, pointy end of the sphere for militaries, especially when it comes to, you know, potentially selecting and engaging targets. What I think sometimes gets lost in the conversation is the substantial degree of autonomy that already exists in modern weapon systems, in that the US military and basically 40 militaries around the world have deployed autonomous weapon systems since the early 1980s. These are often automated systems that are, we're using essentially deterministic, good old fashioned AI, like more or less that are on ships like these enormous Gatling guns called the Phalanx that can operate by algorithm. And so if there are too many threats that are coming in, say too many missiles about to hit a ship, an operator can basically flip on the algorithm which can automatically target and hit those kind, can automatically target and hit those incoming threats. Or, and you also have things, you also have semi autonomous weapon systems that fall into the category of fire and forget munitions. Think about how a radar guided missile works. So you know, a pilot sees that, you know, believes that there's an adversary radar that's a legitimate target. They, you know, press the launch button. The, the radar guided missile fires when it go after going a certain distance, it turns on a seeker, it detects a radar, it goes in and it, and it destroys the radar. There's no human, human supervision or control of any kind after that, after that weapon is launched. And like, hey, maybe that radar's on top of a school Maybe that. Maybe that radar is on top of a hospital. And so that's the status quo, in some ways, of autonomy and weapon systems. And those kinds of technologies have been used since the 1980s. And we tend to think that they're way better than what came before, which was the, you know, like, area bombing, essentially, of World War. Of World War II. And so there's already a lot of autonomy in weapon systems, which then makes this conversation about what we don't want AI to do in the weapon space a lot harder because it can sometimes be challenging to talk about it without inadvertently, in some ways, wrapping in all of these existing kind of weapons, which we generally think are good, more or less, like, in the world where we support military action because they're both more effective and are more accurate, making things like civilian casualties generally less likely.
- [04:12]
A
I was reading to Command the Sky, the battle for air superiority over Germany, as well as Fire and Fury by Randall Hansen. And, like, people forget that those planes, when you drop the bombs, you would be, like, lucky to be within miles of the thing that you are trying to hit. So imagine doing the ayatollah, like, actually succeeding in doing the Ayatollah, like, compound explosion that we saw over the past weekend, like, would have caused tens of thousands of people to die, as opposed to, like, 50 or 100. So you would have.
- [04:53]
B
You would have dropped, like, tens of thousands of pounds of weapons that were, you know, from. From a couple dozen aircrafts that, yeah, like, hundreds or even thousands of people would have. And.
- [05:06]
A
And so, yeah, like, precision strike drones. All of this stuff just, like, tightens the radius of the thing that you end up exploding. And then even. Even with. With drones, like, what we saw with what Israel pulled off of, like, you know, you're going into specific windows and apartment complexes. So, anyways. All right, but let's. Mike, let's take the. Let's take the narrative forward from the. From the. From the 80s to the 2000 and 20s. And I think that's kind of getting a little closer to the contemporary ick factor around this stuff.
- [05:43]
B
So now a thing that is a doable do in the context of weapon systems is imagine a deterministic algorithm that is trained on a very exquisite data set. Like a data set, say, of Russian tanks or Chinese fighters or something very specific. You can now essentially train an algorithm that can go on board. On board, you know, some kind of weapon systems, you know, maybe a loitering munition and that kind of. And it can. It can launch, go to an area, turn on A seeker and then look for Russian tanks. And in some ways it can, it can then use an image classifier to say like, is that a Russian tank? No. All right, move on to the next image until it finds a Russian tank, at which point it will destroy a Russian tank. So you know, that's a weapon then, that is launched by a human who, who in theory then is, is, you know, trained in how the weapon works, understands its upsides and limitations, et cetera. And, but that weapon then is, is after it is launched, not just operates autonomously so you can't recall it, you know, like a radar guided missile or something from the 80s, but is now using an algorithm as the basis for, for destroying a target. And you see like early days for this a little bit in the Ukraine context. There's so much jamming and electronic warfare in the Ukraine context. And so Ukrainian FPV pilots, you know, their one way attack drones were getting jammed constantly by the Russians and they're coming up with different concepts of operation to try to get around that. Or they're working on like connecting fiber optic cables that could stretch for like kilometers to be able to hit a target. Like what if somebody cuts the cable? So there are, you know, now some Ukrainian weapons that essentially have last mile autonomy where you, if there, if there's jamming that occurs in the last kilometer and the, and the data link goes away, that weapon then again trained on an algorithm that maybe has a target library of targets it's allowed to hit, then can still continue on to the target and hit the target. And that then becomes an absolute necessity for militaries fighting in electronic warfare, heavy environments in trying to operate if you don't have access to say satellites or your equipment gets jammed.
- [08:22]
A
So what is the, why don't you give the generous reading of the anthropic case where they say that or actually let's not forget like that.
- [08:35]
B
So like I have no, I have no, I actually have no problem with the, with like what anthropic said Thursday night. I think it's we, you know, you can cut this. But like I actually have no problem with what Anthropic. Like I, I think that they do everybody a disservice when they use the phrase fully automous weapons because nobody knows what they mean and then everybody picks it up because like it's anthropic. And it would be better if like everybody used similar, like, like, like words mean things and it'd be like helpful if people like use the same terminology and we're talking about the same stuff. I think their position is actually like very reasonable, which is LLMs aren't ready.
- [09:06]
A
What are reasonable concerns model providers should have as their models kind of get into the ecosystem that's spinning up weapons like the drones that have the last mile capability.
- [09:22]
B
So I think part of this depends on what you want the role of the human to be in the context of using weapons and what the, and what you, in some ways what you're most concerned about. And one of the things that I think gets lost in the conversation about autonomous weapon systems, at least for the United States, is that for the United States, the United States has a policy on autonomy and weapon systems, but also has both domestic legal obligations and international humanitarian law treaty obligations that essentially require that there is human responsibility and accountability for the use of force. And that's a requirement that exists whether you're talking about a bow and arrow, whether you're talking about a radar guided, you know, missile, or whether you're talking about an autonomous weapon system. And so the, when you start from there, I think things start to fall into place a little bit. The issue is that if you don't start from there and what you are worried about is, you know, AI systems making decisions about whether somebody is a lawful combatant on the battlefield and like turning into, and turning into killbots, and you think that that will happen without a trained commander making the choice to deploy that system in a context that they believe is legal and where they've received legal approval, then you think about it a little bit differently. But if you start from the premise that there's always human responsibility and accountability for the use of force, and you believe, although people might have different views on this, that the Pentagon will follow its own rules and the law on this, on these issues with regard to the use of the use of force, then in some ways that it becomes a question of when we think autonomous weapon systems of different types of are ready for prime time. By ready for prime time, I mean systems that are as good as or better than existing weapon systems, since nobody wants their weapons to work more than militaries. And because weapons that are not reliable or, and aren't safe by definition don't work well. And that means military commanders and operators where the use of these things will determine whether they live or die, are strongly incentivized to get it right, essentially. And the, what this means is you, the incentive for the military has been to incorporate autonomy in ways that they can validate works well and works better. You know, again, as well as, or better than existing, existing weapon systems. And so once you start from that proposition, in some ways you're already starting from a place where some of the worst case fears then about what, what anthropic calls fully autonomous weapons and what the Pentagon calls autonomous weapon systems. Some of those concerns, I think then become, become less, less, less of a, like a broader moral or moral and ethical issue and more of a question of can the weapon do the thing it's supposed to do.
- [12:46]
A
All right, so let's spend a little time walking through the sort of legal strictures that require humans to be involved in this. I mean, it's a nice directive you wrote, Mike, but like, there have been a lot of Biden era over regulations which have been thrown away over the past few, or I guess the past 15 months. So what else besides that directive are kind of keeping, keeping humans involved in, in these sorts of decisions?
- [13:16]
B
The thing that keeps humans involved in decisions on the battlefield actually has nothing to do with the Pentagon's directive on autonomy and weapon systems. The, the Pentagon's policy on autonomy and weapon systems is about the process for developing and fielding semi autonomous and autonomous weapon systems. So the precision guided weapons of today, the autonomous weapons that have been used for decades, and then what future autonomous weapons might look like. The whether a human is actually involved and involved in a substantive way in making the decision about the use of force is actually governed by separate Pentagon policy. Like the Pentagon has guidance on the use of force, you know, written by the lawyers that say when you're allowed to use force or when you're allowed to use force or not. And that's connected to treaty obligations under international humanitarian law where commanders and operators have to ensure that uses of force are, you know, that they're, you know, like proportionality and distinction and all of those, like good legal, all, all of those good legal requirements. And so this is not a case where it is, you know, necessarily. Like, if what you're worried about is like the robot deciding this is not the case where it's like Biden era policy standing between us and the Kilbots. It's a broader architecture of law and regulations surrounding the use of force that again, isn't even specific to AI you can think about Pentagon.
- [14:53]
A
The question is when you have a Secretary of War who, who's telling commanders to kill everybody when they see a boat, like, does any. And there's no like, Inspector General thing that exists anymore, like, who cares? Like, does it. If you're thinking about selling something into the, you know, into the system, like like how much can you hold your hat on, on, on any of that stuff?
- [15:20]
B
That's not an AI issue. Then that's like a Pentagon following. That's like the, that's like a Pentagon following the law issue. And so the like one can believe that and like, you know, and like, you know, muster some evidence for it. But like if you believe that that's not a reason why like autonomous weapon systems are good or bad or like different AI uses are good or bad, like that would be a reason, you know, in theory not to do business with the Pentagon at all and, or why. And that could be true, you know, like much more broadly, much more broadly beyond that. So like take the point. But that's not a, that is that that would be a reason why a company might choose not to do business with the Pentagon, not an argument about autonomous weapon systems in particular.
- [16:06]
A
All right, so back to autonomous weapon systems. This seems like an inevitable force of history that you know, we're going to go from one mile to two miles to five miles to one person controlling one drone versus five drones versus 10 versus 50. I mean like, you know, is there like what are the reasonable, you know, if the drones are actually better than the sleep deprived, you know, like on their fifth cigarette human being in picking out the, the targets. Like what, what are the legitimate ethical concerns around the war bots? You know, it's, it's this if we're, if the analogy, if the actual analogy is like Waymo versus a human driver.
- [17:00]
B
Yeah. I mean in that case I think the, the ethical arguments against autonomous weapon systems are not that persuasive. Frankly. If what we are talking about, that is a weapon system where there is still human responsibility and accountability because there's still the human that makes the decision to use it, that would be responsible and accountable. At the end of the day if something goes wrong and you are telling me that that system will be more effective at hitting a target than you know, a specific target that you, that you wish to hit than the, you know, like 18 year old on their fifth cigarette or like something like that, then the, that seems to me like a better weapon system. And one again like in the world where we, where we're like yay, you're using military force, that that makes sense then for the military to acquire where, where this gets tricky sometimes is the sort of objection to these on, on sort of ethical or moral grounds gets, can get conflated sometimes with what are, I think like, can be pretty legitimate concerns about how like whether they would actually work and that's part of what, you know, Anthropic's beef with the Pentagon was getting at was their belief that large language models like Claude are not ready for prime time when it comes to incorporation into autonomous, into autonomous weapon systems. As a side note, I wish that Anthropic would not call them fully autonomous weapons because that is not a term of art. And it's not clear exactly what they mean by that relative to the like, government entity that they're dealing with which talks about autonomous weapon systems, which like, have a clear meaning in, in policy. But like, putting that aside, the, the I think Anthropic's actually probably correct about the limits of large language models in powering autonomous weapon systems, which is also why the Pentagon isn't doing it right now and wasn't talking about doing it. You know, like, one of the many reasons why this like whole blow up between Anthropic and the Pentagon was so needless.
- [19:07]
A
We had a fun riff on Second Breakfast where I think it was Justin was talking about how usually when you're selling into the Pentagon, you're promising the moon and you sell them like a piece of cheese. And that is kind of like the general dynamic that you're getting. But in Anthropic there, you actually have a provider who's like, no, like we cannot do all the things that you imagine we can. This is maybe like more of a moral philosophy question, but like, is that, I don't know, just what's your sense of kind of going to them and saying, hey, you know, our thing isn't ready for prime time. We're not sure we trust you to not use it in a way which is like going to be unsafe for service members.
- [19:54]
B
This is part of the issue. It's part of the, part of the, part of the issue I think is that the. It is not been clear and from what I know, like not even true that the Pentagon was trying to get Anthropic to develop autonomous weapon systems, like fueled by LLMs, that this was a theoretical concern about like a possible future ask from the Pentagon. And you know, Anthropic even said, I think it was like last Thursday that like they think autonomous weapons systems actually make sense. They just think their technology isn't ready for primetime on this. And further said they were willing to work with the Pentagon to make it happen. And, and so the, you know, which is part of why the Pentagon's concerns in some ways are more philosophical than, than anything else. But like part of the challenge here is so all Military systems, and especially weapon systems go through a testing and evaluation process, which is how the military figures out that whether a system is reliable or not. Because again, as I said before, if it's not reliable, even if it somehow got through the approval process, it's not like commanders and operators want to use it like they want to use stuff that works. But it's been, it's challenging to figure out how testing and evaluation for these large language models should work, especially in the, in like a safety critical kind of use case, like a potential weapon system. And so in some ways there's like, work on the back end that needs to occur to validate these systems, like, in addition to like whatever advances in the systems themselves that Anthropic thinks needs to happen.
- [21:34]
A
What do you think about this cloud versus edge distinction that bubbled up this past week?
- [21:41]
B
I think the. I'm actually reasonably sympathetic to a cloud versus edge distinction as important, but that's because I am very anchored on the Pentagon's definition of an autonomous weapon system. Remember, that definition is the Pentagon's definition of an autonomous weapon system is a weapon system that after activation, can select and engage targets without human intervention. And unless that system, unless there's human oversight of that system continuously, which generally there would not be because then there's a data link that someone could hack or jam. Like by definition there is no cloud access. So the, you know, effective autonomous weapon systems in general aren't going to have data links and cloud access. So if you have a system that only operates through the cloud, then it almost by definition can't be used to, to power an autonomous weapon system. You could use it to do lots of other military operational kinds of things related to the battlefield about, you know, like, like planning military operations, directing things. You know, like there are lots of things you could do with that system, but if it's cloud based, that means it can't operate on the edge in an autonomous weapon system, which means it can't be used for an autonomous weapon system. So I'm actually reasonably sympathetic to that distinction, at least at a, at a high level and think it's a reasonable case to make.
- [23:14]
A
So the idea being if you just think doing autonomous weapon systems is icky, but you still want to help out with command and control and logistics and back office stuff. Being in the cloud API access provision space is like a relatively neat way to make that distinction.
- [23:34]
B
It, it based on where the technology is right now. And, and also keep in mind, I mean, I think Anthropic is correct that this tech just that LLMs aren't ready for prime time in terms of incorporation into autonomous weapon systems. And so even if you could somehow put them on the edge, the. It's not clear like it'd be. It's tough for me to imagine those kinds of systems surviving the Pentagon's own review process for, for those kinds of things. And so which it's very similar, frankly, to what Anthropic said. But I, I think that the, a way to protect that from happening, even if you are. Even though again, I'm, I'm saying, like, you shouldn't worry about it that much because it's not just that it wouldn't be a good idea, like wouldn't make it through Pentagon testing processes, but if, if your system can't operate on the edge, it can't be in an autonomous weapon system, like, period.
- [24:27]
A
Dot, let's talk about command and control with all this stuff. I mean, it just. AI is smarter than me, man. Like, how can it not be?
- [24:39]
B
Come on.
- [24:39]
A
Don't no nobody humans anymore.
- [24:42]
B
Nobody puts Jordan in a corner.
- [24:43]
A
You're very smart, but not as smart as these AIs. They know so much.
- [24:47]
B
I mean, I think the, I think the question asked is like, you know, it's like, I mean, I think like the real conversation to have in some ways is like, okay, like, Mike, you can like, policy and legal. Your, you can like, policy your way to like, persuading me that an autonomous weapon that like autonomous weapon systems are like, basically, are basically fine.
- [25:07]
A
I, I, I see what you're.
- [25:08]
B
Because they only happen with human. Because there's always human responsible, like, whatever. Like. All right, concede your point, but like, is what we're really worried about that like, AI is going to like, do all the really scary stuff before that?
- [25:19]
A
Yeah. Okay. So, um, I mean, I think with the autonomous weapons, like the, the, the, the range of how bad things can get seems to be like, relatively narrow. Like a thing can blow up a school or it can like, turn around by accident and decide to like, you know, blow up the base it came up from. But once you start handing over more and more range, not reigns not to just like a drone or. Actually, no, it is kind of scary. So let's, let's talk about. Okay, so we've got like the rogue, the rogue drone swarm seems like really not great, but one, one or two levels up of like the rogue, like brigade or battalion or like, you know, Combatant Command.
- [26:13]
B
Here, here's why. Let me, let me say two things here. Like, here's I am not that worried about the rogue drone swarm. And the reason I'm not worried about the rogue drone swarm in some ways is because I believe that militaries when it comes to weapon systems tend to be relatively like little c. Conservative institutions. And because of the incentive structures that I laid out before and some of the challenges in developing, like, good testing and evaluation procedures, I think the notion of a, like, super unreliable drone swarm that, like, causes like, major chaos, like, out there, out there in the world, like, maybe there are militaries we should worry about that for. But like, even in the current context, I am not, like, terribly worried about that in the context of the United States. The United States has been way too slow, frankly, in incorporating AI into military systems rather than, rather than too, rather than too quick. But the. But if, but, but again, that's also because even in the case of the drone swarm, the drone storm would have to be activated by a responsible human who would be accountable if something went wrong. So there's, there's a responsible chain of human command and control from the decision to use force, which would be by an informed person, all the way to the varieties of points of impact. And military is actually pretty good at accountability when stuff goes wrong. But sorry, if we want to talk about, if we want to talk, like
- [27:44]
A
at the operation, I think, I think, I think the, the thing is though, right, like when I've been using Claude codes for the past few weeks and it asked me do I want to let it do. And I say, do a thing, and it says, okay, well, you need permissions. Press 2 to give permissions. I, I've been pressing a lot of two over the past few weeks. And then at some point I googled how can I stop pressing 2? And then the Internet said, there's a setting you can put into Claude that says dangerously accept permissions. And now I don't have to press 2 all the time, and it just does the thing I want to do. And it's been totally fine so far. Way more efficient, way more effective, less time. You know, it's, it's, it's, it's succeeding for me. Right? So, like, I just, I mean, maybe, maybe all we have to hold our hat on, Mike, is this idea that these are these slow and bureau and bureaucratic institutions that still have, like, paper trails and, and human beings with, like, legal liability if they screw stuff up as well as, like, you know, the moral weight of killing the wrong person. But there does seem to be something kind of inevitable about more and more parts of Your work, you're just handing over to a thing and like, I don't know, I mean maybe our maybe like test and evaluation can catch it. But when it's just like you know, when you have, when you have a Secretary of War saying I want you to use AI and this is just like the broad ethos like that's going to come across, that's going to come in contact with, you know, it may not always be the sort of inertia that, that wins out here and you could end up getting to a point where you know, we end up handing over too much.
- [29:32]
B
Totally. No, I, I, I think the, like I am far in some ways like I believe that the self interest of commanders and operators and having things that work will generally will make some of the worst case scenarios that people like worry about surrounding autonomous weapon systems a lot, a lot less likely and that it, and that it's possible to capture in some ways a lot of that upside without some of those worst case scenarios. If you want something to worry about more, I think it is this question of operational decision making and the idea that you know, we already have now like tools like Maven smart system which are again like not in and of itself like a large language model. And that is a thing I think people need to get their heads around more that the like risk here is not necessarily connected to whether it's a large language model or not. But the, you have a platform like that that's designed to essentially be a dashboard for commanders out in the field, especially like going up the chain to the combatant command level to try to understand sort of like what the world looks like around them, like what is, like what are the enemy's forces looks like, what is information from public, you know, from open source looks like, what information from classified sources look like, aggregating those things together, interpreting that information in a way that may, may may generate increasingly specific recommendations to commanders for courses of action that the commanders would then take and be responsible and accountable for. And the risks then I think are in some ways more prosaic than we sometimes talk about. Like one risk then is automation bias where people trust algorithms more than they should given the reliability of said, of said algorithm. The There are all sorts essentially of like behavioral decision making biases that can then get triggered if you're just offloading more and more like cognitive judgment to the machine. And you can, the military militaries can develop standard operating procedures and training to try to hedge against that as much as possible. But there's, there's a point at which you've just like given a lot then to the. Given a lot then to the machine. That's a real thing to worry about. So.
- [32:01]
A
But is that okay? So like, what is the. I mean, we're really getting, we're really getting some maven smart system. We're getting, we're getting our money's worth over the past week or so. I imagine this stuff was literally made for like figuring out how to, you know, degrade an air defense system and find missile launchers and blow them up or whatever, what have you. And you can totally see like a anodyne way where this technology helps you find out who needs to fly where and who needs to be bombed with what faster and more effectively than people doing it with pen and paper. So I don't know what's there to be concerned about. This seems actually not super scary just like that humans could have done it better or there's some, you know, bug and they end up bombing the wrong thing or.
- [32:57]
B
Yeah, like, it's like a joke in the Pentagon, the number of like staff officers that it takes to like generate operational plans and like generate, operate, you know, like, like idea, like concepts for, you know, for, for senior military leaders and in decision and the. If you can like automate a bunch of that process and, and especially the process of generating alternatives and like in helping like estimate some of those risks. As long as there are responsible humans involved, like monitoring those feeds and like ensuring that they're, that they're basically right, there are a ton of like real efficiencies you can gain from this that are be like super useful. The problem is if you start using these systems and, but like don't really understand how they operate and don't have people that are really well trained on them and like that's when you, you generate like a bunch of that kind of risk. And it might be frankly that we are in a transitional period where you have these technologies coming online that can be increasingly powerful and a workforce that wasn't raised on these technologies and so might be more inclined potentially towards automation bias or some kind of, you know, like some of those kinds of things where when, you know, like Gen Z is like when you have Gen Z3 stars or something and they've been raised on this tech and understand its limits intuitively because they've been working with it all their lives, maybe some of those risks become. Maybe some of those risks become a little bit less likely and then the question becomes like, how do we get from here to there? But none of that is about autonomous weapon systems, which is like the thing that's been in the news.
- [34:38]
A
So this is, so, I think this is like. Yeah, so the worry about the command and control stuff from your perspective is like, okay, if we were at a 6 in effectiveness and the promise of doing this well gets us to an eight and a half, then if we trusted too much, then maybe we'll be stuck at a seven or a six and a half, but.
- [35:00]
B
Or like really bad accidents happen because we make mistakes.
- [35:03]
A
What. But these are not like doomsday scenarios, Mike. Right?
- [35:07]
B
No, they're not. But like, I'm not, I'm not a doomsday guy. Like the, the. No, I just mean like the, I
- [35:14]
A
think that just not exist. Like, what, what's there to be worried about? You're kind of saying like, nothing, it'll be all fine. I'm not saying 10% scenario.
- [35:24]
B
What I, what I am essentially saying is that I, I tend to think that the process that a military like the United States has for evaluating capabilities, whether we are talking about edge capabilities on the battlefield or operational level capabilities, it is not perfect. There are accidents with every military system. There are mistakes with every military system. Like those are inevitable, and those will be inevitable in the age of AI as well. What I'm, what I'm suggesting is that that process generally works pretty well and that the military is pretty resistant to fielding stuff if they can't know whether it works or not. Which actually will screen out a lot of the things that people are worried about the most and make the, the like, worst case kinds of accidents a little bit, a little bit less, A little bit less likely. And so I think on balance, I'm again like more worried about the Pentagon going too slowly with these technologies rather than, rather than, too quickly. But that's also because of how much, One, how much autonomy I think is already in existing systems, and two, all of the mistakes and errors that humans make today.
- [36:45]
A
So I think maybe then the most. The concern I can sell you into the most is like the, what are we calling this? The autonomy bias?
- [36:56]
B
Automation bias?
- [36:57]
A
Yeah, the automation bias at the strategic level, where if you, if you believe that the, that the tactical and operational people have enough kind of like bureaucracy in their way that they're not going to adopt something which is like totally not ready for prime time because it's their bot, it's their physical bodies that are on the line. Like having the sort of, you know, super cool AI simulation of how like, oh, this, like, this like war that we could start would just end in an amazing fashion because like of course Maven will have like done all the nine dimensional chess to figure out how it's going to work out for you. That in fact that is the thing which is, which is maybe more scary than Skynet is just like a president or a secretary or you know, someone on the Joint Chiefs being like, okay, the AIs just got this like and have a higher confidence that wars would work out.
- [37:59]
B
Yeah, I think like senior decision makers uninformed about AI trusting AI tools too much and like guiding their decisions. If you want like a, like, how does stuff like really go awry? It's like less because the pointy less because of the AI at the pointy end of the spear. It's like much more at the strategic level. And that I think is an absolutely legitimate concern because all of those like standard operating procedures and training and incentives that I was talking about don't necessarily apply to senior leaders. So it's like you really want to worry. Like that's where I would worry.
- [38:32]
A
So it's, it's the President and it's a Defense Secretary just like chatting with their mill.AI, scheming up who to bomb
- [38:43]
B
next, which is the, who even knows one.
- [38:49]
A
Okay. Any, anything else, Mike? I, I think this was helpful. I, I, I'm feeling a little bit better.
- [38:56]
B
You should feel a little bit better. But most of all it would be helpful if everybody just used the same words. Or at least that would make my life better if everybody used the same terminology to talk about this stuff. Autonomous weapon systems, AI decision support systems, like automation bias. If we could all use the same words for the things that we're talking about, it would be like, it'd be like easier at least to like have some of these debates.
- [39:19]
A
And this is, I think this conversation is why I think the, the thing, aside from the personality stuff, the thing that really tripped up anthropic and the government was on the domestic surveillance side because they're like with this autonomous web, like you can, you can fudge a solution, especially if you're willing to do like 95% of it already. But I would imagine, I would imagine that Anthropic was just like, look, we don't want to be involved in finding illegal immigrants. And they were like fuck you. This is, you know, we, we were duly elected to do this stuff. Why aren't you, you know, letting us, letting us rip. So anyways, but I'm sure that will come out in the next I totally
- [40:09]
B
reasonable to be totally reasonable to be worried about. AI enabled mass surveillance. I don't worry about it most from the Pentagon. I worry about it more from, like, other agencies. Totally legit.
- [40:19]
A
All right, let's call it there. Thanks for dropping by, Mike.
- [40:23]
B
Thanks for having me.
- [40:24]
A
Jackson Hewitt handles your taxes and your stress. Inhale our no surprise price of 1.49 or less. Exhale Paying more for complicated taxes you won't inhale. New tax law knowledge. Exhale. Missing out on your biggest refund. Certainly not card. Don't miss paying 149 or less. Rest easy. Jackson Hewitt Scott Taxes guaranteed limited time offer for new clients on the draw turns Participating locations in late turns@jackson hewitt.com 149.