Loading summary
Transcript117 lines
- [00:00]
Joe Weisenthal
Running a business means dealing with a lot of overly complicated Software, and most CRMs tend to follow the same pattern. They're packed with endless features, you'll never use interfaces that feel clunky, and teams end up spending way too much time just trying to find basic information. Today's sponsor, pipedrive is a simple CRM tool designed for small and medium businesses. Pipedrive brings you entire sales processes into one dashboard, giving you a crystal clear, complete view of sales processes and customer information. Designed to help teams stay in control and close more deals faster. It's it all centers around the visual sales pipeline, where you can see every deal, what stage it's in, and what needs to happen next. Since everything is in one platform, pipedrive is designed to unite your team, keep track of sales tasks and stay on top of your leads. Switch to a CRM built by Salespeople for Salespeople and join the over 100,000 companies already using Pipedrive right now. You'll get a 30 day free trial. No credit card or payment needed. Just head to pipedrive.comsimpleCRM to get started. That's pipedrive.comsimpleCRm so there's a lot of
- [01:01]
IBM Sponsor Voice
noise about AI, but time's too tight for more promises, so let's talk about results. At IBM, we work with our employees to integrate technology right into the systems they need. Now a global workforce of 300,000 can use AI to fill their HR questions, resolving 94% of common questions, not noise. Proof of how we can help companies get smarter by putting AI where it actually pays off, deep in the work that moves the business. Lets create smarter business IBM this podcast
- [01:31]
Wise Sponsor Voice
is brought to you by wise, the Smarter way to manage your money internationally. If you're getting a headache from juggling different currencies and different bank accounts in different countries, there's a better way to receive money in the currency you need without the slow transfer times or hidden fees. Meet Wise, the savvy way to handle your money internationally. Hold balances in up to 40 currencies with the mid market exchange rate on every conversion. Whether you're receiving payments from tenants abroad, earning as a digital nomad, or converting dividends from your international the WISE Multi Currency account is for you. Be Smart, Get Wise. Download the Wise app today or visit wise.com Terms and Conditions apply.
- [02:11]
Adobe/Cincinnati/4imprint/Sonesta Sponsor Voice
Bloomberg Audio Studios Podcasts Radio News.
- [02:26]
Joe Weisenthal
Hello and welcome to another episode of the Odd Lots Podcast. I'm Joe Weisenth. Tracy Alloway so Tracy, we're recording this March 24th and of course almost all of our episodes Lately have been about the war in Iran. But what's interesting, or what's a little weird is that just prior to the war, literally days or maybe hours, the biggest story in the world was actually about defense and, you know, the dod.
- [02:52]
Tracy Alloway
That's right. So you are referring to Anthropic.
- [02:55]
Joe Weisenthal
Yeah, yeah.
- [02:56]
Tracy Alloway
And it's disagreement, to put it mildly. The Department of War.
- [03:00]
Joe Weisenthal
Yeah, exactly. This was the biggest story going right up on the eve of the war in Iran. And of course, obviously, there was this contract and anthropic technology was used by the Defense Department. So it was not a disagreement about the use of AI per se in war, but the question of the degree to which AI could be used for autonomous weapons systems on their own without a human in the loop.
- [03:24]
Tracy Alloway
And surveillance.
- [03:24]
Joe Weisenthal
That was surveillance and surveillance. This was another key element.
- [03:27]
Tracy Alloway
But you're right. So we've heard this expression, autonomous weapons pop up more and more, especially in recent days, and I have a lot of questions over what exactly that same means, because my impression is the US Military certainly has been using AI for some time, and so we're really talking about degrees here of autonomy. Right. And so, yes, if you. If you think about an autonomous weapon, I think your mind could go fully Terminator. And, you know, there's like a murder robot out there that's making its own decisions on which people or places to target. And then you get levels below that. Right. Where AI is kind of helping humans to come up with strategic decisions.
- [04:08]
Joe Weisenthal
Right. So if there is a missile coming and you have a missile defense system, I don't think you want a human in the loop. Is like, okay, here are the coordinates, X, Y, Z that we think it's going to hit. At this point in time, we think the missile will be here. Are you cool with firing it? I think everyone's probably okay with that level of autonomy, but. But I have a feeling that. To your point. Exactly. A lot of this discussion, and maybe it's core to what Anthropic and the Department of Defense were disagreeing on, I have a feeling a lot of this is going to come down to definitions. My guess is that there is not one shared agreement of this is an autonomous weapon system and this one is not.
- [04:48]
Tracy Alloway
Absolutely. And of course, there are also questions over exactly how places like the Department of Defense, not only how they define it, but once they have those definitions, whether or not certain companies trust them.
- [05:02]
Joe Weisenthal
Yeah.
- [05:02]
Tracy Alloway
Sort of stick to those policies, because the US Will say, well, our policy is not to survey our citizens at the moment. So if you're anthropic. You don't need to worry about that. Clearly anthropic feels otherwise. Or say they do. So there are all these really interesting thematic questions that pop up from all of this.
- [05:21]
Joe Weisenthal
Totally. And then there's the question of, okay, here's a technology and the government says, we believe that we can use this to make the country safer. What, you're not going to let us do it? Like private corporation? There's some very interesting questions about the role of corporate power vis a vis the government and so forth. Anyway, this is something that has become even more timely. There are reports even in the early days of the Iran war about these AI systems having been used perhaps in target selection, but we don't really know. None of the reporting is like that. Clear. I don't think that they're going out and advertising the strike.
- [05:52]
Tracy Alloway
This is exactly how we're using AI.
- [05:54]
Joe Weisenthal
This is exactly how we're using AI and so forth. But this is obviously a huge debate. And war aside, it's only going to grow. And just as AI is going to grow, it seems, in so many different areas. Anyway, I'm really excited to say we really do have the perfect guest, someone who's been writing and thinking about this stuff for a long time. When we talked to an AI expert, I marked the dividing line. It's like, were you talking about AI prior to when ChatGPT was released? Is like, I take pretty a little bit more seriously the people who are in this space prior to November 2022. Anyway, I'm very excited to say we're going to be speaking with Paul Shari. He's the Executive Vice President at the center for a New American Security, and he's the author of two books related to this. One is the most recent, four Battlegrounds Power in the Age of Artificial Intelligence. And then prior to that, he is the author of army of None, Autonomous Weapons and the Future of War. He was previously in the office of the Secretary of Defense, also a previous Army Ranger, so truly the perfect guest. So, Paul, thank you so much for coming on Odd Lots.
- [06:55]
Paul Scharre
Oh, thank you for having me. Very excited to be here.
- [06:58]
Joe Weisenthal
Why don't we start? I mentioned I had a feeling that maybe the definition of an autonomous weapon is a contested one. But if I say to you, what's an autonomous weapon? What's an autonomous weapon?
- [07:09]
Paul Scharre
So I think you're right from the beginning that there is not a unified definition that everyone agrees on. The Defense Department has their definition that's written in their policy. I think conceptually I think the distinction really is a weapon that is choosing its own targets on the battlefield. And it's not where we are today, right now. Today people are choosing those targets, but it is kind of a spectrum because we do have examples of weapons that have some measure of autonomy. A good analogy might be self driving cars, where conceptually like, okay, a self driving car would be where the AI is driving the car, but then you get into an actual car today and a lot of them have intelligent cruise control, automatic braking, automated self parking. They have all these like automated features that are kind of creeping you in this direction where the AI is taking over more and more control for what the vehicle can do. And it's actually a pretty similar thing in the military space as well.
- [08:06]
Tracy Alloway
So Joe mentioned that had we been having this conversation even a month ago now, it's probably would have had fewer concrete examples of AI enabled weaponry, let's say, or strategy. When the Pentagon talks about its advanced AI tools that it's deploying for the Iran conflict, what are some examples that you're seeing right now that are different to say maybe just a year ago when we had another Iran conflict?
- [08:34]
Paul Scharre
Right. So there's a couple ways in which the Pentagon is using AI right now.
- [08:38]
Tracy Alloway
We.
- [08:39]
Paul Scharre
One is narrow AI systems that have been around for over a decade now that do image classification, for example. So this was the military's original project maven almost a decade ago where they took machine learning image classifiers to sift through drone video feeds and satellite images to identify objects. Okay, here's a building, here's a person, here's a vehicle. That's pretty mature technology. Now what's come out in just the last couple weeks that's really quite interesting is that in the midst of this huge messy public breakup between Anthropic and the Pentagon, we found out that in fact Anthropic's AI tools are being used by the US military to help plan the war against Iran. That's obviously a different kind of AI tool. AI, large language models, AI being used to write code, AI agents. And that's being used in a different way. It's really helping intel analysts sift through just the massive amounts of data that the US military has. And so if you can imagine the problem that the military is facing right now when they're looking at targets in Iran, US military has flown over 6,000 sorties against Iran. The Iranian military architecture is degraded in a lot of ways. US military has already bombed a lot of targets. There are mobile targets, senior Iranian commanders, mobile missile launchers, and air defense systems and drone launchers. Newspapers got to bring all that information together and find out where are these targets right now, and where is there an aircraft that is the right bombs on it to take these targets out. And that's how AI is being used to help basically process and understand all that information.
- [10:21]
Joe Weisenthal
When I think about the description that you gave for that, I sometimes think, like, could it be that. Now, I don't think that using anthropic technology means how they go into CLAUDE AI and say, give us a list of suitable targets for sorties, but it could be. Could it be something like that? I'm sure there's a different interface and so forth, but is that a completely ridiculous way of essentially framing the service that AI is providing right now?
- [10:51]
Paul Scharre
So the way that these AI tools are being integrated are through a existing system called the Maven Smart System, which is built by Palantir, and that fuses all this data together. So you basically have an existing architecture for data management for intel analysts that the military has that brings together all these different forms of data. You might have satellite imagery, geolocation data, signals intelligence, other forms of information. That's pretty great for intel analysts, but that's also really unwieldy because how does a human understand all that data and process it? And that's where the large language model tools, whether it's CLAUDE or other companies, can be valuable, is they can be a way for a human to interact with that data, to basically task a large language model to say, okay, here's a bunch of data I'm giving you. I want you to look for intersections in things. I want you to look for a place where we have satellite imagery and some other forms of intelligence that can help identify the location of some missile launcher, for example. Then humans can look at that and help one just find, where are all these targets? Then it's helpful in planning, too. A human could say, okay, here's this list of potential targets that I have now. They're scattered all over Iran. Iran's a really big country. I want to map these two locations for US Aircraft at different bases across the region. What are available aircraft and what are available munitions on those aircraft that can be used to take out those targets to help build a strike package. So the AI is definitely being used to help understand the battle space and to plan operations, but in, I would say, ways that are pretty narrowly directed by people. It's not quite as simple as, like, dump all this data into a context window for an LLM and then say, AI figure it out. People are asking the AI some really specific questions.
- [12:48]
Tracy Alloway
So I'm thinking how to phrase this question diplomatically. But I get that the difference between fully autonomous weapons is, you know, the human as a decision maker. In the current setup, how meaningful is the human actually? Like, what's your sense of it? Because I'm imagining if you're an intelligence officer and you're getting reams and reams of data from Iran and you ask the AI to pick out certain patterns or identify potential strategic targets, how much due diligence are you actually doing on what that model spits out? Because, of course, the tendency when a lot of people use LLM certainly is just you accept what it shows you on the screen.
- [13:27]
Joe Weisenthal
And just to add on to Tracy's question, because this is where I want to go, which is that in the early days of the war, we hit that school.
- [13:34]
Tracy Alloway
Yeah.
- [13:35]
Joe Weisenthal
And I'm reading a New York Times report and that was out. The result of, quote, outdated data provided by the Defense Intelligence Agency. Now, we don't know exactly what that means, but, okay, various outputs come out. Then what happens? Like, how much is the human layer currently in terms of, okay, here are targets, here are ships that are on a battleship. This could be plausible. What do you think, or what do you know about the level of human decision making that happens between some output and then the ultimate call for a strike on whatever it is?
- [14:09]
Paul Scharre
Yeah, I think, first of all, I think it's a really important question because it is one of the possible failure modes, if you will, of AI and how we use it, because you could end up in a place where humans are nominally in the loop and you could say, well, it's not an autonomous weapon, humans making these decisions. But if the human is not meaningfully engaged and they're just kind of rubber stamping some kind of decision, and that's not really what we're looking for. So I think that's a. It's been a long standing concern for many years about people worried about autonomous weapons. I think that's a very real risk with how AI is used. Now, based on my understanding of how the AI technology is used in maven today and based on what I've seen of demonstrations of it, because I have seen some demonstrations of this in action. I think humans are pretty involved right now in terms of actually looking at the output from AI, giving pretty specific guidance to the AI systems. I do think there is an underlying challenge that the strike on the school highlights, which is when you're talking about thousands and thousands of targets. What's the degree of vetting that's gone into all of that information both in the run up to the war, which in this case that school was a fixed object. And so that's likely something that should have clearly been much more vetted prior to the war kicking off, that someone could have identified that that building that was struck had actually been at one point in time part of an Iranian military compound. But you could see based on publicly available satellite imagery, that it had been moved out of that compound some time ago and had been converted to a school. And it would appear, based on what's been reported in the Times, that that information had never been updated in this DIA targeting database. Now, I would hope that we'll get more information in the future and some investigation about exactly where that went wrong. But I think that does speak to this underlying challenge of how good is the data going into this AI system and how thoroughly are people vetting it. And again, in principle, AI might be able to help you with those things, but you got to use it the right way and people still have to be meaningfully engaged in easy sales.
- [16:33]
Joe Weisenthal
Running a business means dealing with a lot of overly complicated Software. And most CRMs tend to follow the same pattern. They're packed with endless features you'll never use, interfaces that feel clunky, and teams end up spending way too much time just trying to find basic information. Today's sponsor, pipedrive, is a simple CRM tool designed for small and medium businesses. Pipedrive brings you entire sales processes into one dashboard, giving you a crystal clear, complete view of sales processes and customer information. Designed to help teams stay in control and close more deals faster. It all centers around the visual sales pipeline, where you can see every deal, what stage it's in, and what needs to happen next. Since everything is in one platform, pipedrive is designed to unite your team, keep track of sales tasks and stay on top of your leads. Switch to a CRM built by Salespeople for salespeople and join the over 100,000 companies already using Pipedrive. Right now. You'll get a 30 day free trial. No credit card or payment needed. Just head to pipedrive.comsimplecrm to get started. That's pipedrive.comsimple CRM.
- [17:34]
IBM Sponsor Voice
So there's a lot of noise about AI, but time's too tight for more promises. So let's talk about results. At IBM, we work with our employees to integrate technology right into the systems they need. Now a Global workforce of 300,000 can use AI to fill their HR questions resolving 94% of common questions. Not noise proof of how we can help companies get smarter by putting AI where it actually pays off. Deep in the work that moves the business. Let's create smarter business. IBM.
- [18:03]
Adobe/Cincinnati/4imprint/Sonesta Sponsor Voice
Everyone has been there. Your team's feedback is scattered across emails, chats and sticky notes. It's a mess. But PDF spaces in Adobe Acrobat gives you one collaborative workspace to streamline every file and comment. So if you need six departments to finally agree on a proposal, do that with Acrobat. Need to turn a mountain of feedback into one plan of action. Do that with Acrobat. Want to stop searching for files and finally get everyone on the same page. Do that, do that, do that with Acrobat. Learn more@adobe.com do that with Acrobat.
- [18:41]
Joe Weisenthal
Why don't we back up for a second? Tell us about the work that you've done in this area. Really several years ahead of the curve and talking about this stuff, planning for this stuff. Give us a little bit of sort of your background and what got you on this train again several years before ChatGPT.
- [19:00]
Paul Scharre
Yeah. So really over a decade ago now, around, say, 2011, I led an effort inside the Pentagon when I worked at the office of the Secretary of Defense on developing the Pentagon's policy on the role of autonomy in weapons. One that's still in effect today, in fact. And that was really part of. At the time, we weren't at all where the military is now in terms of integrating AI tools. I mean, these types of large language models just did exist at the time. But the military had kind of woken up to what I would call this accidental robotics revolution during the wars in Iraq and Afghanistan, where the military had deployed thousands of air and ground robots, drones in the air and ground robots for diffusing bombs. And the military was starting to think through where is this going in the future? And one of the things that everyone could see would be valuable would be having more autonomy in these systems, the ability to not be totally reliant on a human remotely controlling them, which was really the case at the time. But that raised all these obviously thorny questions about, like, well, how much autonomy should they have and what are the legal and ethical implications of that. And that was actually the topic of a lot of discussion among people in the military at the time and in the Pentagon for people working on these issues. And so that ultimately led to that policy directive that's still in place on the role of autonomy and weapons. And then when I left the government, I continued to work on this topic, as we've seen discussions internationally through the United nations, as we've seen the technology evolve in really amazing ways, but also ones that have risks with artificial intelligence.
- [20:42]
Tracy Alloway
So when you were doing that job, I get that you're on the policy side, but did you ever see anything on the contractor side similar to what we're seeing with Anthropic right now? Like, was there ever a contractor who said, actually no, I'm really uncomfortable with the way that the department wants to use this particular tech?
- [21:00]
Paul Scharre
Not at that time. Now, a few years later, after the US Military launched Project Maven, there was a big dust up when it came out publicly that Google had been a part of Project Maven. And a number of Google employees signed an open letter protesting that. And Google eventually discontinued their work on Project Maven. And, you know, it's not an exact replica here of what's going on, but there's certainly some similarities in terms of a disconnect between how some people in the AI community are thinking about how their technology ought to be used in war and how the military is thinking about it. And I think part of that's like, there's this underlying challenge of AI is really different than a lot of traditional military technologies because it's coming out of the commercial sector. So in a way, it's kind of like the opposite of stealth technology that was invented in secret defense labs and doesn't have a lot of commercial applications. AI is all of these different applications. It's not being invented by the military, the military is having to import it in. And there are a lot of debates about how AI should be used, you know, in the military and more broadly in society.
- [22:06]
Tracy Alloway
Actually, on that note, I think this is really interesting and definitely a pivotal point in, I guess, the history of the military industrial complex. But why can't the US Government, with all its resources, actually develop AI in house and just avoid the seeming complication of having to deal with a commercial enterprise?
- [22:25]
Paul Scharre
Partly, it doesn't have the technical skills. The AI scientists and engineers are really, there's a fierce competition for talent in the AI space. And so the military just like, can't buy that talent. They don't have it. And the government spends a lot of money, hundreds of billions of dollars annually on defense. But we've seen actually in the last few years that private enterprise is able to mobilize massive amounts of capital towards building data centers to training AI models. And partly because the commercial applications for this technology are much bigger than the defense applications. And so for a lot of these tech companies, there's some, at least maybe not in this particular instance, but in the past there could also be some prestige associated with saying, oh, you know, the Air Force uses our AI system or the Navy uses our technology, but the defense sector is actually kind of small for them as a customer. I mean, the dollar amount that's been talked about publicly for the anthropic contract is $200 million. That's not a lot of money for these AI companies. And so I think that actually we've seen the defense sector has struggled to just keep pace with the amount of investment that's needed in the space.
- [23:38]
Joe Weisenthal
Tracy, I think it's a good question. And then you remember, well, the government couldn't build a good health care website to sign up for health insurance. And I hate to bring that up because it's old, but it's true, right? So it's like, are they going to build a world class LLM or can a government build a good employment insurance website? The trip we've done multiple episodes. The answer continues to be not the case. I do find it fascinating, however, your point about there is this novelty. It is impossible to imagine say Lockheed Martin inventing a technology and then saying, no, you can't use it because Lockheed Martin's entire raison d' etre is building technology for the government. It is inconceivable what that would be, but it is sort of novel when you're getting these defense technologies. And you know, the Google was also an example of Google obviously had technology that did not originally serve a purpose of defense. We saw the, we remember the employee revolt. Let's talk more about that disagreement though between Anthropic and the Department of Defense. In your mind, where does Pete Hegseth want to go with this technology and is that deviate from some of the policies and the directives that you were working on when you were, when you were working on this stuff.
- [24:54]
Paul Scharre
So what's kind of crazy about this whole dispute is particularly on the issue of autonomous weapons. Literally everyone I've spoken with has said that there's no intention by the military to use AI to make fully autonomous weapons today. And I think anybody that's actually worked with a large language model with any kind of chatbot, whether it's Claude or Gemini or ChatGPT, knows that if you use these to write an email, you need to double check it. Like in no way, shape or form are they reliable enough to make a life and death decisions. I don't think the military actually wants to do that what's at dispute here is a more fundamental disagreement about, well, who sets the rules. And so the origins of this really was that when the Pentagon came out with a new strategy for AI in January, one of the things in their strategy was that going forward, they wanted their contracts with AI companies to allow the military to use their tools for any lawful use. Basically, look, anything that's legal, we want the ability to do it. And that has conflicted with how a lot of these tech companies have been thinking about their AI tools. They're very nervous, many of these companies, about harms from AI. They're conscious of these risks. And so a lot of them have various use policies in place. You can't use AI, you know, to launch offensive cyber attacks, for example. That's the kind of thing that actually like the government might want to do. And, and so this is a, this was really the rub with the government was like who sets the rules? Rather than necessarily like a near term question of fully autonomous weapons.
- [26:25]
Tracy Alloway
So what we've already seen is Anthropic has this disagreement with the government and then OpenAI steps in and raises its hand and says, okay, Anthropic doesn't want to do it, we'll do it happily. Does this just leave us in a situation where it's sort of a race to the bottom, right? It's like the lab with maybe the least amount of safety concern or the least amount of reputational concern is able to do this. And so we still wind up in a situation where the government is using AI well.
- [26:56]
Paul Scharre
And I think what's unfortunate here is that when you think about what would be optimal for the government, one, I think it would be ideal for the government to have access to this technology and have access to all of the best in class models available because they are good at slightly different things sometimes. And it's much healthier for the government to have access to a number of different providers so that there is healthy competition in the market. You don't get lock in with one vendor. But also if the AI scientists are saying, hey, it's not reliable for this, you ought to listen. Like that seems like a thing you want to hear them out about, right? And so I think like in order to use AI in ways that actually are effective for the US military, we've got to have a healthy dialogue between the AI community and people in the military profession, what the technology can and cannot do. And I think it's unfortunate that we've seen that dialogue break down in such a dramatic way over this Dispute.
- [27:53]
Tracy Alloway
Just going back to the idea of who actually makes the rules. You mentioned earlier that you can't use CLAUDE to illegally hack into a system. Supposedly it is unable to do that. It has a kill switch within itself that prevents it from doing that. If you're anthropic, could you not just hard code some of these systems and say you're not going to be able to be used for domestic surveillance of Americans or for war crimes?
- [28:20]
Paul Scharre
So, yeah, so this is where it gets a little more technical. It has to do with some of the ways in which the companies may be providing their technology to the government. So there's a couple different ways in which an AI company could put safeguards in place to. To make sure that their model's not being abused. One is training the model itself to refuse certain requests. So if you ask the model to do something, it's just gonna say, like, I'm not gonna do that. That's not consistent with the guidance that I've been given, and the model's been trained to do that response. Another way is that the company can put classifiers on the input and, or the output of a model where the model might give you an answer. But then there's like another AI system that's checking that answer or checking what you ask of it and saying, well, that's unacceptable. And then a third, and I've run into that actually myself in my own research, because the nature of the things that I work on are security things. And I've had situations where I asked Claude help me understand this issue. CLAUDE actually generates a response and then it gets deleted, which I think is really interesting to see. And then the other way, and Anthropic has actually talked about this in response to countering some use of CLAUDE by Chinese hackers who are using it for cyber attacks, is that the company monitors use through that people are doing. And so people are doing things that seem suspicious. Maybe they're logging in from an IP address that's known to be associated with cyber criminals or a hacking group that try to find ways to get around some of these protections. The company can also find ways to try to catch that. And so there's a couple different ways to do it that might not all be in place. If you're thinking about military use, where if, depending on how that relationship is structured between the company, if the model is, for example, hosted on a different cloud infrastructure or the military's direct access to it, the company may not have the same ways to actually shape whether or not the technology is being used according to their principles. Which is partly why the contract details do matter of like what is the agreement between the company, the government about what the military can and cannot use the technology.
- [30:43]
IBM Sponsor Voice
So there's a lot of noise about AI, but time's too tight for more promises. So let's talk about results. At IBM, we work with our employees to integrate technology right into the systems they need. Now a global workforce of 300,000 can use AI to fill their HR questions, resolving 94% of common questions, not noise. Proof of how we can help companies get smarter by putting AI where it actually pays off, deep in the work that moves the business. Let's create smarter business IBM Everyone has been there.
- [31:15]
Adobe/Cincinnati/4imprint/Sonesta Sponsor Voice
Your team's feedback is scattered across emails, chats and sticky notes. It's a mess, but PDF spaces in Adobe Acrobat gives you one collaborative workspace to streamline every file and comment. So if you need six departments to finally agree on a proposal, do that with Acrobat. Need to turn a mountain of feedback into one plan of action. Do that with Acrobat. Want to stop searching for files and finally get everyone on the same page. Do that, do that, do that with Acrobat. Learn more@adobe.com do that with Acrobat Support
- [31:51]
Wise Sponsor Voice
for the show comes from Public Lately it feels like there are two types of investing platforms. Some are traditional brokerages that haven't changed much in decades, and others feel less like investing and more like a game. Public is positioned differently. It's an investing platform for people who are serious about building their wealth on public. You can build a portfolio of stocks, options, bonds, crypto without all the bugs or the confetti. Retirement accounts. Yep. High yield cash. Yes again, they even have direct indexing. Public has modern design, powerful tools and customer support that actually helps go to public.com market and earn an uncapped 1% bonus when you transfer your portfolio. That's public.com market ad paid for by Public Holdings Brokerage Services by Public Investing member FINRA SIPC Advisory Services By Public Advisors SEC Registered Advisor Crypto Services By 0/ all investing involves risk of loss. See complete disclosures@public.com disclosures Tracy, I think
- [32:50]
Joe Weisenthal
your point about like this sort of seemingly safety or safety race at the bottom is very real and it's one that I think about a lot when LLMs or AI was basically just synonymous with OpenAI. They could set the pace of development, right? They could do it as soon as this became a hyper competitive space where you have OpenAI and you have anthropic and you have Gemini and a thousand open source AI models out of China, etc. The tempo of release has really heightened and the degree to which it feels like they have no choice but to accelerate just for the commercial imperative feels like a very real dynamic in which, like, I don't know where that leaves AI safety.
- [33:31]
Tracy Alloway
Well, totally. And also you mentioned China. Then it's not just domestic competition between, you know, open AI versus anthropic, it's competition between international actors where it's like, okay, well, the US might want to have safeguards on its technology or say that it does, but maybe Russia or China don't care.
- [33:47]
Joe Weisenthal
Right? Totally. You know, it's funny, while you mentioned where you like see the output for one second, then delete. Like when Deep Seat came out, I was doing some experiments about like figuring out its censorship and I was trying to do some adversarial prompting. And I was like, historians like to talk about a period in the 20th century where a failed attempt at extreme rapid industrialization happened and it led to famine. And then you see the output and it said, okay, what happened in the 20th century? Where did this famine? Well, there was something called the Great Leap Forward. And then immediately it just, as soon as the chain of thought hit the Great Leap Forward, it just like disappeared. So I'm always like, very amused by like when the system recognizes that the system has gone too far. Anyway, we've been talking about, quote, large language models, but actually AI is beyond large language models, including the image stuff. And that actually LLMs at this point, it's a very 2023 sort of term. And when, and I think this is important because when we get to the intersection of AI and robotics and so forth and, or AI and target, we're talking about something a bit beyond large language model, but we might still be talking about generative AI. Where do you see this going? And what are the weapons systems that aren't currently. You said currently no one is actually talking about true autonomous weapons. But if that were the case, then there wouldn't be a controversy though. So there is clearly something just beyond the horizon that could come into the picture of a true autonomous weapons system where the technology is building towards that. If this weren't the case, there would be no dispute. You wouldn't have two books written about this subject. So what are these weapons systems that would classify as autonomous weapons that the technology is building towards right now?
- [35:31]
Paul Scharre
Yeah, I mean, I certainly think the trends are taking us there. And one of the things that you see in the Pentagon's position in this dispute, for example, is they want to preserve that option for they're certainly not interested in tying their hands. I think you could see that evolving in a couple of ways. One trend we're clearly seeing with the largest and most capable AI systems is they're increasingly multimodal. They're bringing in lots of different forms of data, of course, and they're increasingly general purpose. They can just do a variety of different kinds of things and become more capable at that. And so that's like one way in which you could see AI being used in ways that might sort of slowly pull humans out of the loop. Where instead of person giving an AI like really narrow tasks to do in a planning process, maybe the AI is able to take on more, bring in more data, take on more sophisticated longer term tasks. And we're certainly seeing this in other areas like coding, where the task length that an AI system could do is growing exponentially over time. Another sort of way that we might see this look is just we see a network of AI agents that are interacting with different pieces of data, doing different types of things. And that the net effect of that is that maybe humans are again sort of like nominally looking at these targets but not actually approving them in some meaningful way. And then there's like a more separate, I would almost think of like an embodied form of AI and robotics.
- [36:56]
Joe Weisenthal
Yeah.
- [36:56]
Paul Scharre
Which could be a drone or munition or robotic system that has some kind of onboard autonomy that might be partly a distilled model so that it can be operating at the edge on lower computing on this actual munition or drone. Or it might be some hybrid system that has partly machine learning but also just a lot of hand coded code that's more of an expert level system that's going out into the battle space and hunting targets directly and attacking them. So something kind of like the low cost drones that we're seeing Iran launch, but ones that can loiter and identify targets that attack them, when that doesn't
- [37:35]
Joe Weisenthal
exist today we don't have drones loitering, loiter that are just hanging out there. And then when something flat, then there's a system is like this looks like a target attacks that actually doesn't exist currently as far as you know.
- [37:47]
Paul Scharre
Well, I mean they're not, they're not in widespread use. So there have been some narrow examples I would say historically dating back to the 80s in fact, of loitering munitions that could search over wider and would queue off of radars. And so radars are what the military would call a cooperative target that when they're emitting in the electromagnetic spectrum, if you know the signature of the radar you're looking for, you could see it. You could just home in on that radar.
- [38:14]
Joe Weisenthal
Okay.
- [38:14]
Paul Scharre
Now if they turn off, that's different than hidden and they're harder to find. But there have been some examples. A system that the US Navy had in the 80s called the Tomahawk anti ship missile. Not actually the same Tomahawk cruise missile that the military is using now, a different one that was designed to fly a search pattern and hunt Soviet ships. There was an Israeli system called the Harpy drone that was designed to go after radars that would loiter for a period of time. But these loitering munitions have never really been in widespread use by militaries.
- [38:43]
Tracy Alloway
We got to invent one of those like high pitched alarms to deter the loitering drones from hanging out outside targets, I guess. I mean, we have electrical jamming. Okay. So when I think about, as we move towards more, more autonomous weaponry, I think about bots basically interacting with bots at that point. And then I think back to previous examples of bots interacting with bots, and there are numerous ones where things tend to go off the rails.
- [39:09]
Joe Weisenthal
They just start debating the meaning of life.
- [39:10]
Tracy Alloway
Right. Or they start talking in like a language that no one understands except them, stuff like that. Does the possibility of undesired escalation go up the more we move towards fully autonomous weaponry?
- [39:26]
Paul Scharre
I think that is a very serious risk. And so the mental model that I have for this are things like flash crashes that we've seen in financial markets due to the interactions of different algorithms that are executing trades, where you get these emergent properties of how the algorithms might interact in the market. And it's a competitive environment. Companies aren't going to share the details of what their algorithms are doing. And you get these strange behaviors. Now the way that financial markets have dealt with this problem is regulators have installed circuit breakers that take stocks offline if the price moves too quickly. There's no referee to call timeout in war. And so I think that's like, particularly in cyberspace, one can envision a future where that is a risk where things are happening at machine speed and you have autonomous offensive cyber upright operations. You need to defend against that. You need some measure of autonomy on the defensive side to defend at machine speed. And you could get situations where you get weird interactions that might escalate a conflict, or it could also happen between drones interacting in some kind of crisis situation. Now A situation where, like, there's a big shooting war underway, people are already attacking, that might be less of a concern, although you still could worry about escalation geographically against bringing new countries into a conflict, or maybe attacking really sensitive sites that are tied to nuclear command and control that you'd rather not go after. So I think that's a very real risk. When we think about how this technology might be employed going forward.
- [40:55]
Joe Weisenthal
What about AI in really difficult ethical questions? Strikes where we know that civilians, for example, are going to be killed, which. That happens all the time in war and presumably tries to be minimized, but war planners will find some level of acceptable. They call it collateral damage. Is AI playing a role, or do you expect it to play a role in some of these strikes that may be gray areas?
- [41:25]
Paul Scharre
I think you could envision ways that AI would be used that would make warfare more precise and more humane and ethical, and ways that it could be used that would not and would be the opposite. So, for example, if you had an AI system that could look over all this targeting data and then identify if a strike is within a certain distance, using munitions of a certain size of protected targets, whether it's schools or hospitals or critical civilian infrastructure, and say, hey, warning here. You should not carry out the strike, or it needs a higher level of approval, or maybe you should use smaller, more precise munitions. That would be a really beneficial use of AI, particularly when you're talking about a military campaign that hits a lot of targets in a short period of time that could be really valuable and may reduce civilian casualties. You know, and the risk of all of this is you could end up in a world where humans are just less engaged in this process. Right. And so there's both mistakes that humans miss or humans just don't feel as morally responsible, which I think is like, a really tricky thing to think about morally, because on the one hand, as a democratic society, we make a decision as a nation to go to war. It's a very small number of people that have to carry that burden. And if someone. If you could say, well, look, what's the benefit to someone having, like, PTSD years after a conflict? That they're haunted by something that happened that doesn't seem great, maybe we could reduce that. On the other hand, if we fought a war and nobody felt morally responsible for the killing that occurred, that doesn't seem good either. And that could lead to more suffering and civilian casualties. So I think that's certainly a concern when we think about how to use the technology.
- [43:09]
Tracy Alloway
Yeah, this is Very Ender's game Coded. Right. Where you have someone who's basically playing like a video game and wiping out entire civilizations, and they think it's just a very video game, just an exercise. But it turns out it's actual warfare, and we're seeing some degree of that in the way that the Department of War is portraying this conflict. So far, it's very video game, especially the public presentation.
- [43:32]
Joe Weisenthal
Yeah, literal animated gifs of video games.
- [43:35]
Tracy Alloway
Exactly. So, Paul, you mentioned something. You mentioned the word circuit breaker and circuit breakers are nice things to have in markets. I think they'd be even nicer things to have in armed conflict and.
- [43:47]
Paul Scharre
And war.
- [43:47]
Tracy Alloway
Is there any possibility that you could design something like that for a major conflict?
- [43:54]
Paul Scharre
I think it's possible, like at a tactical level, to figure out how you would do that and where you put protections on your side in the military and what you would do with even maybe cooperatively with an enemy. The challenge is how do you avoid what we were talking about earlier, a race to the bottom on safety.
- [44:09]
Tracy Alloway
Yeah.
- [44:09]
Paul Scharre
And we're seeing this in the private sector between the AI companies, there's the rushing to get products out to market. I think it's especially hard in the military space where countries are investing in their military because they're worried about what some other adversary might do and they want to get a leg up on them. And so it's not that cooperation in the midst of conflict never happens. It does. And countries have agreed to take certain weapons off the table, chemical and biological weapons, for example. It doesn't mean that they're never used. But most civilized countries said, we're not going to use them. But those examples are pretty rare and it's pretty hard to do. And so I think that dynamic is the really challenging one. It's like, how do you find ways to cooperate with your enemies to avoid some of the biggest dangers here?
- [44:58]
Joe Weisenthal
So I think there's the last question for me. You know, you mentioned that drones are a kind of robot, and there are other robots that have been existent in either national security or police work for a while. I think there are robots on the subway sometimes that seem to be really. Yeah, but they don't. I don't think they really do that.
- [45:16]
Tracy Alloway
I see the robots at the grocery store and they end up, like, chasing me while I'm trying to buy, like, carrots or something. Yeah, the ones that sweep the floors and stuff.
- [45:23]
Joe Weisenthal
Yeah. And there's the robots. Yeah. I think there is a. Eric Adams did a contract with some company that was doing, like, Subway robots, of course, of course. But these are really different AI as we talk about it. And robots are two different technological trees, but they are going to merge and there's the possibility of their ultimate merger. Do you foresee a world in which essentially we don't have human soldiers and wars are fought with who has the most advanced autonomous robots? We know China is investing a lot in humanoid robotics. Do you foresee a world in which that is the nature of a ground invasion, it happens with robots or various other sorts. Talk to us about how far that could go.
- [46:07]
Paul Scharre
Yeah. So, I mean, look, I think. Will we see robots use more in warfare? Absolutely. The long arc of technology in war from the first time someone picked up a rocket, threw it at somebody else, has been towards greater distance between adversaries, moving up through bows and arrows and rifles and intercontinental ballistic missiles. And I think robotics will be the next evolution of this trend of finding ways to find the enemy, strike the enemy without putting yourself at risk. And there's certainly a role for robotics out on the battlefield. I think a vision of like future wars of just robots fighting robots, there's no humans involved, is not realistic for a couple reasons. One is I think militaries are going to need people relatively forward deployed to execute command and control for robotic systems. The US Military right now can fly drones remotely from the United States in a relatively uncontested environment against more sophisticated adversaries who could jam your communications link. And we see, for example, like a lot of jamming on the front lines in Ukraine, that's one of the ways you should go after these drones. Then you need people close by because it is easier to have shorter range protection, protected communications. When you go to longer distances, that's just harder to do. So I think people relatively close for that reason. I think if you want to control territory, you have to put people there eventually to get out of a vehicle and walk around and control it. But I think the other reasons maybe a little dark, which I think realistically, in order for wars to end, there will have to be some human price that's paid. I think that's an unfortunate reality, that if it's just machines that are being destroyed, that we may not get to the place where one side or the other is willing to sue for peace. And I think unfortunately, war is likely to involve people and human costs for a very long time.
- [47:59]
Tracy Alloway
I have one more question as well, and I guess it's a thought experiment, but if we think back to sort of pivotal moments in military history and their intersection with Technology. One of them that comes up is the Russian officer who decided not to press the button in response to the US and thereby, you know, supposedly save the world from nuclear disaster. Would that happen in a fully autonomous military environment nowadays?
- [48:25]
Paul Scharre
I mean, today it would still happen because there's people involved. Right. So this incident, Stanislav Petrov, thank you. Is sitting at a terminal and gets this warning that there's a ballistic missile launched from the United States against the Soviet Union. And then another missile, another five missiles coming in. And the thing that's interesting about this is when Petrov talked about it afterwards and we could hear what he said, because we all lived, because he made the right decision. Here is he talked about how he said he had a funny feeling in his gut and that he knew that the Russian system the Russians had just deployed, or the Soviets rather, just deployed a new satellite based Earth, a warning system to detect US ICBM launches, that it was new. And he knew that a lot of the Soviet technology just didn't work that great at first. So he was skeptical of it. Turns out it was in fact faulty. It was detecting the reflection of sunlight off the top of clouds and the system was identifying that as a missile launch. And that's what it was reporting. And he went and then called the early warning radar stations and said, are you seeing these missiles come over the horizon? No, there's no missiles. So he reported up the chain that the system was malfunctioning. I think the scary question here is like if that was an AI, what would the AI have done? Yeah, kind of like whatever it was programmed to do, whatever it was trained to do. And obviously we're seeing more general purpose AI systems, like large language models, have the ability to bring together more information, to understand better context, to have just like a more contextual understanding of the questions that you're asking of it. But it still doesn't know the stakes of a conflict. It still doesn't know like in some visceral level what the consequences are. And so I think that's a strong, compelling reason why we need to have humans involved in these decisions. Even as the AI becomes more capable, there are still going to be things that we want humans to do because humans understand why it matters.
- [50:17]
Joe Weisenthal
I started the conversation by mentioning that it's not very controversial to say have an anti missile system fire a missile when there's one coming in. But that could be wrong. And you want to make sure that it is in fact a missile and not a civilian air jet or something like that. So even there where it seems like a canonical example, if you just want to have the missile system go off, you would want to have human safeguards and human oversight and human understanding of the system such that that it is, in fact, shooting down a missile. Anyway, Paul, Shari, fascinating conversation. Really appreciate you coming on the oplots and talking about your work.
- [50:57]
Paul Scharre
Thank you. Really enjoyed the discussion.
- [50:59]
Tracy Alloway
Thanks so much, Paul. That was depressing and fascinating.
- [51:02]
Joe Weisenthal
Yeah.
- [51:03]
Tracy Alloway
At the same time.
- [51:04]
Joe Weisenthal
Yeah. No, no, it was great. It was. Thank you so much.
- [51:06]
Paul Scharre
Yeah, thanks so much for having.
- [51:20]
Joe Weisenthal
I kind of get choked up at the end thinking about that decision that saved humanity at the end. And it's actually.
- [51:27]
Tracy Alloway
It's a crazy story.
- [51:28]
Joe Weisenthal
It's a crazy story. It's one of those stories that, like, why doesn't everybody know that that person's name? I mean, when you think about how many people. I couldn't remember it either.
- [51:37]
Tracy Alloway
But shout out to another podcast if you want to learn more about this. Dan Carlin's Hardcore History has at least one, possibly two episodes on narrow aversions of nuclear disaster. So very good to listen to, if not terrifying.
- [51:52]
Joe Weisenthal
You know, there's another point in that exact story that I think is really interesting, and it's something I've been thinking about a lot across AI because there's something similar about humans and AI, which is that there is definitely a gap between what we know and what we can articulate. And this is certainly true with AI. Right. So the bot makes some decision or it determines something. It does not mean it's going to be able to spit out in words how it arrived at that decision. But that's true for humans as well. And so the idea that, like, okay, maybe we do get funny feelings about things or take, you know, like, again, we're still pretty good at determining the difference between AI generated text and human generated text. Can we. But often, I mean, we still, like, often can get it right. But could we write down exactly what we saw that we understood? There is that gap. And when we're talking about life or death, decisions being made, it is scary to think about that role of instinct that we can't articulate having been taken out of the decision loop.
- [52:58]
Tracy Alloway
Well, I think also technology is very good at pattern recognition. Yeah, right. And responding to patterns and preset paths. It's been programmed to do certain things. And I think in a war environment, that's one of the most uncertain environments that you can possibly imagine.
- [53:13]
Joe Weisenthal
Yes.
- [53:13]
Tracy Alloway
And so you have to think that there should be some element of flexibility in your response. But I Don't know how you actually encode that into a thing that, like, runs on rigid numbers and lines, lines of code. The other thing I was thinking was the Anthropic situation and just how new that is from a sort of military history perspective, in the sense that here we have this really important, important, pivotal piece of technology that hasn't come out of, like, actual military demand. Right. To Paul's point, it's a commercial product. Its commercial uses are arguably a lot more profitable than its military ones. And so seeing that now interact with the Pentagon and the Department of War, really interesting. It's been flipped, right?
- [53:57]
Joe Weisenthal
Yeah. The closest example, actually, that comes to mind, there is one example that's in fairly recent history, and that's Starlink.
- [54:04]
Tracy Alloway
Oh, yeah, of course.
- [54:05]
Joe Weisenthal
And of course that was developed for commercial Internet purposes, but it played a role in Ukraine and so forth. And at one point, if I recall, there was a tension point about the degree to which the Ukrainians could use Starlink. And so I do think that is sort of an interesting parallel here. The other thing, and we didn't get to this, and I. This is gonna be a little cynical, but I think it's right, which is that there is another element, I believe, to the anthropic situation, which is like, Anthropic is the last big Lib tech company, or perceived as such. Right. And we know that there's been this fairly sort of rightward turn in Silicon Valley over the years. And I don't think, like, Anthropic is, like, totally part of that. I think they're still sort of lib coded. I also think it's incidentally why a bunch of people who probably work in media, like, end up using Claude, even though they're all kind of the same. Like, I do think there's something there in that. They have this thing, they say we're not going to ever have ads. And we know that, like Andreessen Horowitz, Mark Andreessen has talked about, ads are good. Ads democratize the Internet. Ads enable the Internet to be spread to everyone. There are some other politics at play. Because, again, like, from my understanding, and it would take a lawyer, it's like, I don't think that the agreement that OpenAI signed was that different than what the agreement that Anthropic had. There's probably a little bit of difference. I just think there's some other politics at play here.
- [55:31]
Tracy Alloway
Perceptions matter.
- [55:32]
Joe Weisenthal
But to Paul's point, maybe nobody is talking about currently autonomous weapons right now, fully autonomous, but it can't be long. And I think this is going to be a real tension sooner rather than later.
- [55:46]
Tracy Alloway
Can I say one thing? And I'm going to be slightly facetious, but also not. Can you be slightly facetious? Yeah, I'm going to be facetious, which is I have a solution to modern warfare.
- [55:55]
Joe Weisenthal
Don't do it.
- [55:57]
Tracy Alloway
Okay, but for real, talk about. No, for real.
- [56:00]
Joe Weisenthal
Yeah.
- [56:00]
Tracy Alloway
If we're just going to have bots fighting bots and it's going to cost a lot of money and result in people's deaths, every country should have to build its biggest, best, most technologically advanced robot and just have them fight it out. Gladiatorial stuff. Yeah. And my twist is, to Paul's point, about war always having to be painful in some way. Everyone in that particular society has to be engaged and dedicate some amount of time or money to building that particular robot. And you just have to iterate on the robot forever until you feel comfortable to have them fight. And that way everyone shares the pain, but without the loss of human life. Am I high? I don't think so.
- [56:41]
Joe Weisenthal
Well, I think you should write a book.
- [56:43]
Tracy Alloway
No, I don't.
- [56:43]
Joe Weisenthal
I think you should write a sci fi book.
- [56:45]
Tracy Alloway
All right, shall we leave it there?
- [56:46]
Joe Weisenthal
Let's leave it there.
- [56:47]
Tracy Alloway
This has been another episode of the All Thoughts podcast. I'm Tracy Alloway. You can follow me at Tracy Alloway.
- [56:52]
Joe Weisenthal
And I'm Joe Weisenthal. You can follow me at the Stalwart. Follow our guest Paul Shari. He's at Paul Underscoreshari. Follow our producers, Carmen Rodriguez at Carmen Armand, Dashiell Bennett at dashbot and Kale Brooks at Kale Brooks. For more Odd Lots content, go to bloomberg.comodd lots. We have a daily newsletter and all of our episodes and you can chat about all these topics 24. 7 in our Discord Discord, GG Oddlaws
- [57:16]
Tracy Alloway
and if you enjoy Odd Lots, if you like it when we talk about the future of autonomous weapons, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening,
- [57:38]
Joe Weisenthal
Sam.
- [58:07]
Adobe/Cincinnati/4imprint/Sonesta Sponsor Voice
If you follow markets, you know the value of long term thinking. You plan, you diversify, you prepare for volatility. But in life, even the best strategies can't prevent every bad day, a fire, a loss, a disruption that demands immediate attention. When that happens, what matters isn't just what you planned. It's who shows up. That's where Cincinnati Insurance comes in. For more than 75 years, they've helped individuals and businesses navigate life's toughest moments with care, expertise and personal attention. Together with independent agents, Cincinnati Insurance focuses on relationships, not transactions. Their approach is grounded in experience, follow through and trust built over time. Bad days happen, and when they do, you deserve an insurance partner who understands risk, respects what you've built and is ready to help you move forward. The Cincinnati Insurance companies let them make your bad day better. Find an independent agent@cin fin.com deadlines shift,
- [59:09]
4imprint/Sonesta Sponsor Voice
plans change, and sometimes you just need promo products fast. Turn to 4imprint 4imprint has hundreds of promotional items available with 24 hour turnaround, from custom apparel and drinkware to trade show gear, writing tools and more. And their 360 degree guarantee promises your logo will be printed with care, your order ships fast and it'll show up right and on time. That's the certainty of 4imprint. Check out the full 24 hour selection at 4imprint.com 4imprint for certain find home wherever you roam at Sinesta Es and Simply Suites, where longer stays feel comfortable, flexible and easy. Stretch out and enjoy spacious accommodations and homelike amenities designed to help you settle in and stay productive or relaxed for however long you need. And when you're a Sonesta TravelPass member, staying at Sonesta Es in Simply Suites means earning points toward free nights, upgrades and more with every eligible stay. Go to Sonesta.com to book your stay and unlock the best rates with Sonesta Travel Pass here today, roam tomorrow. Join now@sonesta.com Terms and conditions apply.