Loading summary
Freedom From Religion Foundation Spokesperson
We like to think the first Amendment is settled law, you know, freedom of speech, freedom of religion and freedom from government imposed religion. And yet here we are. Across the country, politicians are pushing prayer into public schools, blurring church state boundaries and claiming it's all about religious freedom, even when it means the government picking winners and losers based on belief. The Freedom From Religion foundation exists to do one very unglamorous, but very necessary hold the government to the Constitution. FFRF works through education, advocacy and the courts to keep keep public institutions neutral on religion. Not anti religion, just pro democracy. If you care about civil liberties, pluralism and a country that actually belongs to all of us, visit FFRF us newyear or text state to 511-511 to learn more and join text state to 511-511 help protect a country that belongs to all of us. Visit FFRF us newyear or text state to 511-511 today. Text fees may apply foreign.
Jon Stewart
Hello everybody. My name is Jon Stewart. Welcome to the Weekly show podcast. We got a banger for you today. As you know, the world is hurtling in no small measure towards its utter and complete destruction. And there's a new wrinkle in destruction of our world and that is that a lot of the weaponry that we seem to be deploying at the various places around the world are being controlled by not necessarily autonomous but large language model AI. Anthropic Open AI. The same people that bring you Claude and chat GPT and help you break up with your boyfriend or girlfriend using a rhyming scheme that Drake would use that's also being used to target and destroy our enemies. And it is incredibly chilling. And just recently a huge controversy broke out into the open when one AI company, Anthropic, drew a line and said we shall not, we shall not allow our product to be used in this way. And then another AI company there, what do you call them there? The, the OpenAI went, we will. That's cool with us. And but there's it's a lot more nuanced than that. It turns out there may not be heroes and villains in this story, but we are going to discuss it all today in an episode entitled how are we all going to Die and when exactly is it going to be happening? And we have two experts in the field of AI and how it is utilized and especially within a military context. We have with us Dr. Sarah Shoer and Paul Sherry. And let's just get to that. Ladies and gentlemen, we are delighted to welcome today on our continuing episode of I think we're all going to die. Our guests today are experts in the field of how we are probably all going to die. Dr. Sarah Shoker, who is a senior research scholar at the University of California at Berkeley, and Paul Shari, executive vice president for the center for a New American Security and author of Four Power in the Age of Artificial Intelligence. Thank you both for, for, for joining us here today.
Paul Scharre
Thanks for being here.
Jon Stewart
Sarah, let's start with you. You worked in AI. You explain just very briefly your area of expertise as we move forward.
Dr. Sarah Shoer
Yeah, sure. So I used to be the lead of the geopolitics team at OpenAI. That was our research team and we focused on a variety on a portfolio of topics relating to AI and international stability. And, and currently in my role at Berkeley, I focus on new testing and evaluation methods for generative AI models and their potential impact on warfare and military AI integration.
Jon Stewart
Very, very apropos for today. And Paul, for you as well, where do you stand on studying AI on military and AI? What's your background with that?
Paul Scharre
Sure. So I've got about 25 years of experience in the national security field. I was an Army Ranger, did a couple tours in Iraq and Afghanistan. And then I worked for a while in the Pentagon as a civilian policy analyst, actually led the group that drafted the Pentagon's policy on autonomous weapons, which is still in effect today. And then for the last 12 years or so, I've been at the center for New American Security researching, writing on this topic, trying to understand how is AI changing warfare and how do we avoid some of the bad scenarios you're talking about.
Jon Stewart
So this is perfect because I think it brings in the perspective of Paul, you've been in the military, you worked in the Pentagon. You understand the ins and outs. Sarah, you've been at the companies that are developing these products. So let's just start for the basics. And I'm going to say this for my audience. Obviously I understand how AI is used in the military. But Paul, very briefly, how does the military utilize AI and how is that different from their general practices?
Paul Scharre
So, I mean, it's not really the military is using it like any new technology that they're going to try to find ways to be more effective, more efficient, much less they use computers and computer software and computer networks today. So the military doesn't necessarily see this as something special or different, but really a productivity tool, just like I think a lot of people might use a large language model.
Jon Stewart
An optimizer.
Paul Scharre
An optimizer. You're just optimizing for something a little bit different?
Jon Stewart
Yes, slightly. But, Sarah, you. So as you were working at OpenAI, when they talk about optimizing, are they developing at these companies, are they particularly developing for military, or is the technology that they're using just being utilized by military?
Dr. Sarah Shoer
Yeah. So generative AI models are both dual use and also general purpose. They're dual use in the sense that they can be used for both civilian and military purposes, for good and bad, but they're also general purpose in that they apply to a variety of domains. So these are models that can be used in legal applications for software engineering tasks as therapy bots. We now know some people use them as. So it is not particular. They're not trained for particular use in the military, but. But nevertheless, the military, I think, has been a keen adopter in the last year. I think I'd also be remiss if I didn't add that even though most consumers now primarily interact with AI, probably through these generative AI chatbots, AI is in fact a toolbox of methods. It is not exclusive to large language models or generative AI. And the military uses a variety of different AI techniques, such as, for example, machine vision, which is responsible for object recognition, facial recognition.
Jon Stewart
So this is not just in the way I might use it, where I would go on and go. I'm thinking of visiting the Jersey Shore, recommend five different things, and then the AI will say, boy, that sounds like a great trip, because my AI is relentlessly positive, much to my chagrin, and then it'll list me a few other things. They're not just using it in that regard. They're using the other tools of AI, which I guess would be optimizing for anything from targeting to maybe supply chain or any of that. Is that correct, Paul?
Paul Scharre
Yes. You can think about maybe three different types of AI. One is something that's been around for decades that's really like handcrafted software written by humans. Good example of this would be a commercial airline autopilot. We kind of don't think of that as AI anymore, but once upon a time it certainly was. Military has a lot of things like that in radars and sensors and, you know, fighter aircraft, that kind of thing.
Jon Stewart
So already autonomous workings for some of
Paul Scharre
their machinery, maybe bounded autonomy, I would say, like a missile. There's lots of missiles that once you let that thing go, it's not coming back, but the autonomy is pretty bounded in what it can do. Then you've got machine learning systems that might have a narrow application. So they're doing computer Vision, as Sarah was talking about, military uses these to analyze satellite images, analyze drone video feeds. The military is collecting more intelligence than it can possibly put human eyeballs on. There just aren't enough human analysts. But the AI can help you then look through these images and find targets and identify things of interest. And then there are large language models which are these sort of like much more general purpose text kind of machines where you can feed in lots of data, you can have it analyze things. You can combine text and images and other types of data. And that's newer and the military is also starting to use that as well.
Jon Stewart
Now this is so in the public's eye because I want to see if I can fill in the gap between what the public may view this as and what the reality is in the public's eye. It is Skynet. It is, you know, robots, titanium robots that can regenerate themselves, that are walking autonomously over crushed human skulls and just firing what appear to be phasers at all kinds of different things. And you're saying actually it's the same shit that we're all using like at the office? For the most part.
Paul Scharre
I mean, for the most part somewhat different applications, but I mean it's the same types of things. And look, a lot of what the military does, to be fair, are back end functions, right?
Jon Stewart
It's logistics, it's personnel management, administrative and bureaucratic administrative.
Paul Scharre
That's like 95% of what the military does. Now there's a different component that is actually battlefield capabilities. But a lot of the military use cases are kind of mundane.
Jon Stewart
So the battlefield. So let's get to that because that's really where it appears this new controversy is, which is the battlefield. The controversy appears to be. And this began when Anthropic had drawn two red lines. The red line being that there can be no just autonomous kill chains. A person has to be in the kill chain. And that the AI cannot be used for general surveillance on the American public or gross surveillance on the American public. Sarah, is that understanding of the controversy correct? Are those the two lines that are drawn?
Dr. Sarah Shoer
So I make a slight adjustment there, which is that they're they special. They specified autonomous weapon systems, not kill chains in, in particular.
Jon Stewart
Okay, what's the difference there? Tell me the difference there.
Dr. Sarah Shoer
Yeah, so an autonomous weapon system according to the US definition and it's, it's important that I'm noting that it is in fact the US definition because different governments define autonomous weapon systems differently. Are weapons that can select and engage a target without human intervention. A Human can be in the loop, but it's not required. These weapon systems can function without human supervision. The language that's used in the DoD Directive 3000.09 is appropriate levels of human judgment. And Anthropic's position was that they don't believe the models are sufficiently reliable. I agree. And that for autonomous weapon systems, they need a human in the loop, which is essentially already US Policy.
Jon Stewart
So the US Policy is the human is in the loop, meaning. So let's, let's walk through a scenario just to understand a little bit of what we're talking about. Let's say the AI is used to analyze satellite imagery and different targets. A human will then get the results. A human wrote the program, I'm assuming, to analyze it. A human will then get the results of this data that has been analyzed, make their selections, and then give and okay to launch certain weapons that may in and of themselves be autonomous, meaning they'll guide themselves to wherever that target is. And is that a minimalist description of how this might all go? Paul?
Paul Scharre
Yeah, and I think that's right. I think conceptually the idea would be who's choosing the targets. If a human chooses the target, then you'd say the human is in the loop, the human's making that decision. If the AI is choosing it, or the AI is sort of recommending and the human's not really paying any attention, then you'd say, well, the machine is doing that. Right? And so one way to look at this would be after the fact. Something gets blown up and you say, well, who said it was a good idea to blow this thing up? If the answer is all the humans are like, I don't know, I didn't do it. Well, that's not a great outcome.
Jon Stewart
I assume that will generally be the answer.
Paul Scharre
Right. But like, right now, I think we're probably in the case. I certainly have no reason to think otherwise, where the humans are the ones making those decisions. Now, the AI might be helping to process information, helping to even maybe prioritize targets for people. But the debate between the Pentagon Anthropic is sort of a potential debate about where things might go in the future. I don't think, actually it's a debate at the moment about using a large language model to like autonomously make these life and death decisions on the battlefield, and then people aren't paying any attention.
Jon Stewart
Tara, does that sound, you know, is it that we're nervous that the computer will just decide on its own or that it will be wrong when it targets so let's talk about Iran for a second. Describe how a situation like that goes wrong and where the checks and balances are for that.
Dr. Sarah Shoer
So Claude in the Maven.
Jon Stewart
Okay, let me, Let me back you up real quick. You said Claude in the Maven Smart system.
Dr. Sarah Shoer
I can define my terms.
Jon Stewart
I love the fact that it's named after something you could name your cat. Hey, Claude. All right, so Claude is what?
Dr. Sarah Shoer
So Claude is the name that Anthropic gives to its flagship models, which is then used in the Maven Smart system. This is an AI enabled decision support system that does a variety of things, including some of the tasks that Paul mentioned, like helping speed up efficiencies and logistics. But it has also been responsible for targeting in Iran. We now have confirmation there as well. And if public reporting is anything to go by in Bloomberg and the Wall Street Journal and others, the first day that the, the production of 1,000 targets in Iran has largely been credited to the MSS, the Maven, the. The Maven Smart system.
Jon Stewart
Now, who makes Maven Smart System?
Dr. Sarah Shoer
Palantir does.
Jon Stewart
I just. Whoa. Did you guys just feel the room get colder? That was. Oh, the hairs. All right, so. So Claude, who is made by Anthropic, and that is more of a, an interface that we are accustomed to using. What is its role in feeding information to the Maven smart system, which is, I believe, a system we are less accustomed to using and is maybe a little less transparent. So tell us how that operates.
Dr. Sarah Shoer
Yeah. So the Maven smart system has been in use for several years now. The integration of Claude is I think, relatively recent. I believe in the last year, because Anthropic was able to gain access, go through this certifications to gain access to the government's classified networks. As far as we can tell, Claude right now has been used in targeting. And again, according to public reporting, it seems that it has been used for target selection and then also target prioritization. The Maven smart system itself is designed to pull in different data sources, so from sensors, satellites and such, and try then. And Claude then makes those disparate data more readable to the human analyst. So it boosts efficiency in that way. But it does also, reading between the lines a little bit, it does also seem to offload a little bit of human autonomy and decision making as well when it comes to that target selection and prioritization process.
Jon Stewart
Quite frankly, you know, when you brought up a thousand targets, because I have no context, I don't know what I don't know. So I don't know if that's an unrealistic amount of targets. I don't know if that's, you know, I'm understanding that there are target rich environments, there are target poor ones. Is a thousand in a day, you know, I don't know how they count it. Is that an unusual figure?
Dr. Sarah Shoer
Oh, oh yes, it's. I believe Centcom said that it was 2x the number of targets in the 2003 shock and awe campaign in Iraq. Just to have a historical.
Jon Stewart
So 500 targets in a day was, was shock and awe and this was a thousand. Now I think we have to also take into account Trump math, which generally is like this is the biggest crowd ever to see, you know, an inauguration and it wasn't. So how much of that is do you think is Trump math and how much of that is an astonishingly high figure?
Dr. Sarah Shoer
I mean it's being reported by Bloomberg, the Wall Street Journal and the Washington Post all without an extract Rags. I mean they're all taking it at face value and it's acting as though it is seemingly plausible. So there is no indication yet at this point that it's not accurate.
Brittany Momedovic
Foreign
Jon Stewart
stop paying for too much wireless just because I don't know. That's just what I do. It's how it's always been. That's just my company. Mint exists purely to fix that. Same coverage, same speed, just without the inflated price tag. You can change your coverage people. Mint is the premium wireless you expect. You know, your, your unlimited talk, your unlimited text, your data, but at a fraction of what others charge. And for a limited time you can get 50% off. 3 months, 6 month, 12 month plans of unlimited premium wireless. The only thing keeping you from doing it is inertia. Laziness. Don't be the beanbag chair, be the trying to think of something energetic. Pogo stick. Be the pogo stick. That's probably not right. Bring your own phone number, activate with ESIM in minutes. Start saving immediately. No long term contracts, no hassle. With a 7 day money back guarantee and customer satisfaction ratings in the mid 90s, Mint makes it easy to try it and see why people don't go back. Ready to stop paying more than you have to. New customers can make the switch today and for a limited time get unlimited premium wireless for just $15 a month. Switch now@mint mobile.com TWS that's mint mobile.com TWS Limited Time offer upfront payments of $45 for three months, $90 for six months or $180 for 12 months. Plan required $15 per month, equivalent taxes and fees. Extra initial plan term only over 35 gigabytes. May slow when network is busy. Capable device required availability, speed and coverage varies. See mint mobile.com. So you might say, Paul, let's say I'm, I'm working in the military. You work there and you've been researching this. Hey, Claude, I'm looking to take out all the radar installations in Iran. What would be a, you know, where would that, where would I do that and, and how quickly could I get it done? And then CLAUDE would interface with Maven, which has all the data that it's gathered from, I'm assuming satellites, and then it's translating the, that data that they understand through whatever intel they've gotten and they're going to place it into real world menus of what you could target. Would, would that be accurate?
Paul Scharre
Yeah. So let me like like explain what we know and then like what we could speculate reasonably about. Right, because like it is a little. Okay.
Jon Stewart
What we know and what we do not know, we do know before you
Paul Scharre
take a stab at it. So. All right, we know that anthropics AI tool, claude is deployed on US military classified networks. It's integrated through the Maven SMART system, which collects intelligence from different sources and it's been used by the US military in real world operations, including the operation against Venezuelan President Maduro and operations in Iran. And there's been some public reporting that's been used, Sarah was talking about in target generation and prioritization, like exactly how we don't know. So now I'm going to speculate. What might that look like?
Jon Stewart
Speculation alert for Paul.
Paul Scharre
Yeah, so that could look like you're talking to an AI tool saying, hey, help me plan this vacation to the Jersey shore. There's somebody as an intel analyst or a targeting analyst who's going to these tools and instead of having to manually go through all of this data that we have of where are the radars and what is the imagery of them, queries it in natural language. Hey, develop me for example, a prioritization of all of the radars that have already been hit and what the current battle damage assessment is of them. How much have they been destroyed? Didn't hit them again for a follow on strike. How much of them have not been hit yet? And let's put all that in a list, put in a database, let's prioritize it and then let's match it to weapons that would be needed to take out these radars. Different types of radars might need different Weapons. And then let's match that to available aircraft to help build a strike package that would eventually go to, like, an aircraft gets a set of targets and weapons that are assigned to that target. And so, like, the technology is sort of being used throughout that chain to make it just easier for people to access and process this information.
Jon Stewart
So we would be doing that anyway. It would just take longer.
Paul Scharre
That's right, that's right. Now we're talking about basically replacing the things that humans are doing with machines, speeding it up, making a lot faster. The US Military set thousands of targets in Iran. Having the ability to process that information at machine speed is very valuable for the military.
Jon Stewart
And then because it's Claude, you could say, and now give it to me like your Ernest Hemingway. And then it would give you the targets, in short, taciturn. It would just be very terse and go all there.
Brittany Momedovic
So.
Jon Stewart
So, Sarah, are we kidding ourselves then, that there is a line? What is the controversy? And. And how does it break down? What is Anthropic's argument here? Paul was saying earlier, it's really about the future as it stands right now. What is the controversy?
Dr. Sarah Shoer
So I think the controversy in itself is a little mystifying because it sounds like the contract negotiations went south due to some, shall we say, strong personality clashes. If you look at the contract between OpenAI and Anthropic, they're actually relatively similar, if not the same. They've essentially agreed. Both companies have essentially agreed to both
Jon Stewart
red lines, the contract that they have with the DOD or with Palantir.
Dr. Sarah Shoer
Ah, so that's actually. We don't actually know about that yet. So. So stay tuned.
Jon Stewart
All right, let's speculate some more, people.
Dr. Sarah Shoer
Yes, stay tuned. It's not clear what model now Palantir might use or if they'll have an array of different models that they can choose from.
Jon Stewart
So who makes. Who makes the. The contract? Is it. Does Palantir subcontract to anthropic or OpenAI, or does DoD, who is the leading role in integrating these companies together.
Dr. Sarah Shoer
So it's not unheard of, in fact, pretty common for companies to come together and actually combine resources to create a product, especially for defense purposes. So, for instance, the DIU trial and the DAWG trial, that the Defense Innovation Unit and also the Defense Autonomous Warfare Group have a call for building essentially attritable drones, and they've issued that call to industry, and companies have in fact responded to that call by combining resources and submitting joint proposals. So it's not unheard of for Companies to come into contract with one another and then to approach the Pentagon.
Jon Stewart
So they'll do that together. Palantir and Anthropic or Palantir and OpenAI will get together and say we've developed this package using, you know, our product, makes it more readable for humans. Your product makes it more. And so they'll bring it to DOD. And now so the $200 million contract that anthropic had, Paul, do you know what that they had a contract with dod, what was that for and for how long?
Paul Scharre
Yeah, I think so. This is where some of the details we don't really know. We know that they have an ongoing contract with DoD to deploy their AI tools on classified networks. We know they're being used through the Maven SMART system. But a lot of these details of like we don't normally get when defense contractors are working with the government. In fact, the silver lining to this whole thing is the only reason a lot of these details are coming out is because this whole relationship blew up between Anthropic and the Pentagon. Otherwise normally like they would have some deal about what the tools could and couldn't do. We would never know. And so that's like, you know, I think it's unfortunate actually that this sort of feud is spilled over between Anthropic and the Pentagon. But it is really the only reason that we have this kind of insight, which is even still pretty limited on exactly what the terms of use of these contracts are.
Jon Stewart
How opaque are these military contracts? I know the, you know, DOD as it's the only government, you know, agency that's never passed an internal audit. But how, how opaque are these and the $200 million that they use, is that over a five year period just to use their products on their classified networks?
Paul Scharre
Yeah, I'm not sure that we know. Oh, unless, unless Sarah seen more details
Dr. Sarah Shoer
than I have, you know, even as an employee I do not have access to contract details. It's, it's very tented in a lot of these companies and on a need
Jon Stewart
to know basis now these two are is, it is 200 million and on you, I mean to me that's an enormous figure. You know you're talking about the Pentagon budget in, in total obviously dwarfs that 1 trillion now as they're pushing forward. But still it's an enormous amount of money. Do they have it with, with, do they have that with different companies?
Paul Scharre
It's not, it's, I mean it's a lot of money for like a normal person. It's not A lot of money for either the Pentagon or for these AI companies. There's not, they're all dealing in billions and billions of dollars.
Jon Stewart
This is just walking around money. Walking around money to anthropic. OpenAI.
Paul Scharre
It's not quite money under the couch cushions, but like it's, it's not a massive amount of money. And the direct cost to Anthropic of losing this contract is not substantial to them relative to like the scale of AI investment that's happening right now in the AI sector.
Jon Stewart
How much of the contracts for like OpenAI and Anthropic are consumer based? In other words, I pay 1195 to get your latest model. And how much of it is corporate based and defense based? Do you guys have a sense of that?
Dr. Sarah Shoer
Yeah, I mean, I think OpenAI is right now, for 2026, projected to generate about $25 billion in annualized revenue. The majority of that is coming from subscriptions to its models. I think Anthropic is in a similar ballpark where they're on track to generate, I think, about $19 billion in annualized revenue. Anthropic is a little different from OpenAI in that it has prioritized enterprise, prioritized enterprise contracts earlier on. But there is, I think OpenAI's strategy, and this is public, has, has been targeted towards generating more enterprise contracts in the future. But I do think that the majority are still coming from, you know, individual consumers, developers.
Jon Stewart
Right. So the reason I bring that up is, is it does mean, because we're talking about they're opaque and they're tented, but it does mean that the consumer has some influence here in that the government is not their sole benefactor. It really is individuals. Anthropic says, I'm drawing a moral line. Whether that moral line is an actual line or it's already been traversed by whoever knows is a real moral line or not. And OpenAI says, I agree with Anthropic and we are drawing the moral line here. Autonomous weaponry and mass surveillance. Anthropic loses the $200 million contract. And that same night OpenAI announces, hey, we just signed a big deal with DOD. How real is that moral line that Anthropic drew? And how real is the backlash against OpenAI for suddenly appearing to have turned around and said, oh, they won't do it. Okay, we'll do it.
Paul Scharre
Yeah, I mean, look, the backlash is real and it's happened from some AI scientists. Anthropic vaulted after this controversy right to the top of the chart in terms of downloads in The App store that's happening. The dollar amounts for both these companies are relatively marginal compared to all of the other non defense investment. The bigger risk for Anthropic is going to be actions that the government is already taking against the company, labeling them a supply chain risk and going after them in that way, which would designate other defense contractors saying they can't use Anthropic's AI tools in the furtherance of their defense contracts and then other steps the US Government might take to retaliate against the company. They talked about using the Defense Production act to seize control of their AI models, for example. So those are probably like the bigger risk. It's not so much the dollar amount of the contract.
Jon Stewart
And Sarah, was it a real line? And it appeared to an outside observer that OpenAI immediately reversed their moral position. Given what you guys are both saying is a very small amount of money comparatively for their bottom line.
Dr. Sarah Shoer
I'm not sure if there is an actual reversal.
Jon Stewart
Okay.
Dr. Sarah Shoer
I do think that the military usage policies that are often designed by these companies are meant to preserve optionality for its leadership. There was a lot of backlash that, you know, I saw it in real time. A lot of the AI community still congregates on Twitter and OpenAI hosted an ask me anything on Twitter in response to that backlash, which I think illustrates, you know, the fact that the public can act as a pressure point on these companies. But what we ended up seeing as a result of that AMA was not necessarily a, an alteration to their previous policy, but adding more language to explain their already existing position, which in practice again doesn't seem to be all that different from Anthropic. But I think the communication strategy is maybe a little different. I mean, I don't know if it's the cultural fascination with the so called great men of history, but I really would resist any kind of narrative that tries to identify a hero and a villain in this story. I'm not necessarily sure that those are appropriate roles for either Anthropic or OpenAI. But to Paul's point, I think part of the sympathy that's been directed at Anthropic is because they have been the target of government overreach. And so I think it's possible to hold two ideas in one hand here, which is that, you know, Anthropic has been unfairly targeted. But at the same time, these two red lines that have been identified by both companies are probably inadequate and the public does not actually have to accept those two red lines as the threshold, you know, threshold of risk.
Jon Stewart
Imagine if you had some kind of, like, rewards program, you know what I'm talking about, Like a miles program, et cetera. But it's a rewards program that you pay rent through and then earn points for travel, dining, shopping, etc, 20, 26, if you're still paying rent without built. Come on, brother. It's a loyalty program for renters that rewards you for your biggest monthly expense, which is rent. With Built, every rent payment earns your points. You can redeem them. Flights, hotels, lift rides, Amazon purchases, so much more. It's. It's a loyalty program that you pay when you pay rent that helps you get out of the place you're renting for like a week and go have fun. And by the way, Built members can earn points on mortgage payments. You got a house, whether you got an apartment, whether you're sharing it with three of your knucklehead friends. I don't know what your life is. I don't know what you're doing. You can even redeem built points towards your next rent credit or even a down payment on a home. It's simple. Pay rent is better. With Built. Well, you're paying rent anyway. Get something for it. So join the loyalty program for renters@jointbuilt.com TW S. That's J-O-I N B I L T.com TWS make sure to use our URL. So they know we sent you. But are we kidding ourselves, Paul, in that? You know, look, if we think through history, there's. There's no human advancement that hasn't almost immediately been sought by the military for advantage, whether that advancement is sonic or chemical or biological. You know, Sarah mentioned two departments over at Defense that I would pretty much assume nobody who's listening to this has ever heard of. You know, I think we've all heard of darpa, but there are development groups, I'm assuming. You know, they said when they went into Venezuela, they used, you know, the Havana syndrome liquidation, new weapon that, like you pointed at people and their insides melt like we are throughout history. Any advancement that a human can think of, their military wing is immediately going to try and utilize for some advantage. No.
Paul Scharre
I mean, yeah, but look, two of the examples you gave there are chemical and biological. We do have regulations on how they're used. We have conventions banning chemical and biological weapons.
Jon Stewart
Right, but people still use them.
Paul Scharre
People still use them. Right, but not everyone. And they've been sort of, by many states, they've been treated as unacceptable weapons. Now you get some pariahs, you get Some outliers, you get people like Sad Musein or Bashar Al Assad who are going to use them still. But most states have given up those kinds of weapons, and I think it's better that they have. So the question with AI is not actually, are we going to use AI in the military? None of these companies are saying don't use AI in the military. The question is, should there be any rules? And if so, who sets those rules? Because the sort of crazy thing about the dispute about autonomous weapons is, as near as I can tell you, no one is actually saying we're going to use a large language model as an autonomous weapon today. That'd be crazy. If you have a large language model, write an email for you. You better fact check that email, because they do weird things sometimes. The question is, who gets to set the rules? And the Pentagon's answer is, we get to set the rules. We don't want these companies dictating to us. And these companies and many of the scientists working there, they have a lot of discomfort about how the technology might be used going forward in the military.
Jon Stewart
Sarah, when it says about who sets the rules, is it the company or is it the military? So we also, and I've read about this group, they are, they're called Congress. We don't hear much from them. They are, it's this group of generally older white men who, once they're past retirement age, enter into the legislative house. Is Congress. Are they utterly rudderless here? Are they just overmatched? Do they have any role to play? What, what can we expect and what should we expect from them?
Dr. Sarah Shoer
From.
Jon Stewart
That wasn't, that wasn't optimistic.
Dr. Sarah Shoer
You know, let's start small, start asking questions. You know, I am somewhat sympathetic to this idea that, you know, private AI companies cannot be setting the rules in foreign policy. But one of the issues that I see today, and I think this does track with a role potentially for Congress as well, is that AI companies are, in fact influencing foreign policy. It may not always be through the back end and through their contracts, through the Pentagon, but they're certainly donating significant sums to lobbying efforts and tying Those donations to U.S. china Tech Competition and arguing and arguing that a low or no regulatory environment is a requirement to, quote, unquote, be China. And they're supporting potential, you know, political campaigns that agree with that perspective. And so this conversation is in fact coming for Congress and they've probably better be equipped at the very least. And, you know, I actually think Paul may even be a better person to speak on this in particular, since he is, in fact, in D.C. and I would be curious to hear from him what the general reaction has been from Congress on this. On this issue. But I can say that AI researchers typically are very keen to discuss their work. And I've, in fact, never met a keener bunch of people who are willing to talk about, you know, the risks and opportunities related to AI models. So they are, you know, you can always send them an email. Yeah, I think they're pretty eager to have those conversations.
Jon Stewart
Paul? What? So, so what say you down in. Down in Washington?
Paul Scharre
Yeah, I mean, look, I'm here in Washington now. I could see the White House out of my office window here. I'm not going to pretend things are super functional in Washington, but I think we have seen government engagement on some of these issues. And there are a lot of tools that Congress can use to have oversight of the military and intelligence communities. One is passing legislation which may or may not be the right answer in some cases on the domestic mass surveillance stuff, maybe on the autonomous weapons, maybe not. We might want to maintain some flexibility there. But there's other things. Congress could hold hearings. Congress can get people from the executive branch.
Jon Stewart
Yes, they could.
Paul Scharre
They could, right?
Jon Stewart
That is correct.
Paul Scharre
They get people from the executive branch, come in and brief them, say, hey, what are you doing with AI? And if you want to keep it classified, Congress can do classified briefings to educate them about what's going on inside the military. Congress can use tools like procurement and acquisitions. Congress has the money. They are the ones that are allocating money to the military and intelligence community. And so that is a tool that Congress absolutely does use already to fund some projects and not fund others. And so there's a variety of tools that Congress has potentially to influence these things. And I think the model of who should be setting the rules, maybe it's our democratically elected representatives. That is probably the right approach.
Jon Stewart
Well, that's what I was thinking, but to Sarah's point, you know, look, these guys have more money than anyone right now. The money is in AI. Now, obviously, they're using a lot of those billions to build data centers that we have sort of no idea where those are all going, but 25 million here, 25 million there. Elon Musk puts 350 million into political campaigns. The amount of money that's flowing from these, from the tech sector is. Is like nothing we've ever seen before. Do you think that's had the effect that maybe the AI companies want, which is to regulate us, would be. They've portrayed it as national security risk. They've portrayed it as it would cause us to lose to China. Has that been effective? Or is it that they're overwhelmed by not really understanding the gut nuts and bolts of AI?
Paul Scharre
Like, you mean Congress not understanding the nuts and bolts of AI Congress.
Jon Stewart
That's right.
Paul Scharre
Yeah. I mean, I think there's a lot of. I've actually been super impressed when I speak with. I mean, you can always find video clips online of some Congressmember not understanding something.
Jon Stewart
I would use them on the show.
Paul Scharre
Yeah. You know, I mean, like, okay. But I think, like, I've been impressed when I speak with members of Congress and their staffs, how knowledgeable many of them are about the technology and what it can do and its limitations. So I think there's always work to be done in terms of improving tech literacy in Washington.
Jon Stewart
Did.
Paul Scharre
But I think some of the bigger challenge is just sort of getting over the hurdles in passing legislation and getting agreement, whether that's around federal regulation of AI or data privacy or social media or other types of. That's actually really hard for Washington to do, to pass legislation on these kinds of issues.
Jon Stewart
Sarah, you spoke of this earlier. It's this great man. Here's why I'm very nervous. I've met a couple of these folks, and they do not seem particularly enamored with humans. I don't want to say outright misanthropic, but, you know, Peter Thiel was asked famously in a conversation, you know, should humans continue? And, you know, he paused, I think for a pretty considerable amount of time before he went like, well, you know, and. And transhumanism. I once asked Sam Altman about the disruption that AI is going to cause to our workforce and that small amount of time in which it's going to cause it. And his response was just. He literally just looked. The question was five minutes long, and he just went, we'll be okay. You know, how concerned are you with. With these great men and how great they actually are and what is their connection to. Do they understand the damage that they also can do, or are they megalomaniacs?
Dr. Sarah Shoer
Well, I mean, I can't look into anyone's, you know, heart and mind, but I would say that if they're able to cause harm, it's only because they are powered by immense, immense wealth and the high valuations of these companies and also by institutions that allow for corporate donations and excessive individual donations as well. So they're essentially enabled by our current institutional structures in terms of whether these companies discuss the Downsides, I mean, I joined in 2021, I left in 2025. There was a period where I think that was the, you know, dominant topic, you know, topic of discussion, right? Are these tools actually going to increase productivity? Are they going to replace tasks? Are they going to replace workers? Can they enable the proliferation of potentially weapons of mass destruction? And there were, there was testing and evaluations that began to try and answer those questions. So I think certainly the researchers at these companies have tried to make a concerted effort, but these companies are also complex organizations and they're always factions that are butting heads, right? Some people do prefer a low to no regulatory approach. They don't want to see state legislation. They prefer everything at the federal level. And then there are some who are at these companies who are actually quite supportive of state level legislation. So it really depends. I mean, I think of OpenAI and Anthropic and frankly other companies as often going through eras where certain factions went out over others and that's what ends up setting the, the cultural mood of the company.
Jon Stewart
Do they understand the weight of what they're making? You know, I can't help but go back to Oppenheimer and I, you know, when you have something that, that looks like it could be extermination level type technology, positive and negative, I mean, if we split the atom one way we get energy that can power the world. If we split it this way, you can blow it up. And we all know which one we tried first. And it felt like the people who were making that weapon did it under the crucible of the Nazis. And so they developed it with this idea that, well, if the Germans get it, we're all done for. But it was clear that they at least felt the burden of that. Paul, in your experience, are they feeling the burden of this? Because what Sarah's talking about is, well, they did go through all that testing. We don't really know what the results of it was. And they seem to have gotten past that reservation.
Paul Scharre
I mean, the AI scientists and engineers that I speak with, particularly those in the frontier labs, are very concerned about AI risk. They, I think, understand better than anybody actually the downsides of the technology, the way that it could be abused, the way that it could just do sort of strange things that might be surprising. I think one of the challenges here is there are incentives for the companies to move fast to ship their products because there's a sort of perception of a winner take all dynamic in the marketplace that we have seen in other tech industries and operating systems. Handsets well, yeah, I mean, in a way, right. A sort of commercial race to dominate the marketplace. And that does drive incentives. And these companies need a lot of money to build the data centers to train the AI. So I do think the individuals take it seriously. And I think some of the companies, I mean, if you look at what Anthropic just did, they sort of stuck to their guns on this decision in a way that is going to be costly for the company. How costly? I think we just don't know. But they decided to do that. So I do think the companies take these issues pretty seriously.
Dr. Sarah Shoer
And if I can also add, I mean, just at the risk of potentially misspeaking the testing and evaluations that were, that were done and continue to be done at these companies, they are often released publicly. But, you know, of course, in certain areas, like, you know, cbr and so that's chemical, biological, radiological and nuclear testing, and then also cyber, there are greater restrictions placed around what can be shared with the public. But there are even reports, summary reports about what that testing looks like. And then a lot of the benchmarks that are used by AI industry are in fact publicly available. It just so happens that testing and evaluation of these large language models is still in a relatively nascent phase. And it's not always clear what the best way to test these models is, are if what we're trying to do is use them as proxies for social impact or risk.
Jon Stewart
And is that, you know, the famous one is now, you know, if you remember the movie War Games, and it was, you know, the first sort of kind of dystopian look at what would happen when computers take over was the Matthew Broderick movie from when I was a kid. And it was about a nuclear war game gone wrong. And the computer just started launching, you know, nuclear weapons at all the different countries. And at the very end, the computer said the only way to win is not to play with AI. Apparently it was more apt to launch nuclear war than humans or standard computers. What do you know about that testing? And, and is that apocryphal or is that. Did that really happen?
Dr. Sarah Shoer
I mean, it did really happen. You know, I think a variety of researchers at academic institutions have now managed to replicate the findings. The models have a tendency to escalate more aggressively than humans would. And it's not really clear why the models do that. One theory is that in the training data, AKA the intranet, political scientists have a tendency to study wartime escalation rather than de escalation. So that may influence how the models respond to these war game type simulations. But I mean, that in itself is of course a cautionary tale around using these models for approving the use of force or for decision making or frankly, even for war gaming and simulations.
Jon Stewart
Is it possible, Paul, that AI, because of how adept it is at creating these targets and all these other things that it actually made going into Iran more appealing, that before the age of AI, we might have been more circumspect about the type of attack that, that we launched? Are we seeing barriers to military action fall because of how quickly these models can, can they bring a sense of false confidence?
Paul Scharre
I mean, I don't think today that's true. Like, I don't think I was a factor in President Trump making this decision. I think it was based in large part on the US strike against Iran last summer against the enrichment program being very successful and limited. And then the raid against to grab Maduro being very successful and limited. And this sort of like, okay, having a couple, perception of having a couple of wins under his belt. I'm using the military, it seems to
Jon Stewart
be effective, no downside. Sure.
Paul Scharre
Right. So I think those are probably bigger factors. I think what you're describing could be a risk going forward. Right. So one way and this could be a risk is some of the things that militaries count and try to calculate when they measure military power are things that you could see. And you can count, you can count how many tanks somebody has, how many airplanes, how many ships. Then there are some things that matter a lot that are hard to count. We see this unfolding in the war in Ukraine, the morale of the troops on the battlefield. The Ukrainians are fighting for their homeland, the Russians are conscripts, they don't want to be there. The leadership, the quality of the unit, cohesion, those things matter a lot, but they're really hard to measure. So one possibility going forward is you could see a world where as more and more military power gets embedded into software and data and AI, it's kind of hard to measure that. It's like, well, we have this AI and it's amazing and it's wonderful and ours must be great. And there's this. It becomes harder for militaries and countries to sort of gauge what their relative level of power is. And you might see more miscalculation. You might see countries sort of assuming, well, we have this wonderful technology and we can win and the world will be over quickly and we'll all be home. And it turns out not to be true. Countries have Made this mistake before. That's what happened in World War I, right?
Jon Stewart
We've made it quite a few times.
Paul Scharre
Humans have done this, so that's not a, that's, I think that is a possibility that could happen, but we're not there today.
Jon Stewart
Sarah has anybody studied the confidence? You know, there's a certain thing in bars, like there's a beer courage you get. You, you get a couple of shots and you get a couple of beers and you're like, you know, it turns out I'm a tremendous MMA fighter and I think I'm gonna, you know, you get a weird confidence from alcohol, I find you get a weird confidence when you use AI. When you use those models, you tend to be much more assured in your decision making because you feel like you have this kind of infallible being behind you. Has anybody studied AI confidence in decision making? Because I, I feel it when I use it for the, the mundane tasks that I do.
Dr. Sarah Shoer
You know, I'm not sure if, if I've seen anything like that. That's a really interesting, that's a really interesting point. I mean, I think what you're referring to, I've heard some people talk about chatbots or frankly, any type of statistical analysis that's used to make decision making as applying this mathematical veneer, right? It makes us feel better because it's therefore objective and it removes the human qualitative or subjective element to it. The issue that I just keep going back to is, of course, that these models are not always going to be reliable because they are in fact, statistical prediction machines. I mean, they're useful, don't get me wrong, but they are inevitably going to output something that is incorrect. And so being able to keep appropriate human judgment and to create a system in such a way that people do not abandon their critical thinking skills is a very important facet, I think, to any type of human machine teaming that we're seeing today in military AI integration.
Jon Stewart
Is that something the military is concern with? Paul? Because, you know, in looking at it from like, let's say from an educational standpoint, there's been a lot of studies that show that when kids start using this, their ability to do that, to think critically, to reason, and all that falls, that it becomes this crutch that when utilized, you no longer develop those kinds of skills and, and ways of thinking. Does this become a crutch for the military to use? And, and the second part of that question is, is are we ignoring this whole other area which is, hey Claude or hey Maven or whatever it is design me five nerve agents that the world has never seen before. You know, is that another usage that we're not. So far we're only talking about chain of command. Is there a whole other area we, we're not even really thinking about?
Paul Scharre
Yeah, well, that is certainly a risk. The potential for AI to enable biological weapons and to maybe even lower the barrier to countries, to non state groups, to terrorists to do so. Maybe not today, but that's a concern down the road. I think in terms of military uses, I think the military is actually pretty keenly aware of for people in uniform. They understand the responsibility that they have. Okay, if they're going to launch this missile, they own where that missile goes. And I think there's a couple of concerns. One would be making sure that they really understand this AI system. Like what is it going to do? Is it going to do something strange? Is it going to fail? How's that going to work? Ensuring that there's human responsibility and accountability I think is actually quite important to the military. That's part of the military ethos. But it's challenging for a lot of these AI systems because it's not like a traditional computer program where okay, there's an accident, you go back and you say, oh, this is the line of code that caused the problem. Now the answer is embedded in this massive neural network with billions of connections. And you're like, why did it do that? I don't know. And so it gets into these issues of trying to evaluate the model's performance. What are some conditions in which it might be biased in certain ways they tend towards sycophancy, towards basically telling you the answer that it thinks you want to hear. Well, that could really be a problem in some national security applications. You're an intel analyst and you're like asking some questions and it's like, well, ye, you know, this is what I think you want to hear. Right.
Jon Stewart
So that was Napoleon's whole issue. They were like, sure boss. Waterloo, what a great idea. You should go there. Yeah. Now this is going to sound ridiculous, but does it do like what it does with us, which is, would you like me to give you a 10 day bombing plan? Would you like me to add in other targets that may seem ancillary but might have military? Like, is it, is it it, is it that casual when it's describing, you know, what it wants to do next and how quickly does it do that?
Dr. Sarah Shoer
I have never used the Maven SMART system and so I don't actually know what, you know, what the Personality of the chatbot is or is that what
Jon Stewart
they use Claude for?
Dr. Sarah Shoer
Yeah. I mean, you bring up an interesting point though, right? In that these models can be fine tuned with different personality personalities to be either more acquiescing, less acquiescing. We know that users, of course, like to be fawned over a little bit, but it's possible that it's not presenting information in the most neutral way out there. We just don't know publicly, I don't think.
Jon Stewart
Right. Do you know, Paul?
Paul Scharre
No, I don't know. It's an interesting question. I think one way to think about these models is they're sort of role playing. They're playing a role that's in their training data and then that can be fine tuned by additional training that they get from the companies. And so that's why you get this sort of personality, different personalities among the different models. So it's an interesting question of like the ones that the military is using or the intelligence community, what are they sort of trained on? And are there hidden biases that might be kind of subtle that are hard to detect? I mean, that's, I think, a, a,
Jon Stewart
a difficult problem or not so hard. And I just got a chilling feeling that they're training it on the heset and so they plug something in and the model just pops back, hell yeah, let's do this. I told you about my invention, the crumple. The Crumple. It is a topographical blanket for dogs, but not the same topography at each time. Every time you throw it on the ground, it changes its topography. It is an amusement park for your dog to find a place of comfort and warmth, but also with interest. It's not the same old, oh, right, this is where I put my right paw. And this is where I curl my butt. No, it changes every time it gives them. It's like visiting. It's like Epcot. It's an Epcot center blanket for the dog. And have I started this business yet? I have not. That's right. To the great dismay and disappointment of our audience and, and maybe humanity writ large. I have not started my crumple business. And I'm going to tell you why. It's too hard. It's nerve wrack. I don't know how to do this. It's daunting. But, you know, you got Shopify here, makes it easy for people, people you can get started with. They got a design studio, hundreds of templates, help you build an online store. It can match your style. You can, you can do this smoother they also have 24 hour customer service support, world class expertise and everything. It's the commerce platform behind millions of businesses around the world and 10% of all E commerce in the United States. It's time to turn those what ifs into with Shopify today. Sign up for your $1 per month trial today at shopify.com TWS go to shopify.com TWS that's shopify.com TWS.
Paul Scharre
So these are like some, I think some really difficult problems with the technology that we've got to find ways to work through, to use it in ways that are safe and effective. And I don't think there are easy answers. I think the technology has some strange and new challenges associated with, with it.
Jon Stewart
Sarah, you, you, you strike me as having a really balanced but also nuanced view of this. What, what keeps you up at night is, is there something about this that, that you think about as particularly challenging?
Dr. Sarah Shoer
Yeah, there, well, there are many challenges. Let me see if I can narrow
Jon Stewart
them down or throw them all out there and we'll go through them one by one.
Dr. Sarah Shoer
Right. I mean I, I think about the, the challenge related to global governance. I mean for over a Decade now, over 90 plus member states have been meeting at the United nations to discuss regulating or the possibility of regulating or even introducing a treaty instrument that would regulate lethal autonomous weapons systems. But because of the nature of the forum at which these discussions are taking place, it's a consensus based body, it's at the Convention on Certain Conventional Weapons. It's very unlikely that a treaty based instrument is even possible in this, in this space. I mean you can think about how hard it is to, you know, pick a restaurant with you and your five friends. Now imagine that you have 90 plus governments trying to decide that could kill all of us.
Jon Stewart
How have they been able to do it? Why can't they use the model that they used for atomic weapons?
Dr. Sarah Shoer
Oh, I see. Well, so I guess there are a few reasons for that. So the Convention on Certain Conventional Weapons, it's really in the name it is talking about conventional weapons and autonomous weapons. The conversation around them has really focused on trying to preserve meaningful human control to discuss whether that's even possible, whether they can actually discriminate between combatants and civilians. And, and if they can in fact discriminate between combatants and civilians to an extent, then they technically could be legal under international humanitarian law, but militaries would still need to abide by the existing international legal order and international humanitarian legal principles. And the good thing about this particular forum is that, though, you know, regulation with teeth is probably off the agenda, most states have been able to have. Have consented and reaffirmed the norms around international humanitarian law as applying to autonomous weapons systems. So that's, I think, also a silver lining as well.
Jon Stewart
Has anybody kind of gotten it right? And, Paul, I'll ask you, because, you know, maybe you see ways through this from being in Washington, but, you know, has the European Union done a better job with this? Has. Has any governing body, has any international body, Is there any pathway here that you see that could help establish at least the beginning of guardrails?
Paul Scharre
I think that actually the best avenue we have is starting at the level of AI hardware and then sort of building guardrails domestically, eventually globally, kind of from the ground up.
Jon Stewart
Explain the difference between hardware and the software.
Paul Scharre
Right? So the thing about these AI systems that it's kind of amazing is they require massive amounts of computing power to train the most capable models and to deploy them at scale. Now, you can make smaller models that you can deploy on a laptop, for example, or some other kind of edge device, smartphones, but they're not as capable. But the most advanced ones are going to be really big. They're going to have to run in the cloud. They're going to need really advanced chips, and to deploy them at scale as a society, you're going to need a lot of these really advanced chips. Well, these chips are made in one place on earth.
Jon Stewart
Taiwan.
Paul Scharre
Taiwan. Now, that does not, on the face of it, seem great, that it's an island 100 miles off the coast of China that China has pledged to absorb by force if necessary.
Jon Stewart
But it is considered a drawback.
Paul Scharre
I think this is not the best geographic position. However, these fabs that TSMC has in Taiwan, where the most advanced chips are made, depend on technology from three countries in the world. Japan, the Netherlands, and the United States. And without that technology, they cannot make these advanced chips. And so that sort of starting at the hardware level, that actually is like a really narrow choke point to begin to then control the technology.
Jon Stewart
So the deal we just made with UAE to give them the chips, the previous concern had been that they would then sell the chips to China. Did that just blow a hole in the net?
Paul Scharre
Well, I mean, the bigger question is, like, what is the global diffusion of this hardware look like at the tail end of the Biden restriction? Literally the last week when they were in office, they dropped this very complicated rule called a diffusion rule, that basically would take US expert controls on the most advanced chips to China, which we've had for several years now, started under the first Trump administration and expand that globally. And it's kind of tiered system where depending on which country you're, you could get so many chips. It was a little complicated. Trump administration threw that all out the window. But I do think that, like, the chips themselves are a way that we could begin to shape who gets access to the hardware, who can build the data centers because they need these chips to do it. And that's a hook for guardrails. Right. So you could say, all right, you want to buy all these advanced chips. I want to see your domestic regulation surrounding making sure that people aren't going to use these chips to make a biological weapon.
Jon Stewart
Like we did it with enriching uranium and the things that you would need to be able to do that.
Paul Scharre
That's actually not a bad analogy here. Right. And we're, okay, you can get uranium for peaceful, civilian nuclear purposes, not to make a bomb. And we found ways to separate those
Jon Stewart
two, not to enrich it to that level. Right, right.
Paul Scharre
So, like, the idea would be the same thing. You can use these chips for peaceful uses, basically most everything, but you can't use it to make, like an offensive cyber weapon, for example, and put some guardrails around how the technology is used.
Jon Stewart
Right. And inspection. Sarah, is there any fear that, like, by the time we figure this all out, quantum computing is the new standard, and that's pushed us in. So by the time we figure out, okay, these three chips are crucial to any ability to do that, and then somebody else comes in and says, actually that's not state of the art anymore. Are we moving so quickly that suddenly quantum computing is the power that's necessary to drive these, and that's a whole different can of worms.
Dr. Sarah Shoer
I think you're now learning in real time that AI researchers aren't necessarily experts in quantum computer computing. And I am the worst person to answer that question.
Jon Stewart
We just, we. Because the reason why I bring it up is I just read an article about it, and I have no idea what it is. They were. Someone was describing that actually quantum computing is going to be wildly preferable to large language models. And I was unable to understand. Understand the difference. Is there knowing that you're not experts in this, Is there a sort of remedial version of what the difference might be? Paul, do you have any idea about this?
Paul Scharre
Yeah, I think so. So we are seeing some progress in quantum computing, I think. I don't think it's going to like, change this picture in AI for A couple reasons. Okay. One, the quantum computing will become valuable over time for like some very niche kinds of computational computation, but not necessarily everything. And I don't think what large language models or other large neural networks are doing today. It's also like the case that we're not seeing in quantum computing this kind of really rapid exponential growth that we're seeing in AI. So right now the price, performance, the performance per dollar of AI chips is doubling about every two years. It's like really growing very, very quickly. That's not true.
Jon Stewart
That's the productivity of it.
Paul Scharre
That's like the efficiency of it.
Jon Stewart
Okay, right.
Paul Scharre
So that's really powerful. That's what's allowing this massive growth in AI. It's one of the factors data and better algorithms are. Factor two, we're not seeing that kind of exponential growth in quantum computing. It's really hard science. It's like difficult physics. It's much more traditional science where people are making incremental gains. I think we're going to continue to see progress, but I, I'm a skeptic that we're going to see this like, transformative leap ahead in quantum computing and say the next five, 10 years the way that we're seeing with AI right now.
Jon Stewart
So in summation, the drama that we're seeing between anthropic and open AI, that's really the soap opera story and there's not necessarily a lot of there there. It's the general competition between these companies that are going to try and establish primacy in, in the realm of AI models. Military application is just one element of the revenue streams that they're pulling in there. The real sort of where you guys are really looking at is that interface between who are we going to end up trusting more? The humans that are developing the AI models? The humans that are running and integrating the AI models or the models themselves? Would that be kind of where the real tension is going to play out?
Dr. Sarah Shoer
I mean, I think it's fair, but I would just add that it's not going to be only one technical. It's not, it's not only just going to be safety through the technical stack, or only safety through the law or safety through regulation policy. Right. It is truly going to be an all of society effort and in part because AI, again, general purpose and it can be used across a variety of applications. So a one size fits all approach to safety is probably not going to work.
Jon Stewart
Is it, Is it akin to the battle against climate change? And if so, that we haven't done a great job there. So is that. Does that give us a pathway not to follow?
Dr. Sarah Shoer
I mean, I think any pathway towards AI governance is going to be through cooperation. And I don't want to be overly cynical here. And so I'll try and draw on a positive. A positive example.
Jon Stewart
No, go full. Go full cynical.
Dr. Sarah Shoer
I'm going to go. I'm going to give you one positive example. Just one. I think I've been. There's plenty of cynical.
Jon Stewart
Come on, Sarah, hit us.
Dr. Sarah Shoer
So under the previous administration, there was a. They launched the declaration on the political declaration on military, on military use of AI and autonomy. And that was a voluntary declaration with principles and norms and around 60 countries signed onto it. And in that declaration, it really centered international humanitarian law and also civilian protection. Those conversations can resume. Those diplomatic conversations can resume. Really what's stopping right now is political will, and that that process can in fact happen alongside the existing UN processes as, as well. So there isn't really a way out of this that doesn't involve talking a lot to other people, but that there is something there to build on.
Jon Stewart
Is the cynical version of that, that international norms and rules seem to be in disfavor with the current sort of, I guess, what you would call large power politics that seem to be playing out. Would that have been. Is that your downside?
Dr. Sarah Shoer
Yeah, I mean, I, I think that's probably fair.
Jon Stewart
We're dancing around lots of things.
Dr. Sarah Shoer
It's. It is. But, you know, at the same time, people can continue to demand this through Congress. We mentioned Congress earlier. I see a role here potentially. Right. If they want. If they want to do something, you know, if they're, you know, if they have some.
Jon Stewart
I'm counting on your students. Dr. I'm counting on your students at Berkeley to be able to come up with a. Yeah. A way through it. Paul, what, what keeps you up at night and, and give us a nice balance between cynicism and optimism on, on the way forward that you see.
Paul Scharre
Yeah, look, I think the reality is this technology is going to introduce a lot of challenges. How is it used by the military? What are some of the risks in cybersecurity? We talked a little bit about the risks of AI empowering biological weapons. There's a lot of risks of the technology, and that's just in the sort of national security space, not to mention things like job dislocation. I think my takeaway from this fight between Anthropic and the Pentagon is that these decisions are too important to be left up to any one of these entities on their Own right for profit companies or the government deciding on its own. I think we all have a stake in this world that we're living in. Not just on some of the civilian uses, but even military ones. All right, so, okay, we're not the ones building the killer robots, but if people build them, we're going to live in that world. We do have a stake in what that looks like. And so there's democratically elected representatives. All of us, your listeners, have a role to play in weighing in on this debate. And if there's a silver lining of sort of this controversy we've seen in the last couple weeks, it's. It. What would have been a private conversation is now happening publicly. Kind of messy, a lot of personalities involved on all sides, but it's airing this issue and then we're all sort of debating, well, what should be these red lines here. Hold on a second. That's a good conversation to have, and I'm encouraged that we're having that discussion.
Jon Stewart
Fantastic. Guys, thank you so much for joining us on this.
Dr. Sarah Shoer
Thank you for having me in.
Paul Scharre
Thank you. Thanks for the discussion. Been great.
Jon Stewart
Should I. Did I take the wrong. Should I not be calmer?
Gillian Speer
I. Yeah, my hair is still on fire.
Paul Scharre
Still on fire.
Gillian Speer
Sorry to say it did not calm me.
Jon Stewart
Did it help at all that. Because they were still putting it through a process that they were. They still wanted to filter the problem of AI through international cooperation or legislative process or, you know, government incentives for that, rather than saying, look, we're at one second to doomsday, somebody's gotta step in.
Gillian Speer
I think I was kind of calmed by the idea that, like, we have these models for, like, other sort of disarmament that have worked. Like what you said about nuclear weapons, like the nuclear arms deals, but also like the Iran deal, the one he
Jon Stewart
used was biological weapons and chemical weapons.
Gillian Speer
Yeah, exactly. I think that that was encouraging to think, like, we have these frameworks that we could look at as models and like, this isn't totally uncharted territory. And then I think I'm just reminded that we're not doing that. So that's where the nerves come back in.
Lauren Walker
I also think freaking people out too much is not conducive to getting them to act. As we've seen with climate change, I think it's really hampered people's ability to organize. So I did appreciate that. I also really appreciated this is just a personal thing, but over the weekend, I did notice a lot of people framing anthropic as the good guys, which I thought was really odd, considering all of the reporting coming out about these Iran strikes, about the Maduro capture that's already been used.
Gillian Speer
Yeah.
Lauren Walker
And I really appreciated just that we. Someone who's worked at one of these companies, like breaking down that it's not a binary, that there's so many considerations for these people to make. And as you've said, they're not, you know, perfect actors. Everyone makes mistakes. The technology itself makes mistakes. So I just appreciated that nuance.
Jon Stewart
I also like that what they talked about was, you know, in terms of the usage, it really is in some ways a kind of cousin of the way that we use it in that it's there, it's just collating data more quickly and spitting out those pleasantly formatted, you know.
Gillian Speer
Yeah, that did not make me feel better.
Jon Stewart
Here's five great places you could bomb.
Brittany Momedovic
But John, how do you use AI?
Jon Stewart
Oh, like I'll go in to AI and be like, okay, I want to find the best, like, who's got the best pizza in. Blah, blah, blah. Like, generally I use it for like those types of recreational. Yeah, like, sure, I want to try this sport. You know, what's the stuff I might need? How would it be hard to get into it? Like that, sort of. And it's, it's effective. You know, here's five places you could go to get started with, you know, paddle tenant, you know, that kind of thing.
Gillian Speer
And then the government asks what's the best pizza? And then bombs those places.
Dr. Sarah Shoer
And I don't know if that makes me feel better, you know.
Jon Stewart
No, but here's. So here's why, though. Here's. Here's what I'm going to say. So in the same way that I look at autonomous cars as like dystopian, almost everything I've read about it is that it would make it safer. That human error is actually at a higher fraction than the other. Now, obviously letting it just make decisions on its own without any kind of interaction makes me uncomfortable. But I guess the point is, like, how great are we actually at driving?
Gillian Speer
Not good.
Jon Stewart
Because we bought we bomb randomly before computers ever happened. Like, how. What was our track record on bombing? Like, not so great. Like, we dropped two atomic weapons on Japan. Would the computer do worse than that? Like that. That's my only point is like, are we elevating humanity to a higher status than we've earned?
Gillian Speer
I think that the issue is that it makes doing these things so much faster. So maybe it would have dropped five atomic bombs on Japan. I don't know, you know, like. But if we were to look at the. The charts, that seems to be the way that it would go.
Dr. Sarah Shoer
Oh, God.
Jon Stewart
Right.
Lauren Walker
Also, in the Waymo case, there was reporting recently that people in the Philippines were intervening. You know, like, we're just not there yet.
Jon Stewart
It. Oh, really? Okay. Yeah, I didn't. I didn't know that. Yeah, I'm assuming that. I guess what I was saying is sometimes in the battle between man and machine, we tend to look at man a little bit more favorably than maybe man has. Has earned. But I. I absolutely get that. And again, to that. To that point, one of my biggest fears about AI continues to be what appear to be the pathological personalities of the people that run those companies.
Brittany Momedovic
Same.
Lauren Walker
Yeah, I was thinking about that in terms of the. The attitudes and the personalities of these chatbots when you were talking about that in the conversation. And just remembering, like, six months ago, though, Grok or whatever company you know is above Grok for Elon. Yeah, yeah, yeah. Made a contract with the government for, like, 42 cents for like a year and a half. They could integrate Grok into government. And apparently there's, like, posters around DOD with Hegseth's AI generated mug saying, we want you to use AI. They really want to get government hooked on their product. And I just imagine someone in government being to Gillian's point, a little bit like, okay, there's flooding in Texas. What do we do? And they're like, well, Hitler is the best. Best person to deal with this.
Jon Stewart
You know, you're thinking that they contracted with Mecca Hitler as opposed to just normal Gro. No, you're right. That. And. And those guys manipulate algorithms, and they are ideologues. They have a. A lot of them are transhumanist. Like, they are leading us down a path that is. That is not favorable, I think.
Dr. Sarah Shoer
Yeah.
Gillian Speer
When Sarah said, I don't know, the personality of Maven, like, I. My stomach dropped. I was like, oh, my God, we can't be talking about the personality of. Of weapons.
Dr. Sarah Shoer
That's so dark.
Brittany Momedovic
Or when she talked about Sam Altman's heart and mind, I was like, does he have either of those things right?
Jon Stewart
Hearts or minds? But it is like, I don't know, the personality of the Palantir generated war autonomy system. Some wild man and. And not going away. But I loved how measured they were, and I. I loved how they sort of helped us through there. Brittany, what. What. What do the people have for us this week? Week?
Brittany Momedovic
Sure, John. We're still going to get Greenland, right?
Jon Stewart
Oh, I think we already have it we've already won. Like everything else in the Trump administration, we've already. Not only. It's like with the Iran war, we've won and we're doing more. It's. We are Schrodinger's country. We exist in all different. We have Greenland and don't have it at the same time. But they respect our unique and, you know, unparalleled power. And so absolutely we have it and don't have it and could do whatever we want with it and won't because of our largess.
Dr. Sarah Shoer
And I don't know, it's like how
Lauren Walker
the Iran war is almost complete but also could go on for as long as it takes.
Gillian Speer
We live in this middle space. Yeah.
Jon Stewart
Almost complete and never done. Yeah. And we are going to only stop at unconditional surrender. And we've already stopped. We are. We are Schrodinger's country. And it is only the beholder that determines where we are on the existence plane.
Gillian Speer
We obliterated the nuclear program, but they're one day away from it. You know, hard to keep up, guys.
Jon Stewart
Very hard to keep up. Is that it for them?
Brittany Momedovic
One more.
Jon Stewart
One more.
Brittany Momedovic
John, why does everyone ask you where to get pizza?
Jon Stewart
Because I am considered one of the world's leading and this is recognized around the world. Any, any of the larger pizza conglomerates. The pizza they, they recognize now. You know what I think because of that rant I did on deep dish pizza in Chicago. I think that's the only. Oh, and we did something on Trump eating it with a knife and fork. And so those two things, you know, there is no real accreditation other than port noise rating system.
Dr. Sarah Shoer
Yeah.
Jon Stewart
Four pizza. So oftentimes non experts are elevated.
Brittany Momedovic
Sure.
Jon Stewart
To that position.
Gillian Speer
I mean, little do they know. You're just asking. The AI I was about to say reveals earlier.
Jon Stewart
Can I tell you the truth? Like my world there is so small. I go to Joe's on Carmines if I want a slice and I go to John's on Bleecker if I want a pie. And that's kind of my like, as you guys know me, my world is small.
Dr. Sarah Shoer
I am, you know what you like.
Jon Stewart
I am not a man who is out there. It's the same clothes I have eaten. I shouldn't even be telling you guys this. I eat the same lunch every day when I go into work at the Daily show. And I've done it since I've been back. The exact same lunch.
Dr. Sarah Shoer
Well, what is it?
Jon Stewart
I'm embarrassed to say.
Dr. Sarah Shoer
Oh, no, come on. You have to what is it like?
Lauren Walker
Girl lunch.
Jon Stewart
What's a, what's a girl lunch?
Lauren Walker
A girl dinner. Just like little bits of everything. I'm just very curious, trying to prompt you.
Brittany Momedovic
Yeah. We are not ending the podcast until
Jon Stewart
you tell us that's what a girl lunch is, is little bits of everything. All right.
Dr. Sarah Shoer
Yeah.
Lauren Walker
You don't have to really cook.
Jon Stewart
Yeah, no, I, I order, I don't make it. Let me, let me just be very clear. When I go to work at the Daily Show, I don't, I don't cook. Yeah, I call out and I get a bean and cheese tostada.
Gillian Speer
Okay, we have to stop talking about lunch during these recordings.
Jon Stewart
I, I, with all that setup. It's a, I, I know that that was a bit of a letdown in terms of I, I, I should probably be more particular. Like I get every day the same thing. A quarter of a lime spritz lightly on steamed cod. It's a, a bean and cheese tostada. And, and the only difference is it comes with jalapeno. And I generally say no jalapenos. And I've done it every time for three years.
Brittany Momedovic
It's very Jennifer Aniston of you.
Jon Stewart
Is it really? Does she get a, she's a tostada lady.
Brittany Momedovic
Well, not tostada.
Dr. Sarah Shoer
Does she look like a tostada lady? Like a carb lady?
Brittany Momedovic
When they were doing friends, she would eat the same lunch every day.
Jon Stewart
Oh, is that true?
Paul Scharre
Now?
Jon Stewart
What did she get?
Brittany Momedovic
It was like a chef salad. I actually know exactly what it is and I'm not gonna repeat it. Cause I don't wanna look a crazy person.
Jon Stewart
I am the Jennifer Aniston of late night. I think people have always. But you guys know that about. On my 50th birthday, the Daily show bought me. We had one of those staff all hands meetings down in the studio and they had a box sitting on a table and I opened the box and I pulled out. It was a T shirt, a long John's shirt, khaki pants, hiking boots and a thing. And it was exactly what I was wearing that day. And I was flattered and humiliated all in the same moment. But I, I am a creature of, of very lame habits. But, but I hope, man, what, what information they got today. Very, very, very nice. Lovely. A lovely program. Thrilling and chilling and nerve wracking and all those different things. Brittany, how do they keep in touch with us?
Brittany Momedovic
Twitter. We are weekly showpod. Instagram threads. TikTok, Blue Sky. We are weekly show podcast and you can like subscribe and comment on our YouTube channel. The weekly show with Jon Stewart.
Jon Stewart
Beautiful as always, guys. Thank you guys so much for the incredible preparation you did on this episode. Lead producer Lauren Walker. Producer Brittany Momedovic. Producer Gillian Speer. Video editor and engineer Rob Votolo. Audio editor and engineer Nicole Boyce. And our executive producers, Chris McShane and Katie Gray. We will see you next time. Bye.
Paul Scharre
Bye.
Jon Stewart
The Weekly show with Jon Stewart is a Comedy Central podcast. It's produced by Paramount Audio and Busboy Productions.
Freedom From Religion Foundation Spokesperson
Paramount podcasts.
Jon Stewart explores the collision of artificial intelligence, Silicon Valley, and modern warfare. Focusing on recent controversies—such as Anthropic’s refusal to allow its AI to be used in certain military contexts and OpenAI’s willingness to step in—Stewart and his guests dig into how AI is influencing the way wars are planned and fought, who is setting the rules for these new technologies, and what moral, legal, and practical dilemmas arise as AI becomes a tool of military power. The conversation is urgent, wide-ranging, and sometimes darkly humorous, aiming to pull back the curtain on the real stakes as technological and ethical boundaries are tested.
“It’s not something special or different—the military sees it as a productivity tool, an optimizer.”
— Paul Scharre (05:46)
"Both companies have essentially agreed to both red lines... [it's] not necessarily that different between Anthropic or OpenAI."
— Dr. Sarah Shoker (24:36)
“[It] boosts efficiency... but it does also seem to offload a little bit of human autonomy and decision-making as well.”
— Dr. Sarah Shoker (17:21)
“I think the model of who should be setting the rules—maybe it’s our democratically elected representatives.”
— Paul Scharre (41:50)
“Are we elevating humanity to a higher status than we’ve earned?”
— Jon Stewart (79:54)
“Models have a tendency to escalate more aggressively than humans would... That in itself is, of course, a cautionary tale.”
— Dr. Sarah Shoker (51:17)
“That actually [hardware] is a really narrow choke point to begin then to control the technology.”
— Paul Scharre (66:24)
“If public reporting is anything to go by... 1,000 targets in Iran has largely been credited to the MSS, the Maven Smart System.”
— Dr. Sarah Shoker (15:39)
“The question is, should there be any rules? And if so, who sets those rules?... The Pentagon’s answer is, we get to set the rules. We don’t want these companies dictating to us.”
— Paul Scharre (37:36)
On lobbying:
“AI companies are... donating significant sums to lobbying efforts and tying those donations to US-China tech competition... This conversation is in fact coming for Congress and they probably better be equipped at the very least.” — Dr. Sarah Shoker (38:46)
On international agreements:
“It’s really going to be an all-of-society effort... A one-size-fits-all approach to safety is probably not going to work.”
— Dr. Sarah Shoker (71:45)
On pessimism/cynicism in AI governance:
“There isn’t really a way out of this that doesn’t involve talking a lot to other people, but there is something there to build on.”
— Dr. Sarah Shoker (73:40)
The heart of the conversation is not just the technical capabilities or current uses of AI in warfare, but the broader moral, legal, and democratic dilemmas: Who should decide what roles AI plays in war? Can public pressure, regulatory action, and international norms keep pace with technological transformation—or will commercial and military priorities win out? The guests encourage public engagement, transparency, and broad debate, noting that for-profit companies, the military, and democratic institutions must all be part of framing the path ahead.
“These decisions are too important to be left up to any one of these entities on their own... All of us, your listeners, have a role to play in weighing in on this debate.”
— Paul Scharre (74:40)
For listeners and readers: This episode is an urgent, accessible, and occasionally chilling tour through the real world of military AI, offering nuance beyond the headlines and a candid call for civic engagement in shaping the future of war and technology.