← The Lawfare Podcast
Loading summary
Transcript47 lines
- [00:01]
Podcast Host
The following podcast contains advertising to access an ad free version of the Lawfare Podcast. Become a material supporter of lawfare@patreon.com lawfare that's patreon.com lawfair also check out lawfare's other podcast offerings, rational security chatter, lawfare, no bull and the aftermath.
- [00:32]
Kevin Frazier
Avoiding your unfinished home projects because you're not sure where to start. Thumbtack knows home so you don't have to don't know the difference between matte, paint, finish and satin or what that.
- [00:41]
Ben Brooks
Clunking sound from your dryer is.
- [00:43]
Kevin Frazier
With Thumbtack, you don't have to be a home pro, you just have to hire one.
- [00:47]
Ben Brooks
You can hire top rated pros, see.
- [00:49]
Kevin Frazier
Price estimates and read reviews all on the app. Download today Ryan Reynolds here from Mint Mobile. I don't know if you knew this, but anyone can get the same Premium Wireless for $15 a month plan that I've been enjoying.
- [01:01]
Ben Brooks
It's not just for celebrities. So do like I did and have.
- [01:04]
Kevin Frazier
One of your assistant's assistants switch you.
- [01:06]
Ben Brooks
To Mint Mobile today. I'm told it's super easy to do@mintmobile.com Switch upfront payment of $45 for 3 month plan equivalent to $15 per month Required intro rate first 3 months only, then full price plan options available, taxes and fees, extra fee full terms@mintmobile.com maybe China is going to use open weight models to essentially ship their technology, their intangible technology around the world and create dependencies on Chinese industry. The challenge is, if these models have embedded vulnerabilities, then what does that mean for the future of the digital economy?
- [01:47]
Kevin Frazier
It's the Lawfare Podcast. I'm Kevin Frazier, the AI Innovation and Law Fellow at Texas Law and a contributing editor at lawfare, joined by Ben Brooks, a fellow at Harvard's Berkman Klein center and former head of Public Policy for Stability AI.
- [02:02]
Ben Brooks
If the US withdraws, if it pulls up the drawbridge, if it tries to cut off the flow of this open technology through either through enhanced chip controls or through controls on intangible technology, then there's a vacuum there and China will decouple and other jurisdictions will start to fill that vacuum.
- [02:23]
Kevin Frazier
Today we're chatting about open source AI models, a hot topic that is top of mind for everyone from OpenAI's Sam Altman to leading national security groups like the center for a New American Security. All right, so Ben, Open source, like responsible AI or human centered tech, is the sort of phrase that generally has positive connotations but is capable of nearly endless Definition. And so I just want to set the stage for listeners who perhaps aren't deep in the weeds of open source AI. What does that actually mean? And perhaps most critically, why should we care what open source AI means?
- [03:06]
Ben Brooks
I mean, to your point, Kevin, that, you know, you ask a million people to define open source, you'll have a million answers. There is this fascinating and very important tribal war taking place around the definition of open source. The Open Source initiative, which is kind of the custodian of the canonical definition of open source, is a very particular definition which we can get into. But broadly speaking, from a policymaker and a regulatory perspective, I think what matters is simply the idea that the weights, the distinctive settings or parameters for the model are publicly available, which means that a developer can come along, download those weights, integrate that model into their own system, modify the model and inspect the model. And I think for government and for public authorities, that's what matters the most.
- [03:54]
Kevin Frazier
The reason why this open source approach is so controversial, why we have tribes that exist for a open source approach to AI or a closed model approach. Can you highlight what these dividing lines are? What are the general 10,000 foot level concerns about some of the economic concerns that may arise from open versus closed and some of the national security implications that may arise from open versus closed?
- [04:24]
Ben Brooks
Yeah, so I think, and without wanting to tire anyone with too broad a brush, I think you can, you can characterize sort of three buckets that have emerged in this, in this debate, right? So at one end of the spectrum you have people, civil society groups, researchers who believe we don't know enough about these architectures and these models. We can't predict their capabilities, we shouldn't be developing them in the first place, let alone releasing them openly. And I have a lot of respect for that position. I think it makes sense within its own system, values and assumptions. At the other end of the spectrum, you've got the kind of effective accelerationists who say not only should we develop this, but we should absolutely release these models and these technologies openly. Patron Saint Mark Andreessen. It has found favor in this administration or in parts of this administration. And again, and I have a lot of sympathy with that position, I think that that position makes a lot of sense within its own system of values. I think there is this very sort of interesting middle ground that's emerged where you have a set of companies, civil society orgs, researchers who say we should develop these models by all means, but we should be very cautious about releasing them and making them available openly. Just say, making the weights available openly. And I think that position deserves a lot of scrutiny. And that's where I spend most of my time and attention. And, and, and I think if you break that down, there's a sort of underlying assumption there that limiting or restricting access to models or restricting the capabilities of models is the primary and maybe the only effective mitigation against the worst risks. We're talking about catastrophic risks of misuse, the risk of sort of accidental or runaway behaviors. And I think folks in that bucket tend to look at the model as being the primary choke point for those interventions.
- [06:16]
Kevin Frazier
And so I kind of think of this open source debate as a debate reserved for AI insiders, for lack of a better phrase. It's like those who know know about open source and everyone else is just talking about X risk or acceleration of AI, but the folks really in the weeds are having this conversation. But all of a sudden we see headlines talking about open source AI, we see national security folks talking about open source. So can you walk us through this evolution of open source being a sort of issue that only the folks who are nerdy enough to talk about AI on a podcast at 10am on a Tuesday would care about versus having this general conversation? Why? Why is this such an important issue now? And how are we seeing it become even more of an issue both from a national security perspective and in terms of this more general domestic competition among AI labs conversation?
- [07:18]
Ben Brooks
There's always been this kind of latent debate in the regulatory and national security space around dual use technology. And so if you rewind back to the 90s and even before that, you see this fascinating line of debates ultimately ending up in litigation, much of it unresolved, around what are the limitations on government restricting access to useful and capable intangible technology. So we're not talking about hardware and chips and other sort of parts of the physical infrastructure layer. We're talking about access to research software data and latterly model weights. And this kind of, you know, bubbled along in different ways. You know, the copyright space, criminal law, financial regulation. There have occasionally been moments where government is kind of confronted with that question, what do we do about capable software? Software or intangible systems that can be used to do good things, but can also be used to do bad things. And then when you suddenly have this explosion of interest in generative AI, these concerns really started to dominate the conversation. Right, because what do you have? We unpack generative AI. What does that mean? It means you have a versatile system that can do lots of things. Some of them intend to, some of them unintended, they are relatively opaque. We haven't solved interpretability. We can observe the system, but we don't necessarily understand how an input yields an output. And they're fundamentally non robust, which is to say that if you have access to the model and the underlying model weights, you can modify that model for specific purposes or specific tasks. That's a good thing. We can talk more about that. But you can also unwind refusal behaviors and you can direct those behaviors in a more malicious direction. So when I, you know, for me, my first sort of introduction to the changing face of that debate was, was when I led public policy for one model developer, Stable Stability AI, which developed stable diffusion. Stable diffusion was an image generating model, very popular. At one point it, it amounted to maybe 80% of all AI generated imagery. But when the very first iteration of that model went out into the ether on GitHub and on hugging Face, you had members of Congress writing to the then national security advisors saying this model is a threat to national security and public safety. You need to use your export control authority to prevent the distribution of these models. And so there was a kind of a moment around the time of stable diffusion coming out and ChatGPT rising to prominence where policymakers started to think about this in the context of.
- [10:01]
Kevin Frazier
To pause there for a quick second. Fortunately, of the family members I've been able to cajole to listen to this podcast, my dad's among them and he's about as technical as a nail, which is to say not very technical. So what is the technical basis for these national security concerns? So you mentioned that even something like stable diffusion, which you're not immediately thinking, oh no, people can generate images of pigs flying over the moon. Watch out, national security concern. Why is that technically a matter that folks concerned about national security even brought up in these sorts of debates about how to regulate AI?
- [10:43]
Ben Brooks
Just to put a finer point on that, I think there isn't really consensus among policymakers about the risks that ought to justify these kind of interventions. Right. So if you think back to the last session of Congress. Right. 350 bills on AI we've had under the Biden administration the longest executive order in U.S. history on AI. And if you, and you know, 750 something bills at a state level, if you unpack all of that, the motivations are very different. Policymaker, policymaker. So I think if you talk to national security folks in the Biden and the Trump administration, the concern is fundamentally about CBRN and cyber risk. So chemical, biological, radiological, nuclear, Weaponization risks? Could AI. Could these models be used to accelerate the production of a catastrophic weapon, a weapon of mass destruction? Could they be used to accelerate an offensive cyber attack at massive scale? Those are fundamentally the bread and butter concerns of the executive national security community.
- [11:47]
Kevin Frazier
And the concern there in particular for open source is that as opposed to a closed model, where we don't have the weights made publicly available in open source model, and here, correct me if I'm wrong, we're going to have higher odds of, let's say, bad actors, and in particular non state bad actors who are even harder to monitor and police. That's the grave concern, I'm guessing, with respect to open source.
- [12:17]
Ben Brooks
That's right. That's right. If you have access to the models parameters, three risks become more prominent, right? One is misuse, right? You can, you can integrate that model into your own applications, you can deploy those applications and the upstream developer has very little visibility or control over what you're doing. You also raise the prospect of modification. So someone can take a capable base model, they can modify that model through fine tuning, reinforcement learning again, integration with other systems and other tools, and that modification can expose undesirable unsafe capabilities that may not have been in the model off the shelf when it was first released on GitHub or Hugging Face, but can be exposed through that modification. And then I suppose the third and equally important point is this prospect of a mishap, right? Maybe the model has some capability or some affordance, it can be used in a certain way that wasn't clear at the time it was released, goes out there into the ether, and once people start using the model in that way, or once the model's behaviors become apparent, it's too late to do anything about it. You can't withdraw that model very easily. It's a digital file that's being downloaded by millions of people around the world. So that was fundamentally the concern for those national security folks. It's the same in the online safety space as well, which was really what animated concerns about image, video and voice generation is the concern. Not that you're going to create a weapon of mass destruction, but that you will be able to create compelling deep fakes and you use those deepfakes for abusive, fraudulent or politically misleading and deceptive purposes. You think about the Biden robocall, right? So there was this scare around image, video and voice models being used to throw the us, EU and Indian elections. And fundamentally, particularly for a lot of legislators, that is their biggest concern. They're not so concerned about seaburn and cyber risk. They're concerned about these sort of more quotidian online safety risks. And I think that's part of the challenge. Right. It's that what motivates these debates and these concerns changes depending on who you're talking to and changes month to month. I think if you rewound three months, the national security conversation around open weight models and AI in general was catastrophic risk, seaburn and cyber risk, predominantly by non state actors. To your point, Kevin, and you fast forward to post deep sea R1 and the concern is just China, China, China. Right. It's, it's, it's strategic competition, it's, it's misuse by state actors. And, and I think we should pay attention. Why, why is there this sort of unevenness in what motivates these, these objections?
- [15:04]
Kevin Frazier
Yeah. So I, I'd love for you to, to walk through three what I'll call kind of critical points or junctures in this debate. So early on, conversations around Llama. Llama is meta's open source model. And there were a lot of folks who were saying, oh my gosh, you know, Meta, you are basically facilitating Chinese advancement. You're making it easier for Chinese labs to keep pace with leading AI companies. And there was a lot of heat, I think, on meta. So we had that paradigm, which I'd love to start with. Then as you mentioned, we had this deep seek moment. So I'd love to analyze what does that mean for open source. And now the third point I want to talk about is suddenly, famously, OpenAI was kind of the champions of closed models. I'm sure they wouldn't describe themselves as such, but I'm giving them a new trophy. The champions of the closed model. All of a sudden, Sam Altman saying, hey, watch out, we are going to be releasing not only an open weight model soon, but you can expect us to continue to kind of balance open models and closed models going forward. So let's start with that Llama period and the kind of META hate, I'll say, for being so such fervent champions of open source. What was that period like? What were the defining attributes of that initial debate over, oh my gosh, anyone who's doing open source hates America and loves our adversaries.
- [16:40]
Ben Brooks
Yeah. So I think the, the initial release of LLAMA stood for just one proposition. Right. Which is, wow, there is someone out there who is willing to release a capable and expensive model openly. But the challenge is that open source already had a bunch of headwinds. Right. It's a distributed community it's kind of difficult to mobilize. It can be misused and modified in all the ways that we've discussed. There was already, I think, a very low awareness among policymakers, particularly legislators, less so the administration, but kind of low awareness about the importance of open source. The fact that open source sits in all our data centers, it powers most of the world's smartphones, it's in flight control systems for our rockets, it's. It's on our nuclear submarines. Right. Open source is good because it can be inspected, it can be modified, and it can be secured. So there were already these headwinds, and then the first really big player to come out there with a really good frontier or semi frontier model happened to be one of the least trusted companies in America. You know, and I think they're probably quite open about this. Had a really challenging reputation at that time in the, in the Biden administration. And so for them to come out with this fairly provocative release raised eyebrows. And there was, you know, pretty swiftly a sort of angry letter, a bipartisan letter from senators, Senators Hawley and Blumenthal, saying, you know, what, what are you doing? We get that open source is important, but what you're doing is reckless. There doesn't appear to be any sort of systematic process in place to evaluate risks, mitigate those risks before release. What do you have to say about that? But, but broadly speaking, I think that was, that was it. It was finally this watershed moment where you say someone is willing to spend tens or hundreds of millions of dollars releasing the underlying weights for these models. My view is that is, that is predominantly a very good thing. I think, you know, we had up to that point and right through to the present day, a bunch of people telling us this is, this is. This is revolutionary technology. It's going to transform the economy. And the idea that three or four Bay Area companies should be paywalling this transformative technology and that we shouldn't have a capable open alternative alongside those models. That's a very scary world. That's a concentration of power and control in the digital economy and the real economy that we haven't seen for a very long time. So I think the Llama release showed that there was a fork in the road and that there are good open alternatives in parallel.
- [19:12]
Kevin Frazier
Yeah. And I think what was fascinating too, about that moment was somewhat of a failure to include a more robust analysis of the economics of open sourcing a model. So meta from a strategic economic perspective, there are a lot of incentives that come with open sourcing a model if you are the model that is the, the most common one, the most ubiquitous, getting more data, getting more uptake, getting more user adoption, all of those things can be incredibly valuable, especially given that we know having access to data and quality data is so essential for improving these models. So we can talk more about those economics in a second. But it is fascinating to see that this initial period of the open source debate was very much a do you love America or do you want to help our adversaries? Kind of black and white conversation. And then Deep Seek happened. Why was that so important in disrupting this conversation about the pros and cons and adding some nuance to this open source debate?
- [20:21]
Ben Brooks
Yeah, I mean, there's so much to say about Deep SEQ in general, but I think if you, if you really focus on the material facts for policymakers, The Deep seq R1 release kind of showed three things. One, you can get a lot of performance and a lot of efficiency through, you know, a series of familiar innovations in the training and development pipeline. Right. So, you know, and we can go into greater detail, but the, the R1 model and the V3 sort of base model that Deep SEQ released a couple of months prior, you know, it brought together a bunch of techniques that had been pioneered by different researchers and different companies earlier. But they assembled it into a really interesting pipeline mixture of expert architecture, reinforcement learning, you know, so that the model learns to reason, you know, the model learns to explain its reasoning through chain of thought, you know, a number of techniques to improve efficiency and inference. And they yielded this model that was 671 billion parameters. Only a small number of those 30 something billion were active at any one time. And this model was yielding state of the art or near state of the art performance on certain benchmarks. So for policymakers, the question there is like, wow, how much of a moat does US industry actually have? Maybe the moat isn't even compute, maybe it isn't even money. They did this, you know, allegedly with, with $6 million or at least a marginal cost of $6 million. So what does that mean for, for national security policy and industrial policy going forward? How much of these sort of breakthroughs are going to come through your efficiency and through familiar techniques and familiar innovations? I think the second, the second piece was the possibility that Deep SEQ developed this largely independently of export controlled US hardware, technology and infrastructure. So Deep Seq says that they developed this model with 2000 H800 chips. At the time that they procured those chips, they were not export controlled. They were subsequently export controlled by the Biden administration. And as I said, Deep Seq, you know, maintains that they, they spent, you know, just under 6 million on the development of, of these models. So if we take them at face value, the question then for policymakers is, you know, to what extent are export controls on hardware and chips an effective choke point? Like will that, will that work going forward? And there's a lot of pushback on that, right? Like there's, there's know, anthropic musk and others have come out quite strongly. Semi analysis has some great material in this to the effect that, hey, they actually had tens of thousands of A100H100H20 chips, some of which they may have obtained legitimately, others maybe less legitimately, and their overhead cost, their total expenditure would have been much higher than $6 million. But that's sort of a big, a big wake up moment for policymakers.
- [23:33]
Podcast Host
Deleteme makes it easy, quick and safe to remove your personal data online at a time when surveillance and data breaches are common enough to make everyone vulnerable. Yo, it's easier than ever to find personal information about people online. Just try it. You can get your address, phone number, family members, names hanging out on the Internet and that has real, actual consequences in the real world and it makes everyone more vulnerable. With Delete Me, you can protect your personal privacy or the privacy of your business from doxing attacks before sensitive information can be exploited. So I'm somebody with an active online presence. I do podcasts. As you know, I sometimes even make Get Ready with me makeup videos. And you know, I'm kind of out there. I express my opinions. But the dirty little secret is my privacy is actually important to me. And that is why I use Delete Me personally. I have used it since actually before Delete Me was a sponsor of a lot of Lawfare podcasts. I believe in this product and I find that it has made a difference in my life. So you too can take control of your data and keep your private life private by signing up for Deleteme now at a special discount for our listeners. Get 20% off your Delete Me plan when you go to JoinDeleteMe.com lawfare20 and and use the promo code lawfare20 at checkout. The only way to get 20% off is to go to JoinDeleteMe.com Lawfare20 and enter code lawfare20 at checkout. That's JoinDeleteMe.com Lawfare 20 code lawfare20.
- [25:45]
Ben Brooks
What.
- [25:45]
Podcast Host
Does the future hold for business? Ask nine experts and you'll get 10 answers. Rates will rise or fall. Inflation's up or down, tariffs off. They're on. Can somebody please invent a crystal ball? Until then, more than 41,000 businesses have future proofed their business with NetSuite by Oracle, the number one Cloud ERP bringing accounting, financial management, inventory, HR into one fluid platform with one unified business management suite. There is one source of truth giving you the visibility and control you need to make quick decisions. Real time insights and forecasting means that you're peering into the future with actionable data. When you're closing your books in days, not weeks, you're spending less time looking backwards and more time on what's next. So Lawfare is still a little bit small for NetSuite because we're, you know, a tiny little nonprofit. But we aspire to continue growing, and when we get big enough, this is what we're going to plan on using. Whether your company is earning millions of dollars or even hundreds of millions of dollars, NetSuite helps you respond to immediate, immediate challenges and seize your biggest opportunity. And speaking of opportunities, download the CFO's Guide to AI and Machine Learning at netsuite.com lawfair. The guide is free to you at netsuite.com lawFair that's netsuite.com lawfair.
- [27:36]
Kevin Frazier
You shouldn't have to wonder how to make Mother's Day special or how to show her the feelings you can't put into words. That's why we created the Amazing Mom Bouquet from 1-800-Flowers.com crafted to show mom just how special she is after almost 50 years of Mother's Day. At 1-800-Flowers.com we've learned the most beautiful, thoughtful gifts come from the simplest. Yours give beautifully this Mother's Day with 1-800-flowers.com acastra. That's 1-800-flowers. Com acast optimize your nutrition this year with Factor America's number one ready to eat meal service. Factor's fresh, never frozen meals are dietitian approved. Ready to Eat in just 2 minutes? Choose from 40 weekly options across 8 dietary preferences like calorie smart, protein plus, and keto. Eat smarter@factormeals.com Listen50 and use code Listen50 for 50% off plus free shipping on your first box FactorMeals.com Listen50 Code Listen50 Want a workout that actually works? Hydro delivers a full body workout that hits 86% of your muscles in just 20 minutes. Rowing with Hydro combines strength and cardio with thousands of workouts led by Olympians in breathtaking locations. No wonder nine out of ten members are still active one year later.
- [28:52]
Podcast Host
Try Hydro risk free@hydro.com and use code.
- [28:56]
Kevin Frazier
RO to save up to $475 off your HydroPro rower.
- [29:00]
Podcast Host
That's H Y-R-O-W.com code ROE.
- [29:07]
Kevin Frazier
How much of it was a concern as well that we've seen China in other contexts kind of dominate new technological paradigms and dominate new efforts at expanding its sphere of influence by being the sort of core infrastructure upon which other countries build. So when you think about the ports it's helping build in Africa, when you think about the roads it's building in Southeast Asia, so on and so forth, can we think of their leaning into open source as a sort of means of becoming the default AI of the world? Is that a concern that may be animating folks to also adjust their perspective on open source domestically?
- [29:53]
Ben Brooks
Yeah, completely. Right. So I think, I think really the third and maybe biggest bucket of concerns for policymakers was this, this point that maybe China is going to use open weight models to essentially ship their technology, their intangible technology around the world and create dependencies on Chinese industry. That's one thing that's, that's important for strategic competition. But the challenge is if these models have embedded vulnerabilities, if famously these models are censored on questions pertaining to the Chinese Communist Party under Beijing's regulations, then what does that mean for the future of the digital economy? The example I give is imagine if outside of the US and Europe, the next default search engine isn't Google and it isn't complexity. It is some act that is powered by a model developed by or by Alibaba and Tencent and Deep Seq. Right. A model that can't answer questions about Taiwan and Uyghurs and maybe has all sorts of other embedded vulnerabilities, some of them more obvious, some of them less obvious. But that's, that's the kind of concern that I think policymakers ought to have. Now the question then is, well, what do you do about that? Right. So within a couple of days, 48 hours of deep sea R1 being released, you had the House Select Committee on the Chinese Communist Party coming out and saying this censored open weight model again is a threat to national security. You had Senator Hawley in the Senate actually drafted a bill that would expand export controls in a way that essentially criminalizes someone Uploading an open weight model, if it were available to a national in China, would criminalize someone downloading, importing, intangible AI technology from China, something like Deep Seq, the Deep Seq model, and would prohibit research collaboration with Chinese nationals, which is to say that a machine learning researcher would not be able to go to Europe's ICML and share what they've worked on the previous year. Right. So that's sort of the extreme hawkish version of the response. I, I think the better response is to say, hey, we see what's happening here. Open weight models is not only a way to potentially accelerate adoption in AI across our economy, but it is also a way to create dependencies on US industry and to make sure that US trained, US regulated, US aligned models end up powering the world's AI applications. And that's why I think, I think the worst response to the Deep Seq launch is to say we should stop releasing capable open models. I think China is starting to understand that this to your point about belt and road, right? This is a way to project soft power abroad. And if the US withdraws, if it pulls up the drawbridge, if it tries to cut off the flow of this open technology through either through enhanced chip controls or through, through controls on intangible technology, then there's a vacuum there and China will decouple and other jurisdictions will start to fill that vacuum.
- [33:04]
Kevin Frazier
And so now fast forwarding to present day or near present day, we're talking in early April and as you mentioned, it sounds like folks, congratulations Ben, are heeding your advice and saying the best response isn't to, or at least among industry folks, the best response isn't to quash open source models, but instead to lean into it. So as I hinted at OpenAI, which famously has avoided releasing an open weight model for a long time, Sam Altman has now announced a sort of pivot of the company policy. What does this pivot mean for the development of AI in the us? Should we expect that open source may become the sort of new model? And if that's the case, one concern I have you mentioned earlier about the issue with respect to concentration of AI labs where if it is just a handful of companies in Silicon Valley leading this AI effort. We've seen this playbook before, right? Let's just watch the social network again and see the faults that manifested. What does this mean for the economics and larger AI ecosystem if open source starts to become the new default or at least a greater mix of this AI portfolio going forward?
- [34:25]
Ben Brooks
Part of what maybe motivated the announcement is a growing awareness that the models, the model layer is becoming commoditized, right? That a lot of these breakthroughs are replicable fundamentally, they can all be traced back to open research. And given enough compute and enough dollars, you know, maybe other teams are going to be able to yield the same performance in, in, in a relatively short period of time. So the question then becomes, well, how do these firms that have, you know, spent vast, vast sums of money on, on training and development, how do they, how do they monetize this space? How do they create a moat? And I think that is increasingly going to focus on the product layer and the application layer. So it's not just the, the raw model as a kind of dense knowledge digest. It's how you integrate the model with other tools and other systems and then how you productize that and get that out there into the real world and into the hands of deployers who can use it and are willing to pay for it. So I think to some extent that may have motivated the decision. I think, look, I think broadly speaking, it's hugely promising. I think that's what they've said is exactly the right approach, which is it's not open or closed, it's open and closed, and they both have a role to play. I think their commitment to evaluating for risk prior to release is, is expected. It's fantastic. I, I think they will potentially help to, to really advance the state of the art in terms of evaluating open weight models, not just for off the shelf risk, but also for modification risk. In other words, what's the worst you could possibly do to this model through optimization and modification? But there is this kind of lingering reservation I have about the announcement, right? Which is that a lot of the signaling around that announcement and what OpenAI and Anthropic and others have previously said to the Biden administration and potentially the Trump administration, is open sourcing is okay, so long as it's slow or small, right? By which I mean so long as it's some way behind the frontier or it's so small and its capabilities are so diminished that we don't really need to care about it. And that is what I sort of describe as the poor cousin theory of open source, right? And I think it's really troubling, right? It's troubling because, you know, on the one hand it's deferring difficult questions to another day, right? Difficult questions around like what risks do we actually care about when it comes to models? Who determines acceptable risk? Developers as regulators, is the court. What is the standard of care for mitigation before you send these models out into the world? And how do developers satisfy that Standard of care, these tough conversations that we've been dancing around for years now. And I fear that that kind of, you know, open source, the small stuff, open source, the sub frontier stuff, is just delaying that. And then I think the other one is it kind of, you know, it overlooks the fact that to really benefit from capable AI, it's not enough to just contain it behind a paywall. Right. Like, safe containment doesn't really turbocharge the US Economy. It doesn't give the US a kind of strategic boost over its adversaries. What will do that is safe diffusion. It's like getting good, capable models in the hands of as many deployers across the economy as possible, helping them to evaluate your risk, helping them to modify it for specific tasks, and helping them to integrate it safely into what they do. And so I do worry that these announcements may end up just being, you know, we're open sourcing some small stuff. We're open sourcing some stuff that's a year behind frontier. But the capable stuff, the economically transformative technology, we're going to continue to keep behind a paywall and it ought to stay there. And that's a position that I worry about.
- [38:16]
Kevin Frazier
Yeah. It's interesting from a history of technological diffusion standpoint, right. If you could imagine, for example, instead of the Rural Electrification Administration making full electricity available to farms, for example, they said you only you're, you're throttled at. I don't know, I'm not electrician. 30 volts, right. You get to, you get to light up your barn, but you can't light up your bedroom or whatever. That's only for the urban city dwellers. Seeing that kind of limited capacity, I think would, if I'm understanding your point correctly, kind of undermine the whole point of open sourcing, which is allowing for broad analysis of what are the capabilities of these models, what are the risks, what are the potential benefits? And if you're just handing out last year's tech, well, then the possibility of discovering those new risks or those new benefits will be greatly diminished because you're just kicking the tires of, you know, a 2004 sonata instead of looking at, you know, a cyber truck. And what are the potential pros and cons there? Is that somewhat of a way of understanding it?
- [39:28]
Ben Brooks
Yeah, yeah, it's really just saying that. Look, I think 99% of deployers across the economy, whether they're, you know, individual workers, consumers, creators, or whether they are large enterprises, they're not going to need to fold proteins. They don't really. They won't necessarily need the tip of the spear, state of the art frontier model. But they will need something that can do economically useful tasks and do them well and do them cheaply and ideally be capable of being scrutinized, can be modified and can be implemented in a secure private environment, which is kind of what open weight models offer. Now, if you make those capable models openly available, I think that's the fastest path to diffusing this useful technology across the economy in ways that ultimately impact productivity and innovation. I think if we take the view, especially if regulators and legislators take the view that capable models should not be openly available, then we don't, we don't obtain the economic benefit of this technology. Or at the very least we create these, these highly concentrated dependencies on a handful of firms for critical technology. And that isn't good for all the usual reasons. It's not good. Seeking introduces risk. It means that people are transmitting sensitive data back and forth with two or three APIs for the rest of eternity. It means we have very little visibility into the behavior and the performance of these models and very little opportunity to modify and do something about it. So I think that, that, that is, that is challenging. I mean, you also mentioned this wider question around, you know, open or closed, are we heading for a world where the, again, the whole community, the whole ecosystem relies on a handful of companies like OpenAI and Anthropic or Meta? And I think it's a really interesting question, right, because I don't think network effects are going to play out in AI quite the same way that they played out in the Internet, for example. Right. Like fundamentally, the platforms, the big Internet platform, search and social media became big and stayed big because you know, their value increases with every additional user plugged into the network. With AI, especially with models, you know, the calculus is a little bit different. I mean, essentially you've got just a huge publicly available, we can debate that, but publicly available data set on the Internet, you've got very large, very expensive models being trained with huge amounts of compute. The only companies or organizations that can field that capital at the moment, yes, happen to be the companies that did very, very well off the Internet economy and amassed that capital through things like ad revenue. But once they've released that model, especially like Once Meta released Llama 4, for example, that model's out there in the ether. And sure, they, they qualify how you can use it and how you can deploy it and things like that. But, but once the model's out there, I mean, I mean anyone can, can start to play around with it and integrate it into, into their own systems. So you know, I think, I think there is still that concentration risk, but it's a concentration risk that comes from the inherent cost and compute intensiveness of this research and not because of so network effects in the, in the sort of Internet sense.
- [42:53]
Kevin Frazier
Yeah, it's super fascinating because I think that for the AI labs themselves, they have the, a sort of similar economic incentive as the social media platforms, which is to say the more users you have, the more data you have. So in that sense there's a network effects component for them, but for a user it doesn't. I don't care what model you Ben, use. I don't care what model my law firm colleague Alan Rosenstein uses. From the sense that it doesn't give me any additional benefit of getting onto chat GPT and saying Alan just looked up a fun healthcare regime or something like that. That doesn't add any benefit to me. I'm not friending Alan's post or something like that. So it is interesting to see how there are parallels to social media, but it is a distinct set of questions we have to grapple with. And with that in mind, I'm curious. Let's just say Senator Hawley calls into the pod. Senator Hawley, you're welcome anytime. If you're, if you're listening to this and says Ben, you know, I agree. I don't want a Chinese open source model to be the default model for the rest of the world. But these CBRN issues, these cyber concerns, these nuclear concerns, I just can't shake them. Is there a sort of middle ground policy you'd recommend for the people who are just really scared of open weight models leading to this huge increase in the risk of bad actors deploying biological weapons causing havoc. What's a policy that we can latch onto? What would you like to see, for example, in the AI action plan with respect to open source policy, both in terms of promoting its benefits and recognizing some valid concerns?
- [44:40]
Ben Brooks
I mean, I would love to see a clear recognition that safe diffusion is how we're going to boost productivity, innovation and quote, unquote win in AI. And so what does that mean in practice? It means a few things. It means one, restrictions on useful, capable intangible technology should be a last resort, not a first resort. I don't take the position that we should open source everything for all time as sort of absolutist, but it should be a last resort. And too often these restrictions have been talked about as a first resort. There is Lots of low hanging fruit that we can grapple with before then. Right. Like the regulation of transparency in model development, the regulation of deployers, users, downstream platforms. There is a lot of work to be done there and we haven't even done a gap analysis to determine where does our existing regulatory and legislative infrastructure fall to short. In fact, one of the only governments to do this in a systematic way was the previous UK government. And they went around, they asked regulatory agencies, do you feel like you have the statutory authority and the resources to deal with emerging risks in your domain? And of the 12 or 13, all but one said yes. And I think the same you'll find is, you know, the FTC and, and CFPB and others said the same under the Biden administration. So I think, you know, treat model layer intervention as a last resort, not a first resort. Build up readiness on the assumption of openness. So that can mean everything from defensive accelerationism, you know, build up the kind of ecosystem downstream mitigations and safeguards and there's a lot more work that can be done there, including with, with federal support. And then the flip side of all of this is we need a good monitoring capability in government. We don't have a monitoring capability. That's when we start to see really reactive regulation and legislation. And that's why things like the US AI Safety Institute is so important, because that is the monitoring capability for the US government. You can monitor for trends of bipartisan interests and bipartisan significance. They can identify possible proportionate forms of mitigation and they can give advice to the administration and potentially to legislators as well. So I get concerned that on the one hand there are still murmurings, including this administration, around going harder on model layer interventions like intangible technology controls. While on the other hand, it's still very uncertain what's going to happen to the US AI Safety Institute. We need that monitoring capability if we're going to preserve a maximally open regulatory environment. So that's what I'd say to the senators, wholly of law. But I think a lot of this is going to happen, frankly at a, at a state level. I mean, the first. What is it now? April, right. The first quarter of this year we've seen as many more state bills than we saw in the whole of the last two year legislative session. And there's a lot happening there and a lot of that could affect open sourcing and open weight in really subtle but really significant ways.
- [47:43]
Kevin Frazier
Yeah, and I love the sort of chicken and egg issue you're pointing out here, which is the longer we take to just set up a baseline approach to transparency and monitoring, the greater the odds of reactive legislation. That really clamps down on open source because it's just that dearth of understanding that you've pointed out at the state level, at the federal level. If all we're doing is just legislating in response to whatever China does, that's not exactly a great policy posture to have. And ideally we would have a clear national vision for how we want open source to fit into our broader AI portfolio. But that's a tall order.
- [48:25]
Ben Brooks
Yeah, yeah. And I mean, with respect to the very thoughtful legislators and policymakers who are coming up with many of these proposals, some of the most dramatic and interventionist proposals have been ones that were kind of framed reactively in response to some media moment. But if you think about the, you know, there were at least two, you know, developer licensing frameworks that were, that were drafted in the Senate, bipartisan frameworks that were drafted in the Senate in the last session. There was a House bill that would expand export controls to model weights. There was of course the Hawley bill that I mentioned earlier on Deep seq. And many of these kind of pop out in response to the release of Llama or in response to the release of Deep Seq. And I think we can do more to make sure that these are calibrated. And fortunate people forget that the Biden administration, it was the longest executive order in US history, but there wasn't much in there that was actually regulatory in nature. And in fact, if you're a model developer, there was really only one obligation, which was if you're training a model over 10 to the 26 flock, you need to report to the federal government Department of Commerce on your red teaming results. The government didn't even tell them what red teaming to perform. They just said you report it. So we know what's over the horizon. And I think in the scheme of things those frameworks, I would could it be less problematic end of the spectrum, but where you start to see export control regime for multiple weights, licensing requirements for developers, or state legislation that modifies liability rules in ways that are fundamentally incompatible with open sourcing if they require a level of visibility, control and custody over models that is is infeasible in an open source environment, those are the kind of interventions that are really going to impede open innovation, by which I very specifically mean making good, capable technology openly available for third parties to inspect, to modify and to deploy independently.
- [50:28]
Kevin Frazier
Well, Ben, this will not be the last news moment or media moment regarding open sourcing, so we'll be sure to have you back at some point down the road, but we'll have to leave it there for now. Thanks again for coming on.
- [50:41]
Ben Brooks
Thanks Kevin. Appreciate it.
- [50:46]
Kevin Frazier
The Lawfare Podcast is produced in cooperation with the Brookings Institution. You can get ad free versions of this and other Lawfare podcasts by becoming a Lawfare Material supporter at our website, lawfairmedia.org support support. You'll also get access to special events and other content available only to our supporters. Please rate and review us wherever you get your podcasts. Look for our other podcasts including Rational Security, Allies, the Aftermath and Escalation. Our latest Lawfare Presents podcast series about the war in Ukraine. Check out our written work@lawfaremedia.org the podcast podcast is edited by Jen Pacha. Our theme song is from Alibi Music. As always, thank you for listening. If you work as a manufacturing facilities engineer, installing a new piece of equipment can be as complex as the machinery itself. From prep work to alignment and testing.
- [51:48]
Podcast Host
It'S your team's job to put it all together. That's why it's good to have Grainger.
- [51:52]
Kevin Frazier
On your side side with industrial grade products and next day delivery, Granger helps ensure you have everything you need close at hand through every step of the installation. Call 1-800-granger clickgranger.com or just stop by Granger for the ones who get it done.