AI, Cybersecurity, and Securing Model Weights with Miles Brundage and Chris Rohlf

Summary

The AI Policy Podcast (CSIS):

"AI, Cybersecurity, and Securing Model Weights"

Host: Gregory C. Allen (CSIS)
Guests: Miles Brundage (AI Policy Researcher; formerly OpenAI), Chris Rohlf (Security Engineer, Meta; CSET Fellow)
Date: June 27, 2025

---

Episode Overview

This episode offers an in-depth look at the intersection of artificial intelligence (AI) and cybersecurity—with a special focus on the emerging debate over securing "model weights" (the valuable files that encode a trained AI’s knowledge). Host Gregory Allen is joined by cybersecurity veteran Chris Rohlf and AI policy researcher Miles Brundage to explore how AI has changed the defender-attacker balance in cybersecurity, why protecting AI model weights matters, and what policy and practice need to catch up with rapid industry changes.

Both guests share their career backgrounds, the evolution of AI in cyber, discuss how defenders and attackers leverage AI, practical security recommendations for organizations and individuals, and debate whether securing leading AI models is realistic—or futile—in an increasingly open, connected world.

---

Key Discussion Points and Insights

1. Cybersecurity Evolves: From Manual Analysis to AI-Assisted Defense

(Chris Rohlf: 02:20-05:20)

• Chris describes how cybersecurity shifted from painstaking, manual code analysis to automated tools, and finally to integrating classical machine learning and now generative AI.
  • "You sat there, you looked through code line by line, tried to determine what the program was doing, where the developer got it wrong, and then slowly and painstakingly writing an exploit by hand." (02:23, Rohlf)
• The advent of machine learning brought classifiers for malware, but the “paradigm shift” came with large language models (LLMs) automating code writing and vulnerability discovery, dramatically scaling security workflows.

2. Old & New Threat Vectors: “Malicious Use of AI” Taxonomy

(Miles Brundage: 05:51-08:56)

• Miles recounts early AI policy work on the risks of malicious AI use, spanning political (disinformation, authoritarianism), digital/cyber (vulnerability discovery, automated spear phishing), and physical (drones, targeted attacks).
  • "We broke it up into three different domains: political security... digital security... and physical security." (06:38, Brundage)
• Previously theoretical concerns—like LLM-aided phishing—are now "real phenomena that companies are dealing with every day."
  • "It started to be clear that cyber was not just an afterthought, but perhaps one of the most important security domains of AI." (08:00, Brundage)

3. The Offense-Defense Balance: Attackers Still Have the Edge

(Chris Rohlf: 09:50-17:26)

• Chris explains most attackers benefit from scale and automation. Millions of accounts are assaulted via credential stuffing—taking advantage of weak/reused passwords and lack of two-factor authentication.
  • "There are areas... where the cost of attack and the barrier to entry is very, very low for attackers." (11:22, Rohlf)
• High-end, nation-state attacks (e.g., zero-days) are more targeted, expensive, and rare—still relatively cheap compared to other state-level tools.
• Defenders must “be right everywhere,” while attackers need only one slip-up.
  • "The Defender has to be everywhere and do a good job in all of those places, whereas the attacker... has to be right once or twice." (13:36, Rohlf)
• Yet, investing in security meaningfully raises the bar; even if impenetrable defense isn’t possible, making attacks expensive and noisy deters opportunistic hacks and slows targeted incursions.

4. The Impact of AI on Cybersecurity: Leveling the Playing Field?

(Chris Rohlf: 21:17-26:29)

• AI’s main benefit is lifting the bottleneck for defenders—especially in “boring but important” work like vulnerability scanning.
  • "AI is going to tip the scale, eventually back in favor or balance it out toward defenders." (22:08, Rohlf)
• AI helps defenders automate vulnerability detection across huge codebases far more efficiently than attackers benefit in finding novel vulnerabilities.
  • "AI is exceedingly good at that." (24:13, Rohlf)
• Practical constraints: Not all companies are adopting AI-powered tools, especially those below the “security poverty line”—e.g., hospitals and SMBs with low resources and expertise.

5. AI as a Target: Data Poisoning, Adversarial Attacks, and Model Security

(Miles Brundage & Chris Rohlf: 28:30-34:46)

• Newer attacks target AI itself, like poisoning training data, inserting "sleeper agents" (backdoors) via fine-tuning, or exploiting LLMs’ tendency to leak or exfiltrate data.
  • "If you look at a lot of these complex agentic frameworks... the models will generate the code you ask them to." (34:13, Rohlf)
• These risks are real, but the attacks aren’t always trivial—still, vigilance is needed, especially as LLMs are integrated into critical/high-trust settings.

6. Securing Model Weights: Why It’s Hard, Why It Matters

(Miles Brundage & Chris Rohlf: 35:38-59:31)

• Model weights—the files encoding a trained AI—are now multi-billion-dollar assets, but can be (in principle) copied "for damn near free."
  • "We're creating these extremely valuable files that essentially represent billions of dollars in spending... but it's not obvious the security is increasing at the same rate." (36:06, Brundage)
• What does a thief get? Sometimes, transferring weights for practical use may be difficult due to hardware/software dependencies; but in other scenarios, exfiltrating weights could jumpstart a nation-state's or competitor’s AI efforts.
• Security “tiers” exist: basic defense against random hackers up to full hardening (air gaps, armed guards, limited bandwidth, etc.) for state-level threats. It’s challenging to balance high-level security with the desire to serve millions of users and innovate quickly.
  • "Once you ratchet [security] up too high, it becomes nearly impossible to do research or to scale out inference infrastructure." (54:06, Rohlf)
• Even with open-source models, organizations have incentives to secure their own fine-tuned variants/proprietary data.

7. Security “Doomerism” vs. Pragmatic Optimism

(Miles Brundage: 63:08-68:44)

• Miles defines “security doomerism” (aka "security nihilism") as the belief that securing model weights is futile—attackers will inevitably win.
  • "There is a kernel of truth here... security is really hard... in many cases it is actually appropriate to be... a doomer." (66:06, Brundage)
• For some contexts (open labs, non-strategic models), extreme measures aren’t warranted. But for strategic assets, it's both feasible and valuable to invest in high-grade protection—with the right trade-offs for usability and cost.

8. Policy Recommendations: What Should Governments and Companies Do?

(Chris Rohlf: 69:16-71:13, Miles Brundage: 72:28-75:20)

• Promote adoption and access: Policies should focus on broadening access—especially for resource-strapped sectors—so that AI's benefits for defense are realized.
  • "We need policies that allow these companies to flourish and allow them to innovate with as few barriers as possible." (70:14, Rohlf)
• Public investment & shared best practices: Pilot projects, public R&D, and "science in the open" can push the state of the art in secure model hosting and share failures for better community learning.
• Procurement and standards: Use government buying power to set higher bars for AI model security (e.g., “FedRamp for model weights”).
  • "It's totally reasonable to use the government purchasing power that exists to... start to formalize some of these tiers of security standards." (74:02, Brundage)

---

Notable Quotes & Memorable Moments

"You can never make something secure enough. You can never keep up. And that trend of automation obviously at some point collided with AI."
— Chris Rohlf (03:14)

"The Defender has to be everywhere... The attacker only needs to be right once or twice."
— Chris Rohlf (13:36)

"AI is going to tip the scale, eventually back in favor or balance it out toward defenders."
— Chris Rohlf (22:08)

"AI could reduce the cost of attacks, enable larger scales of attacks and, you know, faster paces of certain kinds of attacks. But it could also help with defenders."
— Miles Brundage (08:32)

"It seems like there's maybe a mismatch in kind of the seriousness of protection."
— Miles Brundage (38:28)

"Replication is not innovation. And so we need policies that allow these companies to flourish and allow them to innovate with as few barriers as possible."
— Chris Rohlf (70:14)

"Security doomerism is this idea that... as you said, that it's kind of futile, that it's not necessarily worth the investment of at least getting to the point of robust model weight protection. I think there is a kernel of truth there, which is that security is really hard."
— Miles Brundage (65:41)

---

Timestamps for Key Segments

| Segment | Time | |------------------------------------------------------------|--------------| | Introduction and guest backgrounds | 00:00–05:20 | | The evolution of cyber and AI intersections | 05:21–08:56 | | Offense-defense balance in cyber | 09:50–17:26 | | AI's impact on cybersecurity (defense/offense parity) | 21:17–26:29 | | AI as a target: data poisoning, adversarial attacks | 28:30–34:46 | | Security of model weights; economic rationale | 35:38–47:42 | | Open/closed models & consequences for security | 57:05–59:31 | | Security doomerism vs. trade-offs and best practices | 63:08–68:44 | | Policy recommendations (adoption, procurement, R&D) | 69:16–75:20 | | Closing reflections | 75:20–76:49 |

---

Conclusion

This episode blends practical technical insight and high-level policy analysis on the evolving cyber-AI landscape. Rohlf and Brundage bring clarity to urgent questions: Where and how does AI change the security game? What is realistic to protect? How does success depend on adoption, incentives, and community learning? And as AI model weights become ever more valuable national assets, how do we secure them without hindering progress? The discussion is candid: there are no silver bullets—but a future where defenders can really leverage AI and sound policies can help tip the scale, if we invest and coordinate now.

Essential listening for anyone navigating the new frontiers of AI security and policy.

Transcript

[A]

Foreign welcome back to the AI Policy Podcast. I'm Gregory Allen and today we've got a pretty incredible conversation on the intersection of cybersecurity and AI, and we've got two really good people to talk about this topic. So first we've got Miles Brundage, who is an independent AI policy researcher currently launching a related nonprofit, but he also serves as a non resident Senior Fellow at the Institute for Progress and a Strategy advisor for EPIC AI, among other advisory roles. He spent six years at OpenAI from 2018 to 2024, progressing from research scientist to Head of Policy Research and finally Senior Advisor for AGI Readiness before departing to pursue independent work. I think I've known Miles now for not quite 10 years and he's been an incredible conversationalist and source of wisdom through that whole time. So welcome Miles. Our second guest is Chris Rolfe, who is a cybersecurity expert with more than 22 years of experience at the intersection of cybersecurity, AI and US national security. He is currently a security Engineer at Meta and also serves as a non resident Research Fellow at Georgetown center for Security and Emerging Technology CSET focusing on cybersecurity and AI policy. Chris previously spent nearly a decade on the Black Hat Content Review Board, which if you know anything about the cybersecurity community, is a big deal. And one caveat I should say on behalf of Chris is that he's speaking entirely in a personal capacity, not representing Meta or any of his prior affiliations. So, Miles, Chris, thank you so much for coming on the AI Policy podcast.

[B]

Thanks for having me.

[C]

Yeah, thanks for the invite.

[A]

So I just gave a little bit of background of your careers, but I want to ask a little bit specifically in the case of you, Chris, because 22 years is officially long enough that you were working on the intersection of AI and cyber before it was cool. And so I want to ask, how has your career evolved as AI has changed so much as cybersecurity has changed so much and, and sort of where did you start and where did you end up now?

[B]

Yeah, it's a great question. So where I started was in vulnerability research and exploit development. This was back in around 2002, 2003, and a lot of the work then was pretty much 100% manual. You sat there, you looked through code line by line, tried to determine what the program was doing, where the developer got it wrong, and then slowly and painstakingly writing an exploit by hand.

[A]

What happened is over time you're writing an exploit. Who's paying you to do Well, I.

[B]

Started off in DoD, but I'll leave it there. But over time, the professional penetration testing market also took off. Right. And so I've done my years in consulting for various clients in New York and Silicon Valley, everything from gaming systems to financial networks, embedded systems, phone applications, you name it. But a lot of the work was very, very manual. And along the way, sort of this trend of automation took off. Right, because in cybersecurity there's more lines of code, there's more log alerts, there' developers than you could possibly keep up with. Right. There's a very limited pool of security talent. And so without automation, without scalability, you simply will just drown in these issues. You can never make something secure enough. You can never keep up. And that trend of automation obviously at some point collided with AI. And AI, before the generative AI boom, obviously was a lot of classical machine learning techniques. Things like classifiers to look at, malware.

[A]

To look at logic started out. I mean, you've got this interesting evolution. You started from like manual, handcrafted, hand coded and eyeballs and keyboards kind of cybersecurity work. Then you moved along to traditional rules based automation. Then you brought in the machine learning part of the story. And sort of, where were you when you started really getting interested in the intersection of machine learning and AI specifically?

[B]

Yeah, so I think the so far what I've described is more of like the broader industry trend. For me personally, my area of expertise has always been in vulnerability discovery and exploit development. And the intersection of cybersecurity and classical machine learning was really more on the malware detection classifiers for logulars like incident response, like that style of work. And so I was always sort of aware of it generally as something that you needed to get around if you wanted to write a good exploit. Right. You didn't want a machine learning classifier catching this piece of malware, catching this exploit. I was more interested, interested in bypassing those types of systems. Where my personal intersection and interest in AI really took off was really when I first saw what early models like GPT2 and then GPT3 were capable of, mainly in their ability to write code. And that's when it really dawned on me that their ability to automate what we had been doing manually all along was really going to be a paradigm shift and a game changer all across the board.

[A]

Very cool. And so Miles, let's turn to you. I mean, I always joke, I think the last time I saw you that you and I are what passes for an old timer in the AI policy world, because we've been doing this for like 10 years now. But I don't think you were always interested in cybersecurity in that story, or at least not when I first met you. I don't recall that being a focus area of yours. So talk about your AI work, both in academia, then at OpenAI and sort of when cybersecurity became one of your focus areas.

[C]

Yeah, so I started thinking pretty seriously about AI policy in the early 2010s when I started grad school. But, you know, there wasn't really funding to work on AI policy full time. So it's kind of like a moonlight type thing. While I was technically working on energy policy and kind of gradually over time started to focus on. Focus on it more when I could get grants to focus on it. And one of the first big topics that I got interested in was this idea of malicious uses of AI as a general category. And at the time that was kind of controversial because there weren't that many applications of AI and people are like, well, but there's all these good things. And I was like, yes, but also we should think about what are the worst case scenarios. And there are a lot of bad actors out there. And so I worked with some people. And this is, you know, how you and I originally got introduced on a report on the malicious use of AI and hosted a workshop and kind of came up with a taxonomy of different threats to security that could involve essentially weaponizing AI. And we broke it up into three different domains, so political security and kind of things like disinformation and, you know, use of AI to make authoritarianism more efficient and kind of like detecting disability, those sorts of things. Digital security, so including cybersecurity and kind of use of AI for vulnerability discovery and automating, spear phishing and then physical security, so weaponized drone attacks and things like that. And essentially thought about this very general framework and kind of, you know, we would take what we could from current events, like, hey, there's like a very, very early proof of concept of someone using, you know, AI to help with, with some kind of spear phishing thing. But it was like, you know, very much a proof of concept rather than something that was happening at scale or that was like, necessarily super economical and, you know, a game changer. And similarly, deepfakes were just starting to become a thing. And so I was thinking of cyber as like one of this larger universe of like, things that bad actors might do with AI. And then over time, after I joined OpenAI and started to get more in the weeds of actual, you know, AI systems. And, you know, as the systems got more capable, it started to be clear that cyber was, you know, not just, you know, you know, an afterthought, but, you know, perhaps one of the most important security domains of AI. And that, you know, I think some, some parts of the report that we wrote on malicious uses aged pretty well in terms of some of the high level ideas of like, well, AI could reduce the cost of attacks, enable larger scales of attack attacks and you know, faster paces of, of certain kinds of attacks. But it could also help with, with defenders. But the actual details of like language models being critical and exactly way which ways it will be useful and things like using AI to kind of translate, you know, kind of like Russian into English to do spear phishing better. Those kinds of things have come into more focus and become more, you know, real, real phenomena that, you know, kind of companies are dealing with every day over the past six or seven years.

[A]

Great. And you both have really interesting background. So I think for some of the audience, cybersecurity is going to be a big part of their life and they're very familiar with it. For other folks, they might know mostly what they see in the news, which is XYZ hospital has fallen victim to a ransomware attack, or Russian hackers have penetrated XYZ government system, that sort of thing. So, Chris, it would just be helpful if you could sort of level set folks on where are we in the cybersecurity landscape today? Like who's who in the zoo? What are security professionals trying to achieve? What are the key factors that determine whether they'll succeed or fail? And I think one term that we're going to be using a lot on this podcast is probably the offense, defense balance. So if you could sort of explain that concept for the audience and what it means.

[B]

Yeah, of course. So I spend a lot of time sort of explaining the current state of cybersecurity in this conversation of AI policy, because if all you read are the headlines, you don't really get the full picture of that current state of things. You don't really get the full picture around what is the cost to pull off these attacks? How easy is it? How hard is it? What are the things that tech companies are prioritizing in terms of their defense in general, I would say the state of cyber cannot really be summed up as like good or bad. It really depends on the area, right. The particular domain. It's likely, and I like to use the example, it's likely that your average person has experienced, you know, their email being compromised or being locked out of an account or losing access to an account. And unfortunately those attacks happen at scale because they're very, very cheap. Right? So the a great example here is what we call credential stuffing where someone will compromise one website, extract their insecure stored passwords and email combinations, and then try those combinations on lots of other websites. And when I ask people how many passwords do you reuse and do you use two factor authentication on those accounts? The answer is they reuse passwords all the time and they probably don't have two factor enabled. And so attacks like that tend to be highly scalable and very, very cheap and they affect people.

[A]

When you say scalable, you kind of mean automatable, right?

[B]

Like automatable for sure.

[A]

There's these crime groups that are just trying millions of accounts in millions of places, finding where people have reused passwords, etc. Etc.

[B]

And they're highly opportunistic and you can only really do them through automation. And these threat actors are doing this with accounts on the order of millions to tens of millions. And so there are areas like that where the cost of attack and the barrier to entry is very, very low for attackers. And then you can scale all the way up and look at that spectrum and you can look at these sort of nation state attack where very, very sophisticated and complex zero day vulnerabilities are exploited through multi month operations, lots of operational security, very, very specific targets. Those tend to be more expensive to pull off, more difficult to pull off, have fewer victims. Right. They're not done at scale, they're not opportunistic. But still, when you zoom out, and it kind of gets to the last part of your question, when you zoom out, the cost of all these things is kind of negligible compared to what would be spent on like a missile weapons program or something like that. You know, the cost for a single zero day with a highly reliable exploit is still only going to cost you somewhere in the millions of dollars to develop. And then yes, of course, you need to have like a team around it who can operationalize it and exfiltrate the data you're after. But still, relatively speaking, the cost of these operations is not that high compared to other things, you know, that kind of get bundled up in the, you know, government spending or in the larger national security apparatus. And so when we talk about the offense defense balance, most people are going to say that it's fairly asymmetric in favor of the attacker. The Defender has to be everywhere and has to do a good job in all of those places, whereas the attacker either needs to be opportunistic and of course, because the Internet is very big, there are lots of targets, the barrier to entry there and the cost is very low, or they have to be very, very well resourced, but still only be right once or twice. And that's very easy. If you look at the size of corporations, the amount of lines of code that go into various products, it's fairly easy to get it wrong a handful of times for the developers of these products or the maintainers of these systems. And it's very inexpensive to just break that security one or two times. And so, yeah, the balance is certainly in the favor of an attacker today. And I won't dive too deep into the topic of AI in the intersection because I know we'll get to that, but I think this is a very important thing to remember in this conversation, is that even though things are very asymmetrically in favor of the attacker today, nearly all of the things and capabilities within cybersecurity are very dual use. And so the same vulnerabilities, the same tools that allow you to find vulnerabilities in source code, break into networks, those are usable for defenders as well, for increasing their security.

[A]

Yeah, I thought you did a great job of summarizing where we are in the spectrum of different kinds of attacks. So thinking about, like, why anybody would hire a security professional, because maybe you could make the argument, okay, if I'm asml, ASML is the company that makes the EUV machines that are so critical to chip production. I would wager I don't have access to the internal Chinese intelligence databases, but I would wager they're China's number one industrial espionage target. That's not a straight up weapons company. Right. They would really love to know the blueprints of every single kind of ASML machine and all the research, data, etc. Etc. So make the case for me or don't that ASML is not wasting their time by investing in cybersecurity. If the attacker has this advantage and if they're willing to spend the millions of dollars to hire a world class team, the odds are kind of stacked in their favor. Why would a company like ASML invest in cybersecurity as opposed to just give up? And that's, that's on the extreme end of the spectrum. And then maybe also explain, you know, if you're on the low end of the spectrum, like you're just a Random, you know, person. Why would you, you know, invest in any kind of, any time or money in being better about managing your own personal cybersecurity, you know, if the attacker is done? Because people are still spending a lot of money on cyber defense, so I assume they're not stupid. There must be some return on investment there.

[B]

So I'll start in the reverse order. If you're just a random person and you're looking to increase your cybersecurity, it's actually fairly cheap and should be seamless. If you look at a lot of the products you interact with, you know, your email account, messengers, et cetera, et cetera, these companies try to make it incredibly seamless to onboard with two factor authentication. Security should really just be this thing in this back in the background that you really don't even think of. And so as long as you have a password manager, you're using unique passwords, you have two factor authentication on, you're standing up to date with your patches on your laptop. All of these things should be fairly seamless in the background and it will go a long way towards solving a lot of those problems that make it, I don't want to say impossible, but very costly for opportunistic attackers.

[A]

Tell me if I've got this right. Right, like you're not, you're not creating, you know, the Great Wall of China a flawless, impenetrable defense. But what you're basically saying is if you're going to hack me, it's not going to be effortless. You're at least going to have to really spend time and effort to hack me. I don't want to just give you the keys to my house or my computer. You're going to actually have to break in somehow. And that, that moves you out of the pool of targets of like the idiot, effortless, you know, pool of targets to the, okay, this, potentially we could go after this person and succeed, but it's not effortless exactly.

[B]

All of the opportunistic attackers are going to move on to other targets because it's not you in specific thereafter. It's a collection of accounts. Right. And so yes, doing those things should be fairly seamless, frictionless, cost you virtually nothing because these features are essentially free on all of these services and will significantly raise the bar for you as an individual for opportunistic attackers. The other end of the spectrum is basically any large company with intellectual property that they want to defend.

[A]

Or nation state.

[B]

Yeah, or nation state. Right. So you probably, as a big company are spending billions of dollars developing that intellectual property. It would make sense to invest in cybersecurity to protect that intellectual property. And I think it's helpful to explain that these attacks aren't very like binary. It's not just like they break in, they succeed, they get out. There's lots of different phases of these, of these campaigns and we generally refer to that as the like cyber kill chain. So there's lots of different phases along there, everything from like reconnaissance to actually attacking and exploitation and then exfiltration. So lots of different steps there and opportunities to defeat the attacker. And so these companies don't just invest in hardening systems. They invest in education for their software engineers and writing more secure code. They invest in strong authentication and authorization systems for their internal systems. They invest in insider security programs to detect malicious employees. They invest in penetration testing firms to ensure that it's at least not trivial to break in from the outside. There's lots of different investments. They invest in things like threat intelligence that they can prioritize. That's another thing I mentioned earlier. Defenders have to be right 100% of the time. But if you raise the bar to a certain level, you're able to prioritize on probably a subset of threat actors that are interested in you based on the vertical that your company is in, the type of research and intellectual property that you have. And so you're able to narrow those things down to a smaller subset of threat actors. And once you've done that, it's manageable to look at those threat actors and say, what are the ttps, the tactics, tools, techniques, procedures that this particular set of attackers uses? And now you can sort of prioritize your efforts. These things go a really long way to raising the bar. And that's how we think about it as defenders. How high can we raise that bar so that while it may not be mathematically impossible to compromise any of these systems or that data, it's incredibly difficult. And so if you're going to spend this very, very high end capability to compromise this company, you're not just going to use it willy nilly. Right. You're going to develop it probably specifically for that company. That raises the bar, it raises the cost of the attack. And if we can do that, we can make it less reliable, as noisy as possible, and maybe we catch it a little bit earlier in that attack campaign. And so maybe they make their way partially in, but we stop it and we don't get let them get access to that intellectual property. It's a lot of different.

[A]

Yeah, because the story is often told, and this is why it was good that you brought in the cyber kill chain, that, you know, the story is often told that like, it's like a virus in your blood, right? Like once it's in, you know, you are infected. And like, the story is, you know, quote unquote goes from on or off to on. But in reality, it's like, if you want to break into asml, okay, you've hacked, you know, Jan Peter's email account, but, like, what does Jan Peter have access to? Maybe not that much. And then you're in somewhere, but you have to actually go find out what's interesting in that place where you are and where it might take you. And then there could be honey pots right in the network. So stuff that they put there that they want hackers to find, because if it's ever, you know, activated, it tells like only a hacker where would be looking, you know, in this kind of a place. So there's, there's stuff that you can do that does matter in terms of raising the cost, which I think is cool. So now the story that you just told though, I think was a story that you could have told in 2010. So what do you think that the machine learning revolution and the generative AI revolution, have they moved the needle here on offense, defense, balance? Have they what, what have been the major impacts of the rise of AI technology, either in how we do cybersecurity or in the overall sort of strategic state of cybersecurity.

[B]

So I don't think this is a settled debate. And if AI has taught me anything, it's to adjust my priors every six months or so. That being said, my personal take is that like I said earlier, things are very much in. Like you said in 2010, this has largely been the same playbook for cybersecurity. Things are largely in the attacker's favor, asymmetrically favors attackers. What the generative AI boom has done is allowed a very limited pool of cybersecurity defenders to scale up rapidly. And so if you look at all of these dual use capabilities that AI provides in cybersecurity, and again, the best example I can give is the ability to discover vulnerabilities in code, for example. That is useful for both attackers and defenders. But if you look at who has a bottleneck today in discovering vulnerabilities and doing something with them, it's largely defenders. Attackers only have to find a couple out of hundreds of millions, if not billions of lines of code in order to be successful. But Defenders have to find all of the vulnerabilities. And so there's a much, much bigger bottleneck there for defenders. And so I'm a firm believer that AI is going to tip the scale, eventually back in favor or balance it out toward defenders.

[A]

Interesting. And that's because you, as like a nice ethical cybersecurity professional and not a cyber criminal, are a finite resource and companies are cost constrained. So if they can partner you with generative AI and make you a thousand times more productive, that really does move the needle in the overall equation. And you can partner a bad guy with a generative AI. But a thousand x productivity improvement doesn't help them that much compared to how much it helps you, at least in relative terms. Because they already have the advantage.

[B]

Exactly. The cost is already incredibly low for them. And another thing that I'm often trying to correct in the AI policy debate around cybersecurity is this idea that AI is somehow providing this unique and novel capability to attackers, like it's finding new vulnerability classes. This is not really true. It's just automating what we already know how to do. And the vulnerabilities that AI can find, we largely already know how to mitigate and remediate them. It's just about scale and automation and unblocking that bottleneck. And again, those bottlenecks are largely significantly larger for defense than they are for offense. And that's why I'm a pretty big believer that AI will balance things out a little bit more in the defender's favor.

[A]

Yeah, like, like SQL injection attacks. Like we've, we've been suffering from the pains of SQL injection attacks for ages. And that's just because it takes a lot of time and smart people, you know, to go through a million lines of code and make sure that none of it is.

[B]

Exactly.

[A]

And we know an AI is that kind of obsessive compulsive checker that you want.

[B]

Exactly. And we know how to mitigate and remediate every SQL injection there is. There's nothing new, novel or unique about it. It's a pattern that we have to find across a very, very large data set. And AI is exceedingly good at that.

[A]

Okay, so we see, you know, we see research. For example, a team out of UC Berkeley just published that they were finding, you know, they were using generative AI and specifically generative AI agents to find zero day vulnerabilities, and that this sort of highlighted that we were reaching a turning point at the intersection of AI and cybersecurity. Do you think That's a fair interpretation.

[B]

I do think it's a fair interpretation. And if you look at a lot of the public research that's been done in this area, there's lots of proof to that claim. One of the best examples I can give is maybe like Google's nap time project where you have a existing non AI, non machine learning approach to automation of software vulnerability discovery primarily through the process of fuzzing. If you're not familiar with fuzzing, you basically just bombard a program with millions if not billions of random or pseudo random inputs and you see how the program behaves. We can do that without AI. But where AI is very, very helpful is in taking the code that you want to test, automatically building it, automatically generating the code that creates a fuzzing harness for that code, running the fuzzing harness, triaging all the crashes that come out of it, and then automatically producing a patch for the issues that it finds. And, and so that scalability is enormously beneficial for defenders.

[A]

Cool. Now what you've talked about, and I'm going to after this question, I want to bring you in Miles, but you've talked a lot about how generative AI in principle should benefit the defender. I assume that means that there are steps that the defender has to take to take advantage of this opportunity. So I guess my final question on this line of argument is are there things that, you know, companies, whether they be companies engaged in cybersecurity as like, as their, their profession or whether they just be, you know, companies doing something else like automobile manufacturing, is there, is there some opportunity that they need to seize? And do you actually think that this is being done widespread in the American economy?

[B]

So I don't know if it's being widespread. There's plenty of examples of it being adopted, but they tend to be more forward leaning tech companies who are adopting AI. And this kind of gets back to the classic problem in cybersecurity where you have the perfect mitigation, the perfect remediation, the perfect security control. But if nobody deploys it, then who cares? This kind of goes back to the examples of credential stuffing and account takeovers. Two factor authentication has been around for quite a while. It will stop 99% of credential stuffing attacks. But if you don't have it enabled, it doesn't do anything. And so the same is true when you scale up and you look at companies across the economy. In the security community we tend to refer to this as the security poverty line. There's lots of nonprofits and Hospitals and medium sized businesses out there who would love to have great security and are willing to invest in it, but there's a limited talented pool for them to hire. They can't afford that talent pool. They don't know, they don't have the expertise of how to adopt those tools. And so they tend to be below that line. And attacking them is sort of unnecessarily cheap and easy.

[A]

Bummer. Okay, so that was a phenomenal overview of sort of where we are in the cyber and AI story. I guess there's one other part that we didn't talk about, which is AI as the axis of vulnerability, which is to say, you know, anytime, if you were to hack an autonomous car that is stuffed with a lot of AI capabilities, conceivably you could go after the traditional computer code and try and hack that, or you could go after hacking specific vulnerabilities in the machine learning part of that software architecture. So I think here we're talking about things like data poisoning as a phenomenon, adversarial examples as a phenomenon. Maybe there are some other things here. So Miles or Chris, do you want to sort of run through the part of the story where AI is quote, unquote, the thing being attacked as opposed to traditional software?

[C]

So yeah, I can comment on that. I mean, I, I think the way to think about it is pretty similar to what Chris said about, you know, thinking about the costs of an attack. So it, what, what is the easiest thing to do? Is it to kind of, you know, to kind of just exploit a very fancy vulnerability that you spend a lot of time investing in, or is it just to do something with credentials that works because they, they were sloppy and how they set up their system? And I think similarly, if you're thinking about attacking some organization that has AI, you know, it could be that their, their normal traditional information security is pretty buttoned up, but they're being kind of loosey goosey and how they're deploying AI. They haven't really thought about the fact that, you know, you shouldn't simultaneously be providing, you know, providing an AI with access to private data and also, also allowing it to interact with the Internet and also being able to take inputs and then you can kind of trick that AI into exfiltrating data from that private, private server. And so I think, you know, the way we should be thinking about this is just avoid having a vulnerability where it's like really cheap to exploit. And right now, unfortunately, it seems like I will often be that weakest link where because these Systems are easy to fool, easy to jailbreak. And you know, there are kind of adversarial examples, adversarial examples that kind of exploit the ways in which these systems think differently than humans. Sometimes it'll be easier to trick the AI than to trick the human or vice versa, depending on the context. And so I think it should start to be part of the mental map that people think about when they're thinking about defending an organization.

[A]

So in the case of data poisoning specifically, I mean, this is a, a category of attack in which, you know, rather than exploiting something in the adversary's code, you are manipulating their training data set. So if you're like trying to train an image recognition algorithm, maybe you're using a publicly available library of images as part of your training data. And so if that's a publicly available library of images, maybe it's an open source project, you could throw a bunch of images in there that you have some kind of confidence that when this autonomous car sees a stop sign that has a sticker of a banana on it, you know, it's going to interpret that as a green light instead of a stop sign because you've somehow, you know, manipulated the training data set in a way that was not obvious to the developer. So this was mostly a hypothetical attack until large language models, at least to my understanding. Understanding, because my understanding is that Deep SEQ included confirmed examples because Deep SEQ was scraping, you know, most of the open Internet as part of their training data set. There was people who had put stuff out there on GitHub hoping that it would be scooped up by an LLM crawler and they had injected vulnerabilities into the deep seq model because they trained on that data and did not remove that poisoned data from their training data set. So Miles or Chris, I'm curious, how, how concerned should we be or how should we sort of interpret these attacks? Is this a case of deep seek being sloppy and this is actually a relatively easy problem to fix? Or as, as AI becomes more and more of an important part of the economy, as an important part of national security, and as you know, the, the, the talent pool for coming up with CyberSecurity exploits targeting AI gets more sophisticated, is this problem likely to grow?

[B]

I don't think it's an easy problem to solve. I'll give you an example in code. There is a lot of code on the Internet and of course you want to suck all that code up into your pre training data so that your model is good at generating code. But a Lot of that code out there is going to be of varying degrees of quality. There's also going to be huge volumes of code that are purposely vulnerable for teaching people how not to write insecure code. And if you were able to filter or screen those things out of your pre training data, you should go collect your Turing Award because you've solved this mathematically impossible problem of 100% of accuracy, of always discovering or flagging insecure code. So it's not an easily solved problem at the pre training stages. My understanding is that the risk is a little bit lower. Where things start to get a little ticier is in fine tuning. And I worry more about backdoors being introduced into models in that stage than I do opportunistic sort of supply chain style attacks, if you will, of putting random poisoning data on the Internet and just hoping some model.

[A]

Can you just elaborate? What's this scenario you're afraid of specifically? Why would somebody put a backdoor in? What would it allow them to do? That would be nasty.

[B]

Anthropic did a, I think it was Anthropic did a great write up on this on sleeper agents where essentially, and this has been, people have demonstrated this, where you could do some fine tuning of a model that specifically responds in a certain way according to a certain input, and basically putting that into a model where when prompted, it would produce code that does a certain thing. And so you can imagine if a backdoored model is put into a very complex agentic setup, all it sort of takes is that magical incantation and it will spit out some code that escapes the sandbox that the agent framework is running in for various different purposes.

[A]

Got it. So like 99% of the time this is a helpful AI on my company's network that's doing exactly what I want it to do. But once the right prompt is thrown in it, suddenly it's grabbing data that it maybe has access to legitimately, but it's doing it for the purpose of exfiltrating to some bad character, something like that.

[B]

Right. And I don't want to overblow this risk because if you look at a lot of these complex agentic frameworks and setups, the models will generate the code you ask them to. And so you can, you know, like code doesn't have intent, it just has instructions. And so you can generate code that is, quote, malicious through one lens, but benign through another. And so I'm not necessarily, I don't want to give the impression that that sleeper agent backdoor is required to get one of these models to spit out arbitrary code to do something. It's actually part of the design to more flexible and sophisticated agent setups.

[A]

Okay, so I have to commend you both because I thought that was a really good AI plus Cybersecurity 101. And now I want to move to another topic that I know is near and dear to both of your hearts, which is securing the AI industry and securing the big AI models. So, Miles, I think we're mostly going to be talking about model weight security here, but feel free to correct me if there's other parts of the story that are worth securing. So talk to me a little bit about, like, if you're thinking about your, your past life at OpenAI, or maybe you are working, you know, at a different company, or you're the government and you're trying to think about how they should think about cybersecurity vis a vis these companies, which I think a lot of people now view as strategic national assets. What are they trying to protect and how easy is it to protect it over there?

[C]

Yeah, so I think the right way to think about this is through an economic lens where you think about, you know, what are the costs of building the technology of, you know, the AI IP and what are the costs of defending it. And the reason why I started getting more concerned about this a few years ago is that it seemed like there was a mismatch where, you know, we are kind of getting better at security in the AI industry. You know, companies are starting to either, you know, if they're a smaller company, put in the basics of, of, of, you know, normal cybersecurity practices, or if they're big companies that already have those mature practices. They're starting to think about what are the, the AI specific threats. Like, you know, what is special about protecting an AI model or, or, you know, protecting information about the training process that is somehow different. Different from, from usual cybersecurity. And often it's not different. Sometimes it is different. But what is striking is that the costs of actually creating these AI models has been going through the roof. And it's not obvious to me that the security is kind of increased, that the cost of stealing it is increasing at the same rate. And so just to give some calibration, you know, several years ago with GPT3, you know, estimates of the kind of training costs of these models are on the order of, you know, fifty million, hundred million dollars, you know, and then a few hundred million. And now it's starting to be at the point where people are talking about, you know, specific models that cost billions of dollars to train. And that's concerning.

[A]

That's like one training run or that's the entire universe of experimentation that, I.

[C]

Mean, I mean, you know, you could say billions for both of those. And I think it'll be true over the next few years. You know, first for the overall project and then for the, the big training run. But generally speaking, you know, companies try to be efficient in their use of compute and they'll try and, you know, they won't do the really big, you know, billion dollar training run until they've kind of, they have a good sense that that's an efficient use of resources and that will tend to make up the majority of the costs of the overall project because you can kind of extrapolate from small experiments to, to big ones. But with the, the thing to flag here is that, you know, if you look at stuxnet, for example, one of the most sophisticated cyber attacks of all time, you know, you know, I'm not a cybersecurity expert, so Chris could maybe correct me, but my understanding as a non cyber expert is that this was on the order of $1 billion in terms of cost of personnel and, and kind of preparation and you know, some, you know, some of that was like cyber experts, some the of of that was kind of physical, you know, nuclear, you know, nuclear security experts, all sorts of different types of expertise and operation on the order of $1 billion. And that's kind of the high watermark. That's like a really sophisticated, you know, one or more very sophisticated countries, you know, putting giving their all to a serious attack. But what if we're in a world in which AI models cost multiple billions? That means that if you do even just a stuxnet level attack, you know, just a succinct level attack, you're basically getting a bargain. You can get this, you know, five or $10 billion model for, for a discount if you're able to steal it. And so that's kind of the concern that I have about model weight security is we're creating these extremely valuable files that essentially represent billions of dollars in spending. But it seems if you take the kind of public statements that companies are making about like, well, we are, we are robust against this level of attack and we have these security protections, it seems like there's maybe a mismatch in kind of the, the seriousness of protection.

[A]

Okay, so I think that's helpful, right? Like, training runs cost a lot of money. The output of a training run is model Weights. Model weights are just a computer file and computer files can be copy pasted, you know, for damn near free. By the way, how large are we talking here, like a model weights file for a big honking, you know, modern mile? This is like terabytes, right?

[C]

It's not petabytes, it's like terabytes, which, which is, by the way, like one of the reasons why I think it's at least it's, you know, more plausible that we can, I would argue it's more plausible that we can do a really good job of locking down at least some model weights than, you know, that it is like lock down every single, you know, algorithmic idea that some engineer comes with because those might be like one sentence. Whereas a model weight is a, you know, multi terabyte file that, you know, a set of model weights is a multi terabyte file, typically.

[A]

Okay, so although, you know, there's like terabyte flash drives, right? But sure, sure, yeah.

[C]

Everything's relative. I'm. Yeah. Easier to protect a terabyte than a few bytes is my point.

[A]

Okay, and then help me understand, what do you get if you steal the model weights? Right? So think about, for example, if you are China and you are like, gosh, I'm so mad about all these export controls that are preventing me from having computer chips. It seems like the point of these computer chips is to train models. So I'm not going to train my model because I can't, but I'm going to steal a model and then let's just say hypothetically, they have successfully stolen the latest and greatest, you know, GPT5 model weights. What do they get at the end of that story?

[C]

So I'd be interested in Chris's perspective and this is something that we've been talking about a bit lately. I think this is another area where there are multiple legitimate perspectives and where I think, you know, different people have different priors about, you know, if you've stolen the model weight, to what extent have you, is that the actual AI system like you now have the keys of the kingdom versus over time, will that start to be a part of a larger system? And maybe, you know, it's perhaps even more important or just as important to have the computing power to run that model at scale and to say, sample from it millions or billions of times. And you know, I personally think it's, you know, various scenarios are plausible and that's. And I think we should be prepared for the possibility that it's not just that the model weight costs billions of dollars to train. But it's kind of useless unless you have this specific hardware and you know, it's not really the whole story, etc. You know. But I think it's possible that it will actually be worth billions of dollars and therefore we should consider spending millions of dollars to protect it.

[A]

Chris, do you have any thoughts on this one?

[B]

I largely agree with Miles. I think to me this is a, this is a threat modeling discussion. If you have billions of dollars, if you're a nation state, you're China, you billions of dollars to spend stealing model weights, you probably are also equally capable of building your own models. And so kind of gets back to this core question of why, why would the threat actor want this? And so I don't think it's necessary to steal the model to copy it or something like that because those actors would probably be more interested in those bite sized files that have the intellectual property that talk about how to create the model. What are the latest algorithmic breakthroughs and efficiency breakthroughs that might get through the logjam that they really have the bottleneck they really have, which is access to compute, for example, do the model weights actually break that for them or is it really just a short term tactical win for them to understand something that allows them to make a longer term strategic jump? It boils down to threat modeling. I still think we, we need to lock down and secure model weights. But there's an inherent friction there that I think doesn't get called out enough. Where like why would you spend, as a US American company, why would you spend a billion dollars training a model not to monetize it in some way? And so once you think about it that way, you have to think, well, how do I monetize it if I'm going to monetize? When I spend a billion dollars to monetize something, I probably need to be able to scale access to it. I want millions or billions of people doing inference on this model. In order to do that you need a large complex infrastructure and pretty soon the security controls required to lock down a model that would prevent an adversary with a billion dollars armed with a billion dollars to steal it. You start to become incompatible with what you need to do to scale out that infrastructure to make it available to billions of people for use in a product.

[A]

Well, I still want to kind of understand what you get if you steal a model. So I mean, it's very clear to me what you get if you steal the recipe for creating models or all the insights that represent the current state of the art. But let's just say we're executing a time travel based cyber heist. Okay? So we have hacked into OpenAI circa 2025, we've stolen the model weights and now we get in our time machine and we go to OpenAI circa 2020 and we give it to them. Like what do they get to do now with these model weights that they couldn't do before, you know, and why would that be strategically important for them, et cetera?

[C]

Yeah, so I think there are a couple things and I mean, I think it's worth acknowledging that there might be cases where it actually doesn't add that much, much value. Because say, for example, I mean, so, so I'll give kind of the, the best case scenario and the worst case scenario from the attacker's perspective. The worst case scenario is you kind of steal the model weights. But actually like they, they don't run well on your infrastructure because, you know, they were designed for running on GPUs and it's going to take a bunch of work to get them to work on Huawei chips or something like that. And, and yeah, I mean, so, and also like they're so large and expensive that you don't have the ability to even run them anyway, or at least at large scale. And actually the thing that's really important IP wise is not just the individual kind of model weights or the model brain, but orchestrating it as part of this larger kind of organization of AIs that's debating and you know, running simulations. And you know, there are multiple different types of models that matter. It's not just one model. I'd say that that's kind of the scenario in which the time traveling thief doesn't actually gain that much. The scenario in which they do gain a lot is either because they do have a ready way of using it, but for some reason they're limited in training. So say, for example, you are a very GPU constrained Chinese company that is struggling to build, you know, frontier AI systems in part because you don't have GPUs that are particularly good for training. They're not very well interconnected, which is something that makes it easier to train models. But you have the ability to run the models at a pretty large scale. You have good engineers and you're able to kind of convert, you know, from one, you know, one code base to another one, one type of chip to another. And really the thing that you, the thing that you lack back is kind of those training GPUs and kind of the the interconnects between them. And so that's kind of the scenario where you benefit the most from a use perspective. And that's, that's, that's all, you know, we're imagining the scenario where the, the thief wants to use the models, another scenario in which you might, where, where the attacker might benefit is where their goal isn't so much using it, but understanding it and they maybe want to find some vulnerability. So it's kind of like, like capturing, you know, capturing, you know, an American soldier and then kind of like, you know, asking them about their battle plans or something and then using that against the rest of the, you know, rest of the American army. And so in this scenario, you might imagine, say, you know, the, the thief kind of probing the models to understand what are the weaknesses in the kind of model behavior where it can be deceived and then applying that knowledge in some real world conflict.

[A]

Okay. And then I think that the answer kind of changes if you are a believer that we are headed towards extraordinarily capable AI systems, right, because 2025 AI is very impressive and it's very useful, but it has not yet resulted in annual GDP growth of 50%. Right. And so if we were talking about AI systems that are 100 times more capable than what we have today. So if we're doing this time Travel experiment from 2030 to 2020, well then just the model itself is damn useful, even if you only have one of it. Right? Because maybe the 2030 version of the AI has like extremely profound insights as to the nature of the laws of physics. Right. That you could make an awful lot of money on, et cetera, et cetera. So then we're sort of getting to the capabilities of the model itself are strategic and somebody who exfils the model weights might be able to remove all of the safeguards, you know, from it, whereby you say, like, yes, you know, DOD ChatGPT will help you design missiles, but like Jim Bob Jones in a basement, like, will not help you, you know, design missiles kind of thing. So if the model itself has a lot of intrinsic value and you are trying to secure that from folks, that's obviously a scenario in which security would matter. Okay, so separate, you know, from the, hey, you spent a lot of money on it, you should protect it. What do you, what do you sense is the current state of cybersecurity practices at leading labs and whatever they're doing, you know, is it appropriate? And I'll just take a step back here and say, you know, when, when Google acquired DeepMind back in. Gosh, I forget what it was. Maybe it was like 2015, something like that. Maybe a little bit earlier there was, you know, a lot of concern about the weaponizability of the AI technologies that DeepMind was developing. I mean, there was people at DeepMind who wanted terms in the contract of acquisition saying, you know, DeepMind will never, you know, their technology will never be used for the military, et cetera. So people recognized that what they were working on was deeply profound at the time. But I never heard anybody say that their cybersecurity protections were profound at the time. And so Miles, would you say that's changed? I mean, would you say that the industry has sort of woken up to the need for cybersecurity? And to the extent that they've woken up, is it calibrated appropriately?

[C]

I think it's a tricky question to say whether it's calibrated appropriately because you kind of need to think about the larger context of what's the kind of strategy for making AI go. Well, I don't think it necessarily makes sense to, you know, potentially create a lot of costs for researchers and you know, in order to kind of protect a model weight that you're planning to release anyway, for example, or at least you want to kind of, maybe you, if you're not sure whether you're going to release it, then you should at least have the option of not releasing it. But I think you need to think about the larger strategy. And so, for example, like a small nonprofit that is, you know, fairly unlikely to create, create something that is at the frontier and they're kind of, they're doing open science. I don't think it, it, you know, makes sense for them to worry a ton about weight security if like, you know, they're unlikely to create something dangerous that they need to hold back and they, you know, are planning to release it anyway. I think for companies that are specifically trying and are, you know, credibly aspiring to be on the frontier, I think it makes more sense for them to have an explicit policy of defending, defending against some tier of actors. And you know, and then there's the debate of, okay, should they be defending against just opportunistic kind of spray and pray type amateur attackers or sophisticated non state actors or sophisticated actors and how do insider threats fit into that picture? And so I think it's a, it's a tricky question. And there's been some research on this, for example, from the Rand Corporation articulating this kind of tiered structure where, you know, the Highest tier is defending against highly resourced state attacks. And you know, at the bottom is this kind of.

[A]

And that's like, that's like the, the servers are not connected to the Internet, they're air gapped. There's you know, armed guards patrolling the facility and then anybody who wants to come in and touch the servers, you know, has a background check like that is, you know, some pretty intense.

[C]

Basically. But though let me just add one caveat is that I think there are a bunch of open research and kind of technical questions about like, to what extent does the, does like good security for AI model weights look exactly like. You know, it is usually, you know, it is usually looked in other contexts. And like for example it might be that because of these large model sizes relative to the kind of data of like going, going in and out of the model, there might be cases where, and you know, this is like an unsolved question. I'm not saying that it's easy to do this, but it might be possible to imagine a system that does have some limited connection to an inner, to the Internet. But you, you really lock down kind of the bandwidth going in and out of the data center. And if you see a file transfer that's like over a megabyte, you know, per day or something like that, that you're able to kind of to stop it immediately. And so you can imagine scenarios where, where you kind of have your cake and eat it too. But that's a research question that we need to invest in. And I think right now no one has said that, you know, that they, you know, even have fully, you know, warden, you know, they fully cordoned off even just like a research project, you know, while still having people able, you know, at a scale that is actually useful, you know, be Chris, for Chris's point perspective on this. But I think like, you know, we should start thinking about that because it would be a shame if we got into this world where it was, you know, very costly to have a high degree of security and in order to be secure it's like you know, basically the mo. The security costs are so extreme that you know, it becomes a useless technology.

[A]

Yeah. Chris, did you want to weigh in here?

[B]

Yeah, maybe a few things. I'd add to that, I think of the AI policy debate. There's often like you use the term Greg around like waking up. The cybersecurity community and industry has been here for quite a while defending against nation state adversaries long before the generative AI boom. And so I don't think it is a matter of these companies waking up. I mean you can take Google for example, right? Industry leading models, industry leading frontier AI lab publicly has been defending against nation state threats since at least the late 2000s with like Aurora for example. And so defending the various artifacts, model weights, intellectual property behind AI fundamentally is not very different from defending all the intellectual property a company like that has been defending all along. And I agree there are various, like there's a spectrum here, you can imagine it's a slider, right? And the Rand paper we talked about kind of talks about that. And once you ratchet it up too high, it becomes nearly impossible to do research or to scale out inference infrastructure so that you can serve millions to billions of users. But I think all these things, all these factors certainly go into the threat model of these leading AI companies. When we think about how do we balance securing this technology with also making it available to the world so that all of its benefits can be, can.

[A]

Be, yeah, I think that's, that's one thing that is, is really tricky about this whole story is that at least most of the models that are most prominent right now are general purpose and widely available, right? So the Chat GPT that an astrophysicist is using is the same as the Chat GPT who is your therapist, and that is being served to literally 500 million people per week. And so when you say you want to secure these model weights, well, the fruits of those model weights are being given away pretty damn freely and theoretically they're like banned in China, but you're one VPN away from getting access to that sort of a thing. And at the same time you've got open source projects that are coming up with their own model weights that they're giving away for free. Llama 4 is being given away for free. Deep SEQ is being given away for free. And so given, given those two parameters, the, the, even the closed systems are widely available from for users and there also exist open systems that are not garbage. I mean you can get value out of them even if they're not as good as the closed systems. How does that affect the equation for what is the optimal amount of security that we want these companies to be investing in?

[B]

So I personally see the model weight protection and AI IP protection discussion as a little bit orthogonal to open versus closed weights. There will be a market for open and closed weights no matter what, just like there is for open and closed software. And even where you have open waits that a lot of times you take the open wait, you fine tune it with private data sets. And so now that's like a thing that you want to keep secure. Right?

[A]

Like Deep SEQ gave me a model that knows how to think and my engineering data gave me a Deep SEQ modified model that knows how to design computer chips or gas turbine. Exactly, whatever.

[B]

Yeah, exactly. Right. And so you still want to keep that model and your private data set and the intellectual property around your fine tuning process. You still want to keep that secure. So to me it's a little bit orthogonal those two discussions and again, even those open models that have now been fine tuned with private data sets, you're right back where you started of who do I need to give access to? If it's a very private data set, it's just for a small set of your engineering workforce or maybe you have a bunch of researchers in medicine and you're a pharmaceutical company and you fine tune this open weight model with all of 50 years of research into pharmaceuticals, how many people actually need access to that model? Probably not 500 million a week, probably a much, much smaller number of them. Which means you can take that security slider and you can go a little bit further to the right with it and you can put it in a more isolated environment and that becomes more doable because you don't need as many GPUs, you don't need as many distributed data centers to make it available to just 200, I'm making it up, right. 250 pharmaceutical compound researchers. And so all of these factors I think come in when we talk about what is the appropriate level of security to apply.

[C]

Yeah, I, I agree with that. I'll, I'll just add a couple other things. You know, one, one is just, you know, going better this back to this question of what is the current state of company security on, on, you know, for Frontier AI. And I think, you know, the, I think Chris, you know, presented a reasonable kind of, you know, glass half full perspective in the sense that these aren't novel threats like companies have been attacked and have, you know, sometimes successfully, sometimes unsuccessfully defended against state actors before and you know, and I think there's a lot to.

[A]

But responded, responded, you know, with a mix of success and failure. Give up.

[C]

Yeah. So that's why I said the point.

[A]

Is like Google didn't say like well China's one, you know, like why do we spend a dollar on cybersecurity? So like even though they haven't always succeeded, it's still been ROI positive to invest in cybersecurity.

[C]

Yeah. So I totally agree with that. So, so the point I want to make is that I think there is a lot of institutional knowledge. It's not as if we need to like invent security from scratch. And I think we should build on what already exists and kind of which, you know, is basically Cybersecurity 101 and things like kind of handling credentials well and kind of, you know, employee awareness of, you know, spear phishing threats and those kinds of things. I think we also should be considering the possibility that we need a higher level of protection for at least some sets of model weights. Not all of them, but some of them. And I think, you know, if you read what the companies are saying, I think they, they precisely because they have all this experience defending against state actors. They also are quite candid that in some cases they are not ready and they are not, you know, if they, sometimes they will grade themselves against this kind of Rand framework and they will say, well, we like, would like to get to, to level three out of five or we would like to eventually get to level four out of five for this type of system. And these are the, these are the kind of triggering conditions under which we would say that we have failed if we have a model with this capability, but that degree of, you know, cybersecurity protection. And so I think, you know, if you kind of just like take the companies that there were, they are very candid in acknowledging that they have more to do precisely because they've been through this before. And they, they don't want to say we are fully defend, we are fully robust against state, you know, state attackers because that's an extremely high bar. And so, so that's the point I want to make there. The other thing I'll say is that, you know, you asked a good question of, you know, given that there are these pretty capable open models and given, you know, all these uncertainties about, okay, is it the model that matters, is the system that matters? What should we do? And I think the answer is really about optionality. It's about having the technical capacity and kind of the, the kind of engineering track record of having defended some systems against very robust simulated attacks. So that we can say we have the playbook, we have the recipe, you know, we have the kind of reference architecture of a data center that, you know, has kind of provably, you know, bandwidth limited inputs and outputs, you know, above some size. And we've had, you know, serious, well resourced penetration tests, you know, including, you know, simulated insider threats and so forth. And no one has been able to break into, you know, break into these model weights while still kind of gradually escalating the amount of traffic that these data centers serve and kind of building a track record of success. And I think this is an area where it would be exciting to see more investment from, you know, the US government and from, you know, companies working together to kind of pool their resources to establish this, you know, what might be called SL5 optionality. So security level 5 in Rand's framework is this idea of defending against the best of the best state, state attacker attacks. And I think that, you know, being able to say like, well we know exactly what the cost costs are, we know what the trade offs are and we've driven them as low as we can. Sometimes it doesn't make sense, but sometimes it does make sense and you know, if a company is reaching the point where per their own policies it makes sense to go to this next level, they know exactly what to do.

[A]

Yeah. Okay, I want to ask one other thing and Miles, you know you've written on your sub stack, which is a lovely source of insight for folks who don't know and Chris, you also maintain a blog, I want to say it's on your GitHub page, which has a lot of good stuff on there. But Miles, I want to ask you about the idea of security doomerism which you've sort of railed against in the past and you know, make, make the case that the folks who basically say look, anybody who really, really, really wants to get in is probably going to get in. These files are only a terabyte or so. That's a flash drive that's easily downloadable. So why are we, you know, even worrying about securing these model weights? They're going to get out anyway. And I'll add one element to it, which is the, this is an argument that I've made in the past which is that inference computation changes the nature of what it means to steal model weight. Because there's a big difference between having.

[B]

A.

[A]

Having an AI model that you can serve to one person and doing what OpenAI does, which is serving it to, you know, 500 million people a week. And that really matters in the context of export controls. I mean the rise of inference scale computation requirements has been really important in the export control story because Deep SEQ has a good model that was pretty energy efficient to train and is also decently energy efficient to serve. But there's a big difference between serving it to 10,000 people and serving it to 500 million people. And that's why Deepseek servers all crashed in the run in the excitement where it was really high in the App store. So what I had said sort of before is that the nature of AI competitive advantage was shifting away from secrets that you keep and towards assets that you have, which is namely the computing resources to perform, you know, these massive amounts of inference computation. Now, I wasn't arguing against security, I was just arguing for the export controls. But you know, help walk me through, you know, why you think security doomerism is wrong and why, even if there is this strategic importance of inference time computing, you know, why you still think you wish all these companies were investing more in security.

[C]

Yeah, and yeah, and I again want to emphasize that like, you know, companies are investing in security and so it's not a matter of like, you know, starting from scratch, but it's a, it's a question of relative rates of progress of AI capabilities versus security moving up the RAND scale. Yeah, yeah. And so the way I would put it is that, you know, security doomerism is this idea that, as you said, that it's kind of futile that, that it's not necessarily worth the investment of at least getting to the point of robust model weight protection. I think there is a kernel of truth there, which is that security is really hard and that in many cases it is actually appropriate to be, you know, a doomer in the sense that the costs aren't worth it. And so the example I gave of like a, you know, nonprofit, you know, organization, I think maybe it does make sense, sense for them. You know, that's just like building open source models. Like, I think they should be careful about, you know, protecting their HR files and you know, kind of they're like slack messages and so forth against opportunistic attackers. But I don't think it's likely that they're going to be able to defend against China if China really wanted to break into this group. And similarly, I think even for very sophisticated defenders, I think it makes sense to, to be cautious in, you know, how confident you are about being able to protect very simple secrets. So, you know, this distinction between secrets and assets, I think, you know, for example, like, I don't think it would have been plausible for, and you know, people have claimed that, you know, you know, there, there were, you know, some related leaks before O1 was announced. But like with OpenAI's One model and this idea of kind of reasoning with chains of thought, I don't think, think it's plausible that you could have protected this kind of simple Secret of like, do reinforcement learning on chains of thought and it'll make your model smarter. I think that that's just really hard to protect. And I think in some cases doomerism is appropriate. But I think that, you know, model weights are the kind of thing that is like a, in some sense it's like a very big secret. It's like a big file that you want to keep secret, but it's so big compared to like the one sentence secret that I think it makes sense to think of it as an asset. And it's, it is plausible that you can kind of have kind of, you know, restrictions on bandwidth going in and out of data centers and those sorts of things, and kind of automated alerts and hardware security features like this thing called confidential computing, where essentially you kind of have an additional layer of protection for, for, you know, certain parts, certain, you know, data and processes running on the, on the chip. Right now. Those, you know, all of these things have costs and they aren't necessarily scalable. But I think it's, you know, there's no kind of in principle reason why the cost. You know, I haven't seen any kind of argument why, like, well, it is, you know, it would cost like $100 billion to, you know, build these hardware security features or something like that. And, and you know, even if it did, like, maybe eventually we'll have trillion dollar trade training runs. Like, I, you know, I think, you know, the devil is in the details of like, when are the costs justified and when are they not? And I think there will be some cases where they're justified.

[A]

Got it. Okay, so this is the AI policy podcast and we haven't talked a ton yet about the policy side of the story. So let's just say you two gentlemen were made king for a day of the United States federal government. What would you, what would you want the government to do? And maybe also what would you want companies to do? When we think about the intersection of AI and cybersecurity, whether that's the general case that we were talking about in the first part of this conversation, or the more specific case in Securing Frontier.

[B]

AI Labs, I think we touched on this a little bit earlier around adoption. So you can have the best cybersecurity defenses and tools, but if you don't have adoption, then it doesn't really matter and we know the attacker will adopt these things. So I think policy specifically aimed at improving access and adoption of these technologies is probably the most important. And I mean, this also touches on open versus closed weights as well, because there are limitations in how you get access to these things. And again, I want to be clear, like open doesn't just necessarily mean you get access, right? You still need the hardware to run it, and so on and so forth. So policies that are aware of this and specifically do everything they can to increase that adoption is incredibly important. And I actually think that this is very tied into the previous question around doomerism. And by the way, in the security industry we always traditionally call that security nihilism. Why bother? But it's very hard.

[A]

Miles, take note, you used the wrong phrase in your blog post.

[B]

It's very hard to prove that the proactive and preventative security things you did before prevented a bad thing. But if we had never done anything in cybersecurity, things would be in a lot worse shape. And to this question around like, well, they're going to get the model weights anyway. Why would we bother? Why would we have policies that require companies to strengthen this stuff? I will just say that nation state adversaries have been stealing intellectual property from the United States forever, particularly over the last 25, 30 years, as cyber has become more important. But replication is not innovation. And so we need policies that allow these companies to flourish and allow them to innovate with as few barriers as possible. And once you do that, we can only do that by democratizing access to this technology and making it as widely available as possible. And I think that gets right to the heart of your question around cybersecurity and adoption as well.

[A]

And I should say, I think that, number one, your recommendation is a good one and number two, it has two dimensions where, you know, the United States government has a real interest in the United States economy flourishing, which means that adoption needs to not just take place inside the United States government, which needs to recognize that, hey, generative AI is probably creating this whole new universe of low hanging fruit for upgrading your cyber defenses. Go pick that low hanging fruit. You know, dod, go pick that low hanging fruit. Department of Commerce. At the same time, in the US economy, that whole ecosystem of small and medium businesses where there's not a great ROI case for making robust cybersecurity investments today, it is very possible that neither now or in the not too distant future, generative AI is going to really decrease the cost of upgrading your cyber defenses. So this is a great time to upgrade your cyber defense. And the government really has an incentive in making sure that that's a widely understood reality in the United States economy. And also whatever barriers exist to upgrading cyber defenses, that those Barriers are diminished. That's a great recommendation. Chris Miles, over to you.

[C]

Yeah, yeah, I mean those are great ideas. I'll just add three more. One is I would love to see kind of high profile pilots, projects showing new security technologies and kind of reaching new high watermarks. And we should build on what, what already exists. There are some data centers that are more secure than others. Like Amazon, for example, promotes their, you know, top secret or secret level classified data centers. But I think there should be kind of science in the open about trying new ways of, you know, limiting bandwidth and kind of, you know, scanning the building building materials for, you know, to make sure that there aren't bugs and those kinds of things so that we can learn as a, you know, larger AI and security community rather than those kind of lessons being bottled up within one company. And so kind of public investment in proofs of concept of SL4, SL5 security while still having as much functionality as possible in terms of iteration and the ability to serve, you know, serve wider customers and then kind of showing through experience that these are, you know, that these are actually secure and withstand high degrees of, you know, penetration testing effort. And if not, then you know, share that lesson, share why it failed so that the people who are actually, who are like, you know, I think keeping that knowledge secret is not necessarily going to protect, you know, the, you know, the companies that are actually defending their AI. So I think it's better to share those failures so that we can learn. The second category that I would mention is direct investment in research and development. So I think, and you know, providing incentives to hardware companies that are building these features, like for example, you know, confidential computing features are not universal in every, you know, type of chip. And so making it, you know, incentive compatible and incentivized for these companies to kind of speed up their time timelines for making it possible to kind of run encrypted copies of model weights on, on their chips and have that work across, you know, for big models that are distributed across multiple chips. So kind of speeding up the innovation cycle there through academic research and through incentives for commercial R and D and then lastly using procurement to drive a race to the top. So I think, you know, we're not necessarily in a, you know, political vibe right now where, you know, mandating every, everyone, you know, retail cell 5 is, I don't think that's necessarily a good idea, but it's certainly not politically viable either. But I think it's, it's totally reasonable to use the government purchasing power that exists to kind of drive investment and to start to formalize some of these tiers of security standards. And, you know, again, it's not like we're starting from nowhere. There are things like Fedramp, but it'd be good to see something that's like FedRamp but for a model, model weight security and kind of, you know, make that a kind of carrot associated with government contracts that people are incentivized to try and meet.

[A]

And I mean, your, your, your final recommendation is a timely one, right? Because OpenAI just signed a, I think it was a $200 million contract with the Department of Defense, you know, to get access to AI capabilities. So there are government contracts, you know, being signed with some of these Frontier Labs. Labs. Anthropic had announced that they had partnered with Palantir to deliver more of their capabilities to the DoD. So the dollars are starting to flow to these labs. And what you hope is that that's helping a transfer of knowledge and understanding and best practices of what rock solid security looks like. And you also hope that that's not security theater, because, I mean, from my own time in the dod, we spent an awful lot of time on cybersecurity. I don't know that all of that was high roi, you know, type cybersecurity measures. And so you don't want to just like, saddle these companies with lame process requirements. You want to actually say, like, this is the required state of security that we need for you from you, given the importance of the tasks that we're asking you to take on on behalf of US national security. Gentlemen, this was a fabulous conversation. I think maybe, I mean, if I could immodestly claim to our audience maybe one of the best Cyber 101s out there. I mean, you guys both have an incredible range and an incredible ability to explain. And so I'm very grateful for you both coming on the AI Policy podcast.

[C]

Thanks for having me.

[B]

Thanks for having me.

[A]

Thanks for listening to this week's episode of the AI Policy podcast. If you like what you heard, there's an easy way for you to help us. Please give us a five star review on your favorite podcast platform and subscribe and tell your friends. It really helps spread the word. This podcast was produced by Sarah Baker, Isaac Goldston and Sadie McCullough. See you next time.