Loading summary
Transcript54 lines
- [00:00]
A
Foreign. Hey everyone, I'm James Wilson and welcome to Seriously Risky Biz. This is our Cybersecurity policy and intelligence podcast and it is based on the excellent newsletter that my colleague Tom Uren writes that is called Seriously Risky Business. You can find his newsletter over at our website Risky Biz and subscribe to it over there as well, if you haven't already. Before Tom joins me for this chat, I just want to take a moment to quickly thank Knock Knock for sponsoring this week's episode. Knock Knock is a really awesome solution for keeping risky and sensitive things away from the network. Can't even reach it until a necesso sign in has been done and then magically ports open and they've just released a gray noise integration which is super cool because now you can also tell if someone's logging in legitimately, but potentially from a bad ip. But back to Seriously Risky Biz. Tom's newsletter covers three super interesting topics this week. The first two are kind of related. We talk a little bit about the AI knife fight that seems to be breaking out in the US and the White House around Are we going to regulate models? Are we going to review them? Are we not? Are we distancing? It's comical, but also at the same time you're just like, how is this comical when this is nation state government stuff we're talking about and some pretty serious capabilities in these models. That dovetails into a conversation about the Google analysis that came out this week that essentially points to the fact that much like all the things we've been saying here on Risky Business, the use of AI in the hands of adversaries is now at an industrial scale. And there's some pretty crazy ways that they're working around guardrails and getting access to these models, despite all the precautions that the model makers and Frontier Labs try to put in place. But then thankfully, we leave behind the doom and gloom of what these models can do and what regulations would actually maybe look like to then do a little bit of stargazing. In particular, we talk about the importance of low Earth orbit, low Earth orbit satellite constellations, and how a lot of large governments have suddenly woken up to the fact that having a, you know, an Internet in space that's actually a sovereign capability that they need and they need it urgently. And what's interesting is how each superpower is approaching this in different ways. Some are doing it, you know, quick and fast just for them. Some seem to have more global ambitions. Time will tell. It's going to be interesting. So listen, I'LL drop you into the chat here with Tom where he gives us a quick recap on the crazy week that was and the back and forth about are we regulating AI models? Are we not? Who's fighting for what and who's got their knives out in this fight? I hope you enjoy it.
- [02:53]
B
Yeah. So as I was reading about this, it reminded me of that old joke where the fights in academia are so bitter because the stakes are so low. And so in this case, I was reading the different news reports, and it's like, on Monday, the Trump administration is going to impose AI regulation, and on Tuesday, they're distancing themselves from that idea. And then on Wednesday, it's back on the table. And then on Thursday, it's like a lobbyist is saying, there's no clarity. They're totally disorganized, they've got no idea. And it. It felt like that we don't even know what they're fighting about. But the story that mostly is interesting is that the, for us, from a cyber perspective is that the National Cyber Director has reportedly, according to the Washington Post, proposed that the office of the DNI will have a pretty large center that will be dedicated to analyzing, testing AI models.
- [03:56]
A
But does that make sense? Like, is that why there? What have they got that makes sense to do that?
- [04:01]
B
Yeah. Yeah. So I think it depends what you think the risks are. And I suppose the backstory is that the Trump administration, when they came, when President Trump came in, it was like, AI regulation. Yeah, we don't care. And that was like, What, a good 18 months? Not quite 18 months ago. And I suppose at that point you could put hand on your heart and say, look, the risks of these models amount to not very much at this point in time. And so, well, okay, now it's clear that particularly Mythos and the hype associated around that has made people think, ah, this is actually a really serious cyber security problem. And if it's only a cybersecurity problem, it makes sense that you get cybersecurity expertise. And that is certainly what organizations like NSA have. And NSA has both an intelligence role, but also a defensive security role. So that actually is the perfect place. Is it the perfect place? It is a very good place. It makes sense. You've got that expertise, you can bring that. But I think the thing about models more broadly is that they have many different uses. It's not just cyber security. One of the original concerns was that it could upskill malicious people who want to make chemical or biological weapons. That also, I think, is A serious potential concern. It's not the concern of today's models yet.
- [05:38]
A
Well, that's a good question, right?
- [05:39]
B
Is.
- [05:40]
A
Is it or is it not? Like, how. How much of this is just. Did Anthropic really create a problem for themselves with this, ooh, Mythos, it's so scary. It's so, so dangerous. And then all of a sudden, that piques the interest of regulators. Now, I can't imagine Anthropic is ever going to come out with a model and say, we can't release this to the public because it's really good at making chemical weapons. But nonetheless, that sort of capability could be a latent increase, I think it just happens to be that anthropic so focused on coding, and because they didn't create Mythos to be good at cyber, they created to be good at coding, then found by, you know, side effect, it's really great at cyber. So, yeah, it just makes me wonder, like, I guess, two questions for you there. How much of this is anthropic, almost accidentally creating their own regulatory push? And, you know, to your point around the dangers of models outside of cyber, how would we ever know? Like, I don't think there's a Mythos equivalent here that would raise awareness.
- [06:38]
B
I think the thing about Mythos, well, cyber exploitation, is that it's a yes, no answer. And so it's easy to tell that it's doing really well because you get something that works and you can measure whether it works. It's kind of a binary answer, I think, for biological problems. And as an aside, I actually have a degree in molecular biology, so that's my background originally. It's a wet science, like, you're doing things in test tubes and spinning things, and it's actually very physical. And I think there's less yes, no answers. But the model companies actually test how helpful it could be. On their model cards, there's a section on biological and chemical weapons. And it seems like my reading of it is that they're not as helpful as they are in cybersecurity yet, which kind of makes sense because, like, code is all logic, and if you get something that's good at logic, it will be good at that. Whereas biology is not all logic. There's a lot of physical processes involved.
- [07:45]
A
You basically saying your field that you study is far more complicated than my field that I studied. Is that what I'm taking away here? Okay, good. Yes, thanks.
- [07:54]
B
That's why I moved out of that field into this one. Something simpler.
- [07:57]
A
Yeah, good.
- [07:59]
B
Now, you asked also about anthropic. And I think I probably won't comment too much on that. I think they have fallen into that trap in that they promoted it as dangerous. It seems like so far the administration's focus has been on Mythos, even though by all accounts, OpenAI's latest model is just as good, if not better. And it's kind of just being pushed out and off we go.
- [08:24]
A
Yeah, I think they've learned the lesson to keep quiet on that one. Okay, so bringing it back to. You mentioned that the, the National Cyber Director. I just want to get that term correct. He pitched his own idea of where it should belong in his portfolio, but there's another stalking horse. Who else wants a piece of this?
- [08:43]
B
Yeah, so commerce already has a role in that, and they have since 2024. So it was the US AI Safety Institute back then. It's been renamed to the center for AI Standards and Innovation. And so that actually assesses models. So it's been doing it for a couple of years now. And some of the companies voluntarily hand over their models for assessment. There's also the UK AI Safety Institute, which has looked at Mythos and 5.5. So there's actually a history of this. Now, I think right at the beginning, you mentioned the other piece I wrote, which was about Google's AI threat. I think they called it a threat report. And the piece or the element I pulled out there, and you mentioned it on the show yesterday. I think the most significant thing is that they talk about the industrialization, breaking of guardrails and getting access to models by adversaries. So they talk about like really large scale attempts to, let's see, what does it say? Emerging ecosystem of custom middleware, proxy relays and automated registration pipelines designed to bypass safety guardrails and billing constraints.
- [10:04]
A
Yeah.
- [10:06]
B
So to me, if you're worried about what models can do, I find it concerning that you would have an industrialized process to break those guardrails. Right. And I think that's where AI, well, government should be involved, like the AI companies. Like, they. They tell like a logical and reasonable story about how we grant access to people for cyber capabilities. So OpenAI this week released a piece. You know, this is how our Trusted Access for Cyber program works. You know, you get more if you do this, if you provide a government id, et cetera, et cetera, et cetera. That all sounds in isolation, like, actually pretty reasonable. Now when you have Google coming along and saying there's an industrial scale process to bypass all that, like, well, does it work?
- [10:58]
A
Yeah. It feels a little bit asymmetric, I think one would say.
- [11:02]
B
Yeah, yeah. So I think that a sensible role for government would be to be involved in that place, to figure out how effective the safety guardrails are, how often they're bypassed. Is there anything we can do to make them stronger? Like what, as a government, what can we add? Are there disruption operations we can do? What's the. What should be the bar? You don't want one company being very too strict and another company being too loose. I think it would be good to have a standard. So I think that's really where regulation should be focused is in the short term, at least on understanding how often or how badly guardrails are bypassed.
- [11:44]
A
Yeah. Because at the end of the day, this all comes back to the golden element of in incentives. And I think I agree with you that a Frontier Model lab is doing a best effort to make sure the guardrails are there for their own sort of, I guess, reputation and brand image, really. But that only gets you so far. And I think it's important people understand just how ineffective, if I'm honest, these guardrails are today. I think I talked about this a show or two ago where I was astounded that I could get Claude to go from no, I won't write you that exploit to writing exploit for me simply because. Because I showed it a transcript of another conversation that it had had where it did write it. Now, there was nothing validating that that transcript was real. I could have created it. Right. So we've got to remember all these things are just like tokens and words that are exciting, neural networks that are generating more tokens. And if you can get the right assemblage of tokens, they go right through those guardrails. Yeah. I think I do take the point here and agree that a government's role is to strongly require and enforce those guardrails because there needs to be that backstop.
- [12:50]
B
Yeah, I probably wouldn't right now. I wouldn't go that far. I would just go. It seems anecdotally there are a lot of stories about jailbreaks that work. Like, you just told me one then. Right. The UK AI Safety Institute said for 5.5 with, you know, within about five hours, we found a universal jailbreak.
- [13:10]
A
Right.
- [13:10]
B
So five hours, that's not a lot of time. Like, so these are expert red teamers for AI models. Does China have an expert AI team red team? I bet they do. Is it going to take them five hours or four or eight? Like, you know, less than a day's work. I think that is something to really understand well and then make the next step. Now, it could be that, like, you know, James, you're just one in a million again.
- [13:42]
A
Tom, you say the nicest things, but
- [13:45]
B
I think it would be really good to have metrics and for something like trusted access for cyber, on balance, maybe having more like genuine cybersecurity people involved is a net win, a net positive. Like, I don't know that it's clear that you shut everything off. I don't think that that is the right solution.
- [14:06]
A
Last question on this from me is, as Joe Average user of Model, what tangible benefit do you think I really get if this regulation steps in? Is it going to affect me or is this more, much, much, much larger, larger concerns that are never going to be probably impact me so long as they are. Well, I guess the eventualities that we're afraid of don't occur through the regulation.
- [14:29]
B
I think it's cutting off tail risk. So I think the idea that AI could be used to create some bioweapon that gets out of control and causes a pandemic, I think that's a small risk, but it's also to me at this point, a somewhat unknowable one. Like, how bad is that risk? And I think the point of regulation at this point is to just understand what that risk is. And so if we can be like, I think the other option is to just do nothing and hope that AI model companies do the right thing. I don't really believe in whole societies putting all their faith in a small handful of people who've got their own motivations. Right. And I think that's the point of government, is to try and get some sort of oversight on things. So I think that's true for chemical, I think it's true for biological. There's probably a whole lot of other impacts that AI will have where we want to just cut off those tail risks or manage them as best we can, perhaps.
- [15:33]
A
Okay, makes sense.
- [15:34]
B
So I think you're right. Joe Average, things go well, Absolutely no impact whatsoever.
- [15:40]
A
Things go badly. Joe Average has a pretty bad day.
- [15:42]
B
Yeah. And that's what we want to avoid.
- [15:43]
A
Yeah. Okay, let's take a pivot away from chemical and biological weapons and do some stargazing. Tom it seems now with the advent of Starlink, and I've used Starlink myself, where I live, the cable Internet sometimes goes out, and so we dig out the Starlink dish. It's an amazing service. It's incredible how well it works. But when the only provider is Starlink and you've got Elon Musk at the helm of that, that obviously makes a few people very, very worried. And your piece around low Earth orbit satellite constellations being a must have now is I think centered around the fact that a lot of governments seem to be waking up and realizing that this is a sovereign capability they need and they need it now.
- [16:28]
B
Yeah. So the piece that initially sparked this line of inquiry was one in Wired that talked about Razvent. I don't speak Russian.
- [16:38]
A
Russian R is a bit more rolled,
- [16:40]
B
but yeah, but they, they're investing in a smallish low Earth orbit constellation and it comes basically on the back of Starlink allow listing Ukrainian terminals. And so both sides have been using Starlink terminals in the war in Ukraine. The Russia's had a bit more of a difficult time because notionally they're sanctioned and also there's a geoblock, but at the front they could use Starlink. And then they started using Starlink minis on long range drones. They started, you know, blowing up things deep into Ukrainian territory. And so that seems to have encouraged people to take a more robust approach rather than just geofencing.
- [17:25]
A
Yeah, yeah. And I said that was like an allow listing of Ukrainian terminals, right?
- [17:29]
B
Yeah, yeah. And it's kind of amazed me at the time that it took two years to figure out that allow the sting was a thing. But anyway, and so the, it's clearly this has really been important for the war. There's a lot of stories subsequent to SpaceX cutting off the Russians that, yeah, the Russian war effort is not going well and so they're investing in this constellation. Now the problem for the Russians is they just can't launch very many rockets per year. And so Starlink is like in the tens of thousands. I've got the figure in there. I think it's like 9,000 active right now, but they've launched 12,000 over time.
- [18:14]
A
Wow.
- [18:14]
B
And so you launch them, they're low enough that they gradually decay and fall out of the sky, which is a positive, I think eventually. So the Russian equivalent's aiming for 900 satellites, but it seems like they just won't be able to get there now. The border point is that they feel like they have to try.
- [18:35]
A
Yeah.
- [18:36]
B
So I was looking at all the other satellite constellations and it turns out that the Chinese are launching two and they're both aiming for something like 12,000 satellites. And it turns out that the Chinese actually do have a lot of launch capacity.
- [18:51]
A
The number of the satellites is interesting. Like when I read the story rasavet with its 16 in, I believe it's got 16 satellites up at the moment, but it's already able to achieve something like 48 megabit downlinks, 12 megabit uplinks, 40 millisecond latency. Now I'm sure the coverage is patchy with only those 16 hovering overhead, but that's impressive. Right? That's not to be sneezed at. And the other thing that I found interesting is the number that Russia wants to get to IS 300 satellites. And they're doing this on a very high polar orbit. So it's obviously designed to work for Russia. By Russia. Contrast that to Starlink, which as you said has 9,000 in service, 12,000 already up there. And the reason those numbers are so big is because the goal with Starlink is global coverage, get all the major population centers. So maybe I'm just reading too much into this, but China, if their networks are going to also be up there in the 10,000 odd range, I believe that number applies to both the state network that they're creating as well as more of the consumer one. Should we be reading into the fact that Russia wants this for their own interests in their own territories? China seems to want it though as something that can be used worldwide. And is that, what's the policy implications of that?
- [20:12]
B
Yeah, yeah, so I think you're right there. Both Starlink is a global consumer service and business service, but it's very much consumer focused. Both the Chinese ones will the same global focus to sell to everyone. Now I think the Chinese businesses, they probably will use Starlink if they have to, but they're probably not happy about it. Right. They would rather use an indigenous product and, and vice versa. I think it's one of those kind of geopolitical bifurcations where it depends on who you trust a bit more.
- [20:52]
A
Right.
- [20:53]
B
To look after your interests. And so I think that's one of the reasons China, like you said, they've got a state backed program and then they've got a private program. So two different constellations. They want to have a sovereign capability, I guess, and it can be sovereign for both business, but also for military applications. And I think that it is like super interesting that they're all, we must have something that we can control.
- [21:24]
A
Yes.
- [21:25]
B
We don't have to rely on someone else. We don't have to rely on the whims of Elon Musk or the US President. So I guess one of the interesting things is that in the war in Ukraine There was that period when the US Said, we're withdrawing military support. Like I said earlier, Starlink had been very important in the war. The European. Are they allies? Allies of Ukraine were like, omg, we've got to have a plan B. So they actually have a small satellite constellation. I think it's like some 600 satellites. And it is far more expensive than Starlink. It's slower, but it's a plan B.
- [22:06]
A
The kind of funny flip side of this is this didn't begin with the US Government. Almost like State. Like, you think back to the space program, you know, in this decade, we will get a man on the moon. There was none of this. There was just some guy that really likes launching rockets and he wanted to
- [22:21]
B
put satellites on that everyone thought was crazy.
- [22:23]
A
Everyone thought was crazy. And frankly, a lot of us still do. But, Tom, let's wrap it up there. But just curious if you caught the wonderful little bit of history in the naming of these things. Bureau 1440. The previous company also had 1440 in its name. That was sort of the predecessor of this satellite maker. Did you catch what the significance of that number is?
- [22:45]
B
No, no.
- [22:46]
A
It is the number of orbits that the Sputnik 1 satellite made before it crashed to Earth. And so it's a point of pride, clearly, that number. And it seems to follow along in these satellite makers. But, Tom, this has been great to talk to you. Thank you very much. And we'll see you again next week.
- [23:04]
B
Thanks, James.
- [23:05]
A
Thank you. And folks, again, if you haven't already subscribed, please do head over to Risky Biz, where you can subscribe to Tom's awesome Risky Business, Seriously Risky Business newsletter. Thanks all. See you next week.
- [23:19]
B
Sam.