← Risky Bulletin
Loading summary
Transcript50 lines
- [00:00]
A
Foreign. Hey everyone, I'm James Wilson and welcome to Seriously Risky Biz. This is our podcast all about cyber security policy and intelligence. Earlier today I had a really great chat with my colleague Tom Uren who is our Policy and Intelligence editor, and I sat down and talked with Tom about the Seriously Risky Business newsletter that he is publishing this week. In this week's newsletter he's gone through two really interesting topics. The first one is around, I guess, the blowback from Mythos and the way that it was released and how between what Anthropic has done with the preview of Mythos and OpenAI just releasing GPT 5.5 without any sort of preview or staggered release. This seems to have prompted the US Government to have a bit of a rethink as to whether they want to retain their hands off, just go for it sort of policy towards regulating the release of AI models to potentially considering a more hands on role in regulating the release of these very powerful frontier models. The second topic that Tom covered into this week's newsletter is around Australia getting its own equivalent of the CSRB or Cybersecurity Review Board and Tom talks about whether the blameless framing of this Review board and also the legislation that's behind it, but might render it a little bit less or a lot bit less effective than we would all otherwise hope it would be. But hey, if you're not already subscribed, please do head over to our website at Risky Biz where you can subscribe to Tom's excellent newsletter Seriously Risky Business. And I'd also like to thank this week's sponsor which is portswigger. You can find them@portswigger.net I recorded a cracking interview with James Kettle from portswigger that was released on Monday, which is all about the diabo results that you get when a security expert like James codifies his knowledge and practices in a way that an LLM can access and begin to do research like James Kettle does. It's a scary look at what the future is going to look like. And this, this kind of dovetails into this first part of the conversation with Tom where we talk about this seemingly abrupt change in stance around the US Government considering regulating the release of these powerful frontier AI models that could have some serious cybersecurity ramifications. I'll drop you in here to the conversation where Tom talks about just how did we actually get to this point where the US has had this change of heart. Enjoy.
- [02:41]
B
Well, I think the Trump Admin, when they first came into took office, one of the first Things they did was basically bin a whole lot of Biden era regulations around AI and just out
- [02:54]
A
of spite or with good reason.
- [02:55]
B
Well, I think I would describe it as philosophical in the sense that you've, you know, one camp, any regulation slows down business and slowing down business is terrible. I think that's an exaggeration of their position. But, you know, that's basically what it comes down to. The other camp is, and I'll probably give it a more sophisticated view because it's more aligned to myself, is that companies sometimes do things that aren't in the public interest and we need regulation to manage that. And it is a balance. Right. You don't want too much, you don't want too little. They went with very little. And I think it's easy to go with very little. Like, what was that a year and a half ago? Something like that. When it is, the risks are somewhat theoretical at that point. Models were not very good at cyber tasks. There was no great harm that was coming because they were hacking the planet. And I think it really sharpens the mind, you know, gives you some focus when you've got concrete examples of models doing things that are potentially very risky. So in the last month or so, we've had a whole heap of newsworthy cyber, not incidents, but discoveries driven by, in particular, Mythos Preview, which is Anthropic's latest model.
- [04:15]
A
But, you know, there's a part of me, Tom, that just thinks, what's the old, the old saying, you know, if you've got to go through hell, don't walk slowly. There seems to be an element of a hell of a time is coming for us because these bugs are going to be found, they're going to be exploited. Like, it just feels inevitable. And part of me wonders, like, do we really want the government slowing this down? Because when they slow it down and create these gaps between, I guess, who's got access to models and when the timing of the models, that it's those gaps in time that are going to be exploited by attackers. So what is the real tangible benefit of even getting a government oversight of this?
- [04:53]
B
Yeah, so there's a number of different stories that appeared in a number of different outlets over the last couple of weeks. And one of them is the administration's thinking of setting up a body that would basically decide what the right thing to do is in terms of regulation, and that would consider everything from, you know, government review before models are released to kind of a light touch. And it seems like the administration is keener on what Anthropic has done, which is release Mythos preview to a limited number of organizations. There's one report that Anthropic wanted to increase the number of organizations from, I think, 70 to 120. So, like, that's like doubling the size, but in the scope of the planet.
- [05:44]
A
It's still very small.
- [05:45]
B
And the administration actually pushed back. That's the report, anyway. That seems like their approach is we're keen on the ring fencing and then like relatively slowly expanding that. That seems intuitively like the right thing to do. Right. But I'm at the point where I think I've got no idea what the right thing to do is. I'm not sure that the advantages from doing that are as great as we think. So, for example, like your interview with the Portsquare, what's his name?
- [06:21]
A
James Kettle. James Kettle, yeah.
- [06:23]
B
He's doing all sorts of crazy discovery with an older model. And then there was another report this week from Niels Provost, who was a Google distinguished engineer at one point, and he's basically, yeah, I can find all sorts of stuff with an older model when I give it a harness. So you construct a scaffold around it and it does close to as well as the latest Mythos, and which flies
- [06:50]
A
in the face of any potential benefit of regulating the model in and of itself. You know, like, I think what those. Those two instances, the James Kettle sort of work that he's doing and he's open sourcing that at Black Hat Us in July. So, like, again, it feels inevitable that this apocalypse is coming. But. But those guys have shown us that. I think you cited it in the newsletter. Well, that the difference between a novice with a absolute bleeding edge incredible model like Mythos and an expert with an older model is actually not that different. And so what level of regulation is going to help if it's just at the model level, I guess, is my question. And would they need to go broader?
- [07:31]
B
Yeah. I also think that the most advanced adversaries are the ones that will have scaffolds and will be working really hard. So I guess the argument for holding back models is that there's a whole lot of people who would do some hacking if they could. They can do the easy mode hacking, and I think that's worth considering. Like, what's the unmet demand for random hacking that would be enabled? Right. That, I think is actually what you're talking about by holding back models. I think if you're talking about national security, I don't think that actually makes much difference. The random Hackers will hack random things and they won't. That won't enable China or Russia or whatever, but I think that you're not actually gaining that much when it comes to China, Russia, these sophisticated adversaries, because they'll be trying to like use open source models, they'll be coming up with jailbreaks, they'll use harnesses. And I think for nation states, where it's at is getting the most advanced model you can and getting the most out of it by putting a harness around it.
- [08:41]
A
Yeah, yeah.
- [08:42]
B
And so from a national security perspective, I think that's where the game is at. But like I said, there's a whole lot of other issues as well. And it's not immediately clear to me what the right answer is at all. I just think that locking it down is not as good as we think it is. And so I'm much more keen on the idea of, yeah, let's get involved in model releases. Let's collect a whole lot of information about how they're being used, what, how many bugs they're finding, how quickly those bugs are being patched, what happens when they get patched. Does that actually result in an end day apocalypse? Because attackers can now reverse engineer the patches. And so I think there's a whole lot of second order effects that are not clear at all.
- [09:32]
A
And so you think there's a more meaningful role for government there to collect that data, to sort of be, I guess, the arbiter of good process around observation, surfacing those things. And I think you said it in the newsletter. Well, is it a little bit like maybe we should just wait and see before we start bringing down the ban? Hammer on. Who can release a model and when. Is that sort of a good summary of your take?
- [09:56]
B
Yeah, yeah. I also think that it probably does make sense for each or every company to follow the same processes. And so I dive a little bit into the processes that Anthropic's taking, which is to release Mythos Preview to a small number of organizations. And OpenAI, their latest model, GPT 5.5 is basically just as good as mythos. So the UK's AI Security Institute did testing and it found that it scored better than Mythos did, but within the margin of error.
- [10:30]
A
Yeah. And ironic that poor Anthropic is the one that keeps getting beaten up by the US government and yet they're the ones that are seemingly being safe and cautious.
- [10:40]
B
Yeah, exactly. And they've. Their OpenAI's model is anyone who's paying can get access to 5.5 deliciously capitalism. And I'm not sure that that's wrong either. Right. So they have a kind of tiered model where it tries to funnel people into a trusted access program where you have to verify who you are and whether you're a legitimate defender. And if you can do that, you get access to more tailored versions that are a bit more cyber permissive.
- [11:10]
A
Right, right. A little bit less safeguards still. Same model, I think, is my sense.
- [11:15]
B
Yeah, yeah. And so that's very cautious versus relatively open with safeguards. I think those are quite. I wouldn't say they're diametrically opposed. They both notionally achieve the same goal of safety, but there's genuine questions about how effective is that program? Would we be better off with that program rather than with a more restricted. On the theory that more defenders having access would actually be better on Net?
- [11:45]
A
Well, I think this comes back to your sort of point there around government needs to do a bit of a wait and see. Because if we look at those two different approaches, the, you know, restrict the model, slowly release it versus release the model to everyone, but rely on the safeguards. What we're really coming down to there is the faith in those safeguards and the ability to create a structured cyber. You verified your identity, you can have less safeguards. It all comes down to the faith in do those safeguards work. And is giving a smaller group of people access to the same model but with less safeguards a good way to still enable cyber research. But we're not going to know that unless we have someone that is actually objectively collecting that data, Right?
- [12:27]
B
Yeah, exactly. Yes. Couldn't have said it better myself, James. Thank you.
- [12:32]
A
You say the nicest things, Tom. Well, let's move on to perhaps where you've not said the nicest things around Australia. Launching a hamstrung Cyber Review Board. That's a clear bit of shade that you're throwing there. As I read in the newsletter, and I quote here, that you've called out, their intention here is to deliver actionable recommendations to government and industry to help prevent, detect, respond to and minimise the impact of similar incidents. These incidents being serious cybersecurity incidents in the future. Now, Tom, that sounds like a great intent. Why are you calling this hamstrung?
- [13:10]
B
Yeah, so I've written several times about the US version of this, the Cyber Safety Review Board. I just want to say that a guy I went to school with is on the Australian Cyber Incident Review Board. So congratulations, Baron, best of luck. Now the bad news. So the Cyber Safety Review Board did a number of reports. Some of them were, I would say, quite technical in nature. And the recommendations were very much this way of doing things isn't appropriate anymore. And I'm thinking about, they did one on Lapsus and the kids were basically bypassing cybersecurity controls and they recommended specific things about SIM swapping, for example, or SIM porting or mobile number porting or whatever. That's very technical. And those standards existed because they were appropriate some time ago. Now there's another report they did where they basically said, Microsoft, you just don't care enough about security. And they talked about a cascade of failures. And I would describe that as a kind of political report in that it has a political impact on the leadership of Microsoft. It wasn't you did any one particular technical thing wrong. It was, you don't care about security enough and you should because you're super important. And that had impact. A few months later, Satya Nadella stood up, sent an all hands memo, said security is the top priority at Microsoft. That didn't last all that long, but while the board existed for a few
- [14:42]
A
minutes, it was good. Yeah.
- [14:44]
B
Now the problem with the Australian version is the legislation says the board's report cannot apportion blame.
- [14:52]
A
Right. And you've called this out, you said, you know, the board's impact will be limited by its approach to liability. But as I was reading that, it was quite insightful to me the way that you called out. There's a difference between apportioning blame and also having protections against liabilities such as, you know, like you can, you can blame, but you can then say this blame can't be used in court. So talk me through, I guess, you know, help me understand what should they have done here and just how limiting is this because of the way it's been structured?
- [15:22]
B
Yeah. So the analogy for these Cyber Safety Review boards or the Cyber Incident Review Board is the US ntsb, the National Transportation Safety Bureau or the Australian equivalent, the atsb. And they can write reports and they can say, you know, this person stuffed up, they were drunk, they had failed their medical, they were psychologically unstable, whatever. If the reason exists, they can say it, but that report can't be used in court for liability. Like that's the legislation.
- [15:56]
A
So what is the lasting impact then? Is it just a, you're bad and you should feel bad, like if it doesn't get to court, you know, the
- [16:05]
B
point of the NTSB reports is here's the cause of the accident, here's how we can fix It. And there's no liability because that encourages people to contribute to the report, to speak openly, to tell the truth. And you know that if you do that, it can't be used against you. Now, the Australian Incident Review Board, it's trying to get to the same place where people speak freely, but it chokes off liability at the wrong point. I think those reports should be able to say, you know, company, you were terrible, you just didn't care. The root cause is not some technical issue. The root cause is that you made stupid decisions about risk. And I think, like, it's the nature of, I think businesses that deal with risk, that there's like technical issues and there's like management issues.
- [17:02]
A
Yeah.
- [17:03]
B
And you can't separate the two. And I think the problem with this is that it removes the ability for the board to say, you guys were just so dumb.
- [17:14]
A
Yeah.
- [17:14]
B
And that removes the political pressure to fix that kind of problem. And that I think is a. It means that the impact of the board is limited.
- [17:24]
A
Yeah.
- [17:25]
B
Australia's had a couple of cybersecurity ministers who've been very good at removing the bull that victim companies have run. The PR that they've run.
- [17:35]
A
Right. The sophisticated attacker we couldn't possibly have defended against.
- [17:40]
B
And she's just stood up and said, no, it wasn't.
- [17:43]
A
I think, was that around like, I think, was that the Medicare one or the Optus one maybe, where I think
- [17:48]
B
she really cut through both. Yeah. And I think the way that the legislation is for this board, it removes the ability of the board to cut through in the same way. So I think that's a shame.
- [18:02]
A
Yeah. It is hard to imagine what the real teeth of a report is if it's got no blame. Like I'm trying to imagine in my mind how like at the end of the day an incident happened because something went wrong and something went wrong because someone did something somewhere. Right. You always get to that point.
- [18:20]
B
I think this is particularly timey because of the rise of AI, so I think there will be novel attacks that people can't expect. I think there'll be like, you know, some proportion of major incidents will come from something like out of the blue.
- [18:34]
A
Well, actually, I did want to check this with you as well because again, quoting something here that you put in the newsletter, you quoted, again, the. I think the legislation around this and said that there's a criteria for what an incident has to meet in order for it to be eligible for review. And I found this a bit interesting. It says the criteria is that it involved novel or complex methods or technologies, an understanding of which will significantly improve his traits preparedness. But I would put it to you that making the criteria that the attack had to involve novel or complex methods cuts out a whole swath of stuff that should still be involved because frankly, sometimes what's novel is actually the level of stupidity and incompetence on the side of who's been attacked AI at the moment that we're seeing again, sort of to your point that there is sophisticated use of AI which will be done by state level actors and experts, then there is dumb uses of AI where it's like it's not dumb if it works and it's now working at scale. As I read it, those sorts of things wouldn't meet that criteria. And so is that going to limit kind of what we learn from this as well? Do you think they'll really stick to that criteria heavily. And does that again further limit the
- [19:46]
B
success of each board? I think there's a big difference between pre you've done the review, it looks like a novel or complex attack versus after you've done the incident review, does it still look like a novel or complex attack? And so I think that there'll be, you know, I think these are political things. If there's a major incident, the board will be rolled out regardless of whether it looks like it was complex, novel and certain.
- [20:14]
A
So you think this is, this is a catch, this catches the marketing ball in the pr. If someone dares to say, oh, we were attacked and it was incredibly novel and complex, then they'll be straight on to haha, good, you meet our criteria, please come in.
- [20:29]
B
Yeah. And so I think that stepping back, there'll be one class of attack or some percentage which will be like this, where it is actually something new and different and worth investigating, like what do we do about this? And then there's another class which will be an executive thinks AI is just so wonderful, let's roll it out everywhere without thinking about consequences. And I think there is a line to be struck of yes, this company rolled out AI. Yes, it actually thought about the way to do it correctly and it had mitigations in place and it tried, I guess you'd call it due diligence or whatever. I think it did a reasonable job trying to mitigate those. And I think that would be the sort of straight down the line, here's the technical reasons that failed or why it wasn't good enough. What else can we do? And I think there'll also be some category of we just did really dumb stuff and we didn't think about it. And that's the kind of. And I think that might be quite large.
- [21:32]
A
Yes. Yeah.
- [21:33]
B
And I think that is the sort of thing where you want to be able to say in a report, this was just idiotic and people should not be doing this kind of idiotic thinking. And I think the way that the legislation is, it cuts out that as a possibility. Yeah.
- [21:46]
A
Yep. Well, either way, Tom, we just gotta. We gotta wait and see on this. Fun times. Strangely, I find myself looking forward to the next major cyber security incident in Australia. So we can see whether this actually works. But Tom, let's wrap it up there. This. It was awesome to talk to you. And folks, of course, if you haven't already subscribed, please do head over to our website at Risky Biz, where you can subscribe to Tom's awesome Seriously Risky Business newsletter. Tom, thanks for the chat. This was a lot of fun.
- [22:15]
B
Thanks, James.