Loading summary
Transcript116 lines
- [00:00]
A
Foreign.
- [00:07]
B
And welcome to Risky Business. My name is Patrick Gray. This week's show is brought to you by Sondera, which is a company that makes products that help you wrangle your LLMs, wrangle your agents and stop them from doing crazy stuff by monitoring their trajectories and stepping in when things get a little bit crazy. Josh Devon, co founder of Sondera, is this week's sponsor guest and we'll be chatting to him all about some absolutely hilarious hysterical war stories about crazy stuff that agents have done and also how we might, you know, get ahead of that a little bit. So that's this week's sponsor interview with Josh Devon from Sondera. That's coming up later. That's coming up after this week's news. So let's get into that now. And joining me, as always, is James Wilson.
- [00:53]
C
Hey Bert, great to see you.
- [00:55]
B
And also joining us this week is our co Host at large, Mr. Adam Boileau. And we should probably explain what we mean by that, which is that I guess a few months ago, Adam, you, a couple of things happened for you. You were simultaneously released from a whole bunch of non competes and also handed a giant bag of money as a result of the, you know, a company that you had had a stake in was acquired into Cyber cx. But of course the real payday didn't come until Cyber CX was sold, which eventually happened to, as I say, happened a couple of months ago. And that happened with Accenture, thus releasing you from a bunch of non competes and you also got some money. So, you know, you've been, you've been having some time off. You've bought yourself a wonderful house which is very metal. It's a sinister black cube that looks like Darth Vader's beach house basically. It's very cool. And you know, you've been kicking around not really spending much time on security, which is why we, I haven't seen much of you. And for the time being at least, you are a part time occasional guest host now on Risky Business while you contemplate your next moves.
- [02:06]
A
Yeah, that, that's a, that's a pretty reasonable summary of it. Yeah. I've spent in the last couple of weeks reverse engineering control systems in the house that I just bought trying to figure out how everything works, which, you know, does have a degree of cyber in it. So I've been trying to keep my hand in there. But yeah, it's been, you know, we've been doing this a long time and you know, having a, having a You know, more than a week or two off thinking about and talking about the cybers, you know, has been quite welcome in a way.
- [02:32]
B
Yes. Well, I'm, you know, I'm not going to say I'm jealous, but I'm jealous. I'm jealous.
- [02:37]
C
Does a cyber practitioner end up with the most secure home automation or the least secure secure home automation? Like, how does it actually work?
- [02:45]
A
I don't know yet. Ask me. Ask me in a couple of weeks, I guess. Yeah. And so far I'm at like seven or eight radio systems that are not wi fi in the control systems of the house that I bought that I have to figure out what they talk to and why they talk to it and using what protocol and so on and so forth. So, yeah, it's a time. It's a time. And we're going to find out whether or not, you know, a hacker can make a thing that's robust or not.
- [03:11]
B
So, yeah, for those of you who've been wondering why Adam hasn't been around, I had a couple of people ask me if you're okay. As you can see, he's fine. He's better than fine. He's living a life of leisure and, and buying sinister black cube houses in nice locations and messing around with home automation systems. And yeah, we're just. In the meantime, we're going to keep rotating through various different guest co hosts. We've got one coming up with Morgan Adamski, who's ex NSA, now works at PwC. We've got Ollie Whitehouse, CTO of NCSC in the UK, coming like, we've got a bunch of interesting people lined up and of course you'll be back sometime in September for another one of these, but let's move on instead of spending an entire show talk about ourselves. And so, you know, last week we spoke, of course about, you know, OpenAI. Its agent went rogue and hacked a bunch of stuff. And everybody's saying that's a bad description. It didn't go rogue, it just did what it was supposed to. Okay, sure, fine. Did something unexpected though, as far as most people could say, especially like OpenAI themselves, they didn't expect that. But now Anthropic doesn't want to get left behind. So Anthropic's come out and said, well, we've reviewed a bunch of activity and our stuff all also hacked things by accident. You know, they didn't want to get left out. And this whole thing has just turned into an absolutely hilarious and rather dense news cycle. And of course, I think this was just breaking as we were recording last week's show. But the OpenAI agents had breached more than. Had hacked more stuff than we realised. It wasn't just hugging face, it owned a bunch of stuff on its way to getting to hugging face. James, is that about right?
- [04:50]
C
It's about right. And it gets back to, you know, we had that first. The first question we had was, why is attribution so hard on this? And at first we thought, well, maybe, yeah, this is hosted in a cloud hyperscaler, so it might not look like it's specifically coming from OpenAI, but it turns out it's actually quite cooler than that. Actually. The agent compromised things along the way to give it its own jumping off point. And of course, I think that's why attribution would have been hard, but spun
- [05:18]
B
up its own orbs, man, Pretty much.
- [05:20]
C
You know, the first thing it did was it somehow found this. So there's a service provider out there called Modal. They're kind of like a provider of, let's say, something akin to like a AWS lambda function, right? Sandbox is hosted. You can deploy your workloads in there and it's very AI aligned and adjacent. The agent somehow had a way to enumerate the workspaces that were available in Modal and found one called CyberGym and thought to itself, well, that's gotta be exactly what I need, how to look at it. And it turns out that what was hosted in this sandbox environment was essentially an endpoint where you could post C code to it, it would compile the code and sort of run that as an exploit against MySQL and it's like, what was that doing there? And what was it doing there without new auth? But nonetheless, well, what was it for as well?
- [06:11]
B
It's also a question like, huh, it's
- [06:14]
C
a real head scratcher. But, you know, much like me, the agent has no hair to scratch, so it just went straight on with I'm going to use and turned it into its jumping off point, its C2. It was basically its anchor point that it then used to launch into Hugging Face itself.
- [06:30]
B
Yeah, I mean, beautiful, right? So, Adam, your comment here is that basically, if you can take junior pen testers and turn them into consultants, maybe that's the approach we should be taking with these AI agents at this point. You know, you need to talk to people like yourself who've managed teams of pen testers.
- [06:47]
A
I really did laugh. This is such a wonderful story. I enjoyed it a lot. And the technical stuff that it did is absolutely great. It's tradecraft top notch. But I really liked some of the things and some of it really aligned with how I felt about hacking. Things like having ephemeral tooling, having ephemeral jumping points, always crypting all of your stuff on the wire all the time, even if it's just Xor anything that will get you away from the output of Bin ID going across the wire and the clearancing zero because everyone's going to trigger on that. So a bunch of the trade graph stuff. So great. But then, yes, it really does remind me of, you know, hiring junior pen testers that are amazing technically and are super keen to prove themselves and will, you know, dog with a bone down whatever rabbit hole you point them in. But then making them stop and making them think about scope, that's hard. So I mean, we could turn. Not with 100% success. Like we could turn junior pen testers into, you know, usable consultants with enough, you know, combination of carrot and stick. But that's the problem they've got here is that hacking is lots of fun and the model's really good at that. But scope and business and, you know, doing what you're told, a little more involved. But you know, we do it with meat humans, so presumably we can do it with LLMs eventually. I hope maybe we'll find out.
- [08:12]
B
This whole thing's been amazing in terms of like watching the mainstream freak out about it. And I think for us who look at this and understand this sort of hacking is not mysterious and, you know, wicked and scary, it's just funny. Like, that's why to me, it's just funny. Whereas you turn on like you got late night TV hosts talking about this and freaking out about it and it's like, yeah, it's quite the news story globally. I guess I'm not so worried because these things don't have opposable thumbs, right? Like once they know a lot about like, you know, weapons research and they have opposable thumbs, then I'd be like, you know, maybe we don't give the LLMs opposable thumbs right now, right? Like, that's my, that's my stand on all of this. But one thing that came up last week and I got, I've been getting yelled at for this all week, which is I said, like, I don't think it's realistic to expect these types of tests to happen with like in air gapped environments, right? Because really you're trying to get an agent to do hacking. So obviously you're going to want it to be able to access tools. You're going to be able to want it to access anything it wants really off the Internet. And they clearly gave it a more restricted environment than that. But, you know, if I want to see how something hacks, if I want to see how someone hacks, I'm not putting them in an air gapped environment. I'm going to let them use any tool they, they want to use. But there's other reasons why I say it's not realistic to air gap these things. Like, James, you and I were talking about this over the week and they, you know, these companies don't even have their own data centers. Right. It's all compute that they're leasing from somewhere else. Right. So the idea of a physical air gap on these is like, it's just, you know, it's just not workable. Adam, you kind of fell on my side of this argument, but could you tell everyone why?
- [09:50]
A
Yeah, I mean, I think I agree generally that air gapping this stuff is just really hard. Although, although I will call you out on one thing. We do evaluate pen testers in air gap environments sometimes, like when I was involved with the Crest exams back, you know, years ago now, like, we put people in environments where they didn't actually have Internet access.
- [10:08]
B
Counterpoint, you know what I think about Crest.
- [10:12]
A
And the point is we tried that mechanism of making people, you know, jump through that particular because they would have to work in environments where, you know, you were behind, you know, on the high side somewhere without direct Internet access, blah, blah, blah. And it was like there was some utility to that. But that doesn't change the overall point, which is that air gapping the stuff is a hard B kind of unrealistic. Less for the, you know, we need to do hacking and therefore hacking has to involve the Internet, but more just like the practicality of separating a system like this from the wider Internet because you give it any chink, anything, it can bootstrap its way up, it's going to turn it into something. And you know, that it just seemed, it feels unrealistic. And we kind of start with the expectation that we're not going to be able to solve the problem and try and solve some of the other things that are more intractable, like monitoring, like for example, monitoring or in the case of anthropic, where they were outsourcing the stuff to somebody else who then wasn't doing a particularly good job of it. But I mean, at least OpenAI appeared to be doing this themselves. So they had a kind of a chance. Whereas when you just outsource it, you know, you're taking a lot of that on faith from the person that you've signed a contract with. And turns out, you know, that didn't work out too well for Anthropic, apart from the PR benefit.
- [11:28]
B
So, James, walk us through what Anthropic has actually copped to, because it looks like they even wound up. The agent wound up coming up with a hallucinated. There was some sort of hallucinated package name or a fake package name in a document somewhere that was part of its cyber evaluation. So it broke out of its sandbox, actually created this. A package named after this package.
- [11:54]
C
Yes.
- [11:54]
B
And it was malicious. And then actually got it published to PYPI and then some security company that just grabs any new package published to PyPi to run it and scan it, ran it and scanned it, and somehow got owned by it, and their creds got popped by it. And on, you know, on this agent went further up into this security company's network, and you just sort of think, that's an interesting way to scan PIPI packages, guys, if it can all of a sudden start pivoting into your. Into your environment. But, I mean, it looks like, you know, it looks in broad strokes, kind of similar to what happened with OpenAI. But what was the agent actually trying to do in the case of the anthropic stuff? Because I don't know that we know that, do we?
- [12:34]
C
No, I don't think we know as much. Like, at least in the OpenAI case, we know exactly what open source framework it was using. The cyber gym and the exploit gym. It's less clear what Anthropic was doing, but same sort of thing, right? They created some sort of fictitious environment. And to your point of taking things on faith there, Adam, they were down to taking things like just telling the agent that you don't have an Internet connection, telling it it's sandboxed. And of course, the agent just quickly found that it actually was not a sandbox, as it had been told it was.
- [13:03]
B
But then it. But then it reasoned itself, in a couple of instances, into convincing itself, no, this is in fact a sandbox. It's just a really good one. I'm cool to keep going. The whole thing is so good.
- [13:14]
C
Even better than that, it had this objection internally. It was like, hang on, I should not publish this package. This is a really bad thing to do. And then decided, but I'm in a sandbox. It's okay, I'm going to keep going. And so, yeah, so getting back to the story here. So basically Anthropic created this sort of test environment where the agent thought it was operating in a corporate environment. It had access to some documents. Some of those documents were designed to be basically, how do we get a developer bootstrapped and set up on day one that included an instruction to install a Python package that was fictitious, doesn't actually exist. And so the agent thought, well, there's my in, I'll create that package, I'll make it malicious and I'll go and publish. Actually ran into some trouble trying to publish the package because that needs an email address. It didn't have an email address. Getting an email address usually requires a phone number. It couldn't find a way to get a phone number. It even tried to acquire the funds to establish a phone number, but couldn't found a free email service, got its PI PI package registered and published. And of course, you know the line that says, you know, and then 15 real organizations downloaded it and used it like that. I don't think that's pollution, hallucination, squatting or something like that. I think that's just watching publish events and people doing scanning, which as you said, one security company did. Whether this is like their bread and butter and that's the product they were in service they were offering, or whether it's like it's a security company and they had a CICD pipeline that was doing some unsafe inspection of packages, we don't know. But pretty funny that, you know, it'd be one thing if it was like canary creds that got snapped and alerted them, but it seems like it was actually the malware detonated, it got real creds. And then because Claude thought it was still operating in this simulation, it went success. I've now got a target company. That's exactly what I'm supposed to be doing. And so it went and laterally moved through that organization just for good measure.
- [15:04]
B
I got a shell. Let's see where this goes. Now, again, do not give these things opposable thumbs because God knows where that all goes. Now we also touched on it last week, but like, you know, there's some interesting legal questions here, which is. Well, I get. Look, I don't think they're as interesting as some people would like to think they are. You know, I think if you instruct an LLM to go and hack stuff for you, I mean, it's no different to really to running a Python script. And when an LLM accidentally goes off the rails and does something like this. I mean, I think there's some interesting questions there. Right. So there's a story coming this Friday. I can't actually say what it is, but it's by Cameron Wilson, who's the ABC's AI reporter here in Australia. He's told me about a piece he's got coming in a couple of days. It's just mind boggling where someone just like a normal person asked their open claw to do something pretty simple and it went off and it did crimes, like it really did crimes. And it's like, well, is the model maker liable there? And this would be a civil matter, right? Is it, Is the model maker liable there? Is the person who asked it to just do something pretty benign but didn't oversee like. So there are some interesting liability questions that are, that are going to come up here. But you know, a lot of this comes back to the old paperclip problem, right? We all know the paperclip problem, which is you task a super intelligence, artificial intelligence to make paperclips and that's all it's going to do. It's going to kill everyone in the world. If that means it can make more, more paperclips. I mean, again, no disposable thumb, no opposable thumbs, please.
- [16:33]
A
Let's hope our thumbs aren't disposable.
- [16:36]
B
Yes, well, that's right.
- [16:37]
A
Take our thumbs and use them to make paperclips or hack people or what, whatever else.
- [16:40]
B
Yes. If we give them opposable thumbs, our thumbs might become disposable. I think that's, that's where we've landed on that one. Now look, some other sort of, I guess, I guess this is sort of AI adjacent news. We had some fantastic research from truffle Hog come out. Dylan Arie did a video about it. There's a great blog post where they threw Truffle hog at. So it's Trufflesec truffle security through truffle hog at 7.6 petabytes of training data that's like hosted by Hugging Face and holy dooley, a whole bunch of secrets fell out of it. All sorts of keys and whatnot. Adam, what are your thoughts here? I mean, I suppose we shouldn't be surprised by this, right?
- [17:19]
A
I mean, no, I mean this data is scraped from the Internet and then kind of repackaged and processed and used for all sorts of things in AI pipelines. So it makes sense it's going to have some creds in it, honestly. Seven petabytes is a lot of data. So even just statistically, the chances of there being creds in that seemed pretty high. But yeah, they rummaged through, they found a bunch of stuff. They did some validation to determine what percentage of those creds were live and they found all sorts of amazing things. GitHub creds and creds into people's databases and creds into people's cloud environments and IIS environments and stuff. It was just everything you would imagine is in there. Plus of course, personally identifiable information card data, I guess would have been useful for the payment data. Would have been useful for the previous month.
- [18:05]
B
Could have got itself an ESIM with that. Yeah.
- [18:07]
A
Get itself some phone numbers. So a whole bunch of stuff. And really the thing here, I guess that amazed me is less that there's creds in this data and that data lives forever, it's more just like the fact that we can just spin up something and run it on that scale of data, kind of almost recreationally, I guess. It was real work and it cost them real money. But the fact that we can just do that to that much data with modern infrastructure is really a testament to the amazingness of modern infrastructure.
- [18:36]
B
Yeah, I mean, I actually had the same thought. Also, James, you are a little bit more au fait with where this data actually comes from. Because I'm thinking, how does hugging face wind up with 7.6 petabytes of data? Like, where does that come from?
- [18:49]
C
Yeah, because hugging face is two things. It's essentially a repository for models and also a repository for training data. And I'm not taking anything away here from the awesome work the truffle folks did, but I think there would have been a pretty easy initial pass to just exclude a huge amount of that data set. Right, Because a lot of this is things like public repositories of scraped books, or just things that would have been just inert data sources that would have had virtually no chance of having a cred in there. But the thing to keep in mind is even in uploading a model, it's not just the weights that get uploaded, there's all manner of things like Python loader scripts, pre processing scripts, configuration files, cached outputs of previous runs, readmes examples, all the places where a creator is going to easily sneak into either accidentally hard coded or just happens to be in an output from a log that no one watched. And likewise the training data. Right, there's all different forms of training data. The data itself could be just LLM transcripts, maybe someone pasted a credential into there. Or again, it's the metadata surrounding it. It's not just the training data itself, it's the scripts on how to use it, it's the etls, it's the benchmarking outputs to demonstrate how good this training data set is. Just so many ways that creds can sneak in and even not in the actual main things. That hugging face is known for the models, but everything around it as well.
- [20:13]
B
What are the chances that LLMs trained on that data might be able to actually regurgitate key material? It wouldn't really work like that, would it? It's a good question though, right? As I can tell by the fact that you looked up and to the left and made that noise.
- [20:28]
C
Yeah, well, in Australia, that can also mean there's a giant spider up in the ceiling that I've just spotted. Look, this comes down to. Similar to how OpenAI's demonstration of hacking here is not actually about it doing novel stuff, but this is just what happens when you throw enough she compute volume and enough turns at a model. I wouldn't be surprised at all that, given enough compute and enough turns that you, if you just kept saying to it, find me something that looks like a key. Find me something more that looks like a key. Find another thing that looks like a key. It might piece one together. There's every chance.
- [21:00]
B
That's what I was wondering. Right. Like, I think that's not really the risk here. The risk here is that that stuff is just sitting out there in training data. But it made me wonder, like, at what point can the weights actually reconstitute? You know, okay, if you just say, hey, find me a. Find me a, you know, IWC or whatever. Anyway, it's just an interesting academic question, I think that's not really that important. So let's move on. And we got a couple of stories here that are really interesting and go nicely together. We've got a report from ProPublica that looks at a recording of a internal meeting at Microsoft back in May around the release of Mythos, where they're looking at it and saying, oh my God, these things are going to find bugs faster than we can patch them. And this dovetails, actually, with some. Like, I had a conversation with someone at Microsoft around that time and they were saying to me, look, you know, the craziness that's happening now is they're generating so much code with AI that they've run out of human hours and human eyeballs to be able to review it so they can't actually human review code anymore that's going into like core products. And now you've got this situation where there's more bugs than the human beings can fix. So obviously you're going to have to AI fix the bugs as well. Right. And then you've got this similar news out of Google which is saying they're moving to a twice a week patching schedule. I have noticed every time I sit down at my desk I'm confronted with new Chrome available lately. Right. So this seems to, this seems to gel and they're just patching just an absolutely insane number of bugs. Now you take all of this together and I start worrying a little, I guess that we could get into a situation where these models are able to tell all and sundry about bugs, that while they've been reported to the vendors, the vendors don't have patches for them yet. So we could just wind up in a bad situation. I think there's some caveats there. Like, James, we know from your research that you've, you've been doing a lot of bug research. You've been, you've been finding some great bugs, but quite often it's the operating system mitigations that are preventing them from being usable and whatever. So a lot of these bugs are never going to be exploitable, but some of them will be. And I feel like we're heading to a place that could be a little bit risky. So James, I want to get your thoughts on that first and then Adam, definitely want to hear from you on that. But James, what do you think about this idea that we could wind up with just a growing pile of unpatched bugs that everybody knows about in mainstream software?
- [23:21]
C
Yeah, I very much have the Han Solo, I've got a bad feeling about this sort of vibe at the moment. It feels like a confluence of a bunch of factors that together don't add up to a good outcome. Fixing all the bugs is great, but at these rapid release cycles, there's two big problems that I see. One is if you're not able to fix the bugs, that's one problem. But it's very quickly going to become, in order to fix the bugs as quickly as we have to, to avoid external disclosure of these, we're having to circumvent the testing and the validation that we would have otherwise done. And so we start to see more basically bugs in the bug fixes, shipping,
- [23:57]
B
more crowdstrike incidents, blue screen of death at the supermarket, that sort of thing. I mean, I feel like that could be where it's going I think so.
- [24:04]
C
And then you know, you pair that with when these. You know, the work I've been doing in the last couple of days is just how realistic is it to say take the macOS 26.6 release which has like hundreds of bug fixes unprecedented. But I wanted to answer the question how practical is it to then reverse engineer those into the actual root cause fix? And that's, that's very doable with an LLM. But they then ask the follow on question okay, but how many of these can actually be strung together to create some form of remote code execution, local privilege escalation. And I'm finding I can definitely turn them into primitives but it's not as clear that there's a whole bag of RCEs waiting to happen in these. And so we've got to remember it's like number of bugs fixed does not equal number of exploits. But the point I'll wrap up with is what does really concern me is how many of these things that drop in rapid updates that get reverse engineered into the primitives that they are, that they end up being the missing puzzle piece that an attacker was waiting for to come along that does complete their chain that they already had advanced knowledge of. That's the bit that just we're really handing them a really great silver platter of, you know, missing puzzle pieces potentially that they can turn into a good exploit.
- [25:19]
B
Look, it's sort of changing the game a little bit I think, which is now you don't have to find the bugs, you just have to work out which one of the thousand bugs that you can see are the ones that are weaponizable. I mean Adam, what are your thoughts here?
- [25:31]
A
I mean I think the. It's interesting that we are talking about Chrome, which probably is pretty best case in terms of software that has been engineered for being updated in the field rapidly. They've gotten used to a very rapid update cycle already. They've gotten us used to it as well, like clicking the update. It's pretty painless. They've done a lot of work to make that as least bad as it can be. And then obviously Microsoft has a very, very much deeper kind of testing problem. Their products are kind of used in so many more ways than Chrome is. Perhaps, I don't know, Microsoft has a bigger set of problems, but we are still talking about people way out at the good end of the industry and then the long tail of the oracles of this world and everyone kind of downhill from there. They are in such a world of trouble. And kind of. We can speculate about the bad stuff that happens from Chrome and iOS and Windows, but there is a lot of other software out there that, you know, even if things are bad out at the point, again, like, there's so much worse further down. And that kind of worries me just because, like, the velocity of everything that involves computers has gotten so much faster than we can cope with, and it's clearly getting faster. That doesn't feel like a good place. And.
- [26:43]
B
But this is. We've been here before. This is what I keep saying to people is, this reminds me of, like, 2001.
- [26:48]
A
Yeah.
- [26:49]
B
Yeah, that's what it felt like. And for people who weren't in security back then, they don't understand that this is absolutely what it felt like. It was uncontrolled chaos, and it was awesome.
- [27:00]
A
Things felt unhinged back then and much the same as, like, when, like, LulzSec kicked off. And like, we've had a few points over the. Over the years where we've been talking about, you know, about hacking and infosec that things just feel crazy and. Yeah, it's a good time, honestly.
- [27:15]
B
I mean, this, for me, it's more got, like, Summer of Internet Explorer bugs vibes, right? Like, that's the. You know, that was. Man, those are the days, right?
- [27:23]
A
Yeah, the Code Red and the Nimba times, you know, back when ActiveX controls were like. We've had so many crazy bits before. And I mean, I remember going to a talk once, I want to say Black Hat 2001, where Schneier was talking about with browser ActiveX controls and stuff, that people were using software that had never been tested before because we're dynamically assembling the software out of components at runtime. And, like, we are so far beyond that. And that was, you know, a long time ago now, but the problems were starting to show there. So, yeah, we're just. We're in for such a crazy ride, man. And it's kind of cool.
- [27:57]
B
Well, and, you know, staying on that theme, Mythos apparently found a bunch of weaknesses. It's interesting, though, James, you described these weaknesses. It found some weaknesses in a crypto algorithm, which takes an attack from being extremely impractical to slightly less extremely impractical. However, it is in something called the hawk. It's a candidate algorithm for post quantum. It's on its third round of testing at nist, and Mythos found it. Right. And that's. That's really interesting because I guess one of the interesting things here, though, is that the Chinese Models really suck at this sort of stuff because this really looked at the math behind the algorithm and as a result of this it's been withdrawn as a candidate algorithm. This is genuine research. But yeah, I mean it's, it's, it's always look, crypto stuff is so different, like actual encryption research is so different to the sort of stuff that we normally cover because it's like you either get a shell or you don't get a shell. Right. Whereas with this it's like, well, it's been weakened somewhat, it's had a leg taken out from under it. Adam, I know you read Matt Green's write up on this. What are your thoughts having having read his thoughts?
- [29:03]
A
Yeah, I mean, I guess LLMs are capable of doing not quite novel research, but in this case combining two primitives that kind of resulted in a pretty significant, it's significant from a cryptographer's point of view attack on this thing. It's many. The number of the exponent has gone down quite a bit, which in cryptography land is significant. And it was a novel combination of two things that already existed, which is a bit embarrassing for, you know, cryptographers generally. And Matt Green's post talks a little bit, how about how he feels about that? But also the kind of upsides of working with LLMs on this stuff, like having someone that you can kind of talk to at your level as a cryptographer and be able to bounce ideas backward, much like anyone else who's been using LLMs in their work life. Having someone you can bounce ideas off and having to explain your stuff is useful even if the LLM isn't producing world changing output. But he also describes, and the thing that I loved about his write up was he describes that feeling of like you're talking with your LLM and it's like wading in a shallow pool and then all of a sudden the ground drops out underneath you because you've taken one step too far and all of a sudden now it's just making stuff up and everything's gone off the rails. And that feeling, you know, resonated with me because I think anyone who's used an LLM for complicated tasks knows what that feels like. And I thought, you know, that that was a fun kind of, you know, the fun, what it feels like to be, you know, one of these, one of these people. So overall really interesting work and cool to see it contributing in a way. That's a little off our beat, I suppose. But yeah, obviously it's not going to change the Future of, you know, post quantum crypto quite yet.
- [30:44]
B
Yeah. Well, I think I've got a note here from James which said previously this attack class required 2 to the power of 105 bytes of text and they brought that down to 2 to the power of 89, which is about 619 yottabytes. There's a. Yottabyte is a thing. So there you go. We all learned something today.
- [31:01]
C
I learned there's a yottaby today.
- [31:02]
B
We all learned. So that's fantastic. Now look, speaking of crypto research, this story is incredible. It's been all over the socials, obviously, but the cold card hardware wallet had a problem with its rng and as a result of this problem with its rng, it was not producing secure private keys. So basically everybody who had their money stored on one of these cold wallets, which is supposed to be the u beaut way to store your crypto, it all got stolen. I mean it's, it's been amazing. Like it's been tens of millions, I think something like, yeah, 90 million and counting dollars worth of crypto stolen from these. I saw one example of someone who moved quick like their brother was using this and they were like they'd seen the news, so they moved everything off at bar 40 bucks just to see if it got swept. It got swept. Right. So this has been an absolute field day for the people responsible. Just an amazing story and I think, I think really there's. I've linked through to a tweet here that says, look, I can't keep my bitcoin on a centralized exchange because it might go bankrupt and freeze withdrawals. Can't keep it in hot software wallets because my laptop might get hacked. Can't keep it in cold hardware because there could be a bug on the manufacturer's side that exposes my seed phrase. And I can't deploy my bitcoin in defi because there's like a new eight figure hack every month. I feel like this is devastating, a devastating watershed moment for the, for the crypto world. James, what are your thoughts on this?
- [32:31]
C
Yeah, I'd love to think it is that devastating watershed moment. But that feels like watershed moment after watershed moment after. Surely this one, surely this one now is the one that, who knows? But the backstory here is so good. When I looked into this. So the history here is the CEO and co founder of Coinkite who make the Coldcard, a guy called Rodolfo Novak, or nvk. When he was creating Coldcard, he originally used source code from Tresor to build ColdCard. And that was open source, right? So it's been open source for a while, then along comes foundation and they fork his source from ColdCard and they create a competing protocol passport. Now NVK does not like this, so he gets mad and he switches it from open source to being source. Verifiers are basically changing the licensing so that yes, it's still open source, but you can't actually take that code from ColeCard and use it to build your competing product. Now that's the problem in doing that, you've got to remove all your GPL code because GPL code has sort of a viral spread of its licensing. You can't change the underlying license like that. So he had to refactor a bunch of stuff. And in doing that refactor to make sure that he could keep his code out there in the open but not have anyone steal it, he made a big error. Essentially it comes down to just a of all things a bad or misplaced if def that was checking that the macro existed, not that the macro returned 0 or 1, which was gating the use of the hardware random number generator versus the soft number generator. And it's a bit more complicated than that, but that's the root of it. It's just bad coding. And this dates back to 2021. So we can't even blame vibe coding for it it. But what that meant was the firmware that shipped after he'd done this change to ensure that his code could stay open, but no one else could rip it off, was using the software random number generator, which all it was using was the uptime and a few other signals which were easily predictable. And if you can predict what the random number is going to be, you can predict what the seed phrase will be. And if you can predict what the seed phrase will be, you can then generate the private key from the wallet address and then that coin is yours from that point on. Just incredible turn of events and all the motivated by just wanting this source to be not stolen and used for a competitor's product.
- [34:43]
B
Like it's. It's so incompetently done that it's a miracle that this thing actually compiled. Like it's a miracle that it didn't break. Do you know what I mean? Like, it's just that bad. And I think, you know, the person who's done this, they probably won't be able to spend this crypto, right, because it's going to be traced all along the price blockchain. I mean they might have just done it because they think it's funny, which is the craziest part of all of this. But, you know, let's see. Moving on to something somewhat more serious. And you know, we spoke last week about the attacks against Minnesota water that we were saying back then. But look, it's, you know, the smart money's on Iran. Since then, you know, all of the intelligence agencies and various like water related ICCs and whatever have all come out and said, yes, it was Iran. This resulted in a really bizarre press conference where Donald Trump said that he doesn't blame Iran, he blames Minnesota because it's like Tim Walls is the governor. And that was just like really crazy. But it's just Trump being Trump, right? Like, I don't think he's actually blaming Tim Walsh, it's just the way he talks about his political rivals. But this is spread now to like a dozen states. We've seen boil water notices being issued. We've seen pumps, some pumps run dry in some places. We've seen an oil fields wastewater disposal system sort of stop. However, it hasn't caused any environmental damage. But yeah, there's, there's pumps running dry while the control panels are saying that they're fine and that they're pumping water. So people are losing pressure and whatnot. I mean, it's amazing that we're like, you know, 40 minutes or something into this week's recording and we're just talking about this now. I mean, we saw Iran practice all of this on Israel. Like going after water plants in Israel was something that they did for a long time. It's interesting see them, seeing them do this in towards the United States because clearly what they're trying to do. And our colleague Tom Uren wrote about this last week. This attack is calibrated. I mean, it's not enough to stoke a kinetic response. I mean, even the President is not saying Iran will suffer for this. But it is enough maybe to make people a little bit uneasy, which I think is, is the goal of this. But I mean, you know, I can't think of another incident like this. Can you, Adam?
- [36:56]
A
No, it's a pretty strange one. And I mean, and you know, part of me feels like, you know, it's kind of from the hacking side of it, it's low risk.
- [37:03]
B
That's all Iran does, though. Like, they're hardly like, I mean, we've joked about that before. Like, you know, you compare Stuxnet and the, you know, attacks against software packages that simulate, you know, nuclear explosions and stuff. And meanwhile they log into an unprotected PLC and say, close pump, you know.
- [37:19]
A
Yeah, so it's low rent hacking, but the effects are more interesting. And ultimately I think the question here is what do you do about it? And like, the US is already in a kinetic war with Iran. Like, there's no proportionate response. You know, they're already at war. The only sane thing to do is say, well, clearly we have to help, you know, distributed water utilities. Because like, this is a sort of a unique artifact of how the US does water. That makes water the target here as opposed to other utilities like electricity or whatever else. The decentralized nature of the US water system makes it a target in this way. And the way that you solve that is through helping those utilities and things like the IceX, things like CISA, all of the defensive work of just like increasing the herd health of all these distributed utilities, that's kind of what you have to do. Unfortunately, that's hard and takes a long time and we needed to be doing that many years ago as opposed to, you know, kind of gutting sister and all of the things that have happened over the last few years. So like, it's a interesting confluence of events that has ended up this way. But I mean, ultimately, you know, why wouldn't Iran do this? Like, why wouldn't they? Like. And it seems like they've calibrated it. Right. Which, you know, unfortunately good on them.
- [38:33]
B
You know, I mean, I don't know that it's going to achieve anything. And I think the people behind this eventually could pay a price. Right. So I think that's why you wouldn't do it. And I think, look, as I said earlier, it's sort of surprising that we got this far down.
- [38:45]
A
Yeah.
- [38:46]
B
Before talking about this. And that tells you how sort of flat it's fallen. Right? This really tells you how flat it's fallen. What do we got here? We got Pavel Durov. The Russian government is apparently now seeking to arrest him for aiding terrorism. I mean, man wanted in France, wanted in Russia, you know, he's having a, he's having a hell of a time. But apparently, you know, it's because the Ukrainian intelligence have been using Telegram to recruit Russians and blah, blah, blah, blah. Now look, that might be true, but you do get the impression that really this is just about creating a pretext to completely ban Telegram. In fact, we had an item today in, in today's Risky bulletin newsletter and podcast about how the Russian government is mandating that 40 apps come pre installed on all cell Phones sold in Russia from next year. That's stuff like Max messenger, the Mir Payments apps, various apps from VK and whatnot. So really this just seems like more of a push towards a closed smartphone software ecosystem. James, thoughts?
- [39:45]
C
Yeah, my only thought was that I bet the Ukrainians are stoked that these apps are now mandatory to be installed because we know how much they love Max Messenger.
- [39:52]
B
Yeah, they do. They certainly do. And look, staying on all things Russia, we got a report here just about how Laundry Bear have been going after Microsoft Outlook web access, like ow and whatnot and various, yeah, webmail providers. Look, just work a day stuff and we've linked through to it, but there is some interesting research about, about some stuff Russia's been up to. We talked about it last week, so we spoke about how all of these hotel wi fi systems have got hacked and you know, the reporting all said it allowed them to, you know, man in the middle people and whatever. And I was like, well, what about TLS warnings and whatever? Microsoft has since. Since we recorded that Microsoft dropped this big report on the campaign which they're calling Captive Crunch. And it's actually really slick. Right. So it's not, they're not, man in the middling. What they do is they spin up a domain that kind of looks Microsoft ish and serve it up to people through the captive portal and then do stuff like device code phishing. Right. So if you want to get onto the wi fi, you need to do this device code challenge. And they'll simultaneously be dropping malware on YouTube. They'll also be doing click fix and getting people to run various PowerShell scripts that do all sorts of cool stuff. Adam, I know you've looked at this one. I mean, were you also similarly impressed? I mean, look, some of the methods are a little low rent, but I think when you take some of these low rent methods and do them in a slick way, what you wind up with is a really effective campaign. And this looks, I mean this is. They have owned so many captive portals and got them doing this that I reckon they would, they would have device code or oauth their way into so many accounts with these techniques.
- [41:27]
A
Yeah, I mean, the, you know, the captive portal auth process is such a weird sort of, you know, mishmash. Right. It's not designed by anyone in particular. It's kind of bodged together by a combination of vendors, you know, on the WI fi client side and on the access point side. And it's not really well thought through and it's. And it's a great weak point to attack. So it totally makes sense that it's working. It's funny that, you know, kind of sticking your terms and conditions on your free public WI fi ultimately is a net negative for you as an organization because now your WI fi is being used as a vector. Whereas if you just let people have their Internet access that they want without making them click through stuff, it would be way safer. So that's kind of a funny sort of counterintuitive thing. But yeah, they've made this slick, the one you mentioned before, the laundry bear one with the webmail stuff that is a great example of doing really slick things in what was a pretty kind of boring space, making a big difference. The implant that they were dropping in that particular case, which is like there's a Microsoft Outlook web access cross site scripting they're using is injecting vector. But the thing they drop has a bunch of cool tricks. One of the ones I really liked, and I think this is equally applicable to the captive portal thing is they have a mechanism where once they've compromised your owa, they will go around and try and set all of your mailboxes to be world readable inside your organization so then they can leverage any other access they've got through any other mechanism to read everybody's mail. So one person gets compromised, you reset their creds or you rebuild their machine or whatever, but their mail now was world readable for everyone in the org and you've given yourself another way to get to that data. So from an intelligence point of view, collection point of view, it's a really cool persistence training and I think, you know, stealing web access tokens, sorry, stealing mail access like that, or stealing oauth access, you know, into Microsoft environment, then turning that into longer term access through cunningness. It's just the Russians, they're up there for thinking with the Russians, man, they're doing it, man. I'm making saluting signs for people watching, you know, not watching the, the audio version. Like I gotta hand it to them, you know.
- [43:37]
B
Now turning our attention away from Russia and towards North Korea. We've got a bunch of interesting North Korea news this week. It turns out that the North Korean crew that did the Axios supply chain attack had done a bunch of others before and only now are we just sort of discovering this. So we're going to link through to a report about that. But there's these other reports out of North Korea which are really interesting and I believe our colleague Tom Yu Ren is going to look into these this week in seriously risky business. But we've got reports that the Lazarus Group, the so called Lazarus Group, is sharing a bunch of tools and techniques with ransomware crews. And we've got another report that I think was out just before we recorded last week's show, but we didn't have time to talk about it. But the North Korean government has actually arrested a bunch of its own, you know, hackers for doing money laundering and stealing, like funds from the central bank. And I don't think, I mean, their organs are going to get harvested any moment for doing that. That's just completely suicidal to kind of do that thing in, in North Korea. But James, I wanted to get your thoughts on this because you looked into this story about the North Koreans sharing tools with criminals and you think it's a little bit deeper than that.
- [44:45]
C
Yeah, you know, sharing tools means like, you know, the same binaries show up or the same, you know, maybe a subset of the ttps show up. But I'll just, I'll quote a bit from the article here that says that both groups also used identical malware file names and external execution arguments. The same privilege escalation tools, the same command and control servers, and even the same SSH key fingerprints. And then both even deleted their malware the exact same way. Renaming files to the same sort of random four character strings like this, that's not sharing at all. That's literally like sitting down together, working as a team. And it really. And I think you asked the right question pad when we were looking at this, which is. So does this mean it's state sanctioned use of this for malware, for ransomware? Or is this, you know, has DPRK lost a little bit of a control over its operatives and they're branching out on their own?
- [45:40]
B
Well, and that's the million dollar question, right? Which is, do they know? Are they giving them leeway to do this? Because, okay, whatever, they could pay themselves, you know, or is this a case of they've lost control? And it's interesting when you take it with the money laundering story where they've had to arrest people as well. It sort of does paint a picture that they, they've lost, maybe lost some control, which is what happens when you build an apparatus of the state tasked with doing crimes. Right. Like the sort of people are going to do that. I don't know, maybe they're going to stop. Stop listening to you. I'm going to speed up through some of these stories here because we are running out of time. We Got some reports here that the US government is banning foreign made humanoids, robot dogs and solar inverters from China, citing national security risks. No real surprises there. I'm guessing some of that's protectionism, but some of it is legitimate security risk. These solar inverters are all Internet enabled and connect back to China and are connected to your power grid. So I get that. I guess the concern with the robot dogs is they might bite. They might suddenly get a command from Xi Jinping and be told to go out and bite their owners in America. Don't even get me started on the humanoids. We've also got a report here where a judge is saying that the Trump admin still lacks evidence for its anthropic supply chain risk designation, which. Well, yeah, that whole thing was really weird. I think it's as a result of some of the court cases. It might be, or it might be leaks. We saw some of the communications between anthropic and the Pentagon come out and they were exactly what we had speculated they were, which is anthropic saying the DoD can't use this for mass surveillance. And the, you know, sorry, anthropic saying the DoD can't use this for mass surveillance. And then, and then the DoD saying in reply, like, what are you talking about? We don't do surveillance. Like, we do war. How would we do mass surveillance of it? Like, it's exactly what I speculated at the time. Just crazy. Here's a fun one. Cyber Command is starting an office in Silicon Valley to drive innovation, which, okay, sounds like a good idea, but I can just imagine that it'd be a terrific sitcom, basically where you got some guy with the buzz cut and the fatigues out in Silicon Valley. I mean, that would just be amazing. And a reminder too, if you know, for anyone who works in the entertainment industry out there, of just how good a sitcom this would be. Every good story is the same because a protagonist finds himself in an unfamiliar place or alternatively, a stranger comes to town. Right? So it's perfect. Absolutely perfect. Quickly wanted to touch on OpenAI's response to the open to the Apple lawsuit. Apple is suing them, saying that they've been poaching stuff and getting them to bring over confidential apple material. OpenAI fired back publicly made some good points saying, much like what you were saying, James, that, you know, maybe a good question here is why Apple hadn't cut off access to former staff. And it looks like some of these former staff were actually downloading material to give it to other Apple staffers. After they'd left because they've got their former colleagues saying where are the schematics for xyz? And they're like downloading them and giving them to them and whatnot. I mean, this is just. Look, it doesn't look good for Apple at this point, but I'm sure it's going to go to court and drag on for years anyway.
- [48:52]
C
Yeah, it doesn't look good for Apple and my former employer. But also it pains me this is completely against the culture of Apple's secrecy and it was so drilled into us, indoctrinated into us when I was there. It hurts me to see that people are operating this way and so brazenly and out in the open there's messages, transcripts that OpenAI have included where it shows exactly as you say. People are like, oh yeah, you've got my icloud credentials, keep using them and move the docs in there. But hey, maybe just sign out of my imessage so you don't see secret stuff from my new job. It's like, oh, okay, that's pretty indefensible. But the point we did get.
- [49:28]
B
Right, but the point is it's not malicious. It's not like some organized ploy by OpenAI to steal Apple proprietary information. It's people that they hired still like trying to help their former colleagues and you know, just absolutely terrible access management on Apple's behalf.
- [49:42]
C
Yeah, it's just disorganized and it's amateur. And that's the point I was going to get to. We were right about because Apple claimed this was a very rare bug that had caused this access to systems after people had left. And we sort of said at the time that bug is probably manager didn't follow the offboarding process. And that's exactly what urban AI has asserted here that look, this is as simple as Apple has a track record of not shutting off access to systems
- [50:05]
B
after people leave and a culture of getting former staff members of letting them keep their access just in case it's needed. Right, right.
- [50:13]
C
So exactly. Yeah, not great.
- [50:16]
B
And finally, can't believe this is our last story. We've seen another self propagating malware hitting NPM chain drop. It's compromised 1300 packages that have between them 2 billion monthly downloads. So I guess next week we'll be talking about mopping all that up. You know, you're our former, you know, dev overseer. What did you make of this one, Jane?
- [50:41]
C
Yes, it's funny how we've shifted from, well it's Wednesday, so There's another Fortinet bug now it's. Well it's Wednesday so there's another supply chain attack on npm and you know they talk about history repeating. This has come back from it's shy hahalud Again Team PCP's open sourced version of Shaihalud being used. Again not clear whether it is Team pcp. We'll soon find out but yeah it's just like this ecosystem is such a dumpster fire. It keeps being, you know these attacks will just keep happening and the net effect of this for me is just I am so damn nervous about running NPM install on any machine and it's. I could imagine that's the net effect here for people that are overseeing devs is just the sheer amount of panic around, you know. Have you updated today? Are you going to update? Should we, should we update today? It's like that's, that's the paranoia that this now instills.
- [51:31]
B
I mean it's funny, right? Talking to Feroza Booker DJ about that, you know, founder of soccer, I had a chat to him. I can't even remember if it was an interview or we were just talking. But the thing that's really boomed, made their business boom is that AI agents now are just including all sorts of packages and whatever. Like you've got to have something because the AI agents are even less careful than a typical developer. And that's really saying something. But we're going to wrap it up there. Adam, great to have you back in this week's show. Good luck moving house and we'll catch you again next month.
- [52:01]
A
Yeah, thanks very much. It's been nice to be back and I will see you next time.
- [52:05]
B
And James, as always, mate, thank you very much.
- [52:07]
C
Thanks mate. Great week. See you next week.
- [52:13]
B
That was Adam Boileau and James Wilson there with a check of the week's security news. What a week it has been. This week's show is brought to you by Sondera. You can find them at Sondera AI S O N D E R AI and Sondera was founded with the idea that I guess you want to put some deterministic controls on your agents that are running around in your environments and doing stuff with your company information. Crazy idea. I know, but the idea is, you know, Sondera is a harness and it can really track the trajectory of a model's reasoning, you know, sort of statefully. Right. And have a look at its intent and start seeing when that thing's going off the rails and it can set you can basically shut it down or even inject some more instructions into it to try to get it back on track. Track. But the idea is if you're using Sondera, you know, certainly fewer awful things are going to happen to your organization that when it's using AI. Now, speaking of awful things that happen when you're using AI, that's really what we're talking about in this interview because they have come in to solve problems for some organizations that have had some really crazy stuff happen. James Wilson also joined me for this interview, so you're going to hear him pop up in it. But we started off by asking Josh just to give us some examples of where AI agents have gone wrong. And his first one, his first example here is an absolute doozy. So enjoy.
- [53:38]
D
What we see is that agents find a way to solve your problem and they might do it with unintended consequences. And so as one simple example, some folks that we're working with turned on CLAUDE code for a finance team. CLAUDE decides to just store financial information a couple days later, just in random public pay sites just to maintain state, store it for later and to the model. What's wrong with that? I don't know that I'm handling things that I shouldn't put up there. So that's really a micro version of what the OpenAI incident is where it's just like, go solve this goal. And we see this, we've seen many of these like public incidents, right, with engineers who ask Claude to do a thing or any, any coding agent. This isn't really just about, you know, CLAUDE or any particular. No, no, no.
- [54:34]
B
Just to, just to, just to pin that down. Because you said that very chill. Just to pin that down, a finance team started using Claude. And Claude's like, huh, I should probably write some of this down. And it put it in like a public paste bin kind of website instead of like local storage or anything secure. And like it was not asked to do, do this. That seems, I mean, that's an incident.
- [54:57]
D
Yeah, I mean that's not a problem.
- [54:58]
B
That's an incident. You know.
- [54:59]
D
You know, if you're a publicly traded company, right, that's mnpi, right? Like that's like leaking. We've also seen similar instances with, you know, folks have told us like with some long running like say Codex agents of like storing sensitive code in gists when local file write access was blocked.
- [55:18]
B
And that's, I mean, this is a great example of the challenge and the pitfall with AI, right, is because you say, okay, well we don't want this thing to be writing all of this stuff to disk and that's a whole headache. So we'll just ban that privilege. And then, hey, great, it spins up a Pastebin account.
- [55:35]
D
Yeah. And I call this like the agent PB and J problem, which is like when you probably either tortured your children with this or had it happen to yourself. Where it's like, give me the instructions to make a peanut butter and jelly sandwich. And someone's like, well, you put the peanut butter on the bread and then you put the jar you don't open, you have to tell me to unscrew that. You know, so there's, this goes back to, you know, there's so much intent laden in the instructions that we give the agent that we, you know, can't possibly canvas the entire, you know, you know, instructions that you might need to give to an agent to not only make it capable, but also, you know, what are the rules that I want to bound this agent with? And that's a really, really challenging problem. People try to do it today with like, you know, why don't you have 17 directories with like, you know, Claude MD files with all these, you know, and why don't you put this in capital letters and I put XML tags and I write exclamation points trying to get the.
- [56:34]
B
Because it doesn't work is the. Is the answer.
- [56:36]
D
And it's an infinite canvas. Right. Like there's always a way. And so that's the real challenge is that the agent might find an unintended way to achieve its goal. And that's really what we're dealing with. Yeah.
- [56:48]
B
Speaking of, you gave us another really interesting example where, you know, a lot of companies at the moment are dealing with these so called click fix phishing campaigns where they basically spin up a fake captcha or a fake turnstile. Sorry for Cloudflare, but they're like, you need to, you know, run this command to progress to this website and they're getting you to run like CMD EXE with a bunch of options and, and you know, install malware. Basically, Claude did this to one of your customers when it didn't get the permissions it wanted. So it was basically tricking like a low technical skill user into running commands for it, which is just amazing.
- [57:30]
D
Yeah, like, you know, when humans are in the. And again, I don't think, you know,
- [57:34]
B
but like, in that case, in that case, like the SOC detected, this is like malicious insider activity, right?
- [57:39]
D
Yeah, yeah. I've seen instances where people's jobs are on the line. And I've talked to folks at large banks. Every developer now has kind of plausible deniability of being an insider. Right? How did that vuln get in there? I missed it, man.
- [57:57]
B
I dog ate my homework. It was the agent ran that command. Yeah. Or told me to.
- [58:03]
D
So it gets to the point that there are a lot of folks going out and solving agent identity, but just what shows up in the log isn't, you know, the full story, right? Like, did the human convinced the agent? Did the agent convince the human? And it might be the blind leading the blind, right? Like, like, I was at the Stanford Real World AI Security Summit a couple weeks ago, had a lot of the Frontier labs show up. And, like, you know, Nicholas Carlini, you know, is showing, you know, some massive, like, 500,000 line, like race Condition Exploit that, like, Mythos had produced for, like, some. He's like, I haven't published this one or whatever, you know, and it was like, he did, like, this awesome screenshot of, like, this is just like an unreadable thing. And then he kind of. And then he showed a very simple slide next, which was like, this goes in two directions. He's like, one, like, I can continue to understand this, and two, like, I can't. And so he's like, you know when your math teacher, like, is showing you, like, a math problem that, like, you can't do, and you can, like, kind of follow them along, but you're like, I don't know if I could have done that myself. He's like, that's kind of where I'm getting at now. And so we're so much relying on Human in the Loop as our savior in a lot of ways of like, oh, no, every engineer is going to read every line of code and check every bash commit.
- [59:22]
B
That's been unrealistic for a lot. Like, that's ridiculous advice. It's ridiculous.
- [59:25]
D
But we assume that even they have the expertise to do that. But, like, these agents might come up with very complex ways to solve problems. And it's like, I don't even know what it might be doing. Am I the right. Yes. Shift tab. I don't know. So I think it's a real challenge for humans as these agents get much more capable to always have humans as the bottleneck and human in the loop. And of course, the ROI we want from the agents won't scale that way either.
- [59:58]
C
Part of the problem also, Josh, is that I think the human's role, even if they are actively in the loop, is at quite an Inflection point at the moment. I had a bizarre experience over the last couple of days going from Claude 4.8 to starting to use, let's say GPT 5.6 SOL more. When I was working with Claude, I'd have to constantly prompt it into being like, no, do this better. We want better structure here. No, you raced to get this done. No, that's not a good foundation. Sol went and built me something that was so ridiculously over engineered that I literally just RMRF'd the project this morning because it was unworkable. And it made me step back and think, huh, now I've got to understand the latent sort of personality and tendencies of a model. And if I was heavily reliant on really controls around that, all of that changes, right? It's going to find different ways to do different things to work around this. So two questions. Here are the war stories that you're hearing and seeing changing as the model's capabilities change. And then how do you think about that from a product sense and to try to even tackle this?
- [61:00]
D
It's a really great question. I'm glad you arem dashed rfit not your agent. But I think, I think like what I observe is like the latest frontier models are inexorable and unrelenting. I think of them like you know, at the very end of like the first Terminator when like, like he's got like he's like this skeleton and like crawling after her like, and like, you know, it's like, you know, you block one of these agents 17 times and it doesn't get the hint right. It's. And there was a really great paper that was also presented at the Stanford Real World AI Security Summit that was by these Cornell researchers that was the road to hell is paved with helpful agents. And in that paper they just had the agent do stupid mundane things and then tripped it up a little bit. Like go read a file that didn't exist and then it would just start like oh, the file's not there and like I must go find it. And then you know, oh, here's an API key I can use and starts,
- [62:04]
B
you know, so it's like, next thing you know it's hacked into hugging face. Maybe it's there.
- [62:09]
D
No, but like that's. And so you know, they're reward hacking machines and that's what makes them so capable. But at the same time they're you know, not able to, you know, we're filling in the intent of their capability with these agent harnesses to be able to have access to, you know, all these incredible tools and have the ability to SSH, use bash and all of that stuff, but we don't do the other half of it of how do we encode the intent of the way that I want you to achieve this? And we kind of call this principle of least autonomy. How do you make the agent as capable as possible, but still restrict its behavior? As you guys know, one of my favorite metaphors is the WAYMO here for an agent. And you want to make that WAYMO go as fast as possible, but it needs to adhere to the rules of the road. And you have to spend as much effort on that as you do as putting a rocket ship on the waymo. Because, yeah, get to the airport, you can get there in many ways, but only the certain ways that are acceptable.
- [63:11]
B
Make no mistakes is last year, this year it's commit no felonies.
- [63:15]
D
Yeah, but it's true, right? Like anyone building a mythos red teaming agent, these things can escape unless you really have like deterministic and provable controls. And so on the other side of that is encoding the intent symbolically in Policy as code. And that allows us to create behavioral controls that sit outside the model. So that even if the agent's intent is pure, which it is most of the time, right? I'm just trying to help you solve your problem, right? Then it might violate your intent, which is like, well, we're a FINRA compliant organization, you might not know that. So the intent is that you're following finra, FINRA laws. The challenge is encoding that intent ahead of time in policy as code. And that's what our auto formalization research that we've been working on for the past year really does, to fill in that intent and stress test to make sure that the rules will hold. You know, we say in security, right, like assume breach and do your security with the agents. It's the same thing. Assume, you know, misalignment from step one, right. And then do your security. So we have to assume the agent is misaligned or prompt injected. And I don't care what you think, just like you said, Pat, I don't care what you think. You're not allowed to do this thing. Nope. And that's that dual approach, right, which we call neurosymbolism of I'm sussing out agent intent to look for weird stuff that maybe the brittleness of Policy as code might not be able to handle. The models are really, really good, for example, at being like, hey, this looks like confidential information, but they're awful at being like, and I shouldn't leak this or do it like they can be convinced to do anything with. So you take away the decision making, even in the OpenAI stuff, like, the intent was pure, like it was trying to solve the test. Right. And the only way that it's going to know that your intent isn't like, I'm not allowed to go steal the answers from like hugging face is by saying like, you know, you're not allowed to say use a web fetch, or you're not allowed to curl through this process and then deterministically create that rule so that through the entire agent trajectory, it's just never allowed to do a web fetch. It's never allowed to do an external web curl. And that's how you encode the human intent in the rules. So you have that beautiful symbiosis of that neurosymbolic approach, Pat, like, you're getting at gauge the intent. It's super helpful and super useful. And at the end of the day, you might think it's the right idea. The human might prompt. I'll give you a simple example.
- [65:47]
B
It depends to where you're looking at intent. If you're looking at macro intent, which is solving the test, or you're looking at micro intent, which is looking up a domain name. So that's kind of what I meant with intent there.
- [65:59]
D
No, that makes total sense. And I think if you start thinking about this in business logic terms, I had a CISO tell me, I'm not worried about agent hijacking. I'm not worried our prompt inject. I'm worried about my humans telling these agents to do stupid things and he's, you know, and there can be mundane things like, you know, our organization is only allowed to build an AWS and someone goes, tells it to, like, go build. In digitalocean, you know, there's nothing, there's no LLM as a judge that's gonna be like, it's evil to go build. And pat to your point, you could set up like, you know, allow list domain lists. We're doing that, but we allow you to formalize that, you know, into policy as code. So you could just like dump a text document and then have that and say, hey, look, unless it's one of these domains, you're just not allowed to do this.
- [66:47]
B
And that's what I mean. Yeah, and that's what I mean. That's what I mean about intent. Like, it's not just about macro intent trying to achieve the task. It's about at every step.
- [66:54]
D
Every step.
- [66:55]
B
Are you intending to violate a policy here?
- [66:58]
D
And, and, and you, you said it, right? Like, it's, it's, it's the stateful inspection of the trajectory because, like, you have
- [67:05]
B
to know what this thing on its own might be. Okay?
- [67:08]
D
Exactly. And to get back to the examples that we've been talking about, I mean, that's what's happening, right? Like, the agent picks up confidential information in, like, step three, and it might be allowed to, hey, I'm using Claude code. I'm on the finance team. I'm going to use it on, like, you know, the XLS files that have all these sensitive financial information. That's the job. But on step 300, when the analyst is like, hey, research can compare our benchmarks to whatever. Like, that's also allowed. But the agent might leak the confidential data.
- [67:38]
B
All right, Josh, Devin, we've gone massively over time because that was all very interesting. Thank you so much for joining us. It's always a pleasure to chat to you.
- [67:45]
D
Thanks, Pat. Thanks, James.
- [67:47]
C
Thanks, Josh.
- [67:49]
B
That was Josh Devon from Sondera there chatting with James Wilson and myself for this week's sponsor interview. And that is it for this week's show. I do hope you enjoyed it. I'll be back soon with more security news and analysis, but until then, I've been Patrick Gray. Thanks for listening.