Loading summary
Justin Kohler
Foreign.
Patrick Gray
And welcome to Risky Business. My name's Patrick Gray. We've got a terrific show for you this week. There's been just so much happening in AI and, you know, people complaining about distillation and people writing letters and. Yeah, all sorts of stuff going on. We're going to get into that in just a moment, and then we'll be hearing from this week's sponsor. And this week's show is brought to you by Spectre Ops, makers of Bloodhound, of course. And, you know, a while ago they released Open Graph, which, which really opened up the Bloodhound product or engine or, you know, whatever you want to call it, so that you could actually start looking beyond Windows Active Directory. They extended it out to GitHub and now they've extended Bloodhound out to Amazon Web Services, which is amazing, right, because it is such a complicated beast. And now you can actually map attack paths like from Active Directory through AWS and back if you want. And it's just really crazy. And a couple of Spectrops fellas will be joining me after this week's news segment to talk about all of that. And I recommend you stick around for that one. It is a really interesting interview. That, of course, is coming up after the news, which begins now. And joining us this week, as always, is James Wilson. Hello, James.
James Wilson
Hey, Pat. Good to see you.
Patrick Gray
And joining us also is Pete Ranks. Pete spent most of his career at the Central Intelligence Agency in the United States. He served as the deputy and the director of the center for Cyber Intelligence at CIA. But these days he is the chief strategy officer for a defense tech company called Gravity. Pete Ranks, thank you so much for joining us.
Pete Ranks
Pat. Great to be with you, James, good to meet you. Glad to be here on a week that's not boring.
Patrick Gray
Yeah, definitely. Definitely is not boring. So, look, let's start with the kerfuffle. I guess it was already an emerging kerfuffle last week, but the kerfuffle around open weight models, Chinese open weight models, what to do with them and whatnot. It really did look like the knives were coming out for open weight models, both among US Policymakers and Chinese policymakers. And this has prompted Jensen Huang from Nvidia to publish an open letter defending open weights models as being tremendously important to the AI ecosystem. You read through his letter. It's very difficult to disagree. Basically, every technology firm under the sun has signed on to that letter, with one big exception, which is Anthropic, which has responded, I think, just yesterday with its own letter, which is decidedly less enthusiastic about open weights models. And we'll get to that in a second. But James, look, let's just start with the Jensen letter. At the moment, it's quite a passionate sort of defence of like open source generally, isn't it?
James Wilson
Yeah, it is a great document. It's not often you get a document that you'd be as happy putting in the hands of a CEO and a CIO as well as highly technical folks and have it really resonate with them. It harks back to the origins of open source and how there was a fork in the road where that was the challenging stance to closed source software. And what ultimately prevailed was through transparency, we ended up with an extremely strong software foundation that powers pretty much the whole Internet as well as so many functions in large businesses and government and federal agencies. And so it's a great read, very impassioned plea reads well. And of course it has a very interesting second last paragraph where it essentially says, let's not also forget that not only are open weight models as important as open source, but the techniques that go into creating these, like distillation, have legitimate uses. Which is. It's sort of a very interesting thing that is just in the second last paragraph there of that open letter.
Patrick Gray
Well, yes, this is the most important paragraph here and I want to read it. It says, in shaping this ecosystem, policymakers should be careful not to conflate legitimate model development techniques with misappropriation. Distillation, or the practice of using one model's outputs to help train or improve another, is a widely used technique for model improvement, evaluation and validation. It reflects a long tradition of learning from, building upon and improving existing technologies, a tradition that has helped drive innovation since the rise of the Open source movement. Open Source software movement by contrast, unlawful efforts to extract value from closed models raise legitimate concerns. Those concerns should be addressed through targeted legal and commercial frameworks rather than rather than sweeping restrictions on techniques that play an important role in AI innovation. Now, this reads to me like Nvidia. And you know, some of these frontier folks were getting wind that the White House was going to like completely ban distillation as a technique or whatever. Interestingly enough too, someone pointed out on Twitter that Google offered a distillation service for Gemini where you could take a Gemini. I was like 3.1 or something or 3.2 and then distill it into a sub model for a specific task. What's really interesting is that Gemini distillation service, now you can find it on Google, you click through and it's been 404. Right. So thus is the sensitivity around anything to do with distillation. But I feel like this is the way to interpret this letter. Pete, I want to bring you in here on this. It really does feel like this letter was designed to head off a possible policy change from the US Government that would have seen just distillation regarded as an illegal act. Is that how it read to you?
Pete Ranks
Absolutely. I think it's been fascinating in the span of one week to see at the beginning of the past week, I think there was legitimate fear because of comments from a couple of different administration officials here in the US that we were heading towards a potential ban on open weight models or a ban on Chinese open weight models. And the speed with which that provoked a reaction, including this letter, which really harkens back to the mythology about the beginning of open source and taps into all these deep seated fears in the software community, has really been fascinating. And so I think both the idea that we would ban models and then even the idea of distillation as the boogeyman Anthropic coined the term industrial scale distillation efforts and really obscured the fact that distillation was a much more commonly used technique throughout the software industry. And really quick and I would say massive response from the community to push back on that idea to the extent where I think there's. The administration will have heard enough dissenting voices here that it's going to give them a pause to consider whatever action they take next.
Patrick Gray
Yeah, I mean it really does feel like the policy in the policy in the United States just generally is, you know, pretty unpredictable at the moment. I mean, it's hard to imagine that just a few years ago you had Ben Buchanan, who wrote the book as a policy advisor in the White House under the Biden Admin, you know, a fairly sensible executive order there that was rescinded by Trump and like, and just here we are, right? It's pretty crazy stuff. Now you mentioned that Dario, chief executive of Anthropic, released his own letter. Right. So this is interesting. Instead of signing on to that letter, released his own letter. And in this letter a couple of good points made, right? One of them I really agree with, which is that hardware is very important to this whole thing. Like if America wants to ensure dominance in the AI race, it's going to be more about the hardware than the models. But then he just can't help himself, right, because he has to write about how open weight models are very dangerous because they don't have safeguards and you need to Trust us. And what about bioweapons and cyber attacks and whatnot? I mean we've already seen the extent to which guardrails on on frontier AI models just make them completely useless for things like cyber defense. We've even got a, I've linked through to a tweet here this week where someone was trying to, you know, remediate a Linux bug. It wasn't even an exploitable bug and the frontier models wouldn't help. Right. So they had to turn to GLM 5.2 or whatever to actually get that job done. So this seems like just Daario being Daario can't sign on to everybody else's letter because, you know, he's worried about safety and only anthropic can be, can be trusted to be the safe one. I mean that's kind of like, you know, that's my take here, James. I'm guessing you agree with that.
James Wilson
Yeah, I do agree with that. And yeah, I'll give you the, I'll agree with you on the chip making restrictions I think is the most thing that we all sort of agree or think is the most reasonable point out of it. But then he goes on to say he wants a crackdown on industrial scale distillation and I think that's just. Ship has sailed, buddy. I don't think there's much you can really do there and mandatory safety testing. But then yeah, it deviates into, you know, his quote here is I don't agree with the letter's assertions that open weight models necessarily make it easier for easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. Which I find incredibly frustrating because anthropic is the element that is amplifying the asymmetry between attackers and defenders at the moment in the favor of attackers because their models are so utterly useless for cyber security defense work. Unless you've managed to get into their blessed programs that have an opaque approval process. You don't know what the bar is. It's just you. I mean the old phrase you can't have a cake and Eater 2 just doesn't seem sufficient here. It is just wrong headed, frustrating thinking, frustrated thinking from the very people that are actually doing the largest amount of detriment, I think to cyber security defenders at the moment by having these overly guard railed models and no other options available.
Patrick Gray
Yeah, and look, Halvar Flake has an interesting take here where he said, look, if you consider the concentration risk versus the proliferation risk when it comes to these models, he considers the concentration Risk to be higher than the proliferation risk. Right. The big hacking campaigns will go away pretty quickly. The deepfakes are an enforcement question. This is Halvar's take. You know, he just thinks that open is probably better. Pete, where do you fall in, you know, fall on all of this? I mean, it's a big topic, a lot of moving parts. You know, what, what should happen in your mind when it comes to policing this or regulating it or like what, where do you even begin?
Pete Ranks
Yeah, I'm not one, I do. I, I agree. I think hardware controls are one of the stronger levers that, that the west has. And I think we now have proof that trying to control the software, trying to manage that software advantage is going to be, is going to be a losing proposition. I would not be opposed to some way to try and separate out really large scale distillation campaigns. I just don't know how it would be done compared to what looks like legitimate distillation.
Patrick Gray
Well, I mean, that's the thing, right? What's that, where's that, where's that line? Right? Where is the, you know, legitimate distillation versus wholesale industrial grade distillation? Like what's the, what's the actual, you know, how do you walk that line? And if you regulate it, people are going to walk right up to that line without kind of going over it, you know.
Pete Ranks
Yeah. I think the other thing that I found was a weird. There's a little bit of juxtaposition of distillation as it's used for good and distillation as it's done to us for evil, but anthropic. And everyone else's entire model is really built on reusing other people's work. And so I found the juxtaposition of the $1.5 billion settlement this week for scraping other people's work without their consent, really from a library of pirated books to just be an interesting data point to throw into the middle of this argument that it's unfair for other people to be abusing the investments that we've made. It's a fair point. And I'm not anti guardrail. I do think there is room for gradual introduction of new capabilities. The guardrails obviously have flaws, but I also think they have purposes. And I think as new models come online, it makes sense to have a gentle entry into the ecosystem and do that work. So I wouldn't say that everything is open weights without models, but the argument just doesn't hold up over time. And for Halvar's quote, I'd like to believe the hacking campaigns go away. I don't know that that's true, but I think the point about concentration of power is well made.
James Wilson
Well, there's a good parallel here again coming back to how Jensen harked back to the early days of open source and the parallels now with open weight models. And there's another parallel here that's not being called out, but just comes to mind for me, which is the way that Dario and Anthropic are behaving is very much like the Richard Stallman gpl. You know, it's open, but only if you do it my way. And anything that is a byproduct of this open but only my way has to be done my way. That sort of viral licensing that was a part of the early software, open source software movement feels very much like what Anthropic trying to do. It's like, I'm okay with open, but only if you do open exactly the way I want to do it and only we can do open safely. And you know, that that was something that didn't prove out well, like history shows that although GPL as a license was very important for some of the early components, it's largely seen as a pest today because it's so restrictive. And so if we end up with even a flavor of open that is open but restricted or worse with some viral restriction that carries along with the artifacts, we all, we know that's the worst of the outcomes as well. You know, open needs to be truly open for us to get the full of benefits that are on the table from this.
Patrick Gray
Yeah, I mean it's. Look, I think what we can expect though, just to sum up and Pete, I want to get your opinion on this. I think what we can expect to see in an ideal world I guess is going to be tighter controls on hardware because already we've got the White House saying, well, look, they've been distilling wholesale, you know, Kimi is wholesale distilled from American models on, you know, hardware based in Thailand that they've rented for training runs and whatever. So probably we're going to see more, you know, entities list designations outlawing doing business with some of these companies so that they can't rent some of those chips. Because you know, the people who are operating those Neo clouds might be concerned that they're going to get cut off from being able to purchase that hardware as a knock on effect of doing business with companies on the entities list. You know, I mean, I think so we might see that approach to try to target the Chinese AI industry and yeah, just generally tighter controls on hardware because, you know, there's some Huawei chips but not many and they were all fabbed by TSMC anyway. Like China's very much behind when it comes to the hardware stuff and I think that's going to cover off most of the things that should concern a White House administration. I mean, I think that's probably where this should go. But what are your thoughts there?
Pete Ranks
Yeah, I think the lengths that the Chinese companies and the Chinese government go to to get access through alternative means to Nvidia hardware demonstrates that they still have a fair amount of dependency on there, that the Huawei kind of ascend line of chips just isn't reaching parity at scale in a way that allows them to keep pace. And I just, frankly, it's easier to control hardware across borders than it is to control knowledge across borders. And that's the approach that I think is the strongest hand for the US to play. There's obviously differing opinions on doing that. We've seen different approaches between the last two presidential administrations here. But there is a strong desire to do something and I think they may, as they evaluate options, come back to hardware restrictions as being a stronger hand they can play. You may see some entities list designations. It's, I think yet to be determined whether large scale violations of terms of service really constitute IP theft or something you can put on the sanctions list or that you feel comfortable putting onto the entities list.
Patrick Gray
I mean, they'll probably just say, oh, they're doing work for the pla. Like that's probably going to be the easiest way is to trace some sort of link between an AI lab to some sort of Chinese military or intelligence apparatus bit that you don't like and then you can slap them on the entities list. If I had to guess what the justification will be, it'll be something like that because it's going to be easier than saying, well, there was distillation that crossed a threshold into IP theft. And you know what's even your, your legal theory there for making that complaint. So that's kind of how I would expect that to shake out. One of the great ironies of all of this though, is that Kimik 3, which has kicked off a lot of this, it's really been GLM 5.2 and Kimik 3 that have kicked it off. I mean, Kimi's finally gone open weights, right. It's massive and you kind of need your own Neo cloud to run it anyway. So I think it's, you know, Last time I checked, it had been downloaded like under 3,000 times. And, like, no one really has the hardware to run it. It's a total inefficient pig. Like, it just. I think the main advantage of it for the people who are using it is the lack of guardrails. I mean, that would be basically it, right, James?
James Wilson
Yeah, pretty much. I mean, like, to give you an idea of the scale, the model is 2.8 trillion parameters, which, at a bare minimum, with a floating point Quantization down to 4 bits still requires 8 GB 300. And, you know, very few people have access to that or the budget to afford access to that. Certainly not the number of people that are downloading. Yeah, look, it is a very inefficient model. It is a very token hungry model. And the funny thing is, as much as there's all these claims that it was distilled, I think actually probably one of the most orthogonal or anecdotal reasons that you could say it's been trained on Fable 5 is because it actually scores very badly in cybersecurity, but very good in actual coding tasks.
Patrick Gray
So, yeah, that tracks. Now, look, we've got some updated reporting here on the OpenAI Hugging Face thing. You know, headline here from Reuters saying it's AI agent spent days hacking a company, but sources say OpenAI did not notice for a week. I mean, we knew that, right, When Hugging Face is already in the midst of incident response and blogging about it. Right. And OpenAI hasn't said anything. Well, we know that they, they didn't know. We should point out too, we were joking last week that, that the model probably exploited buildkite where you used to be CTO, because OpenAI is known to be a user of various bits of buildkite technology. And we were joking that it was your fault. Turns out it wasn't. The oday that the model found it exploded was not in Buildkite, it was in JFrog. And it's fair to say that your former colleagues at buildkite were not really amused. They did not find our joke as funny as we did last week. So, yes, there was no ODE found in theirs in their software. It was JFrog and they are patching that now. But, you know, we're just seeing, you know, more write ups on how this all happened. I for one, actually find this whole story hilarious. Like there's people saying, oh, it's deeply concerning and whatever, it's hilarious. I mean, people are saying, people are asking me questions like, well, was this a CFAA violation. Like, no, like robots don't commit felonies. Like, that's not how this works. This is like a waymo, you know, driving into a closed shop or something like that. Like that's the sort of legal, you know, from a legal perspective, I expect it's much more like that where Hugging Face could sue if they wanted to. Instead it looks like they're raking them over the coals and like suggesting that they provide 100 million bucks worth of compute to Hugging Face and its community. Right. To as part of this partnership deal. Because really they can extract a lot out of this given what happened. But I think the most hilarious thing is here I've had people asking me, well, why wasn't it air gapped? Air gap's kind of an old concept, especially when it comes to software tooling like this. Like, I doubt this was running in an on prem anything. Right. It's going to be running in some cloud somewhere and you, you can't really do these sort of tests in an air gapped environment. So I think the controls here were probably reasonable. But the thing that I find really, really funny is that the monitoring just wasn't like clearly if this thing's going off and hacking Hugging Face for a week and no one's noticed, like they're not doing any netflow analysis, they're not really doing any meaningful monitoring of the models. Pete, you had some thoughts on this?
Pete Ranks
Yeah, I mean, and I. So it's old fashioned. I'm probably old. I disagree. When you're doing something at this scale that you wouldn't go to pretty extreme measures for isolation. And to me that's the surprising part about the story. I'm not surprised that the model tried to cheat on the test. We've had lots of examples where the models have found creative solutions to try and achieve the objectives that they're given. I'm not surprised that the model was capable of hacking Hugging Face. I talked to a bunch of the companies building agentic pen testing tools and their challenges are never getting the models to hack. It's constraining them in a way that would allow customers to be comfortable. Right. Just having them stop at a certain point. So not surprised at all that it could achieve that. I was surprised that it was able to escape the sandbox that it was in. And presumably OpenAI would have believed this new model was going to score very well on the exploit gem. And so they've taken a system designed to be the best in the history of the world at exploiting software and put it in a cage made of software, didn't pay attention to it for three or four days and were surprised that it escaped. It's a fireplace made out of wood. And so I wasn't. I did find that implementation of controls and lack of attention to be irresponsible. Now I do think, you know, it's on the scale of irresponsible things that happen in companies. It's not shocking, but it doesn't contribute to the argument that only the closed source model providers can be trusted with safety and security. Because when you're building a tool this powerful, extreme measures for containment until you know what it can do. I think makes sense. Fifteen years ago when we exploited adversarial malware, you know, we had more isolation in systems than we have today because we thought those systems were going to be unpredictable when they were running in execution. So I would have expected a more robust set of controls. And I think I've seen a bunch of criticism of that in the past couple of weeks, which I think is fair.
Patrick Gray
I mean, I agree it's fair, but to me the part that's most egregious is less on the control side and just more on the monitoring side. As in they didn't have any at all. And that's the part that frankly to me is very, very funny because I just wonder what it was like for the people who were doing these tests when they realized what had happened. And every time I think about that, I laugh inside quite a lot. James, if you were designing these tests, what would you have done differently?
James Wilson
Yeah, I was trying to think through that as you guys were talking. And I think the challenge that I would run into is that it's not so much that air gap is an old fashioned concept, it's that I think it's now virtually impossible to actually properly engineer a truly air gapped solution for the reasons being. So let's assume that we didn't have this JFrog.
Patrick Gray
That's why I'm saying they're old fashioned. Right.
James Wilson
But anyway, right, it's not that the concept's old fashioned, it's that the ability to create one is now increasingly hard because of the hardware and software stack we're using. So let's assume we didn't have the JFROG vulnerability. Well, were these things running in a data center? Okay, well they're virtualized. Was there escape, a way to escape the virtualized process and get to the host machine? Could it have got out that way? I mean, even if you think about this as they literally had these things in racks in a basement, zero connectivity to the outside world. You've still got other avenues, right? It's like, how was it connecting to network hardware? Was that network hardware being monitored? Even just the fact that you're saying you needed to have monitoring opens up a vector of a monitoring station that might have wireless radios in it. Right. The point here is these things were told to find a way. They were given seemingly unlimited compute to go and find a way. And we shouldn't be surprised that they found a way. That surprise is no one was watching. And the real question for me is, what the heck were they doing if they weren't watching this?
Patrick Gray
Well, yeah, I don't know. And that's why it's, as I said, as I've said a few times now, very, very funny to me. So that's been quite enjoyable. Okay, so moving on, kind of, because it's still AI stuff, we've got a bipartisan bill being floated in the United States which is looking to give sort of like safe harbor provisions to the AI companies so that they can collaborate on combating distillation. This seems like just a sensible idea. You know, the bill says, well, you're not allowed to collude, you know, on your businesses and whatever, but, you know, you are allowed to talk to each other and sort of cooperate on this. Pete, I'm sure you think this is a good idea.
Pete Ranks
Yeah, I think this was. And I went and tried to read the text of it. Some of the marketing around the bill made it sound like it was aggressively going after Chinese models. What it's really doing is providing safe harbor for, for Frontier Labs to be able to collaborate on things like distillation attacks or anything else that doesn't rise to the level of a cyber incident, but is the kind of thing that reasonable people would already expect. The labs were allowed to collaborate on. So to me, this is kind of unadulterated. Good idea.
Patrick Gray
Yeah. And speaking of some proposals, some legislative proposals that are probably not as great ideas, we've also seen an idea from floated by the Trump Admin that there should be an AI kill switch which would enable the White House, I believe, through the DHS to just kill off AI that it thinks is dangerous or is going rogue or whatever. This seems like a knee jerk response to the OpenAI hugging face thing. Also seems like an absolutely terrible idea. Marco Rubio apparently is out there telling diplomats, hey, maybe we want to talk down the idea of an AI kill switch. When, you know, obviously technology buyers in Europe are already feeling a bit jittery about dependence on the United States. You know, I mean, this seems like just an absolutely insane idea to give the United States government such, you know, the ability to just arbitrarily kill off models. Pete, thoughts?
Pete Ranks
Yeah. I mean, it's technically impractical. It's. It's not going to work in half of the cases where you would hope it would work. And the idea, the terminology kill switch is just a poison pill for selling things overseas. So I don't think this goes anywhere. I do think it speaks to a very real desire for Congress in the US to do something and that there is bipartisan demand for some sort of safety and security controls. And so we're going to see ideas. This idea, in my view, is just not a good one.
Patrick Gray
No, I do think it's funny. We need to do another Marco Rubio meme. Don't. Don't know what. What clothes you should be wearing in that one for this, you know, talking down the kill switch. Meanwhile, we have seen United States governments, government agencies broadening their alert on Iranian hacks targeting ot, you know, targeting a bit of water, targeting. Targeting a bit of energy. When I speak to people like Rob Joyce, they seem much more concerned by cyber attacks targeting the energy sector than they do the water sector. But we actually have seen a coordinated cyber attack targeting or successfully targeting water utilities in something like 30 towns in Minnesota. Not sure what the impact is just yet, but, you know, these are obviously headlines that will be quite alarming to Americans, Pete. And, you know, I understand that you've always been in favor of the United States government maybe being a little bit more robust in its response to these sorts of attacks. I think that's difficult in this at the moment, considering that the United States is already in a. In a hot war against Iran. But walk us through your thinking there.
Pete Ranks
Yeah, I mean, I think you're right. There's nowhere else to escalate right now with. With Iran. But historically, I really advocated that any attack on civilian critical infrastructure deserves a really robust response. And we tend to sweep them under the rug because they're attacks on disconnected water utilities. So something like 17,000 local water authorities in the United States, individual oil wells. So they don't have a ton of impact. And the methods are often really immature. It's finding default passwords on edge devices or control systems connected directly to the Internet. But for me, this was always about the symbolism of crossing. There is still some norms that exist, particularly between the US And China in cyberspace, and I think Every attack by any actor, criminal or State, on U.S. civilian critical infrastructure is an opportunity to message China, to convince them that they should not think about operationalizing their Volt Typhoon campaign. And anytime we don't respond forcefully, we run the risk of sending the message that we would tolerate at least some sort of entry level disruption of US critical infrastructure. It is very difficult. And again here we are in Iran case where there is really no more robust response than already being at war with them. And it's difficult because you don't want to scare the public about these things by playing them up into more than they are. You don't want people thinking the water is poisoned because somebody was messing around with the console in a utility. But trying to find that line of what is a response that would convince people that this is a norm the United States takes very seriously in a world full of other types of hacking has always been a position that I advocated.
Patrick Gray
Yeah, I mean it's interesting to me that you said that the warning to China would be really around them not operationalizing the Vault Typhoon stuff. Because in my mind like the Vault Typhoon pre positioning is already like pretty far beyond the norms. That seems like just crazy to me. And it also strikes me that it is something very difficult to respond to, especially when the Chinese play the game of just denying that they do it right. So they just like what are you talking about? The NSA is the biggest hacker in the world. That's like their pre canned response. So I mean, what can you even do there, right? You're out now, you no longer work for the government. So you can speak your mind. Pete.
Pete Ranks
I will pull my punches probably on how much I speak my mind on some of these things. But the ideally with Volt Typhoon you evict the Chinese, I would say not possible at scale over a long duration. Or you convince the Chinese it's not in their interest to pre position on US critical infrastructure. I would say very difficult to do in their formulation of what total war with the United States would, would look like. And so what you're left with is at least trying to influence their doctrine to say that anything short of full scale conflict, you don't mess with critical infrastructure.
Patrick Gray
So the pre positioning horse has bolted, it's out of the barn. So now you know, probably the most realistic thing you can do is say, well, no pulling the trigger on actual damage.
Pete Ranks
I think even when we've tried to come up with diplomatic statements around pre positioning on infrastructure, we've had a hard time really trying to figure out how to formulate it. Because pre positioning often looks like espionage. Right. Sitting inside of a telco is pre positioning potentially. It's also espionage potentially. And so you have things that kind of live within norms of espionage and you have things that we think violate those norms. Pre positioning is a, it's a hard one to. It's a hard one to identify. And pre positioning is essentially the threat of violence in cyberspace and escalating in response to pre positioning is responding to the threat of violence with violence. And that's a hard thing for governments to do as a matter of policy. Which means we generally complain loudly about the pre positioning and try and get as many countries to sign on to the joint communique about pre positioning. But we haven't really responded very forcefully other than sanctioning individual companies that we
Patrick Gray
can identify as being involved and individuals. Yeah, but I mean, I guess that's the point, right? We're back to sternly worded letters. Right. And for now at least look, moving on from China stuff I guess, well Iran and China stuff now to what the Russians are up to at the moment. They are popping shells on a bunch of Zimbra collaboration suite users which is, I just, I guess it's par for the course.
Pete Ranks
Right.
Patrick Gray
But there is, there are alerts now from NSA about the Russian state actually attacking Zimbra. We've also got this story here about a Russian apt crew hitting WI FI devices at. In hotels, right. To try to redirect people to steal M365 accounts. I'm not sure how that's playing with like TLS and whatnot. And I'm not sure exactly what sort of convoluted phishing attacks they are doing with this. I think you've got. I think they're doing WPAD as well. Wow, that's crazy. They're doing wpad. That's Windows proxy auto discovery WPAD files. You can actually, once you're on a network you can sort of serve them to Microsoft browsers and then proxy all of their connections and whatnot. Very good for you know, person in the middle attacks. But I don't know, I'm still guessing. There's some, some pretty convoluted stuff happening here to get around certificate warnings. James, have you looked at this one in much detail?
James Wilson
Yeah, I have. It's a good write up and you're right, it's two different styles of attacks that really are quest to publish this observed. One is they say that the, these WI FI hotspots just have their. Seems like the DNS configuration change, which I assume means that they're like DHCPing a malicious DNS server and that's returning these attacker control domains when they try to go to legitimate Microsoft 365 domains. But they said they also saw evidence that they were serving PAC files, as you say, for that exact reason. They do say in the article they weren't sure how successful that was. But yeah, I mean this all comes down to the question of have we trained users to just blast through the warnings and they just, you know, don't pay attention to the fact that they're getting cert warnings and this is still successful or is this a bit of a sort of a desperate campaign that, you know, might be netting a relatively small amount of credentials? It's just not clear. The other thing was the, the initial access method for the WI fi appliances was not clear either. And I.
Patrick Gray
Well, that's because it's going to be some like Telnet or SSH Admin, admin, off you go. Like it's going to be really boring. Yeah. You know, and it's from some company that installed them there, you know, 20 years ago or something. And just. Yeah, it's just, it's not gonna, it's not gonna be good. Right. Like it's not gonna be interesting. But I guess this, this to me feels like, yeah, someone in the arena trying things, as they say. Right. Which is like, let's see how many we get this way. We've seen the Russians trying to accumulate as many M365 accounts as possible lately. For whatever reason. I think they've, you know, there's some that are going to be interesting, have interesting information in them, but they seem to like collecting these accounts for whatever reason as well.
James Wilson
Yeah. We're not sure why, but it's exactly like the previous attacks we saw where it's like, sure, this is going to net you some credentials. We don't know why, but it also requires the user to ignore a lot of very loud warnings. So yeah, I agree, we've got this ability. Let's try it. Let's see what happens. I don't think they're going to be particularly successful.
Patrick Gray
Now we've got this few stories here from South Korea. We've got to start speeding up the pace a little if we're going to make time on this week's episode. But we've got LG monitors installing like McAfee pop up ads as part of their driver, like bundle. This is just like sad for LG, sad for McAfee, which is now Kind of behaving almost like, you know, those shifty like pretend fake antivirus things like now just popping up on people's computers. Microsoft has taken action against that, but it's just like that is embarrassing. We've also got this issue issue with LG with their TVs and you know, we've seen this come up before, we've seen the research come up. I think a few weeks ago Catalan covered it in the bulletin. But now LG is banning residential proxies from smart TVs because something like 42% of the apps available in like the LG App Store have some sort of proxy function in them, which is just insane. And this is how a lot of these residential proxy networks are managing to grow. You know Pete, I imagine you would have spent a bit of time at CCI dealing with these proxy networks. Right. And they're a thorny problem. Right. And it just must be so frustrating when you see like a major company like LG kind of enabling the growth in these things.
Pete Ranks
I was floored by the 42% number. I mean I obvious that you could use a TV and the processor and the TV as a res proxy. Of course the idea that, that many of those apps are either just people acknowledging the terms of use and opting into essentially a res proxy network or that there's malicious code installed there. I was shocked by that. It's time to ban this whole computing platform as a TV where you, I mean it's nice to say you're going to take out anything that includes the res proxy function. I don't know why we need apps on TVs to begin with.
Patrick Gray
Well, I mean Netflix for example is an app on the tv, right.
Pete Ranks
So but the built in television are the last ones to get updated. They're the most feature poor. They are the, you know, they're the least secure versions of that that you can get from anywhere else. I mean that's, everybody should go get a, you know, some sort of add on box that gets updated more frequently. So yeah, I think it was frustrating. And so you, you know, the demand for res proxies is only going to go up as people want to do tons of AI distillation. And so you're going to have, you're going to have a lot of demand for this set of services. And the fact that you know, probably all of our TVs have some apps installed on them that are joining us into a botnet makes me want to go check my firewall settings.
Patrick Gray
Yeah, I mean it's funny that you mentioned that. Because I was literally thinking that at that time I need to do a bit of an audit of the. Because I do have an LG tv so I'm going to have to do a little bit of, a little bit of a check there. And you know, as I mentioned earlier, there's been some research out of Lumen, their Black Lotus Labs, looking at how these botnets are continuing to grow despite takedowns. Some interesting stuff in the research too about how they rent from each other to temporarily make their botnets appear bigger. And you know, it's a whole industry now used by all sorts of rogues, from crooks to apts to whatever. I think, you know, if anything this is a sign of the success of companies like Gray Noise in being able to flag bad IPs. The fact that you need a giant residential proxy network these days, I mean it's interesting for me because I'm on the board of a company called Knock Knock, right? And we have a Gray noise integration. But Knock Knock, does network allow listing? So where residential proxy networks are very good at getting you around using Gray Noise as a block list, it's less effective at getting you around using residential proxy networks to bypass an allow list because really like you as, as an attacker can only get added to the allow list if you're behind the same IP or gateway as the legitimate user. And that's not really like, you know, this is, this is a tv, right? Like a TV joining a residential proxy network is not going to be allow listed because a legitimate user is unlikely to be on the same network as that tv, right? So this is stuff that I found very interesting and thinking about like, like you know how network Allow listing is for a lot of applications like remote access for example, just going to be a much better approach in the future to block listing. But you still need the Gray Noise data, right? You still need to spot your bad gateways and bad shared IPs and things like that. So yeah, just an interesting area where I've spent a bit of time lately. So we've also got a story here about this group called Upbound Group. This is a fintech company that handles leases for equipment for businesses. Someone managed to steal a bunch of data and leverage that into $13 million in fraudulent leases. But this involved like handling physical goods, warehousing, dispatching, like really complicated operation. Just goes to show you if there is an angle that'll let you monetize stolen data, people are going to figure it out. What else have we got? Now we got A fake. We've got a malicious Claude artifact hosted on a legitimate Claude domain which has tricked a bunch of people into essentially installing malware. James, you looked at this one. Walk us through it.
James Wilson
Yeah. So Claude has this, and I think it's relatively new functionality where it will go and put artifacts on its public website for you to look at. And I ran into this by complete surprise this week, actually. I was prompting it back and forth and said, I want you to now put together a design proposal for me. And then a minute or so later, a web browser opened. I was like, what the hell just happened? I didn't. Wasn't on a web browser there, and it had launched to Claude's Artifact Store on Claude AI artifacts and it was presenting its design document to me in a web browser. And that freaked me out. I was like, I didn't ask you to put this on the public Internet, but it went and did it on its own. Now, that artifact store has two problems. One, it's being used now to host malware in this campaign. The malware itself looks very detailed. The breakdown of it goes through how it used many stages of loaders, you know, VM protect, packing, shader timing checks, gpu, VRAM checks, all these very detailed things. So it seems like it's incredibly sophisticated malware, but being deployed in this novel way where because it sits on the claw AI domain, it looks very legitimate. The other problem with this Claw AI domain and these artifacts is they were snapped up by Google this week and we're getting indexed in search results. So just a good example of where like anthropics not putting enough thought into the downstream impacts of this functionality and releasing it before it's. Well, before it's really ready.
Patrick Gray
Yeah, Fun stuff, right? So that's. Yes, definitely fun stuff. We've got a story here. We are going to have to speed up now and just direct people to have a look at these stories in our show notes, but we've got a story here which is really interesting about Apple being sued because a user downloaded a fake crypto wallet app from the Apple Store and they lost like $1.8 million. Right? And you would think, okay, boohoo, like things like this are going to happen. You should be more careful. But then you actually read the complaints or you read about the complaint, you realize that Apple had been warned about this a bunch of times and like, it just starts looking like maybe they weren't doing their job properly. I've always had high expectations for how Apple should curate that store, given that they take 30% of revenue right from these app makers. Like the amount of money that flows through that store is just absolutely astronomical. And you would think even if they took 1, 2% of that revenue and applied it to solving these sorts of problems, it would be a solved problem. So I think, let's see, I mean James, you and I discussed this and I think it's really going to come down to like how enforceable is that end user license agreement? And that's, that's going to be what this trial is about. One to keep an eye on. Let's see if they settle it. Probably not because Apple going to Apple Klopp is actually doing the Klopp crew who do data theft and extortion, they're doing another run. I mean the way these guys tend to work is they will find an ODE in a technology, use it to automatically harvest a bunch of data and then ransom it back to these organizations. They are targeting these product lifecycle management services called windchill and flexplm. This is not stuff I know about. Just very quickly, James, you had a look at this and you're like, oh my God, this stuff. You really don't want this data getting out there. You can think of jobs you've had that has used this software where that stuff is very sensitive.
James Wilson
Yes, any retailer at the heart of it, whether it's bricks and mortar or online, has one of these PLMs or product information management systems and they are the source of all the pricing data, the inventory data, the history of pricing, pricing, supplier arrangements and commercial contracts, et cetera. And it's just, it's, it's such a trove of data that when I first looked at this story I was like, well that's boring as, that's as boring as file transfer appliances. But then when I realized what the PLMs were, I was like, oh yeah, no, I, yeah, we might have paid a ransom in some of my previous jobs if this had happened to us.
Patrick Gray
Yeah, that's the crown jewels info basically. We've also got some research here looking at how so called wrench attacks against crypto holders are on the rise. No surprises there. That's when you know, why crack their crypto when you can beat them with a wrench until they give you the passphrase. And you know, we got instances that they cite here about you know, people's family members being abducted and things like that. And finally a late breaking story from Wired that says that the OpenAI agent that breached hugging face may have gone after a bunch of third party services as well. So that is not, not so surprising. I've had a bunch of people ask me did it go after other services? And I haven't known. And it looks like there's some reporting emerging on that now. But gentlemen, we're going to have to wrap it up there because we are out of time. Pete Ranks, James Wilson, thank you so much for joining me.
Pete Ranks
Thanks, James. Thanks, Pat.
James Wilson
Thanks, Pat. Thanks, Pete. My goodness, what a week.
Patrick Gray
That was. Pete Ranks and James Wilson there with the check of the week's security news. Big thanks to them for that. This week's show is brought to you by Spectrops and they make of course the, the wonderful Bloodhound product which started off as a way to enumerate attack paths through Active Directory. Now, of course, you know, Active Directory, there's more to life than Active Directory, especially now. Right. So they gradually broadened out the graphing capabilities of Bloodhound. They released a thing called Open Graph which enabled people to just, you know, use this sort of graphing engine to enumerate attack paths through AD and other things. They themselves then extended the graph from Active Directory and entra into like GitHub for example. Found a bunch of interesting stuff there and now they've extended the graph out into aws. And this is just like if you're an AWS shop, even if you don't want to enumerate attack paths beyond like aws, this is still going to be immensely useful to you because there's all sorts of unexpected attack paths crop up when you start crunching AWS account information and roles information. Right. So, so that's what we're talking about today. We're talking about Bloodhound now being extended into aws. And joining me to talk about that is Justin KOHLER and, and Mr. Jared Atkinson, who's the first person you're going to hear from talking all about Spectrops adventures in aws. Enjoy.
Jared Atkinson
So Daniel Hindson's are kind of one of the researchers that's behind this. Daniel Heinson and Julian Cantrimbone, who got to give them both credit, I guess the. But Daniel started a project called project apeman probably 4 years ago that was trying to understand kind of like assume role policies across aws. And, and this was before obviously Open Graph and all these things that we were adding to Bloodhound and we finally kind of were able to get that to all coalesce, especially with Julian's help, into something that is kind of covering down obviously not every single AWS service that's available to you, but a lot of the main ones like AWS, IAM, EC2, S3, lambda functions, so cloud formation templates, so on and so forth. And the, the general idea here was, I mean, just what you explained, Pat, which is this idea that we had AD. We looked into intra, we started looking into. Into GitHub as well. And then obviously the next big behemoth that we wanted to tackle was AWS and understand not just how do attackers get in, but like, what can they do from once they're in. So how do these policies kind of overlap and allow people to maneuver their way through AWS accounts and overarching across AWS accounts as well?
Patrick Gray
Well, you've had Entra for a while too, right? Like that was even before GitHub. GitHub was a really interesting one because you found all sorts of horrible ways that people could. All sorts of horrible interactions between directories and GitHub accounts and roles and stuff that would allow people to do horrible things. I mean, it was actually quite surprising the extent to which GitHub is a problem there. But AWS, I think, is a more like, it's a more obvious one, frankly, than, than GitHub. Even though the GitHub stuff is awful. I'm kind of surprised you didn't do the AWS stuff first, but it's also a very complicated beast.
Pete Ranks
Right.
Patrick Gray
So I imagine there was a lot more work went into creating, went into extending the graph out to AWS as opposed to GitHub. I mean, is that sort of why GitHub was first?
Jared Atkinson
Yeah, we, with, with Open Graph, we wanted to kind of start with something that we thought was simple and comprehensible. It turns out that GitHub was not what. What we thought it was when we started out. And then it also kind of coincided with a lot of attacks that were targeting GitHub and so kind of.
Patrick Gray
Right, right. So you went for GitHub first thinking it would be simpler and it wasn't.
Jared Atkinson
Yeah, we thought, we thought Basically everybody at SpectreOps understood how GitHub works. And so that's a good starting point because, you know, developers know how repositories work and things like that.
Patrick Gray
AWS Turns out nobody knows how GitHub works.
Jared Atkinson
Oh, 100%. Yeah, they definitely don't know the full impact of different things that you're setting up, especially with GitHub Actions and Things like that. But yeah, AWS was definitely a behemoth that, you know, we didn't know how to like, kind of tackle that. I guess it was just such a big, big undertaking. And one of the big problems that I kind of touched on was this problem with all these different services. So you start off with the IAM policies, right? Like what, what are users, what are roles, what do they have access to, generally speaking. But then you have all the different services that you have to really dig into and they're all kind of like a unique beast in and of themselves. So like, how do you interact with S3? How do you model out kind of like the hierarchy of an S3 bucket, right, because there's files and directories and all kinds of different levels and that can just. Those are gigantic resources. So how do you, how do you model those into nodes and edges? Things like lambda functions and their ability to assume or run as roles and what they have access to downstream and the different types of access that you might have. Right. So can you read a lambda function or can you write to a lambda function or can you, you know, there's all kinds of different, different permissions that you can have and what can an attacker actually leverage those permissions to do? Downstream is kind of the big question that we were trying to.
Patrick Gray
Well, I imagine there's just so much stuff there that you've got to figure out how to plug that into a graph, right? And like, what type of, you know, object is that on the graph and how does it interact between this and that? And like that's just a lot of, that's just a lot of, a lot of work.
Jared Atkinson
I think we ended up with 150 edges or something in that kind of neighborhood. So there's, there's a lot of different ways that you can kind of express what's going on there for sure.
Patrick Gray
Yeah, yeah. But end result is, I mean, you know, we were talking before we got recording and I'm thinking what you've wound up with is something similar in complexity to your og, like Windows Directory product because. Aw. In fact it would probably be even more complicated, right, because of the reasons you just explained.
Jared Atkinson
That's right. I think, I think there's also kind of this very expressive language for how you define these policies of who has access to what, essentially. And there's several different layers that you can apply those policies, right? So you could have like a resource policy, you could have an inline policy, you could have a, like a policy that's defined up at the organization root that then applies down to everything. And then you have to kind of like compare those against each other and understand kind of like what is the actual outcome of all these different policies. And that's like one of the things that the researchers who built this were really proud of is their ability to kind of express that. And I think AD is a little bit more straightforward just because it's. It's like a discretionary access control.
Patrick Gray
No one's, no one's ever said those words in the history of humanity. AD is like, better or more simple. You know, that's amazing. But let me ask you too, like, now that you've got this aws, you know, function or, you know, this AWS feature, are there people using it to enumerate attack paths that exist end to end in their AWS environment? Or is the value more in being able to extend the graph out between different things like Entra and AD and aws? Like, are there people who are just using it now for their AWS in like, just solely looking at aws?
Jared Atkinson
I think Justin's going to share some examples of this, but definitely there are perspectives that allow us to look at, just from an AWS perspective of if somebody has this role and maybe they gain that role through externally exposed service or something like that. Right. Like, we don't necessarily focus on how do they get in in the first place, but once you're in, what can you do with that? We definitely have examples of how only staying in the AWS graph exposes lots of kind of production resources. But we in general, I would say kind of the philosophy behind Bloodhound and Spectreops in general is this idea of how does everything compose together and how as you, as your network environment kind of complexifies with all these different platforms that you're adding and you're connecting through Federation and all that kind of stuff. How do those things create these kind of unintended consequences throughout the entire enterprise?
Patrick Gray
I mean, I guess the short answer there is both. It's both.
Jared Atkinson
Yeah, both. Both for sure.
Patrick Gray
Yeah. All right, now, joining us as well, joining us also is Justin Kohler, also of Specdrops, who's bringing us the examples, bringing us the receipts. First of all, though, Justin, how long's this actually been out there? I'm guessing you rolled it out as a beta or a beta, as you Americans like to say. And you know, you got it out there with a few key customers before it went ga. Yeah. So like, what's been the rollout of this and then what have people found and of what's been found, what's been surprising to you?
Justin Kohler
Yeah, so we started with a very small set of what we call design partners, and we are Announcing our beta actually on July 28th. So this is in beta.
Patrick Gray
So we're like pretty pre beta right
Justin Kohler
now, so kind of like fresh off the press. But we have some really interesting stories. Obviously genericize to protect the innocent here, but I've got three kind of examples and I would say that this is not surprising for any platform. Like the same thing applies, but for some specificity to help people understand what we can map in aws. In one environment, we found one very highly privileged role was trusting an external organization. And that was. That's obviously a bad thing because like what that means is that external organization can just assume that role in that account, but they had no idea what that account was.
Patrick Gray
Well, and who knows what their controls are. And that's not good.
Justin Kohler
Yeah. And so it could have been through an acquisition or a consulting engagement or whatever, but it just was never cleaned up. And so that was a very surprising like, oh, let's fix that now kind of thing. Another example was we had Jared mentioned like hundreds of accounts, I think earlier that's fairly common. You have many accounts atypical to domains. Maybe a large company has many domains in active directory, but in AWS it's frequent that you'll have many accounts. And so this organization was managing its role policies with wildcards, which is fine because they had conditions that said, well, it can only be from our account where you use those roles, unfortunately on one account or on one role that was not properly applied or one account. So out of hundreds of accounts that are doing this at scale, you miss one and you have this terrible outcome that could possibly happen. So it's finding that needle in the haystack that we were able to see. And I think the third one is we keep hearing this. Our AWS account does not have external trust. We don't trust external organizations. And every time we found trust, so sometimes it's legitimate and it's like, oh, I didn't realize that we were doing that because the statement again was we don't do that. And then sometimes it's very surprising like that might not be legitimate and we need to figure that out. So it's kind of like when people say we hear this a lot, I separate my admins from my regular users and it's like, no, you actually don't. And here, let me show you how you are not doing that. I would say broadly, like you asked Jared earlier, is this just an AWS thing or is it like a hybrid thing? And I would say I'm just going to double tap on it, it's a both thing because.
Patrick Gray
No, no, look, it makes 100% because there's enough complexity within AWS itself that you could just use this and only look at AWS and you're going to find stuff. And then when you start extending it out beyond the borders of aws, you're also going to find stuff there. It's funny what you were saying too, just about misconfigurations and this isn't. These examples that I'm about to give, you do not need to do graph analysis to find these sort of things. You just need to audit your permissions. But my two favorites that I've heard of or observed, excuse me, in my career, one of them was there was a hack at the Australian Parliament House by an Iranian apt and they managed to enumerate a bunch of stuff out of a directory and publish it and say, look, we, you know, hacked the Aussies and I got some good sourcing in the end where it turned out that they just happened to have fished a regular user that had extra permissions in that tenant that they weren't supposed to have. And there were like two inboxes across the entire tenant that had those that were over provisioned. And the person who was like in the know on that incident was just like, man, they just like got insanely lucky. Right? And it happens, I guess that's why I'm saying it, it happens where an attacker can just hit that misconfigured account and then they're off to the races. And the other one, which is something I've mentioned a bunch on the show, is when someone was doing incident response and they discovered that every single user in a tenant had what's it called, Intune admin rights. But like every user in a fairly large company. So that, that one was funny. But of course, you know, this is those sort of things you should be finding just doing audits. What makes the graph stuff powerful is. Yeah. Being able to cut across different boundaries and find out the things that are a bit. So what's the response been like from the like alpha users who are using this? I'm guessing, you know, you just gave us some real world examples. I'm guessing they were suitably horrified when they turned this stuff up.
Justin Kohler
Yeah, I would say generally they're all very excited. I mean it gives a level of visibility that they haven't seen before. I would say, you know, misconfigurations. There's a lot of tools that can show you misconfigurations. You mentioned auditing chaining them together and showing you the totality of that impact is what Bloodhound can really show you.
Patrick Gray
Yeah, you thought things were bad, but we're here to show you they're infinitely worse.
Justin Kohler
Just how bad.
Patrick Gray
Yeah, exactly.
Justin Kohler
The other thing that I would say is it doesn't even need to be a misconfiguration. So here's an example. Like, a developer will have access to a lambda function, that lambda function has access to a privileged role that runs in cicd that now gets into prod. So that's not. Each step along the way is not a bad thing.
Patrick Gray
No, I get it. It's like powershells 15 years ago. Same sort of thing. Exactly the same sort of thing, which is these things haven't been thought through properly and there's privilege that pops up where you don't expect it.
Jared Atkinson
I think there's a. There's a bit that's really. We probably talk about this every time we chat, Pat. But there it's. Philosophically, it's useful to think about the AWS as a microcosm and then kind of like, fix what you can there, but then also zoom out, kind of this hybrid talk that we talk about. Because the problem is, is that typically I'll have the ability to see what's going on, and then I could go back out to something that I have more control over, like AD or OKTA or what, you know, whatever you're using as your federation provider into aws, if that's the case. And then you could go back and you can. You can kind of like reposition yourself and then go back into AWS and kind of move around. And so a lot of times it's like, I get my initial foothold, I see what's going on, and then I can use kind of this overarching enterprise to reposition myself throughout, and then go back into a location where I do have that one user that has the inbox that I can access, or the one user that has access to this lambda function, which then has access to this key management store. And I could start to pull that stuff out.
Justin Kohler
I think that's the big thing about SpectreOps, is a lot of organizations think in terms of platforms or applications. I'm the AWS person, I'm the AD person. And we like attackers, and we don't
Patrick Gray
look at it like, who's the everything person? Right. And I think that's the. Yeah, exactly. That's the issue. All right, guys, we're gonna wrap it up there. Jared Atkinson. Justin Kohler. I think you're about to sell approximately a gajillion licenses of this thing. I think it's a great place for Bloodhound to go. I think it's, it's good timing too because it seems like everyone's validating the approach to building graphs at the moment and like you're not the only ones anymore who offer like a graph, graph based tech for ad. So it's great to see you extend it out and be like, well, okay, you've all caught up to us on thinking about active directory graphs. But like look over here. Buildings on fire. Fantastic. Love to see it. Great to chat to you both.
Jared Atkinson
Thanks, Pat.
Justin Kohler
Thank you.
Patrick Gray
That was Justin Kohler and Jared Atkinson there from Spectrops. Big thanks to them for that. And of course, if you want to find more information about on Spectre Ops Bloodhound, yeah, just Google Bloodhound aws. You're going to find all you need to know on that. And yeah, just amazing work from them. I have a feeling that it's like when Nessus was new vulnerability scanning and people would do that first vulnerability scan, a whole bunch of like horror show stuff would fall out. I think that's what's going to happen to most of you all out there. If you throw Bloodhound at your AWS environment, you're going to see some scary, scary stuff. But that is it for this week's show. I do hope you enjoyed it. I'll be back soon with more security news and analysis, but until then I've been Patrick Gray, thanks for listening.
Podcast: Risky Business
Date: July 29, 2026
Host: Patrick Gray
Guests: James Wilson, Pete Ranks; Sponsored Segment: Justin Kohler & Jared Atkinson (SpectreOps)
This episode plunges into a high-octane week in information security, dominated by fast-moving developments in AI policy and open weight models, a headline-making breach involving OpenAI and Hugging Face, and a slew of news on cyberattacks—from nation-state actors to botnet abuse on consumer devices. The tone is sharp, critical, and often wry, with industry insiders weighing in on both technical and policy fronts.
The Open Weight Model Policy Fracas
Distillation in AI: Scapegoat or Savior?
Distillation—using one model to train or improve another—lands in regulatory crosshairs.
Example: Google’s now-defunct Gemini distillation service is quietly scrubbed from public view, reflecting political sensitivity.
"In shaping this ecosystem, policymakers should be careful not to conflate legitimate model development techniques with misappropriation." —Patrick Gray, quoting Jensen Huang (03:46)
Anthropic’s Dario takes a dissenting position: warning of the dangers of open weights and arguing that hardware is more critical for US AI dominance than software.
Guardrails Debate
"Anthropic is…doing the largest amount of detriment, I think, to cybersecurity defenders at the moment by having these overly guardrailed models and no other options available."
—James Wilson (08:25)
Concentration vs. Proliferation Risk
Hardware as the Control Lever
"It’s easier to control hardware across borders than it is to control knowledge across borders."
—Pete Ranks (14:46)
OpenAI/Hugging Face Breach
"They've taken a system designed to be the best in the history of the world at exploiting software and put it in a cage made of software, didn't pay attention to it for three or four days and were surprised that it escaped. It's a fireplace made out of wood."
—Pete Ranks (20:10)
Legislative Moves in the US
Iran and OT/Utilities Attacks
"Every attack by any actor, criminal or State, on U.S. civilian critical infrastructure is an opportunity to message China, to convince them that they should not think about operationalizing their Volt Typhoon campaign."
—Pete Ranks (27:49)
China's Volt Typhoon and Russian Mischief
Residential Proxy Networks via Consumer Devices
"…the fact that you need a giant residential proxy network these days, I mean it's interesting for me because I'm on the board of a company called Knock Knock…"
—Patrick Gray (37:56)
Miscellaneous Incidents
SpectreOps’ Bloodhound Now Graphs AWS Attack Paths
Real-World Surprises from AWS Graphing
Common discoveries: overprivileged roles trusting external orgs, missed exceptions in policy wildcards, unexpected external trust links.
"In one environment, we found one very highly privileged role was trusting an external organization…And so that was a very surprising like, oh, let's fix that now kind of thing."
—Justin Kohler (54:25)
The challenge: AWS is so complex, even seasoned teams find they have dangerous misconfigurations.
Value is both intra-AWS ("inside the behemoth") and interconnections with AD/Entra/GitHub.
Customer Response
| Timestamp | Speaker | Quote / Segment |
|-----------|----------------|------------------------------------------|
| 03:46 | Patrick Gray | "[Policymakers] should be careful not to conflate legitimate model development techniques with misappropriation…" (Reads key Nvidia letter excerpt) |
| 08:25 | James Wilson | "Anthropic…are actually doing the largest amount of detriment, I think, to cybersecurity defenders…by having these overly guardrailed models..." |
| 14:46 | Pete Ranks | "It's easier to control hardware across borders than it is to control knowledge across borders." |
| 20:10 | Pete Ranks | "It's a fireplace made out of wood." (On failed AI containment) |
| 27:49 | Pete Ranks | "Every attack ... is an opportunity to message China...[not to] operationaliz[e] their Volt Typhoon campaign." |
| 37:56 | Patrick Gray | Commentary on residential proxy abuse and necessity for robust allowlists. |
| 54:25 | Justin Kohler | "In one environment, we found one very highly privileged role was trusting an external organization…" (On AWS graphing findings) |
A fast-paced, opinionated rundown of urgent news: Open source vs. safety debates are heating up, AI security remains fragile, nation-states press their cyber advantage, and even seemingly “boring” enterprise or consumer misconfigurations are exploited at scale. The sponsored Bloodhound segment underscores that security pitfalls lurk not just in legacy IT, but in complex, modern cloud deployments.
This episode is a dense, engaging resource on the intersection of AI policy debates, real-world security incidents, and the ongoing chess match between attackers, defenders, and regulators.
Advertisements, music, and outro trimmed; all substance, no waffle.