← Risky Bulletin
Loading summary
Transcript31 lines
- [00:00]
Tom Uren
Foreign.
- [00:05]
Patrick Gray
And welcome to Seriously Risky Business, the podcast we do here at Risky Biz hq, which is all about cyber policy and intelligence, macro level trends, that sort of stuff. My name is Patrick Gray. So in this podcast we chat with Tom Uren, who works with us as our policy and intelligence editor, about the newsletter that he writes every week, which is this Seriously Risky Business newsletter that you can find at Risky Biz. And yeah, before we get going, we would like to say a big old thank you to the William and Flora Hewlett foundation for supporting Tom's work with us, and also to Lawfare, which syndicates the newsletter as a column on its website. We also have a corporate sponsor this week which is Push Security, Funnily enough, a company that I advise and spend a lot of time talking to. They do some really interesting stuff with a browser plugin basically that can prevent phishing that gets missed by other security controls. Because these days, you know, fishing crews are putting their fishing pages behind stuff like cloudflare, Turnstile, they're putting them behind captures. So really the best way to spot that stuff is to be in the browser and see what the user sees. It's a product that we actually use at Risky Business. And yeah, I can't recommend it highly enough, but Tom, thank you for joining me. Let's get into it now. And the first item that you've written about is Trump has sort of issued an executive order to do with cybersecurity that sort of supersedes one that was published by the Byron or enacted by the Biden White House in the final months of that administration. And it was interesting, right, because I saw the announcements about the publication of this eo and it was really like, not, the announcements were really not very specific about what, what they were doing. But it looks like a key plank of what Trump's doing is walking back some of the mandates that would have seen vendors forced to improve the security of their product.
- [02:01]
Tom Uren
Yeah, yeah. The whole thing is quite interesting. And I think the news that you saw reflects that the EO was really just an edit. It's kind of like a Microsoft Word document where you go out and scrub out the previous executive order and it says things like strike out these paragraphs and replace them with these words. And it's quite hard to just reading the EO to know what's actually going on. But the long and the short of it is that in the it was the dying weeks of the Biden administration, they put out an executive order that said, we want to update very roughly, we want to update security standards. And we want to encourage both the federal government and also people selling to the federal government to adopt these security standards. And so it had all these processes for doing that. And basically the Trump executive order says, yeah, updating is still a good idea, we want good standards. But they basically describe the, I guess the, maybe you could call it coercive or the carrot and stick measures as unproven and too much hard work and just busy work for the sake of busy work. And they're, you know, we'd rather let the market come up with its own tailored security solutions.
- [03:23]
Patrick Gray
Well, that's worked out really well so far, hasn't it?
- [03:25]
Tom Uren
Well, yeah, exactly. So that would be my criticism of it. So that, to be fair, that was in a fact sheet that wasn't actually in the executive order. The executive order is much more straightforward. The fact sheet describes it as unproven and burdensome. And I mean, if I was to describe the Biden cybersecurity philosophy, it would be that the market, if left alone, doesn't work its way necessarily towards good security outcomes. So government needs to step in and be more of an active shaper of that market to try and encourage firms and organizations to adopt more stringent security practices. And I think that was kind of right because as you said, like you look at the last however many years of cyber security, when it's just left alone, people don't default to making really good products secure in general. And so that was kind of a change that I think was necessary. And basically this EO is walking it back now. It's still the early days of the Trump administration. What would be nice to see is what is their vision for how to make things better? They haven't advanced one here. I think there's still space for them to do that.
- [04:40]
Patrick Gray
Yeah, I mean, we did see in Trump's first term a whole bunch of actually quite forward thinking executive orders around cybersecurity. I think we can attribute that to the fact that Rob Joyce was in the White House, probably writing them himself. You know, I don't know that we're going to see the same quality of work this time around, but it does point, you know, your piece points out that the White House fact sheet that accompanies this latest EO says Biden imposed unproven and burdensome software accounting processes that prioritised compliance checklists over genuine security investments. It didn't quite seem that way to me. I think updating to like memsafe languages and things like that, that's not a checklist item, that's a meaningful improvement. So I think. I don't know. I don't know why this happened. I'm guessing there was some lobbying involved, but it seems. It seems an odd thing to walk back, actually, because really, the trajectory that the Biden admin had this all on is, you know, you must be this high to enter, you must be this high to pass government procurement standards. And, you know, everybody benefits from that. Like, even me as an Australian citizen, you know, talking to my friends who work in the private sector, they're going to benefit out of that because, you know, if these companies improve the product so that they can still get contracts at the federal government, you know, it's not like they're going to still keep shipping the bad version to everybody else. Right.
- [05:58]
Tom Uren
So, yeah, part of my feeling is that this Biden EO came out, I think it was January 15th or something like that. So, like, you know, was it 10 days, something like that, before Trump's inauguration? And I get the feeling that people are just looking for things to roll back. Like it does, to be honest, seem a bit rich to impose policies on the next administration when you know that you're going. Like, a lot of the time they just say, well, we'll just sit and do nothing because it's Trump's decision to make the call going forward. And so I think there might have been a bit of this dynamic. Let's nitpick and find things that we just don't like and we can roll back.
- [06:37]
Patrick Gray
Yeah, I don't know. I mean, I suspect someone who's friends with someone in the admin, kind of like informal lobbying, which is how it seems to work in Trump world. A lot of the time it does.
- [06:47]
Tom Uren
Feel very much like this is. Let's make it a bit easier for business.
- [06:50]
Patrick Gray
Yeah, exactly, exactly. All right, so let's move on to the next thing that you talked about, that you wrote about, which is a couple of reports from the IA giants, OpenAI and Anthropic. So it's funny because you were writing about this Anthropic report. A friend of mine just sent it to me the other day, so I thumbing through it and. Yeah, I guess someone sent it to you too, but it's a couple of months old. And then there's a OpenAI one from. From last week, which has really looked at how malicious people. Malicious people, how threat actors are maliciously using AI. And what's interesting here is when you look at how it's being used in, like, you know, cyber campaigns, it's kind of pretty dull and doesn't really look like it does all that much. It might make things a little bit more efficient, but it's not a game changer. But what's really interesting is the extent to which these AI, sorry, these North Korean fake IT worker scams, like the degree to which North Korea has integrated AI into every step of the processes that are involved in executing this scam, like they're all about it. Like, you know, North Korea could spin up a sideline in AI consulting at this point.
- [07:56]
Tom Uren
Yeah, yeah, it's really quite extensive. So this is the second time that OpenAI has written about, about the IT worker scam and the first time it said they're. To me it felt like they're sort of experimenting and playing around. They're manually trying to set up resumes and develop work histories. And this time they describe it as they use the words systematically fabricate detailed resumes aligned to various tech job descriptions, Personas and industry norms. The operators use looping scripts to generate consistent work histories, educational backgrounds and references. And it seems like they're coming up with a whole package of things that are authentic seeming because they're generated by AI and consistent. And that helps them get over the bar of skeptical HR recruitment type people. And it's not only the generation of Personas and I guess job packages or recruitment packages, but it's also how do we actually keep track of all the different job applications we've got? So they're asking ChatGPT, I presume, how do we build a tool to manage all the different applications that we've put in? You can imagine that that would actually be quite a nightmare if you're putting in hundreds of applications. Just the paperwork of keeping track of all of those could be quite a pain. And how do we, how do we use tools that stop us getting caught by corporate security, getting pinged by corporate security, and then also how do we recruit people to host our laptop farms? So they're helping with both technology and communications and just seeming authentic. It's really ramped up since the February report. So it's like every aspect of the aircraads business, the fake business, and they're trying to automate it much more. So the company OpenAI says it cannot independently assess the success of these operations. It's sort of taking a bit of a hands off approach. But in the last months there have been many, many reports about just how successful these are.
- [10:13]
Patrick Gray
Out of control, man, it's crazy. And we had a sponsor interview actually at yesterday's weekly show with Alex Tilley, who's a former Australian Federal Police invest investigator who now works in a threat intelligence role at Okta. And what's really interesting is how much of this lines up with what he was talking about, where they could use Okta identity events because they've got identity information from all corners of the, of the Internet. You know, so many people rely on Okta's infrastructure and, you know, through auth0 as well, like, they just see so much and, you know, he can watch these operators actually hitting these different services and using different tools to do these creations and whatever. And like, they tend to maintain the same identity across a bunch of services. And you can actually see, see what their identity and resume factory looks like from an identity perspective. And it's all, yeah, it's all really interesting, but I just think it's, you know, North Korea, we keep talking, we keep talking in like, what seems like really favorable terms about North Korea because, you know, they're doing some great hacking and they're, you know, they're being really innovative here, like really innovative at scale. And it's amazing, really.
- [11:20]
Tom Uren
Yeah, yeah. So that stood out in stark contrast to a lot of the rest of the report. So there's many covert influence operations, basically where someone tries to grassroots a social media campaign and people are using AI for those to generate comments and create Personas. But those campaigns very rarely make any kind of impact at all. And so it seems like the, if you can trick one person, that can be enough to have a significant impact on something. So if you're a scammer, that can be a payday. And if you're the North Koreans, that can be a lucrative job and an opportunity for an insider threat or to compromise or breach something. And so the AI helps you get over that hurdle of tricking a single person or a panel or whatever. But when it comes to tricking huge numbers of people or convincing huge number of people in social media campaigns, that doesn't seem to work.
- [12:19]
Patrick Gray
I wonder if it's less about being convincing, though, and more about just, you know, the stuff that tends to draw attention on social media tends to have a creative spark to it, and models just don't do that, you know. But, you know, still couching, couching it in terms of believability meant that you got to quote Bob Marley in the piece by saying, as Bob Marley saying, you know, you can fool some people some of the time, but you can't fool all the people all of the time. Time.
- [12:43]
Tom Uren
Yeah, I very much Got that sense from these two reports.
- [12:48]
Patrick Gray
Yeah, I mean, I think these reports have generally just got more interesting over the last few years as AI is being used more and more to do all sorts of stuff. And I don't think it's a static thing. Right. Like I think there will come a time when we're going to see some cyber campaigns and stuff that are really making use of AI and doing really well. But the question will be like, will that be observable because they're using ChatGPT or are they going to be using their own self hosted models? Deep seq, you know, llama, whatever. Right. I think it'll be more like that. So these reports won't necessarily be that relevant to that side of it.
- [13:20]
Tom Uren
Yeah, yeah, I kind of agree. I expect that people use self hosted LLMs. Now. One of the really interesting things that came out of the anthropic report is it they found an actor who was selling influence as a service and they had really gone full bell. What is it? Full all out in terms of using an LLM to actually manage campaigns.
- [13:44]
Patrick Gray
They used all the bells and whistles, I think is what you were going for there.
- [13:49]
Tom Uren
So they had Claude, which is Anthropic's large language model, and it was running everything. So it was keeping track of Personas, making sure that they were behaving correctly, that they were consistent with their purported political viewpoints, drafting content, they were sending off image creation requests and then reviewing the images to see if it was. Or Claude was reviewing the images to see if they were appropriate for their purpose. And they also had, I guess you'd call them OPSEC safeguards. So you know how it's quite often people will reply when they think it's a bot and say, you know, do something ridiculous like create a poem. And so they had safeguards to I guess intercept those kinds of commands and make light of them, like joke about it. And so this is like to me it sounded amazing in terms of here's all the things you can use an LLM to do to manage your influence campaign. Yet at the same time it wasn't really getting any traction. And so it did a good job. But the campaign just wasn't kicking the kind of goals that we think it should kick, which is lots of authentic engagement.
- [15:01]
Patrick Gray
Yeah, again, I think that just comes back down to the, you know, the lack of humanness in AI generated content. But look, the last thing we're going to talk about today. So moving on to the last thing and this is something that we covered in Yesterday's weekly show as well. But I'm so glad you covered it and I will confess that I put the headline on this one which is these guys are going to die. Lol. So this group of people on Telegram selling classifier data, they claim to have a whole bunch of classified data and they're selling it to anyone who's got the. Presumably Monero. I think it's Monero that they're asking for. And they gave a free sample to the New York Times, which included some FSB documents that honestly were pretty interesting, right? Like these were not nothing documents. You know, I think, you know, as I say, we covered a bunch of this on the weekly show yesterday. But I think the interesting thing that we didn't really talk about on the weekly about this is that if you're a Western intelligence agency, I mean, would you spend 120k monero for their full dump of Russian documents? And I'm thinking even though you can't trust them, even though you can't trust that these documents haven't been modified or are legitimate, it's probably worth a punt.
- [16:11]
Tom Uren
I would absolutely negotiate down. Yeah, so we can spend 30,000.
- [16:18]
Patrick Gray
Yeah, exactly. But I mean, the point is it's probably worth a look, right? Like, would you, as a Western intelligence agency, you know, buy documents from some shady broker?
- [16:30]
Tom Uren
I think the problems, as I was thinking about this, like intelligence is a cycle, like you come up with what you really want to know, you try and figure out who would have it, you try and convince them to give those things to you for, you know, with whatever lever you can use, money or ego or ideology or whatever. And so this is missing all that part of it, it's missing the targeting. What do you really need to know? But at the same time, if it's not a huge amount of money, why the hell not? It's like, it's like a lucky dip.
- [17:02]
Patrick Gray
Exactly. That was the words. It's an intelligence lucky dip, which is just throw a bit of money at these guys and see what you find. I mean, you might find that you can confirm a couple of data points, you know, that are relevant to what you're looking at. I mean, it's probably not going to happen, right? But I think that's probably more to do with intelligence agencies being sort of lumbering bureaucracies, more so than it being, you know, a bad idea. And also, as you say, it doesn't quite integrate into that life cycle, does it, where you're like, we're spinning up a program to collect on this because this is really important. It's a little bit different to someone jumping up and saying, hey, want to buy a cache of stolen documents? Like, that doesn't fit into the cycle.
- [17:42]
Tom Uren
I mean, the other way of thinking about it is you'll have people who have budgets and they'll have budgets to try and meet intelligent requirements. And it's like, are you going to pay $30,000 for this? And it's like, I don't know. Does it match my intelligence requirement? Who knows?
- [17:56]
Patrick Gray
Yeah.
- [17:56]
Tom Uren
And so I think there's these weird organisational, you know, the way things work that may make it difficult. I still think it's actually a pretty good idea. I think you can get a discount. They were negotiable.
- [18:08]
Patrick Gray
Yeah. Now, you know, as we've said here, it's likely that these guys might come down with a serious case of the Windows. And we've linked through to a list of all of the Russians who've gone out of windows since the. Since the invasion of ukra. Because, you know, they're doing this on Telegram, which is probably completely infiltrated by Russian intelligence agencies. So, you know, what are the odds that these guys just mysteriously vanish one day? I think those odds are pretty high, if I'm honest.
- [18:32]
Tom Uren
I mean, it's. It's just so unknowable to me in terms of. Are these documents even important with the FSB care? I think having them appear in the New York Times or be reported on in the New York Times is embarrassing for someone. So that, I think may be the motivation more than their importance, perhaps. Yeah. So I'd be worried about it.
- [18:54]
Patrick Gray
Yeah. Yeah. I wouldn't want to be the person selling stolen FSB documents on Telegram. Yeah, just same. Same as you there, guy. All right, Tom Uren, thank you very much for joining me to talk through your work this week. Fascinating as always. And I look forward to doing it with you again next week.
- [19:08]
Tom Uren
Thanks, Patric.