Loading summary
Transcript102 lines
- [00:03]
A
Hello, everyone, this is Tom Uren. I'm here with another between two Nerds episode with the Gruk. G', day, mate. How are you?
- [00:10]
B
Fine. And yourself?
- [00:11]
A
I'm good. This week's episode is brought to you by Airlock Digital, who make whitelisting easy to carry out in your organization. Particularly important now that AI powered hackers are running around everywhere. Speaking of which. So this week that was a smooth
- [00:26]
B
segue that you set up right there.
- [00:29]
A
Speaking of which, this week there was news that Hugging Face was hacked. That is an American AI adjacent company. It's heavily involved in hosting models and is quite central. And the story is that an OpenAI model that the company was testing. OpenAI was testing on a cybersecurity evaluation. It had had its safeguards backed off the OpenAI model, hacked its own company infrastructure, got out to the Internet and then hacked Hugging Face in an attempt to find the answers to the cybersecurity evaluation.
- [01:14]
B
So that's, that's a passing grade in my book.
- [01:18]
A
Yes, you would think so, wouldn't you?
- [01:20]
B
Congratulations. Yeah.
- [01:21]
A
So in the wake of that instant, you messaged me and said, I'm just thinking that cyber, cyber is people.
- [01:30]
B
Right.
- [01:32]
A
And so we're going to talk about whether what that means and whether AI changes that. My first thought was just cyber power and states. But anyway, you kick off.
- [01:46]
B
Right. So my argument for cyber is people. There's a lot of stuff where people like Western aid to Ukraine helped with cyber security. And I'm looking at it and it's like it helped with licenses, it helped with, you know, deferring the costs of cloud computing, it helped with that sort of stuff. But all of the actual work had to be done by human beings, and those human beings were Ukrainian administrators. So the Western aid was basically like paying for the Uber ride. But the guy who drove the Uber doesn't get any credit for like taking you somewhere, which is, I mean, that's maybe fair. But someone had to do it, right? Someone had to do the work. There was like all of the cyber stuff that you can talk about, none of it is actually computers. It's all actually people. If you've got a firewall, that, that firewall has to be configured, it has to be installed, it has to be monitored. Like that has to be done by people. So the firewall is a tool being used by people. So the, the cyber stuff is just like it's people doing things, like all of it. And so that's where I was coming from. I was just very frustrated at the idea of Like NMAP by itself is not a hacking tool. It's just a tool that does a thing. If you use it for hacking, then it is, but that's the person doing it. Like objects don't have agency.
- [03:03]
A
So in the case Hugging Face case, the apparent story is that OpenAI set the model there seems like a couple of top tier models in front of a computer, metaphorically, and said, we want you to go hack this, off you go. And they did everything that I described. So the breaking out of OpenAI's infrastructure, the hacking of hugging face autonomously. So there's no actual person, I mean, there's a person to say go. And then interestingly, the hugging face crew, they said that they used another model, an open weight Chinese model, to do instant response, because just the volume, it would have taken them quite a long time. And they said that they were able to rip through the incident in a couple of hours using that model. And so in this case, there's an attack, supposedly autonomous, although directed by a person, set off by a person, and an instant response, again run by a machine, but directed by a human.
- [04:13]
B
So I'd say that one of the other parts of Cyber's People is OpenAI has a network that was set up by people and that network was set up in a way that the test environment was not completely isolated.
- [04:27]
A
Just a bit, perhaps.
- [04:29]
B
Yeah. And that's a mistake. And one of the things is people make errors like humans make human errors. So like cyber as people are the, like all of the attack stuff is people doing things, but on the defense, it's people doing things as well. Like they make mistakes or they cut corners or they misconfigure something, or they, you know, get lazy and stop doing what they know they're supposed to do. They, they go from logging in with a password to leaving it logged in all the time because it's annoying to do otherwise. Cyber is people. And so I would say that even though in this case there's a lot of AI involved, there were still people that allowed the cyber to happen that like created the environment in which that was possible. So it's still, in a way, it's people. I don't know that that's always going to be true. It feels like it might be the end of that, but.
- [05:24]
A
Right. So when you sent that to me, I had a little bit of think about and I read it as Cyber power is people. And certainly so not very good at reading, but certainly. Or maybe you're not very good at remembering what you said, but certainly like three years ago, if you had said cyberpower is people in a state context, I would have said absolutely, yes. So I think the reason that we've got some countries that we would consider cyber powers is that they just have the political will to harness a lot of people to go and do cyber operations.
- [06:04]
B
So, yeah, like North Korea, for example, who don't have other things that would. Like, you wouldn't look at North Korea where like the lights are not on at night except in the capital city. Like, you wouldn't look at that and be like, this is a cyber powerhouse. This is where you want to go for like top tier talent. You look at and be like, these are subsistence farmers living in the Stone Age, essentially.
- [06:26]
A
Yeah, I think that's a. Yeah, there was a long period where cyber security people were skeptical of North Korea for I think, exactly that reason.
- [06:34]
B
Right, right, right. And then now you'll see like multiple change bugs to get access to a thing where they change a JavaScript file for 15 minutes in the right window to capture something and they transfer several million dollars from a hardware wallet that wasn't accessible except at that brief window. Yeah.
- [06:55]
A
So
- [06:57]
B
they're doing magic level, state level, state tier, level. At least there's higher level stuff. But they're definitely, they're definitely very capable.
- [07:08]
A
Argue that cyber power remains people, regardless of AI. I mean, there is a lot of evidence that North Korea is using AI for things like its IT worker schemes, where it uses it to create resumes where North Koreans can get jobs in Western companies and earn an income and perhaps hack them. Right. So that is useful. But if you, I still believe that if you took all the AI away from North Korea, they would still be pulling off outrageous hacks. Right, right. Maybe not as quickly or as many, but still.
- [07:43]
B
Yeah, I think it would be the volume that goes down. That's all Right. AI doesn't give them a capability that they lack, it just allows them to scale better. That's the way I would interpret it, is that if you want to have cyber, you have to invest in cyber. Even if that turns out to be an AI, it's still an expression of state will to have state power. And if you don't have the will, then it doesn't matter what tooling exists, you're still not going to have like, it's not going to be magically available just because AI can do it.
- [08:16]
A
Yes. So another thought or question I had was, does the rise of very powerful AI change the cyber power rankings? Power rankings. And where, you know, maybe, maybe he would place the US or China on top. You could argue about that. But again, I don't know that it makes a huge difference. I think that they will each have tools that are good for doing things.
- [08:45]
B
And yeah, no, I think as you were saying earlier just before we started recording, it's that the countries that are investing the sort of resources to have like 1,000 hackers right now, like those are the same people that are investing the resources to have a lot of compute and a lot of like to develop frontier models and all that. It's the same people. Right. So it's, it's not like suddenly the, the Sudanese are going to become a cyberpower just because all they need is a few Nvidia chips and like access to hugging face to download a model. Like we're not there yet at least. I wonder if we will ever get there because I feel like the counter side of that is if you can download a good hacking AI, you should be able to download a good defending AI that can configure your systems for you so you don't make those mistakes.
- [09:36]
A
Right. So it's a rising tide lifts all boats kind of situation. I suppose you could think that maybe the, the US might end up a bit better because they've got the frontier labs right now. But I think it's like it's not a magnitude of water better off.
- [09:53]
B
Right.
- [09:53]
A
It may be a little bit better off on the edges, on the margins.
- [09:57]
B
I mean it's like they're the world's only superpower. And that will be true in AI I think, but it doesn't necessarily translate into being a magnet as you were saying. Like it doesn't mean that they're like magnitudes better than other people who have to sort of follow behind on that.
- [10:16]
A
Yeah, yeah. Maybe the Chinese will be second because they've got their own labs and then poor old Russia, they'll be living off the dregs of the Chinese.
- [10:26]
B
Chinese second hand AIs.
- [10:29]
A
Now I was also thinking about cyber scammers. So the big industrialized something called pig butchering. Yeah, yeah.
- [10:41]
B
Where they, where they have like compounds.
- [10:43]
A
Yep. Even compounds of people. They've lured their own false pretenses and they kind of like romance scams, crypto investment scams. And these have become massive enterprises, huge industries.
- [10:57]
B
Yeah.
- [10:57]
A
And they seem like an example of where cyber is people. Because they've become huge because they've been able to harness like tens of thousands or hundreds of thousands of people. Right. And, and get them to work.
- [11:09]
B
Yes.
- [11:10]
A
But they also seem like A business that AI could really replace a lot of those people.
- [11:16]
B
It's ripe for disruption. There's an opportunity for the first agentic AI that does pick butchering scams to go in and really shake things up.
- [11:27]
A
You know, I was actually going to say that, but then I thought that might be a bit insensitive. So I'm glad you did it. Yeah, thanks a lot, Tom. So I actually had really mixed feelings about that because, like, scamming is bad. Having enslaved people do the scamming is even worse. So, you know, AI replacing them is. Is that a win? I'd probably have to say win for
- [11:52]
B
humanity, but not necessarily for victims.
- [11:55]
A
Yes, well, you remove half the victims.
- [11:57]
B
Right, right, right. As someone who uses AI, we both know, and I'm sure our audience knows, the one thing that AI is really good at is convincing people. It's very good at being confident and just saying things that it believes are true.
- [12:14]
A
Right, right.
- [12:15]
B
Getting you to accept what it puts out there.
- [12:19]
A
You often feel like you're being dragged in different directions because I'll say this and they go, yes. And then I'll say, but what about this? And you'll go, you're perfectly right.
- [12:26]
B
Yes.
- [12:26]
A
And then you're sort of back where you started. So there's.
- [12:30]
B
And it's worth bringing up the tension. What you're saying is there's a guy
- [12:34]
A
on YouTube, Mo Bitar, I think his handle is, who's very entertaining and he speaks a lot about reinforcement learning by human feedback, where they will train the AI based on what a person likes. So there is an element of like, they're deliberately trained to give you something that will be pleasing.
- [12:55]
B
If they just said you're an idiot, why are you wasting my time? You're probably not going to stay on that chat client very long. You're not going to be like, hey, I should respond to that. Let's keep this going.
- [13:06]
A
I guess going back to scam compounds, that seems like the exact sort of property that would be very useful.
- [13:11]
B
Right, right. And I would say that so like one of the things that holds back AI at home, wherever, is that it's very slow. Right. Because you don't have the compute that's available to like these Frontier Labs. Instead you have whatever you can afford, which is less money than they're spending. And so running an AI on your laptop could take an hour to process. Whatever. That would take less than a minute if you're using the hardware from a Frontier lab. However, because email scams, these email based scams are Asynchronous that cost is not a problem. It's not a thing that gets in the way. If you can only do a chat message every 15 minutes instead of every one, you can deal with it. That's absolutely fine. Right. Like, that's not an issue at all. So I'm bearish on human trafficking and bullish on AI scamming.
- [14:09]
A
Yeah. Because it seems like the people who they've trafficked and are running the scams aren't just winging it. They've got, I guess, empirically developed playbooks that, that they know work based on where you are in a conversation, what you try next. And that seems like if you've got such a strong framework, you wouldn't need a very super powerful AI to run a playbook. Right.
- [14:35]
B
Yeah. It needs to modify a template as opposed to figure out what to do next as a step towards achieving a goal. Right. You get in a thing that says this, you look through your playbook to see, okay, the appropriate response is 17B. You pull it out, you read ziggered a little bit. So it's not the exact same text. Put some right words in.
- [15:00]
A
Whatever the perspective of a scam, Kim Kmin. It seems like using machines instead of people would be good from the perspective of just management overhead. Like, I don't want to have to deal with managing, like, the logistics around a whole lot of people, but also from the perspective of getting caught and getting punished.
- [15:24]
B
So. Right, right. So it's if, if, if China finds out that you've stolen a lot of money, that's one thing. Yeah. If China finds out that you've kept, you know, 10,000 of their citizens in slave conditions and been beating them and starving them, that's a death sentence for
- [15:43]
A
you and everyone involved. Yeah.
- [15:45]
B
Right. So, yeah, if I'm a scam kingpin, I'm looking at this and saying this, like, this saves me so much. Like, it saves me grief. If I get caught, it saves me money because I don't have to maintain a compound with thousands of people and hundreds of guards and getting in food and keeping it. Like, you don't have to pay for any of the stuff that you have to pay. Like, you don't need dormitories, you don't need like working rooms. You don't need a thousand computers. Yeah, you need 10,000 computers, but you can kept. You can keep it in a warehouse.
- [16:18]
A
Right?
- [16:19]
B
Yeah, yeah.
- [16:19]
A
So I mean that I buy the logic that for them cyberpower is maybe could like foreseeable.
- [16:30]
B
It's not people.
- [16:30]
A
Yeah, it's not people. And that makes me wonder what's wrong with my previous logic when it came to states. And like, what's the, what's the difference between a massive scam compound and a state? I mean, other than the obvious. Other than the obvious differences, it comes
- [16:55]
B
down to, like, motivation or like, what's your. What are the objectives that you're trying to accomplish? And if the objectives you're trying to accomplish is trick a whole bunch of people and get them to do something, or just convince a whole bunch of people that you are trustworthy and correct, AI is great at that. That is a good thing. And if you run a business that requires the ability to make people trust you and like you and do what you say, AI is very good for that.
- [17:24]
A
And so scammers are an excellent match or meta or.
- [17:30]
B
Yeah, they're good matches as well, I think, at some point. But I would say that that is useful for states because if you're doing phishing, that's what you're trying to do, Right. So if you're a state cyber capability, then having better phishing that requires fewer people to manage it, allows you to scale better, to do more custom targeted stuff, gives you access to languages that you don't necessarily have at native level. Right. So you can, you could do recruitments against like someone who. Yeah, you could do it in their native language. Right. Which would be good because if you can't write at native level, you can't convince them that you're a fellow native. So I think it's useful in this sort of case. But if you're a state and you're like, so interested in Romania that you need to have someone who can write it, a Romanian native level, you very likely have people like that on staff because you didn't just get interested in Romania today. Right? Right. It's sort of been an ongoing thing. And so you've been getting in, experts getting in, analysts getting in, like all of this stuff. So as a scammer, being able to now speak like Malay. Right. Seems useful. Or like, you know, being able to speak Vietnamese in your scamming stuff, that seems useful. I don't know. It's useful for a espionage unit because if you need to, like, do stuff against Vietnam, you have Vietnamese capabilities already, otherwise. Right. You wouldn't be interested.
- [19:05]
A
I mean, what I'm hearing is that you think there's a difference in the sort of depth of interest, so that a state really, really wants to achieve something in particular, and so investing resources
- [19:20]
B
into it for a while.
- [19:22]
A
Yep.
- [19:22]
B
Already, Yeah, I think.
- [19:24]
A
Yep. Whereas the sort of scam compound, it doesn't care who you are.
- [19:29]
B
Right.
- [19:29]
A
So it's anything that will get us. A small percentage of a lot of people is good. Right. Like an AI is good enough because you don't need motivated workers because obviously we're forcing them to work. I mean, I guess, yeah, you don't need the top tier of worker perhaps is a better.
- [19:51]
B
Right. So like as a scammer, being able to troll and cast a wide net is very useful because you're working on percentages, right. You're going to send out however many attempts and some of them will be successful. And of those, some of those will lead to, you know, actually getting money. And of those, there's going to be a range of money that you get from like a small amount to a very large amount. And so somehow that that small number of whatever has to cover everything. So the larger net you can throw, the more money you make. And if the costs of throwing a net increase with size, which it would. If you have to, like, if you have to get more people, if you have to traffic more people to do work, you have to paying more for that net. And so it's going to impact, it's going to impact the economics. Whereas with AI, I think you get a very large net for a fixed spend and it gives you a larger net overall. And so that seems very useful. But states are harpoon fishers. A state doesn't need to throw out a very, very wide net and just see what they get because they don't have the resources to process that. And it's like, it's not a thing that they want to do. Right. They want to be able to do like these exquisite, important things against people that matter to them, not against every Joe Schmo out there.
- [21:17]
A
Right, right. So what I'm hearing is that scams, big butchering, they have a wide net where they don't care about anything in particular and AI is good enough. It's just particularly well suited to, to computers where you get people to like you and it doesn't matter if you screw up all that much for any individual case. Sort of, you know, law of averages kind of stuff. Whereas states, the jobs are much more diverse. There's places where AI really helps, also places where it's not that good and you actually really care about getting it right because it's, you know, an intelligence priority rather than just
- [21:57]
B
it's not a side project where you're like making a birdhouse in your garage. And if you screw up putting it together, ah, whatever, you'll just toss it and make another one. You'll get it right eventually, and that's fine. It's not a hobby.
- [22:11]
A
Yeah, yeah.
- [22:12]
B
You're not ticking around.
- [22:13]
A
Yeah, that makes sense.
- [22:15]
B
Which, I mean, I feel like that's Almost why the OpenAI and hugging face instance shows why AI cybersecurity would not be good for, like, offensive cyber by AI is not a good spend right now anyway for a state, because the last thing you want is to get close to a network that you need to be on. Tell your AI, you know, solve this problem for me, get on that network. It decides that it's hard to do it directly, so it veers off to the left and hacks a whole bunch of random stuff in some long convoluted path to eventually get to the network. And, like, that's fine to do if that's what you want to do. If that's the pathway you've figured out, it's not fine to do because something autonomously decides, screw it, we'll do this and raise your risk profile, raise the opportunity of being discovered.
- [23:27]
A
So I guess in that example, what you're saying is that the OpenAI models had no concept of risk appetite and had no concept that they were no idea that they were exceeding that risk Appetite. I'm sure OpenAI did not want to write that press release. No one at the organization wanted that. And for OpenAI, that's perhaps, I mean, it's a hilarious outcome. But for a state when it's hacking, I don't know, Xi Jinping, that's not quite so funny.
- [24:07]
B
One of the things that occurred to me, like, the first time I saw just the hugging face report before OpenAI had figured out it was them, when it was this very, very breathless, you know, like, there were all of these. This swarm of AI agents that were doing all of this stuff and like, they did all this crazy whatever. And I was looking at it, I was like, that's really expensive. Who would pay for that? It just, like, that's what didn't make sense to me at the time is like, who would have that much compute that they could spend on hugging face? And so the fact that it's a frontier lab with one of their training runs that got away like that, that actually matches, right?
- [24:45]
A
Like, it feels in some ways a great story and also totally unsurprising.
- [24:52]
B
And
- [24:54]
A
I came across another tweet where they Vercel, which is a company that hosts AI models and runs them, I Think it was talking about how much it would cost to assess a code base for security vulnerabilities, which I assume is pretty much the same as it would be to do, you know, assess it for security vulnerabilities if you wanted to hack it. And they were talking about for a
- [25:19]
B
company sample at random, right?
- [25:21]
A
Yeah. Using OpenAI's top available model. It would be six figures, they said. And so for a state that seems like, oh yeah, that's something that is totally feasible. And especially if we were going to, you know, it was some underpinning software, we could amortize that over many different hacks. Right.
- [25:43]
B
So like a state saying like Xi Jinping's phone is so important to us that we're willing to spend up to $50 on a ChatGPT Max license. No, wait, a hundred dollars on a CHAT GPT max license and see what happens. That's just not the case. Right. They get budget that they invest to do these things. And so that sort of cost is just a cost of doing business that you would accept. A six figure price tag is just, that's what it costs to do this, to use this tool. We've determined this is the right tool for the job. That's what we have to pay. Like, it's like you have to justify it and someone has to approve it. But it's not like we don't have that money. We can't do this. It's a different set of problems. As opposed to just like, I don't have $100,000 to spend on.
- [26:36]
A
Yeah.
- [26:37]
B
You know, auditing software.
- [26:38]
A
Well, I feel that that is actually the thing that, like, if it comes down to a narrow set of tasks, there are some things that cyber is very good at. And I can envision. Envisage. Envisage cyber being very good at both breaking into and defending networks because they're concrete tasks. There are yes, no answers. You can know if you've patched something and know if you've not. You can, you haven't by counting the
- [27:05]
B
number of computers you have. There is a finite number of computers that you have. Like asset management is a thing that you can do because there's a quantifiable solution. Like there's a quantifiable answer of the number of assets that you do have. Yeah, yeah, yeah. So, yeah, absolutely.
- [27:21]
A
Now I think attacking and defending networks are very narrow jobs. Like the vast majority of humanity does not give a rat's ass about those jobs.
- [27:32]
B
The vast majority of humanity is wrong.
- [27:37]
A
Even in the field of cyber security, the Jobs are much broader than just attack and defend narrowly. So for a whole state, when you come to expressing cyber power, there's so many different ways to express it and there's so many different jobs. It does come down to people still.
- [27:57]
B
Yeah. So like, in a way, this does go back to a post I wrote in 2000. Basically I wrote it in 2016 or 17, I think. 17, which was like how to make an apt. And Right. So it was like you need hackers. Right. But then you need sys, administrators and then you need a manager and then you need an admin and then you need like all this stuff. And so it turns out that your 15 people that you need at a minimum, four of them are doing hacking and the rest of them are just doing all of the other things that you need. And so I think if you replace the hackers, then you still now have 11 people doing cyber, even though none of them are hands on keyboard anymore. And I would say that that is going to hold true because AI can sort of help automate some part of those jobs, but it can't replace the people that are involved. And I don't know that it ever will.
- [29:02]
A
So I think that everything we've said has kind of just reinforced what I thought at the beginning and I wasn't sure of that. Yes. Particularly when it comes to cyber power, that is people. And I think that will be kind of enduring because. Right. And it's because there's such a wide scope of things that a state wants to do that you'll need people. But when it comes to cyber, is people like a more narrow. Right on the. Well, and I think maybe you are using it in a broad sense, but I'm using it in a narrow sense that I'm not convinced that AI will not change the at what exactly it is. And I think that's because if you've got a narrow conception of what cyber is, AI is actually getting very, very good at that. And so I guess it comes down to whether what you think cyber really is.
- [29:57]
B
Yeah. So cyber is what we make of it and that might be people.
- [30:03]
A
Thanks Crack.
- [30:04]
B
Thanks to.