Loading summary
Transcript65 lines
- [00:05]
A
Very happy to be speaking to Alex Stamos on the next episode of Frankly Fukuyama. If you do like this series, please like and subscribe to it. So Alex was the head of security at Facebook back in 2016 when all the Russian hacking happened. He's been with us here at Stanford at our Cyber Policy center for a number of years. He teaches a really popular course called Hack Lab where he teaches white hat hackers how to get into computers, among other things. Yes, yes, but really what I wanted to talk about today was the whole question of AI and how that's going to affect computer security. It seems to me that if you listen to people like Jeffrey Hinton, you know, he says that there are these existential threats out there that really may affect the future of humanity as a whole. Elon Musk I mean, there are a lot of people that are into this and there's also another view that says that the threats are shorter term. It's basically not computers doing bad things to people, but people doing bad things to other people using computers. Since you are really one of the foremost experts on computer security, how do you think about the impact that AI is going to have on our well being?
- [01:20]
B
Yeah, I'm much more of a short term worried about humans person. I'm not an existential threat guy. AI Doomer is the shorthand. I'm not a doomer one, I think. You know, over the last year or so, people have become a lot more skeptical of the idea of AGI, artificial general intelligence that a number of experts have either put the idea of AGI in the post 2030. Even further out, a number of people have made the argument that LLMs, the current set of technologies that we're pursuing, will never get us to AGI, that we're going to need completely different types of models to get us there, that LLMs don't have true understanding of the world in a way that human beings do and will never have an understanding of the world that will rival human beings. And that's a good thing. If you're worried about AI taking over the world. Having a real 3D model of how things interact would be necessary to have the Terminator future.
- [02:28]
A
The LLM knows that an apple fell on Newton's head, but it doesn't actually understand gravity.
- [02:34]
B
That's right. The example I often use is like my golden doodle that our family has. She can catch a Frisbee. She doesn't bring it back because she's mostly poodle. So she's not a full golden retrieval. But she can catch a Frisbee. She can't write poetry, right? LMS can write poetry, but pretty much it's impossible to train an LM if you gave it control of a robot to get your Frisbee, right. Language is the hardest thing human beings do. It's hard. One of the last things we evolved. It's one of the highest functions of our brains. And we jumped with LLMs to kind of the hardest things humans do. And then we've kind of backed into it doing these simple things. And so you have all these experiments where folks give control of robots to LLMs and then they're really bad at doing things like navigating simple spaces and such. And what folks are realizing is that like LMS just don't understand the world. They understand like the world through the written word. It's like a novelist's view of the world, but they don't have an actual physical understanding. So people are talking about physical models and physics based models and vision models and other things. Just like, you know, a dog and other animals have a much more innate sense of space and an innate sense of the 3D world that is much simpler and kind of a base brain view that doesn't require that kind of high level. And there's been a bunch of research into simpler models that actually beat the more complicated models and things like solving Sudoku or solving problems that are practical problems, but they can't write poetry, which is a whole set of research that is really effective and I think perhaps gets us to AGI. And perhaps what you end up doing is like the human brain. You end up with a bunch of different models put together. If you want to build an AI system that like a human has the capability to interact in 3D world and write poetry and have instincts and have all these things. But the other thing about AI is it doesn't want anything, right? Like an LLM is just a. It's a bunch of tensors. It's a box of numbers. The most complicated AI model just sits there, right until you do something. Now a human being can prompt it to do something terrible, can give it a system prompt, can give it a goal to go do something, and then can take the output and plug the output into a system that has the ability to do things. And so, yes, and we'll talk about, I'm sure, the bad things you could do with AI. But on its own, AI has no wants, it has no instincts, it has not evolved like we have to have, you know, basic mammalian desires to intention, intentionality, doesn't want to reproduce, it doesn't want to make more of itself. It doesn't want to kill or eat or make more AIs. And so, like, that's the other thing that I think is missing here, that like, if AI was going to do something bad, it's because somebody made it to want to do it. Even in the end, if we ended up with like a Terminator situation, it's because somebody decided to create AI and then to give it the desire to do something terrible.
- [05:39]
A
So it always requires a human in the loop to actually give it that.
- [05:42]
B
At some point a human had to decide, I am going to one, train AI with these capabilities and then two, set it on this path to have to give it the start of like this is what you're going to do and then to plug it into a bunch of capabilities to give it the capability to do better.
- [05:57]
A
Yeah. I think another thing I've heard you say is that LLMs can't be creative in the sense of thinking of things that are genuine, that they've never processed previously.
- [06:06]
B
That's right. And LLMs are not. They simulate creativity. And you know, when I talk about it, you heard me, I was talking to our students here at Stanford about the use of LLMs in a security context. And the warning I give security people is, you know, LLMs are being used in all kinds of defensive purposes. And that's really good. I mean, there's a lot of useful purposes that LLMs can be used for, cybersecurity. But you have to be careful in that they're not going to foresee new attacks and they're not. They can emulate and they can be trained on. We have done all this defensive stuff in the past. Here is a set of attacks that have been seen, but what they're not going to do is, is think, well, what might a human do in the future? What creative things could happen? That's never going to happen. We actually have some interesting examples from the trust and safety world, right, in that companies that have over pivoted into using AI to do trust and safety work end up being outsmarted by human bad guys who end up just changing their approach a little bit and then getting around the AI. Because the AI has been trained on everything that's happened before. And it doesn't take that. You can end up having very quick, very broad, very cheap protections that operate at scale that are also very brittle because it only takes a little bit of creativity to get around them.
- [07:22]
A
Okay, well that's, that's Reassuring in many ways.
- [07:25]
B
Yes.
- [07:26]
A
But as a security expert, you are worried about what's going to happen in the near term.
- [07:31]
B
Yes.
- [07:31]
A
With bad people using these technologies for bad purposes.
- [07:35]
B
Yeah, we're already starting to see, we're right at the start of bad guys starting to use AI to make their attacks more effective. Just over the last six months, we've seen a real increase in the use of AI for offensive purposes. This started with some experimentation around the use of AI to write code. We saw some parts of the kill chain. So, you know, in cyber we have this idea of the kill chain that we stole from the military. Now when the military says kill chain, they mean to actually kill things. In the cyber world, when we say kill chain, it's the steps you take to break into a computer network and then to have an effect on a target. So that might be to steal data, it might be to shut a computer down, it might be to implant a backdoor so you can come back later. And so for those stepping into kill chain, they're a little bit different depending on exactly what you're doing, but it's often you need to do reconnaissance, you need to map out a network, you need to find a way in, you need to build an exploit to do the attack. Then you exploit a network, you deliver a payload. That payload gets you command and control, so it gets you control of the system. Then you might explore, once you get an initial payload delivered, you might explore the network. It's called East West Movement. Bounce onto a couple other computers you might need to escalate your privilege. So maybe you're on like a computer that you don't really care about, but that gets you initial foothold. And then you can get higher and higher privileges until you get into a really powerful computer. And that's a computer you do care about. And so these are the steps of a kill chain. And so what we've seen over the last six months to a year is attackers have systematically figured out how to, how to use AI in more and more parts of the kill chain. And this means one, that as they do that, it means that they can trade. You know, what AI allows you to do is, you know, trade human beings for computers. So they need fewer human attackers. This is useful for all kinds of attackers, both state sponsored actors and for financially motivated actors, where, you know, every kind of actor does not have enough people. Nobody has enough skilled attackers. So if you have a smaller number of skilled hackers, it's great to have them get a force multiplier by them training Computers to do parts of their job. The second is then it allows them to do a lot of work in parallel, right? So that you can have a smaller number of humans then supervise a bunch of AI agents doing their work at the same time. And so as they do, take each part of that kill chain and they automate it. That also means that they can move very quickly. And a friend of mine, Rob Joyce, who worked at the nsa, he likes to say speed kills, Right. From a defender's perspective, if an attacker is really fast, that's a real problem for you, right? Because a defender, what you try to do for all those parts of the kill chain, your goal as a defender is you try to set out tripwires for every part of that kill chain, right? So you have a tripwire for the initial exploitation, you have a tripwire for the command and control, right. So you like your firewall. You try to listen for command and control, record every DNS request from your network. That's a different kind of command and control. You look for east west movement, look for the payload being dropped onto your computer. You look for different kinds of malware. Those are all tripwires you have in your network. And once the attacker hits one of those tripwires, then hopefully it sets off an alert and you are able to respond fast enough to stop before they get all the way to the end of the kill chain. If the attacker is able to automate all those steps and have AI run through it, then even if they trip one of the tripwires, if it takes you time, let's say you still have humans in the loop as a defender. So it's 3am and somebody has to get on their computer and they're like, you know, their phone starts beeping, wake them up, and they're like, oh, no, I've got an alert. So they get their computer and they open up the computer and they do their fingerprint and they have to two factor. And then they have to get in like, oh, no, okay, yeah, this looks real. And they have to get into another system, and from that system, they have to read some details and like, oh, okay, yeah, I have to get in here. And they go and they shut down the system, and that takes 15 minutes. If the entire kill chain's on AI in those 15 minutes, the whole thing could be over.
- [11:32]
A
Yeah, right. But can't you delegate to an AI agent.
- [11:35]
B
Yes.
- [11:35]
A
That what you just described the human being doing.
- [11:38]
B
Right. And so that's what we have to do defensively. I think this is the race that's on now is that attackers are now automating all the parts of the kill chain, and so defenders have to do that too. The problem for defenders is we have bosses. We have Sarbanes Oxley letters. We have. We work for corporations that have to live up to rules. We have auditors. Like, it's much more dangerous for us to do things like hook up all the parts of our network to an AI system that can just shut parts of it down at any point where. If it feels bad versus attackers, where a lot of them just don't care. Now some of them do, right? Like, if you're the Russian svr, if you're like the tip of the spear of the Russian foreign intelligence service, then those guys are very careful. They don't want to get caught. They're probably not automating a lot of their kill chains. But if you're a Russian ransomware actor, those guys hack thousands of targets with the goal of getting a dozen of them to pay a ransom. They don't care. They'll automate everything, right? And so if they can go from 10 ransoms to 50 ransoms, they've got five times as much money. So, sure, from their perspective, it's great. They'll automate even if the AI screws up in a bunch of times. That's fantastic for them. And so, yeah, for them, automation makes a lot of sense. And so this is, I think the big race right now is for defenders to get automation in place that they feel comfortable enough turning on to do the defensive steps necessary. And so you have to have automation within constraints that also the automation. Because, like, being defenders, like, I'm sure you've seen Bridge over the River Kwai. I have to explain this metaphor to our students. I talk about feeling. I'm sure. I'm sure you feel old teaching our joint students, right? But, like, I'm at that point where it's like you make movie references and their faces are just blank. So, like, you know, at the end, right, he. Spoiler alert, right? He blows up the bridge. You know, he goes and he blows it up. They take this beautiful bridge that they build and they rig it with explosives. That's what it's like to be a security person, right? You have this beautiful infrastructure, and you have to rig it with explosives. Because in the end, to stop an attack, you usually have to break the infrastructure in some way, right? You have to cut off a firewall, you have to drop Internet transit, You have to shut down servers, shut down kubernetes, containers often like in a real. I've done a lot of incident response and sometimes I tell the CEO it's time to shut the company down. It's time to turn off all Internet access to your company. You need to send all your employees home now. Right? Like that is the only way we can contain this breach is we have to shut down all Internet access to your company. That. That has happened multiple times. I have told the CEO that it is a bad thing to tell the CEO, but sometimes the only way for them to retain any enterprise value.
- [14:12]
A
Right, and you're saying that you don't want an agent to do this automatically.
- [14:16]
B
That is a hard thing to give an AI agent is the power to do all of. I think you probably don't have to give an AI agent the ability to turn off all Internet transit, but you probably have to give it the ability to turn off accounts, to at least suspend accounts in active directory, to turn off individual containers and individual virtual machines to create firewall rules. And that can be really risky in a corporate environment. It's extremely risky in a production environment. And so this is the race we're in right now as the attackers automate their kill chain for defenders to automate their defensive systems.
- [14:50]
A
So in thinking about AI, it seems to me the real problem is agentic AI because. So I don't know anything about AI itself, but I do know something about hierarchies, human hierarchies. And you know, in every human organization, you delegate authority to some lower level of the organization. And it's almost inevitable that at some point you delegate too much authority because, you know, the people at the lower levels have the ability to act quickly. They can sense the environment, they have more skill and knowledge and so forth. And that's really what gets the whole organization into trouble. It seems to me almost inevitable that this is going to happen. I mean, you've just given an example of that, right? That the automated defensive systems will be much faster and more effective in certain ways. But so there'll be a constant pressure to, you know, allow more of that to be automated. And that's what then gives away too much, you know, authority to these machines.
- [15:46]
B
I mean, the, the delegation problem is a serious problem overall in that we just don't have good ways of delegating authority to agents overall in it. This is actually a fundamental issue we've got is the how do we authenticate AI agents? How do we create, like a temporary constrained delegation for them? We don't know. Like, if you want to tell an agent, go, you are allowed to be me for the purposes of shopping on Amazon for. And you can buy up to, you know, I want you to buy a gift for my wife and it can cost up to $50. Go buy something nice for her. There is no mechanism for you to do that. It will just have your credit card and it can go buck wild. Right? Like there's just no mechanism to delegate its authority within reasonable constraints.
- [16:33]
A
Right? Yeah, yeah.
- [16:35]
B
And it's the same thing on Corporate Net works. There's no good way to say you can go interact with my email for the purposes of sending an email for a very constrained purpose. It could just like basically write an email to tell your boss to go F himself. Right? Like, yeah, you know, m. You know, hey, President Levin, this is what I think about, you know how Stanford's going, right. Love Professor Fukuyama. Right. Like the agent has the ability, it's very hard to build those constraints. Right. We just don't have the semantics for it. We don't have the models for it. We don't run our agents within Sandbox to do that. A bunch of people were proposed mechanisms for that. They're all proprietary. So this is the other thing is that everybody's like, oh yeah, I can solve that for you. If you opt into my sandbox and it's my corporate sandbox and you can only run within my sandbox, yada yada, there's almost no mechanisms that people want to do that are at all open.
- [17:26]
A
And I take it our colleague at the Stanford Law School, Mark Lemle, is working on all the liability issues where you delegate authority that you shouldn't and then somebody gets mad and sues you.
- [17:36]
B
Oh yeah. I mean, once again the lawyers are going to do great out of this. I mean, not the young lawyers because none of them are going to have jobs, but the older lawyers like Mark, who is the wizard in my Dungeons and Dragons group, which here's just a little tip. If you're going to have a Dungeons and Dragons group, do not have any law professors in it because every session turns into a multiple level appellate process. But yes, Mark is working on that. And it turns out the legal issues here are huge because the these AI agents are just going to go out and do a bunch of stuff without you as the human knowing and signing contracts for you and such. I mean, we already have this DocuSign stuff. I sold a company a couple years ago, I sold a company over a decade ago and we had this huge paper signing ceremony and the CEO of the company, we're selling to told me like, son, you should go buy like a nice pen and that'll be like a thing that you remember. And I did like I bought like a nice Mont Blanc and I used it to sign and it took us like an hour like because they walked us through signing all the papers. It was like this big thing.
- [18:36]
A
Yeah.
- [18:37]
B
It was like, it felt really important. And then the second time I sold the company, we did it on DocuSign. Yeah, it was much less. It was very anticlimactic even though it's like for a larger amount of money. And it was like, wow, that was, that was it.
- [18:49]
A
Never understood how DocuSign works because anybody can push a button.
- [18:52]
B
Yeah. I mean apparently people have litigated it, but it does seem very easy to fake. I mean, yes, it could be. You could do it through cross site scripting, you could do it through malware. I mean, do not call me as your expert witness to get out of your DocuSign contract. Somebody. I mean people are going to be getting out of those contracts for sure. But anyway, yes. I mean the AI stuff, people are going to be having AI sign their contracts and say they didn't authorize it.
- [19:13]
A
So let's talk a little bit about the international dimensions of this. Give me an overview of who's really good at this stuff, who we have to worry about in terms of international actors and who are less worried about.
- [19:25]
B
Well, so this is, this is fascinating. Obviously. Okay, so when we talk about actors who are doing this, the, the report that just came out, Anthropic threw like a huge grenade into this whole world saying that they caught a group that they associated to Chinese intelligence using their systems to automate all parts of the kill chain using Claude code. Now this was a huge deal. There's a bunch of kind of conspiracy theories. Anthropic is overblowing this. I will say I wish Anthropic had released more data. Like if you're going to do these threat intelligence reports, you really should have like more IOCs out there and such. I do.
- [20:03]
A
What's an IOC?
- [20:04]
B
I'm sorry, indicator compromise. So like the technical details, they should have released more of like the, the raw, raw details. That being said, I do, I know some of the people in Anthropic who work on this. I trust that they're telling the truth. Anthropic, somewhat controversial because they are, I think the most ethical of the foundation model companies and they have called for more aggressive regulation in this space. And so some people are conspiracy minded that Anthropic is lying about this, but I do think it is true. What we have seen is it is not crazy to think that a PRC entity would do this. In fact, there is a open source tool that you can download from a PRC group that is not that much less advanced than what Anthropic demonstrated that uses Deep seq, which is an open source Chinese LLM that is trained to use Kali. Kali is the virtual machine that I actually use to teach the class here, which is a Linux distribution that has a bunch of ATTCK tools built into it that will automatically do a bunch of hacking for you. This automatic this tool that this Chinese group built that you can talk to it in either English and Chinese and basically ask it, hey, go hack this network for me and it will go do a bunch of the hacking automatically for you. It's super cool.
- [21:15]
A
So Kali is a Linux distro that I can just download and put on my computer.
- [21:19]
B
You can come to my class and I'll teach you how to do it. Frank, you still haven't taken the class. Yeah, you got to do that. If you're going to make the master students do it, you should do it yourself. But yes, you can download it yourself or we run it here on virtual machines here in our tech lab. But yes, Kali doesn't come with the LLM. What this Chinese group did was they take the Kali virtual machine and then they trained an LLM on all of the instructions on how to use all the tools in Kali and then on a bunch of attack patterns that they trained on the open Internet and then hooked it up with MCP servers to use all the tools. And then you can basically ask it, hey, go do attacks for me. And so it'll go run all the tools in the proper order to go do it. So not as advanced as the ATTCK Anthropic talked about on their systems. The interesting thing here about what Anthropic talked about is why if anthropic and OpenAI OpenAI has seen some of this kind of stuff, but not nothing as as advanced as what anthropic demonstrated. LLMs that are being created by Deep SEQ I QEM by the Chinese labs are 70%, 80% as good as the the closed models from Anthropic and OpenAI from some of the metrics, maybe not that good, but they're pretty good. Right? You can go get those models and then just run them on your own hardware. Right? If you do that, you are not leaving logs at Anthropic or OpenAI for the FBI to get for the NSA and such. And so if they are seeing that kind of activity, then there is a ton of activity happening using the open source models, right? Because the other thing you can do is you can't take Anthropic's models, OpenAI's models, and then really retrain them to do hacking. Like they're, they have a bunch of protections in place to try to keep them now you can trick them, you can, you can say, oh, this is authorized and such. And because of what happened, I think Anthropic has made it harder to try to use their tools to do hacking. But the open source Chinese models, like you don't have to do anything to get them to help you to do hacking. And then you can intentionally train them just like this, this Chinese toolkit to, you can take a bunch of stuff and add these checkpoints and then use rags and such and then feed them a bunch of data. And so if I was building these kinds of things, I would just be using the open source models. They're quite good. Like Deepseek R1, the latest kind of chain of thought model from Deepseek is quite good.
- [23:40]
A
Right.
- [23:41]
B
And you know, if you have the hardware, then I would just be using that. So yes, of, of, of the groups I'd be worried about, I think China is by far the first Chinese. China is the only adversary country that is really has their own labs that are equivalent to the US labs, right. And any, you know, all this stuff about the US being way ahead of China in AI is foolish, right? Like the PRC has had a plan for decades to catch up to the United States in fundamental sciences, including computer science. They have been sending students here, including to Stanford for years and that has been affected. Now a bunch of those students stay here, but a bunch of them go back and then they go and they teach there and then they create the next generation. Of all the current fields of computer science, it is one of the most academic. Therefore it is something that is published very broadly. That has changed a little bit in the last couple of years where a lot of the cutting edge stuff has stopped being published. Right? But until, like, until GPT3, almost all of the really cutting edge discoveries were published openly after GPT 3 and the commercialization people stopped. But that's, that was only three years ago, right? So up to that point, everything you needed was public, right? And you know, like our undergrads here build toy LLMs. Our graduate students are doing, you know, cutting edge research. A bunch of them are Chinese nationals And a decent percentage of them will go teach it. Shenhou.
- [25:03]
A
Right, does so, as in the case of nuclear technology, or it seems to me you're saying that unlike nuclear technology, you don't necessarily need to be a state level actor to really be at the cutting edge of this technology.
- [25:15]
B
No, not at all. Because like with nuclear technology, we couldn't control the physics. Right. Like every country in the world very quickly knew how to build an implosion bomb. Right. Like controlling the knowledge of how the physics worked was effectively impossible. What we controlled was the uranium. Well, controlling the compute is effectively impossible. That is something that the Biden administration's controls demonstrated, partially because you don't actually have to have the GPUs in your country for this to work. There's a reason why Singapore and the UAE are on the list of the top importers of Nvidia GPUs. It is not because the UAE is full of AI researchers. It is because they are effectively exporting that compute to China. Because the actual thing you're exporting after you do all that work, it fits on a single thumb drive. So you can do all the compute in Singapore, you can do all the compute in the UAE or Dubai using their cheap subsidized power, and then you can zip the final results in seconds over fiber optic cables back to China.
- [26:17]
A
So if this is something that individuals can really master, it seems to me the threat level then goes way up. Because you're living in a world where there are a lot of people with bad intentions. In the nuclear world, you really had to be a country with bad intentions. Now it seems to me you're saying that potentially a lot of individuals bad with bad intentions can make use of this technology in a very sophisticated way.
- [26:41]
B
I mean, as of right now, you can go on hugging face and you can download retrained versions of Deep Seek that will write you exploit code. Right. You can get retrained versions of Quen that will write you malware. So, yeah, I mean, you could never do that with nuclear weapons. You could never download yellow cake. So I think the, you know, the idea that we can apply something like the nuclear number proliferation, you know, the idea that the CSAC model from the second floor here or third floor is going to apply at all to this world is just foolish. Yeah.
- [27:16]
A
So in terms of the criminal world, what's the cutting edge there?
- [27:20]
B
Yeah, I mean, so the first, the first wave was AI to make phishing better. Right. So a lot of the protections against spam and phishing were based upon the idea that if you were sending out a message to fool a lot of people, you'd send one message, it would be the same and you send two, a thousand people. Now because of LLMs, you can send a thousand unique messages to a thousand people. So the first time people did that, at least it was the same. Like you didn't know who those thousand people were. You were just sending a mail enhancement message that used different languages to try to beat the filter. Now I think what we're going to end up having is you're going to have scammers who are able to do scams at scale where the interactivity with individuals will be based upon bots. And that's going to get pretty scary, right? Like you know that this is Sally, that she's a 63 year old grandmother. And the work that used to be done by a Nigerian scammer who's actually sitting in Lagos, who had to think about Sally and how to trick her can be done by an LLM whose English is better than that scammer and who has information stolen about her from, you know, a data breach and knows what her grandchild's name is and is able to build a lure that's perfect. So I think like that's going to get scary. The other thing we have now is we have the face replacements, we have voice replacements. And so at the high end, what we're, what you end up having is you have people coming in and doing fake video conferences, fake fake phone calls and doing wire transfer scams for enterprises. So if you want to rip off $250,000, $500,000, people are making phone calls or video conference calls into account payable teams, into accounting teams pretending to be executives. The CFO saying, oh yes, we need to make this payment, we make this w transfer or going to companies and saying, I'm your, I'm this large vendor. We've made a change to our payment structure. And this is a old scam. It used to be all on paper. Because it's a significant scam, people then move to, okay, let's have a phone call to verify. And so now you can emulate somebody's voice, emulate somebody's face. The other scam that's going on now is the North Korean worker problem. Post Covid, lots of companies decided there's certain classes of people they can hire remotely to the point of where you might have employees that have never been met by a human being in the company, right, that you interview them remotely, you hire them remotely, you ship them A laptop. They do all their work. When you eventually fire them, you ship their laptop back. Okay, that works out fine until the fact that the North Koreans are taking advantage of this. Where they apply for a job, they use an American mule. So that American has a real Social Security number. They have a real American bank account. They have a real American mailing address. And so they apply with the. With the identity of that American. But the interview was taken by a North Korean who was actually skilled at that job, whether it's a programming job or something. They used face replacement to pretend to be American. They used voice replacement to be able to do, you know, a perfect American accent. They do well in the interview because these people are actually good at the job. They get the job. The laptop gets shipped to that American, the American opens it up, they install a piece of remote access tool, and then somebody in North Korea actually does the job. The FBI has raided these people's apartments. And what you'll have is you'll have some cocktail waitress in Las Vegas who will have 20 laptops open, all of them being controlled remotely by 20 different people in Pyongyang. And then she's collecting 20 paychecks, and she sends 90% of each one of them back to North Korea via cryptocurrency.
- [30:49]
A
This is a real thing that happened.
- [30:50]
B
Oh, it's a huge thing. It is worth now billions and billions of dollars of current hard currency transfer back to North Koreans. North Korea. Now it looks like the two largest sources now of their. Them getting money is one, North Korea is the largest thief of cryptocurrency in the world by far. Nobody else is even close. And then two is the scam. And then they bust that cocktail waitress. She goes to federal prison. And then those 20 North Korean workers have to get 20 new jobs. And the cocktail waitress is like, oh, I had no idea they're North Koreans or whatever. Who knows? This is like those work from home scams. You got recruited to work from home. And then she got pulled in. She knew something was wrong. She probably didn't know they're North Korean, but she knew, obviously, she was getting paid to run these laptops.
- [31:32]
A
Is that the same thing that's going on in these kind of slave farms in Cambodia? They've been in the news a lot lately that people are literally kidnapped and forced to work.
- [31:42]
B
It's possible, yeah. I mean, there's a lot. I mean, certainly the North Korean workers, the reports are they're very hard workers. And it's obviously because they're slaves, right? Like. Like People are getting actually really good output from them. Right? It's graphic design, it's programming, it's data entry. It's things that people can do without a lot of phone calls, without a lot of interaction, you know, asynchronously across time zones. Although they'll work during American, you know, they're up in the middle of the night, their time. The Cambodian stuff. Those are people who are working for, like, Chinese triads, but they work in Cambodia, so they're outside of the reach of Chinese law enforcement, is my understanding.
- [32:16]
A
Yeah, so you suggested that, like the Russian gru, you know, when they get into this, they're more cautious than your typical criminal. Why is that?
- [32:25]
B
Right. So, I mean, you're certainly going to see state actors use AI, but they're going to be more careful. And it's because if you are a state actor, depending on who you are, you might not. You might care about getting caught. Now, the GRU is an interesting example because there's a lot of examples of the GRU not caring. Right? So, you know, as you know, but maybe your listeners don't. There's three major intelligence agencies in the Russian Federation, right? There's the SVR and fsb. Those are the two descendants of the kgb. I always found this interesting. I'd love to talk to Mike McFall about this. It's interesting to me that they threw away the KGB brand, like the Belarusians kept kgb. It's like that sucker's got a huge amount of brand equity, right? Like, what's, like, what's the recognition rate of the term KGB? It's 95%. Who knows what the FSB is, right? But like, the Belarusians still have the KGB because it's like, it's terrifying. Everybody's terrified. Who's afraid of the fsb? Be. I mean, people are now, but they weren't in, like, the 90s anyway. So the KGB breaks up and the first director, like the Foreign Intelligence Service becomes fsb. I'm sorry, becomes the svr. And then domestic intelligence and the near abroad becomes the fsb. And so the svr, and then is the. The svr, they have the best hackers in Russia, right? And then the GRU is military intelligence, so they work for the Kremlin. Gr. You were kind of the thugs of the. Of the Russian state hackers. They're the bulls in the china shop. They often don't care about getting caught. And then the FSB is somewhere in the middle. They have hackers who are very careful. They have hackers who don't really care. They also use a lot of contractors, right? So a lot of FSB work is being done by Russians and people from related countries who are doing work for their uncle in the fsb. When I got to Yahoo, Yahoo had been pre breached before I got there by a group of guys who were working for fsb. A guy named Alexei Balon ran that team. He's actually not Russian, he's Latvian, who had gotten caught by the fsb. And then I guess, you know, given the choice, like you could live in that concrete box or you can work for us. I'm guessing most people take option B, right? And that the FSB recruits a lot of people that way. Right? So the svr, they are famous for not getting caught, right? Because they're an intelligence gathering operation. So a lot of their work, if they get caught, it's useless. They, for example, are famous for the SolarWinds hack where they spent like nine months working their way slowly into SolarWinds, very carefully, very quietly sneaking through the network. When they implanted the backdoor into SolarWinds, they did so with brand new malware on a build server that was in the kernel only. It decrypted itself just for the moment of the build. It patched the software in memory only it never touched disk. It was incredible. Incredible. It's gorgeous. Really careful. The GRU is just like blow stuff away because, you know, they're a defense support, they're a lot about breaking things. If they do intelligence gathering, it's often for a very short term purpose or they're doing like disinformation work, right? So you know, they famously broke. They're the ones who broke into Podesta's email and then they stole that and then they released it, right? So it's like they don't care that people know, people know they know, people know we know they know, like they're fine with it. And so if the SVR is going to use AI, it's going to be for things like exploit development. It's going to be things for like to create malware. It is not going to be to automate their kill chain because they're willing to spend nine months to carefully do stuff. I can see the GRU doing it right now. The ransomware groups, again, those guys operate at scale. They like to hack a thousand things and then traditionally they've had to come back after they hack into a thousand companies and then pick and choose which ones they actually ransom. So if they can automate that process, that's good for Them. The other problem that the ransomware groups have is if you have an affiliate group of 15 or 20 guys working together, one of those guys might be a cop, one of them might be a Australian Signals Directorate undercover agent, one of them might be dumb and go to Cyprus to go to the beach and get picked up on Interpol warrants and they get turned against you, right? So for them, human beings are a liability. And so if you can replace those guys with bots, you're much safer. And so for them I think AI is also a good move because if you can go from 20 guys to two or three using AI, then you have a much safer, a much safer, tighter group.
- [36:41]
A
So maybe just to wrap up in the future, are there new actors that people are not aware of that we ought to be careful about or there new ways and just not companies or big organizations, but just individuals. What do they have to worry about?
- [36:57]
B
Well, I mean one of the things that's gonna be interesting to see is AI is going to really improve the game of lots of people because it's, it's already allowing attackers to find vulnerabilities and write exploit code that they were never able to do before. So like those really high end guys like svr, the Ministry of State Security in China, you know, the us, rnsa, think of the real high end guys, right? So it's a US and the five eyes. Us, Canada, Australia, New Zealand, uk Right. China, Russia, Israel, France, Germany, couple other western nations. Those are like the real cyber powers where you have people who are like writing custom exploit code, doing real zero day development, doing really hot stuff. There's a whole tier below that where it's actually really rare to see brand new exploits used in the wild. So like even Iran they're very active, but they don't use a lot of zero day exploration exploits traditionally because they don't have a ton of people who can write this beautiful new exploit code that is about to change. Because AI both allows you to find new vulnerabilities, but it also lets you write exploit code and test exploit code I think much more easily than you could before. And so it's going to be fascinating to see what happens when you have all of these groups take a step up. So like India and Pakistan, traditionally not a ton of new exploits. I mean it happens, but not a ton of Iran, maybe like South American country, but also like, you know, individual groups, both the ransomware groups, but maybe hacktivist groups. These people have always been doing hacking, but they've been using tools that they Find they've been using exploits that they get that kind of trickle down. You know, you'll see, like the superpowers attack each other, right? You'll see, like, Stuxnet gets used, or the US loses EternalBlue, probably because the SVR steals it and releases it, and then Eternal Blue gets used all the time in all these hacks. Well, if that kind of capability is in the hands of everybody with access to AI, that's a really scary future. And so I think, like, we're going to have to, as defenders, really adjust to the possibility that a much broader set of people have the ability to write really good exploit code and to find new vulnerabilities really quickly. So, one, as defenders, we're going to have to write better code. We're going to have to find vulnerabilities really fast. We're going to have to patch much more quickly, because the number of companies that have actually had to deal with zero Day is. Is. Is really quite small. It's like defense industrial base, oil and gas, banks, big tech, pretty much IT government. Most companies don't actually have to worry about that now. You might see all companies have to play at that level. That's a, I think, a really scary possible future.
- [39:35]
A
Yeah, Yeah. I remember when I was young, I read a science fiction story about some aliens that came to Earth. They had a machine, it was very simple and easy to reproduce. You just pointed at anything made of metal, and the metal basically turns to putty. And so they leave this machine and they fly back to their home planet. They come back 50 years later, and the whole world has basically fallen into complete chaos because any teenager can melt the Brooklyn Bridge, you know, and.
- [40:02]
B
Oh, wow.
- [40:02]
A
So it seems like we may be moving into some cyber equivalent of that.
- [40:07]
B
You think an alien just left the transformer?
- [40:10]
A
Well, maybe it wasn't an alien, but once the technology, you know, becomes that accessible, there's. There's really lots of bad actors.
- [40:17]
B
I mean, the good thing here is AI also helps us write better code. It helps us find these bugs and fix them. So, I mean, this is what the company I'm at now, Corridor, was started by two of our alumni, right? So I joined this company with two of the students I worked with here at Sanford, Jack Cable and Ashwin Ramaswamy. So, you know, so, like, the upside here is AI does help us write better code. It helps us refactor code. It helps defenders find bugs. So it's not all bad. We just have to have the will and the courage to try to move faster than the bad guys.
- [40:48]
A
And it's an ongoing arms race. Or it will be an ongoing arms race.
- [40:51]
B
It will be. And we have to, as defenders, invest in actually fixing things. Like if you look at the salt typhoon attacks, like the, the big attacks against the. The telephone companies, you know, these really never got properly investigated. The CSRB was shut down by the Trump administration. We never got the final report. My understanding is a bunch of the vulnerabilities are exploited. The patches exist. They just weren't applied. Like, we can't allow that anymore. Right. Like, we actually have to go fix things. And if it means, like, we have to have some downtime, we have to spend the money, we have to upgrade the switches and the routers, we've got, as a, as a society to decide, we're going to spend that money, we're going to squish the corporate profits a little bit. We're going to hold people to a higher standard.
- [41:35]
A
Yeah. And you as an individual shouldn't delay on doing that patch on your to hear.
- [41:40]
B
No, no, but like, the truth is, is like us in the tech industry, it's on us to make that automatic. I mean, the truth is, is the things we build, if we ship you a device, the patching should be automatic. It should be secure by default. Like, we should not be putting that on normal people.
- [41:55]
A
Okay. Okay, Alex, that's great. That's really informative. So thanks a lot for talking. And we'll have to do it again, you know, in a couple of weeks. The thing will be completely different, right?
- [42:05]
B
Yes. If the AI lets us have a podcast.
- [42:07]
A
Yeah. Okay, good. Thank you. Thank you for listening to the Frankly Fukuyama podcast. If you like this podcast, consider subscribing to American Purpose and my Frankly Fukuyama column at www. Persuasion.commun.