Wide World of Cyber: How state adversaries attack security vendors

Summary

Risky Business Podcast Summary

Title: Wide World of Cyber: How State Adversaries Attack Security Vendors
Host: Patrick Gray
Guests: Alex Stamos (CIO/CISO, Sentinel One) and Steve Stone (SVP of Threat Discovery and Response, Sentinel One)
Release Date: May 9, 2025

---

1. Introduction and Context

In this episode of Risky Business, host Patrick Gray delves into the intricate world of cybersecurity threats targeting security vendors, specifically focusing on state-sponsored adversaries. Despite initial plans to feature Chris Krebs as a guest, unforeseen circumstances led to Alex Stamos and Steve Stone taking center stage to discuss Sentinel One's latest research on advanced persistent threats (APTs) from North Korea and China, as well as ransomware operations.

---

2. North Korean Threat Actors Targeting Sentinel One

Steve Stone opens the discussion by detailing Sentinel One's experience with North Korean actors attempting to infiltrate their organization through job applications.

• Persistent Infiltration Attempts:
  "[Steve Stone, 02:48]: "We've spent probably five months actively working against the North Koreans applying. We've seen almost 370 distinct personas. They've sent in more than 1,000 applications..."

• Sophisticated Tactics: Instead of discarding suspicious resumes immediately, Sentinel One engaged with the applicants to understand their intentions better. This prolonged interaction revealed patterns and strategies employed by the North Koreans, showcasing their persistence and scale.

Patrick Gray highlights the unusual behavior of these applicants who avoid on-camera interviews, possibly adapting to modern surveillance tactics. This behavior contrasts with other applicants who excel in live interviews.

---

3. Evolution and Scale of North Korean TTPs

The conversation shifts to the evolution of North Korean tactics, techniques, and procedures (TTPs).

• Increased Scale:
  "[Steve Stone, 05:55]: "What the North Koreans have now, which they didn't have previously, is scale."

• Adaptation of Old Methods: North Korea continues to use traditional cyber intrusion methods but amplifies them to operate at a massive scale. Their expertise in areas like cryptocurrency theft underscores their adaptability and resourcefulness.

Alex Stamos emphasizes that Sentinel One is specifically targeted due to their significant presence in the crypto industry, making them a valuable target for North Korean intelligence efforts.

---

4. Ransomware Operators and EDR Bypass

The discussion transitions to ransomware groups' strategies, particularly their interactions with Endpoint Detection and Response (EDR) systems.

• EDR Console Exploitation:
  "[Steve Stone, 14:27]: "They effectively are acting like a prospect or a customer... they're trying to steal credentials and log in to consoles."

• Evolving Tactics: Initially, ransomware operators would simply disable EDR systems, but now they employ more nuanced methods, such as altering configurations to hide malicious activities rather than outright disabling protections.

Patrick Gray underscores the prevalence of successful EDR bypasses, questioning how common it is for well-configured EDR systems to be compromised.

Alex Stamos responds by highlighting the vulnerabilities introduced by single sign-on (SSO) systems, where compromising administrative accounts can lead to widespread EDR control.

---

5. Underground Economy and EDR Testing

Sentinel One's research uncovers an underground economy where cybercriminals acquire EDR technologies to test and refine their malware.

• Front Companies for EDR Acquisition:
  "[Steve Stone, 22:48]: "There's a particular ransomware group that's actually spun up a whole front company just to acquire EDR technology in demos and PoCs..."

• Malware Testing and Deployment: These organizations purchase EDR tools under false pretenses to create environments where they can safely test and enhance their malware, ensuring it can bypass various security measures before executing attacks on real targets.

Alex Stamos adds that Sentinel One actively monitors and dismantles such setups by identifying fraudulent accounts and eliminating the associated malware.

---

6. Chinese APT Threats and ShadowPad Research

Shifting focus to Chinese threat actors, the conversation explores the depth and persistence of Chinese APT groups.

• ShadowPad Campaign:
  "[Steve Stone, 38:27]: "ShadowPad... we found 70 plus victims in just that blink of an eye."

• Pervasive Cyber Espionage: Chinese APT groups like Purple Haze and ShadowPad demonstrate extensive efforts to infiltrate and exfiltrate data from targeted organizations. Their operations are characterized by relentless attempts to breach security systems, including compromising IT providers to gain deeper access.

Alex Stamos remarks on the sheer scale of Chinese cyber operations, noting that thousands of offensive operators are dedicated to infiltrating major tech companies like Sentinel One, making it a relentless battle.

---

7. Changing Threat Environment and Tech Stack Responses

The episode addresses the evolving threat landscape and the corresponding shifts needed in cybersecurity strategies and technologies.

• Limitations of Current EDR Solutions:
  "[Alex Stamos, 43:26]: "...you have to detach the instrumentation from the detection."

• Need for Integrated Instrumentation: Traditional security products often handle instrumentation and detection separately, limiting their effectiveness against multi-faceted attacks that span endpoints, cloud environments, and email systems. There is a pressing need to unify these functions to provide comprehensive threat visibility.

• Advanced Machine Learning: Leveraging AI and machine learning can help in aggregating and analyzing data across various domains to identify malicious activities with greater context and accuracy.

Patrick Gray points out gaps in current technology, such as browser-based attacks and the challenges they pose to traditional EDR systems. Alex Stamos concurs, emphasizing that modern execution environments like browsers create "blind spots" that require specialized detection strategies.

---

8. Conclusion

The episode concludes with reflections on the continuous arms race between cybersecurity defenders and sophisticated state-sponsored attackers. Both Alex Stamos and Steve Stone express optimism about law enforcement collaborations and the potential for disrupting cybercriminal operations, albeit acknowledging the persistent and evolving nature of these threats.

Steve Stone summarizes:
"[Steve Stone, 30:15]: "...If you don't know your customer well, you've just accepted a ton of risk. ... Cyber is people."

Alex Stamos reinforces the need for heightened vigilance and adaptable security measures to counteract the relentless efforts of adversaries:
"[Alex Stamos, 34:07]: "...we have to be hardened up against the human intelligence level, too."

---

Notable Quotes

• Steve Stone [02:48]:
  "We've spent probably five months actively working against the North Koreans applying. We've seen almost 370 distinct personas. They've sent in more than 1,000 applications..."

• Steve Stone [05:55]:
  "What the North Koreans have now, which they didn't have previously, is scale."

• Alex Stamos [07:23]:
  "We're a piece of that larger puzzle they want."

• Steve Stone [14:27]:
  "They effectively are acting like a prospect or a customer... they're trying to steal credentials and log in to consoles."

• Alex Stamos [43:26]:
  "You have to detach the instrumentation from the detection."

• Steve Stone [30:15]:
  "...if you don't know your customer well, you've just accepted a ton of risk. ... Cyber is people."

• Alex Stamos [34:07]:
  "...we have to be hardened up against the human intelligence level, too."

---

This episode provides an in-depth analysis of how state-sponsored adversaries, particularly from North Korea and China, are evolving their tactics to target security vendors like Sentinel One. The insights shared by Alex Stamos and Steve Stone underscore the complexity and persistence of these threats, highlighting the need for innovative and integrated cybersecurity strategies.

Transcript

[Patrick Gray]

Foreign. And welcome to another edition of Wide World of Cyber, the podcast we do here at Risky Biz, which is produced in conjunction with Sentinel One, which is the company that also sponsors this series. Now, this is our first one that we're doing of one of these where Chris Krebs is not a guest, because the original idea behind this was that Alex Stamos, who is Sentinel 1 CISO, and Chris Krebs, who was there then, Director of Policy and Intelligence, would join me to chop it up. But, yeah, unfortunately, Chris has resigned from Sentinel One after the US President Donald Trump signed a memo ordering the DOJ to investigate Chris and whatnot. It's all a big mess. So, yeah, we're actually just going to push through without Chris in this episode. But I just wanted to let everyone know, don't worry, Chris will be back on Risky Business at some point soon. He's just a little bit busy right now. So I recorded this podcast remotely. It was a live event being hosted in San Francisco, and I was around the RSA conference. I was supposed to be there, but unfortunately I chose to cancel my trip to the United States with everything that's happening there right now. So we just recorded it remotely. Now, the focus of the conversation is actually very interesting, right? So Sentinel 1 released a bunch of research looking at various APT crews and various foreign threat actors, and mostly the threat actors that they've observed targeting them, right? So there's the North Koreans trying to get their people hired into Sentinel 1. There are Chinese APT crews and even ransomware crews all trying to sort of attack sentinel 1 and. And do bad things. So really, this conversation looks at what these threat actors are doing, generally, in a few cases, what they're doing to Sentinel 1, specifically. And it's just all around, it's a really interesting conversation. So joining me for the conversation, of course, was Alex Dumos, who, as you'll hear, is also the CIO of Sentinel One now, as well as being the ciso, and also Steve Stone, who is the. And I've got his title in front of me, the SVP of Threat Discovery and Response at Sentinel One. So, yeah, just a great conversation with Alex and Steve talking through Sentinel One's research, which you might have seen pop up in your newsfeeds. A lot of people covered this research. So I'll drop you into the conversation now where Steve Stone is describing what Sentinel One did with all of these North Koreans who were trying to apply for jobs@sentinel1. And instead of just throwing their resumes in the bin, they decided to string them along for a little bit and see what they could learn. Here's Steve Stone. Enjoy.

[Steve Stone]

Yeah, so it actually started we were going through resumes for a position on Sentinel Labs, which is our threat research team. And we just saw some things that jumped out and we started poking and it got really interesting. And we had a couple of late night phone calls actually with Alex. And part of this was it was a risk decision. We had to make a decision on were we going to turn it off right away or were we going to play this thing out and really try to understand the scope and scale. So we've spent probably five months actively working against the North Koreans applying. We've seen Almost, I think 370 distinct Personas. They've sent in more than 1,000 applications and they've been, this is not a one day thing, it is day after day after day. So we just kind of stepped on it and just kept after it. And we got all the way to the point we actually tried to interview them and that's where they would peel off, they would refuse to go on camera. So we could get them on email, we could get them in everything except for on camera.

[Patrick Gray]

I mean, that's not usually the case though, is it? I mean, I've heard of plenty of these applicants at other places actually being interviewed on camera and doing quite well in the interviews.

[Steve Stone]

Yeah, I think it's them learning just like we do. I think they've seen a couple of videos at conferences like this one and they saw their faces on camera and I think they probably adjusted to that.

[Patrick Gray]

Was there any point where they were doing stuff like trying to send over or, you know, trick people on your end into looking at, hey, look at my latest software project, just run this binary. And did it get that far or was it really just too early stage?

[Steve Stone]

That's a great question. What they were actually doing was that before they sent resumes in. So they did exactly what you just described, but they did that as the very first step. And what I think is really interesting on that, if we take a step back and look at North Korean intrusions, that's not new. They've always done that. They did that on LinkedIn, they did that with emails before they sent spearfishes. So it's interesting to watch these old habits carry forward in just their new set of activity. So, yeah, we had multiple people at Sentinel One receive effectively cold emails saying, hey, I'm really interested, would you mind helping me? Would you look at some of this material? And that normally leads to Phishing and this didn't. They were just trying to get more eyes on their application to get a job.

[Patrick Gray]

Yeah, I mean, it's, it's, it's crazy, the breadth that they've been able to spin up. You know, this has been a long standing conversation between a, you know, a friend of mine and, and myself, Dmitri Alperovich, who's one of the founders of your competitor, one of your major competitors. You know, he and I have often talked about North Korea and he's always said, oh no, they're very creative, very, very good. So, you know, he takes issue, I guess, with some of the coverage of them because they're really in the, in the limelight now. But my argument is that what the North Koreans have now, which they didn't have previously, is scale. Right. So you would, you would find that they were able to do some very creative things, but now they're doing these creative things at massive scale. Does that vibe with your understanding of the evolution of North Korea's ttps?

[Steve Stone]

Yeah, it absolutely does. And I think there's really two interesting elements. The first is we always in cybersecurity treat North Korea as if they're the anomaly. But if you look at North Korean cyber activity in the context of North Korea, it makes perfect sense. The US had to redesign the hundred dollar bill because North Korea was the number one currency counterfeiter in the world. The EU had to redesign the euro multiple times. So they have no problem doing illegal activities to make money. They're just now doing that in cyber, and it's not that different than we see them doing with cryptocurrency as a good example. So I think that's takeaway number one. This is all caps. North Korea. This is very them. Second part is if you look at any of the North Korean cybersecurity efforts, once they figure out how to make something successful, they will just stay with it. So I think that's a good indication that someone's hiring North Koreans because they're still doing this and they just don't waste their time in cybersecurity in this space. We see. I mean, again, I'll go back to the cryptocurrency. They are really effective at stealing crypto. They're really effective at understanding crypto exchanges. So if we compare that to the North Korean IT worker piece, we have to make a reasonable assumption that they're being really successful getting employment.

[Alex Stamos]

Well, and I think we have to assume the reason they're coming after us and they're coming after a company that has like an intel team is not because we are the easiest way for them to make money. It's probably specifically because we have significant penetration in the crypto industry.

[Steve Stone]

I think you're right. I think it's about. We're a piece of that larger puzzle they want.

[Alex Stamos]

Yes. Which I would expect that we're getting better applicants, that we're dealing with a better, you know, anybody who does anything at scale, you end up with a diversity of skill sets. You know, if the lesser skill sets are going to be going after the remote jobs at, you know, the business process outsourcers and the folks who aren't going to be doing as much work here, I expect we're getting the better applicants and the people who are better.

[Patrick Gray]

You're not getting the C team. Right?

[Alex Stamos]

Right. I'm not going to say it's the A team, but if we're getting the people whose job it is to eventually do a supply chain attack because we are running in kernel mode on a bunch of machines that are handling crypto, it's not going to be the C team like you said, Patrick.

[Patrick Gray]

Yeah. So just one thing you said there, Steve, about the lengths to which North Korea will go to bring in money. A fun little Wikipedia rabbit hole. Anyone watching this or listening to this might be to look up the Pong Su. So P O N G space Su this was a boat that popped up off the coast of Southern Australia, very close to the beach, and it was a North Korean fishing boat that was smuggling heroin into Australia. And the swell happened to be very big when they were trying to do the drop off and a couple of them drowned and it turned into this, this whole thing. But just fascinating insight into the way. I mean, this was a long time ago, but it's just fascinating when you've got fishing boats smuggling heroin into Australia as part of a state sanctioned activity. So, yeah, crazy.

[Alex Stamos]

It's like, what would you do if you had crime but you did crime at state scale?

[Steve Stone]

Yeah, Well, I think one of the things that's interesting, like in, again, in cyber, we tend to treat North Korea almost as lesser because they're not the Russians, they're not the Chinese, but they're really good at areas where lots of other essence companies and groups struggle. So like North Korean IT worker is a good example. They've got front companies. So if you actually, and we did, we started calling the numbers on their resume, we started tracking down their employment. There are people that answer the phone. There's actual effort to give this structure and we can even Go back to. I spent most of my career working Chinese espionage. We did attribution on them for years because they were really bad at registering domains as a business. North Koreans are doing that all the time and they're really, really good at it. So I think we just have to recognize what it is they're doing, why they're doing it and what works and what doesn't.

[Alex Stamos]

And, you know, it works because sometimes when you check their resume, they really had those jobs at other companies.

[Steve Stone]

Yeah.

[Alex Stamos]

So you're like, we're not going to name any of those companies. But yes, you go and you notify those companies and like, oh, yeah, I'm sure it's fake. And then you don't hear back from them and you're like, oh, okay, guess it wasn't so fake.

[Patrick Gray]

Now, one thing that gives me hope with these North Korean IT worker, you know, situations is that it's very rare that law enforcement gets an opportunity to asymmetrically disrupt, you know, cybercrime. But there is a weak point with these operations, which is they are always, it seems, relying on some sort of laptop farm in a basement where these workers can remotely control computers that are based in the United States. So once you collect a little bit of, you know, you know, a few bits of information about what the originating IP is for these workers when they're, when they're doing their thing, you know, you, if you detect one, you can shut down 100 kind of thing, you know. Do you think that law enforcement is going to be able to keep a lid on this? I mean, you know, you can never completely eliminate a crime syndicate like this. But do you think there'll be some effective suppression?

[Steve Stone]

I do. I'm hopeful on that. And I started my career in federal law enforcement, so that space is very familiar to me. And the reason I have hope in that is two reasons. One, just as a bunch of cybersecurity companies, we are all working together on this and there's not a ton of times we've done that, like the Ukraine, war, Solar Winds, there's like a handful. We are working with other companies and governments literally every day on this. So there's just a lot of really, really good collaboration. And the second part is, to your point, law enforcement's great at pulling threads to get to what they're looking for, and they have lots and lots of threads to pull and there's physically places they can go action that's not very common in this industry. So I think we'll have some good success there. I really Do.

[Alex Stamos]

Seems like the other choke point is payroll, Right? Because these people are using mules in the United States to get payroll. It feels like ADP could just solve this, right? Like, if the law wasn't in the way and they had a team who cared about it. And so I'm a little bit shocked, actually. We've gone this far.

[Steve Stone]

Well, I mean, like, that's a good example, I think, too. Like, law enforcement, they've been doing that for 40 years. They know how to deal with that. So I think in a lot of ways, they're actually happy to get away from all the hacking and go after, like, oh, yeah, money mules that we can do. Like, we got people for that.

[Patrick Gray]

Yeah. I mean, I think the best hack I've seen in years was the Bybit hack recently, which was attributed to the North Koreans. And the way they did that was amazing. They targeted the upstream, like, Safe Wallet supplier to Bybit. And the way they did that is they got someone at SAFE to run something. I think it was. They pretended to be recruiting for a role and said there was a coding challenge. Just run this project and someone did. That's what got them their access. Then they were able to fiddle with the JavaScript that was being served only to one customer, which was Bybit. And they got away with $1.5 billion by subtly changing something so that the multiple parties who had to sign that transaction didn't notice anything was amiss. I mean, it was an incredible bit of work. And my. My co host, Adam Boileau, and I joked that, you know, one strong argument for Korean unification or reunification is that we'll get to sit down and have. Have some drinks with these guys and actually talk to them about their. Their capers. But look, let's move on to another thing that you covered in the. In the report, which was what ransomware operators are getting up to these days. And there's something very telling in the research, which is that one of the first things that they're trying to do is actually get access to EDR consoles and shut them down, which makes so much sense, because edr, you know, just as a category, correctly configured edr, you can pick it from, you know, four or five different vendors where if you're running it properly, it's going to defeat ransomware. But it can only work so long as it's running and correctly configured. And if you, as an attacker can get access to a console, you can effectively shut it down. So, Steve, talk us through what they're doing in terms of trying to get access to those consoles. And I think the other bit that wasn't so much present in the document that I looked at anyway is once they've got console access, what is it that they're actually doing? Are they allow listing specific malware? Are they completely disabling stuff? Like what do they actually do once they get in there?

[Steve Stone]

Yeah, so I think the first part of that is they effectively are acting like a prospect or a customer. They use all this really convenient technology all of us EDR companies build, so you can test it, you can play with it, you can see if it works. That's all they're doing. They're doing that and they're trying to steal credentials and log in to consoles. And that's not unique to us. That's every single edr. And I think what's interesting on that is we actually see them try a new technique or a new tool against all of the EDR technology in one fell swoop. They're just being practical. They want to be able to ransom networks and steal data, and they know they have to deal with X amount of EDR companies. So I guess that's the second part of the answer there, too. That's what they're doing. They're just testing their gear and they're seeing what fires and they're doing one of two things. They're figuring out how to do something a little bit different, to either hit a more generic alert or just avoid it altogether, which actually really rarely happens. Then the second part is they just are testing what fires and what doesn't and how do they change the settings and then they go from there.

[Patrick Gray]

So there's two parts here. Right? So there's the part where they're trying to get access to environments with EDR in them so they can test. Right. So that's what you were just talking about there. But there's the other part where you were mentioning, you know, stolen credentials and whatnot, even authentication tokens, I'd imagine. And that's going to be a feature of this conversation a bit later on, I guess. You know, the first thing I wanted to look at is what are they doing once they've logged in with the credentials for, you know, someone who's administering the EDR for one of those companies? Are they just straight up turning it off, or are they, you know, trying to allow list certain bits of malware or disable certain features and alerts? Like, what do they do once they actually get access to the console in a targeted environment? So this is separate to the testing Side, Yeah.

[Steve Stone]

So that's changed a lot over time. In the early days of ransomware, which wasn't that long ago, they would just turn everything off, which is obviously pretty easy to find. So what they do now is they effectively try to change configurations and hide in the settings noise, for lack of a more technical term. So they really just try to make it so it looks like it's running as normal, but they flipped a couple of bits that allow them to run. And then I think if we look at a ransomware intrusion, like a typical ransomware intrusion lasts about five days, lots of variation there. But typically they're in an environment for five days before they get to a ransomware deployment. So they're not interested in being there forever. They're not interested in learning everything there is about it. They want to know just enough to be able to do what they're trying to do for effectively a week, and then they move on to the next thing. And so they're not. It's very different. Like the Chinese model, the Chinese espionage groups have to hit a certain target, and so they go after that very, very differently. Ransomware groups just need to make money, and if that's at a target and it doesn't work, they'll go on to the next one. And so their level of effort is really around. How do they just make the console not work enough to let them do what they want to do?

[Patrick Gray]

And is this something that has happened, and I'm not just talking specifically about Sentinel 1 customers, but is this something that ransomware operators have been able to successfully do quite a lot? Like how common is it that someone who's doing the right thing has EDR software rolled out across their enterprise, is then getting owned sideways because someone was able to get access to their console?

[Alex Stamos]

Well, it's super common if you have single sign on, right? And there's a certain competitor of ours where their EDR is free, if you buy your email from them, for example, and you can't turn off single sign on. And so if you become tenant admin, right, Then you can just turn off everything. And so that's like, this is where you have to. This is the challenge of buying the product that's free when you get the bundle is you have to be careful because almost by definition, if a ransomware actor is being successful, they're getting some level of administrative access in like a hybrid mode, active directory. And so they can go, these guys know what they're doing. They can pull the PowerShell. They've got their standard Powershell script, they can turn the settings that they need and if you don't have the alerting in place, you won't know that happened until it's too late and you're doing the forensics. So it's really common in those environments because single sign on is always turned on, which is one of the things I'm a big fan of single sign in in lots of situations. This is where though having different administrative domains for your normal IT administrative domain and your security protections I think is actually important. And I often, for high security environments, recommend that people are running a separate administrative domain for their EDR product and, or their logging, their security logging than for their actual IAM and for tenant admin or whatever the equivalent is for whatever cloud service they're using for this specific example. So that if somebody's able to escalate within their IT environment, they're not able to also turn off all of their login and turn off all their security protection actions at the same time.

[Patrick Gray]

Yeah, I was just going to say, like, how do you, you know, once you've, you know, installed all of your edr, can you easily migrate it to another administrative tenant, though? Like, I'm just, I, I don't know, but.

[Alex Stamos]

Well, for us you don't have to, you just don't have to turn on a single sign on. Right? But I'm just saying, like, if you're, if you're running, if you're running M365E5 and you're using Defender, hey, look at that.

[Patrick Gray]

You said the name. You said the name. But earlier a moment ago, it was our competitor who also.

[Alex Stamos]

Everybody knows I'm talking about, so I've just decided to say it. So I mean, there are ways you can do it, right? But it's tough, right? So that's where I've seen it the most from a forensics perspective. Just because bad guy becomes Tenant Admin and they just know, great, I turn off Microsoft Sentinel, I turn off Defender, bam, bam, bam. I have one PowerShell script that does everything I need to do, right? And so that's where, and there are ways to try to prevent that, but most people have not put the steps in place to try to prevent that kind of stuff. And so my suggestion is, if you're running, if you're running Azure Hybrid mode, do not put your Sentinel one console plugged into that. Because the whole point is, if Microsoft has failed you, we're there to keep you safe. Don't make it that you've also lost control of both Unify that authentication in somebody's Fido token.

[Patrick Gray]

Yeah, Steve, you wanted to jump in there?

[Steve Stone]

Yeah. Alex's point on MFA is a really great example. I mean, I think EDR killer, EDR bypass. We really want these to be like these super sexy things. I mean, here at rsa, there's spray paint on the mailbox about EDR bypass, EDR killer pen tests. Love talking about them. Red teams, cyber criminals. It's this big, beefy topic. And we spend an inordinate amount of time, as people would expect, researching this. We research the claims, we research the tech, we go through all of this. Here's what has not happened. An attacker came in from an external avenue and found a way to fundamentally subvert the technology and used it for their own purposes. That has not happened. What happened is exactly what Alex says. You have legacy agents, you have features that are turned off. You have one account, and it's just a password that's been sitting in a forum for two years. That is how that is One of those things I just mentioned is in 100% of EDR killer, EDR bypass, right?

[Alex Stamos]

You have an unpatched Palo Alto box. You pop it, you pull creds out of memory. You have a token that lets you become tenant admin, and you can just turn your tenant admin. You can go to all the consoles, you just turn everything off. That takes no research. You don't have to spend 170 hours coming up with a kernel bypass for us or crowdstriker or any other EDR product that then might get patched out anytime. It's the single sign on mfa, and then it just turns out to be the exact same credentials that allow you to then push your ransomware everywhere with one RM or PowerShell script. So, yeah, that's much more likely. The EDR killer is almost always talking to a SaaS interface.

[Steve Stone]

It's almost always poor hygiene.

[Alex Stamos]

It's poor.

[Patrick Gray]

Yeah, yeah. So now let's talk about the other aspect to this research. Right? Which is, and this was fascinating, and you were touching on it earlier, which is the fact that there's this entire underground economy where people are sort of spinning up, like, you know, virus total, but for EDR and for criminals. Right. So that they can actually test their stuff against various EDRs.

[Alex Stamos]

These guys have figured out that VirusTotal belongs to Google. They finally figured out you can't upload your bad stuff to Google and get away with it.

[Patrick Gray]

Yeah, because someone's going to notice. Right? So I guess the interesting thing is here, like, it's not terribly hard to get yourself a license for an EDR product. Right. There is some basic KYC around that that most of the organizations do. Well, all of them do. But that's a compliance checkbox. Right. Like kyc, no company really wants it to be an impediment to sales. So stuff is always going to slip through the cracks. And your research argues that that's sort of what's happening here, is that these companies exist to sort of bypass kyc, you know, bypassing KYC as a service and essentially setting up test labs for people to throw their ransomware into. You know, pretty interesting.

[Steve Stone]

Yeah. We, we talk about in the research, there's a particular ransomware group that's actually spun up a whole front company just to acquire EDR technology in demos and PoCs to then run at it and then sell it on the dark web in the criminal forums. So I think what's so interesting about this, and this is part of why we released the research the way that we did. We don't want to just say, like, here's some bad stuff. It's all bad. And we're out. Like, have a great day, everybody. We want to talk about how do you really solve these things? And if we look at the North Korean IT piece, we didn't solve that in a traditional cyber way. Working with Alex's team and we're manning the firewalls. That's not what happened. We worked with the people team and we worked with the people team to figure out how this was all going. And now we fast forward to your question here. We had to work with the procurement team, we had to work with the sales teams, we had to work with sdr. These are teams. When you talk about the list of your cybersecurity professionals, they're usually not on that list. But that's exactly how we got to a lot of this and that if you don't know your customer well, you've just accepted a ton of risk. And we're using the customer word when in a lot of ways, actually even prospect. If you don't know your prospect, if you don't know your pipeline, you're just running blind to an entire surface area in your organization. And that's not unique to us as a cybersecurity company. That is every single company who's using technology, period. Hard stop.

[Alex Stamos]

Yeah. Truth is, though, I mean, every security company, including us, has a significant. I mean, we have MSSPs, we have system integrators.

[Steve Stone]

It's tough.

[Alex Stamos]

We don't know lots of our customers we don't have relationships with.

[Patrick Gray]

I mean, this was literally going to be my next question, which is like, once you're selling software at the sort of scale that you do and your competitors do, like, I can't imagine that it's possible to have, you know, foolproof kyc, right, that's going to prevent this. I mean, you can minimize it, but, you know, you're going to have resellers and as you say, like mssps and whatnot.

[Alex Stamos]

We sell our product in the Amazon Web store and the Amazon Marketplace. Right. So well.

[Steve Stone]

And that's why I go back to the procurement teams as a big part of the solution. Here's what we are not going to be able to effectively do. We are not going to be able to look at every agent across the fleet and say, this one is attached to this mssp, they've resold it and now we can technically find our way to see who's using that agent. That is not technically possible. What is possible is saying, this doesn't look right. Let me pick up the phone and talk to the reseller and they can tell you, usually within minutes. Because this is business. It is not hard finding a business person in a company that is an effective and valuable, very quick way to find a thread to pull. And now you can bring in your technical experts and let them do what they do really, really well.

[Alex Stamos]

But what we have done is we've found multiple situations in which either they're front companies or a company has been the identity of a company has been stolen and there's a free trial has been provisioned, but it's not the real company and such. The great thing there is, once we identify it, we don't just shut it down, we then toss it over to Steve's team and we go, pay attention. And now we go and we have all those logs. They're in our cloud, so we can see everything they're doing. We can see the malware they're testing and then we can burn it. Right? So there is a risk to them of doing that and that we are a cloud. They're not getting an on prem product, they're getting a cloud connected product. So if you're listening to this podcast and you're doing this right, it's not like VirusTotal, where that stuff is running In Google's environment, we still have all that telemetry and so we have the ability to roll it back once we figure out that somebody was using a fake identity and then have the ability to burn all that malware later.

[Patrick Gray]

Yeah, that was going to be something I was going to touch on as well, which is one of the opportunities there, because we saw a fantastic bit of research. I think that was what, last year from the Sophos people. Yeah. I mean, that was incredible what they were able to do, which is they realized that people. I think it was a Chinese contractor actually providing tooling to, like, APT crews there. They realized that they were downloading virtual machines like VMs of Sophos products and using them to do exploit development. And I think they were trying to turn Sophos boxes into orbs, basically. Right. And they started dropping some pretty nasty stuff on these exploit developers and. And were able to burn their exploits before they could even use them. And these people had no idea what was going on, which was fantastic. I mean, have you actually done that yet, Steve, or is it something that you're thinking about?

[Steve Stone]

Yeah, I mean, we definitely do that. And I want to give Sophos a lot of credit. That report they put out last year was great, and I think it really did expose how tough of a challenge this is and how pervasive it can be. If you're not looking at it, I will tell you on our end, and I mean, Alex said it right out loud, and I don't think it's a secret. We look at all that and so we're able to go back and say, here is what's working, here is what's not. Oh, by the way, they just gave us all of these files. Let's rip all this out. Let's go alert on this and just keep going here. So I keep using that risk word in this podcast, but that's a big part of that decision. We make a very intentional risk calculus to let some things play out, because we can move fast, faster than the attacker can. And we always talk about the attacker has the advantage. That's not always true, and this is a great example. We get to see what they're doing. And this is all people business. When the day is done, cyber is people. And people get lazy really quick. And if that box doesn't turn off on day one, they're pretty sharp on day two, and by day 10, they are just winging it. And we're gonna pull all that. We're gonna learn all that, just like we are. Just like they are. And at the point we're going to turn it off is when we think there's nothing else to gain.

[Alex Stamos]

Yeah, I mean, once we know that they're doing this, it's like, it's way better. Like, oh, you're going to run our agent on your systems. Cool, that's great. No, you can have it for free now. We won't charge. That's great.

[Patrick Gray]

The idea of doing this used to be a little bit sort of Hollywood movie, right? Like, it just wasn't something that happened. I feel like that's really changed in the last few years where vendors are saying, okay, in order to develop the exploits that are going to be needed to target our technology, or in this case, in order to develop malware that will be able to bypass this technology, we need to be playing with this technology. And there are just some terrific opportunities. I agree with you, Steve, that that was fantastic research from Sophos. And I like that they did that publicly because it sent a signal to all vendors that this is something that they should be doing.

[Steve Stone]

Yeah, I think that's important. And I think what they did, I think they deserve a lot of accolades and kudos. They took a risk and back to that R word again. But whenever a security company goes out loud and says, hey, we're getting messed with and this can be really, really nasty, that is a really risky decision. And I'll use the research we put out today. Here's what also did not happen. Some of our researchers did not write a blog, and it went out and we're all like, okay, great. We had lots of meetings and discussions. We're meeting with our C suite, we're meeting with our lawyers, we're meeting with our tech experts to say, if we talk about this, what are we going to lose here? And it's worth it. We think it's important to show other organizations, one, this is real. This is happening. This is not us making up a marketing story to sell technology. And also, here's how this gets solved. And in many cases, that's solved. You don't have to be a gazillion dollar, Fortune 100 company. It's actually really accessible for most companies once you kind of do things just a hair differently. And I think that's really, really important here.

[Patrick Gray]

All right, so look, one more thing we're going to talk about from your report, from your research is, you know, a look at China, which is obviously the, you know, the most at scale adversary, I would say. I mean, you've got North Korea doing things at scale, but a lot of that's involving, like social engineering at scale. And, you know, that's the way they roll. China a little bit different, more sort of, you know, classic hacking, if you will. You had a look at two APT crews, one you call Purple Haze, one Shadow Pad. I believe with the Purple Haze example, they were actually trying to target your customer environments and then swim upstream into your systems. Why don't you walk us through what you found there?

[Steve Stone]

Yeah. So old school China Watcher, China Cyber is my absolute most favorite part of this. And I think the hardest thing I've ever had as an intelligence professional is accurately conveying just how bad the Chinese cyber problem is. Because you sound crazy. You sound crazy discussing how big and how pervasive and you lose people. And so I think we look at the two different Chinese events that we had. It's a good example of just a tiny slice of that scale. So we had one Chinese espionage group that did subvert an organization that happened to be one of our clients. And in the course of that, they made a really hard run at our console. They wanted to understand our technology for all the reasons we've already talked about. They had also fully subverted the VM it sat on. They fully subverted the server that the VM was on, and they fully subverted the account of everyone who had logged into that vm. So we talk about, like, pervasiveness. That is just one tiny little piece. So we saw them, we dealt with that incident response. We worked with that organization. And what we were able to do because of that, we really dove into that. We could then pivot back and say, oh, wait, they came at us. They really try to take what they learned at that victim and immediately pivot and see if they could compromise Sentinel One direct. So we have that going on in one case with a group of malware that is not unique to that Chinese group, which is not unusual. Then we have a separate Chinese espionage group that went after at least two of the IT providers that we use. And I want to be careful here. I can't tell you they compromised these organizations because they wanted to get to Sentinel 1. I just cannot tell you that one.

[Alex Stamos]

Of them definitely did, because it is really, like me, the odds that they went after them the same month. And this is one of them is like a very important company that services a lot of organizations. One of them is a tiny company that does not. And for them to go after them the same month that we're like, engaged in a knife fight with them.

[Steve Stone]

Yep.

[Alex Stamos]

That is not quinky bank.

[Steve Stone]

Highly likely.

[Alex Stamos]

Sorry.

[Steve Stone]

Highly likely.

[Alex Stamos]

I don't believe in coincidences of that level.

[Patrick Gray]

No, no, it's Alex. Alex can, you know, as a lifelong siso, more or Less can feel it in his waters.

[Alex Stamos]

It's been an interesting couple of months, Patrick. Yeah, no, it's like, it's just, I mean, this is what you're dealing with when you're dealing with a Chinese adversary. I mean, what I tell my team is there are at least 20, maybe 50 people whose job it is just to break into Sentinel 1. Working in the Chinese government wouldn't doubt that at all.

[Steve Stone]

Yeah, absolutely.

[Alex Stamos]

Right. Because it's like that's the size, you know, the Chinese have what, probably 100 to 150,000 offensive operators. So we're not as big as Microsoft. Right. They have at least 1,000 people breaking into Microsoft. But if you had 150,000 operators, you would take the entire security industry, then you divvy up. And so that means for my team, I tell them, like when you stop them, they don't go away, they go home. They come back the next morning, they have their local, a culturally appropriate caffeinated beverage. They get a little talk from their bosses and then they start again. That's all they do all the time is they try to break into us. And if they can't get in, then they're going to look at our supply chain and somehow they found out that these two IT vendors, neither of which were obvious. Right? So neither of which, these vendors were advertised. We're not the government. So beyond trust. And treasury, you can look that up in the database. You can just look up that treasury had to be on trust. There's no way to publicly determine that these two vendors worked with us, which is an interesting thing to try to figure out how they knew to go after them. So, yes, it definitely raises one's level of paranoia when you're playing at this level.

[Patrick Gray]

I think when you're being targeted that pervasively as well. You have to operate under the assumption if you can think of a way to get you owned, they can too. You know, and I think that's. That's changed a little. Right, because, you know, we've always said security through obscurity is no good. But you have to operate under the assumption now that, you know, your adversaries, particularly Chinese apt crews, have a very detailed understanding of your environment, your tooling, how it all works. I mean, they're very good and our.

[Alex Stamos]

People, right, we're now at the level where they'll be using human intelligence. Because, like, this is also what I tell my team is like we're at least at their level when it comes to cyber. That doesn't mean we're Perfect. They could beat us, but we're at least playing the same game. When it comes to human intelligence, we're children, right? When you're talking about the Ministry of State Security, they are the descendants of intelligence agencies that have been doing this since Confucius. It's the same when you deal with the SVR came from the kgb, which came from the nkvd, which came from the Czechists. It's like, oh, we're communists now. That sounds good, right? Like, change the hat. Right? Like, you're talking about, like, a tradition of human intelligence that goes back hundreds of years. And they're good at it, and they're very good at it. And it's the same thing in the People's Republic of China. They're like, oh, Mao, okay, sure, I'll be on that team now, right? Like, almost by definition, it is the people who are the best at changing their stripes are the ones who survived the Cultural Revolution and the Great Leap Forward. And they're like, oh, sounds good. Sure, boss. Right. And we're at that level. That's what Steve's research shows. And so we have to be hardened up against the human intelligence level, too, which is I played at that level at Facebook, and it's not fun to play at that level. It's not fun to have your employees targeted at that level.

[Patrick Gray]

I'm just going to cut in now because we are already. I can see that we're going to get a little bit crunched for time, and my apologies to the event organizers, but we are going to go over time. But just quickly, Steve, if you could fill us in a little bit too, on the Shadow Pad research that made it into the report.

[Steve Stone]

Yeah. So ShadowPad, that group of activity, was one of the two events that I mentioned. And I think what's really interesting there, it keeps going back to the scope and scale. Shadow Pad is not unique to a single group. You can't do attribution off of shadowpad. You have to understand lots of other things around it. And when we were looking at just this chunk of Shadow Pad activity that we were involved in, we found 70 plus victims. So this is a narrow slice of one particular approach that they use that multiple groups use. And in just that blink of an eye, we see 70 plus victims.

[Alex Stamos]

Then why do they. Like, we haven't talked about this. When they know that, then we can key off of something like this. We find. Then we can find and notify these 70 victims. Is it just that valuable for them to have that economy of scale? Is that why they do it.

[Steve Stone]

I think it's just that. I think it's that straightforward and I think it's an issue of where it's non negotiable. There's been so much research that says there is a list of companies and a list of organizations and they must get subverted.

[Alex Stamos]

Right. So this is the flip side of them having such large capabilities and then having such a long list of operations is that you don't give people creativity. You're like, here's your tools, here's what you're doing, go do it.

[Steve Stone]

The best example I've ever seen in research is from. And this is going to be weird now, but our own Dakota Carey over on Pinnacle 1, he had this just incredible piece of research where it actually showed a Chinese company had to take and compare what they were tasked to do in China's five year plan and write out the technologies that they thought they would struggle with that went to a specific government organization that then looked across China and said, we have these technologies other places all or we don't. If it's a no, it went to a third organization which then had the original company go and fill out an actual form. Like there are screenshots of this form that said, do you want us to buy it? Do you want an insider? Are we going to send students? Is it going to be a partnership? And then the classic other at the very bottom. And then you could take that other and you could map that directly to lists of compromised Western companies. And it's just when you actually even like speak with them, you talk like some of the original negotiators for China and us with cyber, they will tell you like, yeah, they're very. Oh, they think we're the fools. They think we're the fools for not doing this.

[Alex Stamos]

Yeah. Because we don't put an other on.

[Steve Stone]

The procurement form and they'll place like, you don't hack other companies. Like why you get so much great research.

[Alex Stamos]

I mean, fine, you don't want, that's your choice. But shit. Okay. Works pretty well, guys, come on.

[Steve Stone]

So I think it's just that we recommend other. Yeah, other work's great.

[Alex Stamos]

Other work's great.

[Patrick Gray]

Okay, okay, okay. So let's move on to the next part of this conversation. Right. Which is about how the threat environment has changed and how the tech stack has changed in response to this. So one thing I would like to note here is we just spoke about attacks from North Korea, ransomware actors and China. Right. One thing we weren't talking about is client side Exploitation of like browsers. This is not how attackers do it anymore. We weren't talking about people just dropping malware on, you know, via email into onto someone's machine and having them execute. I mean in the case of North Korea, kind of, but that's developers. We had edge, case, whatever. My point is attacks have changed, right? A big part of the reason for that is because of the ascendance of edr, right? Appropriate endpoint controls. And not just from you. I mean it's an established category, it's a mature category. There's a bunch of companies that make good edr. The point is that the way that you get on target these days is not the way it used to be, where it was like a watering hole attack and you throw a browser exploit at someone and you get a shell and then off you go. It's not like that these days. You've got, you know, we were talking about some supply chain stuff there. Like the Bybit one is a great example of, of that. You were talking about some Chinese apt crews who were trying to target your suppliers to get into your organization. So supply chain, supply chain, supply chain is a very big part of all of this. We're also seeing particularly the Chinese hitting those devices, vulnerable devices at network edges and then from there, because often they're domain joined appliances. They get all of the material they need to then go and access. Alex was talking about this earlier access the EDR consoles, shut them down. We're also seeing phish kits these days, the sole purpose of which is like pass through phishing, which is even effective against one time codes, right? Not effective against Fido, but that's a whole other conversation. And the whole goal there is to grab a session token out of a browser. We're seeing people starting now to publish malicious Chrome extensions, for example, so that they can get those authentication tokens again. Maybe from there they can pivot into some sort of administrative console, either Microsoft one or for ADR platform. But the point is things have really changed. It's great that we've racked up some wins with edr, but I do feel like the tech stack that we're dealing with now isn't quite equipped for this world. Things have moved on a lot, but I don't feel like, I think we're sort of still focused on yesterday's problems a little bit. Alex, I want to start with you on this. You know, what do you think about that statement that we haven't appropriately moved forward without security? Tech stack.

[Alex Stamos]

Yeah. So I think the thing that now needs to change. And it is changing is the traditional way that security products are built is that one product does the instrumentation and detection itself. And that product is dedicated to a specific deployment scenario. It does endpoint, it does cloud, it does email. A company might have three or four different products, but those are different products. And yes, you might platformize. I'm sorry, I owe Palo Alto Networks 5 bucks now. Every time I say the word platform, Nikesh makes 25 cents and an angel gets its wins. But they're actually different products. Right? The idea, we think about it. So you end up with an endpoint product that does the instrumentation of the endpoint and does the detection of the endpoint, and it might pull that back into a single console, but you still have a cloud product, Endpoint product, an email product. I think where we need to go, and this is starting, we're doing it, and I know other people are doing it too, is you have to detach the instrumentation from the detection. Because as you talked about, we now have this very complex mix of intrusion sets where an intrusion doesn't just happen in one place, it happens in a bunch of different places at once. And the behavior on all of those different domains is sub malicious in those different domains. And it is only malicious when you zoom out and you see the whole thing. You'll have endpoint instrumentation, like an EDR that sees all the stuff that's going on, but the detection can't be there because you can't see all the stuff you need. And you'll have cloud instrumentation, which is mostly just talking to APIs. I don't know why you'd pay $20 billion for something that just talks to APIs. But you know what? I don't have that much money, so I guess I'm not smart enough. And you'll have email instrumentation that looks at all the email. But then what you need to do is you need to have the instrumentation that pulls that data into one place, and then you're looking at the stream of all of the events, all those places, and you have delayed gratification here, where you're delaying the decision of whether or not something's malicious until you see all that context. This is super hard. It is super hard because the mixture of all of those things at every single organization is totally different. It is much easier for us to come up with an alert that happens on a Windows Endpoint because a Windows endpoint at JPMorgan Chase versus ExxonMobil versus Delta, those windows endpoints might be configured a little bit differently. But realistically, somebody is running Word and they open up a Word file with a bad macro that looks the same, right? But those three organizations have a very different mixture of the Windows endpoint and AWS, GCP, Windows, Microsoft 365, how their email is configured, how all these things are configured. And so that is, we've made it this far, nobody's talked about AI, right? But this is where the kind of modern machine learning algorithms can start to let us be a little fuzzier and all this stuff. And I think we can start to get there. And I think that's, I think, the next revolution here. Does it make sense?

[Patrick Gray]

No, it does, it does. I mean, I guess one of my issues though with what you're saying is I feel like there's certain sort of detection and primitives and controls that we just don't have. And a good example of that is a malicious extension that a user installs on their browser that grabs a very important authentication token and from there the attacker just goes immediately to great victory. So the idea that there's such a simple path at the moment, and that's just one example, I just sort of think, well, what are we doing?

[Alex Stamos]

That's an interesting one because you've pointed out a different issue, which is we've created these blind spots that exist in these incredibly complicated execution environments that are self contained. And the browser is the best example of that. And that Chrome is a world unto itself. It is a operating system unto itself. It has multiple security boundaries unto itself that never leave the process boundary. An EDR product which operates in the kernel that expects that you have to leave the process boundary for anything to work, that you have to at least call a dll. If everything is happening in Proc to a single Chrome process, then to us it is just a black box. Unless now you have to do injection into the process and Google updates chrome literally every 72 hours or something. So Godspeed doing that, not crashing Chrome all the time.

[Patrick Gray]

There are options there, right? So you can look on disk at extensions and things like that, but try detecting that there are options.

[Alex Stamos]

But then that's why I'm just saying we create those weird little places and they're like, okay, great, now I have to have an incredibly specific spot solution because Google decided to create this entire pocket universe. It's like looking inside of a neutrino and you're like, oh my God, there's an entire universe in here, right? And it becomes like a dorm room pot session, right? You're like every neutrino is a universe. Right. But that's effectively what Chrome is. And then you have to come up with a spot solution. There's only a couple of those, but browsers are the best example of that. And you're right, the other One's visual is IDES. And we're dealing with this at Sentinel One right now. For all of you CISOs, if you're bored, if you're like, man, my job is really boring, I'd like to ruin my next quarter, go look to see what plugins your developers have installed into Visual Studio code. And there you go. You have ruined your life.

[Patrick Gray]

Yeah, let me just jump in there because I think another thing about all of this is you were talking earlier about GCP and Azure and M365 and whatever. You know, from an authentication and an identity perspective, when you're thinking about what is cloud and what is SaaS, is there even a difference? Talking about dorm room pot sessions. Right? Is there a difference between those things from a. Well, it's just a token in your browser, isn't it? Right. So I feel like, you know, we're seeing some really sophisticated attackers these days, just moving through clouds and SaaS. Right. And intermingling between both of those things. And I just feel like, okay, what's your solution there? I feel like the IDPs and the SaaS providers are kind of letting us down by not giving a standard form logs that can even be that you can even make sense of. And the IDPs might develop some great features that the SaaS providers then don't implement. And I just feel like there's gaps here that are actually quite surprising.

[Alex Stamos]

I mean, you're hitting upon another problem, which is the vast majority of companies, internal cloud authentication is a complete disaster in that the vast majority of companies I have worked with, you're like, we just threw it into a vpc. And so anybody who could talk to another IP address, we're fine. Almost nobody who's not an actual hyperscaler themselves has actually implemented some kind of cryptographically secure dual authentication mechanism. Because to actually do that at scale and for it not to be spectacularly brittle is almost impossible. And the fact that the hyperscalers have not made that easy is honestly on them. That is a huge problem. We have not dealt with identity. You're totally right that identity is the biggest problem in a lot of these cases. And if you drop onto a single endpoint, if you're smart enough, you can grab cookies out of a browser for a bunch of DevOps engineers that then can be used in a bunch of different contexts and that is a problem in a lot of cases. And then people are not tying that to a hardware root of trust. Even though These are all $4,000 totally stacked MacBook Pros with hardware roots of trust and biometrics tied to them. But they're just in the end using cookies that are in the unencrypted Chrome store. Right, that are actually being used. Yeah.

[Patrick Gray]

I mean, I think the reason I keep coming back to this is because occasionally I'll be talking to someone and they'll say, well, what do you think we should do about this? And I just say to them, I have no idea, Alex, before the hook comes out Warner Brothers style to, to pull me off stage, we've actually got to wrap it up there. But that was a fascinating conversation. Alex Stamos, Steve Stone, thank you so much for all of that. That was great.

[Alex Stamos]

Thank you. Thanks, Kathy.