← The Defender's Advantage Podcast
Loading summary
Transcript26 lines
- [00:03]
A
When we look back at UNK 5221's activity and the post exploitation activity in early December 2023 and compare that to the most recent activity, I think it's been a very, it's been a very apparent to me that one, they have a very good understanding of how the appliance works and the nuances of its behavior, and two, they've adapted and learned more as time went on.
- [00:40]
B
Welcome to another episode of Mandiant's Defenders Advantage podcast. I am your host, Luke McNamara. Joining me today I have Matt Lin, who's a senior incident response consultant here at Mandiant, and Daniel Spicer, the Chief Security Officer at Avanti. Matt. Daniel, great to have you here this morning.
- [00:59]
A
Hey Luke, thanks for having us on the podcast today. Really excited to be here.
- [01:03]
C
Absolutely, thank you.
- [01:04]
B
I think from that intro you probably have guessed we're going to be talking related to some of the research and blogs we put out around a threat actors exploitation of the Avanti VPN solution. And I think kind of putting this in larger contexts. As we get into the discussion today, we have the entrance report that just came out that talked about exploits continuing to be the most common way we see threat actors compromise and gain access to enterprises. When this report comes out, we should have our zero day report which will kind of go further into that. But in particular we're continuing to see this theme around threat actors focusing on edge infrastructure as a means to do so. So we're going to talk about some of the specifics of this case that you were involved in working Matt, with Ivanti, but then also I think put it into the larger context of what's happening into the threat landscape and why we're seeing threat actors do this. Perhaps to begin with, we can start with a little bit about this threat actor onc5221 and kind of how they fit into this, the story of what you've been kind of focused on with researching and responding to some of their activity.
- [02:10]
A
Yeah, for sure, Luke, thanks. I think that's a great place to start. So earlier this month, in April 2025, we published a blog post that details the exploitation of CVE 2025 22457, which is a buffer overflow vulnerability that would essentially allow for unauthenticated remote code execution on particular versions of avanti connect secure VPNs. And we saw this threat actor exploitation multiple high value targets beginning in mid March of 2025. And we attribute the activity to ANKH 5221 and ONC 5221 is a Chinese espionage actor. And we actually introduced them for the first time in our four part blog series last year titled Cutting Edge that also talks about Avanti Connect secure exploitation. So it's almost been a year and a half now and it's really been interesting to see how they shifted their targeting and also evolve their tactics to keep pace with defenders in this most recent exploitation campaign of the appliance. And following successful exploitation, we saw the deployment of two new malware families that we're tracking as Trailblaze and Brushfire. And these two families sort of work closely together to ultimately create an in memory only passive backdoor on an appliance. So instead of a traditional backdoor that may potentially call out periodically to a threat actor C2 server passive backdoor would only activate when the threat actor sends a very specific sequence of bytes to the compromised appliance, which helps it remain more stealthy. I remember finding the malware for the first time and just doing some basic triage and I'm picking it apart and just thinking like, wow, this is quite minimal and it's almost elegant in a certain way. Right. It's written in plain C, it uses raw syscalls, and it does exactly what the threat actor wants it to do and no more. And I think the file size for one of these was less than a kilobyte in size, so certainly very impressive what they've done in order to fit that payload into their exploits. And I'll spare our listeners the rest of the gnarly details here, but we do have some more technical details in the blog that I think are a little bit easier to digest visually and
- [04:42]
B
we'll include some links in the show notes to those blogs. Yes, because as you noted, there's a lot of technical details about this activity. Maybe as a follow up to that though, in comparison to some of the activity noted in the Cutting Edge blog series last year, were there some interesting novel techniques that we saw the threat actor do? In the activity noted in these two
- [05:01]
A
blogs we see ank5221 and other actors as well use some pretty interesting techniques that involve custom malware. They Trojanize existing files and scripts on the appliance. They do living off the land, evasion, anti forensics. Right. And all the things that you would expect from a sophisticated adversary. And when we look back at 5221's activity and the post exploitation activity in early December 2023 and compare that to the most recent activity I think it's been very apparent to me that one they have a very good understanding of how the appliance works and the nuances of its behavior, and 2 they've adapted and learned more as time went on and as Defenders and Avanti are responding to these incidents. And they've certainly gotten better at evading detection. They've gotten better at cleaning up after themselves and also finding new ways to persist on the device. I think one of the more interesting techniques that I've seen is a particular malware family called Phase Jam, and we blogged about this in the January 2025 blog from this year with easier exploitation of CV2025 0282. Phase jam in particular has a capability that would block system administrators from performing upgrades or install patches on a compromised appliance. And not only would it silently block the upgrade in the background, but it would also actually render a fake HTML progress bar to the administrator to kind of give the appearance that the upgrade was actually in progress. So the upgrade attempt itself is blocked, and any tools or back doors the actor left on the appliance would remain on the current version of the VPN while giving appearance of a successful upgrade. So if you were an administrator performing the patch or the upgrade, you may not notice that something was wrong. And earlier I mentioned that ONC 5221 has demonstrated that they have in depth knowledge of the appliance, so they use sed. So thinking about living off the land here to sort of surgically insert their malicious code into a legitimate perl module on the appliance that would support the installation package process and that would intercept the function call for the install. So literally in the code that they inserted, you would see them calling the sleep binary to pause execution, append a period to the HTML, sleep one, append a period, sleep one, append a periodOR, etc. And it would display text like finalizing installation when nothing is actually happening in the background. So certainly very interesting.
- [07:53]
B
Yeah, I thought that was one of the interesting aspects of the blog, because of course with a lot of this targeting of edge infrastructure, Whether it's routers, VPNs, et cetera, one of the things that it provides to the adversaries, they don't have to do social engineering like they would through a spearfish, but it was interesting to see you could almost say a form of social engineering in that sort of display of fake updates and making it look like something is happening to administrators. So I thought that that was an interesting kind of piece where they were still thinking through again, how they sort of could manipulate or give sort of a false impression of what's going on to administrators. Daniel, I think to kind of pull in your perspective here, maybe start with, you know, how was this initially discovered? What were your thoughts in when you were kind of initially looking at this activity and what the adversary was doing?
- [08:41]
C
Similar to the last several attempts against the appliance since 2021 when we introduced it, we have been able to discover the activity through what we call the integrity checking tool. And I really like the integrity checking tool as a piece of the defender's toolkit that we introduced for our clients. I know that there's a lot of attempts by the threat actor to try to circumvent the integrity checker tool. So that means we know that it's a problem for them to circumvent, but also it just continues to be successful because it requires the threat actors who are trying to circumvent it to have perfect operational security. And we know ever since Apt 1, the landmark report, right, threat actors do not have perfect operational security. I find that to be a really interesting way that we can continue to put pressure back on threat actors and make it easier for us to detect the fact that they're burning zero days early. I think the other really interesting thing, really looking at the activity from March, April, time to exploitation is really short. This was actually an item that Avanti had identified as part of our continued commitment to secure by design as a memory corruption issue. We originally thought that it wasn't possible to exploit due to a lot of limitations on the ability to the types of characters that could be put it into, into this, into this heap overflow. And then come six weeks later, after we resolved the issue, we saw attempted exploitation by the threat actor. And that's a very, very short time from Avanti fixing an issue, not disclosing it as a vulnerability. So somebody actually having to go in and identify this, this bug as a diff and then determine how to circumvent the character restrictions, the ASLR and several other hoops in order to obtain successful exploitation. This isn't the activity from five, 10 years ago where it was a straight command injection buffer overflow that we're used to seeing. We're seeing a significant increase in the sophistication and just the time to exploitation. I think everybody needs to take that very seriously.
- [11:05]
B
Could you describe a little bit more about that tool and sort of how it fits within security you put around the vpn?
- [11:13]
C
Yeah. So the ICT works very similar to what you would imagine as a file integrity monitoring solution for traditional on prem deployments. There's a lot of limitations on hardened appliances in terms of the capabilities we can do and the performance that we can eat on the system. So the ICT doesn't work quite the same way, but essentially what we're looking for is when the threat actor wants to make a change on the system. So anytime they want to introduce a web shell or basically anything to be persistent on the box or persistent across upgrade, those are the files we check with the ict.
- [11:51]
B
And zooming out here a little bit, you know Matt, your perspective looking at and investigating these particular CVEs, but then also more broadly this trend around edge infrastructure, what are some of your thoughts and your takeaways around this as a trend, as a pattern, not just with this specific threat actor who's obviously shown an interest in going after VPNs, but just more broadly?
- [12:15]
A
Yeah, great question. And I think in our most recent editions of the Zero Day report that we publish, I think what's really notable in there, when we look at exploitation, and particularly Zero day exploitation, we see that China has exploited more Zero days than any other nation state in the past few years and that the prc, they continue to be a leading driver in government backed exploitation and also other various cyber operations, especially for the purpose of espionage. And when we look at the politics and the guiding structures within the country, it makes a lot of sense. But to kind of answer your question about edge infrastructure, I think it's interesting to kind of look at why, and there's a few reasons why, but predominantly I think number one is that edge infrastructure often sits in an elevated position within a network. Oftentimes they are the gatekeepers. Right? We're talking about VPNs, firewalls, routers, different types of application gateways. And you could extend this to things like virtualization platforms as well. And if you compromise one of those devices, it's likely already highly privileged. It may be store some credential material and does some sort of traffic monitoring or brokering within an environment. So if you can figure out an exploit for one of these appliances, one of these edge devices, you essentially now have access to every vulnerable appliance exposed to the Internet globally. But what's interesting is that these gatekeepers often don't get the same level of security attention or investment from defenders compared to things like end user endpoints or things like your Windows Active directory domain controller. You know, that has traditionally been your crown jewel within the classic archetypal view of Enterprise cybersecurity. And you know, I think that's where we're starting to see the conversation starting to shift in recent years as an industry, I'd argue that we've gotten really good with endpoint security and you know, we're starting to see a decreased reliance on endpoints for initial access in the past five or six years. And I can think of a few reasons why. But you know, we have more detection and monitoring capabilities than ever before. EDR solutions are becoming more and more popular and often seen as a default in many enterprise environments today. And vendors, you know, they're putting more investment into making operating systems more secure. But, but right now I think the challenge is that much of the industry is lacking parity on how to secure the edge and also how to respond to edge compromise. And I think it's also worth pointing out that it's not just UNK5221 and Avanti. When we look at other Chinese actors like UNK3886 that have just accumulated a wealth of portfolio of exploits against numerous appliance vendors, or the recent headlines talking about Volt Typhoon or Salt Typhoon compromising US Critical infrastructure or American telecommunication companies, I think, you know, those actors, they're really taking advantage of the sort of technical disparity and I think that is exactly what they're after.
- [15:32]
B
Yeah, I think one other data point you just made me think of from the mtrans report too is also the continued decrease in phishing that we're seeing as an initial access vector, which I think also points to probably that becoming a harder target as there's been investments around sort of phishing prevention and detection. So similar to EDRs, you know, that being an avenue that threat actors aren't leveraging as much and then focusing elsewhere. And I think also Daniel, I'd be interested in your thoughts on this, kind of expanding on, I think a point you made earlier, maybe it was Matt, but the fact that the threat actor really seems to understand and put some effort and resourcing behind understanding these technologies, this is something that we seem to be seeing with a lot of Chinese APT activity where whether it's stuff related to some of the salt typhoon activity, some of the other edge infrastructure compromises, even some of the Juniper OS stuff that we blogged about recently, we see time and time again that there's a very in depth understanding that the adversary has on the technologies. So thinking about that, without going obviously into the secret sauce, things like the ICT tool seems to play a role in sort of how you think about defense, but maybe just more broadly with this focus by adversaries and knowing that this is going to be an area of continued focus for them, how do you think about securing this from a design standpoint?
- [16:54]
C
I think one of the key things that I like to add into how we do threat modeling, which I think originally actually came from Microsoft, is just the assumption that the code is available to the threat actor. Right. I don't think there is a way for us to prevent the threat actor from finding a way to get their hands onto a piece of technology and spending the time reverse engineering it. So with that in mind, I think the threat model just has to assume that. Just assume your source code is available, right? And so assume that they can inspect the source code for vulnerabilities. And if that's the default base of your threat model, right. Then you start talking about how do I eliminate vulnerabilities, how do I create checkpoints, how do I make it easier to disable certain functions in the appliance? And I think that is kind of a key thing that I encourage everybody in the software industry to just assume, don't try to assume obscurity because the code isn't available. And I think a lot of cloud vendors, especially now, who are essentially edge. Right, are failing. And considering that eventually the threat actor will obtain access to the ability to just inspect whatever the piece of software is and be able to find a vulnerability in it. So you have to. You have to make that assumption. You have to go and secure the code, and you have to make it easier to put in mitigations and keep the service running, despite the fact that there may be a problem in the future. Because I think vulnerabilities in general is not an if but a win statement. And so reducing the impact on customers is obviously really important to that extent. Of course, we've been making a lot of investments since we acquired some of the solutions to Avante, including what is Legacy Pulse Secure. I shared an update on where we are in these commitments back in Friday, I'm sorry, back in February, in a blog post. A lot of it involves getting the right resources into play, using those resources to help create these choke points, eliminating certain classes of vulnerabilities, and then very important is just leaning into transparency and letting customers know when we have an issue in the product that we have fixed so they can go and do the update. And that is actually one of the hardest challenges, I think, in the industry is a lack of transparency about when a vulnerability is fixed. And I think this recent example in April where we actually did a misclassification of a vulnerability is a perfect example of how end a exploitation gets taken advantage by threat actors. So that transparency is just really, really key.
- [19:48]
B
Yeah, if you want to expound further on that, that particular point, essentially we
- [19:53]
C
identified the vulnerability as part of our continued attempts to, or continued progress to eliminate classes of vulnerabilities, starting with memory corruption issues. And so we found the issue, we evaluated it, we determined that because of the character limitations, because of binary protections such as aslr, a combination of factors, that it wouldn't have been exploitable. And where we ended up was we patched it as a, as a bug because it still causes a crash. Right. We patched it as a bug, we fixed it in the solution, and threat actors came back six, six weeks later and found a way to circumvent what we thought was just not possible to exploit.
- [20:39]
B
So kind of tying this together with the lessons you learned coming out of this. And I think one of the things that's very apparent, Matt, from what you noted earlier, that this category of threat actors is going to be continued focused on edge infrastructure. There's advantages that they've been able to gain and kind of see the result of as a TTP going forward. How do you think about how organizations can implement security around this? I know a piece of this is logging and having the right logs in place to detect this early on. So maybe curious both of your thoughts of how we will see kind of the adversaries adapt, but then also what are things that organizations can do to properly implement security here, whether it's making sure that they're rapidly applying patches when there are releases available. Maybe Matt will start with you and then go to Daniel.
- [21:29]
A
Yeah, that's a great question. I think one of the challenges of securing edge devices is that oftentimes they are running on proprietary platforms and technologies. So it can seem like a black box. And if you wanted to say, get your favorite EDR agents on one of these appliances, you know, it's likely not going to be possible or very, very difficult to do so. And I think, you know, security shouldn't be done without context or without kind of taking risk into the equation. And so definitely stay on top of the latest trends and exploits out there and any guidance or mitigations, you know, best practices that the vendor may already have out there for this specific version or type of appliance that you have. And I know today we've spent a good amount of time talking about zero days and end Exploitations. Right. And I think that's also really interesting because when we look at how attackers are getting into organizations in the past few years, there's some really interesting trends. Earlier you mentioned that back in 2019, 2020 and before that, phishing was the number one intrusion vector that Mandiant saw. When we look at intrusion vectors in the investigations that we were called in to perform, but in recent years and recently, we're really talking about pandemic and post pandemic era where things started to change quite drastically. Where in a majority of cases, patient zero, when we know patient zero in an investigation, it was the results of exploitation of a vulnerability. Not always a zero day, but it can also be an end day as well. But I think for the average enterprise, I really do think that most organizations can focus on security foundations and also just security fundamentals and put their energy there instead of being, let's say, overly reactive to the latest zero day or headline article. There's no panacea and security, but I think the fundamentals are often where you can kind of make the most impact in your overall security posture while making the best use of the finite resources that you have. So we're seeing this transition from exploits being the number one intrusion vector. It means that things like having a good vulnerability management program and also a good patch management program are going to be in the important conversations that we revisit again. Right. It's not new and it's not the latest or shiniest concept out there, but it is important. And attackers are not necessarily going to burn zero days against organization if there are easier ways to get in.
- [24:17]
B
Yeah. And Daniel, any thoughts as well around that either on, you know, if you're a user of edge infrastructure and products, as most organizations are, but then also for other vendors as well. You know, what are some of the lessons? I think we've kind of touched on this a little bit, especially the transparency component. But things that you think were really important takeaways coming out of these that you think other vendors can learn from.
- [24:40]
C
Yeah, I want to start on the, on the user side, just double down on the importance of vulnerability management. Many vendors have security advisories, have ways to share information. If it's a critical edge asset, you should be subscribed to those. Right. Make sure that you're paying attention to those things early. I'll also just share, not just Avanti, but a lot of vendors in this space where we have edge appliances provide a lot of information for configuration and misconfigurations. Like Matt said, if there's an easier way to get in. Threat actor will misconfiguration is definitely the way that it becomes a little bit easier, but also it exposes you to some vulnerabilities in a higher risk manner than would be if you had configured the appliance correctly or you were using it in the way that a vendor had recommended. Very common issue that we see a lot of people talking about is just exposing things onto the Internet for an edge appliance that were never meant to be exposed to the Internet, especially like administration panels. Again, a lot of vendors in this space are trying to continue to advise the network administrators, the security teams to check for those things and make sure that's not the case because it makes vulnerabilities that would otherwise be much more difficult to exploit very easy for threat actors. And then I guess for, for the, for the industry peers who may not have had a challenge yet. I'll just share the. Transparency is very critical to helping your end customer avoid risk. I think that sharing vulnerability details, registering as a CNA, reserving CVEs for the issues, make it very easy for the customer's vulnerability scanners to identify are they on the latest version? Right. So you're not just relying on the network administrator, but you also have the security team and the infrastructure teams looking at that. Hopefully that, that'll help protect, prevent them from, from exploitation, whether that's nation state or a ransomware threat actor. Because eventually this stuff does roll downhill where we start with nation state threat actors and eventually ransomware actors catch up and start using the same vulnerabilities. So I think those are, those are the key things that I would really want people to take away from this.
- [27:05]
B
Yeah, that's a fantastic point to end it on. And I think, you know, especially realizing, I think conversations like this are important, you know, in sharing kind of these insights as we collectively as a security industry, you know, face this as a continued problem. Threat actors looking to exploit edge infrastructure. To Matt's point, it's, it's unlikely to go away anytime soon. So I think conversations like this hopefully are helpful for the broader security industry. And I thank you both for being here today and helping do that.
- [27:33]
A
Yeah, thank you, Luke.
- [27:34]
C
Appreciate the time. Thank you.
- [27:35]
B
Take care.