B (3:50)

Yeah, as you mentioned, it's not a new technology. And I think kind of to what you're alluding to, I started to notice as well in some of our intel reporting, in some of our blogs in the last several years, seeing it pop up more and more. Maybe before we get into kind of specifically how we're seeing threat actors target that maybe a little bit more about the sort of why. Obviously I imagine that there's a tremendous amount of flexibility that you're offered with this sort of technology. But what are some of like the workloads and sort of the reasons why organizations adopt this technology?

A (4:26)

It's a great question. So most organizations today, particularly enterprise organizations, will kind of have a strategy where they'll consume, I don't know, SaaS based services. And there'll be virtual machines that run in a cloud, perhaps aws, Azure, but they'd also have what's essentially on premise assets as well. So they would be typically in a data center. And to run those, what essentially are virtual machines, they need, need a hypervisor. And in most organizations you will see VMware and for good reason because it's kind of stable, it kind of just works. Yeah, I mean it's interesting that when I first started architecting VMware solutions many years ago and I kind of picked it up again and I didn't touch it for like 10 years or so. When I kind of accessed it and started playing around, it was exactly the same. And one of the reasons for that is it's kind of, it kind of just works. Yeah, it's a product that's stable, it's kind of dynamic and it, and it ticks a lot of boxes for a lot of organizations, you kind of have your virtual center server, which is a control plane. You have your ESXI host. There are some additions like nsx, which is kind of a networking layer. But there's kind of a big ecosystem of products around it as well, like your backup products, products for instance, like veeam and things like this. And it just kind of, for most organizations that run kind of on premise virtual machines and specifically Windows machines, it's kind of, you know, it's, it's really present in a lot of organizations.

B (6:04)

And maybe we could talk a little bit about the sort of why you would go after this as a threat actor. You know, why is this sort of valuable real estate? Obviously, you know, as you kind of note, there's a lot of flexibility, there's a reliability to run various workloads on this. But when you're now approaching this from the standpoint of a threat actor, what makes targeting this very attractive?

A (6:26)

Traditionally threat actors, specifically ransomware type and data exfil, the target would be kind of Windows type machines. Windows operating systems, you know, have, you can run EDR on them, you can run specific security products at that kind of level. You have organizations like ourselves going in and performing assessments that harden the infrastructure. So that whole layer, a lot of organizations will kind of pour a lot of effort and resources into hardening VMware layer, or should I say VSphere layer is quite different and for a couple of reasons. So it runs kind of a. So it's composed of two components. So you have VirtualCenter which is the control plane, which is kind of a virtual appliance. It used to be Windows based. And then you have ESXi which is essentially a custom kernel that runs a kind of Linux type interface. What is key is you cannot run EDR like or any kind of antivirus or anti malware type software within those operating system. So both ESXI and the virtual center are kind of a blind spot for a lot of organizations. So that puts a lot of onus on logging and detection engineering. And from our experience, both from proactive kind of incidents that we have to deal with where VMware or VSphere is involved. There doesn't seem to be that level of detection engineering or logging. So it's definitely a blind spot. So and the other key thing as well is that a lot of organizations will run kind of all in within VMware as well. So they'll have a VSphere cluster, they'll have perhaps a virtual appliance that runs their sim, they'll be running active directory virtual machines, they'll be running Linux virtual machines, they'll be running privilege access management virtual machines. So this becomes a really attractive target for threat actors should they compromise that kind of underlying infrastructure, because they can essentially take the whole infrastructure down.

B (8:37)

So it seems like at least two things that you have going on here are kind of, as you just noted, the criticality of systems that are running on this infrastructure, and especially for disruptive threat actors, like a ransomware or an extortion group, that becomes like a very attractive target.

A (8:53)

Exactly, yeah.

B (8:54)

And then you also have this dynamic, which reminds me a little bit of this increasing shift we've seen over the last several years of threat actors going after edge infrastructure for initial access, where similarly there's a problem with these technologies. Your VPNs, your routers, these don't typically have the capability to run edr, and so they become a blind spot for security centers.

A (9:18)

Yeah, exactly. And I think another thing that if I kind of take, and I keep going back to this, but when I was a kind of an ESX guy and I used to install, I just think at that time it was really around, you know, the question of security. There wasn't ransomware there and there was like malware. Most of the security was at perimeter, so like strategically placed firewalls. And I think what's happened over time is there's, there's. There's kind of been some assumptions around security that have been brought forward with a lot of these architectures. So, you know, it just works. We just leave it. I think a lot of organizations are not really aware of the risk. So kind of we have these VMware environments which, you know, they've been ticking along quite nicely over the years. There's been incremental upgrades, etc. But in terms of kind of having parlance with what's happening today from an attack perspective, it's just simply not there. So there's need to be kind of a strategic rethinking of some of these VMware architectures to make sure that they're kind of in line with the threats that exist today.

B (10:26)

Do you have any. You know, you noted that probably like the last four or five years, we've started to see the targeting of this more and more frequently. Do you have any theories as to why? I mean, again, this is not new technologies, but, but why We've seen this shift of adversaries increasingly looking to target this. We've talked a little bit about the ransomware actors But I know we've also seen for example Chinese espionage actors go after this as well.

A (10:51)

I was thinking about this the other day actually. I think it's kind of a perfect storm. So you know, you have, you know, they adapt, you know, they're seeing barriers, security barriers in place and when they try and compromise environments and they want to perform ransomware or data exfil, it's an easy target. VMware. So essentially, essentially speaking, if you think of VirtualCenters, the kind of brain it controls, all the esxi hosts and VirtualCenters link to an active directory for as an, using it as an identity provider, just one active directory user account that's linked to VirtualCenter which has perhaps, I don't know, control over 100 ESXi hosts, even with limited VMware permissions. Could mean complete takeover of your infrastructure and data xfield possibly as well, which we'll come on to in a minute. So it's, it's, it's, I think it's the ease and it's also the, the velocity as well of, of how these attacks unfold. Like typically speaking, we often see kind of dwell times in typical like Windows type ransomware environments of days, weeks, sometimes even months. What we're specifically seeing with threat actors targeting VMware environments is they'll have that kind of initial access phase where they come in perhaps through a vulnerability in a firewall, etc. And then they're basically looking for as part of reconnaissance, VMware type accounts that are perhaps linked to Active directory. They'll scan SharePoint sites, they'll scan repositories for anything that will kind of give them that golden ticket to the VMware environment. And once they're in, the ability to deploy ransomware is, you know, it's quite easy. It's basically some code they need to transfer to a host or some, or hosts and then execute that. And then because virtual machines are just files. Yeah, you can. That portability and flexibility that they give you also makes them very easy to be, to be manipulated. If it's a file, it's easy to be encrypted basically. So yeah, to sum up, it's really, it's the ease, the ease of being able to access the environment typically through Active Directory. And it's also, you know, kind of the speed and velocity in terms of being able to execute that ransomware or data exfil.

B (13:25)

The speed piece of that was one thing that really struck out to me. I think in one of the most recent blogs that you wrote where We've seen threat actors target VMware vSphere. But one other thing I wanted to ease out in there is you mentioned active Directory and the sort of role that identity plays in this, because I think that has been certainly a theme in a lot of the breaches we've seen over the last several years, whether it's the usage of stolen credentials to gain initial access, whether it's the usage of, you know, vishing the help desk, which of course has been a big player in some of the, the ESXi VMware type incidents we've seen this year. But it's. It strikes me as like that is something that even post initial access, where threat actors are looking for more privileged accounts that plays a role in the specific way that threat actors are targeting this.

A (14:17)

It's an interesting conversation actually, because I kind of alluded to it earlier, like you don't even need a privileged account to compromise and cause a lot of damage in a vsphere environment. Because if you think about, you know, we've got a basic VMware type administrator who's got a role in VM to be able to, I don't know, snapshot a VM or create a VM or copy a VM. Now a threat actor could basically use that privilege to perform data exfiltration. What we see now is an interesting kind of pivot from before. It would just basically be more kind of ransomware and bringing infrastructures down. Now what we're seeing is threat actors actually pivoting slightly and installing kind of backdoors into virtual center appliances and then also kind of in a stealthy way looking for perhaps virtual machines that a kind of tier zero. So typically that would be active directory. And essentially what they do is they would reboot the virtual machine and they would copy the virtual machine disk or snapshot it and mount that to another vm. If that's active directory, that could mean extraction of the active directory database. Once I have that active directory database, then they would transfer it out via their back door. They can, and then they can basically perform standard exfiltration of those credentials. Those credentials could include cloud, it could include backup devices. It could, you know, it can, it, it can be anything. But it's a really successful tactic in terms of being A, undetected and B, performing, you know, quite extensive data exfiltration.

B (16:04)

So we've already started to get into, I think, some of the examples, but maybe, you know, walk us through some of the incidents or examples of incidents where we've seen this either from the espionage side or as you reference also the extortion side, you know, you have a threat actor that gains access by, you know, maybe again it's phishing that the help desk to obtain user credentials, or they come in through a vulnerability in the firewall, or there's an employee who's spear phished. How are the threat actors kind of, you know, what are some of the ways we've seen them carry on from there?

A (16:34)

Yes, so typically it's kind of, it's kind of a standard type approach. So yeah, they will come in through perhaps a firewall vulnerability. They will then look to use that account that, that that has been compromised to then search for a VMware listed account. So typically a lot of organizations will have perhaps an aid and an active directory group called VMware admins or something like that. So then they would look for to either add themselves to that group or perhaps compromise another account via escalation that would have access to that group. And then once they're in that group, the core tactic is to take control of VirtualCenter. Yes, if they have control of VirtualCenter, they have control of all the ESXI hosts underneath that, that are controlled. So then we've seen, this is, I mean, this isn't specific, but we've seen over the last six, seven months or so that definitely they would look to perhaps compromise Virtual center in a way of installing that back door. So that back door is basically malware that's installed on the kind of shell of the Virtual center appliance. Once they have that, they will then look to enumerate all the virtual machines. So if you have a thousand virtual machines, they're essentially looking for those tier zero assets. They then perform the data exfiltration, and then that could be two hosts, it could be 20 hosts. And the key thing to note here is because of the lack of EDR and generally logging and detection engineering, with a lot of organizations, this is completely undetected. And then once they kind of perform those tasks and the command and control is set up, they would then look to transfer the ransomware binary to an ESXI host. So typically that's, they'd have to enable SSH onto a target host and then it's just a simple execution of a basic script. They would look to terminate the virtual machines before applying the ransomware. So typically this is where this will be the first instance that an organization will see that in inverted commons there's something wrong because they would see en masse a large amount of virtual machines being rebooted. But the issue Here is that, and we'll come on to this in a little bit is when this kind of alerts are kind of arriving in an organization sim.

B (19:08)

It's too late if you were to compare this to. Because when I first remembered hearing about this, one thing that struck me is similarities, at least in, in some ways, at least in my mind to how we would see threat actors again in the extortion space, go after GPO and use that as a way to disseminate ransomware throughout the environment. Or as we saw especially in kind of the early parts of the, the war in Ukraine, threat actors that were engaged in deploying wiper malware throughout an environment. And again, kind of group policy object being kind of the primary way they were doing that.

A (19:40)

Yeah.

B (19:41)

If you were to kind of compare this from a stealth perspective, obviously that being more of a Windows environment, is this something that typically has less signature and is stealthier, quieter than those?

A (19:51)

Yeah, absolutely. There's less ic. So the reason why there's that, that kind of. You've, you've raised a valid point actually. So that in terms of stealth, because they're completely undetected, it's kind, and this is, this is kind of a really volatile type scenario, typically. So they're undetected and they have complete control over the underlying infrastructure. And their, their goal is to typically perform the data exfil. That's probably what, you know, the scenario where it probably takes a little bit of time, but the ransomware, it can be executed. I mean, when we're putting together timelines, it's, it's, it's a matter of hours. And this is really what cripples a lot of organizations because typically they'd have AD as VMs. Their privileged access workstations are VMs. The SIM is VMs. You know, everything's a virtual machine. So essentially what happens is it's complete infrastructure meltdown. And in terms of rest, restoring that environment and recovery, it can be really, really painful as well because the whole, essentially the whole infrastructure will have to be rebuilt typically as well. The backup appliance would also run as a vm, so that would also be encrypted. This becomes, you know, a real problem for a lot of organizations in terms of any incident that involves VMware.

B (21:19)

And just for the listeners to put that in context, I think last year when we put out, or rather the M trends from this year that looked at last year's numbers, the median amount of median days of dwell time for ransomware operations was I think around six days.

A (21:35)

Yeah.

B (21:35)

Obviously you have some that's, you know, closer to 48 hours, but the fact that you're talking about operations that are now taking place in a matter of hours versus days is something that is a pretty significant departure from what we've historically seen with ransomware.

A (21:50)

Yeah. And this is again, coming back to your question earlier around, you know, why is this so, why is this so attractive? I mean, these, these TAs, they can get in, they can do this and they can move on to the next one. It's, it's, yeah, it's, it's easy work. If they can compromise a VMware infrastructure, definitely.

B (22:08)

So we'll get into some of the recommendations and like the hardening guide, some of the takeaways that you've written about this year. Yeah, but at first I wanted to ask again, I think this year where we've talked about this activity, at least publicly, a lot of that has been in the context of extortion, ransomware, either data theft or extortion or ransomware. But I know we've also seen this again in context, at least with some of the Chinese cyber espionage operations. Is there any sort of notable difference in how they are leveraging this when we see that sort of apt activity versus the extortion actors?

A (22:44)

Yes, that's a good question. So we don't see a lot of that, but it does happen and we find them. In my experience, it's probably around more of the more complex VMware environments that are. Perhaps their security posture is better. So they would look to deploy like for example, brickstorms are a good example. So they would deploy malware on the Virtual center server. Again, there's no EDR to detect that and they're looking to perhaps deploy what are called as basically ghost VMs. So essentially what they are is a virtual machine that's not registered in Virtual center or seen on a host. So kind of most organizations would be unaware that it's even running. But that VM does exist, it contains malware and they're looking for kind of persistence over a long period of time. But generally speaking, I think moving forward over the next months and years, as organizations start to be aware of the risk and threat around VMware, I think it will probably move down that road of more kind of persistent malware and maybe kind of an area that will probably see more kind of activity would be like virtual machine breakout. So that would be a vulnerability within a VM that breaks out to hypervisor and, and more kind of zero day type things. But essentially what we see today, and I'll be really honest here, is that from our experience of both proactive assessments where we kind of go in and understand an organization's posture and how things are put together, and also reactive, which is more like instant response and dealing with environments that have been compromised, a lot of the organization's posture that we see is very similar. So they'll typically have insecure defaults. They will have kind of security settings that have kind of carried forward over the years that have inverted commas, not really needed to be changed. From a threat actor's perspective. Yeah, I think it will move down that road eventually. But I think there's so much opportunity out there today for these environments to be compromised because they're simply not in the place that they should be in terms of controls. And also there's just. There's no visibility into what's happening into the environments due to the lack of edr. So.

B (25:13)

Yeah. And I think, you know, to make sure that we don't make it sound like this is a insurmountable problem to address, maybe let's talk a little bit about some of the recommendations certainly that you've written about this year that organizations, whether it's around hardening identity. You mentioned some of the logging that can be turned on and this being kind of a key function. What are some of the ways, you know, we don't have to get too specific, but maybe some, some of the categories of things organizations should be thinking about to harden this infrastructure.

A (25:45)

Yes. Really interesting question. So I think, I think the main thing is to, and we see this a lot is to understand the risk. Yeah. So. And part of that is acknowledging that the Hypervisor, so your VMware environment, it's a tier zero asset. Yeah. So you have to treat it like that. So in terms of like zero trust, privilege, access management, those kind of things. So it needs to be a very much a strategic approach. So understand what you have. You know, you're going to have a lot of organizations will have multiple sites, thousands of VMs. So audit what you have. Yeah. And identify them risks that exist in that environment. So if you have AD as an identity provider and that's by default will increase the attack surface layer, then perhaps remove that. Look at introducing phishing resistant MFA across the environment. So if you think of Virtual center is a control plane and ESXi, they both can be added to Active Directory. There's no need for ESXI to be added to Active Directory. So remove that and then you remove that risk and then you Start to centralize all management through VirtualCenter. Yeah, so if you centralize all management through VirtualCenter then you really want to make sure that that's completely locked down. So apply the principles release privilege around roles. We spoke about that earlier. And also if you can, and I know this is difficult for a lot of organizations if you're saying okay, because we've been saying use active directory, use centralized identity providers, but if you can use modern authentication, so you can use something like Azure Entra or something that supports phishing resistant MFA such as Fido 2, make sure you leverage the capabilities such as conditional access points policies so you can identify kind of context aware access. So yeah, okay, that uses him, but then what's he doing? Where is he logging in from? That kind of stuff. And then once you have that kind of, from a, from a high level strategic kind of planning in place, then kind of create a plan. Just coming back to that, that first part in terms of the strategic and understanding what you have and identifying where there may be gaps. Some organizations may not have the bandwidth or the skill so maybe leverage a third party to come in and come and understand and help you along and inform your risk, your risk identification there. That will kind of really help, I think. And then once you have that plan, you need to kind of need like a two fold strategy. So kind of have a plan to implement those controls. And then if you think about the big problem with logging, so okay, so you can't use edi. So what can we do to make sure that I can understand what's going in that environment? Because essentially what you need to do is use the controls. If you have identified via the kind of first phase when you're kind of understanding what you have, understanding the risk and then essentially what you need to do then is use those, any deviation from those controls. Like for an example would be perhaps a user logging on from a specific IP or device that's not allowed at Virtual Center. So essentially what you're doing there is your, any friction around those controls then create alerting for that and then feed that into your sim and then you can create kind of a different way and a different playbook to kind of deal with any potential anomalous activity. And then so you have that you, you, you've determined the risk, you determine the scope, you have a plan, you kind of implemented those controls and then you kind of need to understand how do you monitor and assess that. It's, it's different. I'll be honest, it's difficult for a Lot of organizations who've got big, large VMware environments to really understand what's going on there. Because most, most tasks, for example, we spoke earlier about like for instance, creating a snapshot or creating a VM that might be essentially a daily task. A management task could also be anomalous activity. So what is important there is you kind of create correlation rules with perhaps your identity provider and your ESX site host and virtualcenter. So if you create, if you create some kind of detection engineering that kind of bundles that all together, then that kind of will help you identify what is an anomalous activity as opposed to kind of standard type VMware admin type activity. There are some real quick wins you can do in terms of locking down ESXI host. So use lockdown modes, prevent SSH access. You can't deploy ransomware on an ESXI host unless you have shell access. There's also a really good setting that will prevent the execution of binary such as ransomware called exec installed only. It's just a simple switch. You have to enable it on all ESXI hosts, use kind of TPM modules, kind of get that secure boot every host in place and then once you have that, you kind of, you've kind of locked down your, which essentially your data plane, which is the ESXI host. So it's really difficult to kind of a threat actor, to kind of move laterally there. And then if you think about virtual center. So make sure that, that again, as we mentioned earlier, you're using MFA there. Make sure that there's limited access and users that can perform tasks that are privileged. And I think a really interesting one as well. And I think a lot of organizations are grappling with this at the moment is product versions. So a lot of organizations will be on Vsphere 7 and it's actually becoming end of life. I believe it's next month when you upgrade to a different version of VMware, which should be eight. You're increasing your posture just by default because the vulnerabilities are dealt with. You can have new features that are more security focused. So don't just use it as a kind of an upgrade path. Use it as kind of an inflection point to say, okay, we're going to, we're going to look at, look at virtual sensor, look at my ESXi host. How can I leverage these new features to perhaps increase the posture of my environment? So yeah, there, there is a lot to it, but it's not, as you say, it's not insurmountable and it's definitely something that again, if you, if you take a kind of risk based approach and you understand what's happening in your environment in terms of dealing with that EDR blind spot, you know, you, you have that comfort then of okay, if someone does access the environment, perhaps virtual center level. I know that I've built my detection engineering so that I will know what's happening. I can act upon an event. We spoke about this earlier as well, that it's. You don't want to be alerting that, you know, hundreds of virtual machines have just been shut down as a precursor to ransomware. You really want to be dealing with alerts that perhaps there's been kind of an anomalous logon at a specific time or there's been an enablement of SSH on a host or something like that, that's really what you want to be logging on.

B (33:39)

There is a lot. But I do like the way that I don't remember if this was in one of the hardened guides or the webinar or both, but you kind of note taking a phased approach to rolling some of these things out across identity endpoints, et cetera. And so I think there's the organizations that are looking about, you know, where do I start with this? I think there's a very good crawl, walk, run kind of approach that you lay out as a strategy.

A (34:01)

Yeah, definitely take a more strategic approach. So what we essentially want to do is implement, understand the risk, implement controls typically as well. It helps if you can, if you can use a baseline like for instance assist benchmark for VMware or the VMware Technical Security Guide because they will give you an understanding and inform your decision making around those base controls. So, and then once you have those controls in place, then essentially what you're doing is you're alerting on friction of any deviation of those controls. And then for the. Once you have all the kind of a, kind of a base, a baseline security posture, then of course you can move on to some of the, some of the more strategic and technical stuff like use threat modeling to understand specific fractures and how they operate and kind of use your baseline for your standard controls. But there may be different specific controls that you need to implement that perhaps understanding how one specific threat actor would work would help you mitigate that. A good example is we spoke about this earlier of the ability to data exfil just by simply shutting down a vm, copying the VMDK to another VM and then taking the if this was active directory, the Active directory database. Now you would never see any control recommendation in CIS or the VMware Technical Security Guide, but if you threat model that, then you can understand that there is a way to prevent that. You can use VMware encryption. It's impossible to copy the VMDK from one virtual machine to another if the VM is encrypted. So that's kind of understanding from an attacker's perspective kind of these TTPs and applying the appropriate controls.

B (36:00)

Well, I think we've covered this pretty extensively. I'll include a link in the show notes to some of the blogs I mentioned that go further in depth into this. Stuart, this has been fantastic. I think this has been a great kind of exploration of why we're seeing and how we're seeing threat actors go about targeting this. You touched on this already a little bit. But any sort of like closing thoughts here about where this as a category of activity will likely go in the future, in the next several years? I mean it seems like this technology is here to stay at least for some while. It's reliable, it's dependable. How do you think actors are going to evolve around targeting it?

A (36:37)

Yeah, so interesting question. So I would say that this is just a start, if I'm being honest. I mean if you go into most data centers for a lot of organizations that are, that are running on prem, it will be VMware. Yeah, let's be honest here. I personally, I mean if you see the way it's kind of evolved over the last six months to a year with the kind of specific move away from opportunist to kind of specific targeting of these environments and also the, the kind of shift to data xfield of kind of tier zero type assets. I think this is just gonna, it's just going to, I mean this playbook's effective. Yeah. You know, threat actors will, if it's effective and it works, they will use it. And I think it's just, I think the key thing for me is like, okay, a lot of Organizations are on VSphere 7. It becomes end of life. You have to move to version 8. So use that as a chance to kind of assess your risk, understand where those gaps are. Understand that, yeah, you can't use edi. It's a visibility gap in your environment. Understand that if a threat actor does have access to your environment, it is compromised, that you're going to have widespread impact infrastructure damage. Yeah. And, and kind of treat the whole environment as tier zero. Understand the risk and kind of use the next months maybe into next year and, and the kind of migration to, and transition to vSphere8 as kind of an opportunity to go, okay, let's look at what we're doing, let's look at this risk and let's implement those controls because unfortunately, you know, I can only see this getting a lot worse.

B (38:27)

Well, I think it's a great point to end it on. Somewhat sobering as always, but that's talking about anything in cybersecurity. Stuart, thank you for your time today. This has been fantastic.

A (38:36)

Yeah, thank you. And it's been really enjoyable time. Thank you very much.

B (38:40)

Take care. Sa.