Loading summary
Transcript81 lines
- [00:00]
A
Foreign. And welcome to another edition of the Wide World of Cyber podcast. My name is Patrick Gray. Wide World of Cyber, of course, is the podcast that we do here at Risky Biz HQ with Alex Stamos, the former CISO of Facebook, former CISO of Yahoo, founder of ISEC Partners, and now he's back in startup land. What is it now, Alex?
- [00:30]
B
I am the Chief Security Officer of Corridor Fantastic.
- [00:34]
A
And we are also joined by Chris Krebs, the founding director of cisa, co founder with Alex of the Krebs Stamos Group, currently fun employed, I guess. Enemy of the state. Hold on now. Come on.
- [00:50]
C
That's.
- [00:50]
B
That's like.
- [00:51]
C
I hope my mom's not listening to the podcast because she would not like that very much.
- [00:55]
A
I am.
- [00:55]
C
Let's put it this way, he's not.
- [00:56]
B
The enemy of the state. He's just disliked by the state. I think you would say he's the frenemy of the state.
- [01:01]
C
I'm in stealth mode. Let's just put it that way.
- [01:04]
A
All right, so stealth mode. Stealth mode. Chris joins us as well. Currently, this podcast is unsponsored. If you are interested in sponsoring this podcast, please do reach out to Salesky Biz if you are brave enough. So today we are going to be Talking about the F5 compromise, which of course came to light last week, as it turns out, and of course, details are still coming to hand. We're recording this on Tuesday, 21st October, but it looks like we had a Chinese APT group inside F5 for something like two years. They were doing things like raiding the internal bug tracker, looking for bugs and whatnot. So, of course, now that F5's discovered this, they've rotated a bunch of keys, which is interesting because those keys should be in a hsm, and this rotation kind of implies that they're not. And they've also announced patches for something like 44 bugs. I actually want to start with you on this, Alex, because I've been through the bugs that F5 is patching. None of them look particularly serious. That said, all of them described as dos, conditions, et cetera, look like they actually do have the potential to be quite serious. You being the sort of hard tech guy on this podcast, what was your read on the bugs that F5 actually patched in the wake of this whole thing?
- [02:27]
B
Yeah, so they're rated low so far. These are also exactly the kind of flaws that I would never bet my career on somebody not being able to turn to be exploitable. What we have seen over the past several years Is high end threat actors really liking to go after the sealed network devices because they are identical to each other, you cannot run EDR or other kind of security products on them. So and often they lack good anti exploit technologies. So if you do have a use after free bug, if you have a buffer overflow, if you have some kind of memory management flaw, the odds of you having an exploit that you can make reliable is actually quite good. And they are the kind of like you said, they are described generally as dosses. I would not, I would trust F5 as far as I can throw them metaphorically at this point. So I would not trust them at all. Fortunately, you can get their source code now pretty easily. So you know, there are a number of people who are reversing out these bugs themselves and trying to figure out from the patches. So I expect that we will figure this out pretty soon as well as probably start to see more exploitation in the wild.
- [03:51]
A
Yeah. One thing that initially occurred to me when I heard that they were patching a bunch of bugs and everyone's talking about how these F5s are all at risk is I thought, geez, are all these people exposing the management interface of their F5 to the Internet? Because man, that's a really risky thing to do even without these bugs. But then when I started diving into the detail of the advisory, there was a line that keeps popping up over and over and over in these advisories and it was undisclosed traffic could cause this condition, you know, could cause this, this process or service to terminate. And I was thinking undisclosed traffic. So wait, all you've got to do is get the right bits to hit an interface and you could trigger this condition. That's what made me think, oh, okay, this might be quite serious.
- [04:35]
B
Yeah, and it's, it's not totally clear to me. I mean obviously the F5s include WAF functionality, right? So they're doing deep inspection of everything that goes past. And some of this stuff does include a lot of it is like you said, management interface. And you should absolutely never ever, ever, ever, ever be exposing the management interface of any kind of sealed network device. Those things should always be extremely highly isolated and those management interfaces should only be exposed for the minimal time possible to be doing those that kind of management. You should be VPNing in honestly into those VLANs to be doing that kind of maintenance. But yes, some of it, the language here is, does hint towards it. Some of it is probably exploitable from in band traffic to the applications that are behind the F5s themselves.
- [05:28]
A
Nice real nice. We love it. We love it. Okay, so Chris, what's your take on this whole thing? I mean, so it's bad, but one thing that I found interesting about this whole thing is they had access for two years, including access to the update infrastructure, right. Where F5 prepares and ships the patches. But there's no evidence that they actually tried to ship a malicious patch. And I think that's very interesting when we look back over similar supply chain compromise incidents like SolarWinds is the one that comes to mind, they did manage to ship out malicious updates that would pull malware once they had update to similar infrastructure at SolarWinds. Why do you think they didn't do it in this case? Is that because they, that's how they got caught last time?
- [06:14]
C
Well, it was not them. Right. So first things first. Right?
- [06:17]
A
Like that was the, was that the Russians.
- [06:20]
C
That was. Oh, how soon we forget.
- [06:22]
A
Yeah. Amazing.
- [06:25]
C
Look, I mean, maybe we're making a couple of assumptions here. One, I know they say state actor, and then the rumor mill spits out China. And I think it's probably accurate. The second is how long they've been in. Right. So F5 talks about sometime in August of 2025, and there's some other kind of jumbled reporting that says, you know, it looks around a year. And there's other reporting that says maybe it's two years. I have heard that F5 knows exactly when it happened, and they're providing that information in single soul engagements, not broadly.
- [07:04]
A
That's, that's, that's what's leaking, and that's what's leading me to suggest that the initial intrusion occurred in late 2023. So about two years ago.
- [07:11]
C
Yeah. And all I'm doing here is like, let's just kind of like stitch together the context around this one. Because as much as we like to think these things are cut and dry, they're not. And they just kind of come dribbling out. So maybe there's a comms lesson learned here as well. And we've seen a bunch of that, so why not? I, you know, it, it, it's China, whether it's a contractor or it's MSS or whatever. I, I don't know if I know yet, or maybe someone knows. It just hadn't been shared yet. Maybe they're, they're that much more deliberate. Maybe they weren't in the right spot yet. Maybe they didn't have a payload. They were, they were ready to, to ship. But it also could be like, this stuff's hard and and maybe they just weren't ready to go yet. If the background story is true, if the rumors are true that they had the appropriate logging in place to kind of know when they got in and see some of the movement, then maybe F5 had a better than we would have expected security posture. Given where they sit in the kind of the mid cap space. They're not Microsoft. Microsoft's had their own set of issues. Aws, Google, all those guys. So I think there are a bunch of different possibilities but I do always keep coming back to that private equity backed mid cap enterprise software space. And you know, if I'm a threat actor, I'm trying to paint these guys up because and I'm not saying this is the case with THAT five at all. Saying like that is a super attractive target set because of, you know, the way private equity really looks to maximize profits out of some of these companies. And maybe they're not investing to the extent a larger security company is. So just kind of some immediate thoughts about where they are. And there's a whole other set of issues that we can talk about. Just like what's it like to execute the cleanup and the mitigation set particularly in places like the US federal government right now.
- [09:06]
A
Yeah, I mean F5 we should point out is actually a listed company. It's kind of an interesting company in a lot of ways because they make know these load balances and things that help you do hybrid sort of on prem cloud based stuff like they make some fairly unique equipment which is not known for being particularly secure. They also took on NGINX as well. So just, just an interesting sort of mid tier company. We actually took them on as a sponsor years ago just to find out more about what they were doing on the, on the nginx side. But they are not a PE backed company.
- [09:40]
C
A bunch of institutional investors though, not that that kind of makes the impact. I will say they did something really interesting though. They from a personnel perspective and that's, that's always something to keep an eye on when you have these big events. What happens on the, the personnel front? They took a board member, a board member of F5, a former CISO of Equinix and a bunch of other places. Guy named Mike Montoya, he was a board member and they brought him down. He left the board and he came down as the chief technology and operations officer or something like that. But it's a really interesting move by the board to kind of like send a shock troop in there, get their arms around what they understand is A really significant problem for them. Branding wise, reputation wise, engagement with customers. I mean, Alex, we've seen this before in places like SolarWinds and Avanti. It's just some of these decisions you make right out the gate are huge in terms of stabilizing the company itself and kind of the perception of how you're responding.
- [10:44]
A
Mike Montoya was on the show years ago talking about the Equinix breach and you know, definitely a sharp guy, definitely liked him. That's a, that's a great interview, that one. But yeah. Alex, what do you think of this response, right. Of you know, getting someone from the board to actually go into the trenches while all this is going on? I mean that does, you know, to Chris's point, that does seem like kind of a good thing to do in this situation.
- [11:07]
B
Yeah, I mean it was a quick move. I think the other thing that they have said so far that we'll see if it changes. F5 does not just ship pizza boxes, they have cloud services. Right. So this could get much worse if it does turn out. One of the reasons why you would want access to F5 over a multi year period would be access to their broker services, right? So their distributed CDN and their cloud services, which if these actors did have access to those services, that would be an ingredient, access into many, many the back end of many companies. They are saying right now that there is quote unquote, no evidence of access to those systems. We have all seen that change. Right. And especially since those services obviously use base F5 products. So if they had access to vulnerabilities in those products, it is pretty early in the incident response cycle to be making those claims. So that would be another reason why you would see them not doing something as obvious as backdooring patch cycles. Because the big victory here might be access to those customer consoles that are already hosted in the cloud and then utilizing those basically the zero trust access broker mechanisms to.
- [12:29]
A
But I believe all of that is begat from F5 picking up nginx, which is why I mentioned it earlier. Right. So it's almost like a completely different company. So I'm not prepared to speculate.
- [12:40]
B
Yeah, I don't know how much of that is nginx versus f5 proper.
- [12:44]
A
Yeah, well we don't know.
- [12:47]
B
I mean it's interesting because F5 is like this consolidation of a bunch of different. It is like, like Chris said, like some of these PE backed firms where you have a bunch of different sub brands that get smushed together and then it's extremely hard to tell from the outside of where the technical lines are on the inside. Because from a branding perspective, they try to make it all look the same. From an attacker's perspective, they're all probably quite different.
- [13:10]
A
Yeah.
- [13:11]
B
And you know, and once you get on the inside, it actually gets quite hard probably to. To cross those administrative domains because the internal integration's actually probably gone quite poorly.
- [13:20]
A
Yes, well, I mean, that's kind of what I was driving at, which is to expect that they would be. Yeah. Integrating the old part of the business and the new part of the business. Doing. Doing that. Well, I can. Providing those sort of opportunities to attack is. I'm a little bit unsure about that. So we will get into that sort of private equity led, you know, mid cap vendor discussion in a little bit. But more immediately, you two were heavily involved in SolarWinds response to their incident. This was a Krebstamos group gig. When you went in there, you know, what did you do in there that worked? What was the. Where. Where was the emphasis in that response and where do you think F5 should be focusing?
- [14:01]
B
Yeah, I mean, that was our first project that was actually kind of pushed us off the bench. Chris was tiny, trying to actually take a little bit more time to spend with his family and decompress from his very.
- [14:15]
A
From his running. His first run in with Trump.
- [14:19]
B
Yes, exactly.
- [14:20]
C
And Alex, you were trying to write a book.
- [14:23]
A
Can you finish that?
- [14:24]
B
I was. Which is still not done.
- [14:27]
A
Right.
- [14:27]
B
I mean, it's still, it's partially up. Right. If you go to tsbook.org, our trust and safety textbook is partially done and we continue to release chapters as they get done. But yeah, so basically four winds happened and they had a new CEO coming in who the CEO had. He had accepted the job, not knowing that they had been secretly breached. He finds out that this breach was being announced and he says, okay, I will still come, but you have to let me bring in my own people. And he calls us and says, hey, will you be my guys to help me navigate this? And we formed the company and came in to help with the response. And so it was a big effort. There was tons of people helping on the forensic side, but Chris and I were kind of on the. Really focused less on the hands on forensics for which there, you know, lots of skilled people and more on the corporate recovery of how do you rescue this company and, you know, retain some enterprise value out of what could have been a complete and total write off of the organization.
- [15:31]
A
So what was the advice, Chris?
- [15:33]
C
Sorry, there are Two components of that, right? There's a, there's an up and out and then there's a down and in. And what I mean by that is, you know, working with the board, working with the C suite on the strategy of response where, you know, you're not listening to every single thing your outside counsel says, you're not listening to every single thing your comms team says because if you did, you wouldn't do anything ever. You'd file your SEC required filings and that would be it. So, and yes, that's an exaggeration, but nonetheless, you know, our emphasis was, hey, you've got to kind of meet your customers where they are and you can't withhold information. You have to be as transparent as possible. And really the case here is like okta, right? I mean, we've talked about this on the show before about the, you know, a couple years ago they had their incident, they didn't share enough. The next one, maybe they over shared. But nonetheless it's working with these teams, helping them put into context the advice they're being provided by outside counsel on engagements, on transparency. Sudhakaramakrishnan, who's the SolarWinds CEO, had regular blog posts, he did webinars. Very, very transparent. Not just on kind of what the state of the response was. More importantly like, how are they getting to good? And I think that's the real question right now for F5 is if they have truly been internally owned front to back, where a customer looks at their service catalog or their offerings and says, how do I know that the Chinese didn't own every single one of these? Like, you're going to have to rewrite the entire code base for everything. How can you get me something that makes me feel better about that? And so that's a lot. What the down and in was with Alex working with the team and Tim Brown and those folks over there at SolarWinds saying like, okay, all right, here's how you clean up your pipe, your CICD pipeline, here's how you restructure your security team, put a governance process in place. Here's some of the accountability measures you can put in place. And it's going to take you X period of time. And you know, at the same time you're getting a lot of pressure, right? Because the, the sales organization says, hey, I need an attestation from, from ksg, from Alex and Chris that says we're clean. And Alex and I are looking at each other like, are you out of your mind? We can't do that we can tell you what you're doing is, you know, the right direction.
- [17:59]
B
And so a big part of it is we had to basically build a whole new application security function for them. We also massively simplified their product line. So one of the problems SolarWinds had was they, they shipped too many products. And you see this at these kinds of security companies where they never throw a product way. Right. So I guess not security company. There's kind of a. SolarWinds is kind of a general IT vendor. And what will happen is they'll accumulate this long tail of products that they put into sustained engineering where one of one or two people are still churning out DOT releases every once in a while and it's making a couple hundred grand or half a million dollars a year. And they're like, well, why throw that tiny little bit of revenue away? And they don't realize that thing is actually, it is not an asset, it is a liability on their books because it represents a significant amount of security risk. And they don't see it as that.
- [18:52]
A
The opposite of that, Alex, is something like 20 years ago when some friends of mine came to me and asked me to help them coordinate a vulnerability disclosure to Cisco. And it was like a WI FI product that had come to them via acquisition and was quite old. And we reported these bugs to them and their response was, well, we're just going to AOL this entire line, which was quite funny. But you know, probably like we thought that was crazy at the time, but you know, probably in retrospect that was the smart thing to do.
- [19:20]
B
Yeah, I mean, I think the ethical thing to do is to end of life stuff. You know, before you get the bug report, if people are paying for stuff, you should patch it and then you can say we're end of life in this year out. And their credit, I mean, SolarWinds didn't just kill stuff. They're like, they announced we're end of life stuff in the future. But that's one of the things we did was we helped them kind of triage. Hey, you know, this stuff is like, we're just not making enough money for this to make sense. And so they announced, you know, six months from now, a year from now, this stuff is dead, and simplified it. And then we built an application security process to go through and do assessments to triage. We're doing pen tests, we lined up all these pen test firms, we lined up code review firms to go through all of the products other than the core product that had been backdoored and to Go find. Because the other thing that happens, this is the same thing I did with Zoom when Zoom had the same problem, is you have the core issue and then you have all the hangers on, right? What happens is the media starts paying attention. And so every tiny little thing that is would usually just be a $1,000 bug bounty payout becomes a front page New York Times story.
- [20:28]
A
We had a bug in SolarWinds, another bug in S5. Yeah, no, I know that, like, little.
- [20:33]
B
Tiny bugs in Zoom literally became like front page stories in the New York Times business section. And I would have to like, yell at a New York Times reporter of like, you know, Microsoft Teams had a bug that you could take over all of teams with like a bad JPEG or something. And you guys never mentioned it, but, like this stupid little bug in like a Zoom installer, you guys are now turning into a huge thing just because Zoom is in the news, right? And this.
- [21:02]
A
Well, to be fair, to be fair, there were some pretty decent bugs in those. In those.
- [21:07]
B
Oh, there's, there's some real bugs in Zoom, but, like, that wasn't the real bug, right? So what you have to do is then you have to front run all the stupid bug bounty guys who now want to become famous because, you know, and so you have to go find all the those bugs and fix them. Because everybody wants to become famous. Everybody wants to run to the media with every tiny little bug. And so you have to go do that in every single product and you have to massively overextend. And so we're doing all that kind of stuff while rebuilding the application security team, while doing a bunch of code review and then uplifting the code that's getting built going forward while, like Chris said, rebuilding the CI CD pipeline, which, in the end, this wasn't a security flaw. What happened here was the SVR decided to spend nine months infiltrating inside of SolarWinds and then they built custom malware to in memory. I'm just going to remind people of what happened here in memory. They waited for the build process to happen, and then they replaced in the memory page and the kernel. The code, as it got compiled by Visual Studio, replaced it before it hit disk. So it never actually even touched disk. It decrypted the code, changed the page in the kernel, and then cleaned itself up. The only reason we ever figured out what happened is that one of those build servers was a VMware machine and it got snapshotted. The moment during the build, we had a memory snapshot of the Malware decrypted in memory. The SVR actually cleaned all this stuff up perfectly. They just didn't know. There happened to be a old snapshot that got found. And then one of the KPMG guys I think did forensics on it and we got super lucky. Otherwise we still not describe exactly what happened.
- [22:55]
A
I think you just described the only instance in known history where there was a security benefit from running VMware.
- [23:03]
B
But no, but this is also like why I get kind of pissed about all this SEC bull that happened. Like sowinds made a bunch of mistakes, but the idea that like the SEC who couldn't even like secure their own Twitter account from being hacked because they didn't turn on two factor. The idea that they would be able to stand up against a concerted SVR effort at this level is just a joke, right? Like this was the absolute scalpel. The surgeons of the Russian Federation slowly infiltrating this, this company and then building this incredibly beautiful malware and then doing a really good job of cleaning up after themselves. And like we got spectacularly lucky that we have any idea of what happened here. But anyway, like we, we had to rebuild the whole application security team at this company to uplift. Like SolarWinds did not have an AppSec program that, that they should have. That, that's, that is honestly true of pretty much every software company their size.
- [24:03]
A
Yeah. Now look, before we get on to talking again about what other of these mid tier companies can do, because you both see some opportunities there for that. They can, you know, at least make this problem not so, not as bad. Chris, I wanted to ask you, we've just talked about what F5's response should be. You know, what should vendor, what should users response to this be? And in particular I wanted to ask you about the US government response to this because CISA has issued an emergency directive for government agencies to patch this, but the government's shut down. Are security people still actually at, you know, working and turning up to work in US government agencies at the moment?
- [24:44]
C
I mean, do you guys remember when emergency directives out of CISO, which the authority wasn't even granted until like 2015 or 16. Do you remember these were rare fighting operational directives? Yes, but emergency directives were super, super rare. Like one of the first ones we pushed out was mandating the removal of Kaspersky from civilian networks. That was in the 201718 time frame. And that actually went all the way up, kind of an ancillary case went up to Supreme Court. But immediately after that or not not too long after that there was 2019 and there was the DNS records tampering, the sea turtle case I think it was. And that was January of 2019. And again that was in the middle of a partial shutdown. It was not a full shutdown, it was a partial shutdown. But for the outside observer it doesn't matter. It's the same frickin thing. Nonetheless, a good chunk of IT teams were furloughed. So they were sitting at home and yet we had this active exploitation out in the wild, changing records, DNS records. And there was this whole list of things we had tasked federal agencies to do and there were some agencies that had to recall IT folks. And you know, the thing is like typically security operations folks are not furloughed. But a lot of this stuff, even with the F5, it's not going to be done by the security operations team. Some of it will be maybe pulling some of the CDM data and some of the other, the scanning and inventorying, they can pull that. But in terms of actually patching systems, that's not going to happen probably in the, in the security team, that will happen in the IT operations side. So it's hard. And then at the same time you're dealing with a lot of the communications angles where again your PR team, your comms team, those folks are out too. And then you have the internal coordination piece like reporting up to cisa. So a lot of the important people that would be involved in executing this would not be necessarily on duty. You can recall them when they're furloughed, but that takes time. There's a process, there's a 24 hour process. You got to notify them, you got to say, hey, need you on duty and at station. And you're supposed to kind of be, while you're furloughed. You, you have to check your phone, a certain or email a certain, with certain frequency. You can't go too far away because you have to get back to the job. So it's, it adds just that much more complexity to something where obviously CISA thinks it's a big, big damn deal. But it, obviously it needs to happen. I think there's a kind of a knock on effect as well. Something I learned in 19 was that as soon as CISA pushes out an emergency directive, every other organization, commercial, state and local or whatever, it doesn't matter if they're not in the federal government, they're looking at that and they're saying hey, we need to do this too, because their bosses are going to expect them to do it too. Plaintiff lawyers are going to expect. Hey, did you do this? No, you didn't. You got popped. Well, you should have done it. So there's a lot of, kind of a long tail of those emergency directives and again, in the period of a government shutdown makes it that much more complicated.
- [28:01]
A
So you think probably it ain't going to get done quickly within government, but this will still hurry along the private sector?
- [28:09]
B
I don't know.
- [28:10]
C
So for sure on the latter, it's definitely going to hurry along the private sector because it takes a bunch of different disparate assets, including F5 and other recommendations, puts it down into one really digestible set of guidance and instructions you can kind of checklist off and execute against it. As for what happens inside government, I. Look, I think they're still probably for the most part, going to hit their timelines, the 72 hours and other, other deadlines that are in the, in the emergency directive. It's just going to be like Hurt Locker stuff. I mean, they're going to be working just crazy hours to get this done through the weekends, which, hey, I mean, that happens. And the private sector too. I think they'll get it done. I think, again, on the plus side though, we're, we're, as far as I know, again, like, we're not seeing active exploitation in the wild at least. So. You know what, what I've heard, even.
- [29:00]
B
Before the shutdown, there's basically nobody working on cyber in the United States government. I just want to point.
- [29:07]
C
Oh, that's not true.
- [29:09]
B
Okay, so, but okay, even before the shutdown, there is one confirmed leader working on cyber. It's Sean Kieran Cross, the National Cyber.
- [29:19]
C
Right.
- [29:19]
B
We do not have a director of the nsa. The acting director was just informed that he will not be promoted to be the director. So we have no confirmed director of the NSA.
- [29:28]
A
Yeah, but there are CSOs and admins and people doing the actual work. Alex, come on.
- [29:33]
B
Right, we have an acting director of cisa. Sean Planky has not been confirmed under the Biden administration. There are about 400 people working on the National Security Council, including a fully staffed cyber division. There are fewer than 40 people working in the National Security Council. Fewer than 40 people. There's not been that few people since the creation of the National Security Council. There's nobody, Basically nobody works in the eob. Right. Like, it's like you're going back to the time at which the entire Department of Defense fit in that building before The Pentagon.
- [30:09]
A
Well, look, look, I gotta say, I gotta say I was actually surprised to see an emergency directive come out from cisa. Everything that's been going on with SISA being gutted. Right, Chris?
- [30:20]
B
And it's a testament, I'm going to say, to Chris and Jenny Sterle, who set up these processes and the professional staff who's still there, that they're keeping the ball rolling when you have this complete and total turnover on the, and that there's no political staff left. But like we can only go for so long when there's nobody in charge.
- [30:39]
C
Yeah.
- [30:40]
B
And, and there's nobody coordinating in the White House of any of this stuff.
- [30:43]
A
Chris, Chris, you had, you had something there.
- [30:46]
C
Yeah. Just to write to defend CISA here. I think, Alex, to your point, there is a lot of institutional knowledge and muscle memory that's been built up over the last several years. In fact, I don't think this is the first emergency directive of this administration. I believe there was one not, not too long ago. So the team that knows how to do these can do them and they've been doing them for years. So that's there. I think what you're kind of hitting at though is the more strategic lack of capacity or capability really that's within the, the federal government at cisa, perhaps at the White House. I mean, I, I don't know what's going to happen with Plankies. Sean Planky's nomination to be CISA Director. Ted Budd from North Carolina is with, is holding all DHS nominations, including Planky. So, you know, who knows when that's going to wrap up. But this will hamper for the lack of leadership and the strategic push behind cyber is for sure going to hamper pushing the ceiling on where these agencies can go. And I swear, I mean, this is just the op ed waiting to write itself. We're in sitting here October 2025. The policy conversations around cybersecurity are virtually the exact same as they were in October of 2015. Information sharing in offensive cyber, including hack back, dude, there's not a lot of drift and there's not a lot of net new because it's the only real appreciable difference frankly is that AI is finally here.
- [32:24]
A
Now, now, now is coming. That's a perfect opportunity to segue here because the last thing we're going to touch on is this PA stuff. Right. So, you know, Avanti is a great example of, oh look, it's not even private equity, it's just old software being acquired and the new owners just Turning the handle on this new software, you know, churning out invoices, doing as little maintenance as possible. Now that's one model in the software industry. I'm going to actually stick up for PE a little bit in some circumstances because you can see them do smart things. Sophos, for example, was pretty adrift. Then they got some new PE backers. They're on the march again. They seem to be doing really well. Proofpoint got taken private by Thoma. Bravo. Things seem to be going really well for Proofpoint. So PE is not always a disaster, but there is this certain category of acquisition, not always pe, but often pe, where they acquire some software product that's deeply embedded in government and enterprise, and they just flog that old horse until it collapses in the street. And this is obviously leading to some pretty bad security outcomes. One piece of software like that at the moment is Ivanti. Adam and I were talking about this last week, and that code base is something like 35 years old. Right. So, you know, we haven't really seen a way out of this, but the three of us were talking before we got recording, and you two are a little bit bullish on the opportunities with AI to improve some of these legacy code bases. I want to get your thoughts on this first, Alex, and then you, Chris, and then we're going to wrap it up.
- [34:07]
B
Yeah. So Sophos is a great example. I mean, there are different kinds of PE firms. And I think Joe Levy is actually a great CEO and I think he has. He came up with a good deal for his company and he has made the best of it. And look, I just quit being a public company. Ciso. Being at a public company can totally suck. Right? Like having to live up to quarter by quarter Wall street expectations. I can totally see the benefits of PE in certain circumstances. Yeah. I think Chris and I did some work for Avanti and we've done some work for other companies where you're looking at this old code base and you're trying to estimate what would it take to renovate this code base and fix all the security flaws. And you're coming up with these estimates a couple of years ago and you're like, oh, okay, great. So you would have to hire hundreds of software engineers and they would have to spend hundreds of man years to fix these flaws. It is completely impossible. And the truth is, is now in the AI coding era, that is not true anymore. It is possible, instead of fixing security flaws, to just rewrite significant parts of the code base to refactor the code to eliminate Entire classes of vulnerabilities that that becomes a possibility. So I do think we are moving into an era where that becomes possible under with AI coding. And that is going to become in the next couple of years a totally practical solution to some of these companies. I mean I'm not like going from C to Rust yet, but going from like C to C21, moving to like a standard template library, you know, getting rid of your Mallocs and moving to like a secure allocator. Those are the kinds of things that I think are actually possible.
- [35:51]
A
Well, I mean AI is getting so much better at that stuff so quickly. And in, you know, in the AGI case, I'm a bear on stuff like this. I'm a total bull. I think it's amazing and I see what you mean. But now my question, and this is, this is where we're going to end it. But my question, question for you, Chris, is, you know, Alex has just described how you can turn a gargantuan, very complicated effort into something more manageable. But I think one of the key problems here has always been and is always going to be incentives. Even if it's easier. Are these companies going to be incentivized to actually bother doing this?
- [36:27]
C
I think it depends. I mean you rattled off a couple examples of companies that improve that for PE takeover acquisition. But that's on the execution side. There are at least two other elements they have to drive and that is the quality of the code. That is separate than execution. Execution is getting the product out there. It's the go to market and all that, that stuff entirely different animal. And then the third piece is what the internal security program is. The security program is always the afterthought, always the last thing considered in these sorts of deals. So I think what the incentives are and what I am seeing and talking to a number of private equity folks that we worked with at KSG and even with Sentinel 1, but also just that I know and I think the differentiator here is that if the private equity team comes from a security background, like if These are former CEOs that sold companies to kick ass job and then set up their own fun, they take a really rigorous GRC approach to their portfolio. Do they do a really rigorous job on due diligence? I've seen plenty of others that don't and that Alex and I and our team had to come in after the fact and build risk registries of the existing portfolio and the companies because they didn't want to, you know, get whatever company they didn't want that to happen to them. They didn't want to have to give up all that money in the. The stock crater. So there are a bunch of different things going on here. But again, I think it's. It's who's owning who turns the screws, who has the influence over the company and do they effectively have a security mindset? Yeah, but that's the same everywhere.
- [38:03]
A
No, I mean, I think what you're saying is the obvious answer. Right. Which is the ones who haven't been doing it because it's not been economical but they want to do it, will do it and the ones who just don't think that way won't. Right. Like that's just. That makes a lot of sense.
- [38:16]
B
Yeah. It'll be interesting to see if you have like p. Firms who actually specialize in this. I could totally see firms saying, I'm going to buy this underperforming asset and I'm going to have a group that specializes in. We know how to renovate it. We're going to fire half the engineers and replace them with AI and we're going to rebuild this code and maintain it. And I think that actually might become an interesting little special.
- [38:37]
A
Yeah, I'm not sure about that. Not sure about that.
- [38:39]
C
I don't know if the business model works. I don't know if. Right. You can squeeze in more blood out of that stone. But I think it would be interesting to see the write up for sure.
- [38:48]
A
All right, we're going to wrap it up there. Chris Krebs, Alex Stamos, thank you for joining me for another episode of the Wide World of Cyber. Always great to chat to both of you. Cheers.
- [38:56]
B
Thanks, Patrick.
- [38:57]
C
Thanks, Patrick.