A (3:05)

Awesome. Well, let's dive into the meat of the conversation here. Like I said, we are going to talk all about supply chain attacks and this is something we've definitely touched on at a high level in other conversations, but we're really going to drill into it this time. I know this is an issue that is most prevalent in government and critical infrastructure, but it's certainly commercial businesses are not exempt from that. So diving a little bit deeper into this, I know this is what you specialize in and this is what your business does. One of the biggest risk factors for third party applications is the reliance on open source software. Let's start there. Can you break down why that's such a critical problem?

B (3:43)

Yeah, absolutely. So Open source software really powers everything we know and love today, right? Even this recording that we're doing now, like everything about this is, is powered by open source software. It powers every application. You know, it underpins a lot of AI infrastructure which has been such a reliance on us in the past few years. And really when you scope it down to, you know, what, what are like some data points around this. So like most enterprise applications, about 70, 80%, sometimes higher, is all composed of open source software. And that's awesome because it allows us to move really fast. So we can take components that have been built by wonderful talented developers and reuse those in our applications very quickly and start to build and iterate on software very fast. One of the downsides though is that every single developer that maintains each single library or component from an open source perspective, that your enterprise relies on, that's maintained to a different standard. So their standard on security and on compliance may not be the same as your organization. So when you're looking at it from an enterprise perspective and saying we consume these open source bits, it's really important to understand which ones are your critical kind of open source pieces that you need to focus on in order for your application to be reliable and resilient when you ship that software to your end user. And it's also kind of like that's to the right of the problem, to the left of the problem. You need to be looking at what kind of vulnerabilities or what kind of weaknesses are in this code before I decide to consume it responsibly. Right. So talking about responsible consumption of open source, there's no argument that we can do away with open source. That's just not realistic. Right. But really it's about managing the risk around that as you choose to adopt it and as you choose to ship it embedded in your enterprise applications that you're producing to your customers. Um, and so it's in, you know, to kind of answer your question though, it's in everything. Right. And so we really have to be thinking about what are some next generation kind of like techniques that we could use to counter a lot of these attacks that we're seeing lately. Some of them are just getting pretty out of hand. If I were like a ciso, I would think of them as almost intimidating because the, you know, evolution, evolution of their tactics, techniques, procedures and executing these attacks has increased exponentially. So it's, it's an awesome space to be in. The reliance on open source software and how it gets tied up into software supply chain attacks is, is super fascinating. And you know, I could talk about that for, for hours. Awesome.

A (6:47)

Well, something I do want to kind of drill into that you just talked about there was the idea of these ttps. So can you break down from the entry point of really where these supply chain attacks typically happen?

B (7:00)

Yeah.

A (7:00)

What does that entry point typically look like? And then what happens after that? What are the attackers doing once they're in the environment?

B (7:06)

Yeah. So I can provide kind of like a real world example, a recent one that was disclosed by a good friend of mine, Paul McCarty. He has a great kind of project out there called Open Source Malware and definitely encourage everyone to check it out because it's an awesome resource kind of tracking open source threats. So like a typical entry point for a software supply chain attack that's targeting a piece of open source software. Ironically, the best way to attack Open Source is to contribute. Right. So what we're seeing is people starting to create real accounts and whether it's on GitHub or whether it's like with NPM and they're creating fake accounts on NPM and then publishing packages that people think are legitimate. Right. The attacker on the offensive side is trying to think, how can I get ownership over this code? In the case of the xz, which was kind of an attack two or three years ago, that was really building rapport with an open source maintainer to leverage that and really exploiting trust, which is the kind of critical linchpin of the open source software community, and really taking that and exploiting that to get access over it. And once they have that, there's no like doing multi factor authentication and bypassing that and things you would think of typically in a legacy kind of cybersecurity attack. It's really about just contributing and creating fake stuff so you could create fake, fake packages that you know are full of malware. Which was the case with the disclosure that Paul McCarty made with the Indonesian Foods campaign, which created thousands upon thousands. I think there was a count of around 86,000 basically fake MTM packages that were published just to pollute the marketplace. Right. So there wasn't a ton of like super malicious activity going on. But some of those tactics and techniques scaling beyond initial access are really frightening. So you're looking at, you know, some of the core tenets of that where like a new fake package full of malware being published every seven seconds. And then within that package what gets really frightening is it, is it sources from the package JSON, it will actually source more fake packages. Right. So then you're, it's extremely concerning because you're sourcing the software supply chain attack itself where they establish that initial access is actually bringing in even more bad packages. Right. So if you pulled down one package, you're actually pulling down, you know, maybe in this case eight to ten new bad packages. Right. And so uh, all that started though, if we were to like peel back the onion, how did this really begin? It started with a guy creating a bunch of fake accounts and then pretending to be, you know, open source contributor that, you know, any one of us would just think is just another guy making software. And so that's why you really have to like keep your guard up and really start using, using tools to start inspecting like the code itself in addition to looking at who is this person behind the code, you know, is this a ghost account that was just created two days ago to get access and to start contributing? And that's kind of two core tenets I would really enforce when you're trying to think about how do I do the right thing today? Start looking at is the code good? Which shouldn't be new for anyone, right? That's typical, like Software Security 101. But we need to start thinking about, especially with the like onset of AI now it's really easy to go and scale and create 30 fake accounts at once. You need to start thinking about who's this user account, right? How long have they been around, what's their history with open source, are they a legitimate contributor or are they just here to, you know, cause a ruckus? And you need to start inspecting those starting today.

A (11:14)

You know what it reminds me of is something somewhat like, almost like a Reddit community. Like you have these moderators who are going in and kind of vetting to make sure that everyone's following the rules. And it's like that's kind of a similar system that we probably need to start looking at as, you know, as kind of a North Star, if you will.

B (11:31)

And like, so the open source community has like, you know, certain projects that are very well maintained, right? So these are like cloud native Compute foundation projects, open SSF foundation projects, Linux foundation, all of them do great work where they have maintainers that are basically like the stewards of the code, right? So they're reviewing pull requests, they're keeping an eye out for security issues, bugs, feature improvements, right? All of that's super useful. But it's really hard to scale that across the entirety of the open source ecosystem, where you're dealing with millions of packages, looking at, you know, from various language ecosystems like npm, you're looking at Go, you're looking at Python. And to really scale that for every single person is, is hard, right? So that leaves kind of the burden with enterprises to, you know, take it upon themselves to do their due diligence on open source before they, you know, decide to pull in and include that piece of open source in their application.

A (12:37)

I want to go back to what you were talking about with the idea of threat actors and bad actors contributing packages into these open source communities. So when you were saying that, the first thing that came to mind is threat hunting, threat detection, what roles they can kind of play in helping identify some of these malicious packages and the malware that's baked into them. So can you kind of dive into that and talk about what role threat hunting and detection play plays in identifying these vulnerabilities for supply chain attacks and maybe even touch on the, the idea of this AI powered threat hunting which we're starting to see more and more and more of.

B (13:13)

With the boom of AI. Yeah. So a lot to unpack there. Threat hunting, using threat intelligence to basically dive in and inspect software before you use it. Right. And trying to like, I think when we think about threat hunting, I like to think of it as a really proactive security measure. So we're trying to go out there almost like conduct reconnaissance in advance of using software. Right. So a lot of the kind of attacks that you mentioned, when you're saying, you know, like people are coming in inserting malware into the like open source community, the place that they're doing that at is really these upstream package managers. So like in that Indonesian foods campaign that was really in the NPM registry and you can monitor that registry for packages, for accounts, for how old accounts are that are contributing the software. So you could do inspection on that. So when we talk about using threat hunting to go out and search for this stuff, you really need to drive that with really good threat intelligence. And for enterprises, what I try and focus on is we're not trying to boil the entire ocean here, which trying to just look at your software, the bits of open source software that you use, and then you need to go out and threat hunt against that target set. Right? So you need to say, okay, we rely on these 10 open source packages. We're going to go out there and hunt against every single one of them and start looking at is there malware against these, are there unknown zero days against this? And that's where you get into like some places we're exploring now at Hunted, where we're trying to use models to go out there and interrogate code to find unknown vulnerabilities that maybe someone isn't disclosing. Right. And I made kind of a big fuss about this with a, with a component, a super widely used component called RUNC lately. And they had three vulnerabilities disclosed one week by one engineer that worked at Huawei. But then when you go into the kind of actor's profile, you'll see that he intentionally does not disclose vulnerabilities that he has found against Renci because his boss does not want him to. Right. So that may, you know, if you're relying on that component, you, that makes you kind of think, okay, so what does he have in his back pocket? Or what does any bad actor have in his back pocket that I'm not seeing? Right. And that's where like you could use large language models to come in and churn over the code and say, what am I missing? And look at okay, is there anything hidden in here that I've missed during, like a manual check or a static check that could have gone unseen? Right. And so we're starting to see some really interesting results there. And I think a lot of different security vendors are kind of playing in that space because they know it is being hit really hard right now by various kind of threat actors.

A (16:26)

What does the recovery strategy look like for these attacks where we have some malicious actor embedding malware into some code, an organization applies that into their environment and then it's exploited over time? What role does recovery play there?

B (16:42)

Yeah, so when you look at it from a software supply chain perspective, you typically, if you pull down malware, you have to neutralize that right. First. And then when you're looking at, okay, how do I prevent my developers from continuously pulling it? That's the hard part, Right. So you have to get that reversion and that software you use. So pulling it back and looking at, okay, I was on version, let's say 1.5, that was the tampered version, right. Now I have to pin, right. This comes into a really good proactive kind of software supply chain measure, which is pinning your dependencies so that way you're not automatically just, boom, ingesting stuff, pulling that down. And so really it often involves like a version rollback. So that's really like when you're starting, just restore or recover at that point. And so that's easy. I will say, if you're dealing with an organization that has really good software supply chain measures, right. So you kind of have a constant inventory of what's flowing in and out from a software perspective. So you know what kind of open source you're using because you're doing things like SBOM generation and SBOM management and vulnerability scanning. And you're already out there conducting threat hunts. That gets really hard if one, you're not doing those things and two, if you start looking at the types of attacks that we're dealing with today, right. So again, that Indonesian foods campaign, that was like thousands of different packages. And if you're at a large enterprise and you have to monitor for every single one of those identifiers, recovery gets extremely complicated very quickly. So now it's not hunting for one kind of like zero day event, it's hunting for 20, 30, 40, a hundred, all at once. That hit you over the head. So having good contingency planning in place is super, super important for software supply chain incident response type events. And I like to kind of take this moment to say like a lot of the things you could do to put yourself on like a really good, you know, security posture for software supply chain, don't involve doing anything. I would say that is outside the norm for information security best practices. Right. These are very standard cybersecurity things. So doing things like contingency planning, having awareness. If you rely on 10 critical things in order for your product to work, you better know what they are. Right. You don't want to find out when the attack is already unfolding. You're already on your back foot at that point. And the way you do that. Right. Is really through continuous monitoring. So having continuous monitoring in place so you don't get to the point of having to deal with recovery. Right. So recovery is like really part of that contingency plan that we need to have available at any given moment in time. Because in worst case you could get to a point of like, they stole all my source code. Right. For that was previously private. And that happened to organizations like F5, like Red Hat. And you know, that's a whole different scenario where you're like that. You can't just back up a software version and get your data back. Right. So you have to be, you know, keeping backups of that data, keeping backups of everything you have in git and being able to restore that in that event. Right. So the software supply chain attacks can really be as something as little as one package or they could come in and steal all your source code. So it could get pretty gnarly very fast. Right.

A (20:37)

Right now I'm going to put on some rose colored glasses for a second. Let's say there are some organizations out there that are just not exposed to this threat. I think that's, you know, quite optimistic. It sounds like this is much more of a pervasive issue than most people tend to understand. But let's say people are out there and they're just not exposed at all. Or you know, more likely they are at least unaware of their exposure. In addition to investing in modern threat hunting cyber recovery plans, the things that we've talked about already here, what can organizations do to prepare for these types of attacks outside of those infosec best practices that you were just referring to.

B (21:11)

Yeah, so I think there's a bunch of stuff I don't have like all the time to get into it, but stuff like keeping, you know, I'll just go down into the continuous monitoring bucket because I think that's so important and I think that's really the starting point because a lot of best practices I prescribe are falling under that, that kind of bucket or section of cyber. Right. So doing things like knowing your dependencies, do you have a complete inventory of all the software that you use, whether proprietary or open source? Right. You need to have insight into that. That's really step one. You need to have situational awareness of what you're using. You also need to know what are the most popular and critical pieces of open source and that you need to make sure of those versions. You're doing things like pinning those dependencies. So you need to make sure you're protecting your organization. Just on the way you do software development, you need to look at from the continuous monitoring lens, you need to go upstream and see how are these people maintaining the code that you rely on. So have they not made an update in six years and there's three active vulnerabilities against it, or do they update it every single week and cut a new release every single week as well? Right. Those are two different stories in terms of open source risk. So having that level of insight and then again kind of the final two things there, as I'm like, okay, you need to start getting that inventory. So scanning your code, generating SBoM, storing that, iterating over what are the most critical open source dependencies and also having that continuous model baked into your software development processes for your application. So you need to constantly be interrogating the code to see if it's good or not. So every new release that gets pushed for a dependency you rely on, you need to be interrogating that code. You need to be looking for changes that could be suspicious, anomalous coming. And this is the last part coming from people that you may not know. So these are people that are previously unknown to the project or unknown to the community. Maybe their GitHub account is a day old and it's just some AI generated account generating AI generated code. And that could introduce a ton of, you know, problems for your organization if you're just going to, you know, if that's kind of the state of, of the software that you're choosing to consume. So everything I do or prescribe is really falling into that open, the kind of continuous monitoring of the open source that you're consuming and really driving. For us, we really prescribe using threat intelligence to have that insight available continuously. So if there is something going on with a critical dependency in the open source ecosystem that you rely on, that you just have insight and situational awareness of that risk.

A (24:10)

Absolutely. Hayden, where can folks learn more about the incredible work that you're doing.

B (24:14)

Yeah. So you can go to haunted labs.com learn about all of our awesome stuff that we have coming and we have intercept. There you can learn about our product. We also have the hunting ground where we cover security research or really interesting things that we find unfolding in the software supply chain kind of security space. So those two places is where I point you to have a good time.

A (24:40)

Thank you again for joining. This is a really insightful conversation. I really appreciate your time and the conversation. We'll link to all those resources that you mentioned in the show notes so that they're easy for everybody to access. So until next time.

B (24:51)

Yeah. Thank you.

A (25:00)

And that's a wrap on today's episode of Data Security Decoded. If you like what you heard today, please subscribe wherever you listen and leave us a review on Apple Podcasts or Spotify. Your feedback helps us understand what you want to hear more about. And if you want to reach out to us about the show, email me directly at data-security-decoded2k.com that's the letter N number two letter k.com thank you to Rubrik for sponsoring this podcast. The team at N2K includes senior producer Alice Carrick and executive producer Jennifer Ibin. Content strategy by Mayan Plout. Sound design by Elliot Peltzman. Audio mixing by Elliot Peltzman and Trey Hester. Video production support by Bridget Kirkey Wild and Sorrel Joppy. Thank you for listening. See you next time.