Loading summary
Transcript46 lines
- [00:01]
A
Open source software underpins virtually every modern application. That Ubiquiti is a superpower for developers, but it is also an expanding attack surface Software supply chain attacks were once rare but are now happening daily with malicious actors exploiting the trust developers place in public registries, package managers and CICD pipelines. Chainguard is a secure software supply chain platform. The company started with hardened container images and has expanded to cover domains including VMs, language libraries, GitHub, Actions and Agent skills. Matt Moore is a co founder and CTO of Chainguard and a veteran of Google's open source container and security infrastructure work. In this episode, Matt joins Gregor Van to discuss lessons from recent supply chain attacks, why CICD pipelines are now a primary attack surface, the challenge of meaningful software inventories, the EU Cyber Resilience act, and what the arrival of Anthropic's Mythos model means for the pace of vulnerability discovery and the urgency of patching at machine speed. Gregor Vand is a security focused technologist, having previously been a CTO across cybersecurity, cyber insurance and general software engineering companies. He is based in Singapore and can be found via his profile at Van HK or on LinkedIn.
- [01:41]
B
Hello and welcome to Software Engineering Daily. Today is a fun one where we get to talk to a person and a company again. So at the end of many episodes we say we'll be following along and that's exactly what we've been doing here. So very happy to have Matt Muir of Chainguard back with us. So yeah, welcome Matt, thanks for having me. Yeah, so some of you may remember the episode we did probably around two years ago, I think at this point, something like that with Chainguard, everything talking about hardened containers, about security. So spoiler alert, a lot has happened over the last two years from that perspective. So I think it's going to be super interesting to dive into how chainguard has evolved with that and how it's protecting things and evolved. But as usual, just in case anyone didn't catch that episode, just a very TLDR of who is Matt Mura? How did you get to Chainguard?
- [02:34]
C
Yeah, I'm Matt, I'm one of the founders of Chainguard and the CTO Chainguard. My co founders and I, we all met over many years of collaboration at Google and we collaborated on things across open source, the container space, the security space, the developer experience base, and we have been working on parts of these problems for over 10 years together. But one of my co founders and I had left Google briefly and I was on a break barbecuing and playing video games, trying to figure out what I wanted to do next. And our CEO Dan was trying to recruit me back to Google and I was like, ah, I don't really want to go back to big tech just yet. And so I uno reverse carded him and I was like, have you thought about starting up? And we'd been working on aspects of this problem space for a while and really nobody cared. But at that time solar winds had just happened and overnight everyone cared. And so the time was right and we had spent a lot of time studying this space and so it felt like right team, right time. And we decided to go for it. And it's been a pretty wild ride ever since.
- [03:43]
B
Yeah, amazing. And yeah, it's interesting how you mention solar winds there, like right time. And then we're going to get into almost equivalents that might have happened over the last two years. And then again, for anyone not familiar, just what is Chainguard? What are hardened containers?
- [04:00]
C
Yeah, so we started with containers Chainguard. On the whole, we sort of want to give folks a safe way of consuming whatever open source they're writing. These days you can't really write modern software without Open Source influencing it in some way. Basically every language, runtime, or compiler, whatever is open source. Java, Python node, right? These are open source language runtimes. And then on top of a lot of them, those sit on a lot of open source projects, like the system packages that go into every Linux distribution. The Linux kernel, if you're running containers, the container runtimes are open source. Kubernetes is open source. So Open Source just permeates everything that we do. And it's basically unavoidable these days if you want to write a modern application. But that's also sort of a superpower, right? Like if you think about before the proliferation of Open Source, how much you had to write in order to like ship an HTTP server. Like back 25 years ago, Amazon was racking servers, right? Google was racking servers to build a startup. You were racking your own servers and going in and dealing with the hardware aspects of this. But like Cloud has made that so that you can just take that and use it. Open Source has made it so that the amount of software you need to write to write an API is so much smaller. You write a little bit of glue on top of frameworks that sit on top of HTTP servers. It's this amazing accelerant that allows folks to write much, much, much less software to build a functional app. But really you need a way of safely getting all of that software. And so we started with containers because what we found was the way folks were consuming containers today, you sort of start with a baseline of hundreds of unpatched vulnerabilities. And it's not that patches for these don't exist in the upstream, it's that the way modern distributions work. They're just very slow to get those into folks hands. And so we wanted to give folks a better way of doing it, where we would fully patch everything that goes into the containers that they're getting. And really, for many years early on, one of our big struggles was convincing folks that we weren't somehow lying, that this was actually a tangible result. They had this learned helplessness that like what the result that we were showing them wasn't actually possible, but it is. It's just a lot of work that nobody prior to us had really done to the level that we did it. And so we've sort of evolved as a company and gone beyond just containers really. The way I look at what we do now is being a secure software supply chain as a service, right? We give folks a safe way of consuming that Open source. One of the analogies I really like, and I realize that it dates me a little bit, but like in the early days of digital music, right, like folks got their music off Napster, right? And ignoring the licensing aspects of it, it was a way you could get a lot of music and sometimes you'd get music, but sometimes you'd get malware, right? That's the state of open source today. If you're just consuming stuff from the public registries, it's a lot like the result you'd get from getting stuff off Napster. And so the role fill is a lot like when itunes came onto the scene and started allowing folks to buy MP3s, right? You could buy a song and then you got access to that song. It was coming from Apple, a reputable company, and you had a safe way of getting that music and you weren't going to get malware from Apple. And our initial product, we sort of oriented around unit cost. So you buy an image from us, you get that image just like you'd get that MP3. And then over the years, our catalog grew a lot and we had customers who wanted to standardize on us and wanted access to that whole catalog. And so we shifted towards, we still have that model if folks want that, but we sort of evolved into more of a Spotify model, right, where folks could get access to the whole catalog, listen to what they want. And consume whatever container images they want from us. And it's a developer seat pricing model. So depending on which model folks are interested in, they have a safe way of getting their open source from us. We started with containers and then we've expanded to a variety of other different ways folks consume open source.
- [08:07]
B
Yeah, I love that analogy. I also used Napster back in the day, moved on to, I think, Kazaa and all sorts of other things. And then, yes, now I'm Spotify because it's very quick and easy and it's definitely the actual song and all that kind of stuff. So I think it's a great analogy. When we talked a couple of years ago, as you've called out, it was originally hardened containers, and there's now more of a platform play here. And then almost back then it was like a CVE reduction story, if you want to call it that. We've got smaller images, we've got fewer packages, actually fewer but pardoned, fewer known vulnerabilities. But how has that conversation, I guess, changed with your customers and with just people interested in the product over the last two years?
- [08:51]
C
Yeah, it's shifted a lot. I mean, hardened containers was always just the beginning. When you're trying to solve for a space like supply chain security, it's an enormous place. We had to pick somewhere to start. And containers was a space we knew exceptionally well. So we decided to tackle that first. But the real question is, how do you secure your entire software stack, all of that open source that your developers aren't writing, but it's running in your environment. Right. So it's subject to all the compliance of your environment. Right. And every unpatched piece in there or every piece that may carry in malware. Right. Potentially presents risk to the business. So we have rapidly expanded beyond containers. We started to target VMs and language libraries a bit over a year ago. And then at our most recent user conference, we announced that we were expanding into hardware, hardened actions and hardened agent skills. And so one of the ways I think about the framing beyond CVE reduction is I think about CISA's Secure by Design initiative that launched a couple years ago. And one of the tenets in there is seeking to eliminate entire classes of vulnerabilities. Not just patching vulnerabilities, but this is things like writing things in memory, safe languages, or leveraging an ORM to avoid SQL injection. Right. So part of the way I look at it is by building things from source, we're actually able to eliminate entire classes of vulnerabilities. That come from trusting those public artifact registries. I mentioned the Napster analogy, right? Like just by building stuff from source, we're able to eliminate 98 to 99% of the source of recent malware attacks. Right? It's just a huge class of vulnerabilities. Basically the vast majority of how most of these things happen, right? Like just building from source gets us that, but it also gives us a level of control that allows us to eliminate other classes of vulnerabilities, like folks running unpatched software downstream of us, right? We can make sure that all the patches are applied from upstream. And so I think two things have also materially changed in the past couple years since we last spoke. The number of CVEs being reported has just exploded. And this is only going to get an order of magnitude worse post Mythos, right? We saw data coming from Firefox and some other high profile projects where the number of CVEs patched in a recent release, I think it was 150, was 10 to 20x what most prior releases had found. And so you've got the CVE graph just growing exponentially. But then you've also got malware attacks. These used to be relatively rare and they've gone to monthly occurrences, to weekly occurrences, to daily occurrences. And I posted something on LinkedIn talking about how these things were now happening daily. And someone left a comment being like, soon it's going to happen a couple times a day and literally the next day there were three attacks that happened on the same day. And so while no layer of defense is going to be able to promise 100% protection, one of the things that also hasn't changed in the last two years is that none of our customers have been impacted by any of these attacks where they were using our products. Right? So our library's product has been able to defend against all of the malware attacks that have happened over the last several months. And the rate at which they're happening is just frankly, mind blowing.
- [12:06]
B
Yeah, absolutely. Certainly open source supply chain attacks have had a moment through the last two years especially. And I mean it was kind of, for many of us, we could almost see it coming. I think it was just you have actually taken action on it and started a company to defend against it. Most of us were like, ah, it will happen probably. But hey, it's like nothing's going too wrong yet. But here we are. And it really has become very clear to a lot of unfortunately malicious people that that is a great way to actually, that's how you should distribute your malware. So let's go onto like a case here. So there was the XE or exit Utils case. We did actually cover this and one other that we're going to jump onto in another episode but from a different angle. So some listeners may be familiar, but this was xz. Utils was basically where a maintainer got more and more trust with the community that maintained that set of packages and eventually they were able to then push in a malicious. I think it was a malicious macro basically into the build system. This went into a package lib lzma. I'll stop there. That's the super factual. So someone gains trust, manages to push a malicious macro into liblzma. How did Chainguard I guess respond to that?
- [13:24]
C
I think one thing worth calling out is still in my mind that attack is still in a class of its own with respect to both the level of sophistication as well as the patience. A lot of what we've seen recently is much more opportunistic. Large blast radius attacks that try and just smash as much stuff and then get as many credentials as possible and then figure out what other fun they can have with those. This was an incredibly long period of time.
- [13:55]
B
It's like two and a half years. I think they spent person.
- [13:58]
C
I say that with air quotes, right, like Jia Tan, right. Like they, they spent time, you know, gaining the trust of the community, doing legitimate contributions and like basically it's resembles spycraft, right? Like it is a level of sophistication that I don't think I have seen since. Right? That doesn't mean it's not happening. I'm almost certain it is and I think people are worried about it happening all sorts of different places. But that said, right, like the attacker had maintainer level access, but the attack still required the use of an intermediate artifact. So Jiotan basically published an altered release distribution which a lot of distributions build from, right? And in this case they were like haha, the source is this. But then they altered the release distribution that. They attached the release that they knew downstream distributions were building from and it carried modifications that like activated the malware. Building from source defended against it in this case. But it's also possible that if Jia Tan had taken another path, it might not have because they had done all of that legwork to get that sophisticated and that deep level of control over the project before they were doing their attack and they were doing it effectively in plain sight. But I think this touches on why defense in depth is so Critical. And my point about no one layer of defense is ever going to be 100%. And if someone's telling you that, then.
- [15:22]
B
The Swiss cheese. Yeah, the Swiss cheese.
- [15:24]
C
I love the Swiss cheese model.
- [15:26]
B
Yeah.
- [15:27]
C
So like, building from source, minimizing what you're running, scanning for anomalous behaviors. I think one of the things that's interesting about, like, scanning for anomalous behaviors, not just necessarily scanning for, like, known malware signatures, but scanning for things that, like, programs you want running in your environment aren't doing. So it's not always malware. One of the things that we've been starting to talk about scanning for and blocking is what we're calling grayware. Right. Classes of software that maybe aren't compromised, but do similar things to malware that like, you don't want running in your environment. Like, they may establish remote access terminals. They may read or transmit sensitive credentials to places they probably shouldn't. There are projects out there that do these things that the projects that haven't been compromised and people use them, like remote access terminal. That sounds really great for debugging. Maybe I'll stick it in there. But maybe you shouldn't. Maybe your product security team doesn't want that running in your production environment. So, yeah, I mean, this notion of grayware where, like, you flag behaviors that you don't want running in your production environment, whether it's matching a malware signature or not, is I think, one of the things that I think we need to start getting real about what you'll let in and what you won't let into your environment.
- [16:37]
B
Yeah. What's interesting here as well is, so the packaging of Lib lzma that Chainguard actually uses, it runs on Wolfie as opposed to Debian or x86. So this wasn't actually specifically targeted. It was sitting, I guess, within a build that you had put together. But because it wasn't actually targeting Wolfie, technically it wasn't a threat, at least in that moment, compared to anyone running on Debian. But can you walk me through the mechanics of that as well? Do you think that was accidental, that they didn't also target Wolfie? And I guess also maybe walk us through, if they had been targeting Wolfie, what would that have changed for you guys, for example?
- [17:18]
C
Yeah, I think I mentioned there were a few things. One, it went through the release distribution. I think another one was, I believe there was something in the modifications that targeted specifically systems that were producing, I think, DEBs and RPMs, which we don't build, DEVS and RPMs. And so, yeah, I mean, like we built the release that was tainted, but like it didn't impact us because the way we built it did not activate the attack code. So I think that this is my point about source level access gives folks a way of potentially doing malicious things. And Jiatan had the level of control to potentially do some of those nasty things. But it's also why we complement the fact that we build from source with so many other techniques, including running different behavioral scans around the type of syscalls that are being made in one release versus the next. And it's not to say that any of these things are foolproof. And I think one of the interesting things about dealing with open source is so much stuff tends to be out in the open and how you scan for some of this stuff, actually you almost want it to be somewhat secret, which is counterintuitive to an environment that biases towards transparency and whatnot. But like, say I have an open source malware scanner that we run to detect these kinds of things. If I were an attacker, what's the first thing I'd do? I'd scan it to make sure you weren't going to catch me. Right. And so there's, I think, limits to just how much you can be completely transparent about everything you're doing and still be protected against folks who know how you're going to scan what comes out the other side and can check in advance basically whether you're going to detect them or not. And so it's a balancing act that we have to deal with. But yeah, we are doing various behavioral scans of the package builds where we try and classify behaviors of what they're doing, but also check them for known signatures of malware and things like that.
- [19:23]
B
And I think Chainguard factory flags like an update, if there is a code update that makes a new network connection that's actually not mentioned in chain logs, for example. That's interesting.
- [19:34]
C
Yeah, we try and look at a wide variety of signals. I think one of the behavioral classification things that one of the tools we do and we do differential scanning. So if the previous release, the one release didn't have it and the two release did. Patch releases are typically places where you don't expect significant behavioral changes either at the build time or at in the resulting binary, maybe you enabled some new piece of functionality, in which case flagging is probably the right thing to do. You can review that and then from there on that becomes a baseline. But there was actually a breach on the library side that was able to compromise the source code. And our scanning was able to block that. It was. I forget the name of the recent attack, but it basically embedded a PTH file into a Python library that forced the interpreter to basically load that as even without importing the library. It loads that at runtime and executes it. And that got flagged by our malware scans and blocked. So we didn't rebuild the malicious release even though the source for it was available, because we detected that pth. We actually built the release right before, which was totally fine. And after they detected this, they actually cut a new release and we immediately built that. But we didn't build the bad release because the malware scans that we run detected that malicious behavior and blocked it in our system. So there's a wide variety of stuff that we are doing to both scan the broader ecosystem, scan the stuff that we are building and running through our systems and then taking an approach to how we build these things from source, that tries to eliminate classes of vulnerabilities. We can't eliminate all classes of vulnerabilities, but we try and do what we can to eliminate classes of vulnerabilities where we can. There's building from source. There's things like compiler hardening flags that enable different types of memory safety things. There's potentially exploring replacing components with memory safe alternatives. So things like Sudo rs. I think we were one of the first folks to ever package Sudo rs, which is a Rust reimplementation of Sudo. But there's a whole wide variety of these. We've had the UU Utils project, which is a Rust port of the coreutils project in our distribution forever. And so there's a lot of these different projects that are coming onto the scene that where possible, we try and make those available. RustCls is another really good one where you can swap out the TLS implementation used by Curl, Libcurl, et cetera, with a version that's been written in Rust for handling cryptography.
- [22:07]
B
Yeah,
- [22:09]
D
you're shipping faster than ever with AI coding agents, but those agents don't vet the packages they pull in and they don't have security context built in. ORI by Endor Labs fixes that it plugs directly into your editor via mcp, catching vulnerabilities, blocking malicious packages, and flagging exposed secrets in real time. No separate tool to switch to, no dashboard to babysit security that fits how you actually build teams using Ori. See 10 times fewer security tickets and 6 times faster fixes. Free for developers. Get started at www.endorlabs.com Auri if you're running Postgres in production, you've probably felt the moment analytical queries start fighting your transactional workload. Most teams end up adding a second database and all the pipeline complexity that comes with it. Tiger Data, creators of TimescaleDB, takes a different approach. We extend Postgres with hybrid row and columnar storage, so one table handles both writes and analytical scans. Native compression cuts storage costs up to 95%, continuous aggregates keep dashboards live without bash jobs, and it scales to petabytes without you re architecting. Companies like Cloudflare, Octave Energy, Schneider, Axpo, and Flowco run production workloads on Tiger Data today. No stale data, no second system to operate, just postgres managed for you. Ready for the workload you're building toward? Try it free@tigerdata.com Agents are getting smarter every day, but even the smartest agents get stuck without the right context and the right tools. That's where Notion comes in. With the recent launch of custom agents, Notion became the collaborative AI workspace where teams and agents work side by side. And now their new developer platform is turning that workspace into infrastructure developers can build on. What stands out to me about Notion's developer platform is how much they've built underneath the surface. The platform ships with a set of new primitives that change what you can actually do. Workers are Notion hosted sandboxes where you run custom code, database syncs agent tools, webhook triggers without spinning up your own infrastructure. The CLI lets developers and coding agents read from Notion, take actions, and deploy workers through one interface. And with the external Agent API, you can bring agents like Claude or Codex into Notion as workspace participants with real permissions and triggers, not just a sidecar tab you're copying from, pulling in research sources tracked externally, syncing them into Notion, and having an agent help triage what is worth covering, all in the same place. The team already works. Learn more about Notion's developer platform today at notion.com sed that's all lowercase letters notion.com sed to try Notion's developer platform today. And when you use our link, you're supporting our show notion.cesed.
- [24:41]
B
just moving on, I just want to briefly touch on one other case, which was TJ Actions. So again, we did cover a bit of this from a different angle in a different episode, so some may be familiar. This was related to GitHub actions, so basically this GitHub action, it was compromised. And what that did then was extract CI CD secrets. And we have seen this through other attacks as well, like CI CD pipelines are now a primary attack surface and at the end of the day, chainguard images, they are built in CI. So how do you guys think about security of the build pipeline itself as well?
- [25:21]
C
All day, every day. So I think one of the things we've been saying from the very beginning is treat your build systems like production systems. CICD workflows operate with some of the highest levels of privilege in modern software delivery, right? But they often remain one of the least protected components in the development stack. You worry about least privilege in production. I don't want this one microservice doing more than it should be doing. But then you've got your CI system that has permissions to push everything to production. And so it's got these incredible credentials, this incredible power to publish things into your production environments. And so we need to protect those. So I think the principles laid out in the Salsa framework are a good start. But I think one of the things that in my opinion, I think is the biggest miss in Salsa was not mandating the use of short lived credentials. Because if you look at virtually every supply chain attack that has happened, someone got, and you mentioned some of the fun ones, TJ Action, Shaihalud, they exfiltrate long lived credentials. Some of them were made possible because someone got a hold of a GitHub pad or some other long lived credential, right? I think that Salsa fundamentally missed by not mandating the use of short lived credentials everywhere. And I think at Chainguard we feel a little bit like Cassandra, whatever the story of Troy, right? Like they bring Cassandra back and she's a prophet, right? She can see the future, right? So several years ago we built OCTO SDS to solve this problem for us. It is a credential federation service for GitHub. Because GitHub does not operate one. I've heard ones coming, but I think it's on their public roadmap. But this would have prevented a great many of these attacks. And we operate it as a public good. Anyone can use OCTO sds and the one long lived credential that sits underneath it lives in a KMS system that nobody can access without it basically paging our security team. But like it gives you short lived credentials. And I even flagged that to the TJ Actions Maintainer when this happened because like I said, we operate this as a public good. I think another good one that's happened recently is the Trivi breach. This built on a technique that Billy lynch, who is one of our engineers, disclosed to GitHub three and a half years ago. We call it imposter commits, which is this technique where folks are pinning a GitHub action to a commit. That commit actually doesn't have to live in the repo that it looks like it comes from from the rest of the action line. It can actually come from a fork. If you Google chainguard imposter commits, you'll find the blog post. But this is part of the technique that was used to compromise Trivi and so treat your build systems like production systems, harden them, use short lived credentials, practice least privilege. I make damn sure that every link in your supply chain is auditable, just like you would in your production environments. So you know what came from where, what happened where, et cetera.
- [28:22]
B
Yeah, that's great, great advice. So yeah, if anyone takes away anything today, take that away. So looking at chainguard as a product that, as we've been talking about has evolved over the last couple of years, I'm kind of interested in, I guess a few things actually. So the first thing I guess is actually just the number of images that you now actually maintain, which I think early 23, it was about 400, then it was I think over a thousand by the end of 24 and now we're pushing over 2,000. And I guess it's like, how do you guys think about just maintaining that consistency and quality? I mean that's just a crazy amount of stuff to maintain.
- [29:02]
C
Yeah, there's a lot. I think we're adding two to 300 images every quarter. And we've got an amazing team that has been building a lot of those in some cases by hand, increasingly leveraging automation. But on the upkeep side, right. Like 2 to 300 isn't necessarily growing super rapidly, but maintaining those images keeps growing. Right. It grows by two to 300 every quarter. Right. So we're looking at automation really across the board. And a lot of people talk about human in the loop, right. If you think about the purpose of automation is generally getting humans out of the loop. Loop, right. We're slow, we sleep, and we too can make mistakes. And we've always emphasized automation long before things like agents arrived on the scene. And we would try and automate as much as we could and then humans would get involved when things broke down. And when we did get involved, we would look at ways of avoiding repeating that breakage as you might look at a postmortem. Right. You always reflect and you're like oh, how can we do better next time? And over time those systems have improved, but agents have come onto the scene and we aren't ignoring them. Right. We are looking at ways we can safely use them. But the way we look at them is they're sort of yet another tool for pushing our automation further, handling more of that without needing human intervention. But ultimately, just because our automation is becoming agentic, just because you stick that word in front, it doesn't change any of our commitments or our SLAs. So we're trying to find ways to adopt it responsibly and figure out those best practices. Because the way I see a lot of what we're doing is operationalizing best practices at extreme scale. One of the very first places we actually employed agents was to bolster our testing across our inventory. Right. Like can it generate more tests that exercise different behaviors of libraries or packages and do things that could help us catch regressions, catch coming in through new updates. And so we still very much have humans in the loop. But the mindset shift is increasingly going from how do we solve a problem to how do we solve entire classes of problem.
- [31:08]
B
Yeah. And the average patch time I believe is still incredibly impressive. A critical is less than 20 hours and then high is two days, something like that.
- [31:17]
C
And we recently announced an SLA for the Kev list. So if a vulnerability is known to be exploited, we have a 24 hour SLA way for getting that fixed. And in folks hands that's amazing.
- [31:30]
B
I guess looking from a product perspective, but I guess more on the competitor landscape as well. I think maybe a few people might be thinking, hey, doesn't Docker now do this? So yes they do. And I guess that must have been an interesting moment for you guys. It does happen unfortunately to very great ideas of companies. The bigger guy comes along and says that looks interesting, we can do that too. GitHub I think have done that. And GitHub Actions was basically a clone of other startups into GitHub made sense. So yeah, how has that affected things, how you approach this problem? Docker has a free tier that's I think interesting to talk about as well.
- [32:06]
C
Yeah. And speaking of things that have changed in the last two years since we last spoke, competition has really come out and I mentioned we had to convince people that what we do is possible. Thankfully that's, that's actually a conversation I have to have a lot less now that other competitors have come out with products that at least claim to do the same thing that we do. I think One of the confusing things with DHI is that they claim two things, only one of which can be true. Like, one is that they are building on top of existing distributions like Alpine and Debian. They also claim that they're building everything from source. Right. But if they're building everything from source, then they're not Alpine or Debian. Right. They can't carry patches, they can't build versions that Alpine or Debian aren't carrying. And you know, and build those things from source and then claim it's Alpine or Debian. Right. Those two things are sort of in conflict. I think. The other thing is we are very familiar with the Alpine ecosystem and the Debian ecosystem, and we use them for many years. Our CEO Dan and I really built the first real hardened Images Back in 2017, 2018, which were called the Google Distra list images. And we actually did build that on top of an existing distribution. We built it on top of Debian packages. And it was because of this that we knew that to get to the results that Chainguard delivers, that we couldn't build it on top of another distribution. If you take any of those images which are running on the latest Debian, they're minimized and have the latest versions of all those packages. You take any one of those images and scan them, with the exception of one, they all have unpatched CVEs, because Debian is, you know, deciding what to backport, what not to backport, et cetera. If you scan our equivalents, none of those have any CVEs. And so, you know, I think I'm glad that more folks are starting to take this problem seriously, and not just because I don't have to explain that what we do is actually possible. But I don't think all of the approaches are necessarily created equal. And I think our approach is informed by an extremely long history in this space doing things like this to harden these images and figure out the path that we actually have to take in order to achieve the result that we are getting.
- [34:16]
B
Yeah, I think usually in this case it is so that rising tide raises all boats, I think, or just like a bigger pie or whatever analogy you want to give it. But it should overall be positive. But yeah, it's obviously also interesting when such a large company does come along claiming to do sort of a very similar thing. And I think that distinction that you've already made is very helpful. One other thing that was interesting, I think end of last year you guys announced Emirates oss. You know, this is like maintaining deprecated open source projects.
- [34:48]
C
Emeritus. It's a play on emeritus.
- [34:50]
B
Oh, emeritus. Yeah, I was like emerit. Is that like to do with OSS at the end? Yeah, emeritus. There we go. Nice. I was going to ask what the name meant, but you just. Yeah, there we go.
- [34:59]
C
Yeah. So this is, I think that the origin of this, you know, is that certain projects that are in some cases, you know, critically load bearing for folks workloads stop getting maintained. And so there was one that was near and dear to us. So one of the many, many, many, many projects that Dan I and others collaborated on Google is actually the starter project for one of our engineering managers. When she, she joined reporting to Dan at Google pre up was a project called Kaneko. She has since left Google and joined us and leads our container Steam actually. But so you know, this project like Google never really had, they never productized it, they never supported it. But this project literally closed a three, four year old issue for how do I build a container with a Docker file on kubernetes without mounting the host daemon socket without running with privilege so that I can, you know, start my own Docker daemon? You know, docker and Docker stuff like that. The kinds of things you don't want folks doing in your, you know, cluster if you're an SRE or your product security like Conoco closed that issue. It showed folks like this is how you can do it. And I think GitLab documented it as one of the ways to build container images. I was, I joke that I can't meet with a bank without them talking about how they use Koniko for image builds. And then you know, one day Google stopped really merging any pull requests. And I think it took about six months before they openly acknowledged like they were basically archiving the repo. And so we had a lot of history with this project obviously. And so we were like. And we knew how important it was to a lot of folks and so we decided we would take it on and we would make sure that it was receiving patches for various security problems in the libraries that are in there. But we weren't going to do a lot of feature work as the nature of sort of at some point a lot of open source projects reached this quote unquote done status and it was, you know, it was pretty mature. It was, you know, had a lot of the features that it needed from supporting the Docker file spec and we wanted to give folks a stable and safe way of consuming that piece of software where, you know, we would keep patching the security stuff, but, you know, not do a lot of feature work that would potentially destabilize their workflows. And so that was really the seed from which the this idea for Emeritas grew. And there have been a few other projects that we've taken on with a similar sort of philosophy. Right. We will build images from them, you know, we will patch any vulnerabilities that come in through dependencies. If things are disclosed in the package itself, we will look into that and try and remediate those things. But we're not going to do feature work. Right. I think one of the great things about open source is if someone really wants to take on like making substantial changes to a project, they can fork it and they can add their features to that. And if that fork becomes a successful alternative, then people can just start consuming that. And that's great. We're open source, working as intended and that can become the source of truth that folks use if they find the features that are being added valuable. And so I think there's some great examples of really successful forks that become bigger than the original project. I mean, I think a lot of folks don't appreciate these days that like Jenkins started as a fork of a project called Hudson because Hudson was owned by sun and then sun got bought by Oracle. And the person who started Hudson, Kosuke, left Oracle and wanted to continue his open source project. So he forked it as Jenkins. And now who talks about Hudson? I mean, I think it's probably a, a vanishingly small set of folks who even know what it is.
- [38:40]
B
I didn't know that story. I had not heard of Hudson. So that's super interesting I guess as we kind of like look forwards, I guess in the present and forwards. You have mentioned that chainguard is not just hardened containers anymore. So kind of using that framing, I think looking at things like sboms. So this is software bill of materials. We've covered these in past episodes as well. This is more like runtime behavior, like what is being fetched, like what is the provenance. This feeds into the fact that we do have regulatory frameworks coming in towards the end of 2027. One is the EU Cyber Resilience act, which, okay, it's EU, but it will no doubt frame still a lot of what happens on the US side. And no doubt you have EU customers, so they have to abide by that. And that is saying mandatory SBOMs, things like 24 hour vulnerability reporting. Are you kind of taking in that kind of thing into where the product's going as well, or you're already thinking about that. What are you guys thinking about all of this?
- [39:41]
C
I think it's a nice tailwind for us because I think we do a lot of the things that folks need in order to satisfy these things. All of our images carry sboms that we generate at build time. But we also go out of our way to make sure that SCA tooling that generates SBOMs works exceptional, exceptionally well with our software and the binaries that we include in that software. And so we definitely do a ton of stuff to make it so that folks can either get the artifacts that they need from us or generate them from the images that we are producing. And I think one of the interesting things just to sort of circle back to the Trivi breach, I think the European Commission itself was actually compromised by that, that trivia breach, right? And so they disclosed that they were compromised by it because of that 24 hour reporting deadline. I think it was beginning of May and something like 340 gigs of EU citizens data was leaked as a result of that. Right. And you know, that's, you know, something that wouldn't have impacted them if they were using chainguard's images. Right. And so I think that even if folks are seeking this really just to be compliant, right. There's a really nice security properties that you get out of, you know, adopting hardened images, adopting software that knows exactly what's going into it. I worry that when folks talk about things like SBOMs that they aren't talking about the complete picture a lot of the time though. And so everything that goes into our images is captured in our sboms. And I can assert that because everything that goes into our images comes from one of our packages. And all of our packages capture everything. Like every file that ends up in the final images file system comes from one of our packages. And so we have this sort of chain of provenance for every single file. We know exactly where it came from. And you can trace the builds all the way back, but that's because that we own that full end to end. And so I think that folks who aren't doing this right, you look at the way folks traditionally have built container images, images with docker files, like docker files. Most of the like official images are bootstrapped from like a random tarball somewhere that has a root fs. Hopefully that's the set of packages that that rootfs was built from. And then they start to add stuff up on top of that. Some of those come from the package manager, some of those curl pipe to bash in the Docker file build. One of my favorite things to call out around sboms is what our CEO Dan calls the WordPress test. Right. If I have an SCA tool, can the SCA tool tell me that WordPress is in the official WordPress image on Docker Hub? And I think we found one that can actually tell you that WordPress is in there. I think it's the Black Duck scanner, which uses a sort of signature, like file signature based scanning. But because the official image, like pulls down the source code and builds it in the container and like that's what you get, there's no metadata telling you that WordPress is actually in that container. Right. If you scan our WordPress images, because it comes in through the system package manager, it's represented in our package database. Every SCA tool on the planet can tell you that WordPress is in our images, but it's also in the SBOM that you're getting from us. So I think one of the things that I worry about, it's not just mandating sboms that I think is important, is this notion of dark matter. How much coverage does the SBOM actually have of the files that are in your system? And can I give you an empty JSON document and be like, here's your SBoM satisfies the schema, but there's nothing in it. It has zero percent coverage. Does it pass the mandatory SBOM check? Okay, that's an extreme example. But say I give you an SBOM that covers 10% of what's in the image and terms of packages that are installed, does that count for mandatory SBOMs? This is one of the tensions that I think that I've always struggled with with people being like, yeah, mandatory sbombs. It's like, this isn't a yes or no thing. This isn't a binary thing. We should look at it sort of like code coverage. Right. What is the coverage your SBOM has of your image? Right. Because an empty JSON document would obviously be zero. But I think the intent of it Is to have 100% coverage. But they're not saying that. Right. And I think that's one of the things that I just take issue with in general, when folks talk about SBOMs, they're not talking about the bar you should be clearing when you are generating that kind of metadata about images.
- [44:07]
B
That's super interesting. I guess these frameworks, and especially if they're trying to implement them as mandatory at all, then. Well, I guess it's this question, yeah, is it mandatory? Should it be mandatory or should it be? There's a bar, but then do you say mandatory? And I guess they had to pick one. But yeah, I totally agree. I think this will initially lead to probably a lot of subpar SBOMs. I think at least getting the terminology and getting a lot more. I mean, I can think of so many developers, if I was to say, just say SBoM, they have no idea what I'm talking about. That's going to be helpful. Yeah.
- [44:35]
C
When the original executive order happened in the U.S. there was also this tension where they didn't want to be prescriptive about what format you used. And there's two major formats, right? There's CycloneDx and there's Split PDX, which both have community backing, but they didn't want to pick a winner. And so there's no prescription around what format you use. And I think because of that, they also didn't really do too much around prescribing the structure because you'd have to do it twice and do all this other stuff. I would go so far as to argue that the package database that distributions include in their images is effectively an SBoM. And so I think that you could arguably check the box. Every SCA tool on the planet can generate you a fantastic SBOM from a distributions package database is the first thing any of them look at. It's all the other stuff that doesn't come in through the package manager that where things start to struggle, like the WordPress test I was talking about. And so that I think is one of the things that there's some tension around and you know, yet another gap in the, like. Okay, well, like what mandatory sbom? We're not talking about coverage. What format? Like, what are the minimum elements? What is the depth of resolution? I could tell you at a package install level, I could go down into the file level within a Go binary file. I could tell you the transitive dependencies with respect to every version of every Go package that got pulled into that binary. Then there's other fun things too. Go is really good about including that dependency information, but Rust doesn't do it by default.
- [46:07]
D
Right.
- [46:08]
C
And so you have to enable auditable mode in Rust compiler builds in order to get that same metadata. And we do that by default in our tool chains because like I said, we go out of our way to make it so that SCA tools work really well with chainguard produced images and binaries. But like you can get images today from aws. They build a lot of stuff in Rust, but like they're not turning on auditable mode. And so there may be vulnerabilities in some of the libraries they're using, but you can't tell because that metadata is not there. Right? And if that's not something an SCA tool can recover because the metadata is not there and it's not captured in the SBoM, who knows, right? You don't know whether or not you were vulnerable to that. And so that's part of why we go out of our way to work really well with SCA tools in addition to capturing as much metadata as we can. But there's file coverage, package coverage, there's depth of resolution, there's how you actually like structure all of that information as well. There's so many things that are not well scoped. And so like saying mandatory SBOMs without any of that other stuff means it's really hard to satisfy that in a way that is particularly meaningful. And so as a result of that, right, like we do produce SBOMs but like they don't necessarily have the same shape everyone expects expects because everyone thinks, you know, you ask 10 different tools what the right shape to produce an S bom is, you'll get 10 different answers. But if we work really well with SCA tools that can generate their own SBOMs and you can verify the image came from us via our image signatures, you can generate your own sboms from our images that we know will work with your tool. And so that's why we do it sort of both ways, right? And we don't think that the SBOM space is really matured enough that just generating an S bomb is enough. And it's because of all of these unanswered questions about shape coverage, depth, et cetera, that I think make it hard thing to satisfy, at least in a useful way.
- [48:06]
B
Yeah, absolutely. Kind of related as we're sort of cruising to the end of the episode, but there's a couple of things. First is related to what you've just been saying, Enterprise adoption, what friction points could you, I guess, speak to? That's not a friction point, it's just a perceived friction point. And actually, I guess using chainguard is like, you should just talk to us.
- [48:26]
C
There's definitely some FUD out there about stuff like migration and things like that. Fear of the unknown. It's like, I don't know how hard it is to migrate, so it must be impossible. Right? And so, you know, this is some of what we deal with coming from other distributions, you know, like Debian, like a RHEL based distribution and so I think that we've gone out of our way to accommodate in various ways as broad a base of compatibility as we can. Right. So we use the same package manager that Alpine does. But one of the big sources of incompatibility in Alpine based images is the muscle libc. They use a different LIBC than virtually every commercial distribution on the planet as well as many of the non commercial but widely used distributions. So like Debian, ubuntu, all the RPM based distros, you know, Amazon, Linux, RHEL, etc. They all use glibc. And so when we bootstrapped our distribution, even though we decided to use the APK package manager, we bootstrapped our distribution around Glibc to get that sort of maximally broad compatibility. And so I think there's definitely a lot of FUD around migration, but it's definitely a spectrum. And I think the main thing is, okay, depending on the package manager you're coming from, you know, if you're coming from Alpine, it'll look very familiar. APK ad package name. If you're coming from Debian, yeah, you'll have to change apt get into APK Add or you know, if you're coming from the Red Hat ecosystem, you'll have to change, you know, yum, install into APK Add. But we have tools that help streamline this, make this easy. We have a new tool that we call the Gardener that helps drive a local migration for you, leveraging, you know, our agents to help you, you do some of that package mapping and stuff like that. So if you're coming from Debian, we can be like, oh, this package is called this in the Debian ecosystem, but it's this within the chainguard ecosystem or if you're coming from the RHEL ecosystem, it may be called something else, right. And so mapping package names, doing those kinds of things and really helping you drive that further and further to completion. We have a bunch of other stuff that we're working on to sort of smooth migration, but you know, in some cases is it's really just a matter of dropping one of our images in. So the vast majority of our images are application images which most folks just take and run, right? Like something like an envoy, you typically plug it in and run it as a proxy with some configuration and maybe you're running something like, you know, istio. How many folks are modifying their ISTIO images? They're mostly just taking it and running it through helm chart or something like that. I think the one caveat to that is folks who want to inject their own root certificates into it, maybe they are using something like zscale or something like that. But other than that it's actually relatively uncommon for folks to be modifying the application images. And so those will just drop it and run as the application. You don't have to worry about it having a different package manager, you're not installing packages. And then when it comes to migrating stuff like Docker file based builds, we have tools that will help drive that migration and help you get as close to that finish line as possible, if not all the way there. And then with things like our library's products, a lot of times our customers goal is to effectively just turn a switch. Like a lot of our customers will proxy our language packages through something like Artifactory or cloudsmith or Nexus, right? And they've already done that to sort of insulate their internal networks from developers just being able to pull whatever. And so when you have that kind of mature setup where you can regulate what packages your developers consume and it's coming in through Artifactory, maybe it's just passed through from the outside world, but you have that cut point, you can cut that over from pulling from say Maven Central to pulling those from Chainguard and your developers may not even know that you've switched it to chainguard. So on the library side it's much more transparent in terms of, you know, dropping in and replacing things. That's our goal to be as drop in as possible. But I think some of where that can, you know, deviate a little bit. It's the same migration problem you'd have if you wanted to move to one of our images that doesn't have a package manager. It's the same as moving to a Google Disk for less image, which half of this cloud native ecosystem is using these days. So it's clearly possible. But if I were to move from like a Debian based image to a RHEL based image, I would have to rewrite my package install lines anyways and figure out the package mappings and get it working again anyways. And so any sort of migration like that will have a little bit of work. But folks are already going through those kinds of migrations too, like one lts release to the next and where a lot of system packages are changing. And so we try and give folks something where we aren't an LTS based distro, we are a rolling distro. So if they migrate to us once, which we can help them do as smoothly as possible, they shouldn't have to go through that every four years when there's a new lts as well, we can hopefully be the last migration that you ever do. And we, like I said, are going out of our way to make that as easy and painless as possible.
- [53:28]
B
Yeah, amazing for anyone, I guess, at that sort of enterprise level. But medium, company, whatever you want to call it, like just that on Ramp isn't maybe as scary as it it sounds. Very briefly. Just we were talking about it before we started recording anthropic Mythos Fable model or Fable, which is derived from Mythos. Super curious. Just like this is a very bleeding edge because it's only been announced in the last 24, 48 hours. But how does that affect your day to day or you've been, I think, using it a little bit. What are your first thoughts?
- [53:57]
C
I think it's when it first came out, everyone freaked out. And then there was a whole group of folks who were like, this is just marketing. I've seen some of the findings. It is not just marketing. These are real vulnerabilities. It has very good capabilities. There's several different aspects of it. I think I mentioned the Firefox numbers. Right. They found a whole slew of vulnerabilities that other agentic tools that they had been running weren't finding. Right. And I think one of the things that makes it exceptionally interesting is its ability to chain together multiple vulnerabilities, which, you know, it's in real time. We're watching it close the skills gap that's needed. Like this notion of sort of scarcity. Right. Like right now there's the skills needed to find and weaponize. Vulnerabilities are scarce. It's hard. It takes time. Right. And that time is dropping precipitously. And so I think the Fable model that they released today has all kinds of security controls. The first thing I threw at it, mostly for grins, was to audit Security isolation tool that I've been having OPUS just audit in a loop and find and fix vulnerabilities in this abstraction. I've been very impressed with how good OPUS has been. I've been trying 474 8. So I threw this at Fable and I got this prompt back being like, it looks like you're trying to do security stuff. I'm downgrading to Opus 4.8, so it wouldn't let me do it. So they said in the announcement that they had put in security controls around keeping folks from doing this for vulnerability research. And it caught me. So I think there are controls that will keep folks from using it in that way. Folks always find a way around those controls, though. And so I think the thing to bear in mind is this is not marketing. This is very real. This will find tons of vulnerabilities that other models probably haven't found yet. And folks need to be prepared to patch things at machines and speed because you could already weaponize vulnerabilities at machine speed. And now with Mythos coming into the picture, being able to find even more and more effectively weaponize vulnerabilities at machine speed, if you aren't prepared to patch things at machine speed, which is really what we've been building towards at Chainguard since before we even realized this was a problem folks were going to have, you're going to be pretty unprepared for the world of hurt that I think is going to to play out over the next six to 12 months as at least the initial tranche of Mythos vulnerabilities become known to the world. And then it's a muscle you need to build, even if you get that initial tranche patched. Because I would be very surprised if we didn't continue to see step function improvements in model capability, the quality of the harnesses for leveraging the models to find these vulnerabilities. And it's going to be an interesting next year for sure.
- [56:45]
B
It really is. So, on that note, thank you so much again for coming on again. Learned a lot this time as well as I and I think the audience did last time. So, yeah, who knows, we'll maybe get you back on in another two years. We'll hear. Because, you know, this landscape is just changing so much and so fast. So thanks again and I'm sure we'll catch up again.
- [57:03]
C
Thanks for having me,
- [57:10]
B
Sam.