Loading summary
Transcript48 lines
- [00:00]
A
Foreign hey everyone, I'm James Wilson from Risky Business, and welcome to this sponsored interview with Dan Guido, CEO and co founder of Trail of Bits. In this interview, I'm going to be talking to Dan about an exciting new partnership that's literally just been announced in the last few hours between Trail of Bits and OpenAI. It's under the banner of a new initiative called Patch the Planet, and it is specifically designed to help open source maintainers deal with the deluge of bug reports that they're getting at moment and all other manner of pressures that are placed on them by the Voldempocalypse, bugpocalypse, AI era, whatever you want to call it. What I liked about this initiative is, as Dan explains this, this is not just about fix all the bugs, because as we know, you just can't patch your way to safety anymore. There's more to it than that. And I like how Trail of Bits are really approaching this from the perspective of let's spend just as much time finding and fixing bugs as also pouring time into architectural fix thinking about what are entire classes of bugs telling us about the structure of a project, how do we manage the SDLC to be able to address bugs faster and get our patch cycles under control? All these great things that go above and beyond just the usual rhetoric of
- [01:20]
B
fix all the bugs.
- [01:21]
A
But listen, I'll drop you in here where Dan introduces the new initiative and then talks about why Trailerbits is a great partner here for OpenAI and how exactly this is going to help open source maintainers enjoy.
- [01:33]
C
Trailer bits and OpenAI are trying to help project maintainers stay ahead of the curve when it comes to AI bug finding tools. When you launch an open source project, there's usually just one of you and there's an unlimited number of people on the Internet trying to find bugs in your software. And it can get kind of demoralizing because a lot of those bugs end up being duplicates, they're low quality, they're hallucinations, and you have to spend a lot of effort to review all of them. So we set out to help these maintainers come up with ways to deal with that influx of new issues, increase the pace of their software development and increase the security of their code.
- [02:05]
A
And so what's the sort of the.
- [02:08]
B
I guess, what's the magic that you bring to the table? What's OpenAI bringing to the table? Like, how does this partnership form? Like something that's greater than the sum of its parts?
- [02:17]
C
Yeah. So I think the thing that we've all realized now is that finding bugs is becoming the easy part. Like most people can take a frontier model, especially the best in class GPT5, five cyber models, the current generation or Mythos or whatever, and ask it, hey, find me bugs in this piece of software. And if you have a little bit of skill, you can usually discern what the good bugs are versus what the hallucinations are. But what really you don't have is you don't have judgment about how impactful is this bug. Does this fit into the threat model for the software? How do I chain it together with other bugs that I might know about? How do I judge the severity of it? What's the right way to patch it? When should I patch it? How should I patch it? So all these sorts of questions are things that can only get answered with expertise, true expertise, which is what Trailobits has in spades. So when we take our people and we combine them with the best available open source models and AI coding tools, we can do a lot of help. And that's what Patch Planet was about.
- [03:15]
B
Is that because, as you say, you're bringing the deep subject matter expertise, the people to this, or are you bringing also sort of any sort of custom tooling or skills or. I'm just, I'm curious as to what sort of the collateral is that is sort of bearing down to really help these maintainers, as you say.
- [03:31]
C
Yeah. So Trailerbits has a lot of experience using tools like codecs to find bugs. We got on this pretty early. We've developed a lot of expertise around it and there's sort of a maturity process you have to go through. You know, your first week trying to drive one of these tools around, you're not quite sure how to use it and you end up with a bunch of sloth yourself. So I've been very public about the journey that Trailer Bits has been on to gain expertise in this field and train all of our staff, not just like a couple of people that know about AI, all of our staff, to understand how to use AI. So we have a ton of internal tools, we have a lot of agent skills, plugins, harnesses, and just experience. Way back last summer, the story was the DARPA AI Cyber Challenge. And I think that was helpful in bootstrapping a lot of our expertise. But since then we have not given up digging deeper.
- [04:19]
A
And how do you guys go about,
- [04:22]
B
I guess, doing the triage of which open source projects to move to the top of the queue and help out first? Because I'm imagining there's a Bit of a. I would imagine there's a few different dimensions here. There's projects that are underserved, there's projects that have probably already had a lot of attention placed on them, but they're so critical that you might want to give it yet another look. There's so many open source projects out there. How do you even know where to begin and prioritize who to help first?
- [04:47]
C
Well, so first things, we got a lot of funding from OpenAI to work on this, so we threw like a huge portion of our team at this for a couple of weeks now. At one point we had about, I think 25 to 30 engineers working full time on this project. So we're really trying to address the largest risks on the Internet all in one big swipe. But Trailer Bits has been around for 14 years and through that time we've worked with dozens of the largest open source projects out there. So when we began this, we went out and called up all those old relationships and said, hey, remember us? We did all this great, important work for you before and we had that credibility that could open doors with projects that frankly had written off the utility of a lot of these tools. I think there's widespread skepticism that AI coding tools can help, especially from open source maintainers, because they've been on the receiving end of so much slope that they think that these things are nothing but pain. So when we come in and we have a previous track record of successes and we've already built up the expertise to know how to use them right, there's some likelihood that, hey, this might turn out to be a useful engagement. And turns out we've done a lot of good.
- [05:56]
A
Yeah.
- [05:57]
B
Which I guess goes a long way to answer the question of why trail of Bits in this partnership, you guys have got, as you said, the combination of the long standing expertise in the industry, the existing relationships with some of those projects and the point you mentioned there around, you know, maintainers have sort of written off these, you know, coding models and they, some have even, you know, shut off their bug bounty programs to large language model generated bug reports. I mean, I think the most noisy maintainer in that space has been the curl maintainers. You know, they, even when they got Mythos access, there was that post where they said, oh look, we got Mythos and it found one bug. Before we started recording, you mentioned that you'd had a little bit more success than just one bug in curl. Do you want to tell us a little bit about what you found there?
- [06:40]
C
Yeah, sure. So we reported, I think 22 bugs to curl. Which really just goes to show that these tools, in the hands of an expert, along with tools that have been enhanced by custom skills and deep knowledge of a particular code base, can do a lot more. And that's what you're getting when you work with Trello bids. So, yes, the models get better every single iteration, but you still need this deep expertise to truly judge how to steer them and then what to do with it. So I think it really backs up the fact as well that finding bugs is only half the battle. If you have more, better, higher powered bug finding tools, you're just going to find more bugs. So at the end of the day, the improvements that really matter are things that are architectural enhancements to the project, things that allow them to push out patches quicker and deploy them quicker. And that's a lot of what we focused on in this project. We didn't just look for bugs, although we found plenty of highly severe ones. We also worked on these SDLC type improvements to everyone that we touched.
- [07:43]
B
Which is a good point, right? I mean, we've talked a lot about that on the Risky Biz shows, and I've done a few interviews on Risky Biz features where it's, you know, the message is pretty loud and clear that you just can't patch your way to safety alone. There's got to be other things that you do. If you were to think about, like a rough breakdown of time that you're spending versus finding bugs versus helping with architectural things versus looking into STLC improvements, patching faster, all that sort of thing. Like, is there a bias for any one of those taking up a larger chunk of time?
- [08:14]
C
Yeah, I think we tried to hit about 50, 50 internally. We wanted to spend half the time bug hunting so that we could wipe the floor clean of some of the easiest to discover issues. We want to reduce the incidence rate. I don't want to have some kid with an open source model just drive around without even a harness and start popping out severe bugs. That's bad for everybody. But after you wipe the floor of a couple of the most severe issues, it's all about how do we patch faster, how do we give agents better guidance to operate on this code base? How do we reduce the context required to do good work? It's more than just providing an agent's md. We have to delete dead code, improve testing infrastructure, get feedback in the development life cycle. There's lots of different things that you can do that make it easier for not just agents, but also humans as a Human. If I do a security audit and I get lost in a bunch of dead code, that's wasted time too. So at the end of the day, our improvements matter just as much to humans as they do to agents.
- [09:12]
B
Right. That makes a lot of sense. And talking about those improvements, it makes sense that we've got to spend, I guess, this initial large investment in, as you say, wiping the floor clean of all of these bugs that are out there to sort of level up in the world of AI agents.
- [09:29]
A
But I'm curious, where do you think
- [09:31]
B
this goes in, let's say, three months time, six months time, a year's time? We obviously can't keep pouring huge amounts of money into just patching bugs. And as you say, you can always manage to shake another bug loose. Is this like a blip on the curve that we're going through at the moment in terms of finding and fixing these things? Is this just the new normal in
- [09:51]
A
terms of the rate and velocity of fixing things?
- [09:55]
B
What becomes of this once programs like this, I guess, have done their job or return to wind up?
- [10:03]
C
Yeah, I think this is the new normal. And we've got to help project maintainers keep up and stay ahead of the curve. And that's a process. It's not like a magic switch that you can turn. It's really a human process where you have to work with project maintainers to help them understand and get tooled up on the benefits of these tools and as well as make the code bases amenable to having the tools used on them, because new models are going to come out every single month and there's enormous supply of people that want to use them to find bugs. And we just need to do this in order to skill up a lot of the community to stay ahead of the curve. So this project is ongoing. We're accepting applications from open source projects to get a free week of consulting time from somebody from Trail of Bits paired with Codex to evaluate whether we can instantaneously find any bugs in your code base, as well as improve some of the architectural issues with it. And this process, it's a real open line of communication. We're trying to figure out what the most important changes are from your side and make them a reality using the best available AI coding tools. We got a big page up on the Trailer Bits website where you can sign up and hopefully we can help more people.
- [11:22]
B
And imagine that. Going back to that triage question, I imagine it's going to be a little bit difficult to manage what will probably be a deluge of responses coming into that. I mean, I know it's only been out for a couple of hours, but have you already seen sort of an initial bit of interest from projects coming in?
- [11:37]
C
Yeah, I've seen the scroll back in Slack go up pretty quick this morning. I think it's important to point out that Trailer Bits is a pretty radically open company. So when we work on this, we're also keeping all of our notes about how we consult with these projects out in public. So you can go to Trailabit codecsconfig and read more about how like the process that we follow to get people set up with it. I have similar setups, you know, from my unprompted talk earlier this year where I talked about Trailerbit's journey down the AI native consulting path. I have a whole suite of guidance around cloud code as well. You know, we just want people to start adopting these tools because the benefits they offer are dramatic. And even if you don't work with us, you can still benefit from what we put out there.
- [12:22]
B
Yeah, that's very cool. And I will make sure I check that out myself because I'm increasingly feeling like, although I'm using codecs and Claude code all day, every day, I still feel like I'm largely an organic harness and I'm probably not doing enough to actually put it this way. I bet if I spent a day not coding with these agents and just building better harnesses and loops and all those things around them, then I'd probably have a much better time going forward. But is that the sort of advice that folks can find there in terms of, like, how to just go from, you know, native codecs with a smart prompt is one thing, but surely there's a whole lot more that you can build around that. Is that right?
- [12:57]
C
Yeah. We talk about our agent skills, the hooks we use. We talk about how to use goal to build loops that verify themselves. We talk about MCP servers that we like, various bug hunting strategies. I mean, the whole thing is in there and it's something that's built up from more experience consulting. Like as we use these for ourselves, we try to keep these resources up to date. It's helpful. Like when you get hired at TrailBits, you can study through the onboarding process before you even join the company, so other people can just sit down and do it too.
- [13:29]
A
And if we sort of zoom out
- [13:32]
B
now and think about the broader industry and the crazy volupocalypse bugpocalypse time that we find ourselves in, there's sort of this funny dichotomy, I guess, where it's like we're finding bugs faster than ever and we're finding some really nuts, gnarly bugs out there in some of the most critical systems. Like a couple of weeks ago it was like every morning you woke up there was a new Linux Local Priv. Then the next week it's like every morning you wake up there's a new Windows Local priv and there's all these bugs shaking out in the very bits and pieces that run our entire Internet.
- [14:00]
A
And yet, as you say, this is just the new norm. But we're also not seeing a huge
- [14:05]
B
amount of, I guess, wreckage or.
- [14:09]
A
But is that.
- [14:10]
B
Because that, that hasn't come yet, like that that sort of reckoning is not yet upon us. I'm just curious how you think about the.
- [14:16]
A
How do you tally up all the
- [14:17]
B
things we're finding versus the actual impact that we see?
- [14:20]
C
Yeah, no, great question. I don't think attacker tool chains have caught up to this yet because the opportunities that these vulnerabilities offer to an attacker are really one off. They're unique, they don't fit into a kill chain to a repeatable pattern of how people get hacked. It's all these weird capabilities that they would offer where if you're in this situation, here's a solution to this problem, to that problem, to this problem where a solution is like how to island hop inside of a company or gain access to some database that you're not supposed to. And I actually think that attackers have a lot of work to do before they are able to instantaneously take advantage of these opportunities to perform intrusions. So I'm kind of waiting to see some more innovation on the attacker end of things. Right now I feel pretty good about the spot that we're in. I think there's a lot of bugs and we have this one opportunity right now to fix them before all the open source models get capable enough that we can find them all and before the attacker tool chains tool up to be able to take advantage of them all. So 2026 is a big opportunity and that's why I wanted to work on this project now, because I don't think we have the same chance in 2027.
- [15:34]
B
Yeah, that's right. And talking of open source, did you guys happen to do any comparisons of sort of the results you were getting from Codex with some of the open source models? Like, you can't go anywhere at the moment without hearing about glm and it's sort of taking the forefront of that space. But yeah, curious. What, have you been experimenting with open source models at all or has this purely been with codecs?
- [15:54]
C
We have. I think the way that I think about using open source models is to outsource some of the more reliable tasks that we perform or the less intense tasks that we perform. So like I want to get my entire sales and operations teams on open source models because the things that they're doing to like update statements of work and edit documents and whatnot don't need frontier level knowledge, frontier level intelligence to be able to complete. But when I think about finding critically severe bugs and hard targets, I want to stay on the frontier. So I'm going to be using Codex 5.5 Cyber or mythos or something else. I can't afford not to.
- [16:30]
B
Yeah, yeah, I guess that makes a lot of sense, right? Being diligent about where the right model makes the most amount of sense and getting the, I guess the economic benefits as well as speed and autonomy as well out of using some of the open source models where you can.
- [16:44]
A
Well, Dan, this has been super interesting
- [16:46]
B
to hear about this initiative. Before we wrap up any crazy interesting stories from the work that's come to
- [16:53]
C
light so far, I just am shocked at the amount of work we've been able to do in just five days with some of these projects. We rewrote the entire Release process for Python.org we had several hard targets where we built these end to end fuzzing labs with all kinds of variant testing and other kinds of nice to haves that probably would have taken a sophisticated engineer at trail of bits like three or four weeks to work on. We did in one day. Just being able, like having achieved the level of mastery that our team has with AI coding tools. The things that we can do in such a short time period now are shocking.
- [17:33]
B
All right, well Dan, where can people find out more? Is there a website? Where can they go to, I guess
- [17:38]
A
to learn more both to skill themselves
- [17:40]
B
up, as you said, with that sort of great set of information that you guys are so open and published, but also to sign up to potentially have access to this program.
- [17:48]
C
Yeah, so you should definitely look at our GitHub. We have guides for how to use codecs, how to use Claude code and all the different secondary tools that trailer bits is built to use them, including our sandboxing tools especially. But the Patch of Planet project is right on our front page. You'll find a little icon that describes what the project is and how to sign up. And if you run a big open source project. You definitely should.
- [18:10]
B
Excellent. All right, Dan, we'll leave it there. But thanks so much for dropping by, and super interesting to hear about this. Thanks for your time.
- [18:16]
C
Thanks a lot.