Loading summary
Transcript20 lines
- [00:04]
A
Hello everyone, this is Tommy Wren and I'm here today with Dan Guido, the CEO of Trail of Bits, for a Risky Business news sponsor interview. G', day Dan. How are you?
- [00:13]
B
I'm doing great.
- [00:15]
A
So Dan, you're telling me that you're going all in on AI and I was kind of shocked. Oh really? I was looking at your Claude code plugins, just looking at the titles. It seems to be very specific things where you can get maybe not yes or no answers, but you know that there's a particular problem that you're trying to solve. You're not just talking to it, trying to get an answer. Like my impression is that if you give an adversary the ability to talk to LLM, there's a fair chance they'll convince it to do something that it shouldn't. But these plugins are not that at all like dealing with defined problems where you want. I'm not sure if you have an algorithm or a set of steps that you want to be able to go through.
- [01:05]
B
So I would say these are specific bite sized pieces of behavioral guidance that you give to Claude because the LLM is already trained on the entire universe of knowledge. Like you're not going to teach it something new by like for instance, one of our, one of our skills is about interpreting dwarf binaries and we're not going to give it the entire dwarf binary like reference manual because it already knows that it read that in the pre training. Instead what you're do, what you're, you're doing is you're reminding it like, oh, here's how to use that information, here's how to operationalize it, here's the tools that you can call, here's what you should do with the output, here's how I want you to use it in order to be most effective for the task that I'm trying to work on.
- [01:45]
A
Right. What is a dwarf binary?
- [01:47]
B
It's just like a executable format. It doesn't really matter. It's just, it's a complicated binary format and I could waste millions of tokens repeating back exactly how that binary format works to the LLM, but it already knows it. But what it doesn't know is it doesn't know what tools are installed on my computer. It doesn't know how I want it to be used. It doesn't know the workflow process that I'm going through in order to reverse engineer a binary. And, and the skill is a pack of knowledge, a knowledge pack that guides it to use that knowledge the right way. So here's the thing, all these Claude skills, they are a phenomenal contribution to the public and to everyone else's ability to get security tasks done with especially Claude code, but also all the other agents that interpret this format. But this is the output of a systematic process at Trail of Bits, where we are adapting the company to be AI native. And there is an entire process internally, a whole workflow, whole system, a whole interlocking series of process steps that are helping us rapidly adopt the latest technologies to make them safe to use for privacy sensitive engagements with clients and to build the trust and rigor into them so we know that they work correctly. And as we discover what those techniques and what those outcomes are, we polish them up and we publish them. And that's the output that you see on the trail of its cloud Skills Repo. So this is existential for me. I am fully bought in that the entire industry is going to change over the next year and, and consulting especially is going to have some dramatic effects, you know, placed on it that the difference in output from a consultant that uses AI and one that doesn't is going to be very easy to see over the course of 2026. And I want to make sure we're at the front riding that wave so that we can justify our continued existence. So that's, that's really like where all this stuff is coming from. And it's the most important thing that I'm working on right now.
- [04:04]
A
Do you feel that you're being pushed into this or do you think that you're being farsighted? Like, is this something that is obvious to everyone in the field?
- [04:13]
B
So, no, I don't think it's obvious to everybody. It took a while for me, like I had to spend a lot of time investing in understanding cloud code, the Cloud Agent SDK, the Deep Agents SDK, looking at the work we did in the DARPA AI Cyber Challenge, the aicc and really getting some hands on time with it. And then I think me, like a lot of other people in tech, something kind of clicked in November and December and I think it coincided with some new software updates from Claude Code as well as the opus4.5 release, where now these tools have passed this minimum watermark that I need where I can trust their outputs to a degree that they work the vast majority of the time. So I've been able to get a lot more done with these. And now it's much easier for me to connect, like, oh, okay, now I see where things are going. Like, I don't think that people are going to be using Google and searching for products the same way we do right now. Instead we're going to like, I think the Sequoia people have talked about agent led growth where most people are just going to be interacting with an agent to recommend them products and the agent isn't going to know that your brother or whatever worked for this company. Like it's just going to read the documentation and try to make the best assessment of what meets your requirements and provide you that answer. So that shifted our entire marketing approach. Like how does a company like Trailer Bits go to market in 2026? Is we make our content discoverable. We include comparison tables. We provide as much like a volume of information up front to enable an agent to make a decision. So we started a couple months ago, we're like, ah, the Trailbits website doesn't look too good. It's kind of frozen in time from like 2015. We need to do a visual refresh, we need to choose like maybe a new color or something. And now it's like who's going to look at the website by the end of 2026? I need to make sure all my content is available in markdown. And this is really like, this is great because it aligns with a lot of the rest of strategy from Trailbits. From the early days, we've always been big on sharing our knowledge. We always care about releasing tools that other people can use. And all this aggregated information that's available in public now means that it's much easier for an agent to find it, which means we're going to be at the top of a recommendation list when somebody searches for it. So stepping back, you know, again, this is just sort of an output of a whole organizational reformation that Trailerbits is going through right now. And you're going to continue to see us publish, you know, more Claude skills, more tools, more, more guidance. Change the way our website looks and the way our content is discoverable. But internally, you know, this is a process that I think a lot of companies will have to get through in 2026, kicking or screaming. And I think there's a lot of things that people can learn by observing the way that Trailerbits is acting and replicating those processes internally.
- [07:06]
A
Okay, so to me it sounds like you're saying many companies will be like this. Dan, Is it every single company? Is it just a minority of companies?
- [07:17]
B
Yeah, I think this goes for any company. Like right now you're seeing there are some companies that are just able to ship and they're able to ship a lot faster than everybody else because for people that can build the systems to fully adopt AI, they will be able to be more responsive to customer needs, they'll be able to reach the market faster, they'll be able to achieve objectives that they just couldn't before. Internally, for our team, AI has put things like formal verification in reach for much larger number of projects than we'd ever considered it applicable to before. Because the entire testing process like AI is. I heard somebody describe AI as like a trick to get developers to start writing tests, but it really, it like, it makes writing tests trivial. Just this morning over lunch, while I was eating one of my bowls of slop, yeah. I picked up a legacy project from Trail of Bits that had no tests. And I wrote a full suite of 100% coverage, integration and unit tests with the help of Claude Code. And now what we're really trying to do is find a way to unlock the ability to use AI routinely at the frontier, capabilities that come out like every single week. And in order to do that, you have to have verifications built into all the steps. Like, so internally with Trail of Bits, we've set things like new coding standards, we've set architectural guidelines, we have to build AI native software. You know, we can't have like interactive terminals. Everything needs to be independent. Everything needs to be. If there's an API, it needs to be one for one with a cli, these sorts of things. And we're really leaning on using continuous integration linters and testing tools in order to get that continuous validation cycles that we can actually let the agents do their jobs.
- [09:14]
A
Right, right. So I'm just trying to internalize this and try and understand it. So it feels like to me what you're saying is there's jobs that we want to do, and AI can do those jobs when they're defined and small enough that it becomes like a building block and then we can sort of. I don't know, is it like Lego string those building blocks together or maybe it's like a pipeline and then we can get a lot more done and a lot more stuff that was maybe not necessarily super intellectually difficult, but was time consuming. And so I guess writing tests. Right.
- [09:51]
B
I don't think you're going far enough. So, like, not far enough. Yeah, no, so. So one of my guys this morning, he was like, I'm tired of doing fixed reviews. I don't want to do fixed reviews anymore. I wrote an agent orchestrator that given, you know, one software project from three months ago and the same software project from today and a list of bugs, it goes through and verifies the absence of all the bugs that we reported and it evaluates all the side effects of the code changes that were made. And you know, he did that using a set of tools that we offered internally, using a coding standard that we had published, using architectural recommendations that we had offered, and with the correct application of this, with a model that receives the right behavioral guidance, that has the right gates to verify that it actually did all the things we wanted it to, it's possible for us to take what used to be a three day process and turn it into an automated action that an expert can oversee in about 10 minutes. Yeah, and that is a complicated task, right? It's, it's, it's still, you know, fixed review is not the most complicated thing in the world. But it's January 26th that we're recording this right now. And you have to imagine playing this forward, what this is going to mean about the state of the industry in June.
- [11:14]
A
Right.
- [11:15]
B
Claude code came out only a couple of months ago and Codex and the Gemini, Cli and the rest and open code and all the rest of these things all sort of popped up in the second half of this year. So I'm preparing for a future where like if a client hires me and I can do the job in 10 minutes and there's somebody else that also gets an RFP from that client and that person, that consulting company or that person takes three days for the same task, you know, as long as I can prove that what I'm doing is correct, that I have rigor built into the process, that I can trust the outputs, then I'm going to win every time. And this is the problem that every company is going to be in 2026.
- [11:57]
A
So the rigor and the verification, where does that come from? How do you get that?
- [12:02]
B
Yeah. So from a standard development perspective, I joke that I'm a machine that adds pre commit hooks to projects because lately that's what I've been doing. I've been going around the entire GitHub organization, that trail of its owns, and I've been adding the most stringent, pedantic pre commit hooks to every single project. Because what AI really needs is needs feedback that the job was done the right way. So it needs 100% test coverage, it needs integration tests, needs functional tests, it needs linters, it needs formatters, it needs type checking. You know, it really needs to understand what good is and there needs to be a system to provide that feedback. So from a security perspective, you're also looking at this like, if I want to write a tool that writes exploits, I need a verifier. I would call it. I need a verifier that I have been able to climb a hill and achieve some specific objective. Where that objective is popping a shell on another computer. So some of these quality gates, these verifiers, are very simple. They're running a linter and checking if it passes. Others are more complicated, and it can sometimes be more difficult. Like, how do I fix a linter? Well, I can run auto fix and then it fixes the linter. Or, like, I can strip out an end of, you know, some white space over here and it passes the linter. How do I go from not getting a shell to getting a shell? That's a much harder task. And that's why you have skills that help you point it in the right direction, that get there. But at the end of the day, this is one of the most important things that I think we're developing at Trail of Bits is we have this base layer of quality gates that are applied in our standard development process, where now it is not a negotiation. You have 100% test coverage, and you have all the most pedantic linters turned on with no warnings whatsoever. Everything's an error. But then from a, like, a product development perspective, for trail bits, like the services that we offer, we need to have a whole suite of effectively proprietary verifiers for all the different bug types that we care about so that we can take agents and we can point them at a project and they can eventually seek out what that goal is and achieve them. And those verifiers need to be verifiable. Like, we need to know that. Okay, if you think you found an exploit, you're actually right. So right now we're outsourcing that to humans a lot. Like, you know, the AI gives you an output, and then you have to go, like, check it, and you're not quite sure, and you go inspect all of it. And okay, maybe this is right. But if we make progress on this through the rest of the year, then it'll get closer and closer to what the goal should actually look like until we don't really need to pay as much attention to it at all.
- [14:35]
A
Right, Right. So it seems to me that what you're doing is adding so much feedback that the. I guess the longer an AI has to think, the greater a chance. Well, the longer anyone has to think, the greater the chance that you'll make a mistake somewhere along the way. So if you had a whole lot of feedback in the middle of that process, or in the middle of what would have been a longer process, then you're reducing the chance that it'll make an error. And so then you sort of keep it on the straight and narrow. And I guess in the old days it was just too much work for people to do that.
- [15:07]
B
Yeah, yeah, I mean, look, you know, most projects, they don't have CI set up, so they just wait for a user to complain when the software breaks. Then sometimes you put up a CI, but the CI takes an hour to give you feedback and you know, you can't run it on your own computer. And then some people optimize their CI and get it down to five minutes and that's great. And that's what we've been doing too. Like optimizing your CI is now really important. But then you can keep pushing it left, you can keep going, you can say, oh no, I'm going to make pre commit hooks. Oh no, actually, instead of catching it with a linter, I'm just going to add type checking to my language. I'm going to use Astral's TI in order to make sure that my Python is all strongly typed and then it can't go wrong. That's why a lot of like the, the lingua franca of AI is TypeScript. And there's a really good reason for that. You know, trying to use a language that doesn't have types of is just walking around stepping on rakes. So that's sort of the, the process that we're trying to go through. We're trying to shift everything as far left as possible. And now we have the tools to do it because all the grunt work is easy, but it's also critical to do it. So as we develop all these constraints on the software that we're building, this is something that we're pushing out to trail of bits on an ongoing basis. So we figure out these new constraints, I figure out these coding standards, I figure out, oh, I need this new linter or I have a hook, a policy enforcement for Claude to tell it to do something and not be able to avoid it. So we're also using AI to distribute these things. Like I have a bunch of orchestrated agents that I can use to assess the entire GitHub organization for trail of bits, all thousand repositories, and evaluate it for conformance against a spec that I have of like, hey, everything should Build on Python 3.1, one or more complicated things like nothing should have a vulnerable dependency. So I can enforce these new standards and tools and frameworks that I put out using agents themselves. And all this stuff, like all this is to say that the winners in 2026 are going to be the people that figure out how to compound their effort through the process they have internally where you need to be able to pick up new tools, have people learn how to use them, figure out when people have a working pattern that works for the company and then codifying that into practice, enforcing it with automations and then getting everybody to use it. So that's sort of what we're trying to do at Trail of Bits this year and we're already a lot of the way through it. So these Claude skills, they are one embodiment of a way that we have codified patterns that we realized worked and then pushed them out to the whole company. And that compounds because now everyone in the company can use these skills. We discover new things that you can do with it. New Claude features come out every week. So we have these regular peer to peer learning sessions where last week Claude Cowork came out for teams and three hours after it was released I was on Google Meet training our operations and GTM teams how to use it. And that's what I think a lot more companies are going to need to jump on the bandwagon to do.
- [18:17]
A
Right, right. Well, Dan. Dan Guido, CEO of Trail of Bits. I don't know whether to be terrified, excited or inspired, but I'm looking forward to seeing what happens. So until next time, Dan, thank you very much.
- [18:30]
B
Same here. Thank you. Sa.