From English Literature to Cybersecurity: A Journey Through Blockchain and Security

Summary

Podcast Summary: "From English Literature to Cybersecurity: A Journey Through Blockchain and Security"

Podcast Information:

• Title: Cybersecurity Today
• Host: Jim Love
• Guest: Anton Lavalla
• Release Date: May 24, 2025
• Description: Updates on the latest cybersecurity threats to businesses, data breach disclosures, and how you can secure your firm in an increasingly risky time.

---

1. Anton Lavalla’s Unconventional Path to Cybersecurity

The episode begins with host Jim Love introducing Anton Lavalla, who intriguingly transitioned from a BA in English Literature to a prominent figure in the cybersecurity realm. Anton shares his journey, highlighting how his passion for reading and writing led him to pursue a degree in literature at York. Facing uncertain career prospects post-graduation, Anton landed a role in a startup with a SaaS product akin to "Netflix for Android apps."

Notable Quote:

“I wrote a web crawler that would get me all of the developer names from the Google Play Store... I went from spending all day clicking through to running my bot and actually won first place for most developers contracted in a month.”
[00:44] Anton Lavalla

This automation success ignited his love for programming and set him firmly on the path toward cybersecurity.

---

2. Exploring Blockchain and Its Security Implications

As the conversation delves into blockchain, Anton asserts its continued relevance despite challenges in identifying robust use cases. He emphasizes the success of stablecoins as a practical application, particularly in regions with unreliable financial systems like Argentina and Venezuela.

Notable Quote:

“Stablecoins are much easier to send around as well. It can take a few days, up to a week. And it is expensive.”
[04:22] Anton Lavalla

Jim Love raises concerns about the security of blockchain, noting the prevalence of hacks despite blockchain's inherent security features. Anton clarifies that while blockchain's cryptography is solid, vulnerabilities often arise from poor private key management and traditional security flaws inherited from older systems.

Notable Quote:

“Blockchain is inherently secure. The cryptography... has not been broken yet. But the problem arises from how people manage their private keys.”
[05:28] Anton Lavalla

---

3. Real-World Blockchain Attacks and Insider Threats

Anton recounts notable cyberattacks in the blockchain space, including the $600 million Ronin hack. He explains how attackers used social engineering to install malware via a deceitful PDF, leading to massive security breaches.

Notable Quote:

“Threat actors were able to convince somebody... that when they opened the PDF, it installed malware that exfiltrated keys used for signing off on protocol actions.”
[06:54] Anton Lavalla

They discuss the challenges posed by insider threats and the increasing sophistication of attacks, especially with the advent of deepfake technologies that make verifying identities more difficult.

---

4. Strengthening Security Through Operational Best Practices

Anton advocates for stringent access controls and layered security measures to mitigate insider threats. He suggests implementing multi-person approvals for accessing sensitive data and limiting data access rates to reduce potential damage from compromised insiders.

Notable Quote:

“Technical controls that depend on the sensitivity of the personal information require increasingly more individuals... And layering things like how much data and how quickly you can access.”
[15:19] Anton Lavalla

Jim draws parallels to physical security, emphasizing the importance of making attackers' tasks more difficult to deter breaches.

---

5. Addressing Supply Chain Vulnerabilities and Compiler Attacks

A significant portion of the discussion focuses on supply chain attacks, such as the XE Backdoor incident. Anton elaborates on how compromised libraries and altered compilers can introduce malicious behavior into software, exemplified by the Xcode Ghost attack in China, which injected backdoors into iOS applications.

Notable Quote:

“Xcode Ghost was this fascinating attack where... the compiler was modified so that when you build an application, it injects a backdoor.”
[21:50] Anton Lavalla

He underscores the importance of full source bootstrapping and deterministic builds as defenses against such sophisticated attacks.

---

6. Promoting Deterministic and Reproducible Builds for Enhanced Security

Anton explains the concepts of full source bootstrapping and deterministic builds, detailing how these practices ensure that binaries are built transparently and reproducibly, mitigating the risk of hidden vulnerabilities or backdoors.

Notable Quote:

“Deterministic builds... give you an easy way to check the integrity of your software. If the binary matches what was built locally, you can trust its integrity.”
[25:34] Anton Lavalla

He shares insights into the development of StageX, a Linux distribution designed to enforce these security measures, which is being adopted by companies like Talos Linux for its robust security framework.

---

7. Enhancing Open Source Security and Crowdsourced Code Reviews

The conversation shifts to the challenges of ensuring secure open source software. Anton proposes a crowdsourced protocol, "Sigrev," which involves cryptographically signed reviews of specific software versions to build a trusted repository of secure code.

Notable Quote:

“Let's make it easy to create reviews for specific versions of software and then cryptographically sign them using PGP.”
[34:00] Anton Lavalla

This initiative aims to combat the proliferation of malware in package ecosystems by fostering a community-driven verification process.

---

8. Leveraging AI for Code Security and Documentation

Jim Love raises the potential of AI in enhancing code security, particularly in automating code reviews and documentation. Anton acknowledges the infancy of AI tools in this domain but supports their integration as additional security layers alongside traditional methods.

Notable Quote:

“Why not have an additional input or layer that you can put into your automation... AI analysis tools.”
[39:01] Anton Lavalla

He envisions AI as a complementary tool that enhances existing security measures without replacing human oversight.

---

9. Case Study: Milk Sad Disclosure and Cryptographic Failures

Anton shares a critical case where a misimplementation of cryptographic randomness in a Bitcoin wallet generation process led to significant financial losses. The team discovered that using the Mersenne Twister RNG, unsuitable for cryptographic purposes, reduced key entropy to a vulnerable 32 bits, allowing wallets to be compromised easily.

Notable Quote:

“They used the Mersenne Twister, which is not a cryptographically secure random number generator... reduced the key size to only 32 bits.”
[41:02] Anton Lavalla

This disclosure underscores the importance of proper cryptographic practices and thorough code reviews in safeguarding digital assets.

---

10. Building Distrust: Anton Lavalla’s Vision for Holistic Security

In the concluding segment, Anton discusses his role at Distrust, a firm specializing in high-security consulting for clients like electrical grid operators and hedge funds. He highlights their approach to eliminating single points of failure and leveraging technologies like Trusted Platform Modules (TPMs) for remote attestation.

Notable Quote:

“We think about how do we defend from state-funded actors... eliminating categories of attacks to reduce reliance on single individuals or systems.”
[46:26] Anton Lavalla

Anton emphasizes the importance of open-source tools and community collaboration in advancing cybersecurity measures, advocating for secure, transparent, and reproducible software development practices.

---

11. Closing Thoughts: Purpose-Driven Security

The episode wraps up with Anton sharing his philosophy on impactful work, encouraging listeners to align their skills with societal needs to enhance security, privacy, and freedom through open-source contributions.

Notable Quote:

“Find your sweet spot where your skills meet what the world needs, what you can get paid for, and what you love.”
[53:37] Anton Lavalla

Jim Love echoes this sentiment, highlighting the importance of passion and purpose in the field of cybersecurity.

---

Conclusion:

This episode of Cybersecurity Today provides a deep dive into the intricate relationship between blockchain technology and security, illuminated by Anton Lavalla’s unique journey and expertise. From addressing supply chain vulnerabilities to advocating for deterministic builds and collaborative open-source security measures, the discussion offers valuable insights for professionals seeking to bolster their cybersecurity frameworks in an ever-evolving threat landscape.

---

For more information on Anton Lavalla and his work with Distrust, visit Distrust's Website.

Transcript

[Jim Love]

Welcome to Cybersecurity. Today on the weekend. This is another of those shows where I have a fascinating discussion with someone doing some interesting stuff in the cyber security arena. Advanced Warning. We wander around on a lot of topics, but I think you'll find it interesting. My guest is Anton Lavalla from a firm called Distrust. I'm happy to meet you. You've got had a pretty incredible career. First question I wanted to ask you was, how does a guy with a BA in English Literature from York become a security guy? Except that I have. My undergraduate is in literature from York.

[Anton Lavalla]

No way. Okay, there you go.

[Jim Love]

Just 20 years before you.

[Anton Lavalla]

Yeah. So you would understand better than most. Yeah. How you get from a place like that to being in tech. But yeah, it's. I wasn't really sure what to study. I always liked reading and writing, so that's how I ended up doing my Bachelor of Arts or English Literature. And then when I finished my degree, it was time to get a job. And it was, what am I going to do with this degree? Either become a teacher or an author, maybe pivot into journalism. But I got lucky and my friend got me a job at a startup that had a SaaS product, if you imagine Netflix, but for Android apps that you would usually pay for. So you pay like a few dollars a month and then you get access to a huge catalog of paid apps that have an overall value of, I don't know, $10,000 or something. So I got my first job there and my first job out of university and I had to do business sales and my job was to go on the Google Play Store and look for developers that hadn't been contracted yet to the platform and then check our internal CRM. And then if we didn't have those developers in there, I had to go and reach out to them and try to sell them. On joining the platform, I had already had a little bit of programming experience, but a lot of my work seemed very like something you could automate. So I started practicing my scripting chops with Ruby and I actually wrote a web crawler that would get me all of the developer names from the Google Play Store. And then I had a script that would check in our CRM who's not in there, and it would spit out a list of people to contact and then even further automated that so that would generate my emails based on a template. And I basically went from spending all day clicking through Google Play Store to running my bot and actually very quickly won first place for most developers contracted in a month. And that was my foray into real programming where I really fell in love with the idea of automation. And it just showed me the power of programming and that kind of set me on that path.

[Jim Love]

Now, somewhere between there and here, you got really interested in Bitcoin or at least blockchain. And I'm. I want to. Want to separate the two because I. The first question I have is blockchain. Is it still around?

[Anton Lavalla]

Yeah, actually, it's definitely around. And there's been a little bit of a struggle with finding good use cases, but a subset of these projects have been pretty successful in finding real life use cases that are quite useful. There's a lot of opportunity seeking individuals who are just creating vaporware and not really contributing anything that is worthwhile. But there are projects that are actually doing useful things. So one example I always give is stablecoins, which for those who aren't familiar with is a digital asset that is pegged to the value of some currency. This is useful because not necessarily in places where we have reliable financial systems like in Canada or the United States, but for example in Argentina where there have been events where the banks did predatory conversion of people saving. So people had a lot of US Dollars saved up and the bank overnight without permission, converted that at a very bad rate and not in favor of the user to their local currency and wiped out a huge amount of people's savings overnight. And so for those people having a means to bank themselves and have their stablecoins sitting in a digital asset wallet that they control is a really nice thing because they can no longer trust your banks. And you can extrapolate that to other jurisdictions. If you look at Venezuela, very unstable financially. And so this is actually a very cool use case that's come out. And stablecoins are much easier to send around as well. It can take a few days, up to a week. And it is expensive.

[Jim Love]

There's a lot of clearances. I could not believe the hoops I had to go through because of anti money laundering. Like you said, anything over $1,000, you were just tied up for days.

[Anton Lavalla]

Precisely. I always talk about Bitcoin as well being a great alternative financial system that's been growing and getting a lot more adoption in all sorts of industries, of course, in the financial sector, but that's its own kind of thing. We could spend a whole podcast talking about.

[Jim Love]

We could. But the essence of this, because this is a security podcast, the reason I was tracking back this, I'm always surprised at how much hacking goes on in bitcoin and crypto. Why? Because I thought blockchain was inherently secure. Remember, we were sold that bill of goods. Why is it so pop? Obviously it's popular for the reason that people say you rob banks because that's where the money is. You rob crypto because that's where the money is. But why is it so prevalent in attacks?

[Anton Lavalla]

So it's inherently secure. The cryptography of blockchains is solid. It uses the same cryptography used for TLS and elsewhere and has not been broken yet. So it's designed well. But the problem arises from how people manage their private keys, their cryptographic keys that are actually the thing that enable us to move funds around and issue commands to blockchains. This is where things fail. Because there is this impression, even for people that are in blockchains, that it's this isolated island that is protected from the rest of the world. But in fact, all of the security flaws of traditional systems that came before that, their security footprint is inherited by blockchains. Because at the end of the day, if you're interacting with a blockchain, you're using some sort of a computer with an operating system that you might be using for a number of things. Maybe you're opening PDFs from an email, which is actually very common vector that's resulted in huge hacks in this space one time. This is a crazy story where one of the biggest hacks in Recent history was $600 million with this project called Ronin occurred because threat actors were able to convince somebody, a developer that was part of this project, that they were trying to hire them for a very lucrative position. And it was a multi stage process where they had a screening and then they would pass them off to another person. And eventually they sent them a job offer to sign, which was a PDF and they received this PDF on their work computer and opened it and basically installed malware on their machine. And they were able to use this malware to exfiltrate keys which were used for signing off on certain things happening in the protocol. And they were able to actually, in that way, steal $600 million.

[Jim Love]

This is so funny because we always make these fancy terms for this, but people are talking about linkless phishing now, and what you really described is linkless phishing. Oh, then I ship you the PDF or whatever it is, and I wouldn't open a PDF that I got on the first meeting with anybody, even if I sent it to myself.

[Anton Lavalla]

Exactly. But most people are very naive about this. Back in the day, washing hands before doing things, even for doctors, was like A critical, crazy idea. And it took us about 100 years to normalize that. And I think it's going to take us a while to normalize the idea of isolation. So when I work with clients in my work and personally, I actually use an operating system called Cubes os, which you may have come across. If not, it's just. What was Cubes os?

[Jim Love]

Cube OS with Q U E B E. Yeah, I know the name, I don't know anything about it.

[Anton Lavalla]

So it's a really amazing tool and very simple. Basically they took a hypervisor, the XCN hypervisor, and wrapped it up into an operating system. And so what it does is it just makes it really easy to spin up virtual machines that are for different purposes. And so when you're opening a PDF you don't trust you could just spin up a new virtual machine really quickly that's disposable, open your PDF in it and then just trash it. And so it creates this really simple way based on templates. So you have templates, let's say maybe you have a Fedora template or a Debian template. You can create a cube and they call them cubes, but they're basically just virtual machines. So you can create purpose specific VMs. So all of a sudden it's not such a huge burden to create a separate VM for accessing your AWS account or managing an API token that's for production. So you can create these little VMs and it just makes managing them a lot easier. You could do it by other means. Of course there are other tools for virtualization, but that just wraps everything up into a nice seamless workstation.

[Jim Love]

Let's go back to our blockchain example because I did want to, I did want to go through that and talk about that. So you spent a lot of time working in this area over the past few years?

[Anton Lavalla]

Yeah, I've been in the industry for about 10 years now. And the way I got into it was really funny because one of my early jobs, after I described to you how I discovered programming. But then I spent a lot of time studying and going to meetups. I met some mentors in Toronto that really helped me. Early on I had this friend Josh, who's from a funny named place and I think it's Manitoba, Flint Flawn and I would go and fly is not funny. To me it is. I'm sure it's a lovely place. But I find the name, yeah, just funny.

[Jim Love]

You got bigger. Saskatchewan, Flint fl just Flintflon's dull. That's yeah, you've got Tisdale, the land of rape and honey in Saskatchewan. You've got.

[Anton Lavalla]

Oh my God, there's all kinds of.

[Jim Love]

Funny names in the Prairies.

[Anton Lavalla]

Yeah. So I just watch him program over his shoulder, and that's how I learned. But I landed my first proper programmer job at a company called Wild Apricot. And it just so happened that the owner of that company and the CEO was Vitalik Buterin's father. So the inventor of Ethereum, I work for his dad. And so very early on, when Ethereum was becoming a thing, Vitalik would come around and everyone would be talking about Ethereum. And I was like, oh, cool, that's interesting. I knew about Bitcoin already, but I never. I didn't dive into Bitcoin properly. So my actual, like, full entry into blockchains was through Ethereum, because eventually I started learning how to write smart contracts on Ethereum, joined a hackathon and actually participated in that hackathon with somebody who now runs a big layer two, which is basically just throughput, a way to increase the throughput of a mainnet blockchain. I fell into it and then I really fell in love with what blockchains were about, which was this really interesting cross section of game theory and, and economics and of course, cryptography and distributed systems. And as a young programmer, it really grabbed my attention and that was something that also really pushed me into security. Because all of a sudden your private key is actually money, Right. It controls money or something that's of value. And that kind of raised the stakes and made me go interesting. Yeah, this changes the game in a lot of ways. And eventually North Korea also realized this, and now a major portion of their GDP is crypto theft. And then unfortunately, they use that to fund, amongst other things, their nuclear program.

[Jim Love]

Yeah, that and working for American companies.

[Anton Lavalla]

Yes.

[Jim Love]

Another area that I find just spectacularly amazing, again, where you have all of these things that make hiring someone secure, but you can still hire somebody from North Korea. I find that astonishing. I have, and I've been doing this for a long time back in 2003, had somebody working for me and they phoned me and said, I've been working for you for six months. Don't you think we should actually meet? And I went, yeah, we could do that. But I ran a worldwide consulting practice right after 9 11, and we weren't allowed to travel. So I got very used to doing phones. Primitive sort of conferencing stuff. At that time, it didn't make a lot of sense to Fight traffic in Toronto. So even then I still talked to my staff enough that I think I would have spotted whether they were North Korean or not. Just amazing. I'd love to know how they pull this off.

[Anton Lavalla]

Well, I think now more than ever it's becoming a lot more dangerous because of deepfake technologies. And that's something that's been haunting me for the last few years because even three, four or five years ago, I realized that this would eventually become a thing. Once you can in real time do video and audio deepfakes, a lot of what we use currently or used to use to verify who is who just is out the window. And North Korea seems to have started doing that a little bit. They're still mostly just doing old school social engineering, but I'm sure soon enough there's going to be a lot more deepfake stuff going on and that will make it much harder. But yeah, within the kind of blockchain and digital asset space, I've definitely seen an increase in this type of thing where North Koreans are trying to become an insider. And a lot of my research and interest in what I work on with clients will be around how do we protect from insider threats. And that's a very tricky game.

[Jim Love]

Coinbase just got nailed by people just simply doing the old. We spent all this time talking about the technology of hacking and all of that when just do the old fashioned bribe some guy? Exactly. Basically that whole hack seems to have been they just bribed a few people and they were in positions where they had trust, so they managed to gain all of the access to the blockchain. And by the way, I think they were being paid in stablecoin that even hackers don't want to be speculators.

[Anton Lavalla]

Yeah, it's interesting. I find that there's a lot of regulatory requirements around collecting kyc, but not nearly as much about protecting that KYC or it's not as stringent. If it was up to me, you would have to have technical controls that dependent on the sensitivity of the personal information or data in general, requires increasingly more individuals and maybe across different teams. So you could imagine if you need to access someone's passport, it's not just you're a support person, you can just go in and extract it. You would need to have maybe you request it and then two other people have to, with the correct role, approve that. And on top of that, of course, layering things like how much data and how quickly you can access. So if you're trying to access more than one record a Minute you're just not going to be able to do it and just layer on those types of controls. And I think that would really drastically reduce the blast radius of what would happen in the case that someone gets bribed. Of course it's not going to stop every attack, but it'll definitely reduce the surface area drastically. But companies just don't do that because it's not commonplace. No one's doing it. So why would we do it? We'll see. Maybe Coinbase does something like this.

[Jim Love]

And yet if you had a file cabinet in your office and somebody had to go and look up the personnel files, they'd have to come to you for the key. And the first question you might ask is, why do you need that file?

[Anton Lavalla]

Sure. Yeah. And good luck hauling 100,000 files out of the cabinet.

[Jim Love]

I get that part, but I'm just saying that we don't seem to transport the things that we would find natural to an automated security environment we would like. This whole idea of need to know is something we always had with physical files. Always. And I appreciate that. You're never going to be perfect. My wife always says, when we leave the house, lock the door. We live in the country. Somebody could just smash the window and get in. It's not that hard a deal. They're going to meet a dog they don't want to meet. But then I look at it and say, she's right. You make it a little more difficult. Sure.

[Anton Lavalla]

Making making your attacker's life more difficult. It's never going to be impenetrable, but you can always make it hard enough that they give up.

[Jim Love]

Yeah, yeah. So you stayed with it, with this, with blockchain. What are some of the other things about blockchain that people, you stand back and go, security people should really be thinking about this. What are some of the other ideas that you've seen over the time you spent with this?

[Anton Lavalla]

I've spent quite a bit of time working on vaulting systems, specific or custody systems that are meant to protect private keys that manage cryptocurrency or digital assets. And this is where you really need to go as deep as you can down the security rabbit hole because you are defending from state funded actors and all attacks are basically on the table when someone's trying to steal a few billion dollars worth of crypto, which are cases that I've dealt with and I've, with my colleagues worked on some of the world's leading custodial platforms like bitgo and unit 410 and turnkey and so in these systems you really need to go and explore every possible vector. And so one for example that came up is a very interesting vector is compiler level or more on the supply chain side of trying to. We saw the XE backdoor attempt. I'm sure there are other attempts that are in progress or maybe some that already successfully were deployed that we're not even aware of. And so how do we close off some of those doors?

[Jim Love]

We have a show rule before you go on. You can't just say the XE thing and assume everybody knows it. Playing with that is because we have people who may have lives do other things.

[Anton Lavalla]

So XE Backdoor was this really scary initiative that was a super long campaign spanning about three years where some somebody were unclear who it is, but likely a state funded actor or very sophisticated attacker at the least won the trust of a maintainer of very widely used library that's a compression library and they use this as a means to try to compromise the SSH daemon that's called, that's used to connect to servers remotely in a secure manner. And they were super close to actually successfully getting their backdoor in. And the only reason it was caught was because I believe somebody on Microsoft was doing benchmarking using a tool called valgrind to figure out the speed of SSH connections and noticed that there's maybe a 10 millisecond difference. It was a really small difference, but they noticed and that's the reason we weren't. The fallout would have been crazy. A backdoor where you can just go around all the defense mechanism and connect directly to most servers around the world if they updated to that version would have been horrendous. And so I raised this as an example of a type of attack and there seems to be a trend towards more supply chain attacks because we're not really doing enough on it.

[Jim Love]

This example, this is such a great example and that's. I'm glad you went through the whole story because it's one of those things we had. I had a whistleblower on the program about two weeks ago and the amount of pressure that he got to stop investigating things because he'd find an anomaly and he'd raise it and he'd have to go, I better really figure this out because I can't go talking about this with my boss. And I went, oh my God, this guy at Microsoft sitting there God knows when, but let's say it's 10 o' clock at night, he probably should be home. He sees this little blip and he goes, I got to find out what that is. And God bless them, whoever. Absolutely his management was that he felt that was a good thing to do. I don't know how much damage he saved the world from. And the curiosity of security people is not something we talk a lot about and it's so essential.

[Anton Lavalla]

Absolutely. Caring about security always stays half the game. People are like, so what do we need to do to be secure? First job number one, care about security, Care enough about security and be curious like you said, otherwise you're not going to make it.

[Jim Love]

So we were talking about blockchain and just some of the things that you discovered or seen there that really got you.

[Anton Lavalla]

So I started talking about some methods that we saw work underutilized and that you need to think about because of the sophisticated supply chain attacks. So the two things that I've been spending a lot of time and energy on is full source bootstrapping, which I can talk a little bit about at length, but essentially just building from pure source code and not hiding anything inside of binary. So you don't want binaries when you're building things because they're hard to inspect, they're opaque, and so we don't really know what's inside of them. So when we build, for example, even a compiler, ideally that's a really good thing to full source bootstrap, because we want to make sure that our compiler can't go and introduce vulnerabilities. I'll talk about another really exciting attack now that happened, but I only discovered it recently. I was surprised that I hadn't known about it, but it's called Xcode Ghost and it was this fascinating attack where in China developers were having a hard time connecting to Apple servers. The problem is this version of Xcode was fully functioning, except the compiler was modified so that when you build an application with it for iOS, it actually injects a backdoor. And they successfully compromised a number of different applications and downstream it impacted close to a million devices and it was basically exfiltrating data from these devices. And so this is a cool example. It's a scary example of how a compiler can be the source of compromise.

[Jim Love]

If you don't know, was it intentional or accidental?

[Anton Lavalla]

It was intentional. It was definitely intentional. And relied on the desperation of the developers that were trying to get their tooling and they couldn't because of the great firewall of China. And so they basically used this to their advantage and got people to download from a source they shouldn't have, so. Exactly. And so this is Something that Ken Thompson, who was super prescient back at it was in the 80s at MIT, realized even if your source code is fully reviewed and trusted, the compiler itself could actually inject malicious behavior into your binary. And so this is something that we thought about a lot and realized that when you're building systems that manage billions, you need to actually make sure that all of the software ideally is full source bootstrapped. The other method that we came upon is deterministic or reproducible builds. Right? So the idea that your software should always compile to the exact bit for bit binary. And when you do this, it gives you this really easy way to check the integrity of your software. I can talk about another attack right now that most viewers probably know about, or listeners, maybe not the Solar Winds incident that happened some years ago.

[Jim Love]

We're all familiar with Solar Wind.

[Anton Lavalla]

That attack could have been avoided with the help of reproducible builds. And SolarWinds actually put out a paper after some time saying as much as. But the idea basically, or the failure of SolarWinds was that once the binary was built, because it wasn't deterministic. And so you build it now and then you build it the same version five minutes later. The binaries would be slightly different, not because of being functionally different, but because there might be a timestamp or some artifact from the chipset that the compiling is being done on, or an environment variable that's slightly different. But essentially, once the binary is out of the built pipeline, there's no way to check if that binary is what it's supposed to be be with reproducible builds, because we're forcing the binary to be identical. Now, what that enables us to do is even if the binary is already out of the build pipeline and maybe it's uploaded to a page where you can download it, you, for example, as a developer, could go and build it on your machine and then compare that binary to whatever came out of the build pipeline. And now, even if the build pipeline was compromised, the likelihood that your developer computer and the build pipeline was compromised is much lower. And of course you can extend this and say, okay, we're a really big tech IT company and our tools are used by Fortune 500 and government organizations. So what we're going to do when we build our software is we're going to have three different build servers on three different cloud platforms with separate access permissions for this infrastructure, and we're going to build the same software and then compare it and make sure that all the binaries are exactly the same. Now, the likelihood that all those three systems are compromised in the same way is very low. And so this is a really cool method that isn't really being used as widely. And yeah, we spent some time working on this stuff and actually me and my colleagues and friends built our own Linux distribution that makes this a fundamental part of how we build it. We start by full source bootstrapping a compiler, and then we do that deterministically, of course, as well, and then build the whole tree of all the packages that are available also with that compiler that we now know is trusted fully deterministically. So anyone can go and reproduce the whole thing and see if their hashes match. Of course, that doesn't solve the problem of trusting the source code. You still need to know what's in the source code. But it does eliminate the risk of what if the compiler injects something? Or what if the build environment has something malicious and modifies the code? And most software companies actually are exposed to that risk today.

[Jim Love]

But how do you. These differences are introduced by factors like, as you said, timestamps, the peculiarities of the chip. How do you filter that stuff out and still know that you've got two identical source or two identical binary files?

[Anton Lavalla]

How do you basically get rid of these little variations that make your binaries non deterministic? A lot of the time it's as simple as holding a gun up to the timestamp and saying you're not changing. So you can just fix the timestamp and set UTC to 1, and that will actually force all of the timestamps to always be identical. And for some software that's enough. For other software you have to play with the compiler flags, because, for example, when you're doing certain parts of the build process, it might be non deterministic and it might do things in parallel. And so because it's doing things in parallel, it might do them in different order depending on when you're running it. Sometimes you have to go and actually patch the software to change the way it's actually built to maybe it just intentionally goes and grabs some details about the chipset it's being built on and injects that. And so we've had to actually go in for Node Js, this was the case, and we had to work with the developers of Node JS to change this about how they build their software. So we can make it deterministic, but yeah, it can be time consuming for some software. For other software it's easier. Of course, you also want to do full source bootstrapping for languages so self compiled languages like Rust. If you download a Rust binary from somewhere that you don't know how it was built and then use that to build the next version of Rust, the trust level on that is much lower than if you go and first bootstrap the GCC compiler and then or a tiny C compiler and then you go to the first version of Rust and then the next one and iteratively build up until you have a clear path from before Rust was self compiling all the way to the latest version of Rust. So this is another thing that we discovered as a good defense mechanism on making sure that basically it allows you to say certain categories or classes of attacks are no longer viable. You basically cut those off. And so you still have things to worry about, but at least less things.

[Jim Love]

It. Yeah, and again, it's obviously when we're talking about it in this manner and you talk about doing it the first time, it's probably labor intensive, probably takes a lot to figure out. But I'm just amazed that with all of the problems we've had with open source code that people have not put out a protocol or something that says, hey, this is how you establish that what you've got is a binary compatible version of the original and has not been interfered with. I find this astonishing that no one's really pursued that.

[Anton Lavalla]

There are a few distributions that are working on this. So the one that we built, it's called StageX and there are two other ones that have similar approaches where they do boots, full source bootstrapping and determinism. They're called Nix and Geeks. And Geeks is actually a fork of Nixon with slightly different approaches, but none of the existing distributions were as strict as we wanted to be on this idea of full source bootstrapping determinism. So the one we built, we basically made a rule and said only if something is full source bootstrapped and deterministic, and any changes that are made during packaging, the packaging steps of actually describing how the package is created in our distribution has to be reviewed and signed by two individuals. And so we basically took these things and tried to do something akin to what you were just mentioning. That was the idea how do we make something that's fully open source and free and helps people get determinism and full source bootstrapping into their systems as kind of a building foundation? Of course you need to force your application to also be deterministic to fully close off the loop on that. But that's exactly the Idea how do we get this into more people's hands? And now it's actually fairly easy to use. It's almost an Alpine replacement, like a drop in replacement. So if you're using Alpine, Python or Rust, you could just pull in an image. It's on docker hub of StageX and it's the same software, same version, except it's built differently. So it's built using a full source bootstrap compiler. All the languages are full source bootstrap and it's fully deterministic. So it's pretty cool.

[Jim Love]

And this is available now? You've made this open source, freely available?

[Anton Lavalla]

Yeah, it's open source and we did this because of lot of our clients cared about this kind of attack vector. So it's a cool feedback loop where blockchain companies spent money to do a lot of this work and then we open sourced the work and made it available to everyone. And it's really nice because it's already being used. Actually if you're familiar with Talos Linux, they're very widely used Linux distribution that specializes in kubernetes. They decided to use our distribution to build theirs. Ours is essentially a secure tool chain for building software and they thought, hey, this would be a good way to build our software because we close off a bunch of attack vectors on this compiler level and environment, kind of risk level.

[Jim Love]

I'll be able to say, I knew you win, boy. Can you send me a link and I'll make sure I post that in the show notes at least for the YouTube version.

[Anton Lavalla]

That'd be very helpful because yeah, we spend a lot of and continue to spend a lot of time and energy, essentially all of our free time where when we're not working on projects that make us money, we go and pivot to this and sometimes we're able lucky enough to actually get some of this stuff funded too. But it's a labor of passion and love and we love to see people use it because we believe it can help basically anyone. That's the beautiful thing. Oh, you're using Python in your stack. Great. Just drop in to our Python image or if you're using Rust or Node Js or whatever.

[Jim Love]

Yeah, no, this would be handy. The other thing that just drives me crazy when we talk about blockchain was one of the first use cases for blockchain that I ever heard that made business sense other than crypto was the idea of traceability. And in the food system, like a really messy system full of people who are not necessarily PhDs through this and I'm not making fun of them, I'm just saying this is not an organized system yet. I can trace a piece of lettuce that is here in Minden, Ontario, which is in the middle of nowhere, back to the original field where it was. Because if they get an outbreak of some sort of E. Coli or something like that, they need to be able to pull it off the shelves. And yet with all of the brain power we have in blockchain, we've never been able to establish the providence of. Of software and software elements or modules in the same way.

[Anton Lavalla]

That's a good question and a very difficult problem that people have been trying to solve for a long time. And especially because blockchains fit the shape, it's. Yeah, this should be the solution. I think a part of the problem is that blockchains are very good at preserving data once it's on the chain, but it's very hard to get high quality data into it. And if you think about the blockchain trying to fit into the operational complexity of a supply chain that spans maybe multiple countries and different points where the data has to be entered, I think that's where things start to break down. How do you make sure that reliable stuff gets in the blockchain? Because if you get poo on a blockchain, it'll just be poo on a blockchain and it's not going to be great. So I think that's the operational and logistical parts of how these supply chains work is the hard part. Once you have good data, you can throw it basically on anything. In a sense, the blockchain is just a distributed data layer. It's not really anything crazy. It's not anything crazy. It's actually pretty crazy.

[Jim Love]

Yeah, yeah. It's not something you do in an afternoon, but it is. But this. Have you given any thought to this? And I'm just asked this while we're just talking as to. You've obviously conquered one piece of this and say I can establish the providence of a binary, I can make sure that it's the same. Could we apply some logic to that in the open source world so that we're just making sure we're actually getting the correct modules and that we're not getting somebody's yes.

[Anton Lavalla]

Yeah. The answer is yes. And we've been thinking about this a lot. This idea of how do we eliminate single points of trust in any individual or computer. I think that's like a core idea and, and yeah. Concept to filter ideas through. When trying to think about Holistic security that thinks about the full lifecycle of software that we consume. And so we talked about, yeah, like you said, this part of compilers and determinism and what that can address. But still a lot of holes remain, even on the level of, if you look at compression algorithms and tarball. So with the XZ attack, what was actually happening was that there were additional artifacts in the tarball that were not in the source code. And so it was hiding in the tarball. And this is not something you would think about. You think, oh, source code goes into the tarball, which is just the source code, and then you on tar it and you do stuff with it. But how do you make sure that's actually the case? So that's one step before what we were talking about, where it's, how do we verify that we're actually building from the source code we're supposed to be building from without any extra stuff in there. So here we are trying to build some tooling as well. There's a project that I found that is. I forgot what it's called. It's like Predictable binaries. It's by Fosse, I would have to look it up. But it's essentially a tool that detects binaries and source code. And so that's one idea. You can try to find binaries, eliminate binaries, because those could sneak things in. But then also can we create a way to standardize the source code and make sure that anyone can easily check that the source code is exactly the source code you're meant to be building from. But we can go even earlier than that where we need to review the source code to make sure that there isn't. If someone writes some malicious code into the source code and no one's looking at it, that there's no way to really address it. So we need reviews. So one way that we've been thinking about that in my group of friends and colleagues is how about we create a crowdsourced kind of protocol. We're calling it Sigrev or yeah, sigrev, which is like signed reviews. So the idea is let's make it easy to create reviews for specific versions of software and then cryptographically sign them using pgp. Or you could use, you could sign it whatever you like, but PGP is probably a good case. And then start building a database of software that's been reviewed and you can go and let's say you're an independent researcher. If you reviewed some piece of software, you could say exactly which commit or which state you reviewed have a hash of the tree and then say I reviewed for this and that and then sign it off. Then this is another thing that I feel is a huge hole right now. If you look at the NPM and Python package ecosystems, they're full of literal malware. And for whatever reason, we got to a place where we just feel okay with downloading tons of third party libraries without actually reading the code. And we just say we use static analysis so it'll be fine. But it's not fine because static analysis only catches things that we've previously detected. And so novel stuff we don't have signatures for. Similar to how malware is, right? Like polymorphic malware just slightly changes the shape of the malware and all of a sudden we're blind to it. Right? We're trying to develop tools to fight this. But why aren't we reviewing all the source code? And there are a few answers for this. Often 80% of an application is third party code. That's just open source libraries. How are we going to review all this? It's hundreds of thousands or millions of lines. But the responsible thing to do, especially if you're running a business that is maybe financial institution or manages a ton of personal information, would be to literally review every line of code, right? And you actually say, okay, what if we all collectively as we reviewed software, posted our results and said, yeah, we reviewed this stuff and we actually found it to be secure for this specific version. Now you might use a different version and say, this hasn't been reviewed. So I have to review the difference between those two. But over time, if we were all doing this sort of thing, we would start building up a repository of knowledge of which source code has at least a slightly higher level of trust. Because right now it's just a lot of people use it. Probably somebody looked at it, but we don't really know.

[Jim Love]

I have to. They come and take my podcaster license away if I don't ask about AI at some point in the conversation. It's just the way it is. Forgive me on this one, but why hasn't anybody started to look at using AI to do some of these tasks? Especially when you talk about multi lines of code. One of the things we can have a great debate about whether AI can write code or not. And we can. That's a religion now as to what you believe rather than anything else. But the one thing I think we all agree on is it's really good at documenting code. And I've been very surprised to not be reading A lot about people who are saying, we got this problem, I'm going to apply AI to it. Have you heard of anybody doing this?

[Anton Lavalla]

I haven't looked into it too much, but I am very much in favor of it. I think it still probably is in its infancy. I wouldn't trust a tool like that as, like, oh, we have this now. We can. Hands off. But the way I think about it is why not have an additional input or layer that you can put into your automation. The whole shift left to use that term. You have your linter, you have your static analysis tools and now you have your AI analysis tools. And as people figure it out and they become more sophisticated and advanced, I think, yeah, they could be very effective. I don't know when we get to a point where we can fully trust these tools to make judgments.

[Jim Love]

Why would you have to fully trust it? It's the old thing of that's why we build in layers, because we don't fully trust anything.

[Anton Lavalla]

Exactly.

[Jim Love]

Friend David Shipley would tell me that organizations that believe they're totally protected are the most likely to get hacked. You need. Andy Grove said, only the paranoid survive. I think that's true in security, but having another layer. We're already using AI to look for anomalous things in software that says, hey, we don't have a signature for this, but this looks weird.

[Anton Lavalla]

Totally. Yeah, it doesn't hurt. When I first started thinking about it, I was kind of like, wait, no, but we don't need to rely on AI. It's literally just another set of inputs. So great, I'll take more inputs.

[Jim Love]

Anyway, we rely on people and people make mistakes. I always find this just totally amazing, is that everybody says, yeah, but AI made a mistake. And you go, yeah, and so did a person. And what's your point? Yeah, it's. Nothing's perfect. That's why we have layers. That was at least my understanding of it.

[Anton Lavalla]

Yeah.

[Jim Love]

I have to ask you just in a. Just going on to this thing. I was looking through your LinkedIn and I was reading about this Milk. Sad disclosure.

[Anton Lavalla]

Oh, yeah.

[Jim Love]

And I know, I know I'm drifting into a new topic, but this was so fascinating. We've talked about some of the human things we can do, but this really was a mechanical or a technical error that just went by everybody.

[Anton Lavalla]

Yeah, it was very. A very interesting bug that actually impacted one of my friends and they lost a bunch of Bitcoin. And the really scary part is that they went and followed a very widely used and popular book called Mastering Bitcoin. And this book recommended using a specific software to generate your Bitcoin wallet. And so it recommended that you do it in an offline environment on an air gapped system that's not Internet connected. And so they followed these instructions and did all this. And then years later their wallet got drained. We assembled a research group friends who started looking into how this happened. Eventually we discovered it was a mistake in how they implemented the cryptography around randomness or entropy. And they used what's called the Mersenne Twister, which is a random number generator algorithm. But it's not a cryptographically secure random number generator. So it's actually usually used for Monte Carlo simulations. It's not meant to be used for generating private keys. By doing this, they reduced the key size to only 32 bits. With a well optimized cracking algorithm, you can map out fully in about a day. So this is exactly what we did and we were able to reproduce, basically build the whole, all of the keys that were part of that key space and so on. Milksad.info There are a series of blogs that one of our teammates Christian wrote about. And so it goes into the details, really low level of how it all worked and how we actually reproduced the vulnerability. And we had a responsible disclosure and back and forth with the developers. But it was a very interesting. We actually presented it at the Chaos Communication Congress last year in Germany and Hamburg. So there's also a talk recorded about it. So if you prefer video format you.

[Jim Love]

Can watch that because I found this just fascinating. Again it was, it's for all the technology that we have, it's really nobody being curious enough to go that's only 32 bits, man. Gaming machine will get through that very tricky bug.

[Anton Lavalla]

Because you need to know cryptography relatively well to figure this one out. And also I've seen a lot of books that teach computer science recommend using the current time as the seed for randomness. And time isn't random. And so we also have this layer of just bad information floating around there. And that's partially what leads to these mistakes. And developers can overestimate their ability a little bit when it comes to security and cryptography. And cryptography is something that even I, as somebody who spent a lot of time working on cryptography, take very seriously. And I'm basically scared of like whenever I work with cryptographic algorithms, it's like dealing with nuclear codes.

[Jim Love]

Exactly. But how do you make sure that you're, how do you satisfy yourself that you've done the due diligence I actually.

[Anton Lavalla]

Go and work with, like, multiple different cryptographers.

[Jim Love]

We all go on about how security has to be built in, not bolted on. And sorry if I sound preachy on this thing, but it's just what I would have done because I was the world's lousy. Maybe you've got to be stupider as a programmer. And because every time I did something, I went, I don't really know how to handle this. I'm gonna go to somebody who really knows this. And we would do what we call chalk talk at the time, which became even weird when we were actually in a basement of somewhere like Waterloo where they actually have chalkboards. It probably still happened to this day, but. But, you know, we'd be. We'd be in somebody's. On somebody's whiteboard with markers and still call it chalk talk. But it was that whole thing of go to somebody smarter than you, have them walk you through what you're doing. They may not even be a total expert, but they'll ask dumb questions which are exactly the ones that always get you into trouble.

[Anton Lavalla]

Yeah, I'll go talk to my friends who have expertise in areas that haven't been or delved into as deeply, or I'll go on IRC and find those really arcane knowledge people on something that I need to find out about. And so, yeah, there are people you can talk to, but, yeah, you just need to be a little bit humble.

[Jim Love]

Exactly. Because it's not the times when you don't know it even. Like I said, I. Because I was never the smartest programmer in the world, I always had to ask people, make sure I was doing things right. But it's the guys who really think they know stuff that scare me.

[Anton Lavalla]

Yeah.

[Jim Love]

Because they don't ever have to ask. And I used to work with one architect, and he was like, he just wanted to get the program to be the smallest thing in the world. Even after, like, in the early days when I started, if you wrote a program that was more than 5k, you had to do an overlay. Because we did. That's. We didn't. We only had about 3 1/2 k memory that was available.

[Anton Lavalla]

We still had such constraints. That's one of the things that really irks me. Like, we have so much processing power now, and instead of making our software super fast and efficient, it's actually some of the least efficient software ever.

[Jim Love]

Yeah, I think word has 30 million lines of code. I'm going to do a word processor.

[Anton Lavalla]

I've given up on Microsoft a long time ago.

[Jim Love]

But yeah, well I finally gave up even if so seven bucks a month. I gave up paying the tax to Microsoft to use a word processor.

[Anton Lavalla]

I'm just like living in vim now. Markdown files are my go to and it's been a while now that I've. I've been left all that.

[Jim Love]

This has been a most amazing chat. I want to talk, just close it off a little bit about you. You and I always do these things. I tell people they can't do a commercial on here, but you're working for a firm called Distrust. I always figure if people get to the end of the show that you've done a good job then and they're still listening because the audience that I've got in, particularly in cybersecurity, if you actually try to sell them anything, they'll medieval on me. They'll say, but let's talk about your company Distrust. First of all, who came up with the name that.

[Anton Lavalla]

Yes, it was my co founder and it's basically this idea of you shouldn't trust, you should be able to verify. And so therefore Distrust, it's a company that my co founder started a while back, probably five years back and then I joined and started working with him maybe three years ago. But it's basically a firm that specializes in working with high risk clients. So a lot of it came from working with custodians and vaulting solutions like I mentioned earlier. And we basically learned a lot of really great methods to protect sensitive system. So we work with electrical grid operators and we work with hedge funds and anyone who has really sensitive data or cryptographic keys they want to protect, we help them. And we think very holistically and we think about this idea that I also mentioned earlier of how do we eliminate single points of failure in a system because it's like nuclear launch, like you have two keys. But what if we say we need four keys or we play with this idea of a quorum of people that are required to do something. So maybe you have seven people that have the right permissions and you need four of those seven people for something to happen. So we methodically think of ways to at every layer, including the Linux distribution or the operating system that's built. How do we make sure there's no one maintainer, for example, that can go and insert some code somewhere into one of the dependencies and undermine the integrity of everything that comes after it. And that's basically the game we play. We have a threat model on our website. It basically talks about these different levels of how we think about it. But our approach to threat modeling is basically how do we defend from state funded actors? At the most extreme end, instead of trying to think about specific vulnerabilities like CVEs, we think about how can we eliminate entire categories of attacks and make it so that you don't rely on one person or computer for the integrity of your system. A lot of what we do is consulting and we'll be on retainers with companies like a fractional CS or fractional security engineering team. All of the rest of our time goes to building these open source tools and we have a bunch of them. We have some custom operating systems, work a lot with remote attestation technologies. So getting a server to cryptographically prove what software it's running and a prerequisite for this are deterministic builds. You need the software to be deterministic because essentially the idea is let's say you use a vpn, you don't want to just trust the VPN provider that they don't log anything like pinky promise. What if the server could actually cryptographically attest to the software it's running? So imagine that portion of their stack is open source software and it's basically the thing that you connect to. Now if it's open source, you can read and review the code and say, okay, great, this actually doesn't log anything. Amazing. You build the software locally and because it's deterministic, you expect the hash that you get locally to be the same as what the server tells you is running on that server. And so there is a special type of hardware called a tpm, a Trusted Platform module, which is basically a secure chip that can go and it has its own keys that only it has access to. There are these things called PCRs. So between those two, you can basically get the measurement of the server, figure out what the binary is that's running on the server, and then cryptographically sign it with the secure chip. And now you get a cryptographic attestation from the server, proving that it's running the code that you reviewed locally. So this is amazing building block and a level of trust that I think in the future we're going to see a lot more. But it's just a fundamental building block that is underutilized right now. And so that's another example of where we spent time building. We built an operating system called Enclave OS that's specifically to make this sort of thing easier. So basically wherever we find there are opportunities to make it easier to Access these defenses mechanisms, we create open source software and say here you go, please use it, help us build it. And everything we do basically is open source. So yeah, that's a remote data station is a really interesting topic as well. And then the Caymans I actually came here and opened a company called Caution because Benjamin Franklin once said that the parents of security are distrust and Caution will actually be creating some products that are like managed products to help. For example, one of our products will be a service where you can ask us to build your software and give you a hashback of what we built. So when you're doing deterministic or reproducible builds as a part of your cicd, you could say, hey, what's the hash of this? And so you have another separate system reproducing your software that you don't have to set up for yourself. So that's just one example of what we'll be doing.

[Jim Love]

But I'm going to tell you guys, if you want to get out of the security game, you could go to most of the AI companies and help them name their products so they didn't have stupid names.

[Anton Lavalla]

All of our stuff is named very simply. Like we have an operating system that's for offline operations. It's called Air Gap os. We have an operating system for enclaves, it's called Enclave os. We have the Linux distribution that full source bootstraps everything. It's called Stage X because use stages to bootstrap things. So this is a new stage. And so yeah, we try to use just very simple names.

[Jim Love]

This has been fantastic. Anton, it was great to meet you. And for anybody who's listening to this, if they're still here with us. We met because a friend recommended you and I urge people to do this. If somebody knows somebody who's doing interesting stuff, I love to every once in a while just do an interview that is just talking about interesting stuff that people are doing. So that's. This has been great. Before we go though, I always say that people, when I was heading up things, everybody thinks that the CEO or the head of an organization is running it now we're just selling, trust me. And I always took away one thing was that was when you left a room, what you really wanted, wanted to do when you were in the room was make people remember you when you left. In other words, the question was what are they thinking about when you leave, when you're not there? So at the end of this, what do you want people to remember from this conversation?

[Anton Lavalla]

We had to remember. I mean, I don't want to sound too generic, but that it really matters what impact your work has. And for me it's thinking about what my skills can best be used for to improve the security, privacy and freedom of as many people as I can. And the way I found to do this is through primarily open source software that creates this like building blocks that improve security. And so I would just encourage everyone to try to find their sweet spot for that, where their skills meet, what the world needs, what they can also get paid for and what they love. This is like I ripped the Japanese concept of ikigai, but I would like.

[Jim Love]

Yeah or mo Gad who says stop working for money, start working for purpose.

[Anton Lavalla]

That's it. And I but my passion of freeing these tools and open sourcing everything is definitely the trade I take any day over just making money.

[Jim Love]

This has been fascinating. My guest has been Anton Lavaya and he is with a firm called Distrust. Love to hear your comments and and if you know someone who's doing some fascinating work, or if you're one of those people, or if you've just got an idea for a show, drop me a note. Maybe we'll get a chance to chat and some of the chats actually make it onto the show. You can reach me at editorialech newsday ca or you can find me on LinkedIn or if you're watching on YouTube. You know what to do. Just leave a comment under the video. David Shipley will be sitting in on Monday and I'll be back in the News Channel share on Wednesday. I'm your host, Jim Love. Thanks for listening.