Breaking barriers, one byte at a time. [Research Saturday]

Summary

CyberWire Daily – Episode Summary: "Breaking Barriers, One Byte at a Time" [Research Saturday]
Release Date: March 29, 2025
Host: Dave Bittner
Guest: John Williams, Vulnerability Researcher at Bishop Fox

---

1. Introduction

In this episode of CyberWire Daily's "Research Saturday," host Dave Bittner engages in an in-depth discussion with John Williams from Bishop Fox. The focus is on their recent research titled "Tearing Down Sonic: Decrypting Sonic OSX Firmware," which delves into the complexities of reverse engineering encrypted firmware from SonicWall, a prominent firewall appliance vendor.

---

2. Background on SonicWall Firmware and Encryption

John Williams introduces the core subject by explaining that SonicWall, a widely used firewall solution among their customers, upgraded to Sonic OS X, implementing enhanced encryption measures on their firmware images. This transition posed significant challenges for Bishop Fox’s research team, as the new encryption restricted access to the underlying file system necessary for vulnerability assessment.

John Williams [01:52]: "The research we're discussing today is titled Tearing Down Sonic Decrypting Sonic OSX Firmware."

---

3. Reverse Engineering Process

Dave Bittner outlines the initial steps taken by the research team to overcome the encryption barriers. They began by analyzing the virtual machine (VM) image released by SonicWall, which contained the encrypted firmware and a bootloader. The assumption was that the decryption keys were embedded within the bootloader.

Dave Bittner [03:49]: "We started with a virtual machine image... the keys to decrypt the firmware must be somewhere within that bootloader."

The team successfully identified and extracted the key encrypting keys from the initram volume, which were essential for decrypting the firmware package. This meticulous process involved unpacking the bootloader, extracting the initial RAM disk, and navigating through multiple layers of encryption scripts.

---

4. Challenges Faced

The decryption process was more intricate than anticipated, likened by Dave to a real-life Capture The Flag (CTF) challenge due to the complexity and layered encryption mechanisms.

Dave Bittner [04:50]: "It was one of the most CTF-like challenges that I've ever actually seen in real life."

The team had to reverse engineer numerous bash scripts that governed the encryption and decryption processes, ultimately revealing the use of OpenSSL with AES keys—common but securely implemented encryption standards.

---

5. Development of SonicRack Tool

Upon successfully decrypting the firmware, Bishop Fox developed a tool named SonicRack. This tool automates the extraction of encryption keys and decrypts the firmware, granting access to the root file system. SonicRack streamlines future vulnerability research by enabling quick analysis of firmware updates and facilitating the identification of code changes that may introduce vulnerabilities.

Dave Bittner [08:08]: "We released a tool called SonicRack, which automates the process of extracting the keys from the virtual machine image and then decrypting the actual firmware so you can access the root file system."

---

6. Impact and Vulnerabilities Identified

The decryption efforts led to the swift identification of a critical vulnerability: an authentication bypass in SonicWall's SSL VPN. By conducting a patch diff analysis, Bishop Fox was able to develop a proof-of-concept exploit, enabling them to notify customers of their exposure and assist in mitigating the risk before malicious actors could exploit it.

Dave Bittner [10:28]: "SonicWall had announced an auth bypass affecting their SSL VPN... we were very quickly able to run a patch diff report... find the vulnerability and write an exploit for it."

This proactive approach ensures that customers are better protected against emerging threats by facilitating prompt patching and vulnerability management.

---

7. Responsible Disclosure and Communication with SonicWall

Bishop Fox adheres to responsible disclosure policies, meticulously evaluating the implications of releasing tools like SonicRack. In discussions about vulnerability research, they emphasize the importance of balancing transparency with security. Although they informed SonicWall of their intention to publish SonicRack, there was no subsequent dialogue from SonicWall.

Dave Bittner [12:18]: "We give them a heads up to let them know that we were going to be publishing this. We didn't get a response from them on it."

The release of SonicRack is intended to democratize access to firmware analysis tools, leveling the playing field between independent researchers and more resourced adversaries like nation-state actors.

---

8. Broader Implications for Firmware Security

The episode underscores a broader trend in firmware security, where vendors implement multiple layers of encryption to protect their firmware. While these measures aim to deter malicious reverse engineering, they often inadvertently hinder legitimate security research. Bishop Fox's experience with SonicWall is reflective of challenges faced across the industry.

Dave Bittner [15:38]: "It is pretty typical... when we look at other vendors in the firewall space, they will usually have some kind of basic protection around their firmware images."

The discussion highlights the tension between securing firmware against attacks and enabling security researchers to identify and mitigate vulnerabilities effectively.

---

9. Recommendations for Users

Bishop Fox advises users of SonicWall and similar firewall devices to implement best practices in network security. Key recommendations include:

• Restrict Management Interface Exposure: Ensure that the management interfaces of firewall devices are not exposed to the public internet to minimize vulnerability exposure.

  Dave Bittner [16:48]: "Make sure the management interface is not exposed... pathogens."

• Stay Informed and Patch Promptly: Keep abreast of security advisories and apply patches swiftly to protect against known vulnerabilities, especially those affecting public-facing components like SSL VPNs.

By following these guidelines, users can significantly enhance their network security posture and reduce the risk of exploitation.

---

Conclusion

The "Breaking Barriers, One Byte at a Time" episode provides valuable insights into the intricate process of decrypting and analyzing encrypted firmware. Through diligent research and the development of innovative tools like SonicRack, Bishop Fox exemplifies the critical role of security researchers in safeguarding digital infrastructure. The episode also raises important considerations about the balance between firmware security and the accessibility of security research, urging both vendors and users to collaborate in enhancing cybersecurity resilience.

---

Notable Quotes:

• John Williams [01:52]: "The research we're discussing today is titled Tearing Down Sonic Decrypting Sonic OSX Firmware."

• Dave Bittner [03:49]: "We started with a virtual machine image... the keys to decrypt the firmware must be somewhere within that bootloader."

• Dave Bittner [04:50]: "It was one of the most CTF-like challenges that I've ever actually seen in real life."

• Dave Bittner [08:08]: "We released a tool called SonicRack, which automates the process of extracting the keys from the virtual machine image and then decrypting the actual firmware so you can access the root file system."

• Dave Bittner [10:28]: "SonicWall had announced an auth bypass affecting their SSL VPN... we were very quickly able to run a patch diff report... find the vulnerability and write an exploit for it."

• Dave Bittner [12:18]: "We give them a heads up to let them know that we were going to be publishing this. We didn't get a response from them on it."

• Dave Bittner [15:38]: "It is pretty typical... when we look at other vendors in the firewall space, they will usually have some kind of basic protection around their firmware images."

• Dave Bittner [16:48]: "Make sure the management interface is not exposed... pathogens."

---

This comprehensive summary encapsulates the key discussions, insights, and conclusions from the podcast episode, providing a clear understanding for listeners and non-listeners alike.

Transcript

[Dave Bittner]

You're listening to the Cyberwire Network, powered by N2K.

[John Williams]

Is your AppSec program actually reducing risk? Developers and AppSec teams drown in critical alerts, yet 95% of fixes don't reduce real risk. Why? Traditional tools use generic prioritization and lack the ability to filter real threats from noise. High impact threats slip through and surface in production, costing 10 times more to fix. OX Security helps you focus on the 5% of issues that truly matter before they reach the cloud. Find out what risks deserve your attention in 2025. Download the Application Security Benchmark from AUX. Hello everyone and welcome to the Cyberwires Research Saturday. I'm Dave Buettner and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems, and protecting ourselves in our rapidly evolving cyberspace. Thanks for joining us.

[Dave Bittner]

Part of what we do on our team is we regularly research end day vulnerabilities in major software vendors. And a lot of our customers use firewall appliances like the Sonic Wall devices. And so we, you know, we stay on top of new vulnerabilities that are announced in those types of devices.

[John Williams]

That's John Williams, vulnerability researcher at Bishop Fox. The research we're discussing today is titled Tearing Down Sonic Decrypting Son, Sonic OSX Firmware.

[Dave Bittner]

And with Sonicwall in particular, we had done some research on them in the past. We kept up with a lot of what was going on with Sonicwall. But when they upgraded their platform to Sonic OS X, suddenly we didn't have access to the underlying file system anymore. They rolled out a new version of encryption to protect those images. And so we took it upon ourselves to do the research to try and reverse engineer that encryption so that we could continue being able to research new vulnerabilities when they were announced on that platform.

[Unknown]

Well, let's go through that together. I mean, you mentioned that they enabled encryption here on their firmware. Why would they do such a thing and how does it work?

[Dave Bittner]

Sure. Well, it's pretty common for vendors of appliances like these to use some type of encryption on their firmware images. The reasons why it can be hard to speculate, but I'm sure that the general thought is it just slows down people who are trying to gain access to your files. It slows down reverse engineers, and that much is true. But as we've seen, it rarely actually stops them. And SonicWall had used encryption on their earlier version of Sonic OS, but other researchers had succeeded in reversing that. And you know, there were decryption tools available, which that's what we had used until they released Sonic OS X, at which point we had to do our own research to figure out how to gain access again.

[Unknown]

Well, let's go through it together. What were some of the challenges that you all faced as you made your way through decrypting this firmware?

[Dave Bittner]

Sure. Well, it started off as pretty much as you would expect, if you're familiar with, you know, the way encryption often works on these. And what that means is we started with a virtual machine image, Right. Once they released a vm, it makes it much easier to actually access the static assets. And when you unpack that virtual machine, you basically end up with several volumes that get mounted into the virtual machine at boot time. And in our case for SonicWall, there were only two that mattered. One of them contained the firmware itself as just an encrypted package, and the other one contained a bootloader. And in a scenario like that, we would expect that. Well, because it's a virtual machine, the keys to decrypt the firmware must be somewhere. And if all we have is the encrypted package and bootloader, then the keys are most likely going to be somewhere within that bootloader. And that initial assumption did turn out to be correct. However, it took a lot more. It was a much more complex process finding them and retrieving them than we first expected. In fact, I like to joke that it's one of the most CTF like challenges that I've ever actually seen in real life.

[John Williams]

That's interesting.

[Unknown]

Are there details you can share with us about that process?

[Dave Bittner]

Yeah, I mean, we, we lay it out all out on our blog, so you can go through it on there. If there's anything that's not clear from this conversation, but they looking at it from a high level, the idea is that we had to unpack the bootloader. We had to extract an initial RAM disk from that. Within the init RAM volume, we found an encrypted installer package. But fortunately, because it was a virtual machine, the key for that installer package was also within the initram volume. So we decrypted the installer volume, and in there we found a pair of key encrypting keys along with a lot of bash scripts that actually handled the process of decrypting the firmware and installing the initial file system when you first set up the firewall. And so once we had recovered those, through the process of reverse engineering, we figured out how the decryption worked, and we were able to use the key encrypting keys to extract key material from the firmware image, decrypt the actual keys that were used for the encryption, and then use them to decrypt the package that was contained within the firmware bundle itself. So it was multiple layers of encryption all kind of piled on top of each other.

[John Williams]

Yeah.

[Dave Bittner]

But once we got through all the bash scripts that explained it and had a good, clear sense of how it worked, we were able to automate the process of doing all of that in reverse.

[Unknown]

So as someone who admittedly has only a passing knowledge of how encryption works, when you're looking at something like this, is it fairly easy to establish? I guess I should ask it a different way. How hard is it to establish exactly what kind of encryption you're dealing with?

[Dave Bittner]

Sure. Well, it depends a lot on what your target is and how you are determining the type of encryption used. In this case, because it was mostly bash scripts, once we had gotten to the point where we had access to those scripts, the rest of it was fairly easy to understand. It was a long process because there were so many scripts working together, but in the end, they were using open SSL with AES keys, and those are pretty common, you know, encryption techniques. So once we found those commands, then it was easy to understand what was going on. It was just a matter of digging them out of some really deeply nested functions.

[Unknown]

And so how would you rate the sophistication of what SonicWall was doing here?

[Dave Bittner]

That's an interesting question. I'm not sure sophistication is necessarily the right word, because it was really just. It was more complex than it was sophisticated. Right.

[Unknown]

Resistance.

[Dave Bittner]

Yeah. If you just pile more layers on top of it, does that make it more sophisticated or just more annoying to reverse engineer? Right.

[Unknown]

That's interesting. So you get in and you're successful at decrypting this, what happens next for you all?

[Dave Bittner]

Well, so we released a tool called SonicRack, which automates the process of extracting the keys from the virtual machine image and then decrypting the actual firmware so you can access the root file system. We wrote this tool to simplify the process so that in the future, when we do want to research new vulnerabilities, especially if there's a new end day that is dropped, we can very quickly pull out the binaries that matter from the root file system and be able to, for example, run them through a patch diff process and understand what code has changed between the two latest versions, which makes It a lot easier to, you know, find and write an exploit for a new vulnerability that's come out.

[John Williams]

We'll be right back. Looking for a career where innovation meets impact Vanguard's technology team is shaping the future of financial services by solving complex challenges with cutting edge solutions. Whether you're passionate about AI, cybersecurity or cloud computing, Vanguard offers a dynamic and collaborative environment where your ideas drive change. With career growth opportunities and a focus on work life balance, you'll have the flexibility to thrive both professionally and personally. Explore open cybersecurity and technology roles [e-mail].

[Dave Bittner]

Foreign.

[John Williams]

Investigating is hard enough. Your tools shouldn't make it harder. Maltego brings all your intelligence into one platform and gives you curated data along with a full suite of tools to handle any digital investigation. Plus, with on demand courses and live training, your team won't just install the platform, they'll actually use it and connect the dots so fast cybercriminals won't realize they're already in cuffs. Maltego is trusted by global law enforcement, financial institutions and security teams worldwide. See it in action [e-mail].

[Unknown]

And what sort of vulnerabilities or security concerns did you all uncover here?

[Dave Bittner]

Well, the last major one that came out, Sonicwall had announced an auth bypass affecting their SSL vpn. And because we had put in the work to do this decryption, we were very quickly able to run a patch diff report when they released a patch for this issue, find the vulnerability and write an exploit for it. And as a result that allows us to first warn our customers of our Cosmos platform that where they're vulnerable. So we know exactly which of their servers are exposed and can be affected by this vulnerability. But we also are able to give this proof of concept exploit that we've written to our operators so they can demonstrate the impact and show, hey, not only do we think you're vulnerable based on what we see, but we know you are because we ran this script and we can prove it. And this just gives the security teams for our customers a lot more, a lot more evidence to help them get things patched quickly, you know. So for us it's all about trying to make sure that we are finding the vulnerabilities and exploiting them so that we can demonstrate impact and get things fixed before malicious attackers do the same thing.

[Unknown]

Well, help me understand like some of the conversations that go on behind the scenes with you and your colleagues at Bishop Fox, because you know, it's one, it's there's several layers of your process here as well. It's one thing to internally decrypt this for your own explorations and use. It's another thing to publish a tool so that anybody can do that. Are there any discussions in House about, you know, how. Hey, Ganger, you know, are we sure we want to do this and what's the justification?

[Dave Bittner]

Oh, absolutely, yeah. I mean, we take this very seriously. You know, we do have responsible disclosure policies that we follow, and when something like this comes up, we will talk about it and make sure that we're making the right choice. And in this case with Sonicwall, we felt like it was justified to release the tool. And the reason for that is that SonicWall is a regular target of attacks. It does regularly publish advisories about new vulnerabilities that have come out. And what we will often see is that as soon as the vulnerabilities are announced, they've already been exploited in the wild. And so it's often malicious attackers that are driving this ahead. And so when you have a barrier like encryption that gets in the way of doing good faith security research, we find that it can be helpful to, you know, to break down that barrier and release the tools so that, you know, small independent researchers have the same access to the root file system as do well funded nation state actors, which are typically the ones responsible for this type of malicious behavior. Right. From our point of view, releasing a tool like this helps to level the playing field and facilitate more good faith research, alongside of some of the attackers who already have the capability to do this.

[Unknown]

I see. Have you been in communication with anyone at Sonicwall about this?

[Dave Bittner]

Not back and forth. We did give them a heads up to let them know that we were going to be publishing this. We didn't get a response from them on it.

[Unknown]

I see. Well, how do you suppose your research here informs folks who are out there using Sonicwall's products?

[Dave Bittner]

Well, I think it can help them. If they're technically savvy enough, it can help them to gain a better understanding of how their tools work and what SonicWall is doing to protect them. By unpacking the root file system of a firmware package, it's not like you're getting access to source code. It doesn't go that far. Most of what you'll find in the file system are compiled binaries. So you still need to be technically savvy to be able to reverse engineer them and understand what's happening. So. So it's really only assisting people who are. Do you know who are already well versed in security research to do this. But by having more researchers looking at this code and reporting bugs and vulnerabilities directly to sonicquall, it gives them the opportunity to fix things before attackers get their hands on them. And so, from my point of view, this is something that should inspire more confidence on the part of people who use Sonicwall products.

[Unknown]

Do you understand SonicWall's motivation for encrypting the firmware?

[Dave Bittner]

Yes and no. I mean, I think that from a certain point of view, it makes sense to slow down research efforts. But on the other hand, I think that when you're the developer of a software, you often fail to take into account that those barriers typically only give malicious actors an advantage. You know, and that. But that is what we see from our side of things.

[Unknown]

I see looking at sort of broader trends when it comes to firmware security. How does this research tie into that? I mean, are these the kinds of things that. That you're seeing that are typical for organizations to try to secure their firmware?

[Dave Bittner]

Yeah, it's. It's pretty typical. I mean, not all companies will go to the same length of piling on, you know, multiple layers of encryption, but it is pretty standard. When we look at other vendors in the firewall space, they will usually have some kind of basic protection around their firmware images. Not all of them, but we would usually expect.

[Unknown]

It is firmware an area of security that you feel people have, that it gets the attention it deserves?

[Dave Bittner]

I think it does, yeah. Particularly for these larger vendors, you know, the ones that really kind of dominate the market with hardware appliances that are used to protect corporate networks. From where I sit, I see a pretty even spread of attention across them, both from good faith researchers and from malicious attackers. But we regularly hear from other vendors in this space. So it's definitely not just sonicwall that we're picking on here.

[Unknown]

Sure. Well, based on the research here, what are your recommendations for the folks out there who are using this kind of hardware? Any words of wisdom for them?

[Dave Bittner]

Well, the main advice for consumers of this technology, it's the same as always. You know, if you have a firewall device deployed on the edge of your network, make sure the management interface is not exposed, because anytime there's a vulnerability affecting that side of the device or those components, having it exposed to the public Internet just leaves you vulnerable, and there's no need for it to be even. The vendor advises, make sure your management interface are only facing your internal networks. We still see way too many devices misconfigured that way on the Internet. But when it comes to other vulnerabilities that also affect like the SSL VPN component, that gets a bit more tricky because that interface is designed to be public facing. And so really then the trick there is just making sure that you are staying aware of the news that you're on top of security advisories when they come out and you patch your devices as quickly as you can.

[John Williams]

Our thanks to John Williams from Bishop Fox for joining us. The research is titled Tearing Down Sonic Decrypting Sonic OSX Firmware. We'll have a link in the Show Notes and that's Research Saturday, brought to you by N2K CyberWire. We'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity.

[Unknown]

If you like our show, please share.

[John Williams]

A rating and review in your favorite podcast app. Please also fill out the survey in the show notes or send an email to cyberwire2k.com this episode is produced by Liz Stokes. We're mixed by Elliot Peltzman and Trey Hester. Our executive producer is Jennifer Ibin. Peter Kilpe is our publisher and I'm Dave Bittner. Thanks for listening. We'll see you back here next time.

[Unknown]

This episode is brought to you by Universal Pictures. Today's the day. From Universal Pictures and Blumhouse come a storm of terror. From the director of the Shallows the Woman in the Yard don't let her in. Where does she come from? What does she want? When will she leave? The Woman in the Yard in theaters now.