Loading summary
Transcript159 lines
- [00:00]
Patrick Gray
Foreign and welcome back to Risky Business. My name's Patrick Gray. Yeah, I'm back on deck after having a lovely holiday in Fiji. Part of me in here, I mean I'm still back there in my mind a little. But it is good to be back on deck. Adam and I are going to go through the week's news in just a moment and then we're going to hear from this week's sponsor. And this week's episode is brought to you by Airlock Security, a company that I really love. They're an Australian company that does allow listing software and it works on Windows obviously also Linux and Mac and you know you can use it for allow listing at scale. They've got customers with like 100,000 plus endpoints under management but you can also do like host hardening with it and prevent people from being able to easily move laterally in your environment using LOL bins. It's really good software. So Airlock Digital, this week's sponsor and we're talking to Dave Cottingham who is Airlock CEO this week are about what it's like building a grown up enterprise platform for allow listing where you all of a sudden you need to have like a multi role console and whatnot. It's a conversation that's much more interesting than it sounds now that I've just said what it's about. But you'll see what I mean. That's coming up later. But first up of course it is time to get into the week's news with Adam Boileau. Mr. Adam, thank you for joining me.
- [01:25]
Adam Boileau
Yeah, it's good, good to have you back on the Internet I guess a little bit melted while you're away. So that's good.
- [01:30]
Patrick Gray
Yeah. I tried to not pay attention but I couldn't actually escape this story because I texting me about it while I was over there. So this is a bit of a twisted road this one. It starts off with a piece from ProPublica which was published about a week ago and the piece basically said that Microsoft has a whole bunch of engineers in China who do support work for Microsoft's cloud customers, which includes the Pentagon. So I mean look, getting the Chinese MSS to do your cloud support seems like a weird choice for the Pentagon, but this is the way that they've done it. But don't worry, they have a compensating control Adam, for this which is they have these digital Sherpas who will escort these digital escorts which will escort these Chinese workers into the Pentagon's cloud where there is classified Information to make sure they don't do anything wrong. And you know, that's fine, right? That's fine until you read in this article that the pay rate for these digital Sherpas whose only requirement, like tech skills are nice to have, the only requirement is that they actually hold a clearance and they're paying them 18 bucks an hour.
- [02:41]
Adam Boileau
Oof. Yeah, that one really hurts. Like the idea that you can secure a malicious person's access, admin access to your environment by paying someone to watch them, like is already kind of shonky enough without it being someone who doesn't understand the specific technology. And for that kind of pay rate, like, you know, you're not getting, you're not getting people who could do the actual work for 18 bucks an hour, otherwise they would be paying them to do it. And the idea that you can make this okay, I mean, clearly you can't, right? And that's the sort of the revelation from the story. But it kind of, it looks like this has been going on for really quite some time and that this was part of how Microsoft made the costing work to go sell the cloud to the dod, which, you know.
- [03:33]
Patrick Gray
Thumbs up. Two thumbs up, buddy. What a great idea, right? So yeah, this, this actually picked up real quick. And one of the reasons it picked up is you had, you know, well known right wing loon Laura Loomer picked it up and she had access to some guy who was like a whistleblower who talked about how, yeah, getting support from Microsoft. And it's like the person on the other end of the line is Chinese. So this is, you know, really picked up across like this is why it's a bit of a twisty one, right? Is because you've got ProPublica running with this and then it gets picked up also by the very right wing sort of mega fringe. And everybody agrees, like just a rare moment of bipartisanship, right, A very rare moment of bipartisanship where absolutely everyone agrees this is a bad idea. And Microsoft, except Microsoft, we've already said, oh, we're not going to do that anymore. But this has culminated in Pete Exif, the US Defense Secretary, announcing that, yes, no more Chinese contractors supporting DoD Cloud, please. Here is an excerpt from him announcing.
- [04:36]
Pete Hegseth
That it turns out that some tech companies have been using cheap Chinese labor to assist with DoD cloud services. This is obviously unacceptable, especially in today's digital threat environment. Now this was a legacy system created over a decade ago during the Obama administration, but we have to ensure the digital systems that we use here at the Defense Department are ironclad and impenetrable. And that's why today I'm announcing that China will no longer have any involvement whatsoever in our cloud services, effective immediately.
- [05:13]
Patrick Gray
So there we go, Pete Hegseth and his wonderful hair announcing, yes, no more, no more. The MSS will no longer be supporting the DoD's cloud infrastructure.
- [05:25]
Adam Boileau
I mean, it's just bonkers, you know, it's bonkers. And you got to wonder, like, how many other places have done the same thing because, like, the market competes on price and if Microsoft had to do this to be cheap, presumably everybody else also has to scrape the bottom of that barrel as well to be able to match the kind of rates there. So, like, I wonder how many other people are going and quietly, you know, changing their equivalent program to do the same sort of thing for various bits of, you know, the defence industrial base.
- [05:52]
Patrick Gray
Yeah, I mean, look, full credit to Hegseth though, full credit to the DoD for recognising that this is a problem and trying to turn it around. Although I do feel like Hagseth's like two week timeline to understand exactly how big a problem this is is somewhat ambitious, let us say.
- [06:09]
Adam Boileau
Yeah, I mean, this is a program that has been there for a long time and, you know, getting rid of it overnight is not going to be straightforward. Right. They have to figure out who's going to do this, who's going to pay for it, you know, all those kinds of things. And you. This is not a unique to Microsoft sort of thing we've seen. I know I've been on the other end of this kind of arrangement as well in my professional career where I was allocated the Sherpa to vet the commands that I was going to type into a UNIX box and then type them in. For me, the whole idea is just.
- [06:44]
Patrick Gray
I want you to go into a little bit more detail about what that experience was like, because you have told me that over the last few days. And you did not feel that the person who was supposed to be your guide in this sensitive environment really had much of an idea of what you were actually doing?
- [06:58]
Adam Boileau
No, no, not at all. I think in this particular case I was doing like UNIX host reviews on some UNIX boxes in some sensitive places. And I was given, you know, in order to get privileged access to it, somebody else had to be my terminal and I had to tell them what to type. And then I wasn't even allowed to see the output of the commands. The outputs of the commands got saved to a file and then the person who was my sherpa collected that output, gave it to their manager, who gave it to their manager, who gave it to the project manager at the company that was providing the Sherpa, who then gave it to my project manager, the customer, who then gave it to our project manager at Insomnia at the time and then they gave it to me and then after it had been through 47 hands and this process of course took weeks.
- [07:45]
Patrick Gray
Well, I mean, you know you with a process like that you could really start to see the efficiency benefits of using people in low cost, low cost places. Right. What an efficient process. Amazing.
- [07:56]
Adam Boileau
And like the thing that I found best about that whole experience though was I still shelled the damn thing. It just, you know, it was not a fast process or a fun process, but we still got there in the end, still managed to privesk, still managed to get my job done. So yeah, very, very effective. Good work.
- [08:11]
Patrick Gray
Yeah, amazing, amazing. Now look, the other thing that's going on at the moment, some bread and butter infosec that I'm guessing a lot of the audience are dealing with at the moment, is this in the wild exploitation of a recent bug in SharePoint server? I mean that ain't great.
- [08:29]
Adam Boileau
No, I mean if you leave the SharePoint lying around on the Internet, a lot of people do still. I mean and this is on prem SharePoint, not the cloud SharePoint then yeah, you're going to have a bad time. And in this particular case Microsoft has patched it over the weekend. But this was being used in the wild pre patch.
- [08:46]
Patrick Gray
Yeah, it was being used as odate, right?
- [08:48]
Adam Boileau
Yes, it's not great. And this is like a combination of sort of an auth bypass plus deserialization based code exec which not what you want in your SharePoint but also we've seen attackers using it to steal machine keys which in net applications if you know the machine key you can deserialize and run code kind of by design and the machine key is meant to be secret for that purpose. So they're using this to gain machine key access and then even after it's patched they can come back in The Future and CodExec and given that like everybody from China seems to be hacking all of the sharepoints over the weekend, it's going to be a rough time even if you did patch it, if you didn't patch it quite fast enough may not matter, you know, three months down the track when they decide to come back and hit you again.
- [09:37]
Patrick Gray
Yeah, I mean you and I were talking before we got recording about this one and you know, my comment was I, I do not understand why people who create these sort of web applications expose so much attack surface pre auth. You know, like it's just, it just seems crazy to me that before you've authenticated to some sort of, you know, web service that you can hit anything that isn't the login, like the username and password fields and that should be pretty easy to not have exploits in. You know what I mean? Like it's just nuts.
- [10:12]
Adam Boileau
Yeah, it is, it is. And in the NET case in particular, the fact that their authentication scheme relies on deserializing an object, right. The cookie that you get to auth with is an object that gets deserialized. So the by design the unauthor tax service includes the serialization which they rely entirely on that machine key as that's the only thing that stops you from turning that into arp code exec. So you know, having ATT and CK service is one thing, but then kind of specifically designing high risk stuff into your pre auth attack surface for the sake of what? Right. One wonders. It's just. Yeah, it boggles the mind and there's just so much complexity in modern authentic that you kind of can't really trust it.
- [10:57]
Patrick Gray
Well, yeah, I mean there is that, but there's also the fact that you've got a lot of these sort of services and appliances where they just leave stuff lying around everywhere publicly accessible. Like think of stuff like PHP file include vulnerabilities. Right.
- [11:11]
Adam Boileau
Or anything made by Oracle.
- [11:12]
Patrick Gray
Right, exactly.
- [11:13]
Adam Boileau
40,000 JSP scripts that you can hit unauthed and they're all terrible. Yes, yeah.
- [11:18]
Patrick Gray
And for too long, and it's something I've said before, for too long we've sort of considered that authentication is access control and it's just not. And this is why. So I mean, what can you do about this? If you want to run stuff like SharePoint, if you want to run like file transfer appliances, payroll systems, this, that whatever it is out on the edge of the network for your staff to be able to access and for partners to be able to access, you know, you might consider using some sort of reverse proxy, right. So I think Okta has one. So if you're an Okta customer, you can do that. It's probably going to cost you a bit Authentic, which is an open source IDP that I work with, they've got a reverse proxy as well. There's a company called Pomerium that just does reverse proxies. There's the company that I'm on the board of which is Knock knock, which also has a reverse proxy. So there's a lot of ways to simply set up reverse proxies onto systems like this that basically do not give you anything until you're authenticated. Right. Like you just can't hit the machine. So, you know, I would think that is something people should look at. Go get, run zero, have a look at what's on your perimeter and then start cutting this off. Cut it off.
- [12:21]
Adam Boileau
Everything you can get rid of, you know, everything. Every little bit of attack surface that you know, you have to be at least behind auth4. It just makes attackers miserable and like make them miserable.
- [12:31]
Patrick Gray
Yeah, it does, it does. You know, I just think that's a sensible thing to do at this point. I think like leaving this sort of stuff out at the edge with nothing in front of it is just not. It's not going to be a good time.
- [12:41]
Adam Boileau
I mean, we can't trust the vendors to write safe software to do their own auth. You've got to layer something on front of it.
- [12:46]
Patrick Gray
Yep, that's it. That is it. All right, so now look, what have the Chinese been up to apart from helping the DoD manage its cloud environment? Well, exploiting SharePoint bugs is one thing. They've also been hacking the National Guard as well, apparently, according to NBC News.
- [13:03]
Adam Boileau
Yeah. Kevin Collier wrote this piece about the Chinese group Salt Typhoon. Apparently broke into one of the US State's National Guard. We don't know which particular state. So far that particular detail hasn't come out. But there was a memo, I think from the Department of Homeland Security which described the fact that there had been this intrusion. And obviously National Guard's not a place you particularly want there to be Chinese attackers.
- [13:28]
Patrick Gray
Yeah, I mean, Salt Typhoon is something I normally associate with telco hacks, so I'm not sure quite why they're doing this.
- [13:34]
Adam Boileau
Well, I mean, I guess, you know, why not if you're, you know, if you're hacking stuff to gain access, prepare the battle space etc, etc, then Yeah, I mean, military stuff, telco stuff, it's all high value.
- [13:45]
Patrick Gray
But I mean, Vault Typhoon is the one that's doing that sort of more preparation of the battlefield. Right. Salt Typhoon has just been telcos and like wiretaps and stuff like that. Right.
- [13:53]
Adam Boileau
So, yeah, who knows? Yeah, maybe it's contractor, you know, just fill in their boots.
- [13:59]
Patrick Gray
Well, speaking of contractors that do like hacking for the Chinese government, the Italian authorities, Italian authorities working with the FBI have arrested a Chinese guy in Italy over the hafnium like exchange hacks. And everybody would remember that back then. Oh my God, it was like five years ago now. But remember when every single exchange box was just getting shelled by everyone completely. This was. Yeah, there was a China nexus to that. And it looks like this guy, zhu Zi Wei, 33 years old, he's been charged over this stuff and it looks like he works for a contractor called Shanghai Power Rock Network Company, which awesome name, dude, but they apparently contracted the MSS and Yeah, so now he's in trouble and he's going to get extradited. Now does this set a precedent that China can start arresting NSA people? I don't think so personally because this sort of like massively destructive attack, you know, it's just very messy, creates a lot of collateral damage. It's very different to the sort of operations that five Eyes countries tend to do where you'd rarely hear of them because they tend to be quite careful, they tend to limit scope, they tend to clean up after themselves. Whereas these guys, they're just the proverbial bulls in a China shop.
- [15:17]
Adam Boileau
Yeah, they certainly did make quite a lot of mess. And I think, you know, also, you know, if you work at NSA or some Five Eyes Agency, chances that you're going to go to Hong Kong on holiday, probably not super high. Like, I think that's probably drilled into you pretty hard. Yeah.
- [15:31]
Patrick Gray
But you might, you might go somewhere that might have some sort of agreement with China or whatever. But I, I. Look, there's a couple of reasons why it won't happen. First of all, the OPSEC in the Five Eyes agencies is actually pretty good. Yeah. So I think that's, that's a real thing that would prevent this. But yeah, so I wouldn't just say that like, you know, it's about NSA operators not having holidays in Hong Kong. Like, I just, I don't think it's the same thing.
- [15:54]
Adam Boileau
Yeah, I mean, I guess I don't know what the state of, you know, China's relationships with other countries for extradition agreements are. And I guess that's the problem. Right. If, you know, if they do have relationships with, I don't know, Brazil or whoever, then maybe that makes going somewhere a little more complicated. But yeah, it's the thing, I guess that struck me about this story is like when we have seen indictments of Chinese operators by US law enforcement, there's been quite a lot of people saying, look, what's the point of this? Like, why do we bother doing this? They're never going to face justice. And then you Know, one time in 100, one of them gets on a plane and goes to Milan. So you know.
- [16:28]
Patrick Gray
Well, and it's also, it's also, it's also a mechanism for the US government in this case to actually put pen to paper and publicly do an accusation. Do you know what I mean? Like it's a public attribution, it's an official document. Like I, you know, I don't think it's pointless. I think just the, the act of actually doing these sort of indictments, even if the person is never going to be apprehended, I think there is value in that.
- [16:53]
Adam Boileau
Yeah, yeah, I'm certain. But there has been a lot of criticism I guess of that, of that work and whether it's kind of a waste of effort. But hey, you know, I'm going to be interested to see what happens when this guy gets back to the us like what's that going to look like?
- [17:05]
Patrick Gray
Well, it's going to look like a bunch of charges and probably the Chinese government saying these are all lies, these are all lies, this guy has nothing to do with us. So I can't imagine they're going to go to bat for this guy. Like probably not. You know, apparently Chinese state backed hackers, a group, an Uncategorized group, UNC3886 has apparently been owning up some critical infrastructure in Singapore according to a senior official official there.
- [17:34]
Adam Boileau
Yeah, that makes a lot of sense. I mean Singapore is definitely the sort of place you'd expect there to be Chinese attackers. I think this particular group is one of the ones we've seen, you know, exploiting Fortnite bugs, exploiting VMware bugs. Like feels like one of the, you know, kind of private contractor sorts of companies. So yeah, not surprising to see them there.
- [17:53]
Patrick Gray
No. Now we've got some good news here, which is for Brits young, between 17 and 20 have been arrested over sort of scattered spider ransomware stuff, including the attacks on Marks and Spencer and Harrods. Some of these people were also involved in like the MGM casino hacks back in 2023. And yeah, they're just going to have a very, very bad time I suspect. Although you know, if you had to choose as a computer criminal whether or not you would want to be arrested in the UK or whether you would want to be arrested in the United States, like it ain't even close. I to go to the UK court, please. I'm unsure whether or not there's going to be any sort of extradition here. If anyone's seeking it or whatever over the MGM stuff. But yeah, at the very least they've been bailed. All four have been bailed. But yeah, they're in big trouble.
- [18:45]
Adam Boileau
Yeah, they certainly are. And Brian Krebs. So when they were arrested and then subsequently bail, we didn't see any public reporting of their names. Krebs did some footwork and figured out that one of them, it's a guy called Owen David Flowers, who was the one that was involved in the MGM hack. And he's pulled some threads about some of the nicknames the guy was using on the Internet. Another one was the guy who was behind Docspin, also a member of Lapsus, been kicking around that scene for a number of years now. And Krebs has a good kind of write up of that particular guy's history. Thala Jubeir is his name. So yeah, that looks like kind of bad news sort of guy. He got doxxed by one of the other members of Lapsus back in that particular set of scene wars. So yeah, I've been kicking around for.
- [19:35]
Patrick Gray
Quite some time with friends like those, as they say. It's interesting too because Tom wrote this up. I think it was like around the time I left for my, for my break. That threat intel company's been putting out research saying that the, the idea that Scattered Spider is a vibe, as we like to say on the show, is not actually right. And there's something, there's a very small group of people who direct most of the really malicious bad activity. Something like four people. I think two different companies had settled on that four people figure. I'm not sure if these, if any of these are part of that four, but the idea seems to be, at the moment, the thinking seems to be out there that it will be possible for law enforcement to make a dent on the activity that we've been attributing to this scene and that it's maybe not the, the entire scene that's doing most of the damage. It's just a few sort of skilled and highly motivated people. So that is interesting.
- [20:29]
Adam Boileau
Yeah, yeah. I mean it's, you know, I'm sure when we see the inside details of all of this, like we'll have some, you know, have a better idea of how it works, but that tends to be like these underground communities. There tends to be a few big personalities that drive everything else and kind of like drag everyone else along with them. And then, you know, people want to level up and get better and you know, prove that they have, you know, have skills too. And that Kind of drives a bunch of the, the competing sort of activity they undertake. But yeah, there tends to be some, some big names that once you drop the hammer on them is pretty bad for the scene overall.
- [21:05]
Patrick Gray
Yeah. Now John Greig has a write up about an arrest in Brazil over the theft of $100 million through Brazil's instant payment system, which is called Pix. Right. So this guy worked for CNM Software. And I'm not exactly sure if does CNM actually maintain pics or it just like they use their access in CNM to get to the Pix system.
- [21:29]
Adam Boileau
I think what I had read is that they are a provider that does like Pix software access for smaller banks that don't have their own infrastructure to do it. So they are one of the people that provides access to Pix mostly for smaller banks.
- [21:41]
Patrick Gray
Okay. Point is, this breach was enabled via access into a company called CNM Software. And the police have arrested this guy. He's 48 years old. He sold his account name and password to some hackers for US$2,700 in two separate cash payments after they approached him in a bar. And that's led to a $100 million theft.
- [22:08]
Adam Boileau
Yeah, I mean that's a hell of a return on investment for the attackers. But yeah, that's like this insider kind of threat aspect to it. Like really should make people's spidey senses twinge. Right. Because we spend so much time on cyber controls and computer stuff. But in the end, if one of your employees is willing to sell their access for not a whole bunch of money, then a lot of that kind of edge perimeter stuff really stops mattering when it's a case of spending that kind of money. And I think this guy was, I think I read that he was an electrician or something like some other technical trade who had retrained into it later in his career because he wanted to better himself. And it's kind of sad in the way that probably he didn't really understand what he was, the impact of what selling his credentials would mean.
- [22:58]
Patrick Gray
So we don't know what they told him they wanted to do with that access as well. Right. But I think the, just generally speaking, if someone approaches you at a bar and offers to buy your username and password for your work account. Yeah, that's not go well. It's not going to go well. It's really not going to go well. So yeah, I mean, what do you, what do you even say about this? I mean, what, what can you do? As you said, like, okay, great, okay, give everyone Uber keys. Well, then they'll just sell their Yuba keys.
- [23:24]
Adam Boileau
Yeah, yeah.
- [23:24]
Patrick Gray
You know, if you restrict. If you restrict access into those physical buildings, like, at what point are they. Do they start taking bribes to smuggle people into the buildings and access a terminal and like, you know, when you've got a motivated insider like this. Yeah. Woof. Pretty hard.
- [23:39]
Adam Boileau
Yeah, it's hard. Right, and then. And then what's the. What do you do? Well, you. You pay good wages and you treat your employees well and make sure no one's got any grievances. And like, it's hard and expensive.
- [23:49]
Patrick Gray
Yeah. But even then, we've seen malicious insiders at places where they were being looked after, where they were being treated well, you know, and it can be something as simple as, like, office politics that trips someone into doing this, you know? Now let's talk about CrowdStrike and how it has completely lost its mind. CrowdStrike is disappearing up its own clacker, and it is just bizarre. We've got a piece here from Andy Greenberg looking at some research out of UC San Diego trying to pin down exactly how disruptive their colonel panic was to the healthcare sector in the United States. Now, I think that's a worthy. I think that's a worthy endeavor to try to figure this out. I do. Now, can you pin it down exactly? No, but it seems like the researchers here have made a good faith attempt to see, you know, how many hospitals had disruptions to their systems. CrowdStrike's response to this has been kind of deranged. They've called it junk science. They've said drawing conclusions about downtime and patient impact without verifying the findings with any of the hospitals mentioned is completely irresponsible and scientifically indefensible. And then they pivot from there into a. Well, we recognize we had an incident and we sincerely apologize to customers. So look, we'll get into a discussion on this in a moment, but look, I just want to say, a couple of weeks ago, we actually got an email from crowd. I'm about to spill some tea, everybody. I'm about to spill some tea, as the kids say. We got this email from CrowdStrike's like, APJ, Director of Public Relations, asking us to correct an error in a newsletter that was written and published by us. It was Catalan's Risky bulletin newsletter, and it said, can you correct the sentence? A bug in the CrowdStrike kernel driver took down over 8.5 million Windows systems. And then. This is the CrowdStrike person speaking. The incident was from a defective Rapid Response content update as noted on our website and in the rca also attached to Rapid Response Content updates are not code or a kernel driver. This sentence is inaccurate as written. We request that you please replace blah blah blah blah blah blah blah to accurate language. Now, I ignored this and then they emailed again. Hi Catalyn, hope you're well. As per my initial email yesterday, could you please update blah blah blah blah blah blah now, if you've got a content update that can trigger this condition, you have a bug in your kernel driver. Would you agree with that?
- [26:26]
Adam Boileau
I would agree with you.
- [26:28]
Patrick Gray
So I wrote back. In the end I said, hiya, name redacted. I'm the publisher here at Risky Business Media. It takes a pretty serious deficiency, a bug, you might call it, for a kernel driver to crash when supplied with a content update. We're happy with our phrasing here, but the reason I wanted to talk about that email, right, is because I think CrowdStrike has almost taken on like a bit of a cult like mentality. I think CrowdStrike makes best in class, you know, EDR, right? Like, it is fantastic. I think the Microsoft stuff's good as well. The Sentinel One stuff's good as well. But I think, you know, CrowdStrike really has been the company that has, for the largest proportion of users, kind of defined that space. They've grown a lot, they keep buying and bolting on all sorts of monstrosities onto this to try to platformatize their. They're offering. Right. And by all reports, their additional products just aren't really that good. Okay, I'm just going to say it. They're not that great. People will sort of bite the bullet, buy it, use it. Because, okay, we're already a CrowdStrike customer, we've already got that footprint out there, but no one really thinks it's best in class anything. And yet the people who work there and the PR response just seems bizarre. It seems, it seems cult. Like, do you sort of see what I'm saying?
- [27:41]
Adam Boileau
Yeah, yeah, it does seem like a strange hell to die on, right? Like you could make your product good and do good work and do all the engineering and necessary stuff to not have these problems in the first place. Or you can send your PR people out to, you know, tell a small Australian blogger, like, in what, in what.
- [28:01]
Patrick Gray
Universe, in what universe was that a good idea for them to do that? Like, in what universe is that a hill worth fighting on, worth climbing, let alone dying on?
- [28:10]
Adam Boileau
Yeah, yeah, why not?
- [28:11]
Patrick Gray
They could gargle my, my CrowdStrike. You can gargle my coconuts.
- [28:16]
Adam Boileau
Yes.
- [28:18]
Patrick Gray
Anyway, back to this paper. What do you think about this research? I think it's, I think it's a good idea to try to quantify what the, what the disruptive impact to the US health care system is. I have always said that this CrowdStrike outage, or, you know, crowdstrike blue screen of death across, you know, millions of boxes that happened last year was the best sort of simulation we're going to get of a large scale cyber attack until we actually get a large scale cyber attack. And so, you know, I think this is a good idea.
- [28:43]
Adam Boileau
Yeah, no, it is. The researchers in question, they were already monitoring the uptime, like the availability of hospital IT systems because they had a project to try and determine who was getting ransomware. And so they were, they had this monitoring infrastructure in place and then when the crowdstrike thing happened, they were able to very quickly use their existing tooling to go collect the same data and say, here are a bunch of hospitals where systems that we can observe from the outside that are clinically relevant, you know, things like portals for doctors to log into and so on, are not available that were previously. And they tried to correlate based on timing across the set of hospitals that they were looking at, which I think was about a third of the U.S. medical system that they had some coverage over. And then from that infer, you know, that bad stuff was happening because of CrowdStrike and they controlled for like there was an Azure outage about the same time and they tried to kind of account for that. So from the point of view of the research methodology, it seemed like a pretty reasonable, at the very least, it's interesting. The data they have is interesting and it's not, you know, clearly not perfect because they weren't able to go talk to all those hospitals and get insider data. And they're doing it from the outside, but as a piece of research, it seems pretty representative and relevant and useful.
- [30:02]
Patrick Gray
Well, I don't think they're making any claims beyond what they've said. The data says. Right. Like where, you know, so, so for CrowdStrike to then attack them like this just seems bizarre. And I guess I've already got my backup because they're writing to us because they're saying it wasn't a kernel bug. I mean, really, does that make a difference, guys? Like if it was a kernel bug or a content update, you still nuke eight and a half million machines.
- [30:21]
Adam Boileau
Yeah, if the airport's blue screened, it does not matter if it was a content update or a content.
- [30:26]
Patrick Gray
You know, it's the ultimate, ultimate. But actually, yeah, actually it wasn't a kernel mug. It was, it was a, it was a rapid content update. It's totally different.
- [30:38]
Adam Boileau
Dear, oh dear, oh dear.
- [30:40]
Patrick Gray
So, yeah, anyway, I think, I just think they've kind of disappeared up their own behind lately and it's, it's, it's, it ain't a good sign for the company. You know what I mean? You know exactly what I'm talking about, right?
- [30:52]
Adam Boileau
I do.
- [30:53]
Patrick Gray
All right, we're not going to talk about these ones so much, but John Greig has a write up of two of these over at the Record. The cryptocurrency Exchange GMX had $42 million stolen from it and they've agreed to let the attacker keep 5 million as a bounty. I mean, we keep seeing this over and over and over. It's nuts, isn't it?
- [31:13]
Adam Boileau
Yeah, it really is. And it's just encouraged. Like it's classic tragedy of the commons, right, where every exchange or person that does this encourages attacks on all of them. It's just a bad idea. And the thing is, they're not even really legally binding. Right? I mean you could, we've still, we've seen other people get prosecuted despite having a sort of, you know, like this, you know, white hat reward fiction. And yet, you know, they still end up getting arrested anyway. So, you know, let's hope law enforcement decides they don't care and are going to go after these people anyway.
- [31:42]
Patrick Gray
I mean, I think the days of us joking and it was funny at the time that cryptocurrency theft is a victimless crime. You know, it's, it's, it's, it's, it's starting to get a bit serious. Like there's so much money in this stuff now and normal people are starting to invest in it.
- [31:58]
Adam Boileau
Like you just invest in very air quotes because like, ah, for God's sake.
- [32:03]
Patrick Gray
But they don't realize that. They think it's a proper investment. And I think there's, all the sharks are going to come in. I think there's some deregulation happening now in the United States in particular, which is going to allow all of these sharks to go and like sell people like crypto tulip bulbs into their 401ks and it's just, it's going to end in tears. The question is just how long. So there was another one, an Indian crypto exchange called CoinDCX. They lost 44 million bucks as well, and John Greig has also written up some chainalysis research that says 2.17 billion in crypto has been stolen in the first half of 2020. But 1.5 billion of that was the Bybit hack, which I still think was like, just so cool.
- [32:45]
Adam Boileau
Yeah, we do got a hand to the NOCs for that one because, yeah, they did a good job there.
- [32:49]
Patrick Gray
Yeah. But they're predicting like up to 4 billion in crypto theft this year, which is just wild. Wild. Now let's talk about this research out of expel, which is how to bypass a login flow where someone's using like a Yubikey by using this. What is it? Cross device. What's it called again? Cross device authentication.
- [33:11]
Adam Boileau
Yeah. Yeah. So this story is kind of interesting.
- [33:14]
Patrick Gray
It is, it is. I've read through it and I'm like, okay, I get it now. Yeah.
- [33:18]
Adam Boileau
So we. So Catalyn wrote the story up for, for us whilst you were away. And I remember like, you know, because we edit the script pretty early in the morning, we pasted it in our slack and said, hey Medal, you're gonna want to read this one because it's good read. And I woke up and this was how I spent the first like hour of my day reading this story. The deal is they looked at something that was being authed with Yubikey for the Fido key and this particular instance of it had a mechanism where you could fall back. If you didn't have your Yubikey available, you could fall back to another authentication mechanism. So what we are talking about here is a way to go around a Yubikey rather than a bug in Fido or U2F for Yubikeys themselves. And then this involved basically scanning a QR code with your authenticator app on a separate mobile device and using that to authenticate your login in this case as a second factor. So there was username and password and then you used a QR code to kind of phish the user in an attacker in the middle context. Now, in the way that you would do this in the U2F ecosystem, like there's the official way in U2F of doing cross device authentication. And in that particular like standards way, there is a Bluetooth callback mechanism that binds the browser to the auth device so that everyone agrees who you're authing to so that you can't do this kind of relay phishing or in the middle phishing. In this particular case, that wasn't what they were attacking. This site had another Mechanism, as best I can tell, I had another mechanism that used a QR code. So it wasn't the CDAP with Bluetooth binding, which is the way that you should be doing this. But overall, regardless of the specific mechanism here, the way that you can trick users and the way that we fall back around these robust authentication mechanisms really is the core problem here. Because people are used to having to get their phone out and scan a QR code. They're used to having to do things that are not just username and password now. And yet no one really understands how any of this actually works yet. And nor should they. We're meant to be solving these problems for them. So this is kind of illustrative of the complexity of modern auth and the complexity of non ideal auth situations when you're on a strange device, on a device that doesn't have a keyboard, on a device that doesn't have a camera, on a device that, that doesn't have the compute power or a USB port or whatever. So everything that we build that makes auth robust, we still have to deal with the real world of imperfect end user computing.
- [35:59]
Patrick Gray
Yeah, that's right. And I think we've talked a lot over the last, I don't know, like half a decade about how one of the weaknesses when you're using robust auth is always going to be stuff like when people need to reset a credential. And we've seen a bunch of these social engineering attacks quite recently where that's what people are doing. They're going to help desks, they're getting their, their MFA reset or whatever. There's another incident here that is mentioned in the xpal blog post where someone got, they compromised an account through a phishing email and then they just reset the password somehow and enrolled a Fido device which was perfectly good enough for all of the devices where they had to use that device. So, you know, this isn't a problem per se with like Fido or U2F. The problem is where all of the junk, junk auth reset flows and stuff around those. Yeah, it's all the glue. It's all the times when that's not how you're authenticating and trying to get into that state. And I think we're going to see a lot more of this sort of thing for years to come.
- [37:00]
Adam Boileau
Yeah, I mean, in the end, if we make auth really robust, people are going to keep wanting to get in. And the rise of the success of scattered spider in doing social engineering attacks on those edge bits you know, on the enrollment process, on the reset process, on the my dog ate my Yubikey process. You know, those that have a human part, like those continue to be the weak links and people who can exploit those are the ones that are going to, you know, continue doing the hacking and continue getting the shells.
- [37:30]
Patrick Gray
Yeah. Now Catalan also wrote this up, this one up for us, which was in, in his newsletter. Oh, lordy. So this is some research from John Tuckner and I've mentioned John on the show before. He runs this, this little startup that he's got called Secure Annex. And they're the ones that look into the security of Chrome extensions. And I think it's great. I really hope he does well with this. As I said, I've mentioned it before, I had a call with him a few weeks ago just really talking about what he was doing. So what he found is a bunch of extensions had this code stuffed into them that allowed a company to proxy web scraping requests through browsers that had these extensions installed into them. Right. So this company is like, we can do 100,000 like web scrapes in parallel and the way that they fulfill this is through these extensions which. Okay, is that illegal? I mean, it's probably not illegal if there's like a EULA there that someone has clicked through. Like, I'm guessing it's probably not illegal, but you probably wouldn't want it running in your browser, that extension, because it turns out it like, it disables a lot of like security headers and whatever. Like it's not good, let's put it that way. It's, it's like not, not a great thing. But I just found this, a really interesting story about the sort of things. Now someone is actually in there, in those like Chrome web stores, actually looking at these things, picking them up and shaking them. This is the sort of stuff that's falling out.
- [39:00]
Adam Boileau
Yeah, I mean this was being marketed to developers as a way to monetize their extensions. So instead of selling ads or whatever else, you can in this case sell your users a, quote, unused bandwidth input. So yeah, you'd install this SDK or you'd integrate this SDK into your, or integrate this into your extension. And then, yeah, when the machine was kind of idle or whatever else, then it would phone back centrally, make requests on behalf of the people and then send the results back. Which, you know, I'm sure the end, as you said, I'm sure the end user agreements or whatever said, by the way, we'll use our third party advertisers Code to mumble, mumble, mumble. But people don't expect that to mean we're going to proxy out your connection. And in the case of a private network or a corporate environment or somewhere where your browser has access different to the regular Internet. Right. There are some interesting kind of, you know, if I could on a pen test gig, go buy access to some end user in an organization's browser, I'm going to go hit the internal SharePoint, I'm going to shell all sorts of internal stuff and it'll be party time.
- [40:11]
Patrick Gray
I want to scrape everything on 10, whatever. Right?
- [40:13]
Adam Boileau
Yes. Yeah, exactly. Go find me the password store or whatever, dear. So probably not criminal, but definitely not good.
- [40:24]
Patrick Gray
No, it's definitely not what you want. And I think this is great marketing too for Secure Annex. Because people listening to this, CISO is thinking, God, what sort of controls do we have on our browser extensions? Yeah, you need to be looking at this. I think we're still early in terms of understanding what the potential for badness is with these extensions. And I think the crooks are actually lagging on this and they're going to catch up and we're going to start seeing all sorts of fun stuff happening in extension land. All right, we got to pick up the pace here because a long run sheet this week. We got to get through it. But let's look at this one from Joe cox over at 404 Media. There's this company called Farnsworth Intelligence, which its business model appears to be selling info stealer logs to like law firms, debt collectors and all sorts of companies, which just seems like, you know, there's no allegation that they're operating these info stealers, but they're to going, gaining access to infosteer logs and then selling that data, which seems like, I mean, okay, is it illegal to sell a username and password of someone's account that was obtained from an info stealer? I mean, I don't know, maybe, probably, maybe not. But using that password is definitely illegal. So, you know, like, I wouldn't, I'm not a judge, I'm not a lawyer, but I'm just saying I wouldn't operate a business like this because I would think, in addition to it being unethical, I would think, I would expect there are some legal issues here as well.
- [41:56]
Adam Boileau
Yeah, this seems pretty sketchy. Like the guy behind this, I don't know that he necessarily thought this particularly well through, but yeah, taking data that's on the dark web or wherever else you can get it and then Packaging up and reselling it. Now there is something to be said for buying access to this data in a professional context. Like I know back at Insomnia we would pay subscriptions for some services that provided us with data breaches and data dumps and stuff in the style of have I been pwned? But in a way that would give us credentials. So it would give you unfiltered access to some of this data. And they took care. Like the service we were buying was them taking care of collecting, indexing, sorting, deduping, et cetera, providing a stable API so that we didn't have to spend a day searching shady dark websites, trying to find our targets, passwords and things. So it was useful in that sense, but it was always a little bit kind of like how do we, you know, in those cases we were buying that service from people we knew and trusted. Right. So there was a personal relationship that kind of like established some of the bonafide. So we weren't just supporting randoms doing it. But it is a shady business and I wouldn't.
- [43:09]
Patrick Gray
Yeah, these guys aren't selling it for like infosec purposes or like so you can notify the person that their account's been breached. They're giving people passwords so they can use the passwords, you would think.
- [43:18]
Adam Boileau
Yes, exactly. Which is not great.
- [43:20]
Patrick Gray
Yeah. But I mean, you know, you can imagine like a bunch of the people that they're selling to are like skip tracers. Right. Who are trying to find people who've skipped out on bail. Like oh man, getting, getting their email account. Like, man, that's just making it easy. Right. So of course they're gonna, they're gonna use it. If I'm a skip tracer and I've got an opportunity to log into my target's email address, you know, party time.
- [43:37]
Dave Cottingham
Right.
- [43:38]
Patrick Gray
I'm gonna do that so I know where they are. We've also got this one from Zach whitaker over at TechCrunch, which is I initially thought, eh, because it's about people using SS7 to track people's phone locations. Although what makes this interesting is it's kind of exploiting I guess, weaknesses in SS7 to do this. Right?
- [43:59]
Adam Boileau
Yeah. So this is interesting. Normally as you said, this is just by design. The issue here is that most telcos have dealt with this kind of tracing by putting filters, essentially SS7 firewalls on the edge of their head network. That would say when I see an incoming request for a mobile device, location check to see if the device we're asking about is one of ours. So check the IMSI device identifier and if it's one of ours, don't answer that and drop the request so they can filter them for stuff that is on network. The people who were doing, who were abusing a bug here were, I guess you'd call it a canonicalization flaw where there are ways to multiple ways to encode these messages such that a naive filter which doesn't understand all of the ways of encoding it can be bypassed and they use like ASN1 bur encoding which is like super complex for no good reason. So yeah, the attackers were kind of malforming the MZ that they were asking about in such a way that it was still valid when decoded at the end but not valid when decoded by the SS7 firewall in the middle. So pretty sweet, you know, kind of technical exploit being used to do surveillance like this.
- [45:12]
Patrick Gray
Yeah, yeah, we've got some news out of Ukraine. The Ukrainians are claiming to have wiped a bunch of databases at Gazprom, which is obviously a company that's very important to Russia. It's their gas company. So we've linked through to that one. I mean, how much damage has been done? Hard to know. You're not really going to get some great visibility there. Oh, time to talk about. Yeah, bugs and bugs and stuff. John Greig's written this one up for the record, but there is a zero day exploit for crush FTP in the wild. I feel like I'm having deja vu. Didn't we have another one of these a few weeks ago?
- [45:47]
Adam Boileau
I think maybe this is a variant from a bug. Like I think someone reversed a patch and then figured something out. So this. I don't know the specifics of this one.
- [45:54]
Patrick Gray
They reversed the patch and found a different ode. Yeah, I mean that's.
- [45:56]
Adam Boileau
Yeah, I think it's like a, probably like a bypass in whatever protections. This was a bug in the web management interface, I think. So I don't know the specifics. We haven't seen a park for it, but yeah, it's being exploited in the wild. And if you happen to run any sort of FTP server on the Internet, you're probably already having a bad time. So get attention.
- [46:14]
Patrick Gray
We've got an interesting one here because I'm expecting a lawsuit because I think Cisco is about to sue Hewlett Packard Enterprise HPE because they've started including hard coded credentials in their devices as well. And we know that that's something that Cisco loves to do. They might have some, some patents around that. Not entirely sure. We'll have to see what happens. But yeah, there's some hard coded credentials in the HPE Aruba instant on access points. So like what are these? WI FI access points?
- [46:40]
Adam Boileau
Yeah, these are WI FI access points, like small, medium enterprise kind of WI fi access points. I don't know what the credential is. I'm assuming that it's going to be like a real sick burn and it's actually like Cisco username Cisco password Cisco. Because that would be hilarious by hp, but I suspect it's probably not.
- [46:55]
Patrick Gray
No, that's. That's right. And what do we got here? We got a pre auth. Oh man. More from Watchtower.
- [47:01]
Adam Boileau
So good. It's so good.
- [47:03]
Patrick Gray
We need to look, we need to send them a case of beer or something. We do Watchtower. Watchtower just keep, keep providing the goods. So we got a pre auth SQL injection to RCE in Fortinet fortaweb again. Get a proxy. Get a reverse proxy in front of that thing is my. Is my opinion. But yeah, walk us through this one, Adam, because you told me this one is proper comedy.
- [47:26]
Adam Boileau
It is proper comedy. So this is SQL injection in the auth header of this fortnet fortaweb product. So that's bad enough, right? Like literally it's the authorization header and you stick SQL injection into it and it runs queries in the database. So that's bad. What's even worse is that then MySQL the underlying database is running as root and Watchtower Labs have written up how to turn that onwards into CodExec using the classic select into Outfile SQL Injection where you can have a query write its response onto the local file system and they leverage that with some Python trickery to turn it into pre auth remote root codexec in your Fortinet security appliance. So good job, Fortinet. Good job.
- [48:15]
Patrick Gray
Well done, guys. We've also got and I. Anyone could have seen Blind Freddy could have seen this one coming, as they say. But there was the Citrix Net scalar flaw we spoke about a few weeks ago and I think my comment at the time was, yeah, that one's. That one's going to get a run. And apparently it is getting a run, according to cisa. So this story here is from David Jones at Cyber Security Dive and was published on July 11. I'm not seeing the news document that we work from filling up with links to stories about like this netscaler bug turning into a cyberpocalypse, but I don't know. There's still time.
- [48:50]
Adam Boileau
Yeah. And there's plenty of people out there who are getting known and I have seen, you know, some of the social media has been, you know, keeping an eye on the sort of people who are getting shelled. But this bug in particular, and the Citrix bleed bug that came before it, the real value in these is that they are also 2fa bypass because you're stealing session tokens for live sessions post authentic. So all the two FA in the world ain't gonna help you. And that's worth something.
- [49:17]
Patrick Gray
Yeah, yeah, it sure is. And we got a. Someone's written a backdoor for sonic wall devices, which of course, you know, everybody needs a backdoor for one of those for when you pop them through some really dumb bug in them. And Google, Google's Threat Intelligence Group has published some stuff on that.
- [49:35]
Adam Boileau
Yeah, they did a write up of it and this one really warms my heart because whoever wrote this particular back kind of did it in a very classic trad UNIX way. It's like a user mode rootkit which they trigger from inside the initrd, which is the RAM disk that it uses during boot up. So they backdoor a shared library in the init RAM disk. And then the way that you trigger this backdoor is the user mode rootkit hooks all of the network read and write functions and looks for magic strings being sent across the network. So then you literally show up talking to the web interface on a sonic wall and you can just send it commands like anywhere in the network message and it will get sniffed off the wire essentially by this rootkit and then used to trigger the backdoor functions which is, you know, it's not novel but it's just. It warms my heart that kids old school.
- [50:28]
Patrick Gray
It's nice to see it in the real world instead of someone just theorizing about it in a context talk, you know, it's nice to see people actually out there doing the thing.
- [50:36]
Adam Boileau
Exactly, exactly. So I don't know which, you know, which Chinese, you know, private sector contractor wrote this, but I appreciate you, I appreciate your work, sir.
- [50:46]
Patrick Gray
I wonder if one day it's going to be like after the Vietnam War, you know, how, how Americans went back to Vietnam to see the tunnels and to have a beer with the, the VC though I wonder if that's going to be us. One day we'll be over in Beijing sitting down having a beer with the mss, talking about, talking about their adventures in sonic worlds. This actually reminded me too about a talk you did years ago about writing user mode rootkits for UNIX systems where. Yeah, the whole purpose of your talk was like they won't let you do root, so you have to do everything in user space. Like here's how I do it. And it was a very funny talk. And you had a grab bag of party tricks that you showed.
- [51:19]
Adam Boileau
Yeah, that was my non root rootkits talk. Yeah, that was good fun. This is why I appreciate this kind of unix, Tom Fuldo.
- [51:27]
Patrick Gray
No, I get it. And I can tell that you're like, ah, I should have thought of that one.
- [51:31]
Adam Boileau
I mean, hey, we did this, we were doing this a long time ago in the unix. But it's just really nice that kids still do it. Yeah. Somewhere in, you know, in Hainan island somebody is doing the same thing and I was, yeah, I'm with you brother.
- [51:47]
Patrick Gray
Yeah, yeah. Let's see, let's see though I reckon one day the cyber tours where you get to, as an older gentleman or an older lady, get to tour through the cyber complex in China and meet with those people and you know, talk about how they got you. Now the last thing we're going to talk about today, it's a story, another one from 404 Media by Matthew Galt, which is talking about how hackers can remotely trigger the brakes on American trains and the problem has been ignored for years. Let me guess, this is something that's going to involve. If you have a software defined radio, you can send the magic packet that makes it do the thing, right?
- [52:19]
Adam Boileau
That is exactly what the story is. It turns out. Yes, there are these things they put on the back of trains that monitor it and can also remotely activate the brakes. A train is arbitrarily long as you add more, you know, carriages and things, more carts, trucks. What are they called?
- [52:34]
Patrick Gray
What's the cars?
- [52:35]
Adam Boileau
Cars, more cars to the train. So you know, they don't necessarily want to have to string wires along. So they have radios. Yay. And of course these systems were all designed when the idea that the general public could, would have access to flexible radios was not a threat model, it was not a thing anyone cared about. So yeah, much like so much of our industrialised world that relies on, you know, like 1970s, 1980s radio tech. If you've got a radio, you win a great many prizes. And the American like regulator or whatever for the railway industry has said that they're going to fix this at some point, but it's like billions of dollars worth of gear that needs to be replaced and it's hard not to read a story like this and go, you know, the fact that no one has been activating the brakes on these things meant that the cost of fixing this and doing it right in the 80s with, you know, processors that can't do crypto or whatever other mechanism, you know, whatever controls you would have put in place versus the risks, just not addressing this was probably actually the right choice from a risk management point of view. So. Yeah, well.
- [53:39]
Patrick Gray
And I think make train stop bad, make train go worse. Like if this was a make train go with packet, that's really bad. Make train stop, okay, Inconvenient, make train go kaboom. I think that's the, that's the thing.
- [53:58]
Adam Boileau
Oh, dear. Yeah, no, it's, it's, yeah, it's always fun reading these kinds of stories, you know, because they are, you know, you're torn between, is it junk hacking, is it real, is it risk management, is it business? And the answer is a. It's a little bit of all of them.
- [54:12]
Patrick Gray
Look, if you. I remember seeing a talk from Balance years ago on SDR stuff in Canberra and it is pretty terrifying once you realize like the breadth of this problem. Like it ain't just trains. Like there is so much stuff that you can mess with with SDR and it's kind of surprising that we haven't seen drama with it so far. But I mean, I guess the, you know, flip it, like a lot of Flipper zero stuff would be. That's where you see the mischief. But not people who are hoping the.
- [54:39]
Adam Boileau
Battery charge flap on Teslas with the Flipper.
- [54:41]
Patrick Gray
Yeah, yeah, exactly right. But you're not seeing, you're not seeing really, really serious stuff. Anyway, we've got to wrap it up there because we're over time. No surprises there because it's our first show in three weeks. But Adam Boileau, great to be back on deck. Great to chat to you, man. We're going to do it all again next week. Cheers.
- [54:55]
Adam Boileau
We certainly will, Pat. I will see you then.
- [55:07]
Patrick Gray
That was Adam Boileau with the check of the week's security news. It is time for this week's sponsor interview now and we're chatting with David Cottingham, who is a co founder and the chief executive of Airlock Digital. Airlock makes allow listing software that you can actually use at scale. They have customers with like 100,000 endpoints plus. And it works. It actually works and it's manageable and it's completely unlike trying to do it with the other tools that they are rapidly replacing in enterprise environments. So, yeah, I guess Airlock is a relatively new company. I mean it's under 10 years old. And as part of like becoming a serious business enterprise company, they had to actually work on creating a multi user, multi role console for allow Listing. Which makes sense because you've got different groups who need to do different things with allow listing. Like as you'll hear like it's the support people who might give someone an exception to run something one time that they nearly really need to run and it's the application people who might need to adjust the allow list because they're rolling out a new application and whatnot. So you know, where they started was just to have the one big console and everyone can log into it. But then as you start getting those really big customers, you have to start thinking more and more about like, well, how does this product have to work right, for different people in an organization? Now this sounds like it might actually be a dull conversation, but it's really not. It's an interesting one. Talking to Dave about bringing Airlock up, growing it up into a proper enterprise product with like a multi role console. So here is Dave Cottingham talking all about that. Enjoy.
- [56:44]
Dave Cottingham
So we actually recommend that you have the security people looking at security data, the people that are deploying applications looking at the app data and even though some people might laugh at it a little bit, the support team issuing exceptions. Right, because that creates the most logical sort of business engagement approach to getting apps deployed, getting exceptions out there and ensuring business continuity effectively. So what we have to do inside of the console is essentially for all of those sort of major, I guess, Persona roles, attach different privileges so you can access or invoke different parts of the product depending on whatever your role set is. And also then thinking about how different actions can be dangerous. We spend a lot of time thinking about how we prevent against someone unintentionally or intentionally configuring something in a bad way that, you know, depending on their access.
- [57:39]
Patrick Gray
But yeah, yeah, so that's a whole other part of this conversation. We'll get to that bit a little bit later on, which is, well, we're seeing now like, you know, these comm kids, the advanced persistent teenagers they're going and state actors as well. They'll go after things like EDR consoles, compromise an identity, switch off EDR or like make it blind or something and then onwards from there out to the endpoint. So I'm guessing you've had to put a bit of thought there, but like what was that transition like? I imagine for you it was like, yes, single console, single administrator. And then like, how do you begin to transition from that into these, these multiple roles and permissions? Because that's like, even if you want to add one different role, it's almost like a product rewrite of that part of the product to begin with, right?
- [58:24]
Dave Cottingham
Completely. And also, you know, the way our product worked a while ago was you sort of have this one object to many relationship that, for example, you could have a list of applications that you've trusted and it might be in multiple Windows servers and Windows workstations for example, and customers might want two different teams to administer both of it. However, you have like one application list that links to both. And so decoupling all of that was a huge amount of work in the product. So the first thing that a lot of vendors will do is they will just prevent you viewing those type of objects. Right. So they will say, okay, well this page won't render unless you actually pass this permissions check. But obviously that's quite weak. Right. So you sort of start from that visual part so people can't see it. And then you work backwards to, you know, what we call the application controllers underneath those pages, which is, okay, the controller needs to authorize the user and you know, what actions it's requesting based on what permissions it has. And then down below that, then there is the back end which is about what information am I actually processing and generating. And that needs to be user aware. So the first thing was actually making sure that each component has visibility of what user has taken a particular action and actually flowing that all through. Because when we designed it in the first place, it was sort of like, oh, the system did this because you build like a binary and it would just get parameters to do work, but it wouldn't be aware of like who's actually asking it to do said work. So it was a full approach of front end, middleware controllers and back end making it user aware and then making it permissions aware on top of that and then unpicking all the implications about. Well, hang on, this part is actually related to this part if you want to let's say approve an application and making sure that you can do enough but not sort of too much to influence things that you don't have access to.
- [60:34]
Patrick Gray
Yeah, so I imagine like the early stage of this when you start moving to like more of a multi user, multi role platform is like you fire up version one, you're like, great, everything's broken pretty much, right?
- [60:47]
Dave Cottingham
Completely, yeah. Or I log in and Nothing renders. And it's because there's one call which is down here, which is shared and then you have to work to split that out. I mean even like we're going through a ui, you know, rebuild in the next release of the product, we're in QA at the moment. And yeah, really it is, you know, every call, every, you know, rechecking all of those things meticulously in every single combination of user permission to just make sure that we're really blocking and tackling and having, you know, for lack of a better term, sort of a implicit deny approach to authorization in everything you do in the product.
- [61:31]
Patrick Gray
Yeah, so it's interesting what you said, right, because earlier you were talking about how users have sort of coalesced around these sort of pretty clearly defined roles, right? Like there's the applications person, there's the support person, when you were talking about exceptions too, for those who didn't follow. It's like if someone tries to execute something, they can't, the product blocks it. They can go to support and ask for like a one time code to be able to run it and whatever. And that's what, what a lot of customers want. So you talk about these predefined roles, but then you're talking about, well, all of these combinations and permutations of permissions. So I imagine that would have been a decision for you, which is like, do we have several pre canned roles with permission sets or do we let the users go in there and really mess with stuff? And it sounds like you went that route, but surely there's dragons there, right? In terms of unexpected combinations of permissions, unlocking potential for stuff to go wrong. Like how do you even begin testing all of those combinations and permutations to make sure that they work.
- [62:28]
Dave Cottingham
It's really leaning heavily on automation. As much as you cannot possibly throw people power at this. We've got an automation rig that runs for about 48 hours. It builds up a product, runs through all of those different permutations and then it will be not only sort of like a test harness, but it's functional testing that I think is the most important. So it's actually going in there, having a system, if you're automating it, click something and make sure that it can find the next thing that it actually needs to click on to run through that process.
- [63:06]
Patrick Gray
And then you want to see something actually pop out onto an endpoint, right?
- [63:09]
Dave Cottingham
Essentially, yeah. And then make sure that what you're doing in the console, then you've got to tie Automation in with the endpoint as well in order to functionally test that. We've really led a lot with functional testing, preferencing it over unit tests, even though both are important just because ultimately that sort of tests from the user perspective all the way down through the back end, what's generated and then back up to what pops out in the other end. It's sort of like a prove it approach through the whole stack. So there's a huge amount of complexity there. 1, 1 classic 1 was we have view roles and edit roles, but you can't edit without viewing. And it's kind of this silly thing where it's like, well hang on, you need both of these if you want to edit. And it's just lessons like that that you end up very easily unless you plan it out meticulously, backing yourself into a bit of tech debt that you need to engineer out of.
- [64:07]
Patrick Gray
Well, I mean this is why it's an interesting conversation, right, is because, you know, I've been interviewing you since you were a little itty bitty baby startup and you were not anymore. Right. And this is all part of that, that growing up experience.
- [64:19]
Dave Cottingham
Yeah, one, one was the REST API. So you know, obviously we've, you know, we've got a whole REST API and we try and make the REST API match the logged in interactive functionality as much as possible, but you don't want REST API keys floating around there. So we suddenly realized, well, hang on, if I get an API key, we had a role for Grant, a REST API key, but then once you had an API key you could sort of do anything. And then it's like, well hang on, we now need to put permissions on the REST API. So how do we best do that then? The easiest thing we did was we broke it down to like calling endpoints. So it's like you need to say which endpoints this key has access to and then it will either allow or deny access to an endpoint. But then it's like, okay, well if I, let's say have access to the policies endpoint, but I only want this REST API key for this user to see these policies, it starts to come back into that functional scope as well, where you need to flow it through not only REST API access, but backup into the application. And I think throughout the journey we've sort of learned that constraints are good and there are dragons when you try and satisfy every single use case that every customer wants.
- [65:40]
Patrick Gray
Well, that's kind of what I was asking about pre canned versus like ultimate flexibility. So have you sort of landed somewhere in between? Have you?
- [65:46]
Dave Cottingham
Yeah, we have. So we've got the REST API access based on Endpoint and the user roles interactively and now we're adding in a bit of object access on the REST API side just so we can more granularly sort of segment based on who needs to manage what.
- [66:05]
Patrick Gray
So it has happened that a customer said, oh, we would like to be able to do xyz. And you're like, how about no, Yes.
- [66:13]
Dave Cottingham
I want a REST API key just for this server. You know, and at the end of the day it comes down to, well, create a separate policy for that. And then, you know, there's a level of. You need to structure it in a certain way if you want the level of granularity that you looking for.
- [66:29]
Patrick Gray
Because, you know, yeah, learn the product, do it our way. It's better.
- [66:33]
Dave Cottingham
There are infinite edge cases and if you try and satisfy them all, you'll end up with this unwieldy monster that you need to manage.
- [66:40]
Patrick Gray
Yeah, well, it's how vendors fall. Right. Like it is actually. They just get overly complex and then someone comes along with the next simple thing and, you know, away you go and.
- [66:49]
Dave Cottingham
Exactly. Start again from a clean slate. Yeah.
- [66:51]
Patrick Gray
And given like a big premise of your product was to be the simple one. Yeah. I can understand why you didn't. Don't always want to go there. Now look, I did touch on it earlier, but how do you think about solving this problem of an identity compromise of an admin of your platform of your product? Right. So I spoke to Alex and Chris, oh Alex and Steve actually over at Sentinel One about, you know, what various threat actors would do when they tried to access Sentinel One consoles, you know, and you can't just turn off edr, right. Like if you've got access to that console because that will raise alarms, that will cause some, you know, bells to go off and whatnot. So there are some natural sort of defenses there that you can, that you can build in. But I'm curious how you tackled it as an allow listing company. Like what is it that you do to prevent a disaster when a threat actor turns up with valid credentials into your admin console?
- [67:50]
Dave Cottingham
So the first thing and you touched on it with EDR is visibility for the customer. So the customer needs to be able to really easily see what configuration changes have been made at any given time so that they can see, hang on, this thing was added at this time and you're showing them whenever a change is made. The interesting thing about our product is all the changes are driven by the users. So you go in, you want to allow something, it's you that needs to choose that it's not supplied by us as a vendor. It's like you define that trust. So really bringing forward what's being changed, when and hopefully why. If the administrator wants to fill in that detail, it will tip them off to, hey, these policies have actually changed. So, you know, letting them know is critical. The second one is understanding, especially in a deny by default scenario, about what dangerous sort of policy looks like. So when you're talking about, you know, implicitly not trusting anything, it's a dangerous tool potentially, because what you can do is you could say, oh, I want to block these system files, for example. So what we spend a lot of time on is doing things like let's boot a system, let's understand what the critical items are in actually to get this system up to a shell, get network access so it can get policies and putting detections in place so that if a policy is detected that the endpoint gets that doesn't have, you know, this sort of minimum required trust to allow a system to functionally operate, then what it will do is it will continue operating on the same policy that it had previously and then it will report safe mode.
- [69:43]
Patrick Gray
I'm guessing this is your post Crowdstrike disaster initiative.
- [69:49]
Dave Cottingham
Oh, I look, I think it's been in there for a few years, but we've definitely iterated on what that looks like.
- [70:00]
Patrick Gray
Definitely tighten that one up a little bit after that.
- [70:03]
Dave Cottingham
Yeah, yeah, it was quite simple. And then we sort of like, oh, hang on, what happens if this particular situation occurs and it's sort of like you need to build in the smart on the agent that you're influencing to either you can't have it blindly trust everything the server tells it to, even though the server is the authoritative kind of do what I say, please. You need to also make sure that the agent itself has a level of veto in place to say, no, I'm not doing that, because that's bad.
- [70:36]
Patrick Gray
And what about a malicious allow? So you've talked about someone accessing a console and doing a malicious denial of like a critical system DLL or something about the opposite case where someone wants to get some malware or like some RAT onto an allow list. Like, is there something you can build into the product that makes that more difficult for a rogue admin to do? Because I'd imagine that would be very, very hard.
- [71:00]
Dave Cottingham
Yes. And I guess it's sort of like it's easy when there's known malware that's out there because like we will, we've got a great partnership, the VirusTotal and we will basically flag like, hey, this malicious thing's been either added to your allow list or it's been detected as part of your trust set. Like you've seen something that's bad, that's alert you to that, but when it's stuff that you haven't seen before, I think again, it's back to the best defenses, visibility, seeing when an actual change has been made to add something and giving people visibility of that going forward. One thing that we're building is also segmented approvals of policy. So what you'll have is two different users where one person sets up the policy changes and another person has to go in and actually click accept on that.
- [71:44]
Patrick Gray
Yeah, so I've always wondered about like, yeah, dual key, like in the submarine movies, you know what I mean? The guy takes the key around from around his neck and the other one does and they have to turn him at the same time kind of thing. Yeah, that makes a lot of sense. Do many people actually bother with that though?
- [71:59]
Dave Cottingham
A lot of people want it in the Enterprise because, and it's interesting because they will say, okay, I will trust this team to allow things if I have oversight of it. So they'll want the security team to have oversight, but the application team to actually make the decisions. So they're just sitting there going, look, look, look, approve. You know, that's definitely needed.
- [72:20]
Patrick Gray
Last question is, you know, you keep talking about this visibility. How are people choosing to consume those changes? Is that like pumped out to a seam? Is it like a Slack alert? Like, how do people usually choose to get that information? Because it is quite low volume. Right, right.
- [72:34]
Dave Cottingham
It is, yeah. Especially when you're talking about the change. So, you know, either a push notification to Slack is a big one. The other one is seam alerts, as you said. And then third is definitely actually being in the console. I'm seeing a preference towards people actually wanting to sort of manage the product outside of the product, but that normally happens only for the large enterprise end of town, where they've really got a whole bunch of other systems that they're automating. So, you know, I think it's just about making sure that you have the integrations in the ecosystem for the tools that the customers have so you can can push those notifications and changes up, man.
- [73:07]
Patrick Gray
Awesome chat. I love talking to you about how Airlock's all grown up. It's really cool Great to see you, Dave. And I'll catch you again soon.
- [73:16]
Dave Cottingham
Thanks so much, Patrick. Cheers.
- [73:19]
Patrick Gray
That was Dave Cottingham there from Airlock Digital, a fine company that makes fine software. I do absolutely 100% recommend and improve endorse Airlock Digital. I am not an advisor to them. I just think they're really cool. And that is it for this week's show. I do hope you enjoyed it. I'll be back next week with more security news and analysis, but until then, I've been Patrick Gray. Thanks for listening.