Interviewer (4:34)

Well, you mentioned that it's typically a six week or so cycle. Can you walk us through that? I mean, what happens when these attackers start probing a previously unknown vulnerability?

Bob Rudis (4:46)

Yeah, so actually that's the really cool thing about this. So a lot of people are asking us just from within our data, like you have some older vulnerabilities, like we have some 27, we have vulns going all the way back. But for these enterprise technologies, There are some 2017 vulns in there, some 20, 20 vulns in there. And a lot of folks go, are attackers actually being successful with those vulnerabilities? And sure, I guess if they get lucky they might be able to compromise a host that's on the Internet now if it really hasn't been been patched that long. I know patching is not perfect at orgs, but Generally speaking a 2017 vuln is not going to be in too many enterprise perimeters. So when they're doing this scanning like our belief was, or like the CVE based scanning, like actually testing to see if the exploit works against the particular piece of technology, they're doing that as a cover because what they used to do like four or five. So like back when I was at Rapid seven and I was talking with Morris, like when he started gray noise, we both noticed the pattern of like opportunistic scanning going crazy. There were inventory scans happening all the time. And then as both like rapid and gray noise, we started to tell people about this opportunistic scanning, they kind of went silent for a while. And like they use other stealthier techniques to gather inventory what's on the Internet, but they still need to keep an inventory. So our assertion has been, well, they're doing this probing against older CVEs because orgs don't care about the older CVEs. Like I hear that from customers all the time, we don't care about old CVEs. So they're getting past their scanning defenses, they're ignoring what that signal is on the wire. So to just make sure, oh yeah, there is an Avanti there or there is a Fortinet there, or there is a sonic wall there or anything like that. And then they use that inventory to then do, to launch new attacks with a brand new CVE or an O day or an end day that might happen down the pike. So our hypothesis was again we don't attack people, so we don't know if exactly that's what the attackers are doing. But it just seems to be that they have a sense that there is a CVE coming or they're doing their own exploit creation and they're testing against old TVs to get under the wire, like to get under the radar of organizations. And then they use that to have a more precise inventory because they really want to be able to get as many hosts as possible.

Interviewer (6:57)

Well, how often did a spike in malicious activity lead to an actual cve and what's that timeframe look like?

Bob Rudis (7:06)

Yeah, so if you look at the report and I'm just going to scroll real Quick, just like I could give you some more, like some more concrete information. Yeah, so within there what, what looks, if you look at and this is like page for folks that are listening and you have the report in front of you, it's like page six of the report, it shows kind of the sequence of spike to cve. And roughly on there what you can see is there's 200ish events. I like to round because people like precise numbers and those 200 events are indicative of that within six week time frame. And it's a handful of CVEs so it's like, I think it's under 30 CVEs total for the whole thing. So that for all across those technologies there's at least one to three CVEs that came within that six week time period. That basically means the attackers were doing that scanning and knowing in advance from what we can tell that this new CVE was coming or they were the ones that were creating the exploit that created the cve.

Interviewer (8:03)

And what kind of technologies show the most reliable warning signals?

Bob Rudis (8:09)

Yeah, so that's a really important question too for orgs. So we looked at every single technology that we have in the edge category. And what we first did is we said, okay, we need to see what do the patterns of scanning and attacking look like for these technologies. So I'll give you one that we didn't put in the report and that we didn't analyze. It's like D Link and actually I'll throw linksys in there too. So D Link and Linksys are extremely noisy technologies from like an attacker perspective, like they're doing constant probing. It's almost like a heartbeat. So, and in the report, I think we actually use that terminology, we said, look, if there's a heartbeat, if it looks like some, if it looks like attackers are just going after a particular technology a whole lot, or if there are just scads of CVEs so that we couldn't find any signal in that noise, we ignored that. So D Link and Linksys and a bunch of other ones that fall into that category we basically threw away because there weren't. There was no real spike activity and there was just too much of noise to say that there's signal here. Which ended up leaving us with enterprise technologies, which I was actually surprised at. I actually thought maybe there wouldn't. I didn't actually realize there was that many CVEs for D link or Linksys or those things. I tracked this stuff but my gosh, there's a lot of vulnerabilities in those kits. And so for these enterprise technologies, we ended up looking across all of the edge enterprise technologies that are out there and we didn't deliberately land on, you know, Cisco, Fortinet, Juniper, Palo. I'll list the ones that have more signal in a second. Like we didn't deliberately pick those, they just happened to be the ones that bubbled up from the signal analysis that we did. So Cisco Kit has a, like I would say that it's an 80% confidence signal that you're going to see that pattern happen on there. But for like Fortinet and for Juniper and for Palo Alto and for Sonicwall and Avanti, those ones show a very high signal with this spike to CVE within six weeks. One that I really, I hate to say it this way because I just don't like Citrix, but I was hoping we would see it very concretely with Citrix and we just didn't see that firm of a signal with Citrix and a technology that I used to really not like a long time ago, but Mikrotik is getting a little bit better in their safety protocols. I thought we'd see a lot more signal in Microtik, but honestly attackers aren't even going after Microtik as often as they used to before, so we really didn't see a whole lot of signal there as well too. So again, the big players, Avanti, Fortinet, Juniper and Paolo, like the strongest signal, very high correlation. Cisco about 80% correlation. And then Citrix and Microtik, not a great correlation. I wouldn't be using their signal if I was a defender and I had to spin up more defenses or spin up more logging. I wouldn't choose to do that if I saw a spike on those.

Dave Bittner (11:07)

We'll be right back.

Ben Yellen (11:10)

I'm Ben Yellen, co host of the Caveat podcast. Each Thursday we sit down and talk about the biggest legal and policy developments affecting technology that are shaping our world. Whether it be sitting down with experts or government officials, or breaking down the latest political developments, we talk about the stories that will have tangible impacts on businesses and people around the world. If you are looking to stay informed on what is happening and how it could impact you, make sure to listen to the Caveat podcast.

Bob Rudis (11:42)

Are you feeling more fulfilled now that you're back to work? No, I need a vacation.

Dave Bittner (11:49)

See the movie that critics are saying.

Bob Rudis (11:50)

Is an awesome look at that crowd.

Dave Bittner (11:53)

Pleasing, fist pumping all out brawl of a film.

Bob Rudis (11:58)

You're right about that.

Interviewer (12:01)

They're coming after our family.

Bob Rudis (12:05)

Go Fix this. Oh my. Nobody 2 rated R only in theaters.

Navy Advertisement Voice (12:10)

Now Support for this podcast and the following message comes from America's Navy. The Navy offers new graduates hands on training and experience in careers like computer science, aviation and medicine, plus education and sign on bonuses. Parents, help your grads start their career today@navy.com.

Interviewer (12:35)

Can you help me understand the why here? Like, why are these attackers behaving in detectable ways weeks before a CVE is published?

Bob Rudis (12:47)

Well, so again, like it is detectable if you know what to look for. And most organizations are so overwhelmed with data hitting their perimeter, whether it's legitimate stuff from applications or whatever they host or attackers or scanners, there's just so much volume hitting an organization perimeter they have to throw away a bunch of data that they don't care about. So they will ignore that normal probes, lightweight probes or hits against CVEs that they know they've hatched. They will just ignore that activity because they feel like they've been able to defend against it. And honestly, they have to deal with all sorts of other things right now that they just can't prioritize that. And so attackers. It also looks like a couple things are happening with attackers. One, they either are the ones creating the exploits that they know they want to deploy, or they're following bug bounty hunters or other research teams or even working on at a vendor. I think if people just take a look a couple weeks ago, I believe it was like it was revealed that Microsoft sort of revealed to Chinese attackers that there was volumes in SharePoint which caused the SharePoint problem like, you know, just a couple weeks ago. So like vendors aren't necessarily as tight with their information as they should be. Bug bounty hunters aren't either. The bug bounty platforms might not be, or the attackers are just creating their own exploits for an eventual O day, which then will get a CVE almost immediately after the vendors or the researchers figure out what to do. And the scanning that they do, it's really for, it's an augmentation reason or they don't have resources. And I'll explain that, explain what I mean by that. So we saw starting around three years ago at Gray Noise that what we used to call inventory scans, basically you could count on within this couple days a month you'd see this botnet do an inventory scan of the Internet for this technology. And after enough of like enough orgs like Gray Noise started reporting on that and helping organizations defend against that, that inventory scanning activity went substantially down. And what they do instead is they will take any one of them. Like there's so many Internet scanning companies out there right now. So I don't want to mention, and I don't want to mention any one of them because it's not their fault that attackers do this. But you take any one of the benign good guy Internet scanners out there, they do that scanning for the organization so the organizations can know what's on the perimeter. Anybody can subscribe and get an account though. So if you want to get one into any one of those providers, Shodan census or whatever, and just say hey, show me where all the fortinets are, it'll do that. Because that's what the whole purpose of that database is. And so most of the time we divide attackers into three categories. A listers, B listers and C listers. So the A listers are like nation states and the C listers are like the people that just scrape the bottom of the barrel, rent some time on the Marai botnet and do what they will out there. So the A listers stop doing inventory scans, they just completely stop doing them. But every so often they do need to like augment the inventory that they have with census with something a little more recent or a census showdown. Whoever I'm not again, I'm not picking at any one of them because it's not their fault and they, they'll use that data for targeted scans. But if they really want to have their own inventory and update it and have like a really crisp like immediate thing so that they know they can use it within a certain period of time, they need to do their own scans and they can't do what they used to do, which was like just do random scans for technology or fingerprinting or whatever. So by launching a CVE based exploit or attempting to do so, making it look like they're idiots, that they don't know what they're doing, because why would you try to do that against that piece of technology? It gets under the radar of everybody and that gives them a really fresh crisp inventory. Now there's also a secondary thing where if an org hasn't patched a 2020 CVE or a 2022 CVE, they compromise that host as well too. They get two potential bangs for their buck. They know that there's a piece of technology that they want there and they may have been able to compromise it at the same time. So they get that win and they don't do them as regularly. So these spikes don't happen on a regular time period. If you look again back on page six of the report. Each different graph there has a different time frame and some of them span between January of this year to July of this year. Some go back to December or September of last year to now. Some are just more recent. So they don't do the inventory scans with a regularity like they used to do. They do these bursts to try to get under the radar and then, you know, use that data later on. For what? When they want to. When they want to have it, when they want, when they know a CV is going to happen or they create an exploit that causes the CV to happen.

Dave Bittner (17:20)

Hmm.

Interviewer (17:21)

Well, what are your recommendations then for defenders? What should they do when they spot one of these spikes?

Bob Rudis (17:29)

Yeah, so if I were back, I thankfully am not in a CISO position and I'm not. I don't have to do what real people have real work for a living have to do, which is like defend an org and protect orgs from their stuff because it's really hard. The one thing that like I do know is like if you need to do like extra logging, like either from a system log perspective, or let's say that you want to capture full PCAP or even partial sampled netflow, doing that for like all year across all sorts of devices and technologies is you just can't. Even the Fortune 1000, the Fortune 100 can't do that for all those kinds of technologies that are out there. So if you see a spike happen against one of these older CVEs or just like one of the scanners, because a couple of our scanner tags, just the vanilla scanner tags, also saw some spikes. If you see that in your logs and that you should be able to take time for, get that into some big data system and be able to do spike detection like we didn't do. This wasn't rocket science. This was basic, just core data science statistics that we did on here it is, you can do that, you'll get that notification and that should be a signal that you should flip on maybe some netflow logging, flip on like full PCAPs, at least partial the time for you for different days, or get full system logging going into your expensive splunk data warehouse or whatever so that you have that to look for to see what might be coming against your environment down the road. And that six week cutoff is actually a pretty decent amount of time to be, you know, to get though that logging set up, to get that extra data capture set up and be watching for what happens with that extra data capture. And I think if orgs could do that they, they would be in a better position to see new weird activity happening against them. Or if a CVE comes out within six weeks, they could also plan with their operations team to make sure, hey, can we make sure we patch that like if a CVE comes out within six weeks, can we make sure that there's the emergency patch procedures ready so we can get those patches in and therefore not let whatever those brand new like end day attacks that are going to happen against those CV be successful against that organization. So we wrote it really for those two things in mind to give you the ability to like prepare your defenses and logs and just watch for what's happening a lot closer so you don't like waste a lot of money to do that and resources. Or on the flip side coordinate with all your internal teams that you have to. Because taking, you know, taking down a Cisco firewall or router, taking down a Fortinet appliance, I can get on the list. You are taking that down like it's. There is downtime. You might not have an HA set up or high availability setup. So if you take that down it causes real downtime for real people. It's a real business thing. But if you can coordinate it ahead of time or know in advance, you have to watch for these things. So you can like block IPs on your own if you want to do that. Or watch what we have over here that gives you like real power to maybe not be in that first round of attacks that attackers are doing. And again it does require some resources. It requires a different frame of mind for the teams. You might actually have to get some folks that know how to do the logging so you can do the time series analysis. But honestly it's a small investment for them to do that to potentially save themselves for a really bad attack. Because it just seems lately that CVEs come out or an attacker will launch an O day, a CVV will happen right after that and everyone's taken by surprise and they just don't have any time to react. We were hoping with this analysis to give them some tools that they can use to be a bit more prepared. If you have the ability and time and resources to kind of do that kind of analysis.

Interviewer (20:53)

What about false positives? I mean were there any surprises where a spike didn't result in a new CVE down the line?

Bob Rudis (21:01)

That's a wonderful question. And we chose to not put the graph in the report cause it was a lot more complex. So there were ones that fell outside of that window. But it was like in the 20% range of the spikes that fell outside the six week and didn't fit into there. And they weren't with the technologies that we show in the report. What we chose to put in the report were the technologies where we saw these spikes. There was the propensity of them there. And we included, like I said, the Citrix one and the microtik one to show. Yeah, and even Cisco it's only 80% of a signal. So we did show that. So you could see. Yeah, we aren't saying that this is a complete predictor. There isn't solid 100% correlation across all technologies for this. But there's enough of a correlation for it for certain types of technologies that you can use this to your advantage to be prepared. Like there's actually one graph in the report I want to see. It's. Yeah, so it's on page 10 of the report or sorry page 9 of the report and there's like a, there's like an area graph above what looks like something shooting like little dots out. There's a lot of dots after this. There's not a lot of dots after the six week line, but there are dots after the six week line which means that that spike to CVE correlation wasn't within that six week time frame. And that's really important for folks to realize, look at this, choose to set up your defenses if you want to this way. But there is a huge correlation for again a good number of technologies that it makes a lot of sense for people to do that.

Interviewer (22:29)

So what are your recommendations for folks who want to get started taking a look at this? What's the best way to check this out?

Bob Rudis (22:38)

So most folks do have some type of detection capabilities within their perimeter. I know some of the organizations that just can't afford a whole lot of stuff don't have that. But the ones that if you can afford a decent sized Fortinet or a decent size Ivanti or Cisco, like you do have some capital expense money in your security budget if they're ignoring older vulnerabilities because you believe you've patched them all. I guess the first thing I would say is at least take a look at the CVEs that we noted inside ours, our report for those those and have your detections log the hits to those CVEs and then basically take those things, put them in some kind of. It doesn't need to be like an, or a gigantic Oracle database. You can get SQLite. And they just shove this data into like a small database and just regularly run checks against the spikes with again, the equations that we provided to you. It literally is that simple, is like, hey, log these events, check it for the spike events and then you do your own comparisons. And what's been really interesting about that, actually I was going to say that earlier, so I had no idea how this was going to. We don't pre release reports to everybody, we just release it to everybody at the same time. And I got so many responses from other researchers and other defenders. So we had one person tell us, yeah, I was in my DFIR community, Slack, I have no idea which one it was. And they were going, we all had this gut call that this felt like it was happening too. But none of us could justify the time to go do that log collection or event collection the correlation and do that. And a whole bunch of them are now going to go do that with their own stuff because again, this is hitting your perimeter just like it's hitting our stuff. So I would say there is definitely value there. A lot of other folks have said, wow, yeah, this tracks like we felt this too. And I'm glad someone did some data behind it. And my hope is a lot more folks do take that time and kind of double check our work basically, see, do what we've done against your stuff. If you can go back in time and do that, if you were logging stuff before, that'd be amazing too because I'd like to know that the correlation, the signal is even stronger than we've already shown for the data that we have. And we're going to be continuing to update this and revisit it over time so we can tell people, yeah, like the signal is still there, the correlation is still there. Although by doing what we've done, we may have told the attackers that we know what they're doing and they may stop what they're doing. I have no idea. It's one of those things like the minute you tell somebody that you see this, they might stop doing what? It's really frustrating when they do that.

Interviewer (25:06)

At the risk of being cute, you know, it reminds me of, I think it was an old Bob Seger lyric. I saw the lightning and waited on the thunder. You know, the spikes are the lightning and the CVE is the thunder.

Bob Rudis (25:19)

Wow, I wish you could. Wow, that would have been amazing for the report. That would have been so amazing. Well, feel free to use it if.

Interviewer (25:27)

You ever do a TED Talk on it or something like that. Right.

Bob Rudis (25:31)

Oh man, oh man, I don't know.

Interviewer (25:35)

Well, Bob, I think I have everything I need for our story here. Is there anything I missed? Anything I haven't asked you that you think it's important to share?

Bob Rudis (25:44)

I think maybe the one thing I would just say is like, I know organizations had to be focused on the core defender operations that they have to go do, but they're sitting on a treasure trove of data that if they could allocate just some time to go ask questions of their own data. Like, not like, I work for a vendor. So like this vendor is not going to be happy. I say, what? Well, actually I think they'll be fine with it. You have your own treasure trove of data and it's like the tools to do data science, just basic stuff on that data exist like all over. They're all free too. Like the vast majority are free. And if you can justify or like find some way to justify carving out some time to just start asking questions of your data. Like for all the folks that are listening to this that had that same gut call, you thought, yeah, this, this feels like this is what happens. And you've had that thought for the same couple of years that we've had, go talk to your boss and say, hey, can we carve out like three hours a week to go do this kind of stuff on our own data? And I think you're all going to be surprised at what you'll find if you could actually go do that. And you know, not relying on anybody else except yourself to see that kind of information. I think, I think you can develop some pretty powerful in house, in house techniques and in house tools to repeat this and find more and new and novel stuff that people aren't looking for right now.

Interviewer (26:58)

It also strikes me that, you know, in this age of AI and automation that there is still a really important place for that gut feeling to chase down that gut feeling.

Bob Rudis (27:10)

Oh, 100%. Like the. So the one thing that's really important is you're always supposed to start any type of analysis or process or whatever. Like when doing this kind of work with a hypothesis. And the only way you're going to get a hypothesis is if you're thinking about stuff and generally it's speaking insecurity. Everything starts with a gut call because you're just, you have this like Spidey sense that this is happening and this is happening. I need to now somehow determine whether my hypothesis is true. And most people just don't have the time to actually act upon that and again, I would just, I would just urge folks, if you have that gut call, find some way to get someone to give you the time. The resources are pretty much free. And go do that because I think you'll find a lot more than you're finding right now.

Dave Bittner (28:02)

Our thanks to Bob Rudis from Gray Noise for joining us. The research is titled Early Warning Signals When Attacker Behavior Precedes New Vulnerabilities. We'll have a link in the show notes. And that's Research Saturday, brought to you by N2K CyberWire. We'd love to hear from you. We're conducting our annual audience survey to learn more about our listeners. We're collecting your insights through the end of August. There's a link in the show notes. Please do check it out. This episode was produced by Liz Stokes. We're mixed by Elliot Peltzman and Trey Hester. Our executive producer is Jennifer Ibin. Peter Kilpe is our publisher. And I'm Dave Bittner. Thanks for listening. We'll see you back here next time.

Navy Advertisement Voice (29:01)

You say you'll never join the Navy, that you never track storms brewing in the Atlantic.

Bob Rudis (29:08)

And skydiving could never be.

Navy Advertisement Voice (29:10)

Part of your commute. You'd never climb Mount Fuji on a.

Bob Rudis (29:15)

Port visit or fly so fast you break the sound barrier.

Navy Advertisement Voice (29:21)

Joining the Navy sounds crazy. Saying never actually is. Start your journey@navy.com, america's Navy forged by the sea.