BHIS PODCAST: Network Threat Hunting Runbook

Summary

Podcast Summary: Talkin' Bout [Infosec] News — BHIS PODCAST: Network Threat Hunting Runbook
Host: Black Hills Information Security
Featured Experts: John Strand, Chris Brenton
Date: February 28, 2019

---

Overview

This episode dives deep into the "Network Threat Hunting Runbook", demystifying how to effectively conduct network-based threat hunts. Chris Brenton (of Active Countermeasures, self-described as "Chief Troublemaker") and John Strand walk listeners through a structured, step-by-step process for identifying and triaging compromised hosts using open-source tools like RITA and Zeek (formerly BRO), focusing less on definitions and more on hands-on techniques and decision-making.

The discussion covers:

• Why and how to hunt threats on the network level
• A practical runbook for threat hunting
• Key tools, pitfalls, and methodology
• Weighting and scoring of threat signals
• Filtering and investigating connections
• Fielding listener Q&A on tooling, resources, and advanced threat hunting scenarios

---

Key Discussion Points and Insights

1. Why Network Threat Hunting?

• Purpose: Even the best host monitoring can be evaded; the network acts as the "great equalizer" for finding compromised systems.
• Approach: Seek persistent, suspicious communication that stands out from legitimate traffic.

"Attackers are just really good at hiding their tracks... The network's the great equalizer there." — Chris Brenton [01:54]

---

2. The Network Threat Hunting Runbook

High-Level Steps:

1. Identify Persistent Communication Channels:
   Internal hosts with long-standing or repeatedly-open connections to external IPs.
2. Internal Reconnaissance:
   Investigate what the internal system is, looking for legitimate explanations.
3. Protocol Analysis:
   Examine whether expected protocols are being used as anticipated on correct ports.
4. Reputation Analysis (Last):
   Only after technical analysis, check reputation/blacklists for further context.
5. Decide & Disposition:
   Make a black/white decision: is it malicious? Whitelist if safe; begin IR if not.

"Why are we going to do this in this order? … Identify persistent systems, do recon, protocol analysis, reputation..." — Chris Brenton [01:56]

---

Step 1: Identifying Persistent Connections [06:14]

• Analyze for:
  • Persistent/Open Connections: E.g., VPNs, crypto-miners (which keep connections open for extended periods).
  • Beaconing: Regular connections re-established over set intervals.
• Tools:
  • RITA: Free, open-source, designed to detect such patterns.
  • Firewall state tables: Sometimes useful, but not always available/automatable.
• Pitfalls: Long connections aren’t always malicious (e.g., BGP, HQ-VPNs).

"When you start getting into multiple days, be nervous." — Chris Brenton [07:15]

“Crypto-mining … that's like the number one way people are getting whacked is for crypto mining.” — Chris Brenton [09:14]

---

Step 2: Beaconing, Obfuscation, and Signal Analysis [09:14]

• Attackers balance stealth vs. functionality. Very infrequent beacons are stealthy but not practical.
• RITA looks for beaconing with analysis over 24-hour blocks, not just per-minute.
• Padding/Jitter: Advanced attackers randomize timing and data size—look for averages and patterns over long durations.

“If an attacker gets malware on your network, they want to make sure they can get done … before it actually gets detected.” — Chris Brenton [09:27]

---

Step 3: Recon on Internal Hosts [14:30]

• Identify what device is communicating: Is it an HVAC system? Security software? Is the behavior expected?
• Importance of environment baselining and crowd-sourcing for context.

“Once we figured out it was an H Vac device … this is probably not a backdoor. This is a quote unquote feature. It's just crummy code.” — Chris Brenton [15:35]

---

Step 4: Protocol Analysis [16:24]

• Protocol matching: Does port match expected protocol? E.g., HTTPS on 443 should show SSL/TLS handshake.
• Zeek/BRO: Identifies application-layer protocols.
• Common false positives: NTP, DNS (due to regular, frequent queries in some environments).
• User agents and SSL hellos can be used to detect outliers in homogenous environments.

“If it's TCP 80, that should be web... If I'm looking at TCP 80 and I see SSH version, blah blah blah go by — something's up” — Chris Brenton [21:33]

---

Step 5: DNS Abuse Patterns [23:00]

• DNS as Command & Control:
  • C2 can be hidden in TXT records, AAAA IPv6 queries, or domain keys.
  • Look for domains with excessive FQDNs (>500-1000), especially from unknown vendors.

“Once you start getting up above 500, and certainly once you get up above like 800 or a thousand now, things get a little wonky.” — Chris Brenton [26:19]

---

Step 6: External Recon and Reputation [29:20]

• Use DIG, WHOIS, reverse DNS, and online IP reputation services.
• Blacklists are noisy and easily poisoned:
  • Web crawlers (like Bing/Google) often end up blacklisted incorrectly.
  • Use reputation as a minor component in threat scoring.

“It's a lot easier to get on a blacklist than it is to get back off it.” — Chris Brenton [31:00]

---

Step 7: Disposition and Continuous Improvement

• Decision: Compromised or benign?
  • Benign = Whitelist and move on
  • Malicious = Incident handling/forensics
  • Uncertain = Collect more detailed telemetry
• Repeat process; as whitelist grows, future hunts become easier and more focused.

“If you do that, this makes threat hunting a continuously improving process.” — Chris Brenton [33:27]

---

3. Scoring and Weighting Threats [03:34]

• Persistence scores high (60-80/100 points), protocol matches add more, reputational is a minor component (<15 points).
• Combine to reach a clear action.

---

4. Fielding Audience Questions & Notable Discussions

Hunt Teaming Trends [36:40]

• Demand for threat hunting growing due to "gap in the middle" between prevention and forensics.
• High-profile breaches like Starwood and Heartland exemplify failures of existing tooling to detect persistent attackers.

"This is the big gaping hole in our security..." — Chris Brenton [36:44]

---

Tooling Q&A

• RITA vs. AI Hunter: Commercial AI Hunter automates more, including ASN/IP whitelist management.
• Open-source RITA: Can script around its outputs for greater flexibility.
• Compatibility with NetFlow (v5/v9/IPFix), PCAPs, and on-prem/cloud deployments.
• Hardware: Bro/Zeek and RITA can run together; critical to have proper resources (RAM, disk I/O).
• Scaling: Bro/Zeek perf via PF_RING or AF_Packet; recommendations vary by network speed.
• CDN/Domain Fronting: Still detectable via traffic patterns, size, timing dispersion.

“Our solemn vow at Active Countermeasures is to make sure we're cheaper than the sales tax of Protect Wise and Dark Trace.” — John Strand [39:00]

---

Detection Without SSL Inspection [44:25]

• Attackers can nest encryption; deep packet inspection is not a silver bullet.
• RETA looks at communication structure, not packet contents.

“We stepped away from trying to look deep into the packet and try to get it to tell us its secrets. Because, honestly, you'll never get any answers.” — John Strand [44:25]

---

Practical Deployment

• On-prem and cloud supported; you keep all your data.
• Emphasis on whitelisting legitimate activity early, so SIEMs like Splunk are not overburdened.

---

Notable Quotes

• "Every time you see something unique [like a user agent] — that's interesting. But only if it's on a persistent connection." — Chris Brenton [21:33]
• "If you see a lot of traffic to a domain less than seven days old — that's something to go in and pay attention to." — Chris Brenton [29:00]
• "As Chris mentioned, the really, really smart people, they cringe every time I use this example..." — John Strand on beacon analysis [43:27]
• "Has anyone ever implemented RITA or AI Hunter and not found Beacon? Yes, when it doesn’t work." — John Strand [44:57]

---

Timestamps for Key Segments

• [01:46] Main runbook process introduction (Chris Brenton)
• [03:34] Scoring/weighting threats; blacklist caveats
• [06:14] Persistent connections, beaconing, and detection techniques
• [14:30] Internal reconnaissance; host investigation
• [16:24] Protocol analysis and port matching
• [23:00] DNS signals and anomaly detection
• [29:20] Reputation analysis approach
• [33:27] Disposition and continuous improvement
• [36:40] Audience Q&A (trend of threat hunting, tool questions)
• [44:25] Effectiveness without SSL inspection
• [49:56] System requirements and practical deployment

---

Memorable Moments

• Chris' quip about user agents:
  "We were told, yeah, check your user agents. This is awesome. You can catch all sorts of backdoors. Just look for this specific agent field... that's true, but hear me out..." [21:33]

• On DNS C2: "R1x.com: 63,332 hostnames exposed to the Internet. Something's wrong there." [26:40]

• Distinction on blacklisting: "It's a lot easier to get on a blacklist than it is to get back off." [31:00]

• Lighthearted banter: "Are you a Patriots fan? Because you sound totally wicked, believe it or not. Florida." — [36:15]

---

Conclusion

This episode offers a practical, detailed walkthrough of network-based threat hunting, focused on what to look for, how to filter and reduce signals, and which steps matter most. The advice centers on actionable strategies, not generic frameworks, empowering listeners with both the logic and the tools needed for day-to-day threat hunting at any scale. The extensive Q&A session reflects a thriving, inquisitive community and both hosts' commitment to demystifying security operations.

---

For additional resources or to find the complete episode (including supporting slides), visit:

• Black Hills Information Security YouTube
• Active Countermeasures

Transcript

[A]

Hello from Spearfish, South Dakota. It's the Active Countermeasures podcast. This is the podcast version of our webcast, so the slides we might reference will be missing, but you can find the whole episode on our YouTube page. This is Network Threat Hunting Runbook with John Strand and Chris Brenton. Enjoy.

[B]

Well, everybody, thank you very much for joining today's webcast. With me is Chris Brenton. If you guys are not aware, Chris Brenton is a business partner of Paul and I's with Active Countermeasures. He is the. Can we call you CEO, but since you boss me around, I guess we'll call you the CEO or whatever of Active Countermeasures Troublemaker.

[C]

I'll go with that.

[B]

Chief Troublemaker. We'll go. We'll stick with that and we'll move on. And he is still, to this day one of the few people that walks in a room. And I get nervous because he was my first sans instructor and I keep telling the story. I'm not going to say it, say that anymore. But he is going to be talking about, you know, Network Threat Hunting Runbook. And we're going to be focusing today on tools that we have created like Rita and we're going to talk about BRO and Zeek and all that stuff. Presentation is brought to you today by Black Hills Information Security. If you all need a pen test, a web application assessment, a red team command and control pivot test, whatever it is you're looking for, maybe even a hug, please reach out to Black Hills Information Security because that's where we do all of these different things. If you want to, just shoot us an email at consultinglackhillsinfosec. We'll get in touch with you very, very, very, very quickly. And it is also brought to you by Active Countermeasures. Right, Active Countermeasures as well. And that's really what we're going to be talking about quite a bit, is that network threat hunting side of the house today. With that, I'm going to hand it over to Chris Brenton.

[C]

Thanks, dude. Today we're going to talk about threat hunting, but not in general terms because I think we all know what threat hunting is. Now. The tough part is figuring out how do I do it. You know, in other words, the. Yeah, hunt the network. Yeah, okay, we all know that, but so what are the steps? This gets really convoluted. It hasn't been around long enough that we have really well defined processes. Everybody kind of does their own thing. So what I wanted to kind of talk through Is what, you know, we had active countermeasures and John and you know, some other folks we know have been using his process. That seems to work pretty well for us. Why threat hunt the network? You can read the slide. I don't need to read this to you. Bottom line is we're looking at even the playing field. You know, attackers are just really good at hiding their TR tracks. If you're trying to find them on the host or within the host logs. The network's the great equalizer there. So if you can get rid of the cruff on the network, it's a whole lot easier to focus in on them and what's going on. And from a high level perspective, this is what it looks like real briefly. Start off identifying communication channels, meaning I want to go in and I want to look for internal IP addresses that seem to be carrying on persistent connections without external IP addresses. What does that mean? We'll get in and talk about it a little bit more in just a bit. Identified that once you've identified. Okay, I've got an internal system talking to someone persistently externally. Identify what's the internal host. Is it normal for them start analyzing the protocol. Does this make sense? Scrutinize the reputation of it. There's a reason I'm doing that last. I'll loop back into that later. And then finally we need to figure out, okay, so what are we going to do about this? Those are the steps in a nutshell. So, hey, we're done. No, I'm kidding.

So threat weighting. What I mean by this is I've seen a lot of folks have different processes to go after this. So for example, one of the big ones I see is people check blacklists. Oh, if somebody was talking to someone on a blacklist, that means that it must be something suspicious. Not necessarily. Keep in mind that blacklists are for the most part, anybody can contribute to them. And people don't always necessarily have a high level of accuracy when they're looking at things. I've seen examples where, you know, like Google web crawling, people want to ban that. They want to put that on blacklist because, oh my God, it keeps hitting my website over and over again and what are they doing? And they're running all my JavaScripts. They must be evil. They ran every JavaScript on my system. Well, yeah, that's what a web crawler does. So that doesn't mean Google web crawlers evil. It just means that's its process in life. So blacklists to me are reputational. I Kind of give that a minor think about this from like threat score thing that we keep talking about where, you know, zero means I'm 100% certain it's safe. 100 means, yeah, we got a problem, we need to go in and deal with your. Is it a persistent communication yes or no? That should be worth about 60, 70 points on that thread scale. You know, our idea is to get to zero or to get to 100. If we get to either one of those, we can disposition it, have a, you know, clear sense that yes, we've done exactly the right thing and we're in good shape. I assign anywhere from 60, 80 points if I'm finding a persistent signal, the protocol, maybe that's worth another 20 points. Reputation, you know, you're in like the 10, 15 point range and then once you start adding subtracting points in, you kind of figure out where you fall in that range. And that kind of helps you, helps drive what you do from there.

[B]

And this also, I think kind of underlines the problem with like the blacklist approach, especially with threat hunting or not threat hunting. With threat intelligence, you have a lot of people that say, well, we're going to get a threat intelligence feed and we're just going to use it for the IP blacklist. And that's garbage. And one of the things that surprised us years ago was whenever we just looked at that blacklist. It is very, very, very, very common for internal computer systems to communicate out to a blacklist, but in very short bursts. It could be an advertisement that loads. But malware didn't trigger. That's usually what it actually is. So there's so much more to it, as Chris was saying, for that waiting. And a lot of this Chris, it's kind of funny, you know, a lot of stuff you're talking about, it was interesting more to me, I think, than to you and Bill. See how real networks looked whenever we're looking at that blacklist, when we're looking at persistent connections, because they look very fundamentally different than what most people would think they would look like.

[C]

Yep, absolutely. So step one, is there a constant communication session taking or is there a communication session taking place repeatedly between your internal IP address and an external IP address? This can take on a number of different forms. The easiest is a persistent connection, meaning that your internal system just calls out to some host out on the Internet, leaves that connection open 24 7, or at least until you know, something comes along and kills it, in which case it just goes in and re establishes it again, but it stays Open all the time. The other possibility is that it creates a connection, transmits some data, kills the connection, reestablishes that connection again at some later point. And there's challenges for going in and kind of finding each of these. The continuous connection. One of the toughest ones I've seen is that a lot of devices don't log. This took place until the session actually gets closed, meaning that someone opens a connection, if it doesn't die for a week, they're going to have access to your network for a week. Before you can actually see what's going on there, you need to be able to kind of get around that beaconing. They can go in, they can vary the timing, they can vary the padding. I'll talk a little bit more about that. And then there's. Which are like beacons, but they're just so goddamn noisy it can't be somebody trying to hide their tracks. So let me kind of dig into these a little bit more. So I talked about long connections. That can be a problem. The common place I see people do is they go to the firewall state table. So they go in, they do a dump of their state table to see which state entries have existed the longest. You can do that, it's hard to automate it and not every firewall will actually give you that information. But a really cool way to get that is with reta. RETIR is our open source free tool. So I'm not trying to make a buck off anybody here. You can just grab us off of reta. But RETA has the long connections option. And when you go in and you run that, it automatically lists out what are the longest taking place connections on your network. When you start getting into multiple days, be nervous. Now there are potential false positives here. BGP is a good example, VPNs are a good example. But if you're monitoring inside your internal interface of your firewall, chances are you're not going to be seeing that traffic that much. John, I don't know about you. I have not seen one of these as a backdoor in the wild. What we have seen.

[B]

Yeah, long connection metasploits Meterpreter has one specific communication profile where to get around beaconing detection. They nail open the connection and they leave it open. Which is weird because that's the way it did way, way, way back in the day. And the metasploit three ways. And then the other thing, it's.

[C]

I don't know, I think Cobalt Strike has that option too now.

[B]

Yep, it's one of those, it's one of those C2 profiles. The other one that we can get into debates as to whether or not it's, it's malicious, but it's definitely unwanted was the crypto mining. Cryptocurrency miners, they tend to establish a connection and keep it open for long periods of time.

[C]

Yeah, and that's really hot right now. That, that's like the number one way people are getting whacked is for crypto mining. So definitely worth going in and looking for. I think we've also found people circumventing proxy checks that try to keep you from going to sports sites and other nefarious things. They figured out, hey, I'll just SSH back to a proxy on my home system, on my Comcast network. Tag these with radar as well. Just look for anything that's been open for like five hours or more. So again, long connections, definitely something that's worth going in and paying attention to. So again, we're in that first step. We want to be able to identify is there a persistent connection between these systems. So if I look at that first line, this has been open for at least a day because I'm looking at a day's worth of data. So now I would start executing on steps 2 through 5 for that source IP address that I see there now. Beaconing. I want to kind of clarify this a little bit because you're going to hear people say, oh, beacons, you know, I can definitely write a backdoor that nobody can detect. And yeah, you can, you know, beacon once a week. No one's, there's no tool out there that's going to detect that. The problem is if you're beaconing once a week, you can send a command and get response back every two weeks. That's not a very functional backdoor. There's kind of a timing balance here where if an attacker gets malware on your network, they want to make sure they can get done whatever it is they want to do before it actually gets detected by some host software. So they don't want to be waiting two weeks just to finish one command because they're going to get whacked before they can get anything done. So timing needs to be a little bit lower. You know, we have seen down in the range of like 15 minutes to an hour and some jitter being introduced to be able to kind of vary the timing back and forth. Read is designed to process data in 24 hour chunks. And that's why there are some tools that try to identify beacons but most of them look at 60 seconds, meaning if you don't have multiple signals in a one minute period of time, they're not going to know it's a beacon. By looking at a 24 hour block of information, we're more likely to be able to catch what's going on. Now, could we catch everything? Well, no. Again, if you're beaconing once a week, we'll miss that. But again, as far as it's like being able to catch what's there, it's a pretty useful tool for that. Also padding, meaning that you can look for repeated patterns in the timing. You can also look for repeated patterns in the data size. That's a really useful one, especially when you start looking at people trying to use Office365 or Google as, you know, email backdoor. If you look at the size of the sessions, that could be a really clear indicator that you've got something beaconing. If it's a consistent size all the time. Now attackers will try and pad it to try and throw off the fact that it's a consistent size. But again, you can only pad it so much. If I pat it too much, I'm going to now start getting myself into a boat where you don't need a tool like Reader or anything else to detect that traffic. All you need to do is look at your top talkers on the network and they want to stay below that radar. So again, this whole thing is kind of a balancing act. There's the theoretical can you write a beacon that no one can detect? Of course you can. But would it be functional? No, not really. So again, we want to kind of look for groups of signals that are taking place over a long period of time. The one plug I'm going to give in the slides for our commercial tool, AI Hunter. These, these screen caps are for that because this is the only place I could find something like this. But if you look at the graph on the bottom, so this is a 24 hour period of time. Each spike, the Y axis is identifying how many signals was seen between two IP addresses in that one hour block of time. And if we take the data in one hour chunks and kind of spread it out, you'll notice where I've drawn that little red line. It's almost a flat line. That's a clear indication you've got something beaconing on the network. What's kind of cool about this is you can have something that is jittering its time. So I might say, I might be running Cobalt Strike and I Might be saying beacon once per minute, plus or minus 25%. Okay, so now I'm gonna begin 45 seconds, a minute, 15 seconds. But when you start averaging that out in one hour chunks, it all kind of averages out the same and you end up with something like this. So again, our first step, is there continuous communications or is there a persistent connection between an internal IP and an external ip? There's that continuous connection. There's also beacon signals, something like this. And again, this is something that Rita goes through and analyzes pretty easily. So this circled area on the left, this is a score we go through and identify for it. It's on a scale of 0 to 1. So anything close to a 1, we're pretty certain that's a beacon. And this second red circle here, the 108858, that's the number of beacons that we saw over a 24 hour period of time. When you think about the protocols that tend to generate false positives, meaning they're a be, but it's not actually something you need to be concerned about. The number one protocol people see a lot is NTP and network time protocol. Because if you think about the way that works, depending upon the operating system you're running, every 15 minutes to 30 minutes it's going to send out a fixed size message to a specific IP address which is asking what time is it? And it's going to get a fixed size reply back, which is here's what the current time is. That's going to happen on a very regular basis. Well, so we'll see about a hundred of those a day. So a beacon that's down in that low range, it could be somebody being stealthy, it could be just the protocol itself. That's why we do a protocol analysis. And I'll get into that a little bit. But anytime you start seeing thousands, like tens of thousands plus, oh yeah, that's something you definitely want to go in and pay attention to. So you can, you know, so I'm showing Rita here. Can you go through and do an analysis on this yourself? Yeah, there are some stat tools on the Linux command line that you could go in and you could kind of start processing the data and trying to chunk it yourself to kind of figure this stuff out. But quite honestly, this is what Rita was designed for. It'll just do it for you. And again, it's free, it's open source, you can get it with security onion. This is just the, the easiest way to kind of get all this done. So let's say we go through and we've identified that there's either a continuous connection taking place or I am reasonably certain there's a beacon signal going between my internal IP and an external ip. So anything that doesn't fit that criteria, it's not a current connection or it's not a beacon, get rid of it, get it out of my way. So I've now removed, you know, let's say I've got a million, you know, different connections that took place yesterday. I may be able to get rid of 900,000 of them right off the bat, not have to worry about them anymore because I'm focusing on continuous connections only and beacons only. And that's it. So once I have that action item list, it falls into one of those two categories. The next thing I want to do is I want to do some recon on the internal IP address. What is it? You know, that may help me kind of figure out what's going on here. We had a customer that, that was seeing like 1 million connections a day from an internal IP address, from a couple of different internal IP addresses to an external IP and it turned out to be their H Vac system. My guess is it was crummy code. My guess is somebody in their programming thought they put in a 10 second pause and instead put in a 10 millisecond pause for this thing to call home and say, hi, I'm still online, I'm currently in an on state. Here's the temperature I think it is and you know, here's the temperature that, you know, here's the current temperature and here's the temperature I'm actually trying to set things to. And it was reporting that every 10 milliseconds, duh. So again, once we figured out was an H Vac device and once we figured out, okay, now let's look at where it's calling home to. Oh, it's calling home to the vendor that made the H Vac device. So the management company that's responsible for the H Vac device that made it pretty clear. Yeah, okay, this is probably not a backdoor. This is a quote unquote feature. It's just crummy code. So again, figuring out what that internal system is can be really helpful for that. I will say security software, some of the worst. There's a lot of security software that is, it looks at log entries generated by other systems. It'll call back to the vendor to say, hey, I just saw this digital certificate or I saw a connection to this IP address. Is there you know any known blacklist this stuff is on? Is this digital certificate on a banned list? So it, it really helps to kind of have context and know what that internal system is. It also helps to have a baseline. So I don't necessarily mean a baseline on the system itself, you know, but if I have a desktop that's doing something odd, I should be able to kind of crowdsource a hundred other desktops to kind of see how they operate, to see is that one desktop different from them or, you know, very similar, but just a little bit off. That can kind of help you figure this out too. So our step one, identify if there's a connection. Step two, recon on the internal system. Step three is get into a protocol analysis. So now let's look at what is the communication that's taking place on the network. Is it a protocol I recognize? Is it a protocol that actually makes sense for the port it's on? I'll give you a great example. We had a customer that ended up with a trace where they were seeing a system beaconing on TCP443. So that's HTTPs. That means it's HTTP encrypted, which means I should see an SSL or TLS connection. But when you went in and you looked at the protocol field, there was no client hello going by and there was no digital certificate coming back from the server. So what it was was a back door that was just obfuscating the data saying, well, if it looks encrypted, no one will bother with it. And yeah, for the most part that was true, but we were able to kind of go in. And so again, if you're seeing communication taking place to a port you don't recognize, that's something you need to worry about. I also kind of mentioned the false positive situation. So persistent connections, VPNs. You know, let's say you've got a corporate office talking to a field office. You might have a site to site VPN set up there. You expect that to be persistent and running all the time. Ntp, I mentioned that as a potential beacon. False positive DNS. We've seen false positive as positives as well. We've seen situations where a customer will have an internal DNS server that's acting as a forwarder, not as a resolver, it's acting as a forwarder. So all DNS queries that come to it, it sends it to one of three specific IP addresses out on the Internet that then does the resolution. They're doing that to go through. They're subscribing to a blacklist service. So if somebody tries to look up a domain that's known to be evil, they'll get back a null response. They won't actually end up being able to get to that site. But if you think about it now, internal DNS server is talking to those three external DNS servers. Every time anybody does a query that could end up looking like a beacon. But again, once we figured out, oh hey, that's my internal DNS server. Okay, who's it talking to? It's talking to an external DNS server. Is the protocol actually DNS? Yes, it is. Okay, you know, now we get a pretty good idea. This is something we can whitelist out and it's false positive. But I started talking about this one too, the well known ports. So you know TCP or HTTP, we've seen that on a bunch of different ports. 80, 89,001, you name it. I'm not necessarily talking about checking those up reports. What I'm talking about is if it's TCP 80, that should be web web traffic, I should be able to go in and identify HTTP headers. If I'm looking at TCP 80 and I see, you know, SSH version, blah blah blah go by, or if I see something that looks obfuscated going.

Clear HTTP handshake, somebody's up to no good, that's something you need to go in and dig deeper on. So first pass is to go through and identify if it's communication on a well known port, is the application associated with that port being used. Prozeek is a great tool for this because it will go in, it will look for those headers and it will ID it appropriately for you to let you know, you know, yeah, I am actually seeing DNS on UDP 53 or no, I'm not. You can also go in and validate the integrity of the protocol itself. So let's say we're looking at, yes, it's a protocol on a well known port and it is actually the protocol I expect to see. What else can I go in and do from here? One of the things you can do is you can go in and you look for user agent. Now don't shut me out because I know we've all been down the user agent road. We were told, yeah, check your user agents. This is awesome. You can catch all sorts of backdoors. Just look for this specific agent field and you know, and attackers go in and change it. And now all of a sudden that trick doesn't work anymore. That's true, but hear me out. If I have a standardized environment, so let's say Everybody's running Windows 10 and they're supposed to be using Chrome as a web browser. How that system goes out and identifies itself as a user agent on HTTP, how that system generates an SSL TLS hello packet to start an encrypted session should be pretty consistent. So if I go in and I look at, if I just go in and collect up all the user agents that are, that I see heading out to the Internet, if I collect all the SSL hellos that are going out towards the Internet and I say, okay, show me the outliers. You know, if something has, you know, if most of the systems are using one particular format or one particular setup, I don't care, get those out of the way, show me the unique ones. Because anything unique might be interesting. Now again, this is a weighted field. By that I mean if all I see is a unique user agent I don't recognize, or if all I see is a SSL client hello that I don't recognize, that's not enough to go in and freak out about. I gotta see a persistent connection first once I know the persistent connections there. Now if it has this attribute associated with it as well. Oh yeah, that makes it far more suspicious. But I got to establish there's a persistent connection first. That's got to be my baseline line here. Another slick one to go after. DNS is a really popular command and control channel these days. We've seen queries going out looking for text records associated with fully qualified domain name. And the text records are actually command and control orders. We've seen IPv6 queries quad A queries going out and the 128 bit IP address coming back is not a legit IP address. It's actually command and control orders because you can fit that 100 if you know what you're doing. We've also heard, haven't seen, I haven't seen this specifically, but I've heard tell that domain keys are now being used. So folks are going out looking for the domain keys associated with fully qualified domain name. The domain keys are invalid, but if you go in and decompress them and unobfuscate them, there's actually a command and control channel sitting in there as well. So again, there's three different mechanisms that's taking place over DNS. What they all have in common is that every time that thing wants to call home, every time the internal compromised system wants to go out to check, to see, you know, do you have something new for me to do? It's going to use a new fully qualified domain name. Otherwise, what it may get back is cached information from the local DNS server, which means it'll never actually reach its command and control server. And they can't have that. So the side effect of that is you will see what appears to be a lot of hosts sitting inside of a specific domain. So if you take a look at the first line output here, the r1x.com, actually let me kind of start at the bottom and then we'll work up to that because then it makes a lot more sense. So think about how many fully qualified domain names your environment exposes to the Internet. And for most of us, that number is pretty small. You know, it's usually like 10 or less. We get a website, a mail server, some DNS servers, maybe a customer portal. I mean, you get the ide, the number of hosts we're going to put in our DNS so that people on the Internet can try and get to them is going to be a relatively small number. And then you get into the environments that everybody's heard of. Microsoft, Amazon, Akamai. You know, these are large organizations that have been around forever that offer many services on the Internet. You've heard of them. So to see them with, you know, couple hundred, maybe 500 domain names that are exposed to the Internet, especially if you're using like aws, you know, and you get a lot of systems out there, that's not uncommon. But once you start getting up above 500, and certainly once you get up above like 800 or a thousand now, things get a little wonky. In other words, you know, if a domain has 2000 fully qualified domain names that it's exposed to the Internet, that starts not making a whole lot of sense, especially if it's not one of these big companies that everybody's heard of before. So now let's go up and take a look at that top line. So I've got this domain, r dash1x.com. Who the heck are they? I have no idea. I've never heard of them before. They've never been in the news. So the fact that it looks like they have 63,332 host names that are resolvable, that they're exposing the Internet. Oh, something's wrong there. So again, you know, protocol analysis, these are valid DNS queries, but it's an unexpected behavior. We don't expect to see that many host names associated with remote domain. So long as RETA is installed to see what host names are getting resolved out on the Internet, it'll be able to pull this information for you. So we've gone through the protocol, we figured out, you know, does the protocol make sense? Are they using it the right way? Does all of that stuff look legit? We've added or subtracted some points to our threat score based on that. Now we go in and we start looking at the target, target IP address itself. You know, who, who are they? You know, dig Dash X and then the IP address, what domain is it going to do? I know that domain, do I recognize it? Is it a business partner? Is it, you know, is it one of my vendors? So if my security system that analyzes logs is calling out to the vendor that made that software, okay, I may not like that, but that doesn't mean it's a backdoor. Right? That's really what I'm trying to get to the root of here. Take a look at the GEO information, make sure that makes sense. We had a customer that figured out they had three IP based cameras on their network that were compromised straight from the vendor, simply because the three IP addresses were beaconing out to Quanzu, China and they didn't have a field office in Quanzu, China. That was the first thing that set off a warning bell for them. So even though Geo isn't 100% accurate, it gets in the ballpark, can be really helpful. Also, look at how old is the domain. Any domains that are seven days old or less be really suspicious of. In our previous life I worked for registrar and one of the things that ICANN does with registrars is they mandate that you give a seven day money back guarantee on a domain. So the idea is, in a drunken stupor, I can go out and I can register some stupid domain name and so long as I sober up over the next seven days and say, wow, that was a really bad idea, I should not have spent money for that domain. You can go back to the registrar and say, give me a refund and they have to do it. Well, attackers leverage that. So when I was at this registrar, I started poking through their database and I was able to tie down sources that were using multiple accounts and they were doing the same thing with each account. They usually had three, some of them only had two. But what they would do is they would register a domain, they would spam from it, attack from it, pollute the heck out of it, and then 6.9 days later come back to us and say, hey, I Changed my mind, I want a refund. Meanwhile, they'd have a second account that they would then go through and register and pollute and spam. So while their refund was getting processed on the first account, that Phil overused the second account. And by the time the second account hit 6.9 days, they had already gotten a refund for the first account. They'd now use that money to go out and get an additional account. I found malicious actors that literally had burned through hundreds and hundreds of domain names that way over a couple of years just because they didn't know how to go in and look for that within their system. Once we tagged it, oh yeah, we started kicking those people out left and right. But you get the idea. If you see a lot of traffic going to a domain that's less than seven days old, that's something to go in and pay attention to is the destination on any known blacklist. We're doing this one last, you know, and John and I started talking about this one a little bit back in the beginning and I'm going to kind of rope into why. And here's a great example why. So here is an IP address that has been, you know, there's been 142 reports from 115 different sources that this IP address is evil and this IP address is a bot for Bing. Now we might not, you might not use Bing, you might not care about Bing, but Bing's got web crawlers just like Google does. And they do the same thing Google does, they go out and they index websites. But if you take a look at some of these write ups, you know, oh my God, it's crawling my website and it's running all my JavaScripts. Yeah, that, that's what a bot does. That's what a web crawler does. So 115 different individuals looked at this and said, oh my God, it's evil, ban it. And this got it on a bunch of different blacklists. So if I see one of my systems talking to this IP address, should I, should I be concerned? No, clearly not. It's not. It's a lot easier to get on a blacklist than it is to get back off it at a. So I'm not saying you should not worry about blacklists at all. That's why, you know, that's why I had a slide on it. Make sure you check that. But once you check it, give it a little bit of leeway, you know, again, all the other pieces have to kind of line up for me before the fact that it's on a blacklist actually matters at all. And some additional tools you can use to go in and do some additional reconnaissance on this target IP address. You know, dig, hey, this is a simple tool we've all used in the past, right? Let's go in and look at what's the, what's the PTR record associated with this IP address, what mains responsible for that section of the network you can get rid of plus short and it'll give you a verbose output that'll also show you, you know, who's responsible for the domain name servers and that type of thing. There's a couple of links online, these are just two of my favorites that you can go in, you can plug in an IP address and they'll tell you everything that they know about them. You know, is it on any blacklisted IP addresses you know, is there, is there identified malicious activity associated with this host in the past? Who's responsible for that? I what company is associated with it? You can get a lot of really useful investigative information out of that. So what do you do next? Well, so now we've gone through, we've, we've looked at is there persistent communications, yes or no? If yes, then we went in and we got threatened. We got some intel information on our internal IP address. Then we went in and we analyzed the protocol. Then we went in and we analyzed the reputational information. Now we need to take all those data points, put it together and we need, in it, we literally need to make a black or white choice. There's a very small gray area in the middle and the black and white choice is is this internal IP address compromised? Yes or no? If the answer is no, I need to go in and create a white list entry. I need to be able to go in and have some way to be able to say, don't show me this activity in the future. I've already blessed it and said it's okay. And that might be a note in a JIRA ticket that might be whitelisting the IP within the tool. That really kind of depends on your but you've done all the work to do this investigation. You don't want to have to keep doing it again. And that's actually, if you do that. This makes threat hunting a continuously improving process because the first time you do a threat hunt, you're going to be see a bunch of stuff you don't recognize. And now you're going to go in and you're going to say, okay, here's Eight to ten things I know are, okay, great. Now tomorrow those eight to ten things don't show up. You can focus on anything that's after that, and you can go on from there. Our other big possibility is, hey, I think the system is whacked. Something's really wrong. In which case, what we do here depends upon our incident handling policy. You know, maybe we go into full for incident forensics. You know, maybe we go into full incident handling mode where we do forensics, we start pulling out host log files, and that kind of comes down to your environment, what to do next. But we're definitely into some form of incident handling slash forensics at that point there. Now, I mentioned there's a very small gray area in the middle. What is that? Well, that's when, geez, I'm really suspicious, but not enough to pull that big red lever and get everybody to freak out. So what do you do? Well, what I do is I make a note of that, I pay, and then that's the first thing I go looking for when I go into the data tomorrow. You don't want to do that that often. You know, you don't want to be kicking the can down the road, so to speak. But. But even better, you may go in and say, hey, I'm highly suspicious of this IP. Maybe I start getting full PCAPs of all the traffic leaving that box. Maybe I start collecting host entries off it. You know, maybe I don't kick off full incident handling mode. We need something, you know, something below forensics. You know, we need like a quasi forensics where we go in and we say, I'm suspicious. So we're gonna, you know, start taking, you know, more detail, start collecting more detail up on that. And once we have that information, now we're back to this slide. Is it evil? Is it not? If it is evil incident handling mode. If it's not, we go in and we do a white list on it, and that's about it. I wanted to make sure we left plenty of times for Q and A at the end of this because I know this is kind of like a new thing and everybody's still trying to figure it out. So, John, you said you, you willing to handle questions.

[B]

Hold on. I've been madly responding to them as quickly as I possibly can. I've been doing pretty good at keeping up. Is Chris a Patriots fan? Because he sounds totally wicked, believe it or not. Florida.

[C]

So Chris lives in Florida, but Chris did grow up in New England. I'm not a big fan of sports that don't include at least 500 horsepower in four wheels.

[B]

Do you see this? Okay, this is a good question. Do you see hunt teaming growing?

[C]

Oh, absolutely. This is the big gaping hole in our security that nobody wants to talk about. Because if you think about it, we've spent a lot of time focusing on protection. How do we keep the bad guys out? And we've spent a lot of time focusing on forensics. How do we track down exactly what we did? The missing piece has been that thing in the middle. When I. When our security precautions fail and they got in, how do we figure out they got in? That's what threat hunting is about. I. The biggest tell for me is I think the last one I saw was Starwood, so. Starwood. John, correct me if I'm wrong. They were in that network for what, two and a half years scraping data. Yeah. Before they got caught. That to me. And they're not the only environment that has run into that. What was the. Oh, there was the other one. There was the credit card processing company that got at least one PCI attestation, maybe two, while they were in a compromised state. I mean, that, to me, clearly says our attestations are broken, our processes are broken. We need to be focused more on validating the integrity of things. We've taken steps to try to protect it, but we haven't been validating it to make sure that if those steps are actually working and we are still in a protected system state.

[B]

Another question Mike asked, it was Heartland.

[C]

Yes. Thank you. Thank you.

[B]

That was years. Does RITA provide for ASN and IP addresses? Can you add that to the whitelist as well?

[C]

I'm gonna. I'm gonna have to punt on that question. So I know within our commercial product. Yes. You can create whitelists for it. I don't think so. In ReTA.

[B]

Okay.

Basically what AI Hunter does is, I believe it expands out the ESNs and creates those lists. And we'd have to ask Joe for that. Joe's the one that created that code.

[C]

Yeah. Quite honestly, I think you could easily write. Even if it didn't, you could write a script around it. So you could take the output from Rita and then just pass it through something else that says, hey, if you see any of this stuff, make it go away and give me what's left over. Yep.

[B]

Another question is, what are your thoughts on the tool Dark Trace? And I'll take this one, if that's okay, please. So I love darktrace. I think darktrace is fantastic. If you want to completely re Architect your entire environment. Hook darktrace into absolutely everything and you can afford it. That's great. We wanted to focus on a specific area in the realm of network command and control detection because all bad traffic is going to have to leave your network at some point. And our solid goal, our solemn valve at active countermeasures, is to make sure that we're cheaper than the sales tax of Protect Wise and Dark Trace. So that's the, that's the biggest thing.

[C]

And we are.

[B]

Yeah. Now somebody asked about CDNs and this is one a domain fronting. Alex asked this question, and this is a question after my own heart.

[C]

Didn't somebody write a really good blog entry on that?

[B]

Somebody did. It was a very short one. But yes, we, we actually detect that we have a jackass employee, his name is Derek, at Black Hills Information Security who kind of goes out of his way to try to bypass RETA on a fairly regular basis. And it was a challenge for quite a while. Was a challenge for quite a while. We kind of, I guess the holidays hit and we got, we got slacker on it. But yes, we can actually still detect that. It all depends on the traffic patterns themselves. Like if we try to do randomization, we look at the dispersion of the randomization data. Size is another great thing to look at as well. So yes, we do actually pick up on CDNs.

[C]

So yeah, one of the questions was, so, hey, what do I need for resources to actually get this all set up? Span port, that type of thing? Yeah, I mean, you know, you. So it all comes down to how fast is your link to the Internet. If you're running, you know, like a gig or more. Yeah, you might be looking at like a gigamon and you need to spend some money. If you. Where most folks are in like the 100 megabit range. Yeah, you can build a box to go in and do this. BRO has a mode called PF Ring that works with certain intel chipsets. Definitely do your homework on that. That like increases performance dramatically. It's a little bit of a Peter to get it going, but it's worth it if you can. But having BRO grabbing metadata off everything goes that goes by. Yeah, that's gonna give you a lot of really good intel on what's up as far as what you need for a box. Rita on the same box as bro, quite honestly isn't a lot additional overhead. So really the specs for bro are going to be the same for bro and Rita. If you're around 100 megabit, you could probably get away with like 32 gigs of ram bus IO is more important than anything else. Couple processes, you're in good shape as far as disk storage goes. Now it comes down to how long do you want to keep this information for? For me, unless I find something interesting, I want to keep it for at least five days because if I see something interesting, I want to be able to go back a couple of days. I would definitely recommend SD cards just because the access time to read and write is that much faster. And don't do RAID five because if you do RAID five, you're actually doing two reads and two writes per block. And again, we're trying to grab packets. That just is going to slow everything down.

[B]

Yeah, some people asked about Security Onion and it turns out that I have people asking questions about Security Onion and RITA and and we have people that shared a link on Security Onion and RETA running on the same system. So I just shared out that link directly from the wiki of Security Onion as well. How do you query to find the age of all domains queried? I don't think we do that in.

[C]

Reta, but how do we implement it in AI Hunter? You can go in and if you do a WHOIS query, it'll tell you when that domain was registered.

[B]

And that's just doing it by hand, right? Just running it from the command line.

[C]

Yep. Yeah, exactly. There's another one in here. What factors does RETA use for to score beacons? Just equal quality of period, size, duration or other other factors. So we actually, if you want to read up on it, maybe John has the link handy. We have 23 patents on Beacon analysis takes place. So yeah, there's actually, there's a lot more to it than that. When you start getting into trying to figure out when they've been when, when is it an intermittent signal versus a signal that's actually jittering. That's where the math becomes really important. And quite honestly, I would be blowing smoke if I tried to explain the math behind it. Luckily, JIT hired people much, much smarter than me to actually figure out the math and implement it in the code.

[B]

And we can go over the really easy scenarios. But as Chris mentioned, the really, really smart people, they cringe every time I use this example and I talk about K means clusters clustering and we joke.

We no longer call it K means clustering. We call it L means clustering because it was written by Lisa Woody at Active Countermeasures. But to give you an example of the ways that you can detect it and this is analogy I've used in previous webcasts. If you have a small child trying to get a hold of a parent, they will say, mom, mom, mom, mom, mom or dad, dad, dad, dad, dad. That's consistent data size and it's consistent in interval. That would be a really, really simple beacon to pick out with something like K means or L means clustering. But we go a lot further with reta. RETA will also look at the data size. It'll also look at the durations, the connection times. There's, I think there's like five or six different attributes that we look at at a connection. Far beyond just the beacon ability of it as well. Man, there's a lot of questions.

[C]

Kind of figured it would be.

[B]

Here's a good one. How effective is this without SSL inspection? Well, here's the deal. Whenever we're creating back will use SSL and then we'll encrypt it inside the ssl. So even that deep packet inspection or that interception, you still have a layer of obfuscation that the attackers can use. And that's why we stepped away from trying to look deep into the packet and try to get it to tell us its secrets. Because honestly, you'll never get any answers. Has anyone ever implemented RITA or AI Hunter and not found Beacon? Yes, when it doesn't work, work.

I had this problem the the new netflow IP fix setup for Rita. I configured it at my house and I was an idiot and I sent it to the wrong IP address. So somewhere out on the Internet, there's somebody that's getting a whole bunch of netflow data from my home. It's just.

Their firewall, like, and there's going to be people like Chris and I that'll be like, from that company going, what the hell is this, Chris?

[C]

What does.

[B]

What do they just play Fortnite all the time at this, trying to tell us? So, yeah, if it doesn't work, but that's why the whitelisting is so important. And you want to be able to feed those IP addresses, especially for like security appliances to get those set up. Mike asked NetFlow, yes, we do support Sflow or, sorry, NetFlow version 5 and 9 and IP Fix. As of right now, they just keep coming. Here we go. So, Chris, so if you whitelist correctly, this would team up great with a SIM tool like Splunk, sort of.

[C]

And you want to whitelist it out before you even get into that depth. If you can, you know, just keep it out of Splunk in the first place why have the overhead any different URLs I can get from you that work to view the blog since work is blocking the ACM re. Yeah. Www.activecountermeasures.com of course that'll get you to the same site. So maybe they're just blocking re. If so, that that should work for you. Thank you John. Thank you John. You're the best. Yes, he is. John is freaking awesome. Tried setting up Rita on CentOS with the standard install.

[B]

There's a lot to that 48 hours.

That doesn't seem right. Something very bad has happened. I would say. Hey, set up a demo. Let's talk about about it.

[C]

So Stearns has tested us on CentOS so I know for a fact we work there. So it should not be. That BRO does have some weird settings for defining what your internal network is and if you don't do pass that information along correctly to reta, RETA will miss it, especially with the latest version. So in other words, RETA kind of wants to know what are your internal addresses and it needs to. In fact I borked something recently with this and it needs to have all of them because if it sees what looks like external to external traffic, it'll just kind of ignore that. So my guess is there's a network setting that might be wrong in that setup. But yeah, as John said, book a demo, we'll figure it out.

[B]

That's what we do.

[C]

Or even even better [e-mail] Even though it's RETA, not AA Hunter, we do still kind of support folks with that. Just drop in a note with what you got and I will hook you up with people much smarter than than me.

[B]

Now we do have someone that says is it possible to work with standalone PCAPs? Yep, you just do bro space minus R your PCAP and then it'll bro will parse or Zeke will parse the PCAPs. Then we can pick those up.

[C]

And then also there's actually a little bit more to it than that because you also that that local Net specification, you need that in there as well.

[B]

Yeah, that's right.

[C]

Yep. We actually have a law enforcement agency that is not in the US that uses that to go in and do their investigations. So they collect PCAPs on site, bring them back to a central office, run them through BRO reader and into ia.

[B]

Hunter and Brandon also brought up he said make sure you give the system enough resources like CPU cores and hard drive space. This is not something you just spin up In a vm you would need to make sure you give it proper resources to be able to load all the data through. And I think that we have at RETA the resources that you should use. And then also BRO has the their own website as well that talks about what type of system specs you should have.

[C]

Yep. We do have a pre installed document on a Hunter. I. I don't know. I don't know if we have specs for reta. Let me look into that. Well, we have a misspot. A Hunter. I don't know if we have them posted on the reader side. Hold on. Yeah, because we, we just went through and broke them up based on like 100 megabit link, one gig link.

And beyond. But I don't know if that's filtered over to the reader page yet. I'll have to check that out.

[B]

I thought for sure we put in that request. If we didn't, then our we will be flogging interns.

Oh, here we go.

[C]

Awesome too.

[B]

Oh, thank you, Dale. We do in fact have system requirements. There it is at the reader website. So it's there.

[C]

Awesome. Awesome. Yeah, there's a new set that Stearns and Rebecca just went through and did that break it out because we had the generic requirements. And then we also went and we said, okay, if you have a hundred megabit, this is what you're looking for. And here are two vendor examples that you could buy. I think one was Dell and one was Lenovo and then we did the same thing for a gig just to kind of give people a better idea of what they need for hardware.

[B]

All right, any other final questions? Dumb questions. Threat hub. Oh yeah. So the slides will be released least with us as well. Does AI Hunter and Rita work completely on prem? Yes. You can run it completely on prem if you want. You can throw it up in the cloud if you want. We just.

[C]

Yes.

[B]

Really don't.

[C]

We do not want your data, Andrew. You get to keep all your data. We have no say in that. We don't want it.

[B]

Yep. And then John said, well, it depends on the network card. But John brought up a point and he said we, we used to use PF ring, but we switched to AF packet. And I think that that really depends on the network card that you're using and whether or not it supports efforting like some of the intel cards do. Yeah, this is quickly getting to the point where we honestly should just do an entire webcast on hardware.

[C]

Yeah, that's true. Yeah.

Yeah. Because like Dale's question right after it you know what are some good dedicated network tabs to use with Bro Rita? Yeah.

How to Tap your Network 101.

[B]

All right, let's get out of here, everybody. Thank you so much for coming joining, and we will see you at the next webcast.

[A]

Thanks for listening to the active Countermeasures podcast. Remember, if you enjoyed this podcast, to leave a positive review.