Podcast Summary: Talkin' Bout [Infosec] News — BHIS PODCAST: Network Threat Hunting Runbook
Host: Black Hills Information Security
Featured Experts: John Strand, Chris Brenton
Date: February 28, 2019

Overview

This episode dives deep into the "Network Threat Hunting Runbook", demystifying how to effectively conduct network-based threat hunts. Chris Brenton (of Active Countermeasures, self-described as "Chief Troublemaker") and John Strand walk listeners through a structured, step-by-step process for identifying and triaging compromised hosts using open-source tools like RITA and Zeek (formerly BRO), focusing less on definitions and more on hands-on techniques and decision-making.

The discussion covers:

• Why and how to hunt threats on the network level
• A practical runbook for threat hunting
• Key tools, pitfalls, and methodology
• Weighting and scoring of threat signals
• Filtering and investigating connections
• Fielding listener Q&A on tooling, resources, and advanced threat hunting scenarios

Key Discussion Points and Insights

1. Why Network Threat Hunting?

• Purpose: Even the best host monitoring can be evaded; the network acts as the "great equalizer" for finding compromised systems.
• Approach: Seek persistent, suspicious communication that stands out from legitimate traffic.

"Attackers are just really good at hiding their tracks... The network's the great equalizer there." — Chris Brenton [01:54]

2. The Network Threat Hunting Runbook

High-Level Steps:

1. Identify Persistent Communication Channels:
   Internal hosts with long-standing or repeatedly-open connections to external IPs.
2. Internal Reconnaissance:
   Investigate what the internal system is, looking for legitimate explanations.
3. Protocol Analysis:
   Examine whether expected protocols are being used as anticipated on correct ports.
4. Reputation Analysis (Last):
   Only after technical analysis, check reputation/blacklists for further context.
5. Decide & Disposition:
   Make a black/white decision: is it malicious? Whitelist if safe; begin IR if not.

"Why are we going to do this in this order? … Identify persistent systems, do recon, protocol analysis, reputation..." — Chris Brenton [01:56]

Step 1: Identifying Persistent Connections [06:14]

• Analyze for:
  • Persistent/Open Connections: E.g., VPNs, crypto-miners (which keep connections open for extended periods).
  • Beaconing: Regular connections re-established over set intervals.
• Tools:
  • RITA: Free, open-source, designed to detect such patterns.
  • Firewall state tables: Sometimes useful, but not always available/automatable.
• Pitfalls: Long connections aren’t always malicious (e.g., BGP, HQ-VPNs).

"When you start getting into multiple days, be nervous." — Chris Brenton [07:15]

“Crypto-mining … that's like the number one way people are getting whacked is for crypto mining.” — Chris Brenton [09:14]

Step 2: Beaconing, Obfuscation, and Signal Analysis [09:14]

• Attackers balance stealth vs. functionality. Very infrequent beacons are stealthy but not practical.
• RITA looks for beaconing with analysis over 24-hour blocks, not just per-minute.
• Padding/Jitter: Advanced attackers randomize timing and data size—look for averages and patterns over long durations.

“If an attacker gets malware on your network, they want to make sure they can get done … before it actually gets detected.” — Chris Brenton [09:27]

Step 3: Recon on Internal Hosts [14:30]

• Identify what device is communicating: Is it an HVAC system? Security software? Is the behavior expected?
• Importance of environment baselining and crowd-sourcing for context.

“Once we figured out it was an H Vac device … this is probably not a backdoor. This is a quote unquote feature. It's just crummy code.” — Chris Brenton [15:35]

Step 4: Protocol Analysis [16:24]

• Protocol matching: Does port match expected protocol? E.g., HTTPS on 443 should show SSL/TLS handshake.
• Zeek/BRO: Identifies application-layer protocols.
• Common false positives: NTP, DNS (due to regular, frequent queries in some environments).
• User agents and SSL hellos can be used to detect outliers in homogenous environments.

“If it's TCP 80, that should be web... If I'm looking at TCP 80 and I see SSH version, blah blah blah go by — something's up” — Chris Brenton [21:33]

Step 5: DNS Abuse Patterns [23:00]

• DNS as Command & Control:
  • C2 can be hidden in TXT records, AAAA IPv6 queries, or domain keys.
  • Look for domains with excessive FQDNs (>500-1000), especially from unknown vendors.

“Once you start getting up above 500, and certainly once you get up above like 800 or a thousand now, things get a little wonky.” — Chris Brenton [26:19]

Step 6: External Recon and Reputation [29:20]

• Use DIG, WHOIS, reverse DNS, and online IP reputation services.
• Blacklists are noisy and easily poisoned:
  • Web crawlers (like Bing/Google) often end up blacklisted incorrectly.
  • Use reputation as a minor component in threat scoring.

“It's a lot easier to get on a blacklist than it is to get back off it.” — Chris Brenton [31:00]

Step 7: Disposition and Continuous Improvement

• Decision: Compromised or benign?
  • Benign = Whitelist and move on
  • Malicious = Incident handling/forensics
  • Uncertain = Collect more detailed telemetry
• Repeat process; as whitelist grows, future hunts become easier and more focused.

“If you do that, this makes threat hunting a continuously improving process.” — Chris Brenton [33:27]

3. Scoring and Weighting Threats [03:34]

• Persistence scores high (60-80/100 points), protocol matches add more, reputational is a minor component (<15 points).
• Combine to reach a clear action.

4. Fielding Audience Questions & Notable Discussions

Hunt Teaming Trends [36:40]

• Demand for threat hunting growing due to "gap in the middle" between prevention and forensics.
• High-profile breaches like Starwood and Heartland exemplify failures of existing tooling to detect persistent attackers.

"This is the big gaping hole in our security..." — Chris Brenton [36:44]

Tooling Q&A

• RITA vs. AI Hunter: Commercial AI Hunter automates more, including ASN/IP whitelist management.
• Open-source RITA: Can script around its outputs for greater flexibility.
• Compatibility with NetFlow (v5/v9/IPFix), PCAPs, and on-prem/cloud deployments.
• Hardware: Bro/Zeek and RITA can run together; critical to have proper resources (RAM, disk I/O).
• Scaling: Bro/Zeek perf via PF_RING or AF_Packet; recommendations vary by network speed.
• CDN/Domain Fronting: Still detectable via traffic patterns, size, timing dispersion.

“Our solemn vow at Active Countermeasures is to make sure we're cheaper than the sales tax of Protect Wise and Dark Trace.” — John Strand [39:00]

Detection Without SSL Inspection [44:25]

• Attackers can nest encryption; deep packet inspection is not a silver bullet.
• RETA looks at communication structure, not packet contents.

“We stepped away from trying to look deep into the packet and try to get it to tell us its secrets. Because, honestly, you'll never get any answers.” — John Strand [44:25]

Practical Deployment

• On-prem and cloud supported; you keep all your data.
• Emphasis on whitelisting legitimate activity early, so SIEMs like Splunk are not overburdened.

Notable Quotes

• "Every time you see something unique [like a user agent] — that's interesting. But only if it's on a persistent connection." — Chris Brenton [21:33]
• "If you see a lot of traffic to a domain less than seven days old — that's something to go in and pay attention to." — Chris Brenton [29:00]
• "As Chris mentioned, the really, really smart people, they cringe every time I use this example..." — John Strand on beacon analysis [43:27]
• "Has anyone ever implemented RITA or AI Hunter and not found Beacon? Yes, when it doesn’t work." — John Strand [44:57]

Timestamps for Key Segments

• [01:46] Main runbook process introduction (Chris Brenton)
• [03:34] Scoring/weighting threats; blacklist caveats
• [06:14] Persistent connections, beaconing, and detection techniques
• [14:30] Internal reconnaissance; host investigation
• [16:24] Protocol analysis and port matching
• [23:00] DNS signals and anomaly detection
• [29:20] Reputation analysis approach
• [33:27] Disposition and continuous improvement
• [36:40] Audience Q&A (trend of threat hunting, tool questions)
• [44:25] Effectiveness without SSL inspection
• [49:56] System requirements and practical deployment

Memorable Moments

• Chris' quip about user agents:
  "We were told, yeah, check your user agents. This is awesome. You can catch all sorts of backdoors. Just look for this specific agent field... that's true, but hear me out..." [21:33]

• On DNS C2: "R1x.com: 63,332 hostnames exposed to the Internet. Something's wrong there." [26:40]

• Distinction on blacklisting: "It's a lot easier to get on a blacklist than it is to get back off." [31:00]

• Lighthearted banter: "Are you a Patriots fan? Because you sound totally wicked, believe it or not. Florida." — [36:15]

Conclusion

This episode offers a practical, detailed walkthrough of network-based threat hunting, focused on what to look for, how to filter and reduce signals, and which steps matter most. The advice centers on actionable strategies, not generic frameworks, empowering listeners with both the logic and the tools needed for day-to-day threat hunting at any scale. The extensive Q&A session reflects a thriving, inquisitive community and both hosts' commitment to demystifying security operations.

For additional resources or to find the complete episode (including supporting slides), visit:

• Black Hills Information Security YouTube
• Active Countermeasures