Podcast Summary: "Protecting the Core: Securing Protection Relays in Modern Substations"

Podcast: The Defender's Advantage Podcast
Host: Mandiant — Luke McNamara
Guests: Chris Systrunk, Samat Bisht, Anthony Candarini (Mandiant OT Consulting Team)
Release Date: July 28, 2025
Episode Focus:
A deep dive into the critical role and cybersecurity challenges of protection relays in power grid substations, with a practical, 'Red Team' perspective on both threats and defense strategies.

Main Theme Overview

This episode explores how modern protection relays—core devices at the heart of power grid substations—are actively targeted in cyber attacks, the real-world techniques attackers use (drawing from both historical events and red team experiences), and practical security practices for defenders. The discussion spans attacker reconnaissance, vulnerabilities in IT/OT integration, gaps in defensive posture, and guidance for utility engineers and defenders.

Key Discussion Points & Insights

1. The Role of Protection Relays in the Power Grid (04:23–08:09)

• Protection relays are described as the "brains" of substations, swiftly detecting faults (like an animal on a power line or storm damage) and opening circuit breakers to prevent damage.
• Evolution from electromechanical to digital, microprocessor-based relays has added speed and connectivity (Ethernet, serial ports), but increased the attack surface.

"These smart devices... now [are] done digitally with microprocessors... and now they have connectivity with ethernet ports... we can talk to them, program them remotely or not remotely." – Chris Systrunk (07:35)

2. Why Protection Relays Are an Attractive Target (08:33–13:17)

• Attackers focus on relays since they are critical gatekeepers for tripping circuit breakers—controlling power flow at both micro and macro grid levels.
• Relays are involved in every command path for opening/cutting off circuits, making their manipulation ideal for causing blackouts.
• Beyond just tripping breakers, attackers can alter relay logic—much more dangerous and covert.

"When we manipulate the settings or the logic... it can be a nightmare for the entire substation or the entire country." – Samat Bisht (10:40)

3. Real-World Threats: Historical Attacks and Demonstrations (11:06–13:17)

• Ukraine 2015 blackout cited as real-world example.
• The 2007 Aurora experiment at Idaho National Lab demonstrated how reprogramming relays can destroy physical equipment through deliberate grid desynchronization.
• Even spoofing relay indicator lights can mask sabotage from technicians.

"Instead of allowing them to close when they're supposed to... let's make it sync with the grid in a chaotic cyber-physical way that will destroy the piece of equipment." – Chris Systrunk (12:25)

4. Safety vs Security: The Trade-Off in OT Environments (13:17–19:31)

• Remote management of substations improves safety, but introduces cyber risks.
• Regulations like NERC CIP in North America address large utilities but leave gaps for small utilities and non-U.S./Canada entities.
• Engineers often approach systems with an 'accidents' mindset rather than from an 'attacker' mindset.

"Instead of it being a probability based attack, it is a knowledge based attack... an attacker that knows... can simply just attack it." – Anthony Candarini (18:41)

5. Attacker's Playbook: Reconnaissance to Impact (20:19–32:24)

a. Reconnaissance (20:19–23:42)

• Attackers gather public information (e.g., LinkedIn, RFIs) to target key personnel and deduce technology in use.
• Active scans for exposed VPNs, engineering portals, or internet-facing OT assets.

b. Compromise and Lateral Movement (23:42–28:18)

• Phishing and exploiting VPN vulnerabilities are common initial footholds.
• From IT, attackers enumerate the environment for OT-relevant documents, credentials, and system details.
• Once in OT, attackers look for HMI access, configuration diagrams, and relay logic to plan impactful actions.

"You search through a bunch of shares. We found pretty much all the passwords we needed to get all the way down into the OT environment in two different ways." – Anthony Candarini (28:43)

c. Actions on Objective (28:18–32:24)

• OT networks often lack segmentation ('flat' architectures), making lateral movement easy.
• Default configurations and passwords abound; jumpers or settings left from commissioning provide further opportunity.
• Attackers leverage legitimate engineer tools, which are hard to distinguish from normal activity.

6. Challenges in Attribution and Detection (32:24–34:01)

• Early intrusion phases (on IT) look like typical espionage or reconnaissance.
• Determining intent (espionage vs sabotage) is difficult until attackers take overt actions.
• Even sophisticated nation-state actors can make mistakes or need time to understand each OT environment.

7. Practical Defensive Guidance & Top Security Practices (34:01–39:06)

• Only three publicly known cyber-induced power outages—each in Ukraine.
• Most grid failures are still due to accidental events (e.g., animals), but proactive security is needed.
• Defensive priorities highlighted:
  • Control and monitor remote access to critical systems (preferably with MFA or strong physical security).
  • Change default passwords; remediate insecure default settings or jumpers left after system commissioning.
  • Ensure regular, tested backups and critical spares for quick recovery.
  • Disable unused services.
  • Segmentation between IT and OT whenever possible.

"The simplest thing you could do is go through your environment and change those default passwords. That's the lowest hanging fruit..." – Anthony Candarini (38:15)

8. Looking Ahead: Community, Research, and Expansion (39:06–41:52)

• Encouragement for engineers and security outsiders to join wider OT security communities.
• Future research will include broader OT/TSO (Transmission System Operator) topics—remedial action: schemes, automatic generation control, HVDC.
• Lessons from relay security apply broadly to other specialized, solid-state, OT devices.

"We have a community out there in OT security, ICS security... get involved, reach out to us... increasing your level of awareness around securing these substation systems." – Chris Systrunk (39:28)

Memorable Quotes & Moments

• "When you manipulate the settings or logic... it can be a nightmare for the entire substation or the entire country."
  — Samat Bisht, 10:40

• "[Attackers] can manipulate settings that allow them to... keep on trying to reclose [a breaker] when there's a fault on it, right? That's obviously very bad."
  — Anthony Candarini, 31:18

• "Even for a very sophisticated actor... this still takes time to put together and operationalize. It takes time to actually gain some familiarity..."
  — Luke McNamara, 33:47

• "Engineers aren't dumb. We've been dealing with failure for centuries. Right. That's why we have these systems that are so reliable... what are the top 10 security practices for substation relays?"
  — Chris Systrunk, 34:30

Timestamps for Important Segments

• [04:23] Protection relays explained & why they're essential to the grid
• [08:33] Attackers’ focus on relays & path of circuit breaker commands
• [11:06] Historical attacks (Ukraine, Aurora)
• [13:17] The safety vs security trade-off in operational technology
• [17:43] Why knowledge-based (vs probability-based) attacks are changing OT defense
• [20:19] Red team walkthrough: attacker reconnaissance & intrusion playbook
• [28:18] The challenge of network flatness & insecure defaults in OT environments
• [34:01] Real-world attribution & top 10 defensive practices discussed
• [39:06] Final thoughts—community, broader OT issues & future research

Closing Thoughts

The episode is a practical, field-informed guide for defenders and engineers alike, demystifying how attacks on protection relays may unfold—and, most importantly, how to shore up defenses using not just technology, but also changes in organizational mindset and basic cyber hygiene. The hosts encourage the critical infrastructure community to be proactive, share knowledge, and continuously adapt as technology—and attacker sophistication—evolve.

Recommended Action:

• If you work in, or manage, critical infrastructure, review your relay security practices, prioritize quick wins (like changing defaults), and connect with the broader OT security community.
• Check out the companion blog (link in show notes) for a detailed breakdown of red team findings and actionable advice.

For further information or to join the conversation, reach out to the guests on LinkedIn or at upcoming OT security events.