Podcast Summary: Cybersecurity Today

Episode: The Evolving Landscape of Cybersecurity Training: Effective Strategies and Misleading Headlines

Host: Jim Love
Guests: Michael Joyce (University of Montreal), David Shipley (Beauceron Security)
Date: August 23, 2025

Overview

This episode explores the real-world effectiveness and science behind cybersecurity training, especially phishing simulations and awareness programs. Host Jim Love is joined by Michael Joyce, CEO of the Human Centric Cybersecurity Partnership at the University of Montreal, and David Shipley, CEO of Beauceron Security. The discussion draws from recent, large-scale Canadian research on security awareness efforts, their successes, and limitations, as well as sharp critiques of misleading research headlines in the media. Together, they unpack practical strategies and common misconceptions in cybersecurity education.

Key Discussion Points & Insights

1. The Human Side of Cybersecurity

• Cybersecurity is not just technical—real progress comes from understanding people, culture, and behavior (03:38–04:15).
• Most in the industry recognize that people are the challenge, but revert to technical talk because it’s familiar and measurable (04:15–05:11).
• Research on culture and human factors needs more scientific rigor and outcome focus.

Quote: “The reality is this messy problem of cybersecurity. It ain't just the computer, my dudes, it's the people and the computer together.”
— David Shipley (03:38)

2. Academic & Industry Research Collaboration

• Beauceron Security turned over a unique, broad dataset (700+ Canadian organizations, 227,000+ individuals) to Michael Joyce’s academic team for unbiased scrutiny (09:00–09:50).
• Academic research aims for robustness and long-term knowledge, while industry needs practical, timely solutions (09:00–09:59).
• Large-scale, cross-industry datasets provide rare insights, unlike single-company studies which lack generalizability (10:41–11:48).

Quote: “As an academic, the objective here is to create an output that stands the test of time by itself.”
— Michael Joyce (09:16)

3. Cybersecurity Awareness Month: Does It Work?

• During Cybersecurity Awareness Month (October), companies increase phishing simulations (up 13% in 2023, 23% in 2024) and see a corresponding drop in simulation clicks (~11–12%) (14:02–14:59).
• Real-world reporting of phishing incidents also spikes, but reporting simulated phishes declines—possibly signaling “security fatigue” (16:20–17:52).
• The improvements are temporal; positive changes vanish after the month ends (43:21–44:03).

Quote: “This is a change that happens in October that's ostensibly driven by this program...Awareness does do something.”
— Michael Joyce (14:10)

4. The Goldilocks Zone: Frequency of Phishing Simulations

• Frequent phishing simulations (more than monthly) increase reporting but induce fatigue; monthly simulations offer the best balance:
  • Click rate: ~3%
  • Report rate: ~25% (18:30–19:40, 42:43–43:22).
• Too much or too little training both reduce effectiveness (41:54–42:43).

Quote: “Monthly...turns out to be Goldilocks...average click rate of about 3.05%...report rate of 25%.”
— David Shipley (18:58)

5. The Decay of Security Awareness (“Vigilance” Not Permanent)

• Training is not a “software patch”; vigilance decays rapidly without new stimuli (00:00–00:33, 23:14–28:00).
• After training, reporting rates drop from 98% (immediate) to 60% at six months and 4% after a year. Click rates rise from 3.5% (immediate) to 95% after a year (28:00–39:07).
• Annual training is insufficient; vigilance must be “re-injected” every 90 days for optimal effect (39:07–40:53).

Quote: “This kind of awareness goes away. We expect it to go away...If you train somebody on something once...you cannot expect them to maintain that state of awareness.”
— Michael Joyce (23:49)

6. Motivation and the Value of Reporting

• Reporting is distinct from simply not clicking—motivated by benevolence/community or personal benefit, not just self-preservation (29:18–32:02).
• Closing the feedback loop (giving employees feedback on reports) significantly increases the report rate (from ~28% to over 50%) (31:44–32:49).

Quote: “Reporting is more beneficial to you than not reporting...If you don’t close feedback loops, people stop doing the thing.”
— David Shipley (29:39; 33:38)

7. Why People Click: Beyond Just Knowledge

• Surveyed clickers cited “mimicry” (it looked legit/expected) about 50% of the time, “rushing” 17%, and “don’t remember” 21%—suggesting that almost 40% of clicks were due to workflow/habits, not lack of knowledge (46:00–49:40).
• Emotions (curiosity 6%, fear 5%) played a less significant role than expected.
• Organizational culture (psychological safety) impacts whether people admit and report their mistakes; punitive responses reduce honesty and future reporting (55:00–55:52).

Quote: “21% of people said, I don't remember doing this...almost 40% of clicks, I hypothesize, have more to do with how we work with email than any of the inherent knowledge they brought into the battle.”
— David Shipley (49:01)

8. Debunking Misleading Headlines: Training “Doesn’t Work”?

• Some media/research (e.g., UCSD Black Hat paper) claim phish training doesn’t work—major overstatements not supported by the actual science (54:57–59:26).
• Large-scale research shows annual training is insufficient, but regular, well-designed training does reduce click rates and changes behavior (65:44–69:12).
• Media’s clickbait summaries ignore context, cost-benefit, or overall culture impacts.
• Headlines suggesting awareness training is “useless” are deeply irresponsible, especially in the absence of longitudinal or layered effectiveness measurement.

Quote: “What passes for research at times in this industry is appalling...if you're going to say something like all phishing training is useless, I want receipts, I want data before you say something like that.”
— Jim Love (59:26; 69:12)

Notable Quotes & Memorable Moments

• “There’s a comfort in keeping the conversation about speeds and feeds...then there’s this messy human thing.”
  — David Shipley (05:11)
• “Skills decay is a thing in phishing, which supports our idea that awareness is a temporarily limited phenomenon.”
  — Michael Joyce (43:21)
• “You can still have the best theoretical training delivered the best way...and we’re still going to have...a 3.5% probability of click. Welcome to the human condition. That’s still better than doing nothing.”
  — David Shipley (62:11)
• “Give up the notion of absolute truth. Truth is a percentage and it changes over time. With a change in the evidence, your perspective on what is true must change.”
  — Michael Joyce (65:44)
• “If we're prepared to let your whole defense decay to zero before you retrain, once a year is the way to do it.”
  — Jim Love (40:53)

Timestamps of Important Segments

• 00:00 – The problem with expecting constant security vigilance
• 03:38–05:11 – Human vs. technical focus in cybersecurity
• 09:00–11:48 – How the research collaboration and dataset were constructed
• 14:00–17:52 – Cybersecurity Awareness Month’s impact and limits
• 18:30–19:40 – Evaluating phishing simulation frequency (monthly as best practice)
• 23:14–28:00 – The science of awareness decay and vigilance
• 29:39–32:49 – Importance of feedback loops in increasing reporting rates
• 39:07–40:53 – Empirical decay rates; the 90-day rule
• 46:00–49:40 – Why people click (survey results)
• 54:57–59:26 – The dangers of misleading research headlines
• 65:44–69:12 – How to interpret research and beware of absolutes

Recommendations & Takeaways

• Monthly phishing simulations are optimal (not weekly, not annual).
• Awareness and training DO work—but their impact fades; refresh every 90 days.
• Don’t punish clickers excessively—punitive approaches harm reporting and company culture.
• Close the reporting feedback loop: Employees need meaningful, timely responses to their reports.
• Ignore clickbait headlines declaring awareness training “useless”—the reality is nuanced and layered.
• Integrate academic insight: Collaborate with researchers to measure and interpret security outcomes.
• Critical thinking beats cynicism—scrutinize the evidence, measure what matters, and avoid “absolutes.”
• Layered defenses (“defense in depth”) remain best practice; technical solutions and human elements complement each other.

Final Thoughts

“Annual training, not cutting it...more frequency, consistency, and the benefit of people knowing, ‘Hey, there’s a traffic cop out there ready to pull me over in the email speeding lane—and slowing down is a good thing.’”
— David Shipley (77:34)

“Make sure...you are working to measure and push the field of cybersecurity awareness...forward in a way that is measurably forward, because as we've shown, you can do this right and you can do it wrong.”
— Michael Joyce (78:30)

For security leaders and practitioners:

• Rethink annual, one-size-fits-all training.
• Respect the complexity of human cognition and behavior.
• Demand real data—not headlines—when setting your training policy.

Stay vigilant, but be realistic. People are your greatest defense—if you treat them as such.