Podcast Summary

Podcast: CISO Series Podcast

Episode: It’s a Little Hard to Evaluate New Solutions When You’re Screaming “AI” at Me All the Time (Live in Houston)
Hosts: David Spark, Mike Johnson, Andy Ellis
Guests: Jarek Beeson (CISO, WM), Jack Lydecker (CISO, Gong)
Date: October 28, 2025

Episode Overview

This live episode, recorded in Houston, brings together security leaders to debate real-world collaboration between CISOs and vendors. The focus is on practical advice, the pros and cons of new technologies like AI, the ongoing struggles with cybersecurity basics, and ways for both practitioners and solution providers to work better together. Lively audience participation, games, and candid dialogue fuel a fast-paced, highly engaging session.

Key Discussion Points and Insights

1. Security Advice and Culture: Learning from Mistakes (00:03)

• Jack Lydecker shares his “worst-best” security advice: firing users for repeated phishing failures. The well-intentioned but misguided approach prompted reconsideration. The lesson: focus on changing behavior, not punishing mistakes. Fear of reprisal reduces reporting, making everyone less secure.
  • Quote: “If someone thinks they're gonna get fired, they're not gonna tell you. They're not gonna report.” – Jack Lydecker (00:18)

2. Games to Develop Security Team Judgment (02:11)

• Jarek Beeson details his team's “Pick Your Poison” game, modeling it after the podcast's “What’s Worse.” Team members choose scenarios and attempt to select the response Jarek himself would pick in front of the board. It helps staff preemptively understand leadership thinking, improving autonomy, and risk-based decision-making.
  • Quote: “A lot of decisions are made now without me having to be in the room because we play that game.” – Jarek Beeson (02:32)
  • Diverse perspectives are preserved because the majority is often wrong, driving learning.

3. Open Source: Reciprocity and Responsible Support (03:32–07:17)

• Prompt: Should maintainers of open-source software go on strike to protest exploitative reliance by enterprises?
  • Jarek: Agrees there’s disproportionate consumption over contribution. While large firms do contribute (code, tests, funds), most benefit passively. Calls for strike are dramatic and could be catastrophic (cf. Log4j, MITRE CVE funding scare). He advocates ecosystem-strengthening—“fix the ecosystem, not weaponize it.”
    • Quote: “The solution should fix the ecosystem, not weaponize it.” – Jarek Beeson (05:16)
  • Jack: Recognizes the desire for balance and points to fundraising or hybrid models (e.g., Wikipedia approach), but warns that going full commercial undercuts open source’s spirit.
    • Quote: “If you move straight into commercial, it kind of defeats the whole core of what Open Source was.” – Jack Lydecker (05:44)

4. The Geopolitics of AI-Enabled Threats (07:23–12:39)

• Jarek (citing the 2025 Armist cyber warfare report): Rapid AI advancements mean intent and access, not just resources, are sufficient for nation-state-level cyber threats. The bar to entry is lower, broadening the landscape.
  • Risk Assessment Essentials: Firms must ask about critical operations, supply chain dependencies, hostile nation states, and any business activity that could attract adversarial interest.
  • Quote: “It's not should we include this information? It's how do we operationalize it as fast as we can?” – Jarek Beeson (09:29)
• Jack: Emphasizes that smaller organizations can still become targets, sometimes due to attackers just wanting attention. Even those with limited resources should maintain at least a base level of threat awareness.

5. The "What's Worse" Game: People vs. Technology Burnout (14:44–19:21)

Scenario:

• Spend $1M/year on an LLM-based tool that only reduces analyst burnout (but not incidents), OR
• Spend nothing and rely on overwhelmed human analysts who churn every 9–12 months

Both guests prefer investing for staff well-being, despite the lack of direct security ROI.

• Burnout and turnover kill effectiveness; long-term, human damage hurts more.
  • Quote: “If my SOC is unhappy, I guarantee you that's permeating to other parts of my team.” – Jarek Beeson (17:47)
• The audience largely agrees: preventing churn is less damaging.

6. Cybersecurity Family Feud: Security Culture in Action (19:29–26:13)

A lighthearted segment reveals common security pitfalls:

• Things you should never share online: passwords, social security number, PII, credit card info, nudes (!).
• Mistakes everyone makes but won’t admit: password reuse, clicking phishing emails, risky behaviors like using public Wi-Fi, sharing credentials.

7. Distinguishing AI Hype from Reality (26:19–30:26)

• How do you separate real AI value from gimmickry?
  • Jack: First, ask vendors to quantify impact—can they show evidence? Second, probe their understanding: are they using real models, or just swapping buzzwords?
    • Quote: “If all they're doing is taking your data, throwing it in a giant LLM and spitting it back out, you can do that yourself, probably a lot cheaper.” – Jack Lydecker (27:44)
  • Jarek: Asks vendors what their tool offers that humans can’t, how their models fail, and whether they understand data governance. If they can’t answer, the “AI” is probably just hypercharged automation.
    • Quote: “Vendors with the best failure analysis actually have the best products.” – Jarek Beeson (28:11)

8. The Ongoing Struggle: Basics, Fundamentals, and Accountability (30:32–36:35)

• Why can’t we solve the easy problems, like patching and asset management?
  • Jack: The issue isn’t finding vulnerabilities—it’s managing ownership, patch maintenance, and business/testing processes.
  • Jarek: Security never actually patches systems; IT or dev teams do. When CISOs are blamed for breaches, it ignores that they often only provide context, not the “action.” The root is a business problem, not a purely technical one.
    • Quote: “This is not a security problem. This is a business problem... If a doctor diagnoses you... and that person decides not to do those things... Do you blame the doctor? No. But in security, you blame the CISO every single time.” – Jarek Beeson (34:54)
• Both stress the need to align patching and remediation with business priorities and establish accountability.

9. Audience Speed Round: Career Hacks, Anonymity, AI, and Negotiation (36:41–42:16)

• Hacking the Hiring Process:
  • Find out what hiring managers want and demonstrate impact/conference presentations.
  • Outstanding example: an applicant presenting OSINT about the interviewer.
• Anonymous Forums:
  • CISOs would anonymously discuss “bad actors” in the industry and internal political roadblocks; but full anonymity can limit constructive dialogue.
• Making Space for AI:
  • Encourage teams to use AI to improve efficiency, but always set clear objectives to avoid “flailing with tech for tech’s sake.”
• AI Data Integrity:
  • Start by auditing where AI models and data are used, then apply strong governance and guardrails.
• Negotiating with Stakeholders:
  • Use analogies/metaphors to make risk real and relatable; tie impact to business outcomes, especially customer and sales impact.

Notable Quotes & Timestamps

• On open source “strikes”:
  “The solution should fix the ecosystem, not weaponize it.” – Jarek Beeson (05:16)
• On AI in threat landscape:
  “It's not should we include this information? It's how do we operationalize it as fast as we can?” – Jarek Beeson (09:29)
• On AI vendor hype:
  “If all they're doing is taking your data, throwing it in a giant LLM and spitting it back out, you can do that yourself, probably a lot cheaper and more effective.” – Jack Lydecker (27:44) “Vendors with the best failure analysis actually have the best products.” – Jarek Beeson (28:11)
• On accountability for security basics:
  “This is not a security problem. This is a business problem... But in security, you blame the CISO every single time.” – Jarek Beeson (34:54)
• On supporting burned-out SOCs:
  “If my SOC is unhappy, I guarantee you that's permeating to other parts of my team.” – Jarek Beeson (17:47)
• On stakeholder negotiation:
  “Usually when people understand a risk, they're willing to take an action. But no one's willing to open up their paybook for something that they don't truly understand.” – Jarek Beeson (41:44)

Timestamps for Critical Segments

• Changing Security Culture advice: 00:03–02:11
• Games and CISO team decision-making: 02:11–03:07
• Open Source: Reciprocity & Strikes: 03:32–07:17
• AI & Geopolitics: 07:23–12:39
• What’s Worse Game: 14:44–19:21
• Cybersecurity Family Feud/Mistakes: 19:29–26:13
• AI Hype vs. Reality: 26:19–30:26
• The Basics Problem: 30:32–36:35
• Audience Speed Round: 36:41–42:16

Overall Tone & Takeaways

The conversation is witty, candid, and practical, focusing on real-life dilemmas CISOs and their teams face daily. There is a strong emphasis on empathy (for both staff and vendors), healthy skepticism (especially regarding AI tools), and the need to un-silo security as merely a technical or CISO concern—making it clear that business alignment and cultural shifts are at the root of most persistent problems.

For practitioners and vendors alike: Listen more, question the hype, sweat the basics, and always bridge gaps between technical reality and business urgency.