CISO Series Podcast – Episode Summary

Episode Title:

The Difference with AI Red Teaming is We Added the Word AI

Date:

October 14, 2025

Hosts:

David Spark, Andy Ellis

Guest:

Kush Kashyap, Senior Director for GRC at Vanta

Episode Overview

This episode dives into the evolving definitions and approaches to red teaming in the context of AI, the perennial communication challenges between security leaders and business executives, and the real value versus theory in risk quantification. It also features practical discussion on management style, scenario-based risk evaluation, and lively banter about panel moderation and industry culture. The conversation is insightful and candid, offering a mix of actionable tips, healthy skepticism, and a search for relevance over perfection in cybersecurity leadership.

Key Discussion Points

1. Panel Moderation – “You’re Running the Room” (00:25–05:35)

• Problem with Panels: David and Andy kick off with pet peeves about poorly moderated panels, especially the lack of a clear discussion plan and the unprofessional practice of making panelists introduce themselves.
  • Quote:
    • “If a moderator does not have a storyline on a panel, I know it’s going to be a disaster… It means this is not a panel. It’s a bunch of individuals who happen to be orating near each other.” – Andy Ellis (02:13)
  • Both stress the importance of professional moderation and structure to panel discussion, comparing it to TV/radio where there’s never an ad hoc panelist introduction.

2. Communicating with Executives – “Executives Want Clarity, Not a TED Talk” (06:01–10:45)

• Brevity and Relevance: Focus presentations on risk, revenue, and reputation—always in business terms. Avoid diving into technical details unless directly relevant.
  • Quote:
    • “The most important thing is what matters to you is not what matters to them. If this risk is not real to them, it doesn’t matter how long you go talk about it.” – Andy Ellis (06:54)
  • Painful Lesson: Both Andy and Kush share early-career mistakes mistaking technical thoroughness for communication value.
    • “About three minutes in or so, I would notice phones coming out, eyes glazing over, and I realized I was losing the room. That was a very humbling lesson.” – Kush Kashyap (08:33)
  • Tips for Security Pros:
    • Know exactly what belief or decision you want your audience to take away.
    • Frame everything in terms of business impact and actionable consequences.
    • Practice with non-security audiences for feedback and clarity.

3. Coaching vs. Commanding – When To Take Over, When To Ask (10:50–15:04)

• Coaching for Growth: Instinct is to solve team members’ problems instantly, but this creates dependency and stifles development.
  • “If I kept solving everything, I was creating a dependency problem rather than a capability among themselves.” – Kush Kashyap (11:54)
• Practical Tool: The 15-Minute Rule (13:17): Andy shares a process where team members must try to solve an issue for 15 minutes, document attempts and knowledge, then escalate—building self-sufficiency and faster learning.
• Understanding Leader’s Role:
  • Sometimes leaders are needed as sounding boards, therapists, or for information, not just solutions.
  • Identifying what role the team member wants can clarify the interaction.

4. What's Worse? — Shadow IT vs. OT Blind Spots (16:04–21:52)

• Scenario Debate: Not having visibility into your OT environment vs. not knowing all vendors (“shadow IT”).
  • Both Andy and Kush agree that, for the vast majority of businesses today, not knowing all your shadow IT (untracked vendors/apps) is a bigger risk, especially in the SaaS-dominated, post-Covid world.
  • OT caveat: For manufacturing/utilities, OT gaps could be critical, but that's a minority scenario.
  • Quote:
    • “At this point, we are all IT and SaaS companies… not having visibility there matters far more.” – Andy Ellis (17:51)

5. Risk Quantification—Enough with the Academic Models! (21:57–29:01)

• What’s Wrong: Both hosts and guest call out the fixation on precision and academic modeling (particularly FAIR, Monte Carlo) which may be accurate but are frequently useless in executive decision-making.
  • “If you don’t make them believe about a risk, it does not matter if you walk in and say, this is an $80 million risk, because they can always find one thing wrong in your equation.” – Andy Ellis (23:15)
  • Kush’s Standout Point: Emphasize actionable, relatable, and practical models, not perfect accuracy. Advocates for using real-world insurance claim data to inform risk discussion and move away from overcomplication.
    • “Risk quantification doesn’t need to be perfect, it needs to be useful.” – Kush Kashyap (25:30)

Memorable Exchange:

• Andy’s Reaction:
  • “Let me just say that we need to take what Kush just said and like frame that, hang that in the Smithsonian.” (26:06)
• Clarification:
  • Risk quantification is an internal tool to drive prioritization and decision, not an output metric. The end goal is executive understanding and action.

6. AI Red Teaming – What’s Actually New? (29:01–34:35)

• Distinction from Traditional Red Teaming:
  • Traditional: Breaking into systems, testing controls.
  • AI Red Teaming: Probing the OUTPUT of models, looking for unintended behaviors, bias, prompt injection, ongoing adaptation as models/data change.
    • “You’re not just checking if the lock can be picked, but you’re checking if the lock sometimes decides on its own to open for the wrong person.” – Kush Kashyap (30:41)
  • AI red teaming is continuous, not just pre-release.
• Andy’s Perspective:
  • Many vendors say they do continuous “AI red teaming,” often mimicking old patterns.
  • The real distinction in AI is probing for non-deterministic, emergent behaviors (“corner cases”), which demands new methods.
    • “When people say red team AI, I don’t think they’ve actually thought through what they’re talking about… it’s not a lock on a door, it is the lock on every single door.” – Andy Ellis (32:02 & 33:37)
  • Comparison to data model bias, e.g., redlining in lending or autonomous vehicle fatality distribution.

Notable Quotes & Timestamps

• “For me, the single biggest one is when it’s very clear that the moderator doesn’t have a storyline they’re going to tell.” — Andy Ellis (02:00)
• “You always give [executives] an opportunity to page back in what the problem is, what we’re talking about, what the foundation is. Don’t treat them like toddlers, but recognize that they have the attention span of a toddler.” – Andy Ellis (07:21)
• “That was a very humbling lesson… I realized I was losing the room.” – Kush Kashyap (08:33)
• “One of the biggest sins of security professionals is they do all their prep work in front of other security professionals and then they go talk to somebody who is not in security.” – Andy Ellis (10:28)
• “If I kept solving everything, I was creating a dependency problem rather than a capability among themselves.” – Kush Kashyap (11:54)
• “Before you ask for help, you have to write down what you tried, what you know, what your question is...” – Andy Ellis (13:17)
• “If you don’t make them believe about a risk, it does not matter if you walk in and say, this is an $80 million risk, because they can always find one thing wrong in your equation.” – Andy Ellis (23:15)
• “Risk quantification doesn’t need to be perfect, it needs to be useful.” – Kush Kashyap (25:30)
• “Let me just say that we need to take what Kush just said and like frame that, hang that in the Smithsonian.” – Andy Ellis (26:06)
• “You’re not just checking if the lock can be picked, but you’re checking if the lock sometimes decides on its own to open for the wrong person.” – Kush Kashyap (30:41)
• “It’s not a lock on a door, it is the lock on every single door… and the person you’re asking has the master key to everything…” – Andy Ellis (33:37)

Timestamps for Key Segments

• Panel Pet Peeves & Moderation: 00:25–05:35
• Executive Communication Tips: 06:01–10:45
• Coaching vs. Commanding Staff: 10:50–15:04
• What’s Worse? (OT vs. Shadow IT): 16:04–21:52
• Risk Quantification: 21:57–29:01
• AI Red Teaming: 29:01–34:35

Overall Tone & Takeaways

The tone is frank, energetic, collegial, and occasionally irreverent—balancing in-depth practitioner advice with relatable complaints about industry habits. David, Andy, and Kush challenge security dogmas, focus on practical effectiveness, and champion communication skills that bridge the tech-business divide. Most importantly, the episode stays rooted in real-world prioritization: make security and risk advice actionable and relevant, not just theoretically correct.

For Listeners

• If you want effective executive communication, prioritize THEIR needs, not yours. Use business terms, relevant risks, clear impact, and keep it concise.
• Prioritize team coaching over command—help your people develop judgment, not dependency.
• Risk quantification models should drive meaningful action, not just precise numbers. Build for usefulness, not mathematical prettiness.
• AI red teaming is evolving—focus on probing emergent, often non-obvious model behaviors, not just simulating old “break-in” mechanics.
• Great panel moderation matters. Don’t ask panelists to introduce themselves—set context, tell a story, and run the room.

Guest Plug

Vanta is hiring across multiple teams and is promoted as a modern thought leader in GRC and trust management automation using AI.

[End of Summary]