The Difference with AI Red Teaming is We Added the Word AI

Summary

CISO Series Podcast – Episode Summary

Episode Title:

The Difference with AI Red Teaming is We Added the Word AI

Date:

October 14, 2025

Hosts:

David Spark, Andy Ellis

Guest:

Kush Kashyap, Senior Director for GRC at Vanta

---

Episode Overview

This episode dives into the evolving definitions and approaches to red teaming in the context of AI, the perennial communication challenges between security leaders and business executives, and the real value versus theory in risk quantification. It also features practical discussion on management style, scenario-based risk evaluation, and lively banter about panel moderation and industry culture. The conversation is insightful and candid, offering a mix of actionable tips, healthy skepticism, and a search for relevance over perfection in cybersecurity leadership.

---

Key Discussion Points

1. Panel Moderation – “You’re Running the Room” (00:25–05:35)

• Problem with Panels: David and Andy kick off with pet peeves about poorly moderated panels, especially the lack of a clear discussion plan and the unprofessional practice of making panelists introduce themselves.
  • Quote:
    • “If a moderator does not have a storyline on a panel, I know it’s going to be a disaster… It means this is not a panel. It’s a bunch of individuals who happen to be orating near each other.” – Andy Ellis (02:13)
  • Both stress the importance of professional moderation and structure to panel discussion, comparing it to TV/radio where there’s never an ad hoc panelist introduction.

2. Communicating with Executives – “Executives Want Clarity, Not a TED Talk” (06:01–10:45)

• Brevity and Relevance: Focus presentations on risk, revenue, and reputation—always in business terms. Avoid diving into technical details unless directly relevant.
  • Quote:
    • “The most important thing is what matters to you is not what matters to them. If this risk is not real to them, it doesn’t matter how long you go talk about it.” – Andy Ellis (06:54)
  • Painful Lesson: Both Andy and Kush share early-career mistakes mistaking technical thoroughness for communication value.
    • “About three minutes in or so, I would notice phones coming out, eyes glazing over, and I realized I was losing the room. That was a very humbling lesson.” – Kush Kashyap (08:33)
  • Tips for Security Pros:
    • Know exactly what belief or decision you want your audience to take away.
    • Frame everything in terms of business impact and actionable consequences.
    • Practice with non-security audiences for feedback and clarity.

3. Coaching vs. Commanding – When To Take Over, When To Ask (10:50–15:04)

• Coaching for Growth: Instinct is to solve team members’ problems instantly, but this creates dependency and stifles development.
  • “If I kept solving everything, I was creating a dependency problem rather than a capability among themselves.” – Kush Kashyap (11:54)
• Practical Tool: The 15-Minute Rule (13:17): Andy shares a process where team members must try to solve an issue for 15 minutes, document attempts and knowledge, then escalate—building self-sufficiency and faster learning.
• Understanding Leader’s Role:
  • Sometimes leaders are needed as sounding boards, therapists, or for information, not just solutions.
  • Identifying what role the team member wants can clarify the interaction.

4. What's Worse? — Shadow IT vs. OT Blind Spots (16:04–21:52)

• Scenario Debate: Not having visibility into your OT environment vs. not knowing all vendors (“shadow IT”).
  • Both Andy and Kush agree that, for the vast majority of businesses today, not knowing all your shadow IT (untracked vendors/apps) is a bigger risk, especially in the SaaS-dominated, post-Covid world.
  • OT caveat: For manufacturing/utilities, OT gaps could be critical, but that's a minority scenario.
  • Quote:
    • “At this point, we are all IT and SaaS companies… not having visibility there matters far more.” – Andy Ellis (17:51)

5. Risk Quantification—Enough with the Academic Models! (21:57–29:01)

• What’s Wrong: Both hosts and guest call out the fixation on precision and academic modeling (particularly FAIR, Monte Carlo) which may be accurate but are frequently useless in executive decision-making.
  • “If you don’t make them believe about a risk, it does not matter if you walk in and say, this is an $80 million risk, because they can always find one thing wrong in your equation.” – Andy Ellis (23:15)
  • Kush’s Standout Point: Emphasize actionable, relatable, and practical models, not perfect accuracy. Advocates for using real-world insurance claim data to inform risk discussion and move away from overcomplication.
    • “Risk quantification doesn’t need to be perfect, it needs to be useful.” – Kush Kashyap (25:30)

Memorable Exchange:

• Andy’s Reaction:
  • “Let me just say that we need to take what Kush just said and like frame that, hang that in the Smithsonian.” (26:06)
• Clarification:
  • Risk quantification is an internal tool to drive prioritization and decision, not an output metric. The end goal is executive understanding and action.

6. AI Red Teaming – What’s Actually New? (29:01–34:35)

• Distinction from Traditional Red Teaming:
  • Traditional: Breaking into systems, testing controls.
  • AI Red Teaming: Probing the OUTPUT of models, looking for unintended behaviors, bias, prompt injection, ongoing adaptation as models/data change.
    • “You’re not just checking if the lock can be picked, but you’re checking if the lock sometimes decides on its own to open for the wrong person.” – Kush Kashyap (30:41)
  • AI red teaming is continuous, not just pre-release.
• Andy’s Perspective:
  • Many vendors say they do continuous “AI red teaming,” often mimicking old patterns.
  • The real distinction in AI is probing for non-deterministic, emergent behaviors (“corner cases”), which demands new methods.
    • “When people say red team AI, I don’t think they’ve actually thought through what they’re talking about… it’s not a lock on a door, it is the lock on every single door.” – Andy Ellis (32:02 & 33:37)
  • Comparison to data model bias, e.g., redlining in lending or autonomous vehicle fatality distribution.

---

Notable Quotes & Timestamps

• “For me, the single biggest one is when it’s very clear that the moderator doesn’t have a storyline they’re going to tell.” — Andy Ellis (02:00)
• “You always give [executives] an opportunity to page back in what the problem is, what we’re talking about, what the foundation is. Don’t treat them like toddlers, but recognize that they have the attention span of a toddler.” – Andy Ellis (07:21)
• “That was a very humbling lesson… I realized I was losing the room.” – Kush Kashyap (08:33)
• “One of the biggest sins of security professionals is they do all their prep work in front of other security professionals and then they go talk to somebody who is not in security.” – Andy Ellis (10:28)
• “If I kept solving everything, I was creating a dependency problem rather than a capability among themselves.” – Kush Kashyap (11:54)
• “Before you ask for help, you have to write down what you tried, what you know, what your question is...” – Andy Ellis (13:17)
• “If you don’t make them believe about a risk, it does not matter if you walk in and say, this is an $80 million risk, because they can always find one thing wrong in your equation.” – Andy Ellis (23:15)
• “Risk quantification doesn’t need to be perfect, it needs to be useful.” – Kush Kashyap (25:30)
• “Let me just say that we need to take what Kush just said and like frame that, hang that in the Smithsonian.” – Andy Ellis (26:06)
• “You’re not just checking if the lock can be picked, but you’re checking if the lock sometimes decides on its own to open for the wrong person.” – Kush Kashyap (30:41)
• “It’s not a lock on a door, it is the lock on every single door… and the person you’re asking has the master key to everything…” – Andy Ellis (33:37)

---

Timestamps for Key Segments

• Panel Pet Peeves & Moderation: 00:25–05:35
• Executive Communication Tips: 06:01–10:45
• Coaching vs. Commanding Staff: 10:50–15:04
• What’s Worse? (OT vs. Shadow IT): 16:04–21:52
• Risk Quantification: 21:57–29:01
• AI Red Teaming: 29:01–34:35

---

Overall Tone & Takeaways

The tone is frank, energetic, collegial, and occasionally irreverent—balancing in-depth practitioner advice with relatable complaints about industry habits. David, Andy, and Kush challenge security dogmas, focus on practical effectiveness, and champion communication skills that bridge the tech-business divide. Most importantly, the episode stays rooted in real-world prioritization: make security and risk advice actionable and relevant, not just theoretically correct.

---

For Listeners

• If you want effective executive communication, prioritize THEIR needs, not yours. Use business terms, relevant risks, clear impact, and keep it concise.
• Prioritize team coaching over command—help your people develop judgment, not dependency.
• Risk quantification models should drive meaningful action, not just precise numbers. Build for usefulness, not mathematical prettiness.
• AI red teaming is evolving—focus on probing emergent, often non-obvious model behaviors, not just simulating old “break-in” mechanics.
• Great panel moderation matters. Don’t ask panelists to introduce themselves—set context, tell a story, and run the room.

---

Guest Plug

Vanta is hiring across multiple teams and is promoted as a modern thought leader in GRC and trust management automation using AI.

---

[End of Summary]

Transcript

[Narrator/Announcer]

10 second security tip.

[Kush Kashyap]

Go. Don't chase every risk at once. Focus on the few that could truly break your business and communicate them in business terms.

[Narrator/Announcer]

It's time to begin the CISO Series Podcast.

[David Spark]

Welcome to the CISO Series Podcast. My name is David Spark. I am the producer of said CISO series and joining me, a legendary ciso. And by the way, I want you to know that I'm the one who gave him that title. I don't know if anyone else has called it, but I believe this to be true. A legendary ciso, none other than Andy Ellis. Andy, say hello to our audience.

[Andy Ellis]

Hello to our audience. See, this doesn't work as well as the whole good morning, good afternoon, good evening thing. I got to work on this.

[David Spark]

You know what? Nobody cares about that at all whatsoever.

[Andy Ellis]

I do.

[David Spark]

I know you know what you're doing. You're fighting to find your Johnny Carson golf swing.

[Andy Ellis]

I am. I don't have that moment.

[David Spark]

Yeah, he's one of these. By the way, I'll throw this out to the audience if you want to recommend what Andy could have sort of his welcome, his sign off, whatever it is going to be to be the equivalent of his Johnny Carson golf swing for our younger viewers or younger listeners. Look it up. I will say that. Look it up.

[Andy Ellis]

Look it up.

[David Spark]

We're [e-mail] our sponsor for today's episode, a spectacular sponsor of the CISO series. We adore having them on board. It is Vanta Automate Compliance. Manage Risk and Accelerate Trust with AI. More about just that a little bit later in the show. But first I'm going to ask you this question. Andy and I just attended one of these recently and here's my question. Within the first minute, maybe I'll give you two minutes of a panel session. How do you know it's going to go south? What's a telltale sign? I have my answer to this, but I want to know what you.

[Andy Ellis]

If you're within the first minute, only the moderator has spoken, right? So you have to just based on the moderator if you're going for that one. But for me, the single biggest one is when it's very clear that the moderator doesn't have a storyline they're going to tell, which can be. They ask a question to anybody on the panel. No, no, your first question, you're naming somebody on the panel, they're going to answer it like very similar to how we run the podcast where you toss to me, you toss to our guest, maybe then you toss to the guest, you Toss to me, because you have a storyline. If a moderator does not have a storyline on a panel, I know it's going to be a disaster, even if there might be a great panelist. But it means this is not a panel. It's a bunch of individuals who happen to be orating near each other.

[David Spark]

That is very good. And I agree with you that that is bad. But I think. I know.

[Andy Ellis]

Okay, we can just cut the recording right there.

[David Spark]

Cut the recording. David agrees, but I think far, far worse, and you hear about this way before, and I've said this before, is. And it drives me bat s crazy is when the moderator goes, all right, why don't you all introduce yourself?

[Andy Ellis]

Yes, I know that's one of your pet peeves.

[David Spark]

I want to scream. I want to truly scream when that happens. And I just want to stress there is not a single talk show host on television or radio who does this. Not a single one in the history of all media. They don't do this for the sole reason. It's horrifically unprofessional.

[Andy Ellis]

Well, not only is it unprofessional, you're putting somebody on the spot. Like, look, I will call myself Legendary Ciso because I thought that was funny when you named me, nicknamed me that. I put it on my black hat badge. Like, I have enough healthy ego that I can get away with that, and I'm happy and people make fun of me.

[David Spark]

And there's a story being that I'm the one who gave you the name.

[Andy Ellis]

And there's a story. But I've done panels where I'm the moderator and somebody sends me their bio and they, like, downplay everything they did, and I go look them up, and I'm like, this person is amazing. And if you ask somebody to introduce themselves, they're going to be very timid and they're gonna downplay. Whereas you have the opportunity to say, look, I have this person on the panel because they're the fields medalist in.

[David Spark]

This discipline a thousand percent. Because first of all, two things. People feel better when they are introduced and people talk them up. Second of all, if you're the moderator, it's your show. You invited these people here. It's like when you say introduce yourself, like, who are these four yahoos? Explain who you are.

[Andy Ellis]

Yeah, I think the challenge is that people who are moderating think they're running a roundtable. Like, in a roundtable setting where I've got, like, 15 people, it's fine for me to say, the first time you speak, introduce yourself. And if you Answer. Introduce yourself with just your title, but when you're going to ask a question later, then give your full intro. That's fine. Roundtable setting with no audience.

[David Spark]

Yes, that's a different story. That's a different story, but it's very different.

[Andy Ellis]

But I think people bring that habit in.

[David Spark]

By the way, I have lots more complaints about panel sessions, but that's the doozy. And I just again, and never in the history of time has anyone begun a panel session asking people to introduce himself. Was it a good panel session? It's consistently awful because they didn't know what they were doing out of the gate.

[Andy Ellis]

I will say there are amazing moderators out there. I like to think I'm pretty good at it.

[David Spark]

I also think I'm pretty good at this.

[Andy Ellis]

If you're going to do a panel session anywhere, then you should get a professional moderator, not just a good speaker, and say, hey, we want you to do this. You're doing a webinar. You should have a professional who knows what they're doing.

[David Spark]

You're running the room.

[Andy Ellis]

You want somebody who runs the room.

[David Spark]

Yeah, people don't think that they have to run the room. Anyway, plenty more about this and I'm.

[Andy Ellis]

For hire if anybody needs a professional moderator. So is David.

[David Spark]

Plenty more on this coming up later. All right, let's bring on our guest. So the first time she's been on any of the CISO series shows. So thrilled that she's here. It is the senior director for GRC over at Vanta, our sponsor guest, none other than Kush Kashyap. Thank you so much for joining us, Kush.

[Kush Kashyap]

So happy to be here.

[Narrator/Announcer]

Are we having communication issues?

[David Spark]

Executives want clarity, not a TED Talk, said Will Klosofski of Appalachia Technologies. Too many security professionals lose executive attention to smartphones and glazed over stairs. He laid out some basic principles for security professionals when talking to executives, such as focusing on risk, revenue, reputation. Oh, my God, this is your language. Kush and all of his tips focus on brevity, visual impact, and commanding the room with confidence. So I'm going to start with you, Andy, though, on this. What's worked best for you when presenting to the C Suite? I know you got a lot of experience here. And more importantly, which of those lessons did you have to learn after seeing someone's eyes glaze over in a meeting? Like, did you get a signal like, oh, I'm losing them?

[Andy Ellis]

Oh, I've certainly gotten that signal a lot in my career. And I think the most important thing is what matters to you is not what matters to them. Right. People are so tied into what I know, what I believe. It's all about what the other person believes. If this risk is not real to them, it doesn't matter how long you go talk about it. It's never going to resonate. And so every one of Will's suggestions is amazing. By the way, people should go read that LinkedIn post. It's really quick, very brief, as important. But there's one he misses, which is you should recognize that executives are like computers that are way overworked. Right. They're constantly context switching. The moment you start speaking, they have no idea who you are, let alone what you're talking about. It takes them a few minutes to page in context. You might have talked to them about this problem a month ago. They do not remember that conversation. So if you walk in and you continue from where you were last time, you've lost them. So you have to sometimes say, as we mentioned before, X now. Yes. You don't just lead in with why. You always give them an opportunity to page back in what the problem is, what we're talking about, what the foundation is. Don't treat them like toddlers, but recognize that they have the attention span of a toddler.

[David Spark]

Right. And I also refer to this. I call this often the curse of knowledge. You are walking in the room with everything already in your head, and you kind of have this assumption. Everyone's up to speed where I'm at. Right, right. And they're not usually. All right, kush, I throw this to you. Same thing talking to executives. And I'll kind of ask the same question. Vandy, did you have one of these? Oh, I'm losing a moment. I have to switch gears or for the next time I do it differently. Let us know.

[Kush Kashyap]

Absolutely. Most security professionals are so incredibly smart and technically strong. What happened with me was much earlier in my career, I thought being thorough meant walking through every control, every risk scenario, every framework. About three minutes in or so, I would notice phones coming out, eyes glazing over, and I realized I was losing the room. That was a very humbling lesson. And what I came to understand is that's not what executives are looking for. They don't want a lecture. They want clarity more than anything else. So it's definitely all the knowledge that we have in our mind. How do we distill it for the audience who's sitting in front of us? And how do we make it as relevant for them as possible?

[David Spark]

Very, very good point. All right, so I'M going to ask you, closing this out quick, one or two tips to sort of capture the audience and maybe a tip of keeping them engaged when you're a few minutes in. Andy.

[Andy Ellis]

So I think the first one is recognize what you need them to believe at the end of this conversation. What are you trying to get out of this conversation and how is everything you're saying driving towards that? Because if you don't know what you're trying to achieve, they certainly aren't going to fill it in for you.

[Kush Kashyap]

Kush, on top of what Andy mentioned, I would say also try to clarify what's the real business impact if something or the risk or the incident materializes? How much would it cost us and what can we do about it? Make the business context into your talking points.

[David Spark]

I will also throw this out, have a trial run in front of another audience. I've done this for like really short sub, 15 minute presentations I've done. And I must say I remember doing it like, because I really wanted to nail it. I did like five different times in front of five different audiences. And I swear what it was the first time versus the fifth time, night and day. Like the feedback I got was phenomenal.

[Andy Ellis]

Absolutely. And different audiences is important. One of the biggest sins of security professionals is they do all their prep work in front of other security professionals and then they go talk to somebody who is not in security. And so you didn't get an audience that didn't have your knowledge.

[Narrator/Announcer]

What works, what's not working.

[David Spark]

How do you react when your staff brings you a problem? Now, your first instinct might be to be a problem solver instead. Elizabeth Lotardo in a Harvard Business Review piece suggested asking, quote, what have you tried? So asking the individual. It's always tempting to jump right into the solution mode. Well, you need to go do this, this and this, you know, kind of thing. But Littardo argued that constantly solving your team's problems robs them of chances to develop skills and creates dependency on you. So sounds like great advice, but how much can we apply this to the realities of cybersecurity? I mean, when there's an active incident, you don't always have the time for teachable moments. So when do you coach versus command? And I asked you first, Kush, how do you know when it's time to abandon the coaching and solve the problem yourself? Like give us scenarios, what it is and do you do this yourself?

[Kush Kashyap]

I think it's all about the context. Understand the context of what you're trying to solve for. Is it an Active incident when you don't have the luxury of turning it into a teachable moment. Or are you staying in that mode all the time? Because people would bring me a problem and my instinct was to solve it on the spot because it felt faster and more efficient. But however, the downside was that my team wasn't getting the chance to build their own judgment. And over time, I realized that if I kept solving everything, I was creating a dependency problem rather than a capability among themselves.

[David Spark]

And they would come running to you or running to others at the same time.

[Kush Kashyap]

Yes. And that's when I started practicing the discipline of asking, what have you tried? What options do you see? What worked well, what didn't work as well. It was surprising how often they already had good instincts. They just needed a space to validate them. And when you give people a chance, they grow. The team becomes much more resilient.

[David Spark]

And you bring up a very good point of asking questions. And I'm going to throw this to you, Andy. Often people's success is they don't know what questions they should be asking or answering, for that matter. I mean, I. I know I've run into this when I've tried to do something that I was out of my comfort zone. And I remember someone hitting me with all these questions and I realized, oh, my God, I'm not ready for this at all kind of a thing. I didn't even think to ask these questions.

[Andy Ellis]

So Matt Ringle's coworker I used to have, he had what he called the 15 minute rule. And he said, first you must try, then you must ask. He said, you have to try to solve a problem for 15 minutes, then you can go ask for help. But before you ask for help, you have to write down what you tried, what you know, what your question is, partly so that you preempt that conversation. When somebody says, have you tried? Have you tried? Have you tried? You walk in and you say, look, I'm having a problem with X. Here's what I have done, here's what I know now, what should I do? Or what's next? So it helps you. It also means when somebody asks you novel questions, you can update your own model for that process at the end of 15 minutes. Because what I love about it, the number of times that I have hit a wall, and then I stopped and I said, what do I know? What have I tried? And I solved the problem in that step that I'd already solved the problem, just hadn't realized it. And so it creates this discipline, the flip Side of this, which I also want to talk about, is certainly as you get more senior in the organization, you have senior people working for you. Sometimes when they bring you a problem, it's not because they need help solving it. Sometimes they're bringing you a problem so that you're just forewarned. They're telling you, hey, I'm having this issue with a colleague, whatever. They don't need you to do anything. They're just telling you so in case that colleague escalates. You've heard it from your team before, you hear it from somewhere else. Sometimes they're bringing it to because they need a therapist. Like, I am so pissed off at this person I'm working with. I just need to tell somebody. And the only safe person to tell is my boss. So I'm telling you. So I have found it to be really helpful when somebody brings me a challenge to say, what role do you need me to play? Am I solving the problem for you? Am I coaching you through it? Am I your therapist? Or am I just hearing an escalation?

[David Spark]

This is also good parenting advice when your kids come complaining.

[Andy Ellis]

Oh, absolutely.

[David Spark]

Before I go any further, I do want to tell you about our spectacular sponsor, and that would be Vanta Automate. Centralize and scale your GRC program with Vanta. If you're not already have Vanta on your radar, you should be checking them out. Vanta's trust management platform automates key areas of your GRC program, including compliance, internal and third party risk, and customer trust. And it streamlines the way you gather and manage information. And the impact is actually. It's real. It's real. A recent IDC analysis found that compliance teams using Vanta are, get ready for this. 129% more productive. So get started at Vanta. Go to vanta.comciso the reason you want to do vanta.comciso is they know that we sent you there and they know that they love sponsoring the show. So Please go to vanta.com CISO.

[Narrator/Announcer]

It'S time to play what's the Worse.

[David Spark]

All right. You know how this game is played, Kush.

[Andy Ellis]

I think so your job, your goal is just agree with whatever I say.

[David Spark]

No, don't agree with whatever he says. Your goal is to answer it as truthfully as you would like. So essentially it's a risk management exercise. And you talked about risk at the very beginning. So you're going to answer which is the worst scenario in terms of risk here. All right. This comes from Dustin Sachs of Cyber Risk Collective of which he's contributed lots of wonderful what's worse scenarios. Just so you know, Kush, I make Andy answer first and you can agree or disagree with him. I love it when you disagree.

[Andy Ellis]

I love it when you agree.

[David Spark]

Here you go. Very short, two scenarios. Scenario number one, not having visibility into your OT environment, operational technology, or not knowing all the vendors your company has contacts with, essentially shadow it. Which one is worse?

[Andy Ellis]

So I hate to start with it depends.

[David Spark]

By the way, I'm telling you right now, Kush, it depends. It's not an option.

[Andy Ellis]

It's not an option. So I'm going to say that like 90% of the time it's going to be one of them. And I wanted to tell you the 10% for the other one, right?

[David Spark]

Because like what, like some companies have no OT whatsoever, right?

[Andy Ellis]

Yeah. I'm going to say that for 90% of the companies, it is worse not knowing who your shadow it because that's how you run your business. Your business is run on it, on SaaS. You don't know who these people are. Like, that's gonna be your bigger problem. There's 10% of you for whom. No, no, no, that's not true. Mostly like OT companies. Like, if you're a manufacturing business, everything you do is probably ot. That matters far more for you. So that might arguably worse, right?

[David Spark]

It depends on the actual business itself.

[Andy Ellis]

Like, yeah, if you don't know that GE is running all of your power and all of your manufacturing equipment, we have serious issues. But I'm going to say that's not the norm. So I'm going to go with it's much worse to not have visibility into like the way in which you actually operate your business. Because at this point, we are all IT and SaaS companies. And not having visibility there matters far more than not knowing who your H VAC vendor is.

[David Spark]

But if you're running the electric grid.

[Andy Ellis]

If you're running the electric grid, you are an OT shop, in which case your shadow it is your ot. So I'm just, but I'm just going shadow it's is the bigger challenge.

[David Spark]

But hold on. But your shadow it is your ot. Would that make them equal problems here?

[Andy Ellis]

Well, no, no, sorry. What I'm referring to at that point is the concept of shadow it. Your only it is the electrical grid is the electrical grid, which is ot. So at this point, I could make an argument that shadow it is the same thing as OT at that point, but that would be silly. Let's just go with shadow it for the Vast majority of our listeners and those of you in manufacturing and power can tell me I'm wrong and I'm wrong inside your context.

[David Spark]

Yes. So it depends. Definitely plays a good hand here. I will agree on that. Yeah. All right, Kush, do you agree or disagree with Andy?

[Kush Kashyap]

I completely agree.

[David Spark]

Completely agree. Oh, gosh.

[Andy Ellis]

David hates those.

[David Spark]

I'm so disappointed. All right. Why do you completely agree? For the same reasons or other reasons? So that you can. You can agree for different reasons. That's to me, by the way. To me, that's like disagreeing.

[Kush Kashyap]

I do agree. The supply chain attack surface is a big, big problem for all industry verticals. And I do think for the mass majority shadow, it does pose a more imminent threat. Just like the industry verticals and the companies who rely on OT and OT security is a strong, strong capability for them are far few. And that's why for the majority of them, I completely agreed with Andy on it. And it's. That's my option number two there.

[Andy Ellis]

Yeah. I think seven years ago, if we go back to pre Covid times, we would have anticipated having a different answer. Because at that point, almost all it was in house, in your building. There was very little SaaS. And the SaaS that was there was all big companies like, oh, yeah, we had Salesforce.

[David Spark]

Oh, you're just saying seven years. I would go even further back than that.

[Andy Ellis]

No, I actually think only seven years ago, SaaS was not something. Look, I was investing in SaaS security seven years ago. People thought I was insane. Like, who would have this much SaaS? But yes, I had 500 SaaS vendors seven years ago. But even then, we really thought that, like, OT was gonna be our biggest blind spot, that we really would care about all the OT in our corporate headquarters. Guess what? Most companies don't have headquarters anymore. So I think the shift in Covid the enablement of Everybody to buy SaaS, that, yes, started more than seven years ago. But I think seven years is where, like, we still believed ot would be the bigger risk. And now it's clearly not.

[David Spark]

Well, those in OT would argue.

[Andy Ellis]

I'm not saying it's not a risk, just to be clear, like, this is not a binary. If only care about one.

[David Spark]

But others would tell you, like, you know, if the water stopped pumping or the electric rig goes down, you got a way bigger problem than who's using a different version of Dropbox.

[Andy Ellis]

But I'm not worried about the OT in my house or my headquarters depriving me of water.

[David Spark]

Correct.

[Andy Ellis]

I'm worried about my next Door neighbor cutting into the water main. This happened to me over the summer. Like, I lost water. Wasn't my OT problem. It was my utility.

[David Spark]

Hold on. Why did you hold it? What happened to your neighbor? What did they do?

[Andy Ellis]

They were doing construction, and they cut into the water main. They were trying to widen their main. And occasionally when you do this, you sort of screw up and you cut right into the main. So they had to shut off water for our whole neighborhood. And so we're all out, and it's one of the hottest days of the summer. And so all of a sudden, they've got like, 12 people come out of their houses going, when do we get water back? And they're like, if we knew, you would already have it.

[Narrator/Announcer]

Please, Enough, no more.

[David Spark]

Our subject today on Please Enough, no More is risk quantification. We just played it with what's worse? And I'm going to ask you, Andy, I know you've got tons of opinions on this subject. What have you heard enough about with risk quantification? And what would you like to hear a lot more? So what's going right and what's going wrong, and where would you like it to go?

[Andy Ellis]

So I think that there's this underlying problem under the risk quantification space that people functionally believe that humans and organizations are rational decision makers. And I'm going to put that in big, giant air quotes, which is all you have to do is present an equation that says that we will get more out of our investment by doing this, and therefore we must do it. And that presumption just underlies everything. Like, if we can get better at our estimates, if we can have more data, then here's the reality. Almost nobody is making quantitative decisions anywhere in the business outside of very narrow industry verticals. It all comes down to the belief sets of executives. And if you don't make them believe about a risk, it does not matter if you walk in and say, this is an $80 million risk, because they can always find one thing wrong in your equation, at which point they get to ignore everything you said. So you have to play to their beliefs, because what you're really trying to do is change their worldview. You do not cause someone to make a different decision by presenting them with a perfectly logical case. Very few people operate in that. You get them to make a different decision by having changed the way they view the world such that this new decision makes sense to them.

[David Spark]

I like that. All right, I'm going to ask you the same question. You can feed off of what Andy said or not? What have you heard enough about with risk quantification, Kush? And what would you like to hear a lot more.

[Kush Kashyap]

I want to talk about a different methodology and approach of risk quantification that I agree and don't agree with. For me, I've heard enough about risk quantifications being dominated by fare like models that fixate on asset values and key risk scenarios and then layer on Monte Carlo simulations and more statistical modeling. Don't get me wrong, these models have value, but too often they get so academic that they paralyze the conversation. You spend months, I have seen companies years debating assumptions and by the time you produce a number on some key risks, that business has already moved on. I would also say no to the notion that this is the only credible way to do quantification. The pursuit of precision can sometimes take us further away from actually helping executives make better decisions. What I would love to see more of is using more data that's outside of our purview. For example, using the depth of cyber insurance data because it reflects real world incident losses and claim playouts, and then contextualizing that data for your own environment that gives you a grounded, credible baseline to talk about risk in financial terms with without overcomplicating it. Honestly, the space is ripe for transformation. For too long, risk quantification has been about building models that impress security folks but fail to resonate with executives. It's time we flip that, make the output simple, relatable, and rooted in business reality. I also want to say one thing to Andy. Risk quantification doesn't need to be perfect, it needs to be useful.

[David Spark]

All right. By the way, Andy loves you.

[Andy Ellis]

Yes. I don't want to be inappropriate, but oh my God, I have the biggest work crush ever right now.

[David Spark]

By the way, I may just grab that clip of you, Andy. Just the video of Andy, just the way he was falling over laughing. Oh my God. I've never seen Andy agree with someone with his body language more than what I just saw there.

[Andy Ellis]

Like, let me just say that we need to take what Kush just said and like frame that, hang that in the Smithsonian.

[David Spark]

By the way, there will be a transcript of this episode.

[Andy Ellis]

Make it a requirement of everybody who has a CISSP needs to read that. Like that entire monologue is so correct. Like normally I'm here disagreeing with everybody in the risque quant space. Everything someone said, that's it exactly. Like risk quantification does need to be more accurate. It needs to be useful.

[David Spark]

Okay, so I want to Hear your feedback to that line, because I like that line. But I don't. I'll be honest, I don't fully understand what that means. What does it mean that it doesn't need to be accurate, but useful, Obviously. I'll ask you, Kush. But first you, Andy.

[Andy Ellis]

So the challenge is people try to get to precision. They try to say something like, this is a $40 million risk. And now the problem is the moment you say it's a $40 million risk, everybody's like, oh, my God, is it really 40? Is it 39? Is it 41? And so you keep fighting over precision and accuracy when, look, here's what matters. This is the biggest risk. Why is it the biggest risk? Because these crown jewels are at risk from a credible threat that can do this. You tell the story. The point of risk quantification was not to get to a number that you delivered. It was. So it's an interim step for you to look at and say, oh, 40 million is bigger than $30. Let's go look at the $40 million risk. But now I have to sell it. And if I'm selling it to somebody who wants to hear 40 million, I use 40 million. If I'm selling it to somebody who wants to hear the fairy tale of how Little Red Riding Hood is getting lost in the woods and eaten, and that's going to cost 40 million, I tell them the fairy tale. I don't mention 40 million. It's about the utility of getting somebody else to believe in the risk and invest in it. That's what matters. This number is completely irrelevant without action.

[David Spark]

All right, I'm going to ask the same question. Andy and I want to know your take on the same question of what does it mean? I mean, I think Andy gave a good answer, but I want to know if you want to add or double down or edit. What does it mean? You don't have to be accurate. You have to be useful. What does that useful versus accurate mean?

[Kush Kashyap]

Yeah, because as I was saying before, accuracy comes from. For certain people who have been in this space for a long time, following very precise models, applying statistic probabilistic reasoning to a lot of the numbers for few critical risk scenarios go into asset impact quantification and whatnot. If you steer away from that and focus more on Andy's points around, what is the business value? What are your crown jewels? What is it that is keeping the executives and the leaders stay up at night? Focus on the so what? Focus on the key risk. Focus on why it's Important for us to come here and talk about them and where our investments are already there, where they need to be and where they need to be made. Further, I think that is a much better conversation than fixating on the numbers and the accuracy and the modeling and the academic side of things.

[Narrator/Announcer]

What about this AI security challenge?

[David Spark]

Is AI red teaming really that different or just a specialized version of what we've always done? Patrick Sullivan of Aline makes the case that it deserves to be treated as something new. While traditional red teaming focuses on breaking into systems to test security controls, AI red teaming examines what comes out testing whether models behave fairly, safely and as intended. He breaks down three risk bias training, data that creates stereotyped outputs, model decisions based on probabilities rather than fixed logic, and where users can manipulate responses through prompt injection. AI risks aren't just about external attacks. You're trying to figure out what the model learned to do without being explicitly taught. So he makes some really good arguments that it's very different. There's a lot of new behaviors here. So, Kush, does that distinction make sense to you? And what shifts have you noticed in red teaming methods even in recent months as this tech is changing? And just also a point. I had like three meetings in the first day of Black Hat this year where they were all offering sort of automated continuous AI red teaming, and they were all saying exactly the same thing and just saying, oh, but we're better than them. And I had to believe them. So what's your take here?

[Kush Kashyap]

Yeah, so I do think that AI red teaming deserves to be treated as something distinct. Traditional red teaming is absolutely still very critical, but it serves a different purpose. It's about breaking into systems and testing whether your controls hold up. AI red teaming, on the other hand, is about probing the model's behavior. You're not just checking if the lock can be picked, but you're checking if the lock sometimes decides on its own to open for the wrong person. So what I've noticed, even in recent months, I have noticed something very similar. AI red teaming is moving from one off exercises into continuous pipelines. It's no longer enough to test a model once before launch. You need ongoing evaluation because the data shifts, the model evolves, and new jailbreaks appear every week. So for me, both disciplines are very important, but they're not interchangeable.

[David Spark]

All right, very, very good point. And I like the scenario of the lock just deciding to let someone in for some random reason and lets you.

[Andy Ellis]

Into a different doorway than the one it was. Actually, that's the Biggest problem with some of the AIs.

[David Spark]

All right, so again echoing what was said, there are a lot of these solutions out there fairly recently. Andy too. And are they just trying to mimic a historical behavior of red teaming or are they trying to truly red team AI?

[Andy Ellis]

I think the challenge is that when people say red team AI, I don't think they've actually thought through what they're talking about.

[David Spark]

I would agree with that statement. Yes.

[Andy Ellis]

Because sometimes the way this question is phrased is it actually comes and talks really about models from data analysis. Like we've gone and think through the old redlining conversations that the Community Reinvestment act did away with from what was that 20 years ago? Those were models that was a model that said, oh, if you live in a neighborhood that is poor, you're less likely to repay loans so we're going to charge you more money. Right. And at the end of the day the actual math behind the model is actually sound. But the impact of the model was very significantly problematic. Basically said if you live in a poor reed minority neighborhood, we're not going to loan you money, it's going to cost you more money. We decided eh, we should stop doing that. So there's one set that's all around. When we say the models there, it's not LLMs. These are data models of how are you making decisions based on data that needs one set of red teaming that has nothing to do with breaking in and everything to do with how do you find the corner cases. Imagine autonomous vehicles, right? If tomorrow we could put waymos on the street or whoever you want, like only autonomous vehicles, no more human driven vehicles. And we drop the fatality rate from driving down from 35,000 people a year in America dying to 3,000 people a year in America dying. Would you think that was an improvement? But what if I told you that they were all going to be insert your favorite minority type here. It will only kill black people, you know, Indians, Sikhs, Jews, whatever. But we're only going to kill 3,000. Like look how many lives we're saving. We would all be like, we have serious problems with that model. If like it's going to consciously choose who to kill and always choose the same direction. Seems like a poor choice. So that's a very different problem than the LLM space where often we're taking LLM and we're handing it like it's not a lock on a door, it is the lock on every single door. And so you're walking up to a building and when you're saying, look, I need to get access to room 110A. Can I have the file in the third drawer of the second file cabinet. And the person you're asking has the master key to everything and says, sure, and opens up every door and says, by the way, here's how much money your boss makes. Like, that's a different level of red teaming. Now. That looks a lot closer to like, WAF bypass. There's a lot of comparabilities to things that have happened in the past. It's just a little more natural language than the traditional fuzzing has been.

[David Spark]

Very, very good point. And that brings us to the very end of our conversation right here. I want to thank Kush for joining us for our discussion. The only thing I could disagree with you on Kush is your amount of.

[Andy Ellis]

Agreeing with Andy, because Kush is brilliant. Kush is legendary.

[David Spark]

Already a little too much agreeing with Andy. Kush Kashyap, who is the senior director for GRC at Vanta and Vanta. Huge thanks to Vanta for sponsoring us. Remember, if you want that 1.29x increase in productivity over what you're doing, you got to go check out Vanta and go to vanta.com CISO Kush, I'm going to let you have the last word, but let me also ask, is Vanta hiring?

[Kush Kashyap]

Yes, Vanta is hiring. And we have a lot of positions open, not just in the security team, but in broader Vanta product go to market finance everywhere.

[David Spark]

Awesome. Excellent all the way around. Thank you very much, Kush. Thank you very much. Anti thank you to our audience. I've said this before. I'll say it again. I truly, truly mean it. I'm not blowing smoke. I call you a legendary audience like I call Andy a legendary ciso.

[Andy Ellis]

Oh, see, now you've just taken it away. Like, they have to be something. They're an epic audience.

[David Spark]

Epic audience.

[Andy Ellis]

They're an epic audience, but if they.

[David Spark]

Have stuck with us since we started eight years ago, they are legendary.

[Andy Ellis]

Oh, they can be mythical. One step above.

[David Spark]

I'd be interested to know if anyone listening has actually heard episode one, which, by the way, don't recommend anyone. Go back and listen.

[Andy Ellis]

No, don't.

[David Spark]

It's embarrassing.

[Andy Ellis]

Follow David around at the next conference and play it on a speaker.

[David Spark]

You know what? To get me to listen to it, we'd have to have, like, a Clockwork Orange scenario where you open up my ears and I'm required to hear my very first episode. And I'm struggling to deal with the pain of the that altogether. I would not enjoy that one whatsoever. All right, we appreciate our audience. We appreciate your contribution. Send me more what's Worse Scenarios Please send me lots more what's Worse Scenarios I appreciate that and any other kind of cool segment or topic of conversation you see on LinkedIn or Story. We love that stuff. Thank you for contributing and listening to.

[Narrator/Announcer]

The CISO Series podcast that wraps up another episode. If you haven't subscribed to the podcast, please do. We have lots more shows on our website cisoseries.com Please join us on Fridays for our live shows Super Cyber Friday, our virtual meetup and Cybersecurity Headlines Week in Review. This show thrives on your input. Go to the Participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you're interested in sponsoring the podcast, contact David Spark [e-mail] thank you for listening to the CISO Series Podcast.