AI Confidence: It's a Trap! (LIVE in San Francisco)

Summary

CISO Series Podcast – “AI Confidence: It's a Trap!” (LIVE in San Francisco)

Podcast Hosts: David Spark, Mike Johnson (CISO, Rivian), Andy Ellis
Guest: Sarah Madden (CISO, Convera)
Event: BSides San Francisco 2026
Date: May 5, 2026

---

Episode Overview

This live episode, recorded at BSides San Francisco 2026, dives deep into the challenges and opportunities of artificial intelligence (AI) in cybersecurity, vendor selection pitfalls, the realities of incident response simulations, and the evolving nature of technical conferences. With a lively audience, the panelists debate the “AI confidence trap,” reflect on security leadership mistakes, dissect the difference between technical and people problems, and champion a culture of continuous learning and cross-functional collaboration.

---

Key Discussion Points and Insights

1. Security Leadership: Lessons from Mistakes

• Sarah Madden on “I told you so” Moments
  “It’s not our jobs to say I told you so. It’s our jobs to manage risk and never have that happen.” (00:03)
  • Biggest regret: bending to business pressure against better judgment.
  • Focus on managing risk over proving oneself right after the fact.

---

2. Vendor Selection: Avoiding Long-Term Traps

• Common Pattern:
  Evaluate a few vendors, pick the best demo, sign a long contract, then swap out mid-cycle due to unforeseen changes (mergers, better tech, etc.).

• Mike Johnson’s Strategy (04:49):

  • Don’t sign 3-year deals on first engagement:
    “The first step is don’t sign a three-year deal the first time that you’re working with a vendor, period. Period, full stop.” (04:49)
  • Leverage peer CISO networks to shortlist vendors.
  • Build clear requirements, conduct thorough proof of concept.
  • Accept occasional unavoidable bad outcomes but mitigate risk with careful initial steps.

    “At the end of the day, you can’t predict the future.” (06:34)

• Sarah Madden on Negotiating and People Power (06:59):

  • Got locked into three-year deals during greenfield buildout, but now shifts renewals to 1–2 years for flexibility.
  • Stresses the need to air grievances and leverage renewal threats to push for better vendor quality.
  • Calls out SaaS instability and vendor trend towards shipping lower-quality releases.
  • Importance of matching tools with internal competency:

    “It’s as much about finding the right tool as it is making sure you have the skill sets on your team to manage that.” (09:00)
    “If you don’t have the talent to manage it, you can have the greatest toolset ... but it’s not going to work.” (09:16)

---

3. The AI Confidence Trap

• Quote:
  “Gen AI is deceptively complex. It gives you the false impression it’s simple to use and delivers excellent quality ... This is the trap.” (09:32)
• Statistic: 74% see productivity gains from AI, but only 11% see clear ROI.
• Sarah Madden’s Approach:

  “We don’t trust it. We use AI every day.” (10:38)

  • AI’s results are useful for efficiency (especially in SecOps) but always require verification. “We question your output every time ... It likes to be flashy and dramatic.” (10:44)
  • Human-in-the-loop remains critical.
• Mike Johnson:
  • Evolution in team behavior: no longer blindly trusting AI output.

    “Six months ago ... [people] would just blindly believe them. But nowadays, everybody understands you need to double-check.” (11:48)

  • Draws parallel to false positives in traditional security tools.

---

4. What’s Worse? – Security Team or Technical Debt

• Scenario 1: Security program depends on “a few very exhausted security heroes” – if two leave, everything collapses.
• Scenario 2: Huge technical debt with unmanaged accounts/tokens/identities.
• Mike Johnson’s Take:

  “People problems are harder than technical problems ... if you lose your people, you can’t fix it.” (17:20)

• Sarah Madden:

  “The reason you've got unmanaged service accounts is because you only have three people.” (17:25)

  • Agrees: losing key people is worse, as it makes technical issues unfixable.

---

5. Getting Value from Security Conferences

• Take a strategic approach to conferences: don’t just network, seek out people with advice relevant to your challenges.
• Mike Johnson’s Guidance:
  • Have a plan and purpose before attending.
  • Return value:

    “Bring something back to the team—a debrief, a writeup, links to talks.” (27:23)

• Sarah Madden:
  • Send only those eager and likely to learn/share.
  • Follow up with brown bag sessions to share knowledge with the team. (27:56)

---

6. Injecting Realism into Incident Response Tabletop Exercises

• Central Problem:
  In most exercises, real-world messiness is missing (authority, escalation, political friction).

  “The breach won’t expose your controls. It will expose who is allowed to act without permission.” (28:38)

• Sarah Madden:

  • Quarterly tabletops—growth comes from repetition, trial and error.

    “Every time, my feedback is: you didn’t bring me in early enough.” (29:57)

• Mike Johnson:

  • Perfect simulation isn’t possible, but frequency and external facilitators up the stakes.

    “We’ve actually brought in an outside firm ... It does change the stakes because they have less familiarity ... and you spent money on this.” (32:14)

• Cross-functional benefit:
  Sarah highlights major leaps in cross-team learning over time. (32:59)

---

7. Audience Q&A / Speed Round

AI Adoption, Agentic AI, and Vendor Displacement

• Jensen Huang (Nvidia CEO) Quote:
  “If that $500,000 engineer did not use at least $250,000 in AI code tokens, I’m going to be deeply alarmed.” (36:09)
• Panel Reaction:
  • Conflict of interest (Nvidia sells the hardware).

    “He wants everyone to be spending that kind of money.” (36:14)

  • But agree engineers should use AI to boost productivity.
• Measuring AI Usefulness:
  Sarah:

  “We’re thinking about how to measure effectiveness of AI ... You have to look at goals for AI as a business.” (37:15)

  • AI displacement of vendors is already happening: evaluating every renewal against in-house AI capabilities.

    “Every time we renew a software license, we ask ourselves, can we replace this with AI?” (39:12)

Advances in Agentic AI (38:21):

• Sarah: Faster adoption, improved accuracy, better reinforcement learning.
• Mike: Shift towards orchestration of multiple agents (“master control agent on top”).

Concrete AI Use Cases

• Mike:
  TPM with no coding background used AI to automate part of their workflow.

  “Everybody can augment themselves ... It’s very empowering.” (40:30)

• Sarah:
  Compliance team builds bots to automate audit responses, displacing vendor tools.
  • AI experimentation is now broad across roles, not just engineering.

---

8. Notable Quotes and Moments

| Timestamp | Speaker | Quote | |-----------|---------------|---------------------------------------------------------------------------------------------------------------------------| | 00:03 | Sarah Madden | “It’s not our jobs to say I told you so. It’s our jobs to manage risk...” | | 04:49 | Mike Johnson | “The first step is don’t sign a three-year deal the first time that you’re working with a vendor, period.” | | 09:32 | Quote/Host | “Gen AI is deceptively complex. ... This is the trap.” | | 10:38 | Sarah Madden | “We don’t trust it. We use AI every day.” | | 17:25 | Sarah Madden | “The reason you’ve got unmanaged service accounts is because you only have three people.” | | 27:23 | Mike Johnson | “Bring something back to the team—a debrief, a writeup, links to talks.” | | 29:57 | Sarah Madden | “...my feedback 100% of the time is, you didn’t bring me in early enough.” | | 32:14 | Mike Johnson | “An outside party does help raise the stakes [in tabletop exercises].” | | 36:09 | Jensen Huang | “If that $500,000 engineer did not use at least $250,000 in AI code tokens, I’m going to be deeply alarmed.” | | 39:12 | Sarah Madden | “Every time we renew a software license, we ask ourselves, can we replace this with AI?” | | 40:30 | Mike Johnson | “Everybody can augment themselves ... It’s very empowering.” |

---

9. Audience Games and Moments

• “What’s Worse?”: Team fragility vs technical debt—audience overwhelmingly agrees losing core team is worse. (18:00)
• “What is Dave’s Mom Talking About?”: Audience repeatedly outsmarts panel on deciphering cybersecurity terms from “mom logic.”
  • Memorable moment: “See, this is why we all need to work together.” – Sarah Madden (22:37)

---

Timestamps for Key Segments

• Security mistakes and “I told you so” – 00:03–00:36
• Vendor selection pitfalls & strategies – 03:50–09:16
• AI Confidence Trap & Team Strategies – 09:32–11:44
• What’s Worse Game – 14:18–18:00
• Conference strategies & ROI – 25:10–28:33
• Tabletop exercises & injecting realism – 28:38–33:53
• Audience Question Speed Round (AI-focused) – 35:16–41:57

---

Takeaways for Security Practitioners

• AI’s confident outputs demand skepticism and human review.
• Vendor lock-in can be toxic—prioritize flexibility and talent fit over contracts.
• Security programs are most fragile when dependent on too few, overworked heroes.
• Conferences are an investment; maximize by planning and sharing learnings.
• Regular, cross-functional tabletop exercises drive real incident response improvement.
• Agentic AI continues to accelerate—with benefits and displacements felt across roles.

---

For more wisdom (and some laughs), tune in to the full episode or subscribe at cisoseries.com.

Transcript

[Announcer]

Biggest mistake I ever made in security.

[Sarah Madden]

Go saying I told you so. There's nothing wrong with mistakes as long as it doesn't have a material impact on the business. I love the saying, there's no losses, there's only lessons. But when I think back across my career and I think about the mistakes that haunt me at night, it's the times that I have bent to the pressure of business and I went against my better judgment and I put myself in a position of you can see around corners and you know what's going to happen. And when that does and you say I told you so, it's never a good feeling. Because it's not our jobs to say I told you so. It's our jobs to manage risk and never have that happen.

[Announcer]

It's time to begin the CISO Series podcast, recorded in front of a live audience in San Francisco.

[David Spark]

All right, welcome everybody. Welcome to the CISO Series podcast. I am David Spark. I am the producer of the CISO series. And look this guy to my left, you may know him. It is the CISO for Rivian and my co host since day one, Mike Johnson. Let's hear it for him.

[Mike Johnson]

Hi, everyone. What an amazing audience.

[David Spark]

We have a huge crowd here today.

[Mike Johnson]

This is great.

[David Spark]

This is great. Now by the way, we are at B sides, San Francisco 2026. Let's hear it for B sides. And this is our four, I believe, our fourth time doing a live show.

[Mike Johnson]

I think that's right, yeah.

[David Spark]

Not for you personally. Cause I know you missed one. This is my third number three, four live shows here. Thrilled to be back. This is pretty spectacular. For those of you listening, we are in a gorgeous movie theater with our giant logo and all our sponsors logo, which I do want to mention right now, our three sponsors and I want to hear a huge round of applause for them. Our sponsors, Nudge, Security, Quiller, AI and Zenity. Let's hear it for all three of them. We are going to hear a lot more about them in just a minute. Now I want to bring our guest in who is our far left. We had her on once before. So thrilled with how well she did before. I said, you got to come join us live for our show at B sides. Big round of applause for the CISO of Convera, Sarah Madden. Let's hear it for Sarah.

[Sarah Madden]

Hey, everybody, welcome.

[David Spark]

All right, so here's my question to everybody and this is my challenge to the audience. Now, if you listen to the show regularly, you know, I often talk about this where I love and I don't get to see it often, and I'm hoping you will. Where people are in an event or trying to get into a party. And there's gonna be a lot of events and parties this week during rsa.

[Mike Johnson]

One or two and one or two.

[David Spark]

And they're trying to get in. And the line that is used or a variation thereof is do you know who I am? I so want to witness this. I wanna witness it badly.

[Sarah Madden]

And.

[David Spark]

And I want to throw this out to you. So if anyone witnesses it, by the way, bonus if you get it on video and can send it to me as well. But let me ask both of you, have you actually witnessed someone pull this off? Try the. Do you know who I am? Do you know my influence? Like, why am I not in this event, Sarah? Mike.

[Sarah Madden]

Many times in my life, yes. But I don't have trouble getting into parties.

[David Spark]

Oh, there you go.

[Mike Johnson]

Well, I pull that card all the time.

[David Spark]

Do you?

[Mike Johnson]

Yeah, absolutely.

[David Spark]

And what do you drop?

[Mike Johnson]

See, so serious.

[David Spark]

Do you drop Rivian. What is it you drop?

[Mike Johnson]

Well, I drop that. I know, David.

[David Spark]

And does that not get you kicked out? It usually does, yeah, it does get you kicked.

[Mike Johnson]

It's never been successful, by the way.

[David Spark]

It doesn't count if they say it because they've had a concussion. Do you know who I am? Or ask what year is it?

[Mike Johnson]

I don't know who I am.

[Sarah Madden]

Yes, you're a very important person.

[Mike Johnson]

There you go.

[David Spark]

I believe I am.

[Announcer]

Is this the best use of my money?

[David Spark]

Quote. When you only see 5% of the options, you can't make the best choice. You can only make the best choice from what you saw. End quote. Here's a pattern noted Richard Stennon of IT Harvest. Evaluate three or four vendors. Pick one with the best demo. Sign a three year contract. Now 18 months later, RIP it out because the vendor got acquired and product development stopped. Or a better solution emerged or the market leader turned out to be the marketing leader. Now the implementation time is wasted. And don't forget the squandered political capital spent justifying the ORIG decision. Due diligence isn't just about evaluating vendors, but knowing which vendors to evaluate in the first place. So, Mike, I'm going to start with you. What does your vendor selection process look like to avoid this trap? And second, how do you balance the fear, if you have it, of making the wrong choice against the need to make a decision and just move forward?

[Mike Johnson]

Well, the first step is don't sign a three year deal the first time that you're working with a vendor, period. Period, full stop.

[David Spark]

And let me ask you, have you maybe in your earlier days ever signed a three year deal at the beginning?

[Mike Johnson]

Oh, I think everybody makes that mistake. Okay. Like at some point you go, this was really bad. And then you spend the rest of your career thinking, I'm never going to do that again. So that's step one.

[David Spark]

And by the way, what you just said, that's gotta be the life of a ciso and I'm never gonna do that again daily.

[Mike Johnson]

Like, oh, well, that was bad. Let's not do that again.

[David Spark]

All right.

[Mike Johnson]

But the reality is this really is where our networks come in handy. If I'm looking for a solution to a problem I have, the first thing I do is I turn to my peer CISOs, like, how are you solving this problem? And that helps build the shortlist because you then have context. You're not just going to Google and saying, hey, I need to solve this. Or worse, going to Gartner and saying, hey, I need to solve this. It really comes down to talking to your peers. And then it's like, construct your list of questions, what are your requirements? Make sure, send that out to all of your shortlist, get those answers back, pick a couple that make great answers. Then you do your proof of concept and that makes you understand whether or not it works in your environment. And then you pick one from there. And at the end of the day you're still going to make bad decisions or things are going to happen. One of mine is we worked with a very early vendor. They were great. We actually, we had a one year deal because we signed one year deals first and then we signed a three year deal and they got acquired and product development stopped.

[David Spark]

Ah, just like I said.

[Mike Johnson]

So it can still happen, but at

[David Spark]

the end of the day you can't predict the future.

[Mike Johnson]

At the end of the day, we felt confident in the decision. We felt that we had all the information that we could possibly get. And sometimes you just have to recognize that you're going to make some bad decisions here and there.

[David Spark]

All right, I know, Sarah, you've had struggles with some vendors, vendors that have been implemented. What do you do when things are going awry and you're trying to figure out how to get out of this?

[Sarah Madden]

That's a great question. I'm in a situation where we're stuck in multi year licenses with three year deals. Yeah.

[David Spark]

Were these three year deals you made or somebody else made?

[Sarah Madden]

They were three year deal. So I got to build a greenfield security program at the company that I'm at. Now and four years into it. And so. Well, we did three years initially because we didn't have time to displace vendors. Like we still had to hire and deploy all the controls. And you're not going to mess with that in the middle of one or two year license. And so came up for renewal last year with most of our three year vendors and I put them in either one or two year licenses intentionally for that reason. But you run into a situation where you have to continually air your grievances with your vendors to force them to do the right things and you leverage the stick of I won't renew. Right. We're in challenging times where the majority of the tool sets that we have are SaaS products and they're going up and down, they're unavailable a lot. I think product development has taken a shift where we used to do n minus 1 because you don't trust the latest release. And we've got tons of vendors now that are saying don't trust until I have like a preferred release. And that could be months, if not quarters, if not nine months out. So I've been pressing a lot of vendors lately of just sticking to fundamentals of software development and being confident about the software that you ship to us. And if you get into a situation with your vendors where they're shipping product that isn't stable, isn't good, is causing issues. I'm all over them all the time now because I think we're taking a shift into low quality releases. And this is across most of the vendors we have in our toolset right now. It's happening across the board. So I'm pushing on that a lot as a CISO because I don't want that to be like a new normal. We can't get ourselves into a situation where we used to be an N minus one and we thought that was okay. And then we're dragging further beyond that. Like that's not a good position for us to be in. Right. The other thing I'll say in terms of finding good tool sets is it's also the peer. I agree with everything that you said, but some of the best tool sets we have, I've learned about them from new engineers that are entry level people that got to test out cool tools in their prior job and they're bringing in really cool new tool sets. And it's as much about finding the right tool as it is making sure that you have the skill sets on your team to manage that. Oftentimes we pick tools because the engineers we have on our team have expertise in being able to manage that product because you're never going to get a perfect product. Right? There's always going to be issues. But if you don't have the talent to manage it, you can have the greatest toolset and you could buy and it's not going to work because you don't have the right engineering to manage it. So it's about people, too. It's about their skill sets.

[Announcer]

Is AI going to help us or hurt us?

[David Spark]

Quote Gen AI is deceptively complex. It gives you the false impression it's simple to use and delivers excellent quality, all good or better than your own work. This is the trap. Now, Howard Holton of Gigahome recently shared a striking Gartner stat. 74% of organizations are seeing AI productivity gains, but only 11% see clear ROI. We've convinced ourselves AI output is quote good enough right out of the box. But the reality is that AI presents information with such confidence that we believe it. You know, the polished formatting, authoritative tone, and coherent structure. Holton argues that if you accept lowest common denominator AI output without investing time to develop content about what defines quote good, you're telling your employer you're as replaceable as a public AI model. Sarah, I'm starting with you here. How do you help your teams recognize when they're falling into AI's confidence trap?

[Sarah Madden]

We don't trust it. We use AI every day.

[David Spark]

Hold it. You don't trust it, but you use it every day?

[Sarah Madden]

Yeah. I mean, you question your output every time, Right?

[David Spark]

So you question it. And does your team know to do that?

[Sarah Madden]

Yeah. So we use AI every day, especially in SecOps. Like, it's so efficient for us to run a particular IDE tool in a very big cloud, fork some code, throw it in there, have it, analyze it. We get results back in five minutes. That's contextualized and it's actually super helpful. And then that usually starts like a little bit of a panic and a spin cycle and we're like, oh my God, look at these results. And then we don't trust it. Initially, we go and we look at it and you end up finding that there's pieces of it that's super helpful and there's areas that you could tweak. And it's usually not as bad as what the tool says because it likes to be flashy and dramatic. Right. So you just don't trust it and you look at the results and to me, it's not too much different than our regular vulnerability testing. That we do with the various different tools we have or the pen testers that we use. There's false positives in almost everything we work on, so just simply don't trust it. And then when it comes to just use cases that we're building out, the human in the loop control is just super important until we get more confident with the accuracy of the AI output.

[David Spark]

All right, Mike, same philosophy with your team.

[Mike Johnson]

And I think that really is the new normal. Six months ago it was, hey, these things are really confident. I'm just going to blindly believe them. But nowadays everybody understands that you need to double check and that is the evolution that we've seen. This airs in May. I imagine things have even changed by then. And so we will continue to have that moving forward and we will learn along the way. And Sarah mentioned false positives is something that we actually work with a lot of in security. That is not new for us, that is normal. And we should make sure that we're reminding our teams like, yeah, you need to check these things work. And you're seeing this in the industry where AWS had an outage recently that was AI induced, however you want to call it. And they're making sure that they have senior engineers checking AI generated code going forward. And these are things that we're going to continue to learn. And yeah, nobody trusts these, but that's okay.

[Announcer]

Who's our sponsor this week?

[David Spark]

You know, AI spread to every corner of your tech stack. We were just talking about it. Which is great for innovation, yes, but not so great for security and governance. And that is where one of our wonderful sponsors, Nudge Security, comes in. Nudge discovers shadow AI across your organization. Here's what's good. Also including chatbots, agents, MCP, server connections, AI in the supply chain of other SaaS, tools and even more. And Nudge gives you workflows and automation to scale AI governance without slowing down productivity. The best part, you will have a full inventory of AI assets on day one of your free trial and take advantage of this. By the way, even those introduced before you started using Nudge, by the way, they will scan your environment and tell you, well, how sass loose it is, which it is. So no time machine is required here. Gain visibility and control of shadow AI risks. Get started by going to their website nudgesecurity.com ShadowAI and it is spelled just the way it sounds. Nudge Security.

[Announcer]

It's time to play what's worse.

[David Spark]

All right everybody, for those of you, most of you are familiar with the what's Worse game. If you've heard our show before, our fans, they send in great scenarios of just horrible things that happen, all usually fictional. Every now and then we get a real world scenario and those are actually kind of fun because it was like kind of a Sophie's Choice decision that they had to make. And we find out the real story and we see if our. The panel had actually matched that. So if you've got those, please send them in. We always like to hear great, what's worse scenarios, fictional or real. All right, this comes in from Craig George of Guidepoint Security. Mike will answer first, then Sarah, you can agree or disagree. And then we're throwing it to the audience to find out what your answer is. So here we go, scenario number one. For years you've been running a security program held together by a few very exhausted security heroes. You know it will collapse if two people leave. All right, Your security department is gone if two people go. That's scenario number one. Second scenario, for years you've had unmanaged service counts, API tokens, and non human identities that no one fully owns, kind of partially. But in both cases here, the two scenarios, you're stunned that nothing has exploded because they're both powder kegs just ready to explode. Mike, which one is worse?

[Mike Johnson]

So in the first scenario, what's interesting is these aren't opposite sides of the same coin, which is what we usually get is like this or that, which

[David Spark]

is like, yeah, you got all of this and none of that. Or all of that and none of that.

[Mike Johnson]

Yeah. So you've got a fragile team or a fragile environment. Maybe that's the brain.

[David Spark]

It's a poorly managed environment in general.

[Mike Johnson]

Sure, sure. Realistically. And I'm not trying to change the.

[David Spark]

Yeah. By the way, this is a rule of wtrush. You can't change it.

[Mike Johnson]

But the second one, those are technical problems. Those are far more solvable than people problems.

[David Spark]

But they're not going to get solved. That's the thing. It stays like.

[Mike Johnson]

Right. And the reality is if you have some people who really know the environment and if they leave, everything comes crashing down, you're really gonna be in bad shape. And that is where you're gonna end up with a big issue versus, it's kind of like the known knowns versus the known unknowns and the unknown unknowns.

[David Spark]

Well. But the thing is, first scenario, you're kind of. Your security program's running reasonably well. It sounds like. Second scenario. Eh, it's kind of crappy to start with.

[Mike Johnson]

Well, the first scenario is something Is about to happen. The second one is something has.

[David Spark]

Has happened or something is very visibly weak.

[Mike Johnson]

But both of them are. Something will happen. It's a matter of time. It's a matter of time, and then one is the issue. And frankly, I think the problem that you've built a team around two people, that actually is the worst scenario.

[David Spark]

Well, no, no, it isn't just two. There's a bunch. But if two go.

[Mike Johnson]

But you've essentially built it such that.

[David Spark]

Built it that if 2 go, it's gonna be.

[Mike Johnson]

So I think first scenario is the worst. Is the worst.

[David Spark]

One. First is the worst. All right, we're throwing this to you, Sarah. What do you think?

[Sarah Madden]

I agree with you. The reason you've got unmanaged service accounts is because you only have three people.

[David Spark]

Well, who knows?

[Sarah Madden]

So if you lose any of those, you're probably.

[David Spark]

Well, it's a difference in the second one. It may not be that you have exhausted heroes in the. The second term.

[Sarah Madden]

It's just, well, if you. If you lose your. If you lose your resources, then you can't fix the technical problems, which is the. The principle that I agree with. I think the. The worst scenario is losing your good people when you already have a small team for sure. I mean, none of us have been in perfect environments where we don't have the second scenario. Let's just be honest with ourselves.

[Mike Johnson]

That's true.

[Sarah Madden]

But if you lose your people, you can't fix it.

[David Spark]

All right, you're saying so first is worse because it'll be unfixable. All right, we're going to throw this to the audience by applause. How many people think that the first scenario, that if you lose two people, you're screwed. That is the worst scenario. Oh, it's looking like a lot of people here, A lot of people here, a lot of. All right, second scenario, where it's kind of weak, but, you know, if two people go, you're still running at the same speed. How many people think that's a worst scenario? All right, a few brave souls. I appreciate that.

[Sarah Madden]

Dissension.

[Announcer]

What is Dave's mom talking about?

[David Spark]

All right, we've played this game here before. It's a hit. My mom has become a hit with this crowd.

[Mike Johnson]

Your mom is awesome.

[David Spark]

My mom is awesome. I've known her all my life. So here's how this game works. I know this is going to come as shock to you, but my elderly mother is not a cybersecurity expert. But when her son comes up with a stupid game idea, she plays along and she plays along with this game. So my mother is going to define some terms in cybersecurity. Everyone in this room knows what these terms are. My mother does not. Or she gets varying degrees of correct and wrong. And you'll see in general. So you have to kind of use reverse logic. If you had heard this term before, did not understand cybersecurity, how would you best explain it? I will ask the panel first to answer. If they can't get it, I will throw to you, to the audience. All right, are you ready to play? And I can play. I can repeat them if you don't get them, because some of them are very quick. All right, here is the first one. I think that's just carrying on in some fashion. All right, this is very wrong. I will just say. And this is a tough one. The first one's a pretty tough one.

[Sarah Madden]

An audit report.

[David Spark]

That's a good one. You want to take a stab? I'll play it again. I think that's just carrying on in some fashion.

[Mike Johnson]

Connectionist protocols.

[David Spark]

No, no, I'm going to be honest. This is a really tough one. Anyone want to take a stab at this? Crosstalk? That's a good guess. And wrong. Anyone else carrying on? No. Let me give you kind of a hint. The word itself kind of sounds like carrying on. It is one word and it kind of sounds like carrying on. No, not AI.

[Mike Johnson]

It's good answer to that.

[David Spark]

I'm going to go give a couple more guests and we're going to go to the next one. I'll explain. Anyone else? I'll play one more time. I think that's just carrying on in some fashion. No, that would be hashing. Sounds like a little carrying on. A little hashing. All right. Okay. All right, we're going to play another one. Here we go.

[Sarah Madden]

Taking care of personnel issues.

[David Spark]

Okay, there's a hint of being correct here. There is definitely a hint of being correct. Oh, wait, let them answer first. Hold on.

[Mike Johnson]

I don't know. Human resources.

[David Spark]

No, it's a cyber term.

[Mike Johnson]

Security awareness.

[David Spark]

No, I give up. Take a stab.

[Sarah Madden]

This is Thump d'siso.

[David Spark]

This is not.

[Sarah Madden]

What is Dame's mom of personnel issues.

[David Spark]

And just a hint.

[Sarah Madden]

Account termination. It's like that's where my head's going.

[David Spark]

That is a good one. But again, it's a cyber term. It's a cyber term. Hold my identity. What? Hold on. It's your identity? Yes, it's identity access management. Very good. Very good. Good job, everybody. All right. The audience got it. Not you.

[Mike Johnson]

Good for the audience.

[David Spark]

All right, we got another one. Here we go. We got two more. You need a plan to figure out

[Sarah Madden]

when someone is trying to invade your organization. Incident response plan.

[Mike Johnson]

Yeah, it sounds like an incident response plan.

[David Spark]

Again, that's if she had the correct answer. But as I told you, these are variations of being wrong, so.

[Sarah Madden]

Cyber insurance.

[David Spark]

Okay, you're giving my mother way too much credit here. Let me play it again for you. You need a plan to figure out

[Sarah Madden]

when someone is trying to invade your organization.

[David Spark]

Okay, so there's parts of that that are correct. There are parts of that. Correct.

[Sarah Madden]

Of soar.

[David Spark]

No, not soar.

[Mike Johnson]

Disaster recovery plan.

[Sarah Madden]

Oh, God,

[David Spark]

no. Hold on. Wait. Sorry. What would somebody say here? Cyber. Hold on. Someone say. Intrusion detection. Good job.

[Mike Johnson]

Well done.

[David Spark]

Good job.

[Sarah Madden]

See, this is why we all need to work together.

[Mike Johnson]

Yes.

[David Spark]

All right, very good. Last one.

[Mike Johnson]

So we're 0 for 3.

[David Spark]

Yeah, we're O3. The audience is way smarter than you.

[Mike Johnson]

The audience is doing much better than we are.

[David Spark]

Yeah, way better.

[Sarah Madden]

Make available the best information possible.

[David Spark]

Okay, there is, again, a hint of something correct here.

[Sarah Madden]

Wow.

[David Spark]

Nothing.

[Mike Johnson]

I got nothing.

[David Spark]

Okay, this one you should be able to get. Come on. Make. You got this.

[Sarah Madden]

Mm. Mm. No, I'm thinking this is why you need expertise in this industry to do your job well.

[David Spark]

This is why our industry is so difficult.

[Mike Johnson]

Yeah, it's a hard job.

[David Spark]

It is. All right. Did you hear that? Here, I'll play it again for you.

[Sarah Madden]

Make available the best information possible.

[David Spark]

There you go. They got it. Good job. Once again, the audience is way smarter than you.

[Sarah Madden]

We suck at this game.

[Mike Johnson]

Yeah. Yes. Yes, we do.

[Announcer]

Who's our sponsor this week?

[David Spark]

You've got dlp, you've got casb, you've got alerts. But here's the question no one's asking. Can you tell if an action in your environment was performed by a human or an AI agent? Because today, agents don't just generate content, they take action. They move data, trigger workflows, and change systems without asking. And your existing tools, they'll tell you what happened after the fact. Now, that's the gap Quiller AI was built to close. Quiller AI's decision engine sits inside every interaction, browser, endpoint, SaaS, LLM, and agent workflows, evaluating the content, context, and intent before an action completes. Not more alerts. Better decisions made in real time. Now, if you're serious about securing your AI transformation, I know we're all going there and not just monitoring it. Check out Quillerai. Let me spell that out for you. Q, U, I, L R AI security can't live after the decision anymore. Quiller AI makes sure it doesn't have to.

[Announcer]

What's the best way to handle this?

[David Spark]

How can a CISO really any security professional, get the most out of a security conference like BSides SF RSA so we've actually discussed this many times actually on the show, but this is an interesting take that we saw over on Reddit, because whenever we talk about the go you just got a network. One Redditor on the cybersecurity subreddit advised a more strategic approach and I love this ask yourself quote, is this a person who I can seek advice from if I was in a situation? Now the benefit of networking is to have connections who can find filling gaps that you aren't soon planning on filling. So for example, I don't know much about aimcp, but this guy at this booth seems to know a lot. So another commenter focuses on technical talks, community events and HallwayCon. All great advice. So I'll start with you Mike. How do you set expectations for your team when they attend conferences and how do you measure whether the conference attendant delivered value?

[Mike Johnson]

I think these two examples are really the right way to think about the conference itself. Some conferences it's around the talks, the content. BSIDES is great for that. You're getting these community driven conversations that the attendees have the opportunity to learn from versus the vendor driven events. Maybe your goal is to go speedrun the vendors and understand what all is in that space. The important part is to have the plan going in. Like what is it that you want to accomplish? Why is it that you're going to invest your time in whatever you're going to and then lean in on that? Maybe it is HallwayCon, maybe it is the networking events and so that's really what you should focus on. So that's what I tell my team have the plan going in, but I also tell them I want you to bring something back to the rest of the team. If we're paying for your airfare or your ticket or your hotels or something

[David Spark]

like that, you only do one of the three and or I would assume you pay for the hotel or it

[Mike Johnson]

doesn't have to be exclusive. It could be all of them.

[David Spark]

Who would think andor is what you'd say go on andor I'm just giving you crap. Go on. Thank you.

[Mike Johnson]

Thank you. Never happened before. No, never happened.

[David Spark]

This is the first marked date and time.

[Mike Johnson]

Yes. But it really does come down to there's an investment the company is making in that person that's great, they're advancing, but we'd like to see them bring that back to the rest of the team. So have a debrief, have a document that you write up. Here's what I saw, here's what I learned. Here are the talks that I sat through that you should go watch the recording of, bring that back to the rest of the team, along with the plan going in of what you want to accomplish. And that's how you get the most out of these events.

[David Spark]

Sarah, what do you do with your team?

[Sarah Madden]

I mean, I send my team to the technical conferences so they can sit and learn and then come back and do a brown bag. So to your point, it's purpose built, right? Bsides black hat AWS conference. Like, those are good technical conferences that our teams go to, learn stuff and then come back and share what they learned. We frankly send the people on the team that like to do conferences. There's people that don't like to do conferences, right? And so you can incentivize them with different kinds of training. And so you send the people that actually are going to get out there, go to the talks, meet people, learn things, and then come back and share it. So I think it's another important part of it too, is like, send the people that are going to get the most out of it.

[Announcer]

They didn't think that through all the way, did they?

[David Spark]

Your incident response tabletop, your IR tabletop is lying to you. Not because the scenario is wrong, because the incentives are. Now, this is what Joshua Copeland of Crescendo argues, that in tabletop exercises, everyone talks fast, decisions are clean, ownership is clear, and nobody protects themselves. No real incidents ever unfold in a real breach. The first control to fail isn't a firewall, it's authority. People don't hesitate because they lack training. They hesitate because escalation is political. Detection becomes, quote, let me validate one more thing. Or containment becomes let's wait for leadership and disclosure becomes legal. Is reviewing the language. If nobody in the tabletop worries about consequences, you didn't simulate incident response. The breach won't expose your controls. It will expose who is allowed to act without permission. It's like playing poker without real money. You play the game very differently when there are actual stakes. So I'm going to ask you, Sarah, how do we actually inject stakes into tabletops to reveal those fault lines? Because that's what you. You want to see, fault lines. If a tabletop goes without fault lines, you're like, we did something wrong.

[Sarah Madden]

We do tabletops quarterly. And my feedback 100% of the time is, you didn't bring me in early enough. You should have brought me in here. You should have brought me in here. And it's so hard to sit there because I haven't been called yet, so I have to be quiet. So I'm just sitting there on the edge of my seat. But the reason we do tabletops on a repeated basis is because we get better and better at it every single time. And two years ago, my team was bringing me in way too late, and in some cases, recently, they're bringing me in too early. And so it's just. It's trial and error, it's education. It's keep repeating the tabletop exercises and you get better at it. It's that simple. I think.

[David Spark]

All right, Mike.

[Mike Johnson]

Yeah. I think the key is there's no such thing as a perfect tabletop. Like, simulating the actual stress is not something that you're going to be able to do. But you still learn. Like, you still get better. There's still advantages to doing that. And so you keep doing them, recognizing that they're not perfect simulations and there's still something that you can learn and keep going forward. One of the things that we've done started doing this past year was we've actually brought in an outside firm to do a tabletop exercise for us. And that then does change the stakes because they have less familiarity with the environment. They. They aren't making assumptions in a tabletop exercise of, well, of course we would just do that. And so I think that really is one of the opportunities is continue to uplevel your tabletop game and make that better, recognizing that it's never going to be perfect.

[Sarah Madden]

Yeah. And I love that suggestion. And also just resist the urge to manage them. They don't learn if you tell them how to do it right every single time. So sit there quiet, suck it up, let them fail and fail forward. Right. Like we talked in the beginning about mistakes. Like, there's mistakes are good. It's how people learn. Right.

[David Spark]

And let me ask you that. Getting to the, you know, the line of playing poker without money is not playing poker. And I think if you bring in a third party, correct me if I'm wrong here, because I don't do tabletops, but I would envision they know how to put on a performance. You know what I mean? And so they'll, even though it's not, quote, real, they'll create sort of some false sense of urgency that that will make your team act A little bit more intense. Do you see that to be the case?

[Mike Johnson]

Yes, I do think they are more adept at creating scenarios that still can inject some concern, can inject some urgency again, in a way that your internal team will have a hard time simulating. And there's also the fact that you spent money on this and that does create some additional pressure as well to get value out of it because you could have spent that money on something else. So that outside party does help raise the stakes.

[David Spark]

I have a question for you, Sarah. You say you do it quarterly, which is fantastic. I don't think I've heard anybody do it that often. What are some of the big leaps that you see that they're making from a tabletop, from the earliest to the latest one?

[Sarah Madden]

So I have three different functional teams and we do the tabletops with everybody. I think usually an incident response is primarily your security engineers that are running it. But there's a lot of cross functional knowledge that has been really valuable for the identity management team to be a part of it. Right. They see how to figure out things quicker and they are contributing more and more useful insights as we go through the tabletops. And so I think the cross functional knowledge is super helpful. And then another part of my team does all of our audit and compliance and security programmatic stuff. And so as we run through tabletop scenarios, they'll realize, oh, I could tune this policy this way to make sure that this kind of scenario doesn't happen again. And so I think the consistency of it and the growth that we've had as a team has been cross functional more so than it's been lateral with the SecOps team because they're good at this and they do it all the time. But I think if you broaden it out and have a lot more people in the room, you know, rising tide lifts all boats kind of scenario.

[Announcer]

Who's our sponsor this week?

[David Spark]

Join Zenity and contribute to the future of Agentix Security. On May 27, 2026, the AI Agent Security Summit, hosted by Zenit Labs, returns to San Francisco. Local speakers from leading platforms and industries will provide discussions, panels and keynotes surrounding the most pertinent findings around agent behavior, access and risk. In addition, security pioneers will unveil the most effective practices and that security professionals can take to scale agents across the enterprise securely. Right now, today. Now this event is vendor neutral, free to join and community focus Join to network with brilliant minds across the industry and get ahead of the curve in securing AI agents. I know this is of a concern so you know what you want to do? You want to register? Go to Zenity IO cisoseries and let me spell that for you. Z E N I T Y IO cisoseries. You better already know how to spell that. Go to that website and help shape the future of agentic security.

[Announcer]

It's time for the audience question Speed round.

[David Spark]

All right, I have some questions in my hand from audience members, and these are some really good questions, and we actually have a good amount of time to get through a few of these questions. So they have not seen these questions at all. Let's make it new. But here is the first one. And someone overheard this on the all in podcast just two days ago from Jensen Wang, who is the CEO of Nvidia. And this is his quote. And essentially I'm going to tell you what the question is right now. I want to know your thoughts on this quote. Okay, either one of you can jump in. And this is what Jensen Huang says. If that $500,000 engineer. That's how much he's paying for engineers, if that $500,000 engineer did not use at least 250,000 in AI code tokens, I'm going to be deeply alarmed, End quote. Your thoughts on that, Mike?

[Mike Johnson]

Well, I mean, this is coming from the CEO of the company who makes hardware that burns tokens.

[David Spark]

Yeah. So he wants everyone to be spending that kind of money.

[Sarah Madden]

Can we have a conflict of interest,

[David Spark]

financial interest on that? Yes. So that is pretty aggressive thinking. And if everyone spent that kind of money, their stock would go up a lot more. Yes.

[Mike Johnson]

Yeah. They'd be what, the first $10 trillion company? I think if you look at the gist of it, it really is. Engineers really should be leaning into augmenting themselves, getting more done with AI. And I think that really genuinely is true. That is where we're at today, that you can be that much more effective by partnering with AI rather than assuming that it's just going to do all the work for you or just that somebody else is going to do the work. The gist of it I agree with. I think at the same time, the amount of money raises my eyebrows.

[David Spark]

And by the way, I forgot to give the appropriate credit. This was from Rishe Joshi of Elementrix, who gave this question. Anything to add to this, Sarah?

[Sarah Madden]

I mean, we're thinking about how to measure effectiveness of AI and how much our engineers are using AI and we're looking at what metrics make sense. And it can't be like a conflict of interest in a kind of scenario. So you have to look at the goals for AI that you have as a business and what those outputs are and then measure it that way. If it's engagement with the particular IDE tool that you're using, using in engineering and you want to make sure people are using tokens. Sure. But obviously it has to just be proportionate to what you want to do for your business. But I mean, it's important. There are a lot of engineers that don't want to adopt AI because they think they're going to displace themselves. So what are the kinds of AI use cases do you have going on in your environment that you can incentivize engineers to do and then roll with that?

[David Spark]

By the way, I hate to break it to you, I just realized all these questions have to do with AI. So get ready for. I know it's shocking. All right, so this comes from Colin Dupre of Run Zero. So agentic AI was introduced last year in a significant way. Significant way. It's still the big story this year. What do you believe has changed significantly with agentic AI from last year to this year?

[Sarah Madden]

Adoption speed, accuracy, reinforcement learning models are getting better.

[Mike Johnson]

What I'd say is you've seen more the concept of agents controlling other agents. It used to be this is an agent that it's standalone, it can do many things. I think what we've seen is smaller but more agents along with some sort of master control agent on top of that. And that's what I think we'll continue to see more of.

[David Spark]

All right, this comes from Jordan Kahm, a Morado. Again, another AI question. And you know, there was a little bit of scare in the market with Claude's announcement with security. So Jordan's question is, do you see yourself phasing vendors out because of AI? You're smiling, Sarah. What?

[Sarah Madden]

I mean, every time we renew a software license, we ask ourselves, can we replace this with AI? And I already made that call in a vendor a couple weeks ago. So it's evolving fast. So yes, for sure.

[Mike Johnson]

Yeah. And we've already done it and we will continue to do it.

[David Spark]

You did face the one out already.

[Mike Johnson]

Absolutely. And I think that is the world that we're in today where a vendor needs to set themselves apart from a generic LLM, because if a generic LLM can replace them, they're not differentiated. And if they're not differentiated, we can just do it ourselves.

[David Spark]

All right, good point. And I know you've been very engineer focused with your. All right, so I have two questions that are very similar. And I'm going to go to you first, Mike, because this has to do with a post you put out that got a lot of response. So first from Peter. Let me get through both of these. Peter Zweer is a morado who wants to know from your AI post because you can sort of summarize a little bit more, but you put out a post saying give your team some tokens and let them have at it to try to fix their own problems. And also from Caleb, on a fray, a paradigm. What's the most realistic output for Claude code? So I'm interested to know what have you seen that's pretty cool that either your team or somebody else has done? And I want to also talk to you, Sarah, on that.

[Mike Johnson]

So part of the inspiration for that post was where a TPM on my team came to me and said, can I have a cursor license? This is someone who has never written a line of code in their life. And that really stopped to have me think of. We can really empower the whole team. Everybody can augment themselves, regardless of their role, regardless of how technical it might be. Engineers, it's a more obvious use case, but where you've got folks who aren't used to writing code who are adopting, I'm going to go build a thing to make my life easier. That's really where we are today. And the cloud code example is a good one, where you can go from idea to solution to your problem in a scalable way very quickly, whereas in the past you used to have to go talk to vendors or try and get time from an engineering team. So it's very empowering is what I'm seeing.

[David Spark]

And the coolest thing you've seen, I

[Sarah Madden]

mean, I have my compliance team writing AI little bots and automating responses to audits and customers and displacing vendor solutions we have with that. And so it's not just the engineers that are using AI now. I think it's just the broader creativity that I'm seeing across all of the different roles. We have to figure out different things to automate and then figure out what else are we going to do with our time. That's the fun part about where we're at now. It's everybody's playing with it. It's not just the traditional engineers.

[David Spark]

That brings us to the very end of the show. Let's hear it for our two guests, Mike Johnson, CISO of Rivian, and Sarah Madden, CISO of Convera. Let's also hear from Bside San Francisco and Most importantly, let's hear it for our sponsors. Nudge Security. Remember, go to nudgesecurity.com ShadowAI Quiller AI go to Quiller AI and also Zenity IO cisoseries to go to their event. We see love coming to this. We are thrilled that you came to see us do our show. Thank you very much to our guests, thank you very much to our sponsors and thank you to Bsides. We really really appreciate you contributing and listening to the CISO Series podcast.

[Announcer]

That wraps up another episode. If you haven't seen subscribed to the podcast, please do. We have lots more shows on our website cisoseries.com Please join us on Fridays for our live shows Super Cyber Friday, our virtual Meetup and Cybersecurity Headlines Week in Review. This show thrives on your input. Go to the Participate menu on our site for plenty of ways to get involved including recording a question or a comment for the show. If you're interested in sponsoring the podcast, contact David Spark [e-mail] thank you for listening to the CISO Series podcast.