I Don't Just Guess About Effectiveness, I Make Educated Guesses!

Summary

CISO Series Podcast – Episode Summary

Episode Title: I Don't Just Guess About Effectiveness, I Make Educated Guesses!
Date: November 4, 2025
Hosts: David Spark, Andy Ellis
Guest: Sarah Madden, CISO of Convera

---

Episode Overview

This episode of the CISO Series Podcast brings together hosts David Spark, Andy Ellis, and featured guest Sarah Madden to discuss the evolving challenges and strategies in building effective security programs. The panel explores greenfield security design, the shifting landscape of GRC (Governance, Risk, and Compliance) amid AI advancements, automation’s role in pen testing and vulnerability management, and the difficulties in measuring and proving security effectiveness. The tone is practical, candid, and filled with insightful anecdotes from seasoned practitioners, with humorous interludes to keep the conversation lively.

---

Key Discussion Points & Insights

1. Greenfield IT and Security: Designing for the Real World

[03:54–10:40]

• Main Theme: If given a clean slate—how should one architect IT security?

• Human-Centric Security: Andy Ellis emphasized that relying on users to “do the right thing” is a blind spot; systems must be designed around natural human behavior.

  “Humans actually are the people who provide value in your business... If you say, ‘I’m going to put a security control in your way, take an extra 30 seconds every time,’ they’re going to at some point be like, ‘this is stupid.’”
  – Andy Ellis [05:14]

• Zero Trust Done Right: Andy’s take on Zero Trust involves authenticating the person and the device as a pair, simplifying the user experience.

• Sarah Madden’s Experience: She describes building a greenfield security program at Convera, which allowed her to centralize tools, avoid legacy system bloat, and create clear processes for tool requests and usage. Centralization streamlines security while letting users get what they need efficiently.

  “Control is the name of the game and it’s productivity versus control...we don’t have toolset bloat, we have clarity on what we need to manage.”
  – Sarah Madden [07:24]

• Handling Acquisitions: Andy’s approach was to replace an acquired company's IT stack immediately to avoid supporting decades of merged tech, while Sarah highlighted the visibility benefits of modern cloud tooling for managing legacy assets in new environments.

---

2. AI, Automation, and the New Face of GRC

[10:54–15:49]

• GRC Transformation with AI: With automation spreading into compliance roles, GRC (particularly the compliance “C”) shifts from manual monitoring/enforcement to engineering preventative, automated controls.

• Job Evolution: Sarah Madden notes moving team members from audit-focused roles into “GRC engineering,” directly automating policy enforcement through platforms like AWS.

  “We are shifting the role of somebody on my team...into the UI of AWS and the native tool sets...for automating policy in config and starting to actually click automation buttons.”
  – Sarah Madden [11:43]

• From Compliance Policing to Controls Engineering: Andy Ellis underscores that most GRC is just compliance work; he pushes for true “controls” where violations become technically impossible, not just forbidden on paper.

  “We need to move to a world that is control-systems based, where when you say you have a control, it is impossible for it to happen in a different way—that’s literally what control means.”
  – Andy Ellis [14:02]

• Practical Impact: Both agree that automating controls makes everyone happier and less reliant on retroactively slapping wrists for mistakes.

---

3. Automated Security Scanning vs. Human Red Teams

[24:15–29:21]

• AI Native Security Tools: New AI tools can detect business logic bugs in code that traditional scanners miss, forcing a re-examination of what is possible to automate.

• Limits of Automation: Sarah is cautiously optimistic—excited for internal automation (e.g., vulnerability management in SDLC), but notes awareness gaps in business logic might persist until AI matures.

  “I still think there’s so much of the business logic that a red team has that can’t be automated yet, but I don’t think we’re far off.”
  – Sarah Madden [25:01]

• The Practical Line: Andy suggests most pen test work (“running vuln scanners, break in, look around”) is automatable; exceptional humans are needed only for niche, complex cases.

• Business Model Idea: Andy jokes about "external washing" internal pen test results to appease compliance demands for third-party reviews.

---

4. Measuring Security Effectiveness (and Admitting Our Blind Spots)

[31:09–36:32]

• Hard to Prove a Negative: The insurance industry, with every incentive to find effective controls, often struggles to determine what doesn’t work.

• Why Security Metrics Lag: Andy highlights the lack of standardization in digital environments—unlike houses or cars, every IT environment is a “special snowflake,” making actuarial science difficult.

• Cyber Insurance as a Catalyst: Sarah finds cyber insurers ask better questions than regulators, driven by financial necessity, but both are years behind real-world attack tactics.

• Root Cause Analysis Gap: The panel agrees the hardest (and often overlooked) part is methodically analyzing incidents to address underlying problems—requiring strong communication as much as technical skill.

  “The change management part of security is the harder part.”
  – Sarah Madden [36:32]

---

Notable Quotes & Memorable Moments

• “[Zero trust]: for me, the core…is actually taking a laptop and associating it with a human and authenticating the pair of them…seamless authentication continuously.”
  – Andy Ellis [06:17]

• “Trust is not a control. This is something I say all the time in my role. Our job is to design security controls that limit user impact to create an actual massive security incident.”
  – Sarah Madden [06:56]

• “We need to move to a world that is control-systems based, where when you say you have a control, it is impossible for it to happen in a different way—that’s literally what control means.”
  – Andy Ellis [14:02]

• “Back to trust is not a control. People don’t want to fail controls…The more you can automate them out of making mistakes, the happier they are.”
  – Sarah Madden [15:34]

• “If you say, ‘I’m going to put a security control in your way…they’re going to go around the task. That’s not the human doing the wrong thing, that’s the human doing the right thing from the business perspective.”
  – Andy Ellis [05:14]

• “I will give [cyber insurers] credit—they are better at asking important questions as it pertains to the threat landscape…because there’s a vested interest.”
  – Sarah Madden [34:30]

---

Fun & Light Moments

Earthquakes & Cockroaches

[00:29–03:45]
The show opens with a comedic riff comparing Andy to a cockroach (tough to eliminate), which segues hilariously into stories about living through California earthquakes and using pool waves as earthquake meters.

“What’s Worse” Game: Russian Chatbot vs. Clown Security Awareness

[18:04–24:08]

• Debate: Is it worse to outsource security awareness training to a Russian chatbot or have clowns jump out every time someone clicks any link in an email?

• Consensus: Clowns are worse due to the disruptive fear and panic, with the chatbot’s errors at least mitigatable by other technical controls.

• David cheekily tries (and fails) to convince the group of the viral branding opportunity from clown-prank videos.

  “Would we all laugh at it? Yes. Is it a useful thing to do? Absolutely not.”
  – Sarah Madden [22:31]

---

Key Segment Timestamps

• Greenfield Security Discussion: [03:54–10:40]
• AI’s Impact on GRC: [10:54–15:49]
• Automated Vulnerability Management & Pen Testing: [24:15–29:21]
• Measuring Security Effectiveness/Cyber Insurance: [31:09–36:32]
• Memorable Game Segment (“What’s Worse”): [18:04–24:08]

---

Closing Thoughts

Sarah Madden encapsulates the spirit of the episode:

“I think sometimes it’s hard to step out of your bubble and have conversations with colleagues and there’s so much we can learn from each other, if not just a therapy session. I think we’re all fighting the same fight every day...I would just encourage everybody in the defender space to just keep talking to each other.”
– Sarah Madden [37:26]

Andy Ellis adds to the good-natured vibe:

“They think group therapy with the whole industry.”
– Andy Ellis [38:19]

---

Takeaways

• Design security for real humans, not just ideal processes.
• Automate controls wherever possible—move compliance from audits to engineering.
• AI and automation will majorly change the roles and effectiveness in both GRC and vulnerability management.
• Proving “what works” in security remains an elusive challenge—root cause and clear metrics are needed.
• Don’t underestimate the power of practitioner-to-practitioner learning and group “therapy” among defenders!

---

For more episodes and discussions, visit CISO Series.

Transcript

[Host/Announcer]

10 second security tip.

[Sarah Madden]

Go. Our world is driven by urgency and this causes mistakes that create security incidents all the time. So my tip is it's never too urgent to not fully think through. Calm down, think before you act.

[Host/Announcer]

It's time to begin the CISO Series podcast.

[David Spark]

Welcome to the CISO Series podcast. My name is David. I am the producer of the CISO series. Joining me as my co host because he's here all the time. You can't get rid of him. We've tried. He's like a cockroach. It's Andy Ellis, the principal and legendary CISO of Duha. I've now called you a legendary CISO and a cockroach.

[Andy Ellis]

Yeah, now I'm a cockroach too. I've sometimes been likened to a plague you can't get rid of. But a cockroach is a new one. David.

[David Spark]

There you go. Well, cockroaches are tough to get rid of. Yeah, we're [e-mail] we have lots of wonderful programming over there. Why don't you go check it out? Our sponsor for today's episode. If you've been listening to this programming, you've heard them before because they're phenomenal. Sponsor of ours, IT's Threat Locker Zero Trust Protection Platform. You will be interested in what we have to say about them later in the show. But first, Andy, we were talking about earthquakes off the air and you keep saying you've missed them. So have you never felt or experienced an earthquake?

[Andy Ellis]

Oh, no, no, I have definitely felt an experienced earthquakes. I grew up in LA. I was a senior in high school in 89 when we had all of the pre shocks. But I headed off to college right before the Loma Prieta earthquake. So I was at college during that one. I came back for a while and then head back out east again just in time for the Northridge earthquake. I was born just after the San Francisco earthquake. So none of the big ones have I actually been in California for, but I certainly have woken up many to many times to an earthquake, but never anything that was.

[David Spark]

Now, what was your first reaction to an earthquake? Because mine wasn't. That was an earthquake. Because you don't. I didn't know what was happening the first time.

[Andy Ellis]

I mean, when you grow up in California, like they're just a fact of life. Like at some point you're just like, oh, some things shook. Must have been an earthquake. You learn. I don't know if you've ever assembled IKEA furniture and you had little tabs to nail things to the wall. And everybody here in New England is like, why would I do this? Other than the fact that my, my floors aren't level? It's like, because it will tip over in an earthquake. You're not worried about, like, pulling it over. You're worried about the whole thing swaying and dropping everything onto you.

[David Spark]

My first reaction earthquake was what are the neighbors doing? Because I didn't think it was an earthquake. I thought there was just some loud party going next door and they shook my room. And it took me a little while for me to do the basic math to go, no, that was an earthquake.

[Andy Ellis]

Yeah, that's an earthquake. Especially the difference between there's the S waves and the P waves, and after a while you start to tell a little bit of difference between them. And then there was the year that driving into school every day, and half the time I missed the earthquake. I'd get to school and people were like, did you feel the earthquake? Like, I have shock absorbers on my car. So no, I didn't.

[David Spark]

We have another Californian who is our guest today. She has felt a few earthquakes in her time.

[Sarah Madden]

I have. I actually have a SoCal tip for how to tell how big an earthquake is. You look at your pool, if you have one, and you could judge it by the waves.

[David Spark]

There you go.

[Sarah Madden]

You're also giving me SoCal PTSD, talking about cockroaches, because I live in LA and we've got them everywhere, but. Hi, gentlemen, nice to talk to you today.

[David Spark]

That, by the way, everyone, is the voice of the CISO of Convera, none other than Sarah Madden. Sarah, thank you so much for joining us today.

[Sarah Madden]

Yeah, thanks guys. Great to talk to you.

[Host/Announcer]

Where does the CISO begin?

[David Spark]

For our very first segment, let's take a deep breath and let's think about our dreams for building a greenfield IT infrastructure. Now, these dream makers recently popped up on the cybersecurity subreddit and the responses that they had for doing this range from idealistic, like cloud first with zero trust, everything containerized and hardware keys mandatory. And one commenter just wanted accurate configuration management database cmdb, where every team actually logs their infrastructure instead of building fiefdoms. Now, several admitted the real challenge isn't the tech stack, but that quote, if you're reliant on the end user doing the right thing, it's already too late. Now, one of my favorite takes was quote, as a security expert, it will never work, end quote. Because security people building IT infrastructure ignore business needs. Andy, I'm going to start with you. So when you're designing a security architecture, again thinking Greenfield, you got a clean playing field. Are you optimizing for the world you want or the one that currently exists?

[Andy Ellis]

So I think the answer is both. And I love the comment in there about if you're reliant on the end user doing the right thing, it's already too late. And nothing captures the blind spot of security professionals more than that sentence.

[David Spark]

Well, it's the feeling that it is the user's fault.

[Andy Ellis]

Right? It's the user's fault. You need them to do the right thing. And if they don't do that, they're doing the wrong thing is the implication, when the reality is you have to design for humans. Humans actually are the people who provide value in your business. Humans operate with some pretty well understood rules. Like you can model how most humans will interact with things. And if you say, hey, I'm going to put a security control in your way and you have to take an extra 30 seconds every time you do a task, they're going to at some point be like, this is stupid. Why am I wasting my time with this? Let me go around the task. That's not the human doing the wrong thing, that's the human doing the right thing from the business perspective. So always design with humans in mind. Human centric design that deals with the infrastructure you have today, but is aimed at the future. I loved Zero trust. And I know it's a big buzzword, but for me, the core of zero trust is actually taking a laptop and associating it with a human and authenticating the pair of them. And everybody gets this wrong. They try to authenticate the human and build a barrier between them and the laptop. If you say, look, the laptop is yours, we authenticate it with you attached to it. All of a sudden you can do things that are very user friendly, like the user logs in once, proves their identity once, and then the laptop carries on for them. And it's not, oh, I have to do MFA 17 times. No, I proved to my laptop that I'm me and now I and my laptop can go get stuff done. Seamless authentication continuously.

[David Spark]

That is a very good example right there. Sarah, I'm going to put you to the test. I do like Andy's setup there. You got a greenfield, you're building an IT environment. What do you want that is friendly to this IT environment that you're building, but also thinking about the humans?

[Sarah Madden]

Yes, and I have some good perspectives here because I was able to build the ground field security system at convera. Which is the reason I came here. I do want to address something, Andy, that you said real quick though. Trust is not a control. This is something I say all the time in my role. Our job is to design security controls that limit user impact to create an actual massive security incident. Right. So I have to have enough controls in place where users, they do something too quick, something too urgent, something stupid, it doesn't cause an incident, Right?

[Andy Ellis]

Absolutely.

[Sarah Madden]

At the end of the day, they're trying to do the right thing for the business. They're trying to be productive. And with this growth of a bunch of SaaS tools available, it's very easy for users to go rogue and use their corporate credit card, come up with some new SaaS solution for something, and they're doing it with best intent in mind. So control is the name of the game and it's productivity versus control. Right. So bringing me back to the greenfield scenario, one of the things that I think is the biggest bang for the buck from a greenfield perspective is we don't have toolset bloat. We have clarity on what we need to manage. We were able to deploy in one cloud, so we don't have multi cloud bloat, we don't have tool set bloat. We know what things go where. And we had the opportunity to say, we are going to use this control for this thing, we're going to use this tool for this type of business use case, and everybody is clear on what we use for what. And when you come into a scenario where a user doesn't have what they need, they go through this intake process and they say, hey, I want to do this thing. What's the tool for this? And you centralize everything. Right. One of the challenges you always have in other environments is it's just a business that's grown over 5, 10, 15, 20 years and you've got all this toolset bloat and it's very hard to get out of. I'm used to using this thing for this thing and now I have to go do this. We didn't have that problem. Right.

[David Spark]

That actually minimizes problems dramatically. The, the funneling of the requests that they don't go outside.

[Andy Ellis]

Yeah. Well, especially the not carrying forward the legacy. One of the best things I ever did at Akamai, and it will sound very, very silly because I fought so hard during M and A's, said whenever we acquire a company, we're throwing out their entire IT stack, everything. Day one, they get brand new laptops that are pre provisioned as if they're new employees here, and there's a lifespan for their old infrastructure. We're throwing it all away. You have to choose what to migrate. There's no presumption that we keep anything. And every CIO forward from that. When they came in the door, they're like, wait, I don't have stacks of things I have to support from the 30 acquisitions we did. This is amazing.

[Sarah Madden]

Wow. So you just ripped the band aid from the jump instead of trying to unravel things over time.

[Andy Ellis]

Right. Because you'll never unravel.

[Sarah Madden]

Yeah. So we had to do a balance. Right. So we acquired a division of Western Union and we lifted and shifted their assets into a new data center environment that we created. And that was the green field that we got to create. But anytime you buy old legacy systems, it's kind of. And you put it into a brand new network. It's like you're lifting a Pinto into a Ferrari engine. Right. Or Pinto parts into a Ferrari engine. And so there's a part of it that you have to unravel. But I think one of the benefits of modern tooling in the cloud is you have levels of visibility that you never had before. So even when you bring in old stuff, the amount of control you have and the amount of visibility you have gives me comfort that we can manage even old, clunky stuff in a new, shiny system. So I. I love your idea from an acquisition standpoint, where you just say, hey, you are now coming into this world. This is the new system. Ways of working have changed. X, Y and Z. Like, that's a really easy way to integrate the user experience. And that's a different approach that a CISO can take versus you have to bring the systems in no matter what. Right?

[David Spark]

Right.

[Andy Ellis]

You got to bring the production systems over, but you don't have to bring the it.

[Sarah Madden]

Delineating the difference between the user experience and the IT versus the systems that you have to manage from a production standpoint. Yeah. It's a good way to look at it. I like it. I might steal that for the next acquisition.

[Andy Ellis]

Go for it.

[Sarah Madden]

Thanks. Isn't this what we're here for? Sharing tips for making our lives easier and being better at our jobs? Right.

[Host/Announcer]

How is AI going to solve this problem?

[David Spark]

What is the new GRC archetype? All GRC players are AI infusing their products, which calls out for staffers to engineer AI enabled governance rather than just monitor policies. Proposed Nikhil Sarnot of Accenture. Assuming we get to the point where we can automate routine Compliance work, humans will shift into areas requiring ethical reasoning, foresight and creating escalation protocols. Is that where GRC is shifting? And if so, how does this change how we hire for GRC roles like this sort of AI enabled or we were talking about the bionic hacker, so it could be the bionic GRC person. And if you're already working in the field, how do you stay ahead of the sea change? So if I'm in grc, how do I become this bionic GRC person?

[Sarah Madden]

Sarah so we've been on a path of automation for quite a few years, right? And that's the impetus for AI, right. So I think every security person's mindset should be in, if it's not already, how do I automate this? And then eventually AI is going to pick that up. Right. We're at a stage now from an efficiency and scale perspective where we're looking at GRC engineering, where we're looking at policy enforcement and config and code AWS reinforce in Philly a couple months ago had tons of workshops on DRC engineering that several of my team members went to and got really, really excited about. We are shifting the role of somebody on my team that used to be in a primarily audit role, in a security audit role where they're doing all the audit and compliance and testing controls. They're shifting their day to day work into the UI of AWS and the native tool sets that exist there for automating policy in config and starting to actually click automation buttons. Like if this is our set of controls, I can automate it in AWS and I can prevent somebody from ever violating this control. Instead of me having automated audit tool suites that tell me somebody violated a change control policy or an access policy, we're just preventing it in the config. I am really excited about this evolution and the way that we work. So as my team members, they're looking at it from a sense of excitement in their day to day jobs versus kind of doing the same old audit nuance every single day. It is the future of how we will manage compliance in an automated way and it gets us out of this mindset of writing people up or slapping people on the wrist for doing silly things and causing issues and it's just all automated in code and we're getting out of the firefight. This is the part of security that I'm actually really excited about right now. Beyond broader capabilities of AI. I'm really excited about GRC engineering now.

[David Spark]

All right, do you have the same Excitement. Looks like you're totally on target here, Sarah. What about you, Andy?

[Andy Ellis]

So I love Sarah's approach and I've got a slightly different one, but I'm more and more convinced that people should just listen to what Sarah said and ignore me for the moment, by the way.

[David Spark]

Oh, God, that is a soundbite I am taking.

[Andy Ellis]

There you go. Right there. She nailed it. Because the challenge is, I actually hate the term GRC because let's be honest, 90% of the profession is the C. It's just compliance and governance and risk is what the CISO does. And oftentimes we have people whose job is enforce descriptive compliance that we said we had rules, they're not actually controls. We say, oh, we don't do X. No, of course you do X. You have compliance people who hunt for people who did X, and then some Security Ops person comes in and tells you to undo it. And we need to move to a world that is control systems based, where when you say you have a control, what that actually means is it is impossible for it to happen in a different way. That's literally what control means. And I think that's going to be the evolution that AI and really better systems automation is going to give us. We say, oh, look, we bought a bunch of tools. We actually have to verify implementation success, that we turn these tools on, we set them up in a controlling fashion, and that's what our compliance becomes. Now, if today you work in the GRC field, hopefully you're one of the 10%, if you're listening to this, that isn't just doing the compliance validation, but you already know what you need to be doing, which is understanding how do we implement control systems, how do we identify the risks that are being created by bad controls and the risks that are being created by not having controls. That's an architectural function. That's not a traditional compliance analyst function. That's where the humans are going to need to be. They're not going to be the ones who are validating that we have all of our paperwork, because that's automatable. Humans need to be the ones saying, do we actually have the right story? Do these controls fit what our customers need us to do and how our business needs to operate? And what are the gaps that we're going to go forward with?

[Sarah Madden]

Is it designed correctly and can I automate the enforcement?

[Andy Ellis]

Yep.

[Sarah Madden]

Back to trust is not a control. People don't want to fail controls, right? They do it because of mistakes or misunderstandings or just lack of knowledge and awareness. And the more you can automate them out of making mistakes, the happier they are. Right?

[Andy Ellis]

Right. If I don't have to type numbers in, I'm happy.

[Sarah Madden]

Yeah, totally. Totally. And I think the outside of the compliance part of the governance, risk and compliance. The risk part is really important to me too because we've been playing whack a mole with vulnerability management for our entire careers and that's the biggest opportunity with AI is automating vulnerability management. That part I'm equally excited about, if not more probably. Right.

[David Spark]

Automate. Automate.

[Andy Ellis]

Automate.

[David Spark]

Who's our sponsor this week? It is Threat Locker. I told you about this. And we have something new you may not have heard before. Even the most reliable employees make mistakes. We were just talking about this. An unauthorized USB device or accidental click can expose sensitive data and create serious risk. Traditional user based access controls rely on trust. And trust alone isn't security. You remember what Andy was talking about? Securing the device and securing the person. Like the two together. So threadlocker takes a different approach. By enforcing program based policies, it ensures only approved applications can access, read or copy data. Sensitive files stay locked down while approved software continues to run without disruption. And when exceptions are necessary, it does happen. Administrators can jump in and they can improve them in seconds, keeping productivity high without sacrificing protection. Also with ThreatLocker, every action is logged in a detailed audit to capture the exact user file, application and device serial number. Ah, we're getting into compliance issues and watching behavior. So this is all zero trust action in precise, enforceable and it's simple to manage. Discover how ThreatLocker can help you gain more control over your environment. Go to their website, it's threatlocker.com and I'm going to ask you a small favor. When you go to threatlocker.com, do this, just add a/ciso that lets them know that you heard about them through us. The CISO series, It's a small thing. They just want to know that people are coming because they heard about them from us. Threatlocker.com CISO Foreign.

[Host/Announcer]

It'S time to play what's Worse.

[David Spark]

Sarah, are you familiar with this game?

[Sarah Madden]

Yep.

[David Spark]

All right, I'm ready. I'm gonna set you up. This. This is pretty darn goofy. All right. This comes from Howard Holton, who is now the CEO over at Giggo and he's a good friend of the show. And here we go. I make Andy answer first. You can agree or disagree. Andy, this is very different than what you've had before. Okay, you're outsourcing your security awareness training to a random Russian chatbot. That's option one pretty bad. Or you hire a group of clowns to jump out at people every time they click a link in their email. Now that's any link, phishing link, or anything. Which one is worse?

[Andy Ellis]

Wow. Okay. This one definitely wins so far for being the silliest. And honestly, I don't even care about my answer. It's so silly.

[David Spark]

It is pretty silly.

[Andy Ellis]

These are both ludicrous environments. So I think. I think I'm going to go with the clowns is worse. And that is strictly because it is so disruptive and disorienting in the workplace that I would rather have somebody catch me having used a Russian chatbot than have clown running around the environment. And it's my fault.

[David Spark]

But the Russian chat. The Russian chat bot is doing the quote training. The Russian tripod is probably training you, as I put it in air quotes, to do the wrong thing.

[Andy Ellis]

Oh, sure, but you've heard my opinions about security awareness training. That's outsourced anyway. Like, 99% of it is pretty awful. There's a couple of decent vendors, but primarily, most of security awareness training is pretty bad. So, like, fine, I will take that badness over. I have clowns jumping out at my CEO, by the way.

[David Spark]

Hold it. I think there's a net benefit here. I want to get Sarah's answer, and then I'm going to get your take on something else. All right, so Andy thinks the worst of these two scenarios is the clowns jumping out. Sarah, what do you think?

[Sarah Madden]

I definitely agree with Andy. I think clowns jumping out is worse because you didn't say anytime they clicked on a malicious link. You said any link. So now you're going to have users in a state of panic. They're going to make mistakes constantly, constantly worried that they're gonna have ptsd, whereas I am going to have appropriate defenses. So even if the Russian chatbot tells my employees to do the wrong thing, I have controls in place that I mitigate bad ideas and bad logic and psychological manipulation, because that's the spirit of phishing in general.

[David Spark]

Okay, I want to just say all of these are good arguments, but let me throw out this couple things.

[Andy Ellis]

I will point out that Sarah agreed with me.

[David Spark]

So just for keeping score, Sarah agreed with you.

[Sarah Madden]

You're right.

[David Spark]

But I'm going to throw at this. That may change your opinion here. If you truly have the clowns jumping out scaring the crap out of everybody in your office, and everyone's having a panic about it. Let's just think about how incredibly viral these videos would be. If you had video of this, of the clowns jumping up, people go, oh my God. Like that. This would do an amazing branding of. Look at what we put put our employees through and they're still working for you. I think this would be an amazing viral video moment that would put the business on the map and be huge for the business. Andy, your thoughts?

[Andy Ellis]

So do you remember the company that went viral when their security awareness training sent out links that employees had been laid off and they had to click here? That's in the same category.

[David Spark]

I do remember this. I can't remember who it is. No, it's not. It is not. Because this is visually very amusing.

[Andy Ellis]

Oh yes it is.

[David Spark]

That was just cruel.

[Andy Ellis]

But how's this? I'll put Sarah on the spot. Sarah, let's just imagine for a moment that you authorized this and tomorrow a fleet of clowns was in the convera offices or apparently in people's houses jumping out at folks. Do you believe you will be employed at the end of the day?

[Sarah Madden]

Absolutely not.

[Andy Ellis]

There we go. Thank you.

[Sarah Madden]

I think the drive for viral content.

[David Spark]

No, hold on, wait, wait. She was authorized to do this, obviously from the business, so of course she's gonna stay employed.

[Andy Ellis]

Doesn't matter. Even if you went, you went and told the CEO you were gonna do this and the CEO said, this sounds like a great idea. Go do it. You do it. Everybody complains to hr, it goes viral. Do you still think you're gonna be employed by the end of the day even though the CEO had said go for it?

[David Spark]

I'm telling you, the employees are giggling, watching the videos of their colleagues jumping out of their seat because they're scared the crap by a clown.

[Sarah Madden]

Would we all laugh at it? Yes. Is it a useful thing to do? Absolutely not. I argue with regulators all the time that I don't do simulated phishing tests for a reason. Because the data doesn't prove that it's effective. For a lot of different reasons. Right. Like there's so many things that factor into whether or not somebody is susceptible to a spear phishing attack or a general phishing attack. And even security professionals fail them all the time. Right. Like there's so many different things you could do. The age of the employee, like it doesn't work. It's better to do customized training. Do not do out of box security awareness training. We don't buy that from anybody else. We make it internally and we consistently.

[David Spark]

So if I'm a Russian chatbot. I should not be approaching you.

[Sarah Madden]

Try if you want to.

[David Spark]

All right. I try to argue that this could be a huge boon for the business with the viral videos.

[Sarah Madden]

It would be hilarious. It would be hilarious.

[David Spark]

It would be hilarious. It could put the business on the map. This would be a great way that security could be an enabler for the business.

[Sarah Madden]

I mean, we've used embarrassing tactics for different things in the past, too. Remember when people used to leave their keys unlocked and you'd go change their desktop background to, like, getting Hasselhoffed or something?

[Andy Ellis]

Yeah, I have gone a little far in that one. Like, I've got stories from 30 years ago of sending a breakup note from somebody to their significant other who was trying to come up with a reason to break up with them and was like, oh, thank you.

[Sarah Madden]

So if we've evolved in our jokes and our enjoyment, I could get behind this being one of them. Or cattle prods, clowns.

[Andy Ellis]

There are things that are funny in small groups that do not scale organizationally.

[Sarah Madden]

Yeah. Yes.

[David Spark]

That is a good point.

[Sarah Madden]

Yeah. It's not going to be effective.

[David Spark]

Yeah.

[Host/Announcer]

Is this benefiting the company or just making my life easier?

[David Spark]

Quote, these AI engineers find the wildest of bugs and they just keep finding them after every run. That's security engineer Joshua Rogers describing AI native security scanners that recently found hundreds of real vulnerabilities in critical open source software that didn't show in traditional scanners. Rather than pattern matching, these tools can tease out business logic and spot mismatches between developer intent and code. I've previously said automated tools lack human creativity and fall into familiar patterns, but these tools might make me rethink that. I will ask you, Sarah, what does a red team do that can't be automated? What do you think?

[Sarah Madden]

I still think there's so much of the business logic that a red team has that can't be automated yet, but I don't think we're far off.

[David Spark]

Can you give me an idea of what can't be automated?

[Sarah Madden]

I would want to run simulations here. We're not using it yet. I think we need to test this out a little bit more and see where we have gaps. And I think only trying it will tell us and we're not there yet. I think I'm very excited about the evolution here and there's a lot of buzz in the industry and excitement around it and we are going to start using different AI models in the testing that we do, in the tabletops that we do. I think there's a potential for replacing some of our pen testing with this in the future, but we're not, we haven't tried it yet. I want to, I'm interested in it, but I suspect we're going to end up with kind of awareness gaps from a business logic perspective that it might not see. But that's assuming we don't also feed it code bases to figure these things out.

[David Spark]

So we're kind of in a wait and see because I know I met with a bunch of automated pen testing companies at Black Hat. They would like us to believe that you just need their solution. Andy, do you think that there is something that just even with how good AI is getting, that these automated solutions will never be able to really fully understand business logic, that we'll also need a human pen testers?

[Andy Ellis]

So absolutely there is things that an AI will never be able to do. But mostly people don't pay for pen testers to do that. They might say that's what they're paying for. But honestly, if you took every pen testing contract on the market today, I'd be willing to bet AI can do 95% of that. Because honestly, most people aren't even having pen testers read code and do static source analysis, which is what this blog post is about. Like, most people are hiring pen testers to like run a vuln scanner, figure out what they could break into, you know, break in now, look to see where they could go next. Like, this is all automatable. This doesn't require complex AI. This is literally just iterative automation. On the AI side. I'm very excited about the source code analysis capabilities to be able to look at code and say, here's how it's being called, here's where the gaps are. I'm looking forward to MCP almost being inverted and people saying, let's take an MCP based analysis of an API and say what else could it do that isn't advertised, but that represents a risk. That's the things that humans sometimes have a hard time finding unless they're really good and that AI is more likely to find for you. It's like, I have bloated code that has 1000 capabilities, but I'm only calling 20 of them. That means 980 of them are potential vulnerabilities that you need to understand. This is never going to be called. Why do you have it available there? So one day it can be the reason why you're the headlines of the New York Times.

[David Spark]

Mm.

[Sarah Madden]

I think the near term opportunity is just using it internally for Vulnerability management within your Software development lifecycle, STKs, refactoring old code vulnerabilities, things like that. Like we're, we're going that route and then externally red teaming I think will be next. Right. I can get more out of some out of box pen test that you buy just from those third party sites that do your security scanning. Like I could get more out of that than paying a pen tester. Right? And we, we do both, right? We do the pen test that we have to do for compliance, that produce reports and then we pay real people that are really good at their jobs to find that we can't. Right. And that's the part that I think would be hard to replace in the next year or so with AI.

[Andy Ellis]

I actually think there's a business model here which is just external washing your internal pen testers. Because actually most of the value of having an outside pen tester is you get to say, oh, an outside pen tester found this, therefore it's real even if your team knew about it. So you can almost like take your own internal pen test reports, like upload them.

[David Spark]

Oh, by the way, that didn't start with external pen testing, just any external consultant. Their value is greater than anybody telling you inside.

[Andy Ellis]

Right? But imagine a world where you have your own AI based tool, you get all the results and then you have it washed so it looks like it came from an outside company. Show that to your management. Like here's all of our exposures found by a third party. They can see them.

[Sarah Madden]

A crisitunity.

[Andy Ellis]

There you go.

[David Spark]

A crisis tunity. I like it. That's.

[Andy Ellis]

Yep.

[Sarah Madden]

Sometimes you have to create them.

[Host/Announcer]

It's time for this week's security tip. This week's AI infused security operations tip is sponsored by Anvilogic.

[David Spark]

AI can't defend what it doesn't understand. Too often SOCs deploy machine learning models that have been trained on generic datasets, which means it has been tuned for someone else's risk environment or even no one's risk environment. The real power of AI defense is realized when you feed your organization's unique context into the system. From business critical assets to common adversary tactics, data flows and compliance priorities. This contextual knowledge teaches the model what your normal really looks like in your world and helps it recognize those faint signals that indicate when something's off from a user logging in from an unexpected region or a workload suddenly communicating with a new cloud service. These can sometimes be subtle signs that might otherwise get buried in the noise. You could think of it as like training a guard dog. A well trained dog doesn't react to every sound or scent it learns and then knows what belongs and what doesn't. Having AI as part of your defense means letting it understand your specific landscape, which means every detection, correlation and risk score becomes sharper, faster and more meaningful. Rather than having more AI, think about having your AI.

[Host/Announcer]

To learn more about saving costs and optimizing analyst capacity with a hybrid SIM and data lake, go to anvilogic.com.

[Sarah Madden]

What.

[Host/Announcer]

Works, what's not working.

[David Spark]

Quote it seems it's really hard to prove that something doesn't work statistically. Definitely sounds like cybersecurity. That's Jeremiah Grossman, who's the CEO over at Root Evidence. After talking to cyber insurance folks, while they can point to a handful of controls that measurably work, like MFA and edr, they struggle to say what doesn't work at all. Are we just terrible at measuring security effectiveness? And if the insurance companies who have every financial incentive to understand what can prevent a cyber attack and what can't, if they can't tell what's useful and what's worthless, what does that say about how we're making security investment decisions?

[Andy Ellis]

Andy well, I love the question because this is we really don't understand what we're doing. We don't ask the right questions.

[David Spark]

So people should stop hiring cybersecurity professionals because they don't have a clue, right?

[Andy Ellis]

That is one outcome one could take from my statement. I don't know it's the right one. But here's the challenge. How many security professionals look at all their controls and say what would be different if I didn't have this control? Like third party risk management is one of my favorite ones. What changed in your company in the last year as a result of your TPRM program? Are there vendors that the business was going to buy from that now they didn't because you identified a problem? Are there vendors that actually made substantial changes to their security roadmap because you identified a problem? If you can't identify a place in which TPRM helped you, then functionally TPRM is completely and utterly ineffective. Except you have to check the box that you have TPRM because your insurance carrier will get cranky if you don't like this is one of those funny ones is we do things to satisfy our insurance carriers and our regulators that they themselves can't point at how those are being effective. And that's an interesting challenge. But I think the core reason is insurance is actually, and I'M going to get hate mail for this one. It's a real simple problem because it mostly deals with simple systems. Like building houses is functionally the same technology for the last 3,000 years. We stack boxes on top of one another.

[David Spark]

So you're comparing cyber insurance to home insurance.

[Andy Ellis]

Yes. Think about all the insurance markets that work relatively well are all relatively simple systems.

[David Spark]

I always argue that if you looked at an actuarial table of car insurance incidents and home insurance, you would see patterns. Do you think you still see the same patterns in cyber?

[Andy Ellis]

No, you don't see the same patterns in cyber because when we look at homes and automobiles, we look very carefully at things like what were the building materials, where were the houses and built. Like, there's a lot of norms that don't have huge variation in them.

[David Spark]

Well, right, but that's what I'm saying is that I don't think. I think cyber is kind of all over the map, isn't it?

[Andy Ellis]

Because cyber isn't replicable.

[David Spark]

Right. Don't you think this is a more difficult problem?

[Andy Ellis]

It is a very difficult problem, but that's why we don't have data, is because we don't have this large corpus of identical entities doing the same thing every day. Everybody does something different. They all think they're special snowflakes.

[David Spark]

Special snowflakes. Sarah, what's your opinion here?

[Sarah Madden]

I am not going to own that one at all. I am not a special snowflake. I grew up in Alaska. My dad was a fisherman. I could be tough.

[David Spark]

All right.

[Sarah Madden]

If I take the cyber insurance side of it. No. Knock on the dozens and dozens of global regulators that knock on my door. But cyber insurers ask better questions than regulators and auditors and they have to because there's a vested interest. They have to pay out money if something fails. Right. They have been hyper focused on ransomware because that's where they've been paying out the most of their premiums over the last couple of years. And so I think they're overly focused in the things you need to do to prevent ransomware and phishing. And some of the conversations we were having about how to test employees before, I think they've been optimized over that for the last couple years. But I will give them credit that they are better at asking important questions as it pertains to the threat landscape, as it pertains to what will cause a breach that will cause them to pay out a premium. Right. And I think we need them there because we need policies in order to land customers and so I think this is a good driver for security, even better than regulators and audits. From my opinion over the last couple of years, I think it's important to note that regulation and even cyber insurers are years behind the attack landscape. And I think what we don't do a good job about, because that was one of the questions, was we don't do a good job at root cause analysis. We don't focus on understanding the actual root of the problem and then using that as our narrative of what needs to get fixed. I think there's a lot of really great security people that are good behind a computer and they're good communicating with a computer and they're good at analyzing things. But when it comes to articulating risk to get budget to make changes that impact productivity, it requires good communication, a solid way to communicate risk, and a clear understanding of root cause of a problem. And I think that's the hard part. Right? And it's too easy sometimes to go from one incident to the next because, you know, we're defenders. At the end of the day, it's hard to go back to that longer tail process of root cause analysis, risk treatment plans, budgets, arguing with people, changing controls, communicating with employees, that's the harder part. The change management part of security is the harder part.

[David Spark]

I would agree with that. We're going to leave everyone on that cliffhanger. The change management part is the harder part. We are not going to answer that question. You'll have to tune in next week when we may or may not answer it. We'll see. This show is not serialized. Sarah, you are excellent. Thank you so much for coming. Let me just make a few mentions and I want to hear your last word on today's show. Our sponsor for today's episode with Threat Locker. Remember, go to threatlocker.com add the/ciso. It's a great way to let them know that you heard about them from us from the CISO series. Or just tell them, hey, we heard about you through the CISO series. But they really have a very impressive entire platform of zero trust tools. Take a look at it as you're building out your zero trust environment as well. Sarah, any last thoughts on today's episode? And why should everyone be checking out convera?

[Sarah Madden]

Well, if you want to move money around the world, check us out. That's what we do. We move money around the world fast. So you can move money, say for instance, from somewhere in Africa to the US in less than a couple of days or China or Within a day, we move money around fast. Last thought for this podcast Podcast. I thoroughly enjoyed this conversation. Thank you guys. I think sometimes it's hard to step out of your bubble and have conversations with colleagues and there's so much we can learn from each other, if not just a therapy session. I think we're all fighting the same fight every day and just taking a step back and talking to each other and helpful tips and tricks that we learn from each other or just again, having a therapy session. This, these conversations are really valuable. So I thank you for inviting me today and I would just encourage everybody in the defender space to just keep talking to each other.

[David Spark]

So are you saying that I'm your therapist?

[Sarah Madden]

You did help me a little bit today.

[Andy Ellis]

Awesome.

[David Spark]

But everyone heard this therapy session today.

[Sarah Madden]

So we all need it.

[Andy Ellis]

So they think group therapy with the whole industry.

[David Spark]

Well, I'm not bound to any ethical guideline of not disclosing this because we're recording this and it's going to go out to everybody. So everyone's going to hear it. Hear all your dirty laundry, Sarah. And yours as well, Andy.

[Andy Ellis]

There goes your social worker license.

[David Spark]

Thank you very much, Sarah. Thank you very much, Andy. Thank you to our sponsor, Threat Locker, and thank you to our audience. We greatly, and I do not mean this lightly, appreciate your contributions. And for listening to the CISO Series.

[Host/Announcer]

Podcast that wraps up another episode. If you haven't subscribed to the podcast, please do. We have lots more shows on our website, cisoseries.com Please join us on Fridays for our live shows, Super Cyber Friday, our virtual meetup and cybersecurity headlines. Week in Review. This show thrives on your input. Go to the participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you're interested in sponsoring the podcast, contact David Spark [e-mail] thank you for listening to the CISO Series podcast.