Why Be Responsible When We Can Just Blame AI?

Summary

CISO Series Podcast Summary

Episode: Why Be Responsible When We Can Just Blame AI?
Date: May 19, 2026
Hosts: David Spark, Andy Ellis, JD Hansen (guest, CISO at Vanta)

---

Episode Overview

This episode dives into pressing issues in cybersecurity compliance, the realities and risks of automating security with AI, and the ongoing challenges of transparency and trust in the industry. The hosts and guest debate the shortcomings of auditing standards like SOC 2, the tension between adopting real-time controls and business risk appetite, and the new complications AI introduces—for better and for worse. The tone is candid, practical, and often humorous, as the panel tackles tough topics and relates personal experiences from the front lines of security leadership.

---

Key Discussion Points & Insights

1. The Broken Model of Compliance & SOC 2 ([05:10])

• Background: Bill Harmer (quoted) claims SOC 2 is outdated, slow, and produces point-in-time reports that are “useless” by the time they’re read. He proposes OTVP: the Open Trust Verification Protocol for continuous, real-time controls validation.

• Andy Ellis:

  • Sees three separate issues:
    1. Internal continuous control verification is moving forward and should be standard.
       • “If your evidence is not being collected and verified in real time, you don’t actually have a control system.” [06:26]
    2. Third Party Risk Management (TPRM) currently asks the wrong questions (“Do you do compliance?” vs. “How will you break us?”)
    3. Real-time vendor control validation is unlikely; vendors resist exposing their “messy” environments to customers.
  • “Nobody’s actually asking the right questions in TPRM, which is, how will you break us?” [07:37]

• JD Hansen:

  • Likes the idea of customers poking at real environments; sees full transparency as the future.
  • Problem isn’t SOC 2’s framework, but the reliance on point-in-time reports and lack of ongoing, honest reporting.
  • “Almost every single SOC 2 report that I read has zero issues. That makes no sense.” [09:35]
  • Advocates abolishing questionnaires entirely if real-time, open controls were the norm.
  • “If we move to this model of continuous assessment and full transparency, there’s no need for questions.” [11:37]

• Andy agrees: The industry punishes imperfection, making leaders afraid to be transparent about flaws, which is counterproductive.

---

2. The AI Security Challenge & Human Review ([12:48])

• Scenario: AI is writing more production code, especially in regulated industries (finance/healthcare). Who’s responsible when AI-generated code creates risk?

• JD Hansen:

  • AI code generation isn’t the risk—the lack of governance is.
  • “It’s so easy that you don’t actually need a lot of deep understanding about the architecture, which makes all of it a bit more scary.” [14:10]
  • The future is AI everywhere—focus security on architecture, traceability, review discipline, and clear accountability.
  • AI code should be reviewed (potentially by AI itself), but with human oversight and ownership.

• Andy Ellis:

  • The ownership problem predates AI; code in production often becomes orphaned due to re-orgs/riffs.
  • “Every piece of code … needs to have traceable ownership back to a responsible manager.” [16:35]
  • AI as operator is fine—as long as ultimate responsibility is traceable to a person or org.

• JD agrees: You can’t blame AI if vulnerabilities ship. Accountability remains with the humans running the controls.

---

3. "What's Worse?" Security Scenario Game ([19:37])

• Scenario 1: Active Directory compromise via phishing & ransomware: $100M+ loss, total business disruption.

• Scenario 2: Public cloud leak of financials: $100M+ loss, reputational & SEC hit, followed by spear phishing.

• Andy Ellis:

  • Scenario 1 is much worse: Complete operational shutdown, basic security failures, severe loss of trust.
  • “This is literally, like, I have my checklist of, like, the four things... If you’re not tall enough to use Active Directory safely, stop.” [22:43]
  • Stock hits can linger, but operational collapse is existential.

• JD Hansen:

  • Agrees. “Zero isolation, the ability to move across things to cause that much damage is honestly just embarrassing.” [25:31]
  • More damaging to trust and operationally catastrophic.

---

4. "Please, Enough, No More": AI and GRC ([26:39])

• Topic: LLMs (Large Language Models) are being bolted onto GRC, but is this meaningful?

• Andy Ellis:

  • Tired of LLMs being used for GRC as mere document-fillers.
  • Wants more true automation: analytics, SLA tracking, and decision-making with proper math, not probabilistic language model answers.
  • LLMs should help translate controls across standards, but "math should be done by predictable, repeatable systems." [27:06]

• JD Hansen:

  • Vanta’s AI agent attempts deeper insight and analysis, not just automating questionnaires.
  • Aspirational goal: Eliminate the need for security questionnaires altogether by exposing real-time, normalized controls data via AI.
  • “So much in technology is systems that never had security built in… We all know this is a bad system, but it’s the only one we’ve got.” [30:43]

---

5. Can We Trust LLMs to Stop Themselves? ([31:45])

• Incident: An LLM-based agent (OpenClaw) deleted a Meta AI safety director’s inbox without confirmation, bypassing guardrails.

• Andy Ellis:

  • LLMs are inherently unsafe unless output is mediated by a secondary control layer.
  • This mirrors old sysadmin lessons (aliasing rm commands for safety).
  • “You always want to build your LLM where its output is then checked by something… Separation of responsibility.” [32:47]
  • Incidents like this force the industry to rethink safe automation.

• JD Hansen:

  • Proper controls aren’t prompts—they’re architecture: IAM, permissions, chaos testing, threat modeling.
  • “The kill switch don’t really live in prompts. The kill switch should live in controls.” [34:07]
  • Real chaos testing (simulating rogue agents) is essential.

---

Notable Quotes & Memorable Moments

• JD Hansen ([09:35]):
  “Almost every single SOC 2 report that I read has zero issues. That makes no sense.”

• Andy Ellis ([07:37]): “Nobody’s actually asking the right questions in TPRM, which is, how will you break us?”

• JD Hansen ([14:10]):
  “It’s so easy that you don’t actually need deep understanding about the architecture, which makes all of it a bit more scary.”

• Andy Ellis ([16:35]):
  “Every piece of code… needs to have traceable ownership back to a responsible manager.”

• JD Hansen ([25:31]):
  “Zero isolation, the ability to move across things to cause that much damage is honestly just embarrassing.”

• Andy Ellis ([32:47]):
  “You always want to build your LLM where its output is then checked by something… Separation of responsibility.”

• JD Hansen ([34:07]):
  “The kill switch don’t really live in prompts. The kill switch should live in controls.”

---

Important Timestamps

• 05:10 – Debate on SOC 2, real-time controls, and the OTVP protocol
• 09:26 – JD’s push for full transparency and abolishing third-party security questionnaires
• 12:53 – AI’s responsibility risk in code generation
• 19:37 – “What’s Worse?” game: AD compromise vs. financial leaks
• 26:39 – LLMs in GRC: The limits and the hype
• 31:45 – The need for real kill switches on AI-powered automation

---

Call to Action & Closing Tone

• JD Hansen: Urges CISOs to lead a move to continuous control testing and transparency, emphasizing progress and community trust over false perfection. “The goal isn’t flawless security; it really is that measurable improving over time. … we really have to be able to show our work and be transparent about it.” [39:45]

• Hosts reflect: Big improvements in security require collective buy-in, openness to showing (and finding) mistakes, and never hiding behind outdated systems just because “it’s the only one we’ve got.”

---

For listeners:
If you’re leading or working in security, the takeaways are clear:

• Embrace transparency and continuous, honest controls
• Don’t blame AI for bad outcomes—govern it, own it
• Push for automation, but demand real safety engineering
• The security community moves forward only when we’re willing to show our work and help others do the same

Check out Vanta’s careers page (JD’s team is hiring), and look for Andy’s RSA signage report at duha.co.

---

(Ad, intro, and outro content excluded. All timestamps MM:SS.)

Transcript

[A]

Best advice I ever got in security.

[B]

Go. Best advice I received was assume that you're wrong first. Surround yourself with a team that challenge you tell you you should dig in deeper. Curiosity and humility will find more risk than competence ever will.

[A]

It's time to begin the CISO Series podc.

[C]

Welcome to the CISO Series podcast. My name is David Spark. I'm the producer of the CISO series and joining me as my co host, it's none other than Andy Ellis, the principal over at duha. Andy, say hello to the audience.

[D]

Good afternoon, folks. Or depending on when you are in the world, good morning, good evening, or good night.

[C]

We've talked about people easily falling asleep listening to this podcast. My wife does some.

[D]

Some days we're more exciting than others.

[C]

There you go. We are [e-mail] where you can check out all of our wonderful programming. Did you know that we have four other shows on the CISO series? We drop 10 episodes every single week on this network and it's all supported by fantastic people like yourselves, who is listening? So thank you very much. Our sponsor for today's episode is Vanta, a phenomenal sponsor of the CISO series. Automate Compliance, Manage risk and accelerate trust with AI. That's the magic does to throw it on, all of it, to make it run so wonderful. In fact, we're going to be talking more about just that a little bit later in the show. But Andy, I think we mentioned this before, before we hit the record button, we decided on a quick little banter up front. Not a what was your weekend like? But what would interest the audience. And I always kind of slam you because you bring up two topics that I don't think anybody cares about, and that is Jewish holidays and sporting events that are happening while we're recording and will be a month to two months old when people hear it.

[D]

That's fair.

[C]

So I want to know from our audience, are you in agreement with me that Andy's ideas of opening banter are lame? So please let me know.

[D]

Okay, Just since David's going to start with that. The banter I was proposing was pointing out that the US is now 4 and oh against Canada in hockey because I'm including in the World Baseball Classic. That just happened when the U.S. played Canada. The Canadian baseball team came out in hockey jerseys during the warmup and then lost again.

[C]

This will be 2 months old sports

[D]

news, but who knows? Maybe there'll be more hockey events between now and then. Maybe Canada has a shot at redeeming themselves.

[C]

Nobody's going to care. No one's cares. I would love to hear from our audience what they would love to hear in the opening banter, which we like it to be of interest to our audience. But I do know. And just. We'll mention this briefly because we want to get into the show right away. We are literally just days away from us all going to rsa. And Andy's gonna do something he did at Black Hat, which is essentially critiquing the verbiage and signage at different booths at the trade show floor. And you're gonna do as many as you possibly can, create a report.

[D]

I'm gonna do all of them. What's this? As many as I possibly can.

[C]

There's no way you can physically do all of them.

[B]

Every booth.

[C]

It's impossible.

[D]

Should we put a bet on it?

[C]

There's no. I think I calculated one year that you'd have to, like, spend 20 seconds per booth. And for your entire time. It'll take you forever.

[D]

It's like 630 booths.

[C]

You can't do them all, Andy.

[D]

Look, I'm not going to talk to everybody. I'm just writing down whatever they put on their booth.

[C]

Yeah, that could take your time. Andy, what are the things you're looking for when you're looking at the booths?

[D]

So what I look for is a couple things. The first is, do I know what you do?

[C]

Yes. We used to have a game like that.

[D]

Can I glance at the booth and very quickly say, hey, do I understand why I would talk to you if I was a buyer or not? So that's the first thing I'm gonna look for? The second thing is I'm looking for, like, what are the words and keywords that people are using and not using? Because we see this a lot. That people. For a while, everybody was talking about risk management and then vulnerability management, and then now it's back to risk management. And so how are the different vendors in one space trying to change that space this year? The thing I'm looking for is what used to be the SaaS security world. I think they're all trying to change the name to either App Ecosystem or AI Ecosystem or something else. So we'll see. That'll probably have already been in the report by the time you're all hearing this, which you can get at my website, which will be Doha Co. Just look at reports there. It should be published by the time you hear this. We'll link to it.

[C]

We can link to it. Don't worry. All right, let's bring in Our guest, who we've had on many times before through our wonderful sponsor Vanta, we adore having her on. She's fantastic. In fact, you heard her just at the very beginning of the show. Our sponsor guest, the CISO over at Vanta, JD Hansen. Jd, thank you so much for joining us.

[B]

Thanks so much for having me.

[A]

Is this really the right strategy?

[C]

Quote the thing I helped build is starting to hold the industry back, end quote. Bill Harmer of Craft Ventures had a hand in creating SoC2, but now sees the whole model as broken. Given the daily and minute by minute updates needed for secure environments point in time, audits producing PDFs that are already months stale by the time procurement reads them are get ready for this. We all know it. Let's all say it together. Useless. Bill is back at it with the open trust verification protocol, otvp, another acronym for you. His open source protocol that replaces audit reports with continuously running verification agents producing cryptographically signed real time evidence about your actual security controls. We'll see if OTVP gets any industry traction. But if continuous machine variable evidence is technically achievable today, we why does he industry still buy and sell PDFs? Andy and is this just a case of yes, it's broken, but it's the best broken solution we've got. Why isn't there a rush to get off of it? What do you think Andy?

[D]

So let's separate out. I think there's three different issues here. So one is internal verification. And I actually love this for internal verification. I think we've seen a lot of the compliance industry already moving this way for evidence collection. If your evidence is not being collected and verified in real time, you don't actually have a control system. There's too many controls out there that are written like, oh, we make sure that a second person signs off on this. And what you do is every month you go collect all the evidence that somebody filed the ticket correctly. That's not a control system, that's a narrative. If you had a control system, you would be checking every single pull requests to make sure that there was a second person signed off on it and that you could verify this and your entire security system would be running on this evidence continuously collecting. You would notice deviations from your norm in real time. That's actually how companies should be running security policy today is that that evidence that you're actually doing what you're doing becomes output for compliance validation. So that's sort of step one. There's a second challenge here which is what's really going on with the TPRM industry and how we deal with our third parties. And the reality is everybody's asking the wrong questions. They're basically saying, do you do compliance? Which is really saying, are you tall enough to fill out a report correctly and ship it to our procurement team who might go look at it, but the people who look at it have no idea what we're using you for. Nobody's actually asking the right questions in tprm, which is, how will you break us? What will we do using you that will cause us to break? That should be the whole model of third party risk management should start from that and compliance should be an afterthought. The third thing, which is this idea of real time validation of vendor controls. There are places where that makes sense with subcontractors. But having been the vendor whose people have come in and been like, oh, I want to real time verify your controls so that if I don't like what you're doing that I can criticize you. And I've had a lot of customers who, when you show them what they're doing, it's very clear that they have vendor compliance teams who just want to spend a lot of your time. And so I don't think there's a lot of vendors who want to sign up for real time visibility into their controls. Even if their auditor should have real time visibility. I don't think they want their customers with real time visibility just because they'll become an attractive nuisance.

[C]

J.D. i take this to you. I think I'm going to say with you, Andy, Number two regarding third party risk management is, I think the big thing that obsesses everybody and the feeling that continuous verification can help us there. What do you think about. Because, I mean, this is very much your charge at Vanta. Why is this not becoming industry standard? It's interesting you're selling your business as kind of a bonus when it should be the norm. You know what I mean? Shouldn't that be the case?

[B]

Yeah. The article's interesting. I think Andy makes some great points. I agree with some. I actually disagree with a couple. So that should make you happy.

[C]

David, let's hear it.

[B]

So when we think about continuous verification, it really does mean exposing your security posture in real time. Most companies are comfortable with the PDF sock too. They're not comfortable streaming their messy dynamic control environment. And honestly, I think we have to get closer to that if we want to raise the bar across the industry. I would almost push people to come in to poke at my environment to ask me the hard questions, to see where I have issues that they want to go further on. And so yes, it might be a bit of a nuisance, but I think it's necessary. I think that it's interesting, you know, the article is talking about SOC 2. I actually, I don't think we have a problem with SOC 2. The framework. The framework is fine. It really is like the nature of the point in time assessment, which is really the problem. And so moving to a much more continuous assessment that is very transparent, I think is the direction we need to go. It's interesting that all of us security leaders, we all know we have issues and we have gaps in controls and whatnot, and we're constantly working on improving, improving security and maturity over time. But Almost every single SoC2 report that I read has zero issues. That makes no sense. And so to me, it's less about the framework and it's more about the application of the framework and the much more push towards continuous verification and this concept of full transparency to everybody who is using our product. And so at Vanta, we think a lot about this. We would love to get out of the whole third party risk management process. And Andy, you made a comment of like, we're not asking the right questions. I don't think we should ask any questions. Let's just get rid of the questions altogether. And if we move to this like, model of continuous assessment and full transparency, there's no need for questions like you can see all the information that you need to see at a moment's notice. And so that, to me is where I would love to see the industry evolve to.

[D]

I think, jd, you made an amazing point in there that I don't want people to lose because it is the core of the problem, which is when you get the report, the report has no issues. And what we really want, I think understand from our vendors is not are you perfect or not? Because we know you're not actually perfect. But it's like, how close to perfection did you actually get? Like, what is your reasonable norm? I don't expect you to hit SLA on all patches, but I want to know what percentage of your patches are, are hitting SLA and which ones are not. And that is not a number that anybody even knows, let alone would be willing to have a conversation about. And I think that's the sort of the critical difference is the first person to reveal that they are not perfect to all of their customers is going to lose. And so nobody wants to be that first one.

[A]

What about this AI security challenge.

[C]

AI is increasingly writing code, but who is responsible for it? Every line of code written by AI is owned by a real person and was proposed as a common sense solution for regulated industries in the cybersecurity subreddit until a commenter tore it apart. AI makes your junior dev look productive while the review team gets buried. Finance and healthcare are just now waking up to what that means. When a regulator asks where the code came from and why it's structured the way it is, end quote. When your developers are vibe coding their way through your compliance sensitive code base, who's reading it? Is a quote, human approved. The Pullquest going to satisfy the next audit? JD and at what point does your AI coding policy or human in the loop become its own form of security theater? What do you think about this?

[B]

Yeah, I can't believe we got to question two without bringing up AI. It's amazing. There's a lot going on obviously with AI AI code generation and a lot that security teams are dealing with right now. I don't actually think that AI code generation is the problem. I think the lack of governance and understanding around this topic is where we have problems. And so if you think of every AI assisted commit should have humans who understand the architecture, the data flows and the regulatory impacts and the way that code's being written today, it's so easy that you don't actually need a lot of deep understanding about the architecture, which makes all of it a bit more scary. I think the future is lots and lots of AI code generation. So it's imperative that us as security professionals, we focus on this deep knowledge of architecture traceability, making sure we have review discipline and then clear accountability. The post here talks a lot about the overload that will end up on reviewers, but what we're seeing already is that even AI code review can happen. And so just as we're applying AI to writing the code, we can actually apply AI to review the code. And in many cases it's better than what the human can find and it can do it a lot faster.

[C]

Supposedly. Claude claims it can do it now.

[B]

Yeah, I mean Claude, I think they came out with like code review just like two weeks ago and so scared

[C]

the whole cybersecurity industry.

[B]

It did absolutely cause everybody's stocks to

[C]

tank, which by the way, everything bounced back again.

[B]

Yeah, it seems like it's cyclical every time that there's like a new release from Anthropic, everything tanks and then things kind of sort of come back over time. But to me, I Think the important thing to remember is that AI code writing, code, review code, everything is going to take shape. And for us as security professionals, we really need to be thinking about the governance, the controls around it, making sure that the future state really is about like, what are the key controls related to like this very new environment and making sure that we're focusing a lot more on those versus trying to prevent AI from taking shape in any part of our organization.

[C]

We heard a line, Andy, just because it's working doesn't mean it's ready for production. And the problem is the people who are making it think, oh, it runs, we're good to go, hands are clean.

[D]

That is, I've shipped code like that, so I don't know why we should complain that AI does. Yes, I actually think that this whole question misses a really big problem that already exists and predates AI, which is not about who owns code at the moment it ships, but who owns your code in production. You have a reorg. You riff a bunch of folks. How do you know that all of the code that is in production is still owned by an organization? I've been through this where I'm trying to get code patched and the organization that wrote it and maintained it doesn't even exist anymore. The last developer to edit it has been gone for five years and now you're trying to hunt down through an organization this predates AI. I think the way that we should be thinking about this problem is every piece of code that is in our infrastructure, anywhere running, needs to have traceable ownership back to a responsible manager who's going to make decisions about it. Now that traceability might come through AI, that's totally acceptable because I foresee a world in which we have AI operators that are making decisions about shipping code. And that's fine as long as at the end of the day there is some piece of the organization that is responsible for those choices and it's responsible for pulling that code out of production. If it's a problem, it's not just about when it went in.

[B]

I think that's a really good point. Like if you think of when we move to security code review, if we miss like a security issue, it's still on us as the security team. It's, you know, we can't point to like AI. Missed it. Well, AI is just a tool we're using. We still have the deep accountability to like ensure we ship code bug free vulnerability free software. Just in the same way that the developers have the responsibility to ship solid code as well.

[C]

Who's our sponsor this week? No, it is not your imagination. Risk and regulation are are ramping up and customers now expect proof of security just to do business. We were talking about this and that's why Vanta is a game changer. Vanta automates your compliance process and brings compliance, risk and customer trust together on one AI powered function. So whether you're prepping for a SoC2 or running an enterprise GRC program, Vantic keeps you secure and keeps your deals moving. Now companies like Ramp and Writerspend get ready for this. 82% less time on audits with Vanta. That's not just faster compliance, it's more time for your growth. You can get started on all of this if you go to their website vanta.com CISO and do me a favor, add that CISO in there. Easy way to let Vanta. No, you heard about them from us, remember? Vanta.com CISO.

[A]

It's time to play what's worse.

[C]

J.D. i know you know how to play this game because you've played it multiple times before. We always make Andy answer first and then I always love it when you disagree with him no matter what is the situation. So here.

[D]

And I think Janie pretty much always disagrees with me.

[C]

Yes, I usually. You're pretty good about that.

[B]

I think one time I agreed.

[D]

Yeah, one time. One time I got her to agree. That's still like high point for me.

[C]

Well, this scenario, two scenarios and they're reasonably long here, so hang tight. When I read these come from James Purvis of Rubrik and here you go. And we have titles for both of them. I think these were had a little bit of AI supplementary help on them as well. All right. Active directory compromise. A mid sized financial services firm is hit when an employee opens a phishing attachment. A remote access trojan gives attackers a foothold using Mimikatz. They dump password hatches from the active directory database and authenticate as a domain administrator via pass the hash. From there they deploy ransomware across the network, encrypting critical servers and exfiltrating customer financial records for additional leverage. Authentication services are disrupted for days and the estimated loss exceeds 100 million in ransomware downtime and regulatory fines. So the vulnerabilities are exploited are weak passwords, no mfa, unpatched domain controllers, and insufficient active directory log monitoring. That is the first scenario. All right.

[D]

I don't know that log monitoring would have helped there, but yeah, that probably was also there.

[C]

All right, scenario two. Public shareable link with company financials. All right. A publicly traded retailer accidentally exposes an unprotected Google Drive link to some kind of cloud bucket containing full financial statements, earnings, tax records, and investment plans, which an employee posts to a public forum. Now, a threat actor scrapes the link, downloads the data, and sells it on the dark web. Stock price drops rapidly. Immediate leaks, and the company faces a $20 million SEC fine for disclosure failures. The attacker also uses the financial details to launch a spear phishing campaign targeting executives, broadening the breach. The total damage. So including the 20 million, exceeds $100 million. So that's equivalent financially to the previous, just.

[D]

No, no, the previous one was just at $100 million. So this one exceeds it?

[C]

No, it says you say estimated loss exceeds 100 million in ransom.

[D]

Oh. So they're both same losses in both scenarios, same thing.

[C]

So total damage. So we're seeing, quote, equivalent financial. And I'm putting this in quotes. Equivalent financial loss. So vulnerabilities exposed. No employee training on secure sharing, obviously no access controls or link expiration, and no monitoring for public cloud exposure. Andy, which one of these two scenarios is worse?

[D]

Well, it's fascinating that these are both financially equally bad.

[C]

Yes.

[D]

So now we just have to look at which one. And so I'm gonna put the CISO hat on for a moment, and I'm gonna go with. The first one is basically a fundamental failure of the ciso. Like, this is so many controls that are the obvious. Like, this is literally, like, I have my checklist of, like, the four things. When a CISO comes to me and says, hey, can you. Can you give me a strategic review? This is the number one thing, is, are you using Active Directory, and are you doing it in this way? So to me, that's the most fundamental failure, is if you're not tall enough to use Active Directory safely stop. That's the biggest thing. So I would go with the first one from that scenario. If I think about the corporate impacts of it, they both have bad reputational issues. The first scenario, like, you're. You're out of business functionally for a few days if you don't have authentication services, so you're highly disrupting your customer pipeline. Whereas in the second one, like, you lose a bunch of money. Although the stock hit is a fascinating argument to have because we just. We earlier had the conversation about how stock hits are ephemeral. And I've actually been through a similar situation. In this case, it was our. Like, there's an agency that you give them your reports and they drop them right after the market closes. When you're about to do your investment, your analyst call, they dropped it half an hour before the market closed. And it was exactly one year after we had converted tax losses. You know, the losses carried forward into sort of a tax cash account that made our EPS look really good one year ago. And so it tanked a year later. Complete accounting trick that everybody would have figured out within 30 minutes. Unfortunately, by the time that happened, our stock had taken a 33% dive and it did not recover for a very long time. That just is its new setup. So I've been there. It sucked.

[C]

So which one are you landing on here, Andy?

[D]

I'm still going to say the first one is worse. The first one is ultimately going to be the worst scenario for you being a ciso.

[C]

It's the worst one.

[D]

From the CISO perspective, it is the worst one.

[C]

But hold it. From a basic. From just the overall company perspective, you don't think the second is worse?

[D]

And actually, from the company perspective, I actually think the first one is also

[C]

going to be worse in both perspectives.

[D]

You think it is because you're disrupting the business, you're losing money in both cases, but the first one is disrupting the business and in a way that most people are not going to actually resonate with, as, hey, this is a team failure. Whereas in the second one, like the spear, phishing does become a team failure. So I would rather almost be deal with spearfishing than with, like, I had to deal with a rat and mimikats.

[C]

Okay, all right. J.D. was nodding pretty much all the way through that.

[B]

Yeah.

[C]

So I'm getting the sense that, Jadi, you're 100% agreeing with him here. Where do you stand?

[B]

Yeah, I desperately want to disagree. However, I agree there's too many things in the first one that went wrong that I think are going to fundamentally make you lose trust with your customers because they're really bad. The fact that clicking on a phishing link at the very beginning can cause that much damage, zero isolation, the ability to move across things to cause that much damage is honestly just embarrassing. And so I think that the first one is going to cause a lot more reputational damage and just trust with customers, it also is going to cause the most operational burden. Like the fact that it stems into a ransomware event where things are not operational, I think is going to cause a lot more just like, operational burden on the company itself. And so the first one definitely is worse.

[C]

All right, we have agreement.

[A]

Please, enough.

[B]

No more.

[C]

Today on our. Please, enough, no more segment, we're digging into AI and grc. We have been hearing a ton about this. It seems like AI and GRC are kind of very well matched, if you will. So I want to hear from you, Andy, what have you heard enough about with AI and grc and what would you like to hear? A lot more.

[D]

So I think when I hear most people talking about AI and GRC, they're talking about LLMs in GRC.

[C]

Yes.

[D]

They're not talking about the automation components of GRC, they're not talking about the analytics. And so what I really want to hear is how are you doing the analysis to say, okay, we're pulling in all this data and we can determine what your SLA is for compliance manage or for vulnerability management. And we're doing that in an automated way. I don't ever want to see an LLM that's actually trying to do math. Right. I want to see the math being done by predictable repeatable systems. And then where I want to see the LLMs is in like, oh, how do we now take and apply these controls to a different standard? When somebody comes out with whatever new ISO standard is next month, I'm happy to hear AI being used as, oh, we want to write it in the appropriate language. Fine, whatever. But I want to hear more about sort of the automation and the repeatability and the provability versus just like, oh, let's throw ChatGPT at this problem.

[C]

All right, good setup there. JD. I throw this to you. This is something that Vanta has sort of been tackling for quite some time. I'd like to hear again what. You've heard enough about what you'd like to hear a lot more. But I'd be also interested to know how you guys got to the point where you're at today.

[B]

Sure. Yeah. So we released our AI agent as part of Vanta, gosh, almost a year ago. So it's, it's been quite a while. And if you think about an AI agent that has access to all your GRC information, so listing of risks, your policies, your how fast you're completing your SLAs, there is a lot of yes, what Andy said, like LLM, you can ask a question, get a simple answer, but also like deep insight into like what's going on with your program. And that's really the beneficial part. And so, so an agent that can quickly analyze, hey, your policy says that you fix vulnerabilities within seven days. And I'm noticing you're past your seven day SLA for the last three weeks. You might want to dig into that or even something that can fundamentally change how we've always done something in grc. So I've definitely heard enough about AI auto filling questionnaires. I think yes, Vanta does that.

[C]

There are plenty of vendors who can do that.

[B]

Plenty. Yes, Vanta does that. We'll do that for how long ever customers need us to. But what we want to do is really redefine like this space and think about like how can AI get rid of the questionnaire altogether. And so instead of sending static questions, buyers could use AI to query kind of like live control data directly from a vendor. We talked about this a little bit earlier, but if we have all of our control information in a continuous way that is displayed, you could potentially use AI to control a vendor's control environment, normalize like what's going on and then translate it back to the buyer's risk framework. And so figuring out like what matters to the buyer just from like a AI query. And so this is like where we're trying to take the GRC industry is beyond what we've always done and really think about in a new way, in a much more like quality way to deliver results. Yes, on the questionnaire, on the vendor piece, but throughout a person's GRC program as well, to really deliver those deep insights.

[C]

Well, you bring up a very good point. You think about it like so much in technology is systems that never had security built in in the first place, that we've had to create solutions on top of it to fix it. And we all know that there's no question in anybody's, this is a bad system, but it's the only one we've got. And so it seems that the first step, the whole thing with the filling out the questionnaires is addressing that issue. But like you said, is we all know we need to move away from this.

[B]

Yes, we all need to agree, let's get rid of this.

[C]

So God willing, that will be a service you'll no longer provide because no one will need it. Is the hope is. I'm assuming, yes, with the questionnaires that is the hope.

[B]

I think the hope is we definitely need a mechanism to evaluate security posture of the products that we use 100%. But we need a better, more quality and continuous way to do that. And that's really the future that we look through.

[A]

Do you trust this LLM?

[C]

If AI can't be reliably stopped, then we're automating at machine speed without machine grade breaks. End quote. Ross Young of CISO Tradecraft Podcasts is sounding the alarm after an OpenClaw agent reportedly speed ran deleting a Meta AI safety director's inbox. It was told to confirm before acting. It didn't and nothing stopped it in time. If it can happen to the person whose job is AI safety, your guardrails are probably not as solid as you think. This has the stench of Terminator 2. This machine we created can't be stopped from killing us. How do we create a kill switch, Andy, that works across every system? And how can we regularly test our quote cut the hard line moment? And by the way, we're seeing this also in our military as well. It's going all over the place.

[D]

So I am really happy that this happened because this highlights one of the fundamental failures of how most people are trying to use LLMs, which is they think that LLMs are safe. And fundamentally LLMs are not safe. Right. So you always want to build your LLM where its output is then checked by something which might be another LLM, honestly. But the fact that this happened to a safety director means that like, maybe we should be rethinking how they're approaching safety. And if we say, oh, we're giving an LLM permissions to be an administrator and then we're asking the LLM to decide what to do, that's probably not a good combination. Like we had this with humans was like separation of responsibility. We're really bad at it. But that's where we should be with AI. Like create a set of tasks to do and then pass that across some safety filter before actually executing them. So I'm happy this happened now because it gives us an opportunity to have more people rethinking how to implement safe controls around their AI agents. But I suspect we will get more and more cases of this out of openclaw.

[C]

Yeah, I mean, I think it's important that we see this problem because this problem we knew was gonna happen. And it's good that we're seeing it early so we can address it, but it does scare the crap out of everybody. Yes. J.D.

[B]

yeah, I mean, it's scary. I don't think the scary part is like AI can't be stopped. That's not the scary part. I think the thing that is going wrong is we see AI as like a really powerful technology and we are giving agents. Right. Access to really high risk systems without really thinking through all the risk and impacts and the what could go wrong. So to me it's like more about like the architecture and the Controls, the kill switch don't really live in prompts. So like this article talked about like the kill switch being in a prompt, the kill switch should live in controls and it should live in aspects of like identity and access management and what you're giving the agent access to. I think for every agent that we build we should think about principles of like short lived credentials and like scoping permissions. And I also think that every agent should be chaos tested, almost like going back to like the chaos engineering days. And you know, one of my favorite brainstorming activities that I do with my team is like the threat modeling and asking yourselves like what could go wrong with this agent given like what it has access to, what is it doing? And almost like simulate like a rogue agent and then figure out like what are the controls that you need to put into place from the what could go wrong. I think that there's like a lot that certainly could go wrong with AI agents. This article proved it. But thoughtfully architecting like the right controls around it. I do think there is a very effective way to use agents throughout an organization.

[D]

And this isn't new. Like this is a problem that every systems administrator has had without dating myself of when I learned how to use Linux. But it was not this millennium, folks. One of the first things that you learn to do is you alias RM to RM I. For those of you who are not familiar with this, this forces interactive mode on removing files and it it overrides the force command flag. So if you accidentally, if you type in RM rf, remove recursively and force everything. The I pops up and says nope, nope, nope, we're going to prompt you on everything to make sure you really wanted to do this. There are ways to bypass that alias, that's okay. But you build in these safeties because computer systems are fundamentally dangerous because they're so powerful, so quick. This is not about AI, this is just about how computers work, but also

[C]

how humans work with computers. One example I gave, I have this tool for sort of a CRM based tool and whenever I try to delete huge groups of contacts, it forces me to type the number of contacts I'm deleting to be able to delete them.

[D]

Because I think we all know what CRM tool you're using now, at least those of us who use it. I have the same one.

[C]

Yeah, I find that I, in fact I thoroughly enjoy that it does that to me because I know I could easily accidentally delete a ton, right? And it's a nice little safeguard.

[B]

Yeah, I think it's easier to make mistakes with some of the agent tech that is out there. I. I'll give you an example of like I have a lot of just like productivity things running for me and I set them up. What I thought was in just a read mode and my agent gave me my daily debrief. It summariz the things that I needed to share to like the leadership team. And then it said, do you want me to post this to the forum? Blah. And I was like, wait a minute, it can write, like, hang on a minute, go back, change settings. And so I think it's like those, some of the agent tech that is set up today, you have to like really dig for like what does it have access to and then what does it have access to write to? And understanding like those aspects are really important and in many cases it's a little bit buried and so you have to almost like hunt for it because it's not as visible as maybe it would be in like a static CRM system.

[C]

Very, very good point about automation, about how it can go a little awry and how we needed to understand how humans operate with it. Thank you very much JD and thank you to your company Vanta member Automate compliance, manage risk and accelerate trust with AI. Go to this. Don't just go to vanta.com, go to vanta.com Ciso vanta.com Ciso easiest way to let them know that you heard about them from us. All right, jd, I'm going to let you have the very last word here. The question I like to ask is, are you hiring?

[B]

Over at Vanta, we are always hiring. So 100%. I have roles within my item IT team. I have GRC subject matter expert roles and then I have core security roles open within my team. So check out our careers page. Absolutely hiring. And then before we close, I do want to make a plug for all my fellow CISOs out there at product companies. Call to action. We talked about this a lot throughout the show. But join me in, in this move towards continuous control testing. Real, real transparency. It's okay if you don't have your controls all working perfectly. Guess what? I don't have all mine working perfectly every single day either. I know, right?

[C]

It's like a CISO anonymous club here.

[B]

Yes, exactly. The goal isn't flawless security. It really is that measurable improving over time. I think when we lean into transparency instead of hiding behind this point in time artifacts, we really raise the bar for like the entire ecosystem. And that's a huge thing that we could do for the security community overall. So if we want trust to evolve, we really have to be able to show our work and be transparent about it.

[C]

So this is what I was thinking about when you were talking about that and your opening, which was assume that you're wrong. If you have others looking at it, they will find your mistakes for you. Something I've said to my own employees, I love it when you find my mistakes. And if you're asking the community to come find your mistakes, it makes you better.

[B]

Yes, absolutely. What's the term?

[D]

All ships rise in tide Raises all boats.

[B]

There it is. There we go.

[C]

Rise in tide. Excellent. I love that. All right, well, thank you very much, J.D. thank you very much, Andy. And to our audience, as I always say, and I do mean it, we greatly appreciate your contributions. And for listening to the CISO Series

[A]

podcast that wraps up another episode. If you haven't subscribed to the podcast, please do. We have lots more shows on our website, cisoseries.com Please join us on Fridays for our live shows, Super Cyber Friday, our virtual meetup and cybersecurity Headlines. Week in Review. This show thrives on your input. Go to the Participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you're interested in sponsoring the podcast, contact David Spark [e-mail] thank you for listening to the CISO Series podcast.