Not Enough Hallucinations? Let’s Outfit Your LLM with Another LLM

Summary

Podcast Summary: CISO Series Podcast – "Not Enough Hallucinations? Let’s Outfit Your LLM with Another LLM"

Hosts: David Spark, Eddie Contreras (Frost Bank), Andy Ellis
Guest: Anthony Kandeas (CISO at Weight Watchers)
Date: July 8, 2025

---

Episode Overview

This episode delves deep into the evolving chess match of cybersecurity in the age of AI and LLMs (Large Language Models). The conversation centers on the practical and philosophical challenges of deploying AI agents in cybersecurity, how to treat and supervise automation, pitfalls in current AppSec training, and the increasing complexity (and sometimes circular logic) of using AI to manage other AI systems.

---

Key Discussion Points and Insights

1. Cybersecurity as an Evolving Chess Match

• Opening thoughts: Cybersecurity is a never-ending game of adaptation as both threats and defenses continuously evolve.

  “Threats are evolving, which requires defenses to evolve. This is a continuous cycle with no end in sight and it makes cybersecurity so much fun to work in.” – Anthony Kandeas [00:04]

---

2. Food at Cybersecurity Conferences – A Lighthearted Intro

• Bad conference food pushes attendees out of vendor-sponsored areas, hurting vendor visibility and engagement.
• Insight: The experience and environment around conference food matters—plain trays and lackluster options send people elsewhere, impacting event ROI for sponsors.
• Quote:

  "If you don't have an experience built around the food, it's going to go and eat and you're not going to be able to get them back in." – Eddie Contreras [03:18]

---

3. Treating LLMs Like (Junior) Team Members

• Main question: Should AI agents/LLMs be treated as junior team members, given some recommend onboarding and educating them similarly?
• Eddie draws parallels to the familiar "90-day onboarding" for new hires but notes it's more nuanced with AI.
  • You can't just drop AI into an environment; context matters for both people and machines.
  • Context adaptation is essential.
• Anthony’s take: AI shouldn’t be credited as "junior team members" but rather as "tireless interns," stressing a practical, low-expectations approach.
  • Supervising AI is about scalable quality assurance – trust but verify.
  • Quote:

    "Treating them as junior team members is giving them too much credit...they are tireless interns." – Anthony Kandeas [08:05]

  • Output QA is critical; creating audit logs and artifacts are essential for trust.
• On measurement:

  “How do you understand that these outputs that these AI models are coming to... are accurate?” – Anthony Kandeas [09:09]

---

4. Hiring for Potential, Not Just Credentials

• Prompt: How do you uncover if a candidate truly has a willingness to learn and problem-solve?
• Anthony values ingenuity and evidence of active learning (e.g., following Brian Krebs) above rote knowledge.
  • The how of solving unfamiliar problems, setting up safety nets, and readiness to fail and pivot is more important than technical definitions.
• Eddie: Looks for comfort with feedback and the ability to read the room.
  • Hiring shouldn't just be about the technical resume; dialog, feedback, and problem-solving instincts matter just as much.
  • Quote:

    "Don't always fall in love with the smartest resume...it's the conversation and the dialogue in the interview around constructive feedback." – Eddie Contreras [13:15]

• Anthony expands:

  "Did they create any safety nets in their solution?...That shows a level of sophistication higher than other candidates." [14:45]

---

5. What's Worse? Security Skill vs. Trust

(Game Segment: "What's Worse?")

• Scenario 1: Highly skilled, effective security professional no one trusts or wants to hear from.
• Scenario 2: Popular, well-liked, but technically incompetent, can't improve company security posture.
• Both guests agree: The first scenario’s isolation and lack of influence is worse—for morale and for the company’s ultimate security posture.
  • The technically skilled person at least can respond in crisis, but being powerless is deeply unpleasant and organizationally dangerous.
  • Memorable exchange:

    "In the latter, ignorance is bliss...until the day the S hits the fan." – Anthony Kandeas [21:03]
    "If they really love me that much and something really did hit the fan, I'm gonna get funding to bring in a third party..." – Eddie Contreras [22:28]

• “I told you so” culture is unhelpful. Real change requires top management buy-in and awareness, which isn’t always there—even after a breach.
  • Quote:

    "If you look at every breach that's been out there, there's always artifacts where this was being discussed." – Eddie Contreras [23:26]

---

6. AppSec Training: Compliance Box-Checker or Valuable?

• Reddit’s critique: Is AppSec training just checking a compliance box?
• Eddie: It's what you train on that matters. Make it relevant and engaging; tie it to business outcomes and actual code practices.
  • Cookie-cutter, detached content falls flat.
  • Quote:

    "If you're just pulling stuff off of NIST and OWASP and saying these are the things that we want to talk about, it can be dry. But if you're making it relevant to your organization...that's when you start to talk about, okay, this is energizing, it's engaging." – Eddie Contreras [25:49]

• Anthony: Threat modeling and immediate feedback are better for awareness.
  • Questions the real ROI of mandating hours of generic AppSec training for all developers – significant opportunity cost.
  • Envisions "vibe security code coach" as an IDE agent that provides real-time feedback, fitting the workflow and passions of developers.
    • Quote:

      "Why isn't there this instant feedback in the IDE as these developers make mistakes, learn how to get feedback immediately, and get reinforced on secure coding practices?" – Anthony Kandeas [27:17]

---

7. AI to Solve AI Problems? (Stacking LLMs)

• Discusses Google DeepMind’s new tool that increases transparency between APIs and LLMs.
• Should we use one AI to check or moderate another, or is this just compounding complexity?
• Anthony: Disagrees with “don’t use AI for your AI problems.”
  • Argues the solution is domain-specific LLMs—not just stacking general ones.
  • Observability/logging is critical. Specialized, focused models can validate outputs from broader ones.
  • Quote:

    "We should be using AI to solve our AI problems 100%. But...it shouldn’t just be these broad, large language models...they need to be intentional." – Anthony Kandeas [31:17]

• Eddie: Compares it to "Inception"—stacking layers on layers can be valuable if each has a clear, bounded purpose.
  • Understand what AI is for—detection, quality control, repetitive work, etc.—and integrate accordingly.
  • Quote:

    "AI is a part of the ecosystem...it's really understanding how to make the most out of it, as opposed to just putting it back on the shelf and pretending like that the spinner stopped..." – Eddie Contreras [32:34]

---

Notable Quotes & Memorable Moments

• On AI supervision:

  "AI has the ability to have context, it has the ability to understand environments...so you can't just launch an army of agentic AI agents and say it's going to perform just like my team." – Eddie Contreras [07:11]

• On problem solving in interviews:

  “Did they create any safety nets in their solution?...That shows, I think, a level of sophistication higher than other candidates.” – Anthony Kandeas [14:45]

• On training effectiveness:

  "Everyone's vibe coding...where is the vibe security code coach in this stratosphere?" – Anthony Kandeas [27:17]

---

Timestamps for Key Segments

• Opening Cybersecurity Chess Match & Conference Food: [00:00–04:59]
• LLMs as Team Members/Interns, Supervision, QA: [05:05–10:30]
• Hiring for Learning and Problem-Solving, Interview Approach: [10:30–15:38]
• Game: “What’s Worse?” Skill vs. Likability: [17:22–22:49]
• AppSec Training: Value or Checkbox: [24:47–29:51]
• AI to Solve AI Problems, LLM on LLM Debate: [29:51–34:20]

---

Conclusion

The episode weaves together practical strategies and high-level thinking on managing both human and machine actors in cybersecurity. The hosts and guest emphasize contextual understanding for both people and AI, the importance of tailored training and real-time feedback, and a pragmatic embrace (rather than rejection) of AI complexity—so long as it’s supervised and intentional.

Final Takeaway:
You can’t just plug in new talent—whether human or AI—without attention to context, culture, and continuous supervision. Security effectiveness hinges on aligning technical capability, learning agility, and clear-eyed management of both tech and team dynamics.

---

For more episodes and show notes, visit cisoseries.com

Transcript

[A]

What I love about cybersecurity go the lifelong chess match.

[B]

Threats are evolving, which requires defenses to evolve. This is a continuous cycle with no end in sight and it makes cybersecurity so much fun to work in. What we did 10 years ago isn't what we need to do today.

[A]

It's time to begin the CISO Series Podcast. Welcome to the CISO Series Podcast. My name is David Svark, producer of the CISO series and with me, who is becoming a regular co host, usually on the other show, but I invited him to be on the show and he's been on this show before, but not as a co host. It's none other than the CISO over at Frost Bank, Eddie Contreras. Eddie, welcome. Thanks for joining us.

[C]

Thanks for having me, David. It looks the same on this side of the pool as on the other side of the pool. So I think we're familiar now.

[A]

He's going to be able to handle it. I have confidence in you, Eddie. You'll be able to handle this. We are [e-mail] where you can find all of our other wonderful programming. We have a total of five shows now on our network, so why not check them all out and enjoy our sponsor for today's episode, a spectacular sponsor of the CISO series. And that would be Vanta Automate Compliance, Manage Risk and Prove Trust Continuously. More about just that a little bit later in the show. But first, Eddie, here's a topic I want to bring up with you and that is food at conferences and trade shows. And I'm not talking about like you go to the local neighborhood restaurant. I'm talking about the food that they actually serve you there. Now here's the point I want to bring up. If it's a one day event and you get good food, wonderful. You feel great, that's fantastic. But if it's bad food, it's like you didn't know you were going to get served bad food and you just kind of got to suck it up, right?

[C]

You do, you do. I think it's. It can be dangerous sometimes, depending on the venue. Sometimes I think they experiment on some of the attendees. You may see things that you may never see anywhere else. So you may be watched, right? You might be in a fish tank and say, hey, what happens to this person as they eat this new hors.

[A]

D' oeuvre if we give them a lot of calamari, right?

[C]

Exactly.

[A]

Well, that's actually an attempt at good food. You've seen it where you just look at the Spread and you go, why were we served this? I mean, it paid a lot to be here. Now when it's a multi day event, I'm going to use a big one that just passed rsa. And on day one, you realize there's bad food. What happens on day two, three and four? Eddie, what do you do?

[C]

You find the vendors that have the steakhouses and then you go ahead and make your reservations early on. So there's always great eating outside the event, Right.

[A]

Or if it's lunch, you go somewhere else, right?

[C]

Correct.

[A]

And often you're on an expense account with the company and it's no big whoop to you. It's a free meal to you either way. Now here's the problem for the vendors who sponsored the event. If people are leaving to go get lunch, they're not necessarily in a rush to come back. Especially if it's an event like RSA where there's a lot going on that's not in the conference hall. So have you run into this nonsense as well?

[C]

I have, and I think that's why you hear about dinner in a show. It's not always about the food, it's about the environment. And I think if some of the sponsors. Yes, you're going to be carting out food, snack bites, finger foods. You have to make it exciting for people, want to stay there and have the chat. But if it's just kind of off in the corner, if it's just a tray in the middle, if you don't have an experience built around the food, it's going to go and eat and you're not going to be able to get them back in.

[A]

So you're taking a different angle than I was thinking because not all vendors are off offering up food. But the point I'm trying to make is if I'm a sponsor of an event that is serving bad food and it's a multi day event, those people wandering the floor are going to not be as numerous because they're going to physically be out of the venue. So my call to other vendors sponsoring big events that choose all of a sudden to serve bad food, which by the way, I just want to mention RSA used to serve great food. I was very happy. It's just this past year that kind of not so good. So I just call on them, please. And for the vendors as well, pressure them, just make the food. Not a situation where people want to run away.

[C]

I agree completely.

[A]

That's, that's all. I mean, it should just be at the level of I Don't want to run away. All right. With that being said and the irony of who the guests I'm bringing on, we're talking about food. Didn't even dawn on me that we're talking about food. And this is the guest we have on. He's actually the cease over at Weight Watchers. Actually been wanting to get him on for quite some time and now he's finally with us. It's Anthony Candeas. Anthony, thank you so much for joining us.

[B]

Thanks for having me on, David. I appreciate the time and looking forward to a fun discussion.

[A]

Do you trust this LLM? Start treating AI agents like junior team members. Train them, test them, watch their outputs and never assume perfection. Now this narrative around agentic AI is it will be the next big thing given AI systems the ability to make decisions and do actual work across a variety of workflows. But that doesn't mean we should let them run unsupervised. As Peach George of Mantech Digital Transformation Consulting pointed out on LinkedIn. Now I like her idea of treating them like junior team members. But I'm going to ask you Eddie, where does that metaphor start to break down? Because you really can't treat an AI like an employee. So how do we scale that approach when your direct report of AI agents could be in the thousands? What's your take and like what is it similar to a junior team member? Where is it not similar?

[C]

Yeah, and I'll add senior members as well. When I look at my leadership team and they join our organization typically I'm going to ask them. You're going to hear a lot, you're going to learn a lot. I'm going to ask you to hold your position for about 90 days. Really just take it all in. Don't make decisions until you understand why we do the things that we do. Because there's always a reason behind it. And I talk to our leaders that way because sometimes they'll come in wide eyed, have a really good opinion, they know where it was done better and they have to really understand why we did that. And that's why leaders typically have that 90 day plan. They're learning the ecosystem, they're learning the environment. Same thing with junior employees. And that's really when you bring in a junior employee, they're sitting there shadowing somebody, they're really watching, they're understanding the environment matters. And that's why you just can't pick up one person from one company and just drop them into another one splunk environment doesn't look like the next. Right. So you have to learn context. You really do have to understand.

[A]

Can you teach your AI context then?

[C]

You have to. Right? And AI has the ability to have context, it has the ability to understand environments. AI also has as well aware the notion of feelings. Whether it's mechanical or mathematical, it gets there at some point in time. So you do have to be able to bring context to the decision making process. But it takes time to get to that context. And so you can't just launch an army of agentic AI agents and say it's going to perform just like my team. No, they need the context to be able to perform just like your team. So you would never just bring an employee and say, go at it, read the runbook and now make some decisions. They have to sit, they have to shadow, they have to learn, they have to be molded into that environment. And you look at agentic AI as a very similar way. Follow that model.

[A]

All right, Anthony, I take it to you, you agree, disagree. What's your take on this?

[B]

I think first and foremost, treating them as junior team members is giving them too much credit.

[A]

Okay.

[B]

When I started socializing this in my organization, the way I teed it up was that they are tireless interns. And the reason I approach it from that perspective is that it really level sets on the expectation of the output. Right. I think the junior level members can produce some high quality work in particular areas and projects, but the interns are just a little bit of a lower bar. Right. And that just level sets expectations. When I'm talking to the C suite on the level of sophistication that we can get out of AI, I think AI is a ton of use cases, obviously, but we should also like step back and set expectations. I think to your point, we should really step back and think about how do we trust but verify these AI agents, Allow them to do what we've programmed them, let them have some autonomy, obviously, and perform the actions that they're trying to take, but also have supervision along the way.

[A]

So let me dig into that. Hold it. What is that? Supervision and trust bus verify what is it you're doing there?

[B]

Yeah, it's really quality assurance at scale. Right. How do you understand that these maybe not actions, but outputs that these AI models are coming to the conclusion of are accurate? Right. How do you, when you think about like this AI implementation, the happy path is very easy to figure out. Like we're going to do these five things and AI is going to solve these things. But now how do you close that loop on the back end to say, let's start measuring how effective the outputs actually are of the AI to make sure it's actually going back to the initial use case and it being effective.

[C]

Yeah. The thought process around quality control, and you think about audit logs, you think about being able to produce artifacts that show exactly what Anthony's talking about. Okay, it said it did A, B and C. Now let's look at the audit logs. Did it do A, B and C? Did it skip B and go straight to C? And so quality control is vital to be able to say, I trust what's in front of me. And I think, you know, if you look at assembly lines, if you look at IoT, they do that very well. Agentic AI is going to probably have to follow a similar path.

[B]

Yeah. Because, like, one of the things that we're very prescriptive on is that we want hospitable customer service bots. Right. So now how do we measure hospitable? Right. That's actually challenging to do in some ways. So that's a continuous, iterative process that we have to use QA to help us hone in.

[A]

Would this person be a good fit for the job? There's no certainty when it comes to new hires, and it can lead to a lot of misaligned expectations that came up on the cybersecurity subreddit with the comment, being new isn't the problem, but there has to be a willingness to learn. What I've seen instead is people talking a big game, then barely putting in the effort while the rest of us clean up after them. And when they do try to contribute, we end up spending an entire day fixing what they broke, end quote. So willingness and ability to learn are important skills that any CISO would obviously want in a new hire. Now, by the way, this quote came from a Reddit discussion, but my question to you, Anthony, is what are the questions to ask to see if your staff can educate themselves to learn new skills, solve new problems, or try to make sense of something completely unknown?

[B]

Well, we know Reddit is a high quality source of information, so I know.

[A]

There'S a lot of humor in that, but we have a good relationship with the cybersecurity subreddit. Yes, there is a combination of garbage and good stuff. Go on. I'll let you say it. Go ahead.

[B]

So, I mean, the area of focus that I really hone in on when hiring is potential and desire to learn. So it's actually very relative to the conversation in my perspective.

[A]

Yeah, pretty. And I would Say pretty much every CISO we have. If you don't have that capability, what are you even doing in cybersecurity? Telling all this truth?

[B]

Yeah.

[A]

Yeah.

[B]

But I think it's actually really hard to distill and measure that. Right. When I think about some of the questions that I've been asked throughout my career about like oh, OSP and cloud and tcp, that's pretty easy to regurgitate. Right. That's pretty easy to go Google and read a textbook and be able to repeat that same phrase and those same kind of key items that show that. Yeah, I did some research on that. I think that's actually less relevant because I think most of those topics you can actually learn on the job from my perspective. So for example, if a candidate I was talking to today mentioned to me, oh, I was just researching and reading Brian Krebs blog on the latest DDoS attack, that goes a million miles longer for me than somebody that can repeat, oh, I know the explanation of a SQL injection from a textbook. So that's sort of my high level perspective is showing that they have ingenuity and aspirations and interest in like the day to day that really hones in on their sophistication around cyber rather than knowing some textbook definitions.

[A]

Yeah. So the active participation in just their own learning and their community seems, I mean that seems like a very clear way. I mean, I got to assume you agree with that. Eddie, is there anything else you would add?

[C]

Yeah, you know, I talk to my managers when they go through their hiring process. Don't always fall in love with the smartest resume. It's really easy to see, oh my goodness, this person has been with company A, B and C, oh my goodness, they have these coding capabilities and it's easy to see the resume that sticks out. What's. I think a better insight or perspective is the conversation and the dialogue in the interview around constructive feedback. And we always try to build in constructive feedback comments into the interview. Is somebody going to get uncomfortable hearing something? Whether the feedback is about something they just said, something that's on their resume, something like what Anthony just referred to. I want to see whether or not somebody is able to hear the room and understand if they can read the audience. And I think that's a, that's a trait where you can say, okay, this person is not trying to be the smartest person. They're actually trying to understand the problem. And I think if you can walk through that scenario, the hiring process, you start to weed out, you know, who's there to Earn a paycheck and make the big dollars versus who's trying to solve a problem. There's nothing wrong with either one. You're going to have both profiles on your team. But if you're overwhelmed on one side or the other, then it becomes a challenge. And so, you know, I always talk to my managers, look at these other skill sets, understand that it's these types of qualities that can really solve problems and it's not just the hefty and beefy resume.

[B]

And I want to double down on something that Eddie said and really focus around the problem solving aspect. One of the things I actually look for is that during an interview you should assume that the candidate, whatever solution they come up with during maybe a problem solving exercise, probably won't fit in your environment. Right? Because they don't know your environment. And that's okay. But the thing I actually look for when they do the problem solving is that did they create any safety nets in their solution? Right. Did they figure out a way that they can assume that this might fail and allow them to fail fast, to be able to reverse and pivot and learn? And I think that's so much more important in going kind of that second and third effect around creative problem solving to say, like, hey, I'm going to propose a solution. It might not work. It might work, but if it doesn't work, I have another plan B and another path to go take. And that shows, like I think, a level of sophistication higher than other candidates.

[A]

Well, before I go any further, I do want to tell you about vanta. They've been a phenomenal sponsor and if you are not clued in, Savanta, I'm going to do it for you right now. Compliance regulations, third party risk and customer security demands are all growing and changing fast. Is your manual's GRC program actually slowing you down? Probably is. So if you're thinking there must be something more efficient than spreadsheets, screenshots and all manual processes, you are right. GRC can be so much easier while strengthening your security posture and actually driving revenue for your business. Vanta's trust management platform automates key areas of your GRC program, including compliance, internal and third party risk and customer trust, and streamlines the way you gather and manage information. And the impact is real. A recent IDC analysis found that compliance teams using Vanta are, get ready for this 129% more productive so you get more time and energy to focus on strengthening your security posture and scaling your business. You don't want to spend all your time in grc do you so Vanta GRC how much easier trust can be. So go to their website vanta.comciso do me a favor. Do the V A N-T A.comciso so they know we sent you there. Go to vanta.comciso to learn more. It's time to play what's Worse. Eddie, you've played this game before. Anthony, you have not. But are you aware of this game? Yes.

[B]

Yes.

[A]

Okay, so we give you two crappies scenarios. And what's interesting is these have like a pro and a con to both of them. But you have to determine which sort of combination of pro and con is better. If you will. I will make Eddie answer first and then you will answer second and agree or disagree with Eddie. And this comes from Inet Siegel of Iprove. And here are the two scenarios. Number one, you are a highly competent and effective security professional and have an incredible plan to improve your company's security posture. Sounds damn good, doesn't it? But no one in the business trusts you or cares to know who you are. Okay. By the way, I'm sure many listeners feel this. All right. I think there's some pangs of ooh, I think that's me. Okay, so there may have a lot of reality to this. That's scenario number one. Now, next scenario. You are a very popular security professional and have forged incredible relationship across the business. But you have little to no technical expertise and you struggle to deliver anything that will improve your company's security posture. So they love you. They want to keep you, but you're an ineffective buffoon. Pretty much. What's. What is worse?

[C]

All right. While tempting the buffoon path, it is. I'm going to go. What is worse is going to be option number one. I think having a great plan that just does not land is really unfortunate. And I know like you said, half the audience may be like, oh my goodness, are they talking about my company? Is that my team? Is that. Is that my last three roles? I think the reality is that is a very difficult situation to be in. Understanding that you have really a good way to achieve success within information security and it's just not resonating. And that is hard.

[A]

Yeah. And you're miserable every single day and that's it. But the second scenario, you're not doing much for the company at all. Whatever. So it's interesting. There's two of these, like, what's worse for you as an individual versus what's worse for the company overall?

[C]

Okay, so that's a different lens, right?

[A]

Yeah, yeah, yeah. So, I mean, there's two different lenses here. But when you get right down to it, neither scenario is the company getting a good security program. Because in the first scenario you pretty much don't get to implement it because no one trusts you. Or maybe you implement it and it's not supported, which is kind of the equivalent of not being implemented. And the second scenario, it's just, you know, you've slapped duct tape to the business. So I think the business is kind of failing in both scenarios.

[C]

It is, which is why. It's a good question. It's a good question of what's worse. But if you think about that second option. Right. The second option is, you know, you're cordial, you're loved.

[A]

I mean, day to day, you're having a great time.

[C]

But I think when you look at that one, the reason I didn't pick that one is most likely you've hired some very talented people. And so even though you may have some incompetence aligned to you, well, you're.

[A]

Adding a lot to this. That may not be the case.

[C]

Yeah, I'm adding this, I'm adding to this scenario. My assumption here is team no, but you're.

[A]

No, I think, I think it's good because the second scenario, you're happy as a clam because everyone loves you even though you're not doing much of anything. All right, Anthony, he thinks the first scenario is worse. What do you think?

[B]

I think so as well, because you have the self awareness to understand the situation that you're in. Right. In the latter, ignorance is bliss in this regard. Right. Like you can just live in la la land doing nothing and everything's hunky dory and everyone loves you and everyone.

[A]

Adores you until the day, you know, the S hits the fan and then you're in problem.

[B]

Right. In both scenarios that will occur. Right. And then.

[A]

Yeah, because you had a crappy scenario either way.

[B]

Yeah.

[A]

Which, which, both. They'll hold it. Now, now wait, you bring up a very good point. Both scenarios happen, but in the first scenario you're competent and so when it does hit the fan, you can handle it. In the second scenario, you cannot. So.

[B]

Well, I guess it really just depends on that Runway that you have until it occurs. Right.

[A]

Well, it's gonna. It sounds like it's gonna occur one way or the other.

[C]

Well, it's inevitable to occur in both paths.

[A]

It's inevitable. So do you. It's going to hit you. So one day you're going to be able to deal with it. One day, you won't be able to deal with it. So the thing is, it's good until it happens, and until it happens, it's not good. So does anyone want to change their state, their stance here, or are we sticking to it?

[B]

I'm holding firm because you're miserable much longer in scenario one than you are in scenario two. Because in scenario two, you're probably gone. Right. Eventually, at least. Scenario one, when the. When everything does, you know, hit the fan, they're gonna be like, oh, yeah, I'm glad we have this guy.

[C]

Yeah. I think I will stick firm as well. In scenario two, if they really love me that much and something really did hit the fan, I'm gonna get funding to bring in a third party, and. Great. We're gonna be. We're gonna be past this event in a couple of days. In scenario one, I think. Yeah. You're. You're still. I've been telling you all this for a long time. Yeah.

[A]

And they still don't want to listen.

[B]

To I told you so.

[A]

Okay. You don't need to name any names, but. Cause one of the things that a lot of security professionals used to say, and thank God I don't hear it now, but is the classic I told you so. And thankfully, I don't hear it like I used to hear it, but have you had any situations where there was an I told you so moment, but it wasn't said, and yet there. The company did not wake up to it? And it could have happened to you. It could have happened to a colleague. I mean, have you had a situation like that where even when it hit the fan, the company still didn't get it?

[C]

You know, well, we're going to protect Anthony here. Right. I'll say. If you look at every breach that's been out there, there's always artifacts where this was being discussed. Right. Every. Every breach out there has some remnants of. There is dialogue. Go back to the target breach. There is the red flashing light on the. On the sock wall. There's always something that says something, someone new. There was insights and perspectives, and unfortunately, it didn't land, which is why I still think that first option is the worst, because the fact that upper management is just not really grasping onto the concept. I've seen leaders not pick jobs because upper management just really doesn't understand truly what information security is. And so it's hard.

[A]

Okay. I've heard many CISOs say they won't take a job unless you know they get it because they want the support. Understandably.

[C]

Exactly. Exactly.

[B]

All right, yeah, I similar sentiments. I mean, when we look at some of these major breaches in the news, someone knew something was going on and just couldn't get by and to fix something. Right. The reality of it is, is that you always have to create this balance and trade off of like, can we lock down everything and get all the money and resources and maybe create a poor user experience or accept some risk. Right. And unfortunately, from a business perspective, accepting the risk is most likely on the table.

[A]

What works, what's not working? AppSec is critical. So why does AppSec training seem like a dubious proposition? Now, a post on the cybersecurity subreddit asks if it's completely useless and only there for compliance purposes. Now in the comments, several developers said the training is there just to check a box or is too basic to be useful. The best training is threat modeling and working directly with dev teams and architects, said one Redditor. Now, even those with positive experiences with relevant training admitted it's often seen as a sunk cost by the business. And even with a good platform, you're not turning every dev into a security champion. All right, I'm going to start with you, Eddie. Is AppSec training useless? And if so, why? And I know I'm painting a broad brush, I know some do it well, some don't. But there's this argument on Reddit, so does it have to be this way? What could make it better? Or what should AppSec training be? Maybe just simplify it to that.

[C]

And I think I'll focus on that last part of the comment because I, I feel if you're engaged in this conversation, the question should be what are you training on? If there is a debate of value, then the question really is what is the content? Because you can train on a multitude of things, on the delivery process, on coding practices, on fast fails. You can even train on here's how we test you. Here's the issues of our last audit finding you can make it energizing. Okay, here's where we saw time to market collapse by X percentage and these are the coding practices that they use during that timeframe. So I think it really does matter what your training the developers and it can't just be cookie cutter stuff. If you're just pulling stuff off of NIST and OWASP and saying these are the things that we want to talk about, it can be dry. But if you're making it relevant to your organization, if you're talking about Business success. If you're talking about things that are in their annual review cycles, hey, this is things that they've committed to from a goal perspective. That's when you start to talk about, okay, this is energizing, it's engaging and most likely bidirectional. And when I mean bidirectional, they're asking questions in the training. Okay, I see this. Okay, that's, there's value there. So, yeah, I think if, if you find yourself in that conversation, you should really look at the content that's being presented as opposed to should I be presenting? Yes or no.

[A]

All right, I like that. Take your take, Anthony, on AppSec training specifically.

[B]

Absolutely. So I mean, I definitely do agree that threat modeling and awareness are an effective way to get developers buy in and have them understand the why. But I actually would kind of challenge it and say, do you get a good return on investment by having your entire engineering and development force spend what, maybe an hour a quarter doing AppSec training? If you're like really ambitious, probably more of an annual type of thing because that's opportunity cost. Right? That's time that developers are not improving product, not fixing bugs, not building on new go to market features. Right. So now you have to find this balance. And I think what I hope to see, kind of going to the later point of the question is everyone's Vibe coding. When we went out to SF, I looked on the plane and there was like 12 laptops all just like vibe coding with cursor. And that's fantastic and it should be embraced. But where is the vibe security code coach in this kind of stratosphere? Why isn't there this instant feedback in the IDE as these developers make mistakes, Learn how to get feedback immediately, immediately and get reinforced on secure coding practices so it doesn't take seven days later for someone to get an alert to triage to go reach out and talk about a security bug. Like we can shift even further left on security awareness in terms of development training right into as developers are writing every line of code.

[C]

Anthony brings up a good point. There are certain things developers really like to do and you can see when they're energized. Inversely, you can see when they're not. Right? And maybe even some of that training could be, let's see what you. Maybe you don't like peer reviews. Maybe that's your Achilles heel. You just hate doing peer reviews. Well, maybe your training is how do we shorten your peer review cycles? How do we get you to the point where you can get through that peer review process. And so not always just amplifying the fun stuff, but how do you start to eliminate some of the nuances of the role that you just feel, you know, you could do without? So yeah, look at what's passionate for those teams and you can see very easily, like Anthony said on the plane, you can pick up really quickly who are true coders, right? They, you know, they, they, they get mad when they hear that bell ring and it's like, oh, put your laptops away. Reality is, you know, you can find passion and then, you know, either train on to allow them to focus more time on their passion or you start to eliminate things that they just really prefer not to do.

[B]

Right? And me working in a B2C business, we always talk about going to where our customers are. We need to follow that same philosophy. When we think about us as security practitioners, who's our customers and developers are those people?

[A]

How is AI going to solve this problem? Quote don't use AI to solve your AI problem Google DeepMind debuted a new Camel tool that is empowered by machine learning. It's a simple debugger that lets humans see what's actually being passed between LLMs and APIs. And that's the point, according to a blog post by Simon Willison, the creator of Dataset, an open source tool for publishing and exploring data. Now, instead of adding more models to oversee misbehaving agents or filter bad outputs, what we need is less AI and more visibility. As Willison said back in 2022, prompt injection is fundamentally unsolvable. I get the sense that getting one AI to look at another AI is simply stacking LLMs on top of each other. So does this method work or is it just layering on more complexity instead of inspecting what we've already built? And we've heard this a lot like run something through one AI tool like ChatGPT and then have copilot look at it or Claude look at it or whatever the heck it is. Just have them sort of. They're all kind of built on different LLMs and what will one say to another? So are we creating an AI on AI arms race when what we really need is basic engineering discipline, logging boundaries and human readable insights?

[B]

Anthony so there's a lot to unpack there. I disagree with the initial quote. Okay, we should be using AI to solve our AI problems 100%.

[A]

Okay.

[B]

But I would say the different approach that I would take is that it shouldn't just be these broad, large language models to do so. It should be Very specific large language models for the use case. Right. Like, I think we've all seen the news that Gemini is going to be rolling out a security specific large language model. Right. So now can the application of that be applied to something that's generated by maybe a broader one? Could there be large language models that are maybe SOC trained specific or AppSec trained specific or Red Team trained? Right. I think that's where the differentiators are. I do agree with the observability problem. That's probably one of the biggest challenges, to be able to see the prompts and responses and what's going on behind the scenes. That's definitely very needed from a visibility logging architecture perspective. But then how you going back to that kind of QA aspect and be able to leverage multiple AIs? They need to be intentional large language models, not these broad large language models that are prompt. So they need to be very contextualized and purpose specific.

[A]

I think you're on the same wave length here, Eddie. Yes.

[C]

It sounds like the movie Inception, right? Where you stack on stack on stack. The part at the end of that movie that I think is the most energizing is the last 30 seconds when you're watching that top spin and you're looking and you're saying, okay, any second now this top is going to stop. And it sparks a debate, it sparks a dialogue. Right. It sparks conversation. Did it stop? Did it not stop? Are you still in the dream? Are you not in the dream? And I agree with Anthony. Right. I think AI is a part of the ecosystem. Whether we realize it or not, it's going to be there for the foreseeable future and it's really understanding how to make the most out of it, as opposed to just putting it back on the shelf and pretending like that the spinner stopped or pretending that the spinner keeps going. I think the reality is you don't really have to know what happened to the spinner other than the fact that there's a conversation that has to happen. And so, yeah, it sounds like an absolute Inception question, but I do agree AI is fine as far as how to bring it into your ecosystem, into your technology stack, so long as you know what its role is. And kind of like the questions we started off with, is it making determinations? Is it doing quality checking, is it helping me find things that I'm not even looking for, I don't have the time or bandwidth to do, or is it just repetitive and is it just repeating things that are already going on in the organization and just cutting some steps out of a elongated process. And so I think it has a role AI on AI and if you know how to do it correctly, you're going to get some benefit out of it. I think what most people don't realize is large language models are huge catalogs. They're huge number catalogs. And so if you're nervous about a huge catalog and you're okay with data warehouses, then you may not really understand where the risks lie. And so I think AI and AI is perfectly fine.

[A]

All right, so go to it. AI on AI. Both of you believe the AI versus AI arms race? Well, there's a certain level of respectability to it to it and value for that matter. I want to thank my guest Anthony Kandeas, the CSover Weight Watchers for joining us. Thank you Anthony for being here. And also to Eddie Contreras, the CISO over Frostbank, for stepping in as the co host for this very episode of which you will hear a lot more of him over on Defense In Depth. And a huge thanks to our sponsor. That would be Vanta. Remember, go to their website vanta.com CISO remember to add that CISO so they know we sent you there. Remember Automate Compliance, Manage Risk, Improve Trust Continuously with Vanta. Huge thanks to our audience. We greatly appreciate your contributions. And for listening to the CISO Series podcast that wraps up another episode. If you haven't subscribed to the podcast, please do. We have lots more shows on our website cisoseries.com Please join us on Fridays for our live shows, Super Cyber Friday, our virtual meetup and cybersecurity Headlines Week in Review. This show thrives on your input. Go to the participate menu on our site for plenty of ways to get involved, including recording a question or a comment for the show. If you're interested in sponsoring the podcast, contact David Spark [e-mail] thank you for listening to the CISO Series podcast.