Omer Akgul (14:04)

So we were initially interested in understanding if we can put any sort of bounds on the the truthfulness of LLMs, right? They they say stuff, but it's pretty hard to fact check what they're saying. They're pretty confident in what they're saying. And so they lie all the time. They call these things hallucinations. So that was our that was our initial curiosity, right? How do you go about this? And then we ran into a line of work called consistency. Some people call this accuracy prediction. Some people call this accuracy calibration. They call it confidence. The terms are somewhat conflated, but the general idea is if you were to try to get the model to tell you how confident it is. Is it going to give you the same answer every time? How much is that going to happen? Right. And turns out that might be a pretty decent predictor of when it's making stuff up.

Interviewer / Host (15:13)

Well, you use the word consistency here. What does that mean in the context of large language models?

Omer Akgul (15:19)

Right. So the very simplest example or the definition I'd give is how likely is the model to produce the same output given a prompt? So say I ask it, tell me what two plus two is. How likely is it to say four each time? Or is it going to say something different every once in a While? So say 60% of the time it says 4, and 40% of the time it says 1, 3, and 5. So that's the simplest way of putting it, I think.

Interviewer / Host (15:55)

Well, let's go through the research then. I mean, how did you go at this? How did you compare human judgments to automated consistency metrics?

Omer Akgul (16:05)

Right. So we were looking at the state of the art, right? What have other people have done to try to measure consistency? Because turns out it's not actually super, super simple, as the way I put it, to measure the model's consistency. What we notice in a lot of prior work is that they find these automated ways of trying to understand what consistency looks like. So they would, for instance, have the model respond to the same question multiple times, and then they have these algorithms to automatically compare these answers to one another and come up with an answer. Right. They would say, 60% of the time it's consistent to this prompt. But those automated ways, turns out, aren't super ideal because they aren't the same way that humans would compare answers and say these answers are consistent. So I would maybe say writing down four, spelling it out, and the numeric for single character is the same thing. But these automated ways might not necessarily say that all the time. So we noticed that flaw in prior work and thought that was worth investigating more.

Interviewer / Host (17:27)

And what were your core findings here? What did you discover?

Omer Akgul (17:31)

What we find is our initial intuition was right, that there is somewhat of a discrepancy between all these automated metrics. And if you were to ask human intuition, if you were to ask humans directly, this has consequences. This means that the consistency metrics out there that people are using already have flaws. They're not perfect. And we identify in how they're not perfect. Right. But we also have some mitigations to this. We do find that if you combine a couple of these different methods and you calibrate it with human intuition. So there's like a little bit of a training loop going on there. You can get pretty close to human numbers and you can be pretty efficient at that.

Interviewer / Host (18:24)

So is this, as you say, is this a matter of training your LLM and giving it positive reinforcement when it gives you what's perceived as a correct answer?

Omer Akgul (18:35)

There is some work that does that. So that is a method that could be explored. But the way we do it is we basically have an auxiliary model where it looks at some of the. Basically, before an LLM is going to give you an answer, it produces these internal states and these probabilities of what I should say. Essentially our method looks at those and it's calibrated by human intuition to say, based on what I'm seeing, this model is outputting. A human would have said, this is this much consistent. That's the general idea. We look at that and this auxiliary model gives you a number. It says, this is how consistent this model is with this answer. That's how it works.

Interviewer / Host (19:34)

Just so I'm clear here, the part about human intuition is also modeled. That's a separate model.

Omer Akgul (19:45)

So we do collect a bunch of data from humans.

Interviewer / Host (19:48)

Okay, that's the part I wasn't clear about. To what degree are humans actually in the loop here?

Omer Akgul (19:55)

Right. So we did collect a bunch of data from humans, and that's how we get our first results. Right. There's a discrepancy between these automated metrics and what humans would have said. But then we're like, well, can we make these automated things better? Can we make it more aligned with humans? And that's where that auxiliary model I was just talking about comes in. So, yes, the human intuition is somewhat modeled in one of our solutions. And the reason we need a solution to ever begin with is you can't have a consortium of humans rate the answers of models Every time they produce an answer that's very impractical. So we try to distill it down a little bit. It's not perfect by any means. There's still a discrepancy, but it's better than what it was.

Interviewer / Host (20:41)

Yeah. So from your research here, what sort of steps do you recommend organizations take when they're deploying LLMs in their own critical workflows?

Omer Akgul (20:52)

Right. So this is the tricky part, and we did try to come up with guidelines, especially in the blog post that I wrote later on. But what it boils down to is actually pretty similar to how the machine learning or the model deployment lifecycle works in other contexts. What you generally do is you train and develop your model, you put it out there, but what you train on and what you deploy on aren't matching one to one. Your model won't perform as well in the real world as it did when you were training it. There's a similar thing going on here with consistency. So say you developed or you borrowed your consisting metric from someone because consistency is pretty useful to understand if your model is doing well or not. Right. So in the case of LLMs, it can tell you if your model's lying or not with some confidence interval. So it's not a fail safe thing, but it's pretty useful. And what could happen is you pick your consistency metric and you think it's giving you pretty good data. All you're saying is your model is doing great, but in reality it's not doing all that great. So there needs to be some calibration going on to show that what your consistent metric is saying is what is actually going on in the real world. That's what we do with this paper. We calibrate it on humans. I suspect depending on how critical your deployment is. Right. You might need to do something similar to really get use out of these consistency metrics. It's not like they're completely useless without this. But again, this depends on how much risk you're willing to take. Depending on that, you might want to do some of this calibration and that could mean collecting data from humans. Maybe depending on their scenario, these humans might need to be specialized in whatever domain. Say these are like logs of computer systems, right? Maybe you need people who understand systems, maybe more cybersecurity people that look at these things. But you might need to do some calibration on your consistency metrics for them to be really useful to you. I see.

Interviewer / Host (23:24)

Where do you suppose this is going to go in the future? I mean, do you envision that this, the, the, the things that you've learned here could be just integrated into everyday LLMs?

Omer Akgul (23:38)

I hope so. It, it's, it is tricky. Right? The, the human calibration part is it's not cheap and it might change based on domain. But I, I certainly hope that there is more attention paid to this as more and more consistency metrics come out. Because this isn't a solved problem. Right? There are new versions of these things coming out practically every other week. The pace of LLM papers being put out there is pretty fast. But I certainly hope more people will pay attention to the flaws that we've discovered and the solutions we've proposed. To make this stuff more robust for real world deployment.

Interviewer / Host (24:27)

That's Omer Akgul from RSA Conference.

Dave Bittner (24:42)

This episode is brought to you by Indeed. Stop waiting around for the perfect candidate. Instead, use Indeed Sponsored Jobs to find the right people with the right skills fast. It's a simple way to make sure your listing is the first candidate. C According to Indeed data, Sponsored Jobs have four times more applicants than non sponsored jobs. So go build your dream team today with Indeed. Get a $75 sponsored job credit at Indeed.com podcast terms and conditions apply.

Interviewer / Host (25:17)

And finally, two penetration testers walked into a courthouse to do their jobs. Eventually, Dallas County, Iowa agreed to pay them $600,000 for their trouble. Back in 2019, Gary DiMacurio and Justin Wynn, then working for Coal Fire Labs, were hired to test security at the Dallas County Courthouse. Under written authorization from the Iowa Judicial Branch, the rules explicitly allowed lock picking and other physical attacks. They found a door, popped a lock, tripped an alarm, and promptly showed deputies their authorization letter. So far, textbook red teaming. Then the sheriff arrived. Despite confirmation the work was approved, Chad Leonard had the pair arrested on felony burglary charges. They spent 20 hours in jail, posted $100,000 bail, and endured months of public accusations before all the charges were dropped. The fallout was career threatening, the message chilling. Even authorized hacking can end in handcuffs. After years of litigation, the county settled days before trial. D Mercurio now runs Kaiju Security, and the lesson stands. Sometimes testing defenses exposes a different vulnerability altogether. And that's the Cyberwire. For links to all of today's stories, check out our daily briefing@thecyberwire.com we'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity. If you like our show, please share a rating and review in your favorite podcast app. Please also feel fill out the survey in the show notes or send an email to cyberwire2k.com N2K's senior producer is Alice Carruth. Our Cyberwire producer is Liz Stokes. We're mixed by Trey Hester with original music by Elliot Peltzman. Our executive producer is Jennifer Ibin. Peter Kilby is our publisher and I'm Dave Bittner. Thanks for listening. We'll see you back here tomorrow. If you only attend one cybersecurity conference this year, make it RSAC 2026. It's happening March 23rd through the 26th in San Francisco, bringing together the global security community for four days of expert insights. Hands on learning and real innovation. I'll say this plainly. I never miss this conference. The ideas and conversations stay with me all year. Join thousands of practitioners and leaders tackling today's toughest challenges and shaping what comes next. Register today@rsaconference.com cyberwire26 I'll see you in San Francisco.