← Data Security Decoded
Data Security Decoded
AI-Driven Cyber Defense in Action: How AI Agents Are Saving SOC Analysts From Burnout
00:22:54
In this episode of Data Security Decoded, join Caleb Tolin as he sits down with Grant Oviatt, Head of Security Operations at Prophet Security, to explore the transformative impact of AI agents in SOC environments. From reducing false positives by 95% to dramatically improving incident response times, discover how AI is augmenting human analysts rather than replacing them. Whether you're a CISO looking to optimize your security operations or a SOC analyst concerned about AI's impact on your role, this episode offers practical insights into successfully implementing AI-driven security solutions while building trust in automated systems. • Learn how AI agents handle tedious security tasks, freeing analysts for strategic work • Explore real-world success stories of AI-powered threat detection and response • Understand the critical balance between AI automation and human expertise • Get practical steps for deploying AI agents in your SOC
Loading summary
Transcript23 lines
- [00:00]
A
AI agents never tired, never sleep, and can perform these actions. Our ChatGPT moment was effectively seeing that AI agents can successfully perform security tasks, and we've been starting to see the results of both identifying threats that people have been missing and removing all that tedious work that analysts hate to do and didn't get into Security 4 in the first place.
- [00:32]
B
Welcome to another episode of Data Security decoded by Rubrik zero Labs. I'm your host, Caleb Tolan, and if this is your first time joining us, welcome to the show. Be sure to hit that subscribe button so you're notified of new episodes. And if you're already a subscriber, thanks for coming back. We encourage you to give us a rating. Drop us a comment below letting us know what you think about the episode. Your feedback is invaluable to us now. In this episode I had the pleasure of sitting down with Grant Oviat, who is the head of Security operations at Profit Security. Now, Grant has a wealth of experience in security operations, having held senior roles at companies like Red Canary, Mandiant and Expel, but his current role at Profit Security focuses on leveraging AI to enhance security operations, automate threat detection, and improve response times. Really important topic of today. Grant's a really cool guy with a really unique perspective on the role of AI in cybersecurity and I hope you enjoyed this episode as much as I did. Let's dive into it. Grant, thank you so much for joining us for the podcast. I'm really excited for this episode and I'd really love to start with what you're currently doing in kind of your current role. So you're currently head of security operations at Profit Security, which is an AI driven SoC platform, and you're kind of acting as patient zero for what seems to be like the pinnacle example of AI driven cyber defense, which I think a lot of people have a lot of questions about. So, based on your experience so far about with AI for cyber defense and how it's shaping security, how is it impacting the organizations you're working with and how is it working with responding to cybersecurity threats?
- [01:57]
A
Great question. Background on US is really building AI agents to solve those tedious tasks that analysts have had to just deal with over and over. So the false positive alerts as security perimeter has expanded into the cloud, there's only more red blinking lights that people have to go and deal with and people just don't have time in the day to go and triage and manage that risk. And so AI agents never tired never sleep and can perform these actions. Our ChatGPT moment was effectively seeing that AI agents can successfully perform security tasks. And we've been starting to see the results of both identifying threats that people have been missing and removing all that tedious work that analysts hate to do and didn't get into Security 4 in the first place.
- [02:40]
B
Right, that's very interesting. And so I think I could imagine some concerns of people who would be deploy a solution like this might be that there could be some of those false positives that the AI agents themselves are uncovering. Has that been something you've run into an issue with or like, how's that been addressed, making sure that you're not just getting this massive influx of false security alerts or just a massive amount of alerts period from these AI agents. Does that happen?
- [03:06]
A
Yeah, it's a great question. I think it's something that would be the opposite of the antidote here of like, wow, we've got even more false positive activity that's coming from these AI agents. What do we do? I think our focus has really been around doing evidence backed investigations. And so there's clear lines of transparency and explainability behind the evidence sources that we're producing and showing to customers. And also we have a really high degree of efficacy in the investigations that we are performing. And so everything for us is audible and explainable and so customers can build that trust and see the investigations that have been performed. The net impact so far has been a dramatic around 95% reduction in false positive activity that folks have seen.
- [03:46]
B
Wow. Wow, that's really cool. And I think that concept of building trust with AI agents is something we'll touch on a little bit later in the conversation because I think there's a lot to unpack there. But kind of going back to what we were just chatting about, this is all AI for cyber defense. And so many of the headlines are talking about how attackers and threat actors are using AI to fuel their own social engineering attacks and all this kind of stuff. So are there any really cool stories that you have to share from like the battlefield of using AI for cyber defense within a stock?
- [04:17]
A
Yeah, a few different ones. I mean all the headlines are really flashy for these big breaches, but I think it's also important to spotlight like the real security issues that most companies experience every day and some of the things that we've observed there. And so business email compromise is obviously something that has just only continued to grow for customers from the insurance space. For folks that have cyber insurance policies bec is the biggest category of cyber payout, which effectively means the most incidents are happening from that category. We've been able to stop and identify BEC activity that's occurring from initial lock on events that are weird, those are super voluminous, 99% of the time they're false positive. But when they're bad, it's potential access to your environment and data loss and financial loss. And so we've been able to take investigations that historically have taken people hours or days and compress that to five, 10 minutes. And we've seen that with several of our customers now looking at BEC activity where a user has gone and entered their credentials in a fake sign on page. We saw them log in later and start to perform actions and before they were able to make real progress, able to identify, capture the remediation activity and go ahead and contain the threat. I think the other side is important too, where there's a bunch of activity that looks like high risk activity that you can spend a whole day looking at when it's really nothing. And so we've had customers, customers that get ransomware alerts. Funny story, we had a POB going on. Now a customer, and this is one of the big things that closed our relationship and started our partnership together is they had a ransomware incident that fired on their edr. They realized it wasn't testing, they realized that they didn't know what was going on and started digging into it. They told us they spent a full day doing this investigation manually. They forgot they'd started a relationship with us the Friday before and they're like, oh, we should go check out profit in eight minutes. They said we did a better investigation than they did and would have saved them the rest of their day not running something down that wasn't actually ransomware. So I think those are the real things that people experience, that exhausting work, that it's a dead end, it's not a real thing, but you of course want to react and respond appropriately and then prevalent cyber threats that we're starting to see getting knocked down and reducing real risk for organizations.
- [06:27]
B
Right, right. That's crazy. That's a really cool story and it leads really well into the next question I had for you. So I know anyone listening to this may kind of be a little bit concerned of like, is AI going to replace my job? That's something that everybody's saying from not just in IT and security, it's across a lot of different professional domains. But is that something that you've run into in conversations with any clients. And what's your reaction to that? Will these AI agents be replacing actual humans on keyboards with hands on keyboards?
- [06:57]
A
Yeah, Caleb, I get this question probably once a week at least from analysts and folks, and I think it's a legitimate concern. I know there have been other companies like you mentioned that have really make it made a marketing splash on this sort of platform of like, hey, use fake human to go and do tasks. Our focus has really been around leveling up folks in security and letting them do the, the things that they're best at and the reason they got into security in the first place. Like any analyst I talk to, I follow up and say, like, did you get into security to look at 99% false positive alerts and run things down that weren't important or send slack messages to team members to check if they're actually supposed to be traveling to Mexico today? Probably not. I think you wanted to go and build something that was really meaningful for your organization and respond to real threats, address risk. You just haven't had time to do it because all these other things are wearing you down. And so our goal is to take all the boring, ugly, but important responsibilities away from security operations so they can effectively level up and do the things they got into the industry for. And I feel like that really resonates of like job satisfaction increase when it's such a high turnover position because it's just so exhausting of, of seeing the light at the end of the tunnel and being able to do the 5 to 10% of the job that they really enjoy to do but don't have time to do. And that has huge business impact, by the way. And so that's more of where we, we come in and sort of connect with analysts.
- [08:24]
B
That's really cool. And so obviously AI in the use case of a soc is for cyber defense is this new skill that people are going to have to develop. And so I realize that people who are already in the industry may say that this could just be a new skill that they have to develop for the rest of their career. Has there been any discussion that you've been a part of in terms of how this impacts people entering into the cybersecurity workforce? Like how? Because I can imagine that this is replacing a lot of the more like junior level SoC analyst jobs. Is that something that is a conversation topic that you've explored before?
- [08:58]
A
It hasn't been one that we've been exploring, but I definitely have opinions on it and I think it's actually going to lower the barriers to get into cybersecurity, truthfully. So when I think about being an analyst or a SOC analyst, it's all about asking really good questions. You're a detective, but basically with security data and logs, right. And so whether I've seen some of the best security analysts I've ever worked with come from fields in biology research or come from fields in finance. And the big barrier isn't the way that their brain thinks, it's understanding the evidence and how computers represent information to tell a story. And so I think you can translate those skills of people being naturally curious or analytical and just start to ask questions in plain language and get answers back to make decisions. And so the huge barrier that I see in folks that are self study or getting into cybersecurity is really less around thought process. And thought process. There's just sort of traits that drive that curiosity, but more around like I don't know where to get this data to answer this question or I don't know what that looks like. And so if we can remove that technical barrier and folks can just ask like, hey, is this normal activity for the user or, or what should I be expecting in this scenario? And get reasonable answers that are evidence backed, I think you start to build up that technical capability really quickly and you take someone who hasn't been in the field and allow them to operate more successfully. Maybe as a direct example there, one of our customers brought in some new college grad interns to start and start working with us. We basically were onboarded the same time they they were or just a few weeks after. The feedback from their manager has been that they're operating more like a senior analyst today than someone that's entry level because they understand the investigative thought process better and are able to ask questions questions without getting bogged down with their SIM or other security tools. And so we're starting to see trappings of what you're describing. But my hypothesis is it's actually going to reduce the barrier for folks that are inclined to move into these types of roles to actually be successful faster.
- [10:57]
B
That's really cool. That's really cool. And a really optimistic outlook. So I dig it. I hope it goes that direction too. Cool. Well, for anybody who's been listening to the podcast for a while, you may recall that we used to do hot takes every once in a while. And so I wanted to shake it up and throw those in, you know, every once in a while here. So no better time than the present to do that. So we're talking all about AI agents here. And so, Grant, my question for you is, if you had to choose one famous agent from a movie or TV show that you would turn into an AI agent to back you up in an incident, who would it be? Would it be James Bond agent Cody Banks, one of the members of the Cortez family from Spy Kids? I'll follow on the knife first and take the first dab at it, but I'm going to go with Juni Cortez from Spy Kids. He had this really cool watch that had all these really awesome holographs around it that I think was in, like, the Spy Kids 2 movie. When I was a kid, I thought it was, like, the coolest thing since sliced bread. And so I've always wanted one of those watches. And I think working with him, I would get one step closer to getting one of those watches. So it's probably not the best answer, but that is my answer, and I'm dying to know what yours is.
- [12:03]
A
That's a solid answer. It's hard to be Juni. He's now married to Meghan Trainor, which is a plus. And then also has the robot frog, which is cool. So there's certain perks. I totally understand. I'm trying to go, like, thinking out loud. The AI agent path is sort of never tires, never sleeps, deals with the work that you don't want to do. Maybe it's just the, like, TV commercials I've been watching recently, but new Mission Impossible probably resonates most. I forget what Tom Cruise's character is, but does all of his own stunts. Jumping in, kind of doing crazy things to support the mission so that other people don't have to. I don't know that we all want Tom Cruise to be our security agent, but I think there's a good corollary between how he operates in those films and what we do for our customers.
- [12:50]
B
Good answer. Good answer. Mad respect for it. So I love it. I love it. So, back to the meat of the conversation. You were kind of just talking about some of the things that AI agents do really well, and I'd love to kind of explore that topic a little bit more. What, in your experience so far, have you seen AI agents are really, really successful at? Are there things that you have seen that they're really not that successful at that really still require that human magic touch? And I'm interested in your perspective, both from a preparation and the response and recovery perspective as well.
- [13:20]
A
Cool. There are certain things that. I mean, I think the evolution of this space is Interesting. Where was sort of all the rage where you can build your own custom playbooks of if this then that and perform actions. The hard part is it's super static and in line and if something deviates just slightly, then you've got to spend a bunch of engineering resources to go and fix those problems. I think where we've been seeing AI agents be really successful is in those reasoning gray areas of things don't line up exactly as you might expect. But still being able to make a really informed security decision based on whether this looks like legitimate activity or not and pulling that full story together or without having to write thousands of lines of Python code dog meant the capability. So we've seen a ton of success in that front. I think the thing that requires the human touch is still around context today. I would say AI agents jump off the starting block having a really strong base understanding of how to perform these investigations given the data that they're provided. But everyone is a bit of a snowflake. Like every customer has their own crown jewels, everyone has their own different naming conventions and things like that. And so we're really big personally from a profit security perspective of raising our hand and saying we don't know the answer to a question we mentioned before, the space is all around trust and so you don't know. I think it's way better to say like we need your help versus like making a call either way. But prodding the user for some additional bits of context like hey, is this allowed in your organization or is this a policy violation or is this this cool? Has been an element that's of customization that continues to be really important for customers and one that's hard to just native derive out of the box truthfully. So from like a preparation perspective, I think that fits into like having context to go in and make decisions accurately, just like a person on the team would. That's a place where humans really support. Well on the response and recovery side, we've been seeing some great benefits in coming up with remediation actions. I think the analyst experience is very emotional that folks may not realize you feel the weight of the organization on you. As a SOC analyst, you find something bad and there's that panic moment of like what do I do? And you've seen this activity before or not, but you need to start getting things moving to reduce the response time. And so we'll dynamically come up with remediation actions or response actions for customers in those situations. So it's less of a moving towards like a Brains off moment or it allows you to drill in and see like, hey, these are the things that look like they're most important. I can go and perform these actions and contain a host, reset passwords, things like that to stop the bleeding without having to jump around to four different screens and give you a reasonable first start on where to go.
- [15:58]
B
Cool. So that's really fascinating. So what about hallucinations? Is that something that is a issue that you run into with the AI SoC agents? I mean we see this with every AI tool under the sun. They have these crazy interpretations of data and their input and create some kind of out of the box output. So what does that look like so far? Has that been something you've run into and how are you addressing it?
- [16:20]
A
It's one of the biggest issues that we try to solve for when building initial product because accuracy is everything with security and trust is far easier to lose than to gain. And so when you're doing a job like this or trying to take this work off of someone's plate, you need to replace the steps or repeat the steps that they would do or feel like you're doing as good of a job or better. And also show your work. It's kind of like math class. Like showing your work and your answers to how you got the answers to the problem is more important than the answer itself. And so we spent a lot of time making sure that all of our agents are using evidence backed answers. So effectively they have line of sight to the raw data that composes the activity, the decision that they make on it. Like it may not be the same interpretation of that information that a human may make for better or worse, but they are always constrained to the raw evidence. And so when I think of hallucination, it's just making something out of thin air. We've gone to great lengths to ensure that there's no magical answers. Everything is tied to the data sources that a human would be using in order to make a conclusion. So that's how we've solved it. But it's totally a risk area when you think about AI agents being used in security operations or, or anywhere else for that matter.
- [17:42]
B
Right, right. So you talked about how the building trust into the system early was really important to make sure you're addressing things like hallucinations. Are there any other steps you took to build trust within these AI systems? Because that's something even admittedly myself, when I'm going through using AI tools in my day to day work, you're putting a lot of trust into These systems to make your life easier, produce something that's high quality that you can fully say, like, hey, it saves me time. So what mechanisms have you put in place to build trust in other ways outside of just addressing hallucinations?
- [18:14]
A
Yeah, totally. I think it's important from the customer journey perspective too because everyone's skeptical and appropriately so of AI, I think AI demos really well. In initial conversations it's obviously very buzzy, but during our customer journey, we're pretty pushy around, like, hey, go try this in your environment. Like don't take my word for it, like, go see this in action. Let's do comparison to what you're doing today to what you see here and like build some real evidence that this is going to solve your problem. And so I think that sort of approach is very encouraging and at least shows our trust in what we've built and sort of asking people to take it through the paces. Additionally, we're really big, I mentioned before, but showing our work, it's for hallucinations, but it's also for the skeptics and building trust. And so what queries that we use in order to gather evidence from your environment? What was the data transformation or what actions do we take in order to make this decision? What was the raw data we used in the first place? And so the idea is you can take this investigation and if you wanted to manually repeat the steps with the information we have, you could go and do it yourself. And so we believe and hope that it's the most useful feature that people never end up using ultimately. Right. So you get that initial trust and feel. So it resonates with the way that you think about investigations and handles security operations. And then you can move to just our decision making process and feel more confident in that. So we spend a lot of energy in making sure that we build trust early, both in like the customer journey, but also in the product. Even building things that are probably overly transparent just so that people feel like they can start working with this easier and believe what we're doing.
- [19:53]
B
Right, that's great. So I only have really one last question for you and it's for our listeners who are tuning into this. I'd love to know what they could start doing to deploy an AI agent in their SoC. What are the steps that you would recommend to get started with something like this? Are there anything that they should be looking out for? Any like red flags, any absolute necessities that they need in order to succeed?
- [20:15]
A
It's a great question. I think there's A few things to be thinking about there one is doing this yourself is really challenging. I'll say from personal experience and kind of going about this journey for the past 18 months with some of the more talented engineers who've been had the opportunity to work with, it's a really daunting task to take on. I think there's some really easy ways to start bringing AI or gen AI into your workflow related to summarization and kind of formatting investigations to a way that makes sense but making decisions. I would encourage going from a more specialized product where folks have invested in that space to make sure that things are accurate and transparent. So I would say from like reporting and response perspective, tons of benefits. If you're going to send an alert to ChatGPT and ask it to make the right decision, probably just about as good as a coin flip than getting the right answer back. And so yeah, be cognizant of that. And I think when you're looking at AI agents really of, of anything that you look at, but certainly in security, just be aware of kind of your testing criteria of what you're looking for. I think most folks bar for AI is higher than humans. It needs to be better than what that output is in order for it to be worthwhile or take the risk of having some other outside system be involved. So be looking at how AI systems reduce your risk in terms of response time. Ask how folks measure accuracy, like how do you know that there aren't false negatives and things that you're doing in my environment, how do I know that there are they're not unsung threats that you are avoiding or that AI missed? Ask those hard questions around like how is a product that I'm operating with ensuring that I'm getting the right results both in terms of speed and accuracy and quality control. And then I really encourage testing things out for yourself and engaging vendors and getting this in your environment. And the proof of the pudding is in the eating, as we like to say. So always give things for a spin and make sure it works for you.
- [22:14]
B
Right. Great advice. And something you said there that really resonated with me was that people have a higher expectation for AI than they do for themselves. And I think that's really true. And so a lot of interesting insights you've shared with us. This sounds like a really cool platform that you're building and makes me feel really strong about the direction we're heading with AI driven cyber defense. So Grant, thank you so much for the conversation. It's been wonderful speaking with you. Thank you to our listeners for tuning in and really appreciate your time. So till next time.
- [22:42]
A
Yeah, thanks for having me.