David Moulton (4:44)

And Patrick, you've worked in security consulting, engineering and operations across banks, inventors, and now here at Palo Alto Networks, you've seen the socks evolve firsthand. What's changed the most in how teams manage their workload and their tools?

Patrick Bile (5:03)

I think what's changed the proliferation of tools has exploded. And looking at legacy socks grew organically. And they would. They would bolt on additional tools and they would bolt on things like, we need threaten t management now because we've got so many indicators of compromise. Oh, we've got a massive deluge of stuff that we need to automate. Let's bolt on saw technologies like, like Liz said, how shi came into the market at that kind of opportune time. So we've kind of seen that explosion from a singular point of view to multiple screens, multiple chair. You know, the swivel chair analogy, that people in socks are looking at too many things and they are, as a result, being unproductive as far as we're concerned.

Liz Pender (5:49)

Yeah. You've got to remember so on Paddy first started in the sock like the wheel wasn't even invented. So it was a long time ago.

David Moulton (5:58)

I also was there 3,000 years ago.

Liz Pender (6:02)

Wow.

Patrick Bile (6:04)

I'll explain to you what a Z series is, Liz, and how you can manage security risks related to that for a large financial organization.

Liz Pender (6:12)

Thank goodness. I'm looking forward to it.

David Moulton (6:16)

So when we were talking about putting this podcast Together we were talking about this idea of context switching and I ran across this HBR article that talked about workers switching apps something like a thousand plus times a day. It seems kind of wild, but then you start to observe your own patterns and you realize, yeah, you're moving back and forth in between desktop applications and web apps through your browser and your tab gets. Your browsers and your tab get to the point where you can't even read the tabs anymore. There's so many there and each one is a different action or a different capability. And I imagine in the SoC that that kind of context switching shows up and that it's really costly. Can you talk about what the cost of that disruption and or the inability to focus because of all those tools looks like? Liz?

Liz Pender (7:10)

Yeah, definitely. I mean when we talk about the impact of that actual screen switching, I feel like there's kind of two overall issues that happen. There's kind of the issues on the analyst side. So what I definitely experienced and then we see issues as well on that visibility and detection side and just kind of to talk more about the analyst side because it's obviously like my personal experience but I kind of like to think about, I don't know if you've ever heard of the, the, the article by Paul Graham which talks about kind of maker and manager time. So it's a quite an old piece of research quite a few years ago now. But essentially it goes through that maker time is something where you have long interrupted blocks to actually build and create something. Whereas manager time is kind of split into meetings, check ins, quick decisions and it's really that kind of make a time that you can directly associate with an analyst. You need to have that time for deep thinking with no interruptions, especially when you are going through an incident when you're triaging. And of course if you, I mean just thinking back to my experience, if you're having to continuously collect data for an alert, an alert comes in and I'm going to have to go to different sources to collect this data either through logging into a firewall platform or kind of going and querying logs in my SIEM solution, or even contacting someone, like contacting the owner of this kind of misconfigured S3 bucket for example. All of that time adds up on its own. But it's also that kind of mental overhead that you have like that's not really kind of thought of if you know, if someone interrupts you and you're kind of in the zone and then you get a slack message come in or you have a meeting put in as you're doing this task that requires that deep thinking. For me personally, it takes me like a good 30 minutes to actually get back into the task that I was originally presented with. So imagine that constantly when you are just triaging alone and how much time that adds to actually resolving that alert. That's a lot of the reason why we have such long mean time to respond is because of that jumping across different tools and gathering all that information.

David Moulton (9:52)

You know, Liz, you're talking about being interrupted by a different application or, you know, something coming in. There'll be times when things run slowly and, you know, I'm like, okay, I'll let that run and I'll come back to it later. I interrupt myself because I'm not willing to wait. And, you know, I guess I'm ashamed to admit this a bit, but sometimes I'll come back and be like, what was I even doing here? And I can only imagine if you compound that and then you run that, that same scenario under stress. You're trying to look into an incident, figure out what's going on, and you're holding a lot of different ideas in your mind as you run those alerts down. It doesn't help to have multiple applications adding to that cognitive overload.

Liz Pender (10:36)

Yeah, exactly. And you say it's like a shame to admit, but that's just the way the human brain works. And like you said, especially when you have that pressure there, and it's not even that, like, you know, if you're dealing with a big incident that's come in, I'm talking like, maybe like a low or informational alert that's come in even then. Just that overhead of like having to think about that one alert that may be a false positive just takes up so much time. So I have less time to then or less mental kind of energy to then focus on what matters.

Patrick Bile (11:15)

I think one of the best things about working in a SOC is that no day is like the same. The worst thing about working in a sock is there's no two days the same. It can be chaos, especially without automation. And we see this a lot with legacy socks, where if you don't have a grip on inconsistency on how you're going to respond to a type of incident, then it's the Wild West. I'll ask three people in a SOC how they tell me about how you deem whether something is malicious, and I can get three different answers, complete different answers, and who's to say, which is Right. Why? Who's to say which is right and which is wrong? So it's really up to the design of the SOC to decide what is the appropriate way to respond to that incident.

David Moulton (12:01)

Liz, you were talking about how you came into security and then saw the rise of automation. Talk to me about where you see automation making the biggest dent in reducing the cost of this switching that we've been talking about.

Liz Pender (12:19)

I think automation is just a great start, especially when it comes to bringing tools together, integrating those tools together. And it's something that I experience, as I said before we brought in, it was Demisto at the time, now exor into our own soc. And that really changed just the way of working. My daily job became so much more interesting. So instead of receiving and having to deal with a phishing email coming in, for example, or a user submitted email, instead of me having to go through that, and nine times out of 10 at the time it was either emotet or spam and it was just so boring and mundane and going from that to then having these kind of auto resolved, closed down and focusing on actually really interesting phishing emails, really interesting malware. I'm not only able to actually use the skills that I've been training for, but also of course, what was going on about before, extremely reducing that. Meantime, to respond by bringing together all of the integrations that we use in a phishing response process. Just automatically dealing with that just made a huge change to not only the soc but also my experience in my role as well.

Patrick Bile (13:51)

Yeah, I think everyone who gets into security analytics, maybe they don't know what they're getting into. But like how Liz fell into it and how I kind of fell into it is. That sounds interesting. I want to go and do that because it sounds like a challenge and we want to be challenged. I think the SOC Personas want to be the hero. They definitely want to be useful. Nobody wants to be dealing with false positives. They are inevitable, of course, that there will be a time when you will work on something and you will invest your hard time on it and it will result in an unfulfilled incident, or an incident that cannot be resolved, or an incident that's false positive. But if you do have automation, then you can reduce the, you know, the context switching and you can ensure that the SOC can be fulfilled in their job. So they can work on things that are genuinely interesting to them. They can potentially be the hero and we play on the hero complex, but they can potentially be the hero and they can do what they are Interesting. And they will have a material impact to the organization that they work for.

David Moulton (14:53)

How can leaders streamline those SOC environments that they're in charge of without sacrificing their detection and response quality?

Liz Pender (15:06)

We hear customers talking about streamlining and this happened even when I, so I first started the SoC, like must be nearly eight years ago now. And even when I first started, and I'm hearing customers say it as well, they always say, I want this single pane of glass. I remember my manager being like this enigma, this single pane of glass. I want to get there. It's like Harry Potter and the single pane of glass. Honestly, it was like the goal. But if you think about it, because the overall dream when they talk about single pane of glass is what we talked about before. Fewer loggings, less tool hopping, like a central pace. But if you think about a single front end or UI consolidation, some automation may lead to kind of fewer tabs open. But then the problem is that that only really goes so far in actually helping an analyst understand the full story of an event. So when we kind of brought in this whole unified front end single pane when I was there in the SoC, you still had that issue of I have to, as an analyst manually still connect the dots. I still have to manually trawl through logs to connect the dots myself and get the full kind of causality, the full visibility of this alert or of this incident. And as a human, we can only go so far. And actually in actually correlating that data together, I'm not machine, so you're really losing out on, you're really still increasing that mean time to respond because you're having to manually trawl through that data and establish that causality, which really adds to that response time.

David Moulton (16:58)

Let's talk about alert fatigue. When analysts jump from tool to tool and alert to alert, how do we ensure that they can stay focused on what matters?

Patrick Bile (17:08)

So yes, when we speak to socks and we say, what would you like to automate? And it's an intentionally provocative question, we normally get two answers, everything or we don't know. And that's, you know, I'm not sure which one's. Which one's scarier, to be honest. But probably we don't know because like, if we're talking about alert fatigue, they should know the type of alert that is causing them to be fatigued or alerts. So really, like when we're talking to socks, don't pick that one horrible task on that horrible system that you don't like doing. That you have to do once every six months or every year, do the things that you do little and often. If you can shave off 30 seconds here, a minute here, and you do that numerous times a day, week, month, then there's your return of investment on your sock and there's automation being key for you and there's your reduction on burning out because you're not doing the same thing over and over again. And that's the stuff that drove me up the wall, you know, repeating those mundane tasks and also thinking about from the risk perspective again, that's the stuff that people in the soc would forget to do or intentionally not do because they have a bias to know what that the result is. So they assume it's benign or they assume it's pernicious and they'll just quickly try and close the incident down. That's the wrong behavior that introduces risk which we, the sock is there to avoid.

Liz Pender (18:38)

Right.

Patrick Bile (18:38)

Or reduce the risk. Sorry.

Liz Pender (18:41)

It's all about giving them something interesting to look at. Right? Because we talk about, you know, how do we, how do we not interrupt that flow? And first of all, we can go by automation. So not just automation in terms of all the way to resolution, all the way to, let's block this straight away, really simply, we can utilize automation to enrich an alert. So instead of me having to go to my various open source intelligence tools to look up this one IP address, I can have all that information provided to me straight away. So I can just make that informed decisions or analysts can make that informed decisions to then isolate that machine or close that alert down. So it's really that low hanging fruit almost that helps combat that alert fatigue.

David Moulton (19:36)

Patrick I think that humans seek out the new and novel and when we're running through something that is extraordinarily repetitive, especially if it's a set of tasks in a row and we tend to forget things or skip over things because it's not new and novel, it's repetitive. I think this is why pilots and surgeons have pre flight or pre surgery checklists that they go through to make sure that they've flipped the right levers or washed their hands in the right order. And when you're talking about some of that automation, I feel like it's that checklist, but it actually goes through and it does that, that behavior. And that's the behavior that you want so that you can get to, you know, case of the, the pilot taking off in a safe way or landing in a safe way. And is that what you're seeing Here is like the, the part that's new and novel, the investigation, running down and understanding. If this is a malicious activity, that's what you want somebody to be able to focus on and bring all of their, their talent and their flow to, and not forget, oh, we've got to log this or write that down, or check here for information or make an assumption. You want to move those things into automation because it's 20, 30 seconds, but it counts that you had the discipline to do it every single time.

Patrick Bile (20:55)

Absolutely. There was one soccer engagement and it stuck in my mind because we were talking about their value to the business and how he engages. And he was there, level three analyst. So he was, he was the man in their sock, one of the most senior people they had. And his frustration was, which he, you know, he didn't necessarily say in front of certain management, but he said to me, every 30 minutes I'm expected to stop investigating or stop analyzing this incident and provide management with a summary of what's going on. And I was like, dude, that's, that's a process. Like, that's something you should be automating. And let me show you how to do that. You, you know, you email this distributionist with this report format. If the alert is open and in this status, I. E. Critical, then you can tell them automatically, here's what's going on, here's the next step. That's just stuff that you're doing as part of an investigation and that's kind of case management stuff which can be extrapolated and sent to people. So that was not the typical use case for solving SOC problems, but it was a burden to him and that was a risk to the business because if he left as one of their two most seniors, engineers or analysts, that was a problem for that business because that's something that's really easy to fix and that's often not something that a soc would step back and go, this is causing us pain. This, this could be a quick fix because it's a process, it's a thing. It's not, it's not resulting in a happy analyst and it's increasing our mttr.

David Moulton (22:27)

I'm sure you could calculate the, I don't know the, the cost of, of the automation or the savings, but from a overall risk reduction, it seems like almost invaluable. Liz, you've designed playbooks and that's what we tend to call these pre flight checklists. And some of the automations in security is a playbook or a workflow with a lot of customers. What mistakes do you see teams make that actually increase their context switching during instant responses?

Liz Pender (23:30)

Yeah, so something that was probably most common, and I think Patrick will agree, is you can't automate, you know, without having that process there in the first place. Right. So quite often, you know, customers come to us or, you know, I'm building a playbook and that process either doesn't exist in the first place or it's a bad process, you know. So for example, we had a customer that wanted to simply just reset a password and remove them from ad if there was like an insider, insider threat. So what they, they didn't have the proper process written down. So it was really difficult to kind of then automate it. They didn't think about, you know, what if this, what if this person was a VIP user? Do you want to like change the password of a ciso, for example? You know, so things like that, if you, if you put in a bad process into automation, you know, into create a playbook out of a bad process, you're going to have a bad playbook. So really you need to think about that process and go through it beforehand before you, you know, think about automation.

Patrick Bile (24:45)

It's not all or nothing. And I think there is that fear of, well, we want to automate, we want to automate fully, but we don't trust it that you can and should implement guardrails for break glass situations like potentially putting yourself at risk of losing your job by resetting someone's password who's in the exact position that you shouldn't. But you could argue as well that they're probably the people who would like you to be targeted. And let's test it, keep testing it. It's not a set and forget type thing. It's an iterative process that you want to test and refine. Unless it's not the high fidelity, 100% accurate things that I said, like who is user, what is ip, what is mach, what is cve, all those things.

David Moulton (25:29)

As we're talking about this, I'm realizing more and more that the human factor is critical. If you can't focus, if you're burnt out, if you're incentivized wrong, all of those things can come in and compound to increase your risk and lower the outcomes that you're looking for. If you were to talk about what a more human centered sock design would look like, what are the ideals there and how do organizations get closer to that?

Liz Pender (25:59)

When you first start the SoC, you are especially fresh out of uni, right? You are the keenest being, you're super keen. And just like what we were saying before, around, you know, you're hired, you joined cyber because it's exciting and, you know, you're hired as an analyst because you have all this background. Like my job description when I applied as an analyst, you have to have Python skills, you have to have, you know, reverse engineering. And it was like a bit of a shock to the system when I actually got down and did the job that I did. None of that, you know, it was, it was still, you know, kind of interesting. But after a year of, you know, closing down false positives and copy and pasting indications of compromise, you think, you know, there's my brain, you know, needs something more. It needs assistance. So really it's about achieving a role for an analyst that encompasses what you actually were trained for. And I think if we look at our own SoC, Palo Alto SoC, I think that's a perfect model of that, where they don't spend all of their time responding to incidents, triaging alerts. A lot of that time is on threat hunting, things like research. That was my favorite thing in the soc was you're doing that more proactive instead of reactive threat hunting. So just kind of, yeah, relating back to our own SOC and the way we've achieved that is through that unified platform, not only having that unified front end, but that unified backend as well and normalization of that data.

David Moulton (27:47)

So it'll come as no surprise. I'm going to shift the conversation to AI, hoping by now you guys have heard of these AI co pilots or assistants, a lot of which are being pitched as a fix for the overload that we're talking about today. I'm curious what's realistic and what's hype when it comes to using AI to reduce some of the SOC fatigue.

Liz Pender (28:15)

Yeah, so obviously AI, it's become such a buzzword. You know, I think every single vendor, every, every soc comes up to us and wants to talk about AI and what we're doing around it. And I think, I mean, in reality, if we, you know, look at the umbrella of AI and what's within that, PALO has been using, or Cortex has been using machine learning for years in terms of, in the analytics on that normalized data that we're seeing in xiam, and also just like taking it all the way back in our prevention modules around behavioral threat prevention and local analysis. And I mean, obviously that's. We've seen huge benefits around AI within that context. Instead of relying on those static rule matches, machine learning Allows that analyzing of that behavior over time and identifying what's normal to a machine or user. But again, just going back, what is overhyped in my opinion is the use of AI and more specifically those large language models or LLMs in soft tools. So obviously everyone's, I'm sure everyone listening to this has used ChatGPT. They are non deterministic in nature. You know, I can ask them, can you plan my Disney holiday to one and it will come out, you know, completely different Disney holiday to the next. And I'll ask the same again, it will plan a completely different, you know, the same model can plan a completely different holiday. So imagine that in the context of a copilot. You know, we get questions like this all the time and we say, do you have AI to summarize an incident? And if you say can you summarize this incident? It will come up with different answers every time. You'll likely get a response that's inconsistent and inaccurate. And often these are sold or pitched as reducing that stock fatigue as helping out first line untrained analysts. But in reality you just make it worse because first line analysts or people that have just joined a soc like myself, when I first joined, I would have taken something like that as face value. I wouldn't have been able to determine if that summary of that incident was correct, was accurate. And another thing that it does as well is it completely takes away the skill that you actually brought in that analyst for that logical and analytical thinker and instead it's been replaced by that vague assistance.

Patrick Bile (31:04)

Sometimes we get that question of tell me about your AI. And I think that is a telltale sign that they are looking at potentially too much marketing material and not looking at what are the outcomes that I want from my function in this case security operations. It's like cool, we can talk about it, but what do you want from it? Take it back to the problem of where is the pain. So yeah, we see it being really effective where there is a specific goal in mind. But yeah, there is stuff that is innovation for the sake of innovation. And yeah, some of it will get better, absolutely. But some of it is, is adding another thing for the soc to do. And it's in vogue, it's sexy, but it doesn't contribute to the outcomes of the SoC.

Liz Pender (31:55)

And it's not to say that copilots with LLMs aren't valuable, especially when we think about more kind of RAG LLMs, retrieval augmentation generation based LLMs that have that relevant information, it retrieves that relevant information from a knowledge base or anything like that. And that's useful, especially when it comes to helping a new analyst with a specific tool. So you know, how do I isolate a machine? And it'll look at kind of the work documents around that because let's face it, you spend a lot of time getting used to a tool. So I think it's really valuable when it comes to things like that.

David Moulton (32:37)

Yeah, Liz, I tend to agree. I sometimes look at the tools that we have in front of us and you're Talking about the LLMs or at least that's what I'm thinking about. And your knee jerk reaction is go do the summary or write this for me. And I prefer the model of here's what I think about this situation. Am I missing something? Have I got a bias or a blind spot or I need to learn about how to use a tool and walk me through how to use this interface or how to use these tools in a way that I hadn't considered because I don't have the time to go through and you know, build up that experience and having a coach. And it sounds to me that the right use case and the right application of some of these tools, great. But just a wholesale handing off ends up putting you at a point where you end up taking longer because you're not getting a good outcome that's deterministic or repeatable and you don't have confidence in it. And I think that's the, that's the lovely thing about automation is that once you have it going and you've built to that point of trust, you can run it and start to then take that savings of time and apply it to the things that we humans are uniquely capable of doing.

Patrick Bile (33:55)

Yeah, it's robust and it would have been something a human would have had to do multiple times. And that's the thing. If you do it once, it's not going to be beneficial. If you do it 5, 10, 15, 20 times, that is massively beneficial.

David Moulton (34:10)

Patrick, what's the most important thing a listener should remember from our conversation today?

Patrick Bile (34:16)

Realizing you need to reduce context switching is a great start. Change is very rarely as scary as you think it is when you do it once you go into future proof your security operation center. I think we have to kind of prepare ourselves to be constantly disruptive because the attackers are continuing to innovate. And I think if you're staying static or you're afraid to change or you have Stockholm syndrome because you've poured loads of effort into this tool or these tools and we can't possibly change things. Then that will become a risk to the business, either through attrition from the experts have done that. Or. Or through tools not innovating or through more tools being acquired and, you know, increasing the demands on the sock. So they have to switch context more often.

Liz Pender (35:08)

Yeah, what Paddy said. No, I. Yeah, just to echo that, I agree. You know, I mean, just going back what I've said before previously, I've been in the industry almost, you know, eight years, nine years, and it's still an issue like alert fatigue, still spoken about, you know, high MTTRs. And it's, it's. It's getting worse because we've got more data, you know, than we've ever had before. So I guess my overall kind of comment would be that something has to change. We have to do something different. You know, legacy, legacy SOC tools just aren't working anymore. And yeah, essentially it's what Paddy was saying. It's really scary. You need to make the change, otherwise it's going to get worse, I think as well.

Patrick Bile (35:58)

Like validation. If you are unsure of something, then you can validate. You can have a purple team exercise where it's kind of a collaborative thing, or you can take a step back and look at what is our strategy for the SoC. What are we going to, you know, we've implemented this thing, now it's working or we're unsure it's working. How can we validate that's happened and how can we may maybe perform some sort of collaborative exercise like something with unit 42 where we do our purple team, or we'll do a tabletop exercise or play out different risk scenarios to see how the sock would act and how that should be a continuous thing as well.

David Moulton (36:38)

Patrick, thanks so much for a great conversation today. I really appreciate you sharing your experience and your insights on context switching in the SoC and maybe how we can design a smarter, more sustainable analyst workflow.

Liz Pender (36:52)

Yeah. Thank you so much for having us. It was really fun. We took a trip down memory lane. It was traumatic, but we got through.

Patrick Bile (37:00)

Yeah, we had fun. We were allowed to have fun occasionally. So it's good that we got to do it with you, David. Thank you.

Liz Pender (37:06)

Foreign.

David Moulton (37:11)

That's it for today. If you like what you heard, please subscribe wherever you listen and leave us a review on Apple Podcast or Spotify. Your reviews and feedback really do help me understand what you want to hear about or reach out to me directly@threatvectoraltanetworks.com I want to thank our executive producer, Michael Heller, our content and production teams, with which include Kenny Miller, Joe Bacourt and Virginia Tran. Elliot Peltzman edits the show and mixes the audio. We'll be back next week. Until then, stay secure, stay vigilant. Goodbye for now.