Loading summary
Transcript19 lines
- [00:00]
A
Foreign.
- [00:05]
B
This is Tom Uren with another Risky Business News sponsor interview. Today I have with me Edward Wu, the CEO and founder of DropZone AI. G' day Edward. How are you?
- [00:15]
A
I'm doing well.
- [00:16]
B
Dropzone makes AI SoC analysts and they've published a first of its kind study measuring actually how AI helps society stock analysts. And that's what we're going to talk about today. If you're interested in the study, you can find that at DropZone AI and we'll have a link in the show notes. So Edward, tell us about the study. What did you do?
- [00:38]
A
Yeah, it's a very interesting study actually. See, first of its kind. So when we think about AI soc analysts or any application of AI agents, see number one question is always, does having AI agents give you more benefits than the potential downside risk of being inaccurate or creating more distractions that end up costing more time? So obviously we have been building our AI Soc analysts for over two years now and when we looked around in the industry it didn't seem like there has been any definitive study on the actual efficacy of human security analysts working with and versus without AI. For this study we partnered with Cloud Security alliance and what we have done is we gathered a group of 148 analysts across different experience levels and specializations and we essentially had them specifically split into two groups. One group investigated two security alerts manually. The other group investigated the same two alerts with the help of our product. And at the end we compared, you know, the accuracy, the speed, the completeness and the fatigue resistance of the two groups.
- [02:10]
B
So Edward, what did the study find in terms of speed and accuracy and I guess even fatigue Resistance?
- [02:19]
A
Yeah. With AI support analysts were 22 to 29% more accurate and finished their investigations 45 to 61% faster compared to the manual control group. And the AI assisted group was also significantly less fatigued measured by the relative small decrease in completeness of the second investigation.
- [02:48]
B
Right, so I guess that's measuring how well you do the job rather than how quickly, is that?
- [02:53]
A
Right?
- [02:54]
B
So if you're tried, people would tend to, I don't know, skip a step or not do entirely as complete a job or something like that.
- [03:05]
A
Exactly, yeah.
- [03:06]
B
Does that match your experience? Like you must have rolled out these kind of deployments and does that match what you hear back from customers?
- [03:15]
A
Good question. Directionally, it definitely matches our experience. Frankly, the evaluation or the benchmark study, the measured differences were actually smaller than what we have seen across our early adopters and customers and we attributed a lot of that to the fact that these are actual real life security analysts who have never seen or interacted with dropzone before. So this is like the first time they have used the product. And as you can imagine, with any new technology, there are some learning period where when the analysts or the participants were leveraging the AI assistance, they were not able to fully benefit from 100% of its capabilities. And in comparison, when we talk to our early adopters and customers, those folks have already acquired the technology, they are using it every single day. So the actual impact or size of the impact is much larger once they get more familiar with it, once they get to customize and tune the system based on their likings and environment.
- [04:35]
B
Yeah, I was looking through the report and what struck me as really interesting was that there's this section which looks at the respondents, how they felt about AI after the experience. And almost all of them said it was efficient and helpful, but the number for trust was actually relatively low. I think it was something like 35% of people actually trusted it. And I thought it was really interesting that you could have a technology that you think is helpful, useful, not actually that trustworthy. And that sort of matches my experience trying to teach my daughter maths, where the AI can be very helpful in giving you a framework, but you don't entirely trust it. I just thought that was a sort of fascinating data point because the discussion is often, if you can't trust it to be as good as a person, it's no good at all. Whereas I don't think that's right.
- [05:29]
A
Yeah, a lot of people say trust is earned. Right. It's earned over a period of consistent performance. Right. So obviously within this study, the participants were only able to experience drop zone within the context of two investigations. I agree that across two investigations it's hard to trust a product. But from what we have seen in the field, once a security team has experienced consistent, high quality investigations and consistent performance across 50, across 100, across a thousand investigations, they do, over time start to trust the system more.
- [06:12]
B
Yeah, that makes sense. Now, in terms of you're not trying to replace SOC analysts is my take. Is that right? And also how do you think this affects learning and development? Is it upskilling like total noob SOC analysts helping them to work better from the get go, or is it something where you still need an onboarding process and then drop zone or whatever AI could fit in later and help them as they become more experienced?
- [06:44]
A
Good question. I know how the different job roles change after AI augmentation is a heated Topic across every single white collar job family. I'm sure a lot of developers have also been thinking about, now we have cursor and cloud code, what are we going to do? And one consistent theme I have observed is for a lot of these roles, what AI agents are going to do is they are going to handle the toil, they are going to handle the manual repetitive work. And by the virtue of doing that, they can up level the existing human engineers or human analysts to be more like generals or special forces. And would I envision AI SoC analyst influence the security job market in a very similar way? And more specifically, I do anticipate a world where in the future there is no such role as a tier one SOC analyst. You are either a student learning cybersecurity or once you kind of learned enough, you join a company and start immediately doing tier 2 work. And you could argue, yes, the gap between being a student and the entry level job has increased, right? Because previously you can start as a tier one security analyst and be useful for a company. But now because of the increasing automation, the barrier of the entry has been raised a little bit similar to again all sorts of jobs who have historically benefited from automation. With additional automation, it does force folks to transition. Where previously you can actually get hired by automating and working on a lot of the entry level tasks, but now with automation you have to uplevel and become a tier 2 or tier 3 before the company really start to see value. And this is where one thing a lot of people are concerned about is this increase in barrier of the entry will really hurt the talent pipeline. From my perspective, obviously we don't agree with it because we have seen in many other industries where increasing automation can actually benefit the development of talent. If we look at airline pilots, most airline pilots are leveraging autopilot, oftentimes auto takeoffs or auto landing as well. But at the same time, the same technology used to perform autopilot auto takeoff are also being utilized to build simulators which where they can train airline pilots in all sorts of interesting situations. You know what happens if an engine goes out when you are taking off or landing? What happens if one of the tires blew off when you are landing? And this is where we have seen many cases by having an AI SoC analyst, it actually increases and accelerates the learning process from a cybersecurity student to a tier two analyst. Because historically in order to cross that chasm, one has to spend years within the SOC grinding through thousands of security alerts and gather additional context and experience and techniques along the way from their colleagues or team members. But now with an AI SoC analyst, it's kind of like everybody having your own personal coach, a junior security analyst or a security analyst intern can now compare notes against one of the best human security analysts on the planet on every single alert. And that's where the junior security analyst can quickly up level themselves, maybe just within six months, instead of having to go through the tier one phase for a couple years.
- [10:59]
B
So on that very positive note, Edward, thank you for an interesting discussion about how AI actually helps in the SoC.
- [11:07]
A
Thank you.