A (4:58)

So you're training the model on your own data inside the company. You didn't just grab some tool of GitHub and all of a sudden it works. Right.

B (5:07)

There are those tools out there that exist in the platforms and some of them are good at what they do in the basic. And I say basic because it's very complex and not my complete background of, of creating those. But you know, there are certain things that are unique to a business and those unique things are where we are focusing our time and effort to create our own custom on top of the, you know, and kind of supplementing the basic ones or general generic ones that come out of, out of the box with some of these.

A (5:47)

So right now is this tool working in real time or do you need like five or ten minutes to get an alert?

B (5:54)

So right now it is responsive or reactive? Right. And so what we are driving towards right now is kind of A, I want to call it an intelligent decision where you get an alert that data comes in, it gets processed by the language models and it comes back with a category of should you investigate it further or does it appear to be kind of a false positive on the ones that are the positives. Right. That we're truly going into investigating. We are taking those positive items as well as the negatives and we're, we're working on the back end to become more proactive where you can now use those language models and the AI to not only protect the data before it leaves the company, but truly go after it and find out where does it live, Are there proper security controls? Do they have, you know, the proper data sensitivity labels and whatnot on those, you know, whether it's a device, a site, a share. And so we're leveraging now. Once we become more confident in the responses we're getting on the response side, we're going to take that build on kind of what we learned and become more proactive so that you're stopping it before the data actually even leaves the company.

A (7:17)

So how long have you been using this so far?

B (7:20)

Conceptually, we've been trying to do it for about eight months. Realistically we have been about four or five months developing it and implementing and trying to, to get the right responses back from the language models and the, and the systems.

A (7:38)

Okay. So there's basically around a one year implementation pH until get something useful. Right. So it's not just a plug and play solution. I think that's important.

B (7:46)

No, I mean there are, there are some out there that can do it. You know, let's, let's take some of the platforms like you know, Microsoft, they have their trainable classifiers that go in depending on what licensing you have. And, and you know, there's, there's many others out there that, that do this and, and like you said, the, I'm kind of tired of hearing the AI like oh, AI can solve everything, you know, sales pitch from everyone because I think it's, there's a danger in relying in on it too much. But then to be able to supplement and educate in our space. Right, Educate very technical analysts about what the data is that's very technical but totally outside the scope of their work. Helps them make a more, you know, informed decision about, you know, how, how serious is this is this event that we're looking at and it's been a journey and you know, out of the box things work to some extent, but they're going to miss, you know, some of your more proprietary formats that are, that are specific to, to your type of business.

A (8:56)

Okay, so this has benefits for your legal team as well, not just your security team, because they get clearer alerts, they don't waste their time on false positives. That's some sort of downstream effect, right?

B (9:09)

Yeah, absolutely. I think, you know, to say that it doesn't help the incident handlers on our team as they're going through an incident, it 100% helps them. Right. Because it now can summarize some of those documents that might have taken, you know, days and hours to go through in some capacity. But on the response side of it, absolutely, we can go to a business owner and we can say, you know, we've seen 15,000 documents that were, you know, exfil out of the company through whatever means. But of those 15,000, we believe only 25 of them are related to a certain category that we've worked with them to build out the models. And now instead of asking to go through all 15,000, we're just saying, hey, let's go focus on these 15. And depending on what they are like, that helps you. Instead of creating a triage process where everything is looked at and then we decide whether it's low, medium, medium, high, this helps bubble up some of those things because it's. You're never going to have enough staff to get through all of those events.

A (10:21)

Is this your first time trying to use AI internally? Because just as much as I'm interested in proper ways of using AI, I'm just as interested in the wrong ways of using it. So did you try it with something else and then decided that doesn't work, it's just creating more work for us?

B (10:36)

It's very new for my team. Right. It's very new in how we're leveraging it. I am sure, you know, there will will come times where, you know, it's not working as we, as we expected it to. But we're still in the phase of training the models on a lot of the data we have. We have a couple use cases completed, but there's many more that we want to go leverage the capability on. I think, speaking from security perspective. Right. I think the downside would be if people rely 100% on what is returned without, you know, making sure that it's validated, like it's just another piece of data that an analyst should use. It is not the end all answer that analyst, I believe should be using when it comes to an incident. And so if we leverage that additional data point, you Know, it just helps our investigations, our matters, our alerts become that much better. Right. Because now we're going to learn from that matter and then the next one's going to be a little bit better and then the following matter we'll learn from the 10 previous. And so I, I think the one place I get a little nervous is if people or analysts rely 100% on what's coming back versus doing what analysts should be doing, which is digging into the data and validating the ones and zeros and the facts that are presented to them.

A (12:07)

What kind of internal metrics or analytics are you using to see if this actually works in any way? Because I'm interested in how people evaluate this kind of tools in their workflow. Like, do you look at certain things and then decide six months later that it's just not worth it?

B (12:24)

You know, again, we're, we're still a little early in it, but we have the way our process works. You know, we have analysts that accept or reject the tickets, right. It's either true positive, false positive. And right now what we have implemented is a suggested triage decision based on some of the categories that we have. And so we're seeing about a 10% of the time where the analyst rejected and the model said reject it. Right? So that's a 10% potential reduction of work that obviously, as they get better, as the models learn more, as we work more with the business to get them kind of better data sets to evaluate, we expect that number to go, you know, and be, be better, but it's not going to ever be 100%. Like, I don't expect that. But what I do think some of the metrics are that we could look at and that we're going to look at is speed to triage. Right. Incident comes in at a certain time, how long does it take you to triage the incident, and then how many of the incidents are being bumped up from a low to a high or critical. Right. Based on some of the language model feedback. I also think another way that you could measure the success of it is we have a metric around time to, basically time to the manager, right. Or the subject matter expert of when we get their decision. So time to decision on the importance of the data. And so I think you would find those numbers would significantly, I would expect them to significantly drop if you're asking someone to review, you know, 5 or 10 or 20 documents versus 1500 or 3000 documents. So I think there are definite ways you can measure the success of it still, I Still think it's still very early in kind of that flow. But there's definitely, I believe, KPIs that you could look at that will show how effective it actually is in your environment.

A (14:30)

So you're basically looking at it like a time optimizer, right?

B (14:34)

Correct. Yep. Yeah. Because you know, I mean there look, there's, there is, there's only 24 hours in a day. You only have a certain number of people. And so when the alerts are coming in at such a rate, there has to be some way to automate some of those risks that maybe will show up in a ticket. And I think again, NLP and the AI piece of it is not the end all, be all. It is just another data point to help raise that priority. So as an analyst goes and looks at a dashboard, they can start to see like, huh, this one's a little bit higher. I should probably focus on this one right now or next versus, you know, one that hit on a term that was confidential in, I don't know, a resume. Right. And that one could have been auto rejected.

A (15:23)

Budgets are always a hot topic when it comes to corporate security. So I was wondering what is the cost of running something like this? Money and human resources wise? Is it something expensive? Do you need two, three people to constantly look at it or is just easier to automate and leave alone?

B (15:40)

Yeah, I mean, not getting into the specifics, but yeah, it's definitely a time consuming area and I think the more, the more time you put into it, the better those results are going to be. Right. So I specifically have someone on my team who we've, we've tasked with working with the business kind of on a weekly, bi weekly basis of finding, you know, where sensitive documents live, making sure that we can categorize them, building out the strategy on how we're going to go, kind of scale that across the enterprise. But we're also leveraging a lot of our relationships we have with various vendors that have existing contracts with the company. So you know, as far as cost, it can be quite expensive if you don't have any tooling already in place. And also depending on how complex your environment is, you know, if, if you, if you have an environment with, you know, 500 people, it's going to be, you know, probably, you know, pretty inexpensive compared to an environment with, you know, 250,000 people around the globe. So, you know, it's, it's definitely an investment of human hours for sure with no real, you don't know what the outcome is going to be. And what the return on that investment is until you get to a point of where you start seeing analyst triage time, time to remediation, kind of all of those time metrics that we're going to measure start decreasing.

A (17:13)

So what's the reaction from upstairs? Do the bosses have any idea what you're doing?

B (17:18)

So they do know. And we are, the direction we are going is around taking some of these models to discover where the data sits in the various data repositories on our networks. You know, where, where we may not have, you know, great kind of visibility. Right now we're going to take those models and now we're going to be able to make sure that if we say this document, because of partnering with the business now we have the language models and we have the scanning capabilities and all of that to kind of go do it. We're now going to say, okay, this document is sitting out in a repository where it should maybe have a different setting of permissions. And so that is really where we're going to start truly locking down more than it already is and implementing some tighter controls around the crown jewels, the very valuable intellectual property, better leveraging it for monitoring of some of the key systems that could have crown jewels and things like that. So you know, it's already taking what we have in place and enhancing it and making it better. And I think that's the key any security organization should be doing right. It's, it's lessons learned. And if you're not learning from your incidents, then you're probably doing security the wrong way. You should always be trying to look for better ways to improve it. And so, you know, I think we, we do a very good job and this is just the next iteration of how do we improve our ability to not only protect the data, but respond more intelligently when we have, at least in my case, right insider threat type matter. How do you arm the analysts that may not know the data with the information they need to kind of make an informed decision and make sure that the right people in the business are aware of what that data is.

A (19:25)

See now I'm glad we had this conversation because I would have never thought you could use this system as some sort of asset discovery for intellectual property. Because documents usually find their way outside your walled garden. They reach workstations you never thought they'd reach. And this is a good way to find those leaks.

B (19:45)

Yeah, I mean, and you know, you could have someone open up a document and copy that data into a brand new document that maybe has a lesser classification and is very kind of generic. And instead of what I really like about this, you'll be able to piece events together. But if you relied 100% on DLP in that scenario, you'd never find it. There's not a chance that you would find it. And with this, now what you have is you leverage all the events that you have, all the logging. You now maybe have a little bit of the DLP component. But now, like, even if you didn't have that, you can have the language models and the AI scan the document and whether OCR it or not, it'll be able to tell you, hey, this document fits something in this category. Even though it's not labeled as sensitive, it's going to be treated as if it's sensitive. And now you can block it, you can prevent it, you can send someone a message saying, hey, you probably should put tighter controls on this. And in some cases, even if they were to have the ability to potentially email it out, you can then encrypt it and do all kinds of, of things to it. So it becomes, you know, not, not usable to people. But it's going to be a really, you know, and that's why I say like the, the first piece is getting the language models right. Right. So it can identify what are the categories of things that you're, you're interested in, then taking those, learning it into a different type of system that will help you proactively identify those systems that contain similar data that maybe you didn't know about before. Maybe someone copied it down to their PC or, you know, so I think it's, it's really neat and really excited on kind of where, where my team is taking it.

A (21:45)

Brian, thank you for this insight into how companies could be using AI instead of, you know, using simple ChatGPT queries and rewarding documents.

B (21:56)

Yeah, Yep, I, I'm right there with you. You know, please write me this email. But, but yeah, this is, this is more where, where I think, you know, teams like, like mine can leverage. It is summarization, right? Data summarization. And I think it's going to be a game changer for a lot of the security team teams.

A (22:17)

Brian, thank you very much.

B (22:18)

Thank you.