Loading summary
Transcript38 lines
- [00:02]
A
You're listening to the Cyberwire Network, powered by N2K.
- [00:13]
B
Welcome to Threat Pick for the Palo Alto Networks podcast, where we discuss pressing cybersecurity threats and resilience and uncover insights into the latest industry trends. I'm your host, David Moulton, Senior Director of thought leadership for unit 42.
- [00:28]
A
You don't need to test the model, you need to test the system. Because it's not just the model, it's not just the provisioning cloud provider. It's all those other things you connect to it. And the more things you connect to it, the more problems you can have just with scalability production. Does it work consistently? You know where the bugs exist, but you also introduce a lot more places where you're going to have risk from security or safety standpoint.
- [01:17]
B
Today I'm joined by Brett Kinsella. He's the General Manager of Fuel IX at Telus Digital. Brett has led marketing strategy and product growth at scale across both startups and major enterprises. He's also a widely published author and speaker with work featured in Harvard Business Review, USA Today, and Wired, and he's hosted over 400 podcasts on AI innovation. Today we're going to talk about securing generative AI systems, the real risks behind shadow AI, and what leaders need to know about hallucinations, data leakage and testing the full system, not just the model.
- [01:55]
A
Foreign.
- [02:02]
B
Welcome to Threat Vector. I'm really excited to have you here today, David.
- [02:06]
A
I'm excited to be here. I've been listening to the show for a while. I caught a couple of your recent ones. In fact, one of my friends was on the show who I've known for many years, and it's an honor to be sitting here across from you.
- [02:18]
B
Let's open up and talk about your journey a little bit. How did you end up leading Fuel IX and then becoming this leading voice in AI and synthetic media?
- [02:30]
A
There was no grand design. I've been in new technology for a long time, and if you're in new technology and develop a passion for it, you wind up working in a number of different technologies and across a number of different industries, because new technologies typically aren't limited to industries. So I worked in consulting, I wound up in software companies. I transformed a services business into a software business over the years. Worked with a lot of software companies, particularly SaaS back in the day. But one of them, in late 2012, launched a very innovative AI product and that was my first introduction. And then I wound up working with a number of startups in that space and then realized that I didn't have the type of information I needed in order to make good decisions about what was going on the market, how things were being adopted, what technology mattered. And so I started doing some work on that, wrote a couple articles, and then just by accident, started a research business and publication for several years that I did as well. And then I got to know a lot of people in the AI space, obviously. In fact, I've hosted about 400 podcasts myself. Like, just interviewing AI innovators over the last decade. Um, and not too long ago, I guess a year and a half ago, the president of Telus Digital called me up and said, hey, I've got something that I want you to take a look at. We've developed some really interesting technology. We're using it internally. Our customers are asking us if they could use it. So you've done a lot of software, Brett, you've done a lot of AI. Could you come in and take a look at it? And that's basically how it started.
- [04:09]
B
Before we kick off, I want to come back to the conversation that we started off mic. We were talking about ultramarathons as something that you enjoy doing. And I'm curious. Can you talk to me about that first spark that got you into ultramarathons?
- [04:28]
A
Well, it actually was quite a bit before I actually even thought about signing up for one. I remember reading an article about Leadville 100 miler, and that's in the mountains. It's. The whole race is above 10,000ft. There's a lot of vertical gain and things like that. And I said, that was amazing. That sounds really interesting. And then I didn't think of it again. Seemed just something that would. Sounded cool, but other people do it. And then when I was living in Massachusetts, I had some friends there who ran a lot. So I started running a lot and did a half marathon, did a couple marathons. And then at some point was like, oh, it'd be really interesting to do an ultramarathon. And there just happened to be one that the North Face was sponsoring. First year down the street for me, like, five miles from where I lived, and I was like, oh, I'm going to sign up for that. But a hurricane came in, so they canceled it, which usually they don't cancel ultras. Me and this other guy were the only two people that showed up. The weather was terrible, but it was super fun. And we wound up doing not a full 50 miles, but we did probably 25 that day. But it was a great story. But then he. He had Done dozens of ultra marathons. And then I just started running with him every Saturday for years. And then once you're in it, you're just like, oh, I'll do this next one, because the one was canceled. I signed up for a 40 miler the next month because I was already trained, that type of thing. And so I might as well get one in. And then it's just one thing leads to another.
- [05:58]
B
Brett, do you see any parallels between running an ultra marathon or ultra running in general and cybersecurity?
- [06:06]
A
I'm not sure that, like, I mean, I bet a lot of people who work in cybersecurity would relate to the fact that their life is about pain and problem solving and an occasional sense of victory and relying on others and problem solving again and dealing with new conditions every day. I mean, I think it's, there's, there's a lot of parallels.
- [06:29]
B
You know, Brett, enterprises are increasingly adopting AI. I don't think I've had a conversation in the last weeks or months doesn't land on how are you using AI to go faster or unlock a new capability and, or how are attackers using AI to speed up their attacks. I'm curious if you can talk about the hidden risks that are introduced when employees use consumer AI tools for work tasks without good IT oversight.
- [07:01]
A
So I think when we think about shadow AI, it is the same type of thing. And so if you think about, like, over half of people who are in large companies are in, are using some sort of AI, assistant, generative AI, about 2/3 of those are also admit that they've put sensitive company information into the chatbot. And so, like, what does that, what does that mean? This is not going to surprise anybody that's listening to this, I don't think. What are they doing? They're putting customer data in there. Well, why are they doing that? Because they want to do their job better. It's not because they want to disseminate customer data. They want to do some analysis on it. And the AI tools are just better than the tools that they've had in the past. They're putting company proprietary information. They're putting your code base for some sort of proprietary firmware that you have that runs all of your machines and your manufacturing plants or something like that to try to find a, a bug. They're putting company financial information in there, which could be before it's announced to the market. And where is that going? I think people think about this, say, okay, so the data, like, it's going so people could intercept the Data, maybe it's not encrypted, maybe it's stored. All these other different things could be used to train the model. These are things that every IT organization, every CISO has to be aware of. And the fact is, these tools are so much better than what we've had in the past. People are going to use them. They might even use them if you provide something. If you provide something to them. This is where you go back to this other thing. If it's not as good as what they're used to using at home, they might just use it and they might just use it anyway. Just even if it is as good, because it's just a habit in this thing. So you have to break that. So you can do things like policy, but I think you can't really hold back the tide. You can't just whitelist and blacklist certain websites and just hope that's going to solve all your problems, because it doesn't. We've seen it. People just use what they have because most people want to do their jobs better or worried about losing their job or want to get home sooner.
- [09:05]
B
Are there specific security vulnerabilities that you think leaders at these organizations should be focused on first and. Or specific behaviors that they should try to encourage, you know, their staffs or their organization to adopt?
- [09:23]
A
Well, I think in general, leaders should be encouraging their organization, not just the people, but the other organization, to adopt AI tools because they are really so much better than the things that we've had. Everybody's recognized this. You don't have to spend too much time with these tools to realize that they fill a gap. Often a gap that people didn't know they had, but certainly a gap. So. So that's the first thing they should be pushing the organization to do it. They should set up some policy about what's okay to do and what's not okay to do. But then they have to provide some tools because people are going to start using things anyway. Now, within those tools, I think the things that they would be most concerned about is first of all, data leakage, like data loss protection. We all know about that. This is just a new, very broad funnel to get things out there. And it's not just the chat, it's not just stuff that they copy in there they write about or anything like that. Many people have heard the term rag retrieval, augmented generation. It's a vector database. Like if you take large data stores, let's say you've got 50, 500, 5,000 documents, it's very hard to Find information in there. Traditional search, semantic search, get you part of the way there. But what if you have to concatenate things from several different parts? Like you become an information archaeologist. That's what search as we know it is. Tells you where to find something and then good luck. And then you put it all together, right? And the AI tools are really good at this. They just like, we'll find things that are similar in different parts of documents, different documents, all these things and put it all together, give you a nice synthesis. It just takes seconds. It's better than things you would normally even be able to do. And guess what just happened. You just uploaded and vectorized 5,000 of your documents or 50 of your documents, or your customer documents, or your contract documents. Right. Your IP research, all those things. And then those are potentially available to anybody that can access that database. The vendor that does the vectorization, the model maker does the embedding model, the model maker that you use to do the retrieval. I mean there's just all these different points. So those are all the things that I think you want to think about. And if you just recognize this going to first principles, that people are going to use these anyway, try to give them something that you least have some control over, some governance that you can apply to.
- [11:43]
B
Yeah. A couple months ago I was at south by and the researchers out of Carnegie Mellon were presenting and I realized that the data leakage issue is a huge concern. They were actually talking about the data return from the LLMs having a 40 to 70% of the time a hallucination, which is such great branding for outright lying or inaccurate, you know, made up new things. And I thought, what a wild thing to go into this clearly better tool but not have the expertise in the topic. And it returns with confidence. Something that has, let's say, on the low end of 40% of the time something's inaccurate. 70% of the time it's inaccurate. Either way, that's a lot of running down and trying to verify that information. And do we have the discipline to do that? And then that becomes part of the lore, part of the conversation, part of the fact set of an organization. So while you're slipping data out through the system, new and fantastical fantasies are coming in at an incredible rate. And it's one of those areas where I'm like, I don't know if that's a security vulnerability in the traditional sense, but it's certainly one of those areas of massive concern. Maybe it's, you know, brand and reputation issues when you go out to the world. And you say very confidently, you know, up is down, and everyone scratches their head and goes, nope. But you didn't know better. You know, you had this. This confident tool that took your data out for you and gave you new things that don't make any sense.
- [13:24]
A
Yeah, human error. It's hard to get rid of human error. It still exists, I would say. There's an interesting study. I think it was cohere that did it. And one of the things they were thinking about, they're one of the big model makers. People aren't familiar with them, but one of the things that they were looking at was what makes a user like a response. And this was. I think this was done probably close to a year ago, nine months ago now. So things change. So, but at the time, at least this was true. People liked longer answers. So that was one of the things that increased the perception of the user in terms of the answer. And that makes a lot of sense because when you get search, what do you get? You get links or even the snippet boxes that we used to have. It's like, okay. All of a sudden it's like, wow, you're giving me a lot of things. This is like, much more robust. I like that. The other big determinant, the only other thing that really made a big difference was the confidence or the decisiveness in which the information was returned. And even when the user knew the information was wrong or was told the information was wrong, they still like the decisive answers more. I kind of understand it like, that's what people. People want that. They want certainty in their world, but that's what they liked. So you got to be careful.
- [14:46]
B
Yeah, I would go one step further. It seems like lately there's a bit of flattery that has shown up in the pattern of, oh, Brett, that is a brilliant idea. You caught something, you sharpshooter. And as a former designer, I'm looking at a couple of those things of longer answers, a level of confidence that isn't warranted, flattery. And I'm going, we used to call that an anti pattern, and yet it seems to be pervasive as these, you know, these tools try to figure out where they are in your stack and how to get more users. And, you know, you don't want to run somebody off with no flattery, lack of confidence, and a short answer if in fact your next growth model depends on those things. And it's. It's a real weird push and pull.
- [15:37]
A
Sam.
- [16:05]
B
Listen, let's shift gears to AI chat security Vulnerabilities. The what, what emerging vulnerability patterns are you seeing in AI chat implementations that, that security teams seem to be overlooking?
- [16:20]
A
When you think about layered defense, you have your model makers, you have the, maybe the provider, the cloud provider or somebody else who. They all have different types of layers of defense. Your guardrails would be another layer of defense. Your prompt engineering would be a layer of defense. But the question is, how do you know where your problems are? And I think this is the place that's often overlooked and that's how do you identify the vulnerabilities and how do you do that at scale? Today, it's a needle in the haystack problem. You've got a very small number of skilled red teamers who have some tools that are basically new and somewhat immature, I think in a lot of ways, some more mature than others, and they're just trying to find the problems. So I think that the prevention side is actually much harder right now. Intervention side, I'd say with guardrails is okay, needs to be better. But the prevention side has been a gap. And that's something that I've personally. And I've got a team that's worked a lot on that problem because we were doing red teaming and it was like, oh, this takes too long to get through all these requirements. And then people would want us to do blue teaming. And we're like, well, where do we start? Well, where do you start? You start with where your vulnerabilities are. So how do I figure out my vulnerabilities Faster, right? And so that's when I think about the different patterns today. Relying on third parties, then to guardrails, maybe a little bit of prompt engineering and then sort of best efforts on the vulnerability detection.
- [17:54]
B
What role should human monitoring play versus automated detection and identifying malicious prompt patterns?
- [18:02]
A
This is going to be really important. I think, like AI, as we've seen across the years, is actually better than humans at finding a lot of things, particularly pattern recognition, needle in the haystack stuff where you have just a lot of information. But we see this, I've got research teams and the things that we're doing is we are working with, you know, what you would have called classifiers, but using LLMs now for them, judges, we would call them in this case, and they're very good. But judge, all these judging systems that you're using are probabilistic. So what you're going to do is you're going to have certain things, you know, it's going to be, it's going to be important and you're going to intervene. If it's a guardrail, then you might be more aggressive at intervening because like certain risk you want to get rid of, but you're still going to have all these things which are sort of mid level, low level probability that are going to get through. You might even get some high probability things you need to look at and then you can start to put those things together. And I think this is one of the key things when we think about it from a human standpoint. One is you have to use the technology for what it's good at, which is identifying things, giving you probabilities, clustering for different things that are like. But then you still really benefit from a human taking a look at some of that. And this is one of the things I think that gets overlooked. A lot of people just want the AI to do the work for them or they want a human to do the work for them. Well, the human can't look at enough data to be consistent across humans to see enough to really understand all the patterns, signal and noise. We know that for a fact. And the AI does miss things as well because it also tends to look for certain types of things that may or may not be something you want to do. That's why we do reinforcement learning with human feedback. Right. So we're really big on this idea of AI elevating human capability and ingenuity. That's where we start with all the things that we build, the things we build for internal users, the thing we build for customers. And you know, that's the first thing we think that there is a lot of low hanging fruit about just helping humans be better, more consistent, have more reach. Adding automation on top of that then can take care of some things so they don't have to take care of them. But a lot of that comes back and you want the human in their loop either to verify it or to look at it and review it. This is a, I think it's an important question. I think you still need to look at logs. Anybody who's used AI, who's used other systems, like, come on, we got a lot of cybersecurity people here. If you don't look at the logs, you miss stuff.
- [20:38]
B
So it sounds like your philosophy is use the AI to make humans even more human and to get the unique capabilities that ingenuity I think is the word that you mentioned out of humans. Because they're not bogged down with the hunt, they're not bogged down with the fact that they have not seen the data or they've seen too much. And that's really interesting because I think on the flip side, humans are really good at pattern matching and eventually adversaries will figure out, here's the patterns that actually get past the AI, no matter how good it is, because it's new and novel. And that will be that moment where we do need to have those skills and those capabilities and that time to go in and look and say, why did this get through? And figure out how do we adjust, how do we continue to strengthen our defense?
- [21:32]
A
Yeah, that's a really good point. That systems are much more consistent than people. Therefore it's easier to identify patterns, that systems will or will not identify that then you can try to use that to the systems, to the disadvantage of whatever you're trying to attack. Whereas humans are much harder to predict what they're going to do. And that's a, I think as you were indicating, that might be a weakness, but it's really a strength very often.
- [22:04]
B
So let's shift gears a little bit and talk about cross platform AI risks. What governance frameworks should be established for managing data flow between different AI systems.
- [22:17]
A
This is actually an important thing for people to understand where we are now with generative AI and the enterprise and where we are rapidly heading. So today most of the use of generative AI is that open text box. It's doing transformation steps like rewriting that email, or it's doing a summarization of a document, or it's doing something like search. Right, okay, great. In most of those cases you are really limiting access or you're providing access to like public sources or something like that. What we're starting to see with like the emergence of new protocols like model context Protocol, mcp, if people are familiar with that, some of the direct integrations as well. The next thing that's coming is people are starting to experiment a lot with agents. And when you look at like model Context protocol, you look at agents, very often what they're trying to do is they're trying to figure out, based on what you've asked for, what is available that they could provide back to you to meet your need. And with the agents you're going to not just have like a user doing one or two tool calls at a time, you might have them calling two or three agents which are calling other agents, which are calling multiple systems simultaneously. So all those we want standardized security protocols, standardized communication protocols, all of these things that we've seen some movement on over the last six months. But what you've just done is you've created a lot more risk because you just have a lot more touch points, a lot more systems, any one of which could fail. And this is something that I think we should leave everybody with, or I want everyone to leave this with, is that you don't need to test the model. You can test the model, you need to test the system. Because it's not just the model, it's not just the provisioning cloud provider, it's all those other things you connect to it. And the more things you connect to it, the more problems you can have just with scalability production. Does it work consistently where the bugs exist, but you also introduce a lot more places where you're going to have risk from security or safety standpoint.
- [24:38]
B
Based on the experience that you have at Telus Digital and your work with few ix what emerging AI security threats should organizations be preparing for now that aren't really widely discussed?
- [24:53]
A
First, I think the things that people deal with today, I think you really have to think about information that's coming out of your system that is in violation of how you want your system to operate. So a violation of code of conduct. That's sort of the first thing. The next thing that's going to be, people are going to be dealing with more and more is going to be some of this data poisoning things where there's something going on in the data store, it gets put in there somehow and it's causing problems for you down the road. And parallel with that, but requires a little more sophistication. They should be looking at things which are what you think of as like model exfiltration, sort of trying to understand how the model works so that they can develop more and better attacks. When I move on from there, I say then the next things that I think are the biggest concerns is operational systems. Anytime you're hitting them just directly or you're hitting or you're allowing agents to have access to those, and then you have to differentiate between sort of human in the loop, human on the loop, or human out of the loop. But those are the ones that I think are going to cause the most consternation, rightfully so, but probably the most enthusiasm, because people like the idea and I like the idea of having bots out there just solving problems that I never even have to know about. Right. Because they're designed to do that and they have some autonomy and authority to do those types of things. But at the same time, every one of those, you could create a scenario where you just basically Have a bunch of super users that just happen to be bots. And that means you've got to test your system, which includes the agents. But you should be testing your agents independently as well. And there aren't really, I mean, there's, you can use some existing techniques, but there aren't really tools for that right now. So those are going to have to be developed as well.
- [26:47]
B
And so when you say test those agents independently, is that a outside party test them for you or is that.
- [26:57]
A
It could be a. Well, I think first, like your developers or your IT team that does the implementation, whatever they should be testing. So when you think about an agent based system, I think you should do unit testing on the agents as well as system testing in concert with the agents. So both of those things, because it might pass a unit testing, but you might have a problem in the system testing and vice versa. Right. So I think we all know that it could be that you do want a third party to do validation, depending on what you're doing. We do that at Telus. We have third party audits as well. So if you look at what Telus has done, and Telus has been, I guess, legitimately a pioneer in this, has won a number of awards, but developed a purple teaming approach where you've got end users, red teamers and blue teamers all working together, kind of a purple, purple team approach, running through a standard set of protocols which can reach into the hundreds each and then reviewing all the outputs and coming together. That's like the human policy standpoint. And that's the type of thing that is really important because mostly testers who are developers miss a lot of the outliers because they're not users. They don't think about how the users actually would come into a problem with bugs. But this is the same thing with lack of imagination in terms of how a malicious actor might go. So I think that there is a process for policy, for people, for developers, for third parties, and then for automated test suites that would do both unit testing and system testing through different configurations.
- [28:39]
B
You know, Brett, one of the things I've said for years is that our technology reflects our humanity and we build things in a way that reflects our culture norms, our expectations and how you're going to use whatever that technology is. And then when people don't use it in the way that we expect, I think that's where we run into security issues. Right. Like we expected that you would never do this and then people do and, you know, whatever, whatever the technique is and, or whatever the social engineering effect is. And then we're left with like, okay, how do we clean that up? And when you talked about this idea of like a user using something in a way that the tester didn't expect, you know, I just had shuddered and went back to my design days where I was like, oh yes, I put this together in a way that is so obvious. And then we'd watch a user use it and I'd go, oh no, we are so off. Like we had these assumptions. We're off. We gotta fix that. All right. Most important thing for a listener to remember from today's conversation, a couple things.
- [29:43]
A
I would say if you're thinking about AI security and AI safety and security, because they're both, it's different than you have to think about it with a framework which is different than what you have in the past where you had procedural systems, it was easy to trace. Not always easy, but more deterministic and tracing what's happening, what the sequence events is. So the first thing is you have to understand that this is different, that there is more risk because they're probabilistic systems, open ended on both ends. And you really should be doing test system testing, not just model testing, but system testing in order to identify vulnerabilities like extensive and use people, use users, use developers, use automated tools. But do that, use that in concert with intervention tools such as guardrails and you're going to be in pretty good shape to get started. And then as we move forward, just understanding that every time we introduce a new tool or agent or something like that, or new connection to third party, we might have existing standards and how we handle that, but we might need new tools in order to understand what the capabilities are, what the exploits are, because our expectations have enough imagination to understand how they might actually be compromised.
- [31:18]
B
Brett, thanks for a great conversation today. I really appreciate you sharing your insights on shadow IT and enterprise risk in general.
- [31:26]
A
Well, thank you very much. I appreciate the what you do for the industry and providing insights for so many thought leaders that I've learned from over the years and just exposing that so everybody doesn't have to learn the hard lessons themselves. They can learn from the people who have been there before them.
- [31:43]
B
You know, one thing that really struck me about today's conversation with Brett was his point that we can't just test AI models in isolation. We have to test the entire system system. It reminded me that in cybersecurity we often get so focused on the new shiny technology that we forget or skip the fundamentals. It's not that the individual component creates risk, it's how everything connects together. Brett put it perfectly when he said that people are going to use AI tools whether we want them to or not, because frankly, they're just better than what we had before. So instead of fighting that tide, we need to get ahead of it with smarter governance and testing frameworks. Because in the end, shadow, it isn't really about the shadow. It's about the light that we're not shining on our systems. That's it for today. If you like what you've heard, please subscribe wherever you listen and leave us a review on Apple Podcast or Spotify. Your reviews and and feedback really does help me understand what you want to hear about. I want to thank our executive producer, Michael Heller, our content and production teams, which include Kenny Miller, Joe Benecourt and Virginia Tran. Elliot Peltzman edits the show and mixes the audio. We'll be back next week. Until then, stay secure, stay vigilant. Goodbye for now.