Loading summary
Transcript40 lines
- [00:02]
A
You're listening to the Cyberwire Network, powered by N2K.
- [00:12]
B
Welcome to threatvector, the Palo Alto Networks podcast where we discuss pressing cybersecurity threats and resilience and uncover insights into the latest industry trends. I'm your host, David Moulton, Senior Director of thought leadership for unit 42.
- [00:25]
A
You shouldn't put any AI live in any enterprise use case without securing it first. AI has a ton of promise. AI can drive major transformation within a company, whether it's again, reducing operational costs through automating of processes, all the way through to co development, to improving customer experiences with better products. AI can do some pretty amazing things, however. It can go off the rails, it can generate malware, it can execute attacks that live within these artifacts. It can exfiltrate very important data. It can even lead to perhaps brand reputation if it doesn't have the right guardrails. Even though AI is so impactful, it can be so amazing in many different ways. We need to make sure that it's safe, that it's trusted and that it's secure, and that there really should not be any AI in any enterprise without security of AI.
- [01:36]
B
Today I'm speaking with Ian Swanson, AI Security Leader here at Palo Alto Networks. Ian is a founder of three Successful Exits, most recently as CEO of Protect AI. Before that he led AI and ML at Amazon Web Services and served as VP of Machine Learning at Oracle. Today we're going to talk about securing the AI supply chain, why it matters, where the risks are hiding and how leaders can take practical steps to close the gaps. Ian Swanson, so glad that you're here on threatvector. We've had a little bit of a, of a slow start here as we're getting started, but I'm expecting a great conversation with you today.
- [02:19]
A
Hey, thanks David. I really appreciate you having me on threat vector.
- [02:23]
B
Ian, you have this really interesting journey and I noticed something off mic when we were putting things together the way that you think in memos and you know, that alludes to your time at, at Amazon where you're leading AI there. Now you're at Palo, leading AI. Talk to me a little bit about that journey and how those different pieces string together to get you to this moment and maybe what's the secret of what's next?
- [02:51]
A
Yeah, so I've been in AI for roughly 20 years, so I've been pretty fortunate. Definitely AI is having its moment in the last three years, but I've had multiple companies that I helped start and was CEO of that ultimately became successful exits to American Express to Oracle and now Protect AI, which I started about four years ago as part of Palo Alto Networks. And the genesis for Protect AI was really when I was at Amazon, I was leading the worldwide AWS business for AI and I saw risks firsthand. And so we had over 80,000 customers AI at scale back when I was running that business. And I didn't see any cybersecurity companies focusing on the risks, specific risks that could be inherent in artificial intelligence. So I chose to start Protect AI to go after this kind of greenfield space. And it's been a phenomenal ride over the last four years. And this past July, we finished the acquisition where we become part of Palo Alto Networks. And it only took us two and a half months to fully integrate the Protect AI product set into what is now called Prisma Errors at Palo Alto Networks.
- [04:06]
B
Well, I'm glad that you went on that journey. And as somebody who is involved in a lot of the conversations in and around the risks to AI from AI, you know, how do we use AI? I think this will be a very clarifying conversation. And today we're going to get into AI supply chain. We're going to talk to you about your thoughts on what can go right and wrong when teams are too heavily reliant on things like vibe coding. So let's get into it. Let's go back to this concept of MLSecOps. And it's a concept that I think that you helped pioneer and maybe for the listeners that might be new to it, can you explain what MLSECOPS is and why it's so incredibly crucial right now?
- [04:51]
A
MLSECOPS stands for machine learning, Security Operations. It's a term that Protect AI helped coin about three and a half, four years ago. And it really was a play on DevSecOps, you know, as we think about secure design and shift left moments. And again, there wasn't any company, or I should say companies in general were not focused on the security of AI. And so we released frameworks around how you should think about securing AI all through the development lifecycle, from how you build AI to when AI is in applications and in production workloads and all that sat underneath this banner of ML SecOps. Again, how do we secure AI by design?
- [05:34]
B
I want to ask you something. A lot of times I'll get into these nerd conversations and somebody will say, like, oh, the AI supply chain. And you just kind of nod along, you're like, I kind of have an idea what that is. But before we get into some of the deeper questions, I Think it's important that we define exactly what we're referring to when we talk about something like the AI supply chain. And then if you could go a touch deeper and say, what component should CISOs or security leaders really be paying attention to in that AI supply chain?
- [06:04]
A
Yeah, no, it's a great question. As I look at the supply chain, you know, clearly data is the fuel to AI and machine learning. And there's been a lot of security around data for the last 10, 20 years. But what's something that is, is new, that we really talk to CISOs about at depth, is the machine learning models themselves. And so if data is the fuel, the machine learning model is the engine to an AI application. And there's a lot of these great foundational models that live in the open source environments. And so, you know, you could go to Hugging Face, which is the world's number one AI community, where there's over 2 million models that companies are able to pull in and train on their data sets and release. But there's all these various model repositories that have a really rich supply chain of building blocks that companies use as they are putting forth their AI applications. Now, what are the risks? So again, oftentimes when I meet with the ciso, I say, how many machine learning models do you have live? A common answer that I get is somewhere between 100, 150. The real answer is tens of thousands. And we have many customers that have hundreds of thousands of models that are live in production. And as we scan our team's devices, the network, the cloud, we also need to scan machine learning models for risk. We need to scan the engine that powers AI applications. And within that engine can be a lot of malicious code, unsafe operators, neural backdoors. And so that's one of the first areas that we tell companies to really look out for because they have this deployed and have had it deployed in production at quite a large scale.
- [07:43]
B
So as you're talking about that, you're basically saying that the perception is they have a couple hundred and reality is they have tens of thousands, maybe even more. And that lack of visibility is I suppose, a first problem. But then specifically, are there common AI or ML vulnerabilities that you see out there in the wild that companies are consuming today that really concern you?
- [08:09]
A
Yes. So I think there's multiple areas in the development lifecycle where there are hidden risks and important risks that CISOs need to pay attention to. As I said, if data is the fuel to AI, the engine is the machine learning model. We need to deserialize these models, look in them for risks. And we found real risks, that if you deploy these in for example, your cloud environment, it's going to try to steal credentials, it's going to try to exfiltrate data. But as those engines, those models go into AI applications, we should test drive these AI applications before we put them in production. What does that mean? Test, benchmark, evaluate red team these applications and models before you put them in production, at the point of inference, let's say in customer facing applications. So throughout this development lifecycle, we need to run continuous testing and find real threats. I'll give you an example of a threat we saw within the supply chain of open source models. We found a model pretending to be from a well known healthcare life sciences company. It was a name squatting attack. It wasn't the company that put that model live, but it was an attacker, a malicious actor. And that particular model we saw was downloaded tens and tens of thousands of times. If you put that model within your AWS infrastructure and at the point of deserialization, one of its core goals was to steal and exfiltrate your credentials on your cloud. And so we see a lot of attacks that 10, 20 years ago were just in the typical software supply chain that are remanifesting themselves within the AI supply chain, specifically around data models and now agents.
- [09:52]
B
And do you think that there is a tension between the securing and looking for those vulnerabilities and making some of those mistakes again and then the desire to move fast that's causing a lot of organizations to quickly skip the validating where did this come from? And just inserting it and moving quickly, what stops that human behavior?
- [10:17]
A
Yeah. So first off, I truly believe that there should be no AI in any enterprise without security of that AI. And if you take a look at the boardroom conversations over the last few years, the CEO is saying we're going to move faster with AI, we're going to reduce our operational costs, we're going to improve the customer experiences of our various products. But in that same boardroom, they're turning to the CISO and saying what are you doing to make sure that this is safe, that this is trusted and that this is secure? And so at that point they ride that fine balance, David, of how do they act as an enabler to all these innovative teams, but yet do it in a way that is again, safe, trusted and secure. And it's a healthy dialogue. And one of the most important things is this is truly a team sport of how we develop AI that drives true value, that is secure. We need to make sure that these teams are having a discussion, that they're both being educated on the risks, but also the opportunity of AI.
- [11:17]
B
So you mentioned a moment ago this idea of name squatting, and that's an old tactic and it's showing up in a new domain. Are there other attacks, other ways that threat actors are out there that you're seeing as real world campaigns, or is really some of this very theoretical and hypotheticals? Uh, I'm curious if you can illuminate that for our audience.
- [11:43]
A
Yep. So as I said, we, we see attacks that manifest within the supply chain, you know, within data, within models. But a lot of attacks are also happening at runtime. So let me explain runtime. Runtime is at the point of inference, it's inline. It's in the example of, let's say a chatbot. As somebody is communicating with this chatbot, they're giving it a prompt and they're getting a response. And in many cases we see malicious actors that are trying to fool these AI systems and manipulate them to leak sensitive data, leak personally identifiable information, bank information on that. And so this is at the point of runtime that we need to take a look at all the inputs, all the outputs, run it through many different security checks across security, but also safety concerns for brand reputation. So there's two places that attacks manifest at the point of. When you're building AI within the supply chain, it can be third party as well as first party threats. But also as you get it in production, you need to be have constant guardrails looking at all the inputs and outputs, including embeddings, code tool calls and agentic workflows. And you run that through a series of policies that make sure that we shield ourselves from attacks at the point of inference.
- [13:03]
B
So, Ian, I know you've talked about this idea of Vibe coding quite a bit. First off, what is Vibe coding? And then why is it so especially dangerous in AI development?
- [13:16]
A
Yeah, Vibe coding is a slang term basically for a process in tools where a developer is able to utilize AI for code generation. And there's a lot of amazing companies, startups as well as big cloud providers that have many different Vibe coding solutions. It really is a force multiplier for a development team. I see within our team 30% gains, you know, in terms of just the value and the velocity that that happens there. Now the challenge is how do we make sure again that the AI doesn't go off, you know, off the rails and introduce malicious content malicious URLs, that it's not as it's understanding what it is that it's going to build, that it's not reaching out to repositories that might have a, what's called indirect prompt injection attack. The bottom line is, as you are vibe coding with these solutions, you have AI on the side that is making plans, it's perceiving, it's executing steps, and you need to make sure that you have controls there, otherwise these processes can perhaps go rogue. And that's where we see that there are solutions for, again, runtime security checks, looking at all the code to make sure that we're not introducing anything malicious within our various environments.
- [14:33]
B
So as you were talking about that idea of 30% gains, was it 30%? Did I get that right? That's, that's incredible, right?
- [14:41]
A
It is.
- [14:42]
B
And, and you have this allure, Right. Like, I've got an idea, I want to go faster. I want to be able to build something quicker, maybe the, the blank page or the blank cursor. You know, that's a daunting space to. So you're able to build the boilerplates and some of the different pieces of code very, very quickly. And I could understand why you'd want to go fast, but how do you advise teams to strike the balance between the dangers of using an LLM to pulling in some of that malicious code or having an injection and also then still getting that value out of it so that it's fast and safe?
- [15:22]
A
Yep. So I think a lot of our development teams want to use these tools. And so it's really important that security enables the tools because of all the efficiency gains, you know, that I, you know, previously brought up. That being said, we do see that these engineering teams are cognizant of the fact that there could be introduced security concerns. So they're okay with security teams putting in some guardrails, runtime protections, analyzing all the code at the point of read and write on it. And they're okay with maybe adding 100 milliseconds of latency to be able to go through these various security checks again, because we're able to enable them with a tool that's incredibly powerful. Now, these tools. I'll give a quick analogy. If I gave a Formula one car to my daughter, who just learned how to drive as she's 16, she'll probably crash it in the wall. That car is a little bit too high performance. But if we give a Formula one car to a highly trained driver, they're going to just Smash it on the race course, they're going to be able to use that car to its fullest potential. I think the same way in Vibe coding, I see my senior distinguished engineers building incredible things, but also doing it in a safe and trusted way. But if I give it to maybe somebody a little bit more junior that doesn't understand the basics of security or the foundational building blocks of engine, sometimes things can get a little bit hairy there. And one way to be able to control it in an even and balanced way is to put in place security at the point of inference so that we can make sure whether you are a very senior distinguished engineer or junior engineer that just came out of the university system. We basically put a firewall in between all the interactions that you have with these Vibe coding solutions to make sure again that they're safe and secure. Sa.
- [17:44]
B
You know, a couple months ago I got to talk to our CIO Mira, and she talked about needing security to be a strong break so that she could go fast. And I think you're getting at the exact same point of you can go fast. You can have that Formula One. I don't recommend it for a 16 year old, but, you know, maybe your daughter really likes to drive, but you do need that strong break and sometimes it needs to kick in for you before you go. And you said, smash it out there. I know what you mean, but you don't want to, you don't want to wreck. Right. So I like the idea that there is this moment of here's the risk. As a business, you can accept the risk because the reward is so good, but we're going to put in mitigations and be thoughtful about where could things go wrong and how do we stop before there's a real problem. I think one of the ways that you can go and determine if those, those controls, those hypothetical controls that are going to stop things actually works is through red teaming. And I'm curious what your perception of where does red teaming play in securing an AI system?
- [18:48]
A
Yep. First off, red teaming is incredibly important. And so where does it live within the development lifecycle? Again, if data is the fuel and the engine to an AI application is the model, when we put that engine into the AI app, we need to be able to test drive it, we need to be able to red team it before we put it in production. And so it's incredibly important that before any application goes live, we need to test it. And so that testing, what is that? First off, it's continuous and it's integrated within the Developer tool sets that as various points of the lifecycle. As they're training these models, we need a red team as we put them in production, we need a red team it. And what we're doing is we're running a series of attacks. We have a couple different approaches to this at Palo Alto Networks. Number one is we have this static attack library that's tied to all these frameworks, nist, Mitre, Atlas and more. And it runs through these series of attacks and creates a scorecard. And that scorecard right now is being used by our customers as they make go no go decisions for these AI applications. They're using that scorecard as it starts to build policies that they're going to put in production to enforce guardrails around data leakage, how susceptible these applications might be to prompt injection attacks and stop that. So it's really important that we test and that we understand where the applications are vulnerable to one, improve on the application side, perhaps improve the model, but two, to inform these runtime policies so that we can de risk the application even further, mimicking it as if it was in a production environment.
- [20:32]
B
And I think it's important too to run those red teams because you've got that human ingenuity, that craftiness, and you're going to go in and say like, things don't stay the same over time. What didn't work before might work now. Something else comes up and you've got that continuous attack going on so that you've built your resilience and you've hardened things. Another, another area that has emerged as more and more tools have become available. Everyone in the enterprise, everyone in a business is this idea of shadow AI, right? And I can't imagine the difficulty of getting your arms around shadow AI as a leader. The tools are better than what we had before in whatever work we're doing, and we want those tools. And so sometimes you skirt the rules, right? You, you say, like, I'm going to go around, I'm going to use a different model and I'm, I'm going to skip the governance and not necessarily acknowledge that visibility is needed. How do you talk to security leaders and advise them on getting that under control?
- [21:36]
A
So that's really the first step, you know, which is discoverability. Discover all the assets that you have. And what makes this incredibly complicated in the AI space is that these assets can live in multiple different forms and places. And so let's think about an agentic workload. They can live on end devices, they can live on your laptop, they can Live in your infrastructure, on premises, in your cloud. And then you also have SaaS agents that live within Salesforce to ServiceNow and many of these SaaS solutions that are opening up to agentic workloads. So shadow AI takes form in many different places. And I like to break it down to two categories. Number one is employee usage of AI. So whether it's through the browser or other methods, we need to understand how we can govern what your team members are able to access, what generative AI solutions like perhaps ChatGPT, what are they allowed from a governance perspective to share with these solutions, but get complete visibility on the employee usage and all the applications that happen on that side. The other side is as you build, train, deploy models, AI applications, agentic workflows, we need to figure out where all those live and we need to bring that to light so that we have visibility. And then the next step is how do we audit and assess risk on all these assets.
- [23:03]
B
So that visibility piece that you're talking about is really critical. And I'm curious, Ian, if you were to say, like, what's the one first task or policy that you would recommend for that, that discovery, you know, and monitoring, where do you, where do you tell concerned leaders to go first?
- [23:25]
A
Yep. So again, I break this down in those two different categories. On the employee usage, it's, hey, let's, let's start figuring out from a browser perspective all the AI that's being consumed and used and catalog it. And then on AI that's being built and deployed, it's really going into your buckets, your object storage, trying to understand where are all these artifacts. And then once we get a handle of all the artifacts from there, we can start to assess what's the real risk, scan them if they're models, if it's applications, how do we red team them if there's agentic workloads? Let's start to understand the security profile of the identity, the permissions, the tools it has access to. And so again, two different areas. The employee side, start at the browser and then as you're building your own AI applications or perhaps using SaaS AI solutions, start to scan all of your infrastructure to see where that lives.
- [24:23]
B
And so I want to close the loop a little bit here on something you said earlier, where organizations think that they've got, you know, hundreds and they end up with tens of thousands. This is exactly what you're talking about, right?
- [24:33]
A
Yeah. So I was, I was meeting with the top 10 bank, you know, just in the past few months, and I asked the question, how many models do you think you have live? And they said, in the hundreds. And I said, message your teams, your AI team. So I was talking to the security team and they messaged, you know, their team, and their team came back and said, no, we actually have, you know, I think it was like 93,000 plus models. And that was enlightening to the security team. Security team was like, wait a minute, where are all these models? Who's building these models? What are third party assets and supply chain versus first party? And it started to really paint the picture for them of, wait a minute, maybe we don't have the visibility that we thought. And in some cases it requires different tools. And that's why we invented this category of secure AI and we brought it over to Palo Alto is to help customers understand the differences, but provide them unique tools and that can actually assess the risk and stop it as well at runtime.
- [25:32]
B
I have a question that is related to your role starting and building Protect AI. And now you're here. You know, you talked a little bit about the gaps that you saw in the industry. It drove you to start a company, and now you're inside of Palo Alto Networks, which has maybe, we'll say, a slightly bigger view of things, maybe more data. How has your thinking evolved now that you're part of the Palo Alto Networks enterprise?
- [25:58]
A
Yeah, so first off, the acquisition of Protect AI closed at the end of July 2025. Within two and a half months, we completely integrated all of Protect AI's offerings into what we call Prisma Airs at Palo Alto Networks incredibly fast. One of the ways we were able to do that is we had a lot of microservices within our architecture that we were able to kind of tie together, if you will, with Palo Alto's existing offerings. And what we were able to get was a truly better together scenario where Palo Alto had amazing capabilities in data loss prevention all the way through to detecting malicious URLs. Things that kind of added a lot more foundational capabilities to what we also brought to the table with Protect AI. And so one of the learnings, to answer your question was Palo Alto Networks was already doing a ton in this space and we were able to really provide our customers with this one plus one offering, Protect AI plus Palo Alto Networks that solves end to end AI security. I hear from some other people in the space that we have a platform for AI security, but they only work with AI security at the point of production. We go all the way back into how it's being developed. All the way through to it being in production. And we are able to do that from the power of all sorts of integrations we were able to get from Palo Alto Networks. We're working with the Cortex cloud team of how we pull in posture. We're able to share with these teams what we think about security of model artifacts as they see them in S3 buckets, perhaps. So there's all these capabilities we're able to bring under, you know, this common tent and platform that is parallel to networks to be able to secure AI end to end.
- [27:47]
B
So we're sitting here in early 2026. I'm wondering what you see as the biggest blind spot for security leaders when it comes to AI risk this year.
- [27:59]
A
So I'll say right now I meet with probably 7 to 10 CISOs every week, Chief Information Security officers. My broader team meets with many more than that. So we're having a lot of leadership discussions. And top of mind for all these CISOs are these agentic workloads and specifically where they start to lose control and visibility as it relates to building within software as a service offerings. So one of the first things that we did here at Palo Alto Networks as it relates to agent security was leverage all of our great partnerships to be able to build in native security solutions that can give our customers a single pane of glass of all the AI agents that are being worked on in SaaS environments. So think ServiceNow, Salesforce, Microsoft Foundry. We've developed native integrations into all that, which is incredibly powerful. And it's the thing in 2026 that is, I think, the biggest risk, which is agents, because we've given AI arms and legs to go carry out tasks. We need to make sure they don't go rogue.
- [29:07]
B
If there's one thing that a leader could do to improve their AI security this quarter, what would it be?
- [29:14]
A
It's a good question. I think it starts with education, and that education starts to drive that visibility of understanding all their AI assets and where they live. Why did I start with something as simple as education? I'll tell a story. I was on a call with a top five software company in the world probably a year and a half, two years ago, where this situation took place, where their research team was in one room, their security team was in a different room, zoom meeting, cameras pointing down, and we're talking about all the security risks. My team is with them and the research team goes off mute and they say, ian, we're going to leave early because none of this really pertains to Us, we are the AI research team. We don't put stuff in production. We do experiments. Doesn't apply to us. I didn't have to say a word. The application security team at their own company comes off mute and goes, whoa, wait, are you pulling in AI artifacts from third parties supply chain? Their answer was yes. They said, are you training it on our customer data? The answer was yes. Are you putting it in this particular cloud environment? Yes. And the security team came back and said, hang on, here you are pulling in grenades and pulling pins and you don't even know it. But I would also say the security team didn't know it. Right. And so what needs to first start with is this team approach of let's understand the gaps that we have. The AI teams might not understand security risks. The security teams might be blind to all the AI that's already in development and how it's being developed within companies. So even though it's really simple, I think we need to start with internally at a company. Let's catalog and let's understand all the AI that's being built. And that needs to happen through conversation across all these teams.
- [31:06]
B
Ian? I like that. And it reminds me of a conversation I had with Noah Russell last spring, and she talked about this idea of a baby tiger. And that research team, you know, they're just building something. It's a cute little tiger. We're not sure what it's going to do, but it suddenly grows and it grows quickly and it still has its claws, it still has its teeth. As you said, it's a grenade with pinpole. Right? These things get big fast and all of a sudden you're left with a menace. You're left with a lot of risk. And if you're not communicating that well, if you're not aware of what's going on or aware of the risk that you're accepting when you bring in code from somewhere else and put it in your own systems, trained on your own data, that's a, you know, sure, maybe they didn't know, but it's a wildly risky behavior. And I like the idea that, you know, transparency and a lot of good conversations allow for a company to understand what they're doing or any team to really understand what they're trying to secure. Ian, thanks for this awesome conversation today. I really appreciate you taking the time to hop on threat vector and, and share your insights around the AI supply chain and some of these recommendations on education and specific behaviors that, that you're seeing out there that are risky and how leaders can reduce their risk around AI.
- [32:21]
A
Thanks for having me. David.
- [32:31]
B
Foreign if you like what you heard today, please subscribe wherever you listen and leave us a review on Apple Podcast or Spotify. Your reviews and feedback really do help me understand what you want to hear about. I want to thank our executive producer Michael Heller. Our content and production teams, which include Kenny Miller, Joe Binocourt and Virginia Tran. Original music and mix by Elliot Peltzman. We'll be back next week. Until then, stay secure, stay vigilant. Goodbye for now. Sa.