← Reshaping Workflows with Dell Pro Precision and NVIDIA RTX PRO GPUs
Reshaping Workflows with Dell Pro Precision and NVIDIA RTX PRO GPUs
Tackling Security Risks in Modern AI
00:34:31
Learn how to secure, govern, and take control of enterprise AI systems. Discover overlooked AI risks and solutions from Prediction Guard’s CEO.
Loading summary
Transcript28 lines
- [00:00]
A
Foreign. Welcome to Reshaping Workflows with Dell Pro Precision and Nvidia, where innovation meets real world impact in high performance computing.
- [00:20]
B
Welcome back to another episode of Reshaping Workflows with Velcro Precision and Nvidia RTX 4 Pro GPUs. I'm your host, Logan Moller. So got a great episode on the docket today. We are talking to someone new that we've never talked to before and going to get all into kind of how to protect from an AI perspective, how to kind of enforce security across sovereign AI systems. So with that being said, I have Daniel here with us today. So Daniel, take a second, introduce yourself. Elevator pitch. Where are you from, where you've been? And then we'll jump right into it.
- [00:55]
A
Yeah, definitely. So great to be here. Thank you for the invite. Daniel Whitenack I'm CEO at a company called Prediction Guard and my background just personally is originally in physics. Did a PhD at Purdue and then went into industry after that kind of at the start of the data science era, kind of 2011, 2012 and have been in data science, AI machine learning since then that time have been. I founded Prediction Guard three years ago. What we provide is an AI control plane for those that are deploying kind of building agents or AI driven applications that they intend to deploy in high trust environments. So think about whether that's, you know, aerospace and defense manufacturing on a factory floor or in regulated industries where you're processing very sensitive data. So that could be anything from air gapped on prem to your cloud, vpc, wherever that high trust environment is. We allow you to ship an AI control plane into that environment which enforces governance policies, integrates with your monitoring stack and allows you to track supply chain risk.
- [02:11]
B
Okay, so you kind of. Well one, this is why I love this because I get to speak to people that are way smarter than me because I definitely don't have my PhD. So let's start. So Prediction Guard, you said you founded it, when did you say you founded the company?
- [02:26]
A
January of 2023. So it's been a few years now.
- [02:29]
B
It's been a couple of years. Okay, so because you hear this word sovereign AI, it's coming up quite a bit more maybe let's say start with how do you define sovereign AI? And then I always feel like it's one of those things where one of those terms that can mean a lot of different things. What do you think it means to, to you and you know, the context of your business and then just as an industry term, overall yeah, yeah.
- [02:54]
A
There's a whole range of semantic meaning here for this term. And you know, like a lot of things in AI, there's a lot of confusing jargon. What we mean by the term sovereign is that we want our customers to have control of what we call the control plane, which is where they're enforcing this AI governance or security, you know, how they're handling, monitoring, et cetera. So a good parallel would be recently in the news we heard about the light LLM hack, which you may or may not have followed. But kind of what we learned from that is if you have a thing which is outside of your security boundary, meaning outside of your network, outside of your operating environment, and you kind of centralize or outsource AI governance in into that place, it's kind of like a big target on your back, right? Because it's sitting outside of your operating environment, your network, your firewall, et cetera, and your traffic is necessarily routed through there. So with Prediction Guard, what we allow customers to do is they actually deploy our control plane into their environment. Like I say, could be on prem in a data center, air gapped at the edge in your cloud, vpc, wherever that high trust environment is. And we allow you to have an easy interface to configure that. But ultimately it lives in your environment and you're able to set the governance rules, policies, et cetera, which if you look at some of the stuff with for example, Anthropic and the federal government, you have these things happen where all of a sudden a certain vendor is not applying governance in a way that you need it to operate and thus is all of a sudden a supply chain risk. And again, if you're managing your governance and your policies, how those are enforced, before anything leaves your operating environment, or even in an air gapped environment where nothing leaves that environment, then that is under your sovereign control. So that's kind of ultimately what we mean by that sovereign is you are now not just a consumer of AI functionality, but you're actually an operator. You're in the operator seat. You manage that kind of the control of that system. So tied very much to control.
- [05:17]
B
Okay, yeah, that makes total sense. So let's take it back. Let me ask kind of a simple question, right? Is that I think, you know, AI, when it kind of came to the forefront. Let's just use OpenAI as an example, right. Is that, you know, everything was kind of cloud based, not necessarily on prem. Right. Obviously I think that's changed and people are kind of shifting more, you know, to local Compute whether that's in a data center or whether that is in, you know, a kind of an edge device or wherever it may be. Right, but what percentage of customers, maybe not customers, but folks in the industry, business in the industry still are outsourcing kind of the security to, you know, don't have to name anthropic, but not necessarily, but any model provider or any, you know, any AI company. What I'm curious the percentage on that.
- [06:06]
A
Yeah. Well, I was recently at RSA out in San Francisco and I would say that definitely our solution kind of that main differentiation was this architecture that I just described. There's a majority, I would say. What I would say the majority is, I don't know an exact percentage, but I would say the majority right now in terms of their maturity are either kind of making assumptions about what their external AI vendor is doing governance wise, maybe they're locking into that which like I say, has its own risk associated with it, or they're potentially being exposed to these SaaS offerings or open source offerings that again might operate outside of their security boundary and not kind of fit the environments that we're talking about. And so I think that would be, I guess, the majority. And you know, as things happen like, like I mentioned, like the light LLM hack or other things, I think people are realizing that hey, if, if I'm just outsourcing that vendor, that, that security to external vendors, then I'm kind of again just consuming their take on what governance means. And I think it, as we see kind of right now, the forcing function around doing things in a more sovereign way might be those industries that are necessarily constrained environment wise, those operating at the edge or in air gapped environments or in defense or other environments like that. But I think what we're seeing, if you look at things like the frameworks that are being developed, nist, AI Risk Management framework, what's coming out of owasp, ieee, the AI underwriting company and the certification process that they're doing. We're going to shift very soon, I think, into a necessity of not just saying, hey, trust me or trust my vendors that I'm doing AI governance correctly, but I actually want to export evidence that I have alignment with these frameworks and these standards in a similar way to, you know, you do SOC 2 auditing, right? And you have to export evidence of that. I think right now the forcing function for that sort of rigor is in these higher trust environments or industries. But as we move along, that's going to be more and more kind of reaching these other industries where they'll be kind of need to be exportable proof or evidence of alignment with good AI, security and governance policies.
- [08:48]
B
It makes sense. Right? So you said, you know, prediction guard 2023, obviously OpenAI kind of hit a little bit before that. I'm curious, was there like a specific moment or like that, you know, watershed moment where you were like, wow, Prediction Guard needs to exist. What was that? Like light bulb where you were like, this needs to happen.
- [09:07]
A
It's interesting. I. There's kind of a couple ones, I guess, which is not, not an uncommon story for startups, I guess, where you kind of have these epiphanies as the market develops. But you know, in that initial phase, you know, I had been working for years and years deploying models, AI systems into real world enterprise environments. Everything from, you know, data centers to laptops to, you know, cloud environments, the whole thing. And I realized as models were being released and people were interacting with AI, integrating it into their systems, there's going to be a whole host of enterprise realities that aren't being accounted for. So that was the initial motivation. I think in those first couple years we focused a lot on the privacy element, much less the security and governance kind of control element, because we were kind of in a phase where people were figuring out, do I want to send traffic to, you know, like you were saying earlier, different cloud vendors of models and what we've learned over time, and especially I think the most recent epiphany kind of this last year was we're moving into this age where the model is not the primary thing. Right. Actually, the model is a piece of a wider puzzle that is the agent and maybe agents plural, and each of these agents has a harness around it, which is actually a distributed system of things. And what I mean by distributed system is you do have models, you have embedding models and LLMs and vision models and re rankers and then you have things like MCP server connections to your salesforce or to your netsuite or to a local database. You have Maybe connections to APIs or other tools within your organization. And so there's this distributed system of things that is driving the agentic workflow. And so really kind of the epiphany that we had last year I think was that no one is really thinking about, as we move to this world of physical AI and AI, like you say, you were saying, being embedded in more on prem environments or local environments. Who's talking about how you ship an AI system like this, put it in that Sort of operational edge environment and ship with it governance configuration that's actually enforced in that environment. And because we had led kind of in the previous two years with a very infrastructure ops focus because we were deploying a lot of models in a private way across all of these environments, you know, multi cloud on prem air gapped, we were already set up infrastructure wise to really address that problem. And so we've kind of gone all in on this. Hey, you know, you can ship an AI system, governance included, into any environment from a single node, you know, heavy client in a manufacturing floor to your cloud, vpc, whatever that kind of operational edge looks like for you.
- [12:22]
B
I agree. Like, I mean and I think every good business kind of starts from you know, that moment. Right. And I will say is, you know, I've gone to GTC, you know, first year was 2025 and then this year and I definitely saw a lot more, you know, on the security front. Right. Like I, you know, did a couple interviews with companies that are doing, you know, and I won't name names but you know, encrypted. Right. Because a lot of times when you're looking at use security and like, you know, when it comes to AI it's very like kind of prompt focused, you know what I mean? It's kind of like looking at kind of the outputs or the inputs and the prompts. Hey, let's you know, encrypt those, let's do whatever to try to focus on that. But it seems, you know, after doing my research before that you're focusing on more of like a system level governance. Can you kind of talk to me what that would mean for someone who you know is maybe an IT that is their company saying hey, we want to deploy an enterprise knowledge base. What does that mean to you as a system level governance versus like just kind of focusing on encrypting of prompts, inputs, outputs, tokens, etc.
- [13:31]
A
Yeah, I think there's two elements of this. One would be kind of a shift from, and I guess this is a realization of the actual Personas involved. Right? Because if you think about that situation where you're deploying a rag based system or automation or whatever it is into your environment and we assume that's a high trust environment, then there's a developer side of that which is concerned with the business logic and how well that operates. And then there should be a concern around like the security side of that and governance and policy enforcement and, and the reality is that at least what we found in all practicality is you want to have a control plane through which the security governance, policy enforcement can be applied and then make the developer side as easy as possible. So they just know they're using an API with access to great models and with all the right functionality and you know, the right connections to systems and all of those things. And, and they don't have to worry about, hey, do I need to put in PII masking for this application or how should I handle prompt injection logging and that sort of thing and where should it go? You know, they're not doing that plumbing of things to, you know, maybe Splunk and other systems generally. And so what we want to do is first apply the governance and enforce that at the, the control plane level and make the developer experience as easy as possible so they don't have to feel the burden of getting all the right governance in place. The second then would be making sure that that governance enforcement is across this kind of multi layered system. And so in the case that you're talking about, like let's say it's the rag case, you would probably have multiple models like an LLM and, or an language vision model, potentially a re ranker, an embedding model. You would have a vector database, you would have potentially, you know, web search or something like that as a tool if you're enabling that. And so you've already kind of, you're already stacking up five, six, seven different things. And so the way that we would approach that is you would register all those things into what we call an AI system. And, and then from the admin side you would enforce the proper governance policies onto that system, which might be the prompt injection, how you want to handle prompt injections. It might be what you need to require for authentication on MCP servers, how you're handling toxicity out of the models, whatever those policies are. Then the developer who's developing the RAG system, they just get an OpenAI or Messages or Responses compatible API regardless of what models they're using, they use it. And let's say a prompt injection comes into that. They don't have to worry about hey, is that going to be detected? That's taken care of at the control plane level. As the input comes in that is then detected and either blocked or logged, sent to an integration to your monitoring alerting stack. So all of that's tied in on the back end side to your security infrastructure. And then the developer just gets to do the cool stuff that they, that they want to do.
- [16:59]
B
Okay, so it really sounds like, you know, and I like to Think about any sort of AI system or applications. There's really two sides of the fence, right? There is the traditional kind of IT management, governance, that side, then you have the developer side, right? And at the end of the day those sites don't always see eye to eye. It sounds like you kind of have a solution for that. You know, another kind of question to tie into that, right, Is let's say, you know, you're running, you're a customer, you know, ABC doesn't matter and you're running Prediction Guard. And we kind of expand upon that example a little bit. Let's say you're like, hey, I'm interested in running in a corporate environment, open claw, but I also want to do the rag like how fine tuned can kind of that control layer be for different applications? And then like are. And is it really set by one central IT team or is there, you know, I'm just curious like the level of flexibility because I can see, depending on how you answer, I could see it really loving it. And then the developers being like, no, they shut down. I can't even call out to anything. This is ridiculous.
- [18:05]
A
Yeah, so there, there's obviously kind of always this tension between the best in class functionality and what you can do in secure environments. And this, you know how we think about this at least is we think about this in our product by letting you manage systems, plural, not singular. And that could be because you know, you want. Let's say you're a software vendor and you are building an AI feature into your software and that's going to be external, customer facing and you're going to need to answer infosec questions about what you're doing in that product and you want to have that extremely locked down. You may create an AI system with A1 governance, you know, configuration or policy in place that applies to those public systems or system. And then you might have another system which is your development system where you're trying new models or there's more flexibility, but it's not what is powering your kind of live production features, for example. So there's a variety of ways and each of these, the way it works in our architecture is that each of these would have its own control plane which is just a set of software services, very lightweight, running on CPU that lives in a single tenant way in your environment that you're calling out to. So that developer environment would have its own single tenant AI system with the proper governance. The production one would have its own AI system with a different set of governance and you could imagine this kind of again as we get further into the physical AI world. Oh, I'm going to deploy 100 systems into retail locations or I have five different, you know, manufacturing facilities each that are manufacturing different levels of, you know, defense related things that have different requirements or regulation. Or maybe you're an MSP or you're a service provider and you're doing services work for you know, one set of customers in manufacturing, one set of customers in financial services, one set of customers in a different industry. You could have AI systems that you're building off of with specific, you know, enforced governance for each of those scenarios in a single tenant way.
- [20:35]
B
So not to going in this next question, not to necessarily scare people but just to bring in kind of the reality of the situation. Right. And you don't have to name names or share any proprietary stuff but you know, you get invited in. I'm assuming there's some sort of proof of concept, there is some sort of audit that you will do and you know, maybe before when they're interested in procuring, you know, a company's interest in procuring. Prediction guard. But what are some donate names but like some things you've seen that are scary that people need to be aware of. Right? Because I feel like it's one of those things, it's like you never buy. You know, I was just in Germany, right for a conference doing some speaking stuff and you know when I was at the car rental counter is they asked me for the insurance. I did it because you know, Dell paid for it, it was 10 bucks. Right. But if it was personal I wouldn't have spent that money until you have an incident. Right. And I think having people know in this podcast that are listening are some of the real risks that they could face or maybe they're not even aware of is a good thing to share. Right. So maybe go through, you know, couple that you've personally experienced or seen that maybe affect quite a few customers that they're not even aware of.
- [21:48]
A
Yeah, I think there's a couple of these that fit into even things, you know, I, I can share about that are public in the news that you know, there's kind of a set of things that's the more malicious and you know, the, the scary things from the malicious standpoint. And then there's things that are like you're talking about that are maybe things people don't even realize. I think there is general confusion around this difference between like AI assets, models and products. And so when I use a productized system from an AI vendor that has a model in it, right? But it is subject to, it's essentially a SaaS product, right, which has its own T's and C's, right? And one thing that, that came out in the news recently is where a court judge actually ruled that an individual's chat logs from, I believe it was Claude, had to be introduced into court because even though he was talking about like his legal situation and all of that, there's no attorney client privilege with a tool, right? And so that was discoverable, right. Another example would be I often hear our customers say I'm going to like accumulate all this output that I'm getting from these AI systems and that's like my new ip. But it's not only that external vendors could potentially train models on your data, but what you're getting out of their systems, right, Is subject to their T's and C's and you can't do certain things with them legally. Right? So you, you can't train your own models in many cases and, and that sort of thing. So it, it creates these gray IP areas on the more malicious side. What I would say we've seen and what kind of concerns me is people talk a lot about these maybe direct attacks that seem kind of almost childish, like ignore your instructions and do this bad thing, right? Someone puts that into one of these systems. I think the, the deeper problem is that now that we're in this distributed system world and we're connecting agents that can take actions to our enterprise systems, email, our file store, our databases, our erp, whatever that is, right? There's these indirect things that can happen, right? Like I can get an email and if I have automation built off of my email and that email has an attachment, it's a PDF. I look at the PDF, I just see a bunch of words, right, about something. And you know, I don't see that there's a whited out big prompt injection in that, in that PDF, right? But the problem is now that's plugged into an automation that actually automates something off of that, off of that email chain, right? Or I'm processing 10,000 documents out of this file store and that's completely private, right? But there could be all sorts of things in, in those files. And so it's these kind of indirect things and things that result from distributed systems that take actions across your enterprise that I think increasingly make this complicated and difficult to think about in the absence of an appropriate kind of governance harness.
- [25:18]
B
You know, it's interesting, right? Like I Mean, and I'll show a little bit of my lack of knowledge here is that, you know, I thought we were kind of beyond the days of, I'm not going to say jailbreak, but like, you know, trying to break an LLM or trying to break a system, but it sounds like we're not. So I mean, I found that really interesting. But yeah, I mean it is something like that. At the end of the day, you know, companies need to be aware of and I feel like there's always been a lot of focus on it and I think kind of a follow up question to that is, you know, a lot of times, you know, in AI adoption, I think we're just kind of getting there, right? Like, you know, there is still a long ramp and a long way to go. And I mean, where do you think from your perspective and background, you know, and I know the answer, but I'm going to ask it anyways is, you know, you're building a system, where should security come in that chain, right? Is that, you know, is it the first thing you should think about? Is it, hey, let's bolt it on at the end? Like where is the ideal part or ideal time to really think about security? Because I do feel sometimes companies get wrapped up in the security piece and ultimately that will derail the project.
- [26:25]
A
Yeah, I think that, and maybe I could use a bit of a metaphor here. So I, let's say that I want to be a healthy individual. Is it bad for me to, let's say, take my temperature, do a temperature check? Well, no, that's a, that's a good thing, right? Like that's a good data point. It's a good isolated point check, right. That I, you know, would indicate something related to my health that is very different than me as an individual being plugged into a healthcare system where I have, you know, healthcare records. I'm part of a system that's governed by policies and has experts involved and is giving me feedback about a whole spectrum of things related to my health. Right. And so ultimately if I want to be healthy, I need to be part of that sort of system with policies and governances associated with it. And so I think a lot of what's going on right now, if you look at some of these kind of individual model endpoints, is there are point solutions and point checks, like is it good to have a guardrail for prompt injection in front of your model? You know, it's not, it's not bad. But would I say that that like helps you really in the grand scheme of things, not a ton, because what really kind of matters is how you enforce governance systematically and do it in a way to your point that doesn't block development and time to value. And the way to do that is, I think, not try to do AI security and governance and forget all the intuition we have from security generally, but to say, hey, we already have these systems that help us monitor an alert, right? So if I had a control plane like a prediction guard that would integrate with my monitoring and alerting stack without blocking a very convenient, easy to use API for my developer, then I could know if, for example, over time there were 80% prompt injections coming into this API key in the past five minutes. And you know, I need to rotate that API key, but I could do that at the governance enforcement level without affecting the developer and the API that they're using. Right? And so I think that that's really what we're trying to do because ultimately I'm a, I love building things. That's my background. And so I want to see our team use more AI. I want to see our customers use more AI. This is extremely transformative and there's a way to get there without kind of treating security as an, as an afterthought, but kind of starting with something that provides that ability for governance, but again, doesn't block the developer, to your point.
- [29:21]
B
So we're getting kind of towards the end of the episode, a couple final questions here. You know, what is kind of one AI security risk that, whether it or developers need to be aware of, or maybe it's just starting to budget, that's going to become something they need to prepare for in say the next 12 to 24 months. That's not getting enough attention in your opinion today?
- [29:42]
A
I think generally like security best practices around integration and management of MCP servers and tools is a big one because again, people put a lot of focus on the model. And to your point, oh, I can jailbreak this model or do this thing? Well, it doesn't really matter if you protect that part. If your output of your model is then connecting to insecure tool calls and making destructive changes in tools and systems that are connected through via mcp, which is model context protocol for, for those that aren't familiar. So I think the real distributed nature of this problem is going to hit people really hard when they, when they start realizing that. And it's partially tied to mcp, but partially just the general distribution of these, of these tools. But I recommend if people want to think about this, OWASP does some great work in their Genai project around agentic risk and MCP servers. And you can look at that and kind of understand a bit about kind of the range of things that you may need to be thinking about in the coming years. The other one I guess I would mention is some of this is already happening in Europe. I think it's on its way here, especially from insurers and industry regulators and auditors is again, that ability to export proof of your AI governance is much different than saying, hey, trust me, my developer seems smart and they probably put in the right thing. So I think that of course elevates this discussion and could block if you don't have the right system in place to allow you to quickly export that. That evidence.
- [31:31]
B
So, final question before we wrap up the episode. You know, what do you think if there's one thing that a listener should take away from this episode, maybe a mindset shift about AI security or Prediction Guard, what would be that takeaway or thing they would need to, that you would want them to walk away and know and be able to remember, you know, two or three senses. What would it be?
- [31:54]
A
Yeah, I think that would be related to thinking about how much control and ownership you have over the, the distributed set of AI things that's coming into. Coming into your environment. So just being kind of approaching that with radical honesty of, you know, where is, where is governance being enforced? What are the set of things? How much control do I have over these set of things? That I think is a. Is a great question and a great place to start, which reveals kind of a lot of downstream things that people need to think about.
- [32:34]
B
Okay, perfect. So with that, Daniel, really appreciate you taking the time. Take a second. Tell everyone, you know, where they can. The name of your company again. Prediction Guard, obviously, but the website, the URL where they can connect with you and we'll go ahead and close it down.
- [32:47]
A
Yeah. So people can find a little bit more information about what we're doing@prictionsguard.com also I co host a separate podcast called Practical AI which you can. If you just search for Practical AI podcast on any podcast platform, then then you'll be able to find that which has been really fun to, to host over the, over the years. So we'd love to. Would love people to connect either way there and reach out for, you know, a call schedule something on our, on our website. We're, we're thinking about these things very deeply and would love to even if it's just being a good sounding board in this kind of noisy, frothy environment. We would love to be that for you.
- [33:30]
B
I love it. Well, Daniel, really appreciate you taking the time. So, once again, another fantastic episode on reshaping workflows. So moral of the story, takeaway. Whether you're, you know, heaven forbid, in the cloud, but you're in the data center, you're the edge, wherever you have an AI system, security should be paramount. And don't let the illusion of security or the illusion of what you think you're getting actually take away from the reality of what you're getting. And definitely check out predictionduard.com and with that, I'm Logan, and we'll see you on the next one. Do what you want. Do what you want.
- [34:21]
A
Do what you want.
- [34:22]
B
This podcast was produced in partnership with Amaze Media Labs.