Shadow LLMs, Agentic Identities, and Securely Integrating AI

Summary

The Defender's Advantage Podcast Episode: Shadow LLMs, Agentic Identities, and Securely Integrating AI Date: July 22, 2026 Host: Luke McNamara (Google Threat Intelligence Group) Guest: Mohamed Manir (Technical Manager, Incident Response, Mandiant at Google Cloud)

---

Episode Overview

This episode explores the emerging risks and challenges enterprises face when adopting artificial intelligence (AI) and large language models (LLMs). Host Luke McNamara and Guest Mohamed Manir dive deep into topics like “shadow AI,” identity management for AI agents, supply chain threats, governance frameworks, and the “living off the land” abuse of LLM-enabled tools by threat actors. The conversation offers practical insights for CISOs and security leaders striving to safely harness AI while bracing against evolving attack vectors and internal pressures like cost control and decentralized development.

---

Key Discussion Points & Insights

1. AI Adoption in Enterprises: Beyond External Threats

[03:05–05:46]

• Challenge: Organizations are rapidly adopting AI but often without a strong, overarching governance framework.
• Governance Distinction:
  • For AI: The overall organizational governance structure and policies.
  • Of AI: Ensuring that individual AI deployments adhere to the set framework.
• Shadow AI: When business units develop or use AI tools outside centralized policies, often increasing risk through unchecked integrations with open source models or third parties.
• Legacy Security Debt: Weak identity management or vulnerability controls compound risk as AI systems are introduced, leading to higher probability of data leakage or misuse.

Quote:
“If you don’t have the [AI governance] framework defined, then it’s pretty much the wild wild west.”
— Mohamed Manir [04:23]

---

2. Misconfigurations & Identity Risks

[06:52–09:33]

• Agentic Identity Issues: Agents (AI systems performing tasks) often use user accounts instead of distinct agent identities, making it near impossible to distinguish between human and AI actions—a critical investigative challenge.
• Living Off the Land: Threat actors/malware abuse legitimate LLM-enabled CLI tools already present in the environment to evade traditional detection.

Quote:
“Users that have privileged access, the agents are now using those same identities. How do you distinguish between a human using that account versus an agent using that account?”
— Mohamed Manir [00:02, 06:55]

---

3. Threat Actor Tactics & LLMs as Attack Tools

[09:33–11:03]

• Reconnaissance and Blind Spots: Threat actors leverage agents with broad permissions to access sensitive cross-departmental data, exploiting misconfigurations in access controls.
• Dwell in Plain Sight: Malicious actors leverage LLMs and agents to blend in with legitimate operations, increasing dwell time.

Quote:
“Role based access controls have been backdoored using these agents ... by virtue of having access to that agent, they now have access to the data.”
— Mohamed Manir [10:33]

---

4. Supply Chain Vulnerabilities in AI/LLM Usage

[11:54–17:01]

• Open Source Model Risks: Blind consumption of open source LLMs exposes organizations to backdoored or tampered models—often with no verification like hash checking.
• Software Supply Chain: Vulnerabilities in transformers, packages, or dependencies often go unchecked, multiplying risk through tools like coding assistants.
• Lack of Bill of Materials: Organizations rarely track the full software/components stack for their AI deployments, making audits and remediation difficult.
• Data Lineage Challenges: Unclear or undisclosed model training data raises compliance and ethical risks.
• Distillation Attacks: Attackers use high-volume prompts to extract sensitive model details (weights, training logic).

Quote:
“Not extending your vulnerability management practices into this pipeline is another area that we’re seeing deficiencies around.”
— Mohamed Manir [12:17]

---

5. Insider Threat & Overprovisioned Agents

[15:50–17:01]

• LLMs and agents change the landscape for insider threat, enabling access to data and tools through poorly governed and overprivileged agents.
• The ability to write code or perform unauthorized actions with AI coding assistants heightens risk.

---

6. Shadow AI and Shadow Governance: Definitions & Organizational Blind Spots

[21:19–26:25]

• Shadow AI: Evolving definition. Initially, AI outside enterprise licensing; now, any AI in use without centralized security review/logging, even if licensed.
• Shadow Governance: Emerges when departments independently set parameters, either due to a lack of communication or absent policy, creating fragmentation and inconsistent security postures.
• Cost control pressures lead some organizations to encourage “bring your own LLM,” increasing Shadow AI risks.

Quote:
“Anything that we don’t have logging for, visibility on, is shadow AI.”
— Mohamed Manir [24:41]

---

7. Identity & Secrets Management for Agents

[26:25–30:03]

• Inadequate onboarding/offboarding of agent identities leaves orphaned and highly privileged accounts—a major security risk.
• Mixing agent credentials with human accounts blurs audit trails and response ability.
• API keys for LLM endpoints have been stolen, incurring millions in unexpected costs before detection.

Quote:
“We have run into situations where API keys have been stolen for LLM endpoints and the organization only realized...when they got the bill...a few million dollars.”
— Mohamed Manir [28:25]

---

8. Recommendations & Common Pitfalls

[30:48–33:47]

• Address Security Debt: Honestly evaluate and tackle legacy weaknesses (identity, vulnerability, data governance, supply chain).
• Clear Ownership: Define both who owns AI, and who owns AI risk in the organization.
• Bridge Compliance and Use Cases: Connect compliance requirements directly to AI implementations via operational governance.
• Define and Mitigate Shadow AI: Know what “shadow AI” means for your org and curtail unsanctioned usage.
• Visibility & Telemetry: Prioritize meaningful logging/telemetry for explainability and transparency, tailored to each AI use case.
• Open Source Caution: Treat use of open source LLMs with rigor—validate, track provenance, and mitigate risks.

Quote:
“Who owns AI within the organization?... Who owns the risk for AI at the organization? No hands go up. You need to get ahead of that.”
— Mohamed Manir [32:14]

---

9. Looking Forward

[33:47–35:06]

• Many of the technical and organizational risks with AI are evolutions of prior security challenges, not fundamentally new. However, attack surfaces, controls, and the pace of innovation are accelerating.
• Subsequent episodes may witness evolving risks as adoption patterns change, but foundational security themes persist.

Quote:
“I don’t think the security challenges would change necessarily, but how we’re seeing it, I think that’s going to continue to evolve for sure.”
— Mohamed Manir [34:52]

---

Notable Quotes & Moments

• “If you don't have the [AI governance] framework defined, then it's pretty much the wild wild west.” — Mohamed Manir [04:23]
• “Bring your own LLM as something that is part of the organization's approach...” — Mohamed Manir [22:54]
• “Anything that we don't have logging for, visibility on, is shadow AI.” — Mohamed Manir [24:41]
• “Who owns the risk for AI at the organization? No hands go up.” — Mohamed Manir [32:14]
• “We have run into situations where API keys have been stolen ... the organization only realized...when they got the bill… a few million dollars.” — Mohamed Manir [28:25]

---

Timestamps of Important Segments

• 00:02–01:07 Introduction to agentic identity risks and LLM-enabled tool abuse by malware.
• 03:05–05:46 Foundational governance risks and introduction to shadow AI and shadow governance.
• 06:52–09:33 Real-world misconfigurations: agent vs. user identities and incident insights.
• 09:33–11:03 Why threat actors are targeting enterprise AI agents.
• 11:54–17:01 Supply chain threats, model provenance, and open source model risks.
• 21:19–26:25 Shadow AI, evolving definitions, and organizational governance fragmentation.
• 26:25–30:03 Agent identity, onboarding/offboarding, secrets management, and breach anecdotes.
• 30:48–33:47 Mohamed’s practical recommendations and common mistakes.
• 33:47–35:06 Closing reflections on enduring and evolving security challenges.

---

Summary Takeaways

This episode is a must-listen for security practitioners and leaders navigating the tidal wave of enterprise AI adoption. With incisive real-world examples and guidance, it lays bare both technical and human factors—urging organizations to go beyond “checking the box” to build robust AI governance, clarify responsibilities, and proactively secure agents, identities, and supply chains in the AI era.

Transcript

[A]

Users that have privileged access, the agents are now using those same identities. How do you distinguish between a human using that account versus an agent using that account? And that leads to, from an investigation perspective, challenges for those organizations. So that's definitely something. And we've responded to a few incidents where threat actors and I'm going to pivot a little bit into LLMs being living off the land risk here where threat actors or malware that looked for CLI tools that are LLM enabled and the malware being able to leverage that CLI tool.

[B]

Welcome to another episode of Mandance Defenders Advantage podcast podcast. I am your host Luke McNamara. I have the privilege today of welcoming back Mohamed Manir, who is a technical manager on our incident response team within mandiant at Google Cloud. Mohamed, great to be talking with you today.

[A]

Likewise, Luke. Thank you for having me.

[B]

So we're going to build off of a conversation that we touched on briefly, I think when you were last on we touched a little bit of kind of your area of focus, I believe, when that part of the Mania practice was being spun up to kind of the services around helping customers secure their AI implementation. You gave a talk at Cloud Next this year building further on that. But I wanted to kind of dive further into this topic because I think it's one that there's a lot of interest in and it's not one necessarily that shows up as much in some of the data sets and reporting that within GTIG we push out. So maybe I'll kind of start with this framing and you can correct me as we're diving into this, but people hopefully are familiar with the AI Threat Tracker report we put out every quarter here at gtig, and that is primarily looking at threat actors that are misusing or abusing AI. But there's a whole other area to this that we are increasingly seeing as AI is being adopted by enterprises, which is the security risk, the misconfigurations, the governance issues that organizations run into as they start actually implementing this in their own environment. So beyond just for example, how a threat actor might conduct a prompt injection or a distillation attack against any one individual model, as these models get adoption, as these AI tools get adoption, now we are starting to think through and kind of run into risk where when done insecurely, those present issues for the organization. And so this is something I know you and your team are working with enterprises to help address, help them work through this. So maybe I'll start super broad question, you know, what are you seeing out there? What's going on these days when it comes to organizational challenges in the enterprise space in adoption of AI, yeah, absolutely.

[A]

I think the logical place to start there would be organizations that dive headfirst into adoption of AI, but they don't necessarily have the appropriate governance defined for the organization. So when we're talking about governance, we're talking about identifying the risks, the strategy for adoption of AI, the type of AI use cases that the organization is either going to acquire from third parties or will be developing in house. And that really leads to a few different offshoots. You've got shadow AI as a risk where business units or different teams within the organization will develop their own strategies in a silo, execute on those strategies, whether it is engaging with third parties, developing in house, and to go one more layer deeper into that component where some teams might be testing open source models, for example, or integrating third party AI platforms with data and tools, that leads to a higher risk. But that's a blind spot for the leadership teams and the security teams that you know, enhances the risk for those organizations. Part of this is also, again we make a distinction between governance for AI and governance of AI for AI. Is your overarching governance framework for the organization. The way that I described it, governance of AI is ensuring that the AI deployments, the applications that are either developed in house or required from third parties are adhering to the governance that has been defined for the organization. So obviously if you don't have the framework defined, then it's pretty much the wild wild west. But in the event you do have IT defined, how do you operate? Operationalize that for agents, for example, ensuring that they have the appropriate role based access controls for agents like you would for employees. You have the right profiles. You're applying the least privileged approach when it comes to agents like you would to users. That is something that we're finding organizations trying to figure out and in some cases struggling with. One other area I would call out here is legacy security debt. So organizations that have deficient practices around identity and access management, around vulnerability management, supply chain management, vendor management, they are at a higher risk as a result because again they have already deficiencies around those programs. But now with adopting AI capabilities that further I feel the impact of the risk is greater as a result. I've been asked this question by a couple of clients. Is an AI risk just IT risk? It is, obviously it's part of the IT risk, but it's also a matter of probability. You know, you've increased the probability of data leakage, of data theft as A result of integrating these capabilities with your data, of abuse of your tools. So that's. Those are some high level areas that we're definitely seeing challenges around.

[B]

And is that mostly just to kind of highlight or hone in on that point there? What you're talking about in terms of the realization of the amount of tech debt or other sorts of legacy security issues that haven't been addressed, those coming to the forefront is that on the sort of. Now that they're thinking about architecting AI into this and they're kind of looking at the sort of how is this going to change our threat posture? Is that where that is becoming apparent? Or are you starting to see like there's an uptick in incidents that are being traced back to the deployment of AI being put into an enterprise that already has a lot of these legacy security issues and that's just kind of highlighting all the problems. Problems under the rock. So these may be a way to. Maybe that was a bit of a mouthful, but I guess what I'm trying to hone in is like, are these. Because this is coming to the forefront because more incidents are happening where they're able to kind of tie this back to AI, or this is something where they're just now thinking through the implementation and as they're kind of tabletopping, they're realizing we have a lot more long standing issues we need to address.

[A]

So it's definitely more related to them realizing the misconfigurations. And I don't want to say it's as a result of necessarily an uptick of incidents. So I'll give you an example. Organizations that have agents that are using user identities instead of agent identities because they haven't enforced agentic identities. That's as a result of having lax controls around identity and access management and not having a strategy in place for agentic IDs. So when we are working with organizations and we see that type of behavior, we help them understand what the risk there is. How do you distinguish between a developer using an API key that is assigned to them versus the agent making those calls? Or similarly users that have privileged access, the agents are now using those same identities. How do you distinguish between a human using that account versus an agent using that account? And that leads to, from an investigation perspective, challenges for those organizations. So that's definitely something. And we did. We've responded to a few incidents where threat actors, and I'm going to pivot a little bit into LLMs being a living off the land risk here, where threat actors or malware that looked for CLI tools that are LLM enabled and the malware being able to leverage that CLI tool to basically enter prompts and perform reconnaissance and further actions as a result of that tool being in place. Now the developer is using that LLM enabled CLI tool, the malware use that same LLM enabled CLI tool. So from that perspective, there's no way for them to make a distinction when that happens. Benign usage versus malicious usage, not having the appropriate guardrails, allowing for that type of activity on endpoints, even if it is sanctioned. Fair enough. But then you need to have that security layer in place to say we have appropriate visibility and we have the ability to detect when anomalies happen. And it goes right back to the type of behavioral analytics that we used to perform before Velocity and Volume, where back in the day we used to look for evidence of scripted attacks that would be evident as a result of the velocity of the execution, the files being touched. Same thing here. Prompting can be indicative of an agent making API calls as a result of the volume and velocity of those calls versus a human actually doing it or a script executing those API calls. So it goes back to again, misconfigurations organizations realizing that as a result of some of the assessments that we've done, realizing that they have facilitated this type of scenario to propagate in their environments. I feel with the AI capabilities it just exacerbates the problem.

[B]

So you mentioned we are starting to see these examples of threat actors using LLMs or AI agents. AI agents in the environment as a method of, you know, whether it's reconnaissance or living off the land activity more generally. What are, what are some of the things that are attractive about targeting and leveraging those tools? Obviously we've seen threat actors use AI on the back end to support their operations from a reconnaissance or attack perspective. But using those. And maybe this also kind of ties into the MCP component and where you get sort of over permissioned infrastructure. But what makes this attractive for a threat actor, that once they're in the environment they want to target it?

[A]

I would. And again, this is a bit of a judgment statement on my part, but I feel due to a lack of visibility on these tools, due to a lack of detection capabilities that makes it attractive. And the permissions and the access to data that these agents typically have. We've run into situations where role based access controls have been backdoored using these agents. So for example, if somebody in finance doesn't have access to data in hr, but they have access to an agent that does have access to data in hr, by virtue of having access to that agent, they now have access to the data in hr. Same type of misconfiguration that the threat actors are relying on to say that we can do reconnaissance in a blind spot, we can dwell in plain sight, leveraging these agents, leveraging these LLMs to perform that reconnaissance.

[B]

Talk a little bit more about what's happening in the larger landscape right now. So when you're working with organizations and you're helping them kind of think through and architect the secure usage of these tools and infrastructure. I know kind of earlier we were talking about some of the things that you're seeing in landscape, and one of the things you said that was interesting to me is in the interest of maintaining and kind of managing costs, which is something obviously a lot of organizations are now kind of thinking through with respect to token usage and AI costs, leveraging and hosting open source models and having the ability to kind of select between different models depending on different tasks. But you were bringing up some of the interesting things that you've seen where maybe organizations haven't thought through some of the risk associated with just even architecting things in that way. Maybe you could talk about that a little bit more.

[A]

One of the primary blind spots that I'm seeing is organizations not really having an opinion on the lineage of the model. So that is where who's the publisher of the model? That is one of the starting areas that I'm seeing. Organizations haven't thought things through. But when you do pivot into that open source model element, scanning of the model, scanning of the weights, ensuring that they're not backdoor, having a process through which you're actually getting assurance around that to say that, you know, there's no misconfiguration there. We've seen situations where, for example, something as simple as just doing a hash check, something that is posted by the publisher of the model, and then somebody going out to hugging face and grabbing a copy of the model, but not ensuring that it's a valid hash. So what that means is it's quite possible it's been modified. So that again is something that is a miss for organizations. Not extending your vulnerability management practices into this pipeline is another area that we're seeing deficiencies around. So for example, taking a look at the different components, libraries and packages, transformers, for example, have vulnerabilities that are associated with them. There are some transformers that have remote code execution vulnerabilities, but they're being used by organizations because they're not keeping track of the versions for transformers. They're not keeping track of the different libraries and packages that are being used. Again, here's a multiplier impact here. Coding assistance that are being used and coding assistance that can actually introduce non standard libraries and packages that are not being reviewed are being pushed into production. That speaks to the lack of AI bill of materials where they're not tracking all the different components. And again as a result of that, that's leading to an increased risk. To expand that a little bit supply chain, we've talked about identifying the data lineage. So what type of data has the model been trained on? Not a lot of publishers out there are making that clear. Well, they're not disclosing that necessarily. That can result in a bit of a challenge that's actually come up with a couple of our clients where they've asked that question says, hey, how do we validate what type of data the model of the open source model has actually been trained on? That is another risk that you have to consider. Further to that, I would say organizations need to also consider when they're hosting models, whether they're hosting it in cloud or private cloud, they have to consider the attack vectors that are now going to be part of their landscape as a result of hosting a model. What are their capabilities around data quality, around data integrity, data access, model access. It definitely solves a problem from a costing perspective, but it doesn't necessarily, or rather I should say it introduces a host of other security risks that you have to consider. So while yes you can, you can address the costing element there, you do then also have to address the security risk, the potential security risks that the organization faces as a result of these models being used. Open source models being used. Now you're fine tuning, you're grounding that model, open source model with your corporate data. So data integrity, data lineage is going to be something that you have to be cognizant of. Organizations that don't have mature data governance data tagging in place, that becomes a problem for them to say, what type of data are we going to expose to this agent? Or is it just going to be again a blind spot? We don't know what type of data is being used or where the data is being sourced from. We've had some clients that are basically stating that we use whatever data is available within the repo, so the data owner is responsible for the data integrity and quality. But again there's no assurance there. So that's another risk that you run into when you're Using open weights and then security of the weights themselves. Distillation attack is something now that the organization needs to consider. And again, distillation attacks do not happen with one prompt. It's going to be a high volume of prompts that result in somebody trying to reverse engineer your weights and trying to understand how the model's been trained. But if you don't have the ability to detect and prevent those types of velocity attacks, you're pretty much a sitting duck at that point. And again, it does depend on whether or not the applications, for example, the chatbots that might be using the open source LLM model, whether it's exposed externally or internally. But regardless, you need to have that visibility. I do want to make one more comment here. I missed talking about it when I was talking about the LLM as being living off the land. Risk organizations aren't necessarily tracking how this actually impacts the insider threat landscape. So when we're talking about threat actors using LLMs for reconnaissance, we're talking about agents that might be overly provisioned, backdooring, role based access controls. Well, that does open a door for insider threats as well, because now they're able to again, do exactly the same thing. Reconnaissance, getting access to data they typically wouldn't have access to beyond just researching different technologies that are available within the organization in the event they're trying to figure out how do I bypass controls or being able to write code to perform certain actions that they wouldn't be able to before. So that's something else that we want to keep track of, how this actually changed the insider threat landscape.

[B]

Yeah, and I want to return to the identity piece a little bit later, but to just kind of linger a little bit more on the supply chain component of this, because you noted that. And I think it's important to also highlight this is not something that is just a theoretical, theoretical risk. This is also something we've seen this year in several supply chain compromises with malicious NPM or PI FI packages or others, some of which are being downloaded and utilized in a variety of CICD pipelines. But we've also seen some that have been specific to AI applications, harnesses, et cetera. So maybe I think, kind of to go back to an earlier point, you know, are you seeing an environment where enterprises are very multimodal in terms of the models that they are leveraging to include open source, and then are leveraging these open source supply chain pipelines to develop their own kind of harnesses or you know, to, to work with some of the more open source models? And they're exposed to some of this risk because of the supply chain threats that we're seeing and how they're integrating these multiple models and capabilities together. Yep.

[A]

So we're definitely seeing organizations that are again, like you said, multimodal in the sense that they've got multiple different solutions and use cases that they have in place. So for example, and even from the perspective of what's being deployed. So for, for example, they might have a specific application that is using a third party LLM, an enterprise LLM, and they might have another application that is using in sort of open source, fine tuned model that they're hosting themselves. An interesting deployment that I've seen is organizations that are actually tiering their agents. So for example, and again this goes to the token cost element where the more tasks that are geared more towards the third party Enterprise LLM solution, they have that at Tier 3 and then they have Tier 1 and Tier 2 for the more simpler prompts where they redirect those prompts to their open source models or agents that are using those open source models. So to do that they're basically using variations of what I'd call AI gateways where it infers the prompt based on the complexity of the prompt. It decides where it's going to direct that particular prompt to. Now one other element here is, and again a challenge here for organizations is people within the organization that are running agents locally that are beyond these platforms that they're allowing people to actually interact with these agents from, and then also allowing people to actually leverage, for example, cloud code, where developers will be using the API key for cloud code directly from their endpoints. We're seeing that organizations have deployed their API gateways and the intent is to get visibility on usage of all LLM API keys going through that API gateway. But they're also allowing developers to hit the LLM endpoint directly from their workstations, which means it's not going through the API gateway, so they lose visibility there. Now they're again having to go through two different platforms. Yes, it's still going to be through the firewall, but whatever controls you have in place within that API gateway, including payload inspection, is not in place now. So you're allowing a path out to those LLM endpoints that bypasses the guardrails that you're setting up. And unfortunately we're seeing this very often. Something to be said about a typical comment I get from the security teams is, well, the reason we have to facilitate this is otherwise it's seen security is seen as A headwind limiting element that would slow down the pace of development. And I think this is where organizations need to make that informed decision, risk based decision to say, how much risk are we willing to accept to be able to facilitate this type of an approach? What does the risk look like for us? And if we do actually enforce the security, how much assurance do we get? How much does that mitigate the risk for us to be able to enforce that and to basically state that this is the path, this is the enforcement that we have in place. To me, again, I might be coming off a bit biased here, but more visibility that you have, more controls that you have, the better. Not to the detriment of development, but I feel once you have that visibility plane and control plane in place, it should in fact increase the adoption and enhance the adoption because you have more assurance and limits the risk for the organization.

[B]

Returning back to the topic of governance which we opened this with, but this is something that I think is since the advent of this technology, probably one of the earliest areas of concern of risk that I've heard from CISOs and some of our customers about shadow AI, right? How do you define shadow AI or even shadow governance of AI? What are some of the things that you've seen kind of run into in your work?

[A]

I'll quickly state again that the, for example, organizations that are allowing usage of non standard LLMs is something that we're seeing pervasive. The other thing I want to point out here is some organizations are doing that deliberately again to address the token cost element, to say, hey, if you have access to your own LLM, great, use it. Because that means we don't have to pay for it. You're responsible for whatever token burn rate you're going through and they're applying that to developers as well. I worked with one organization where their developers were using their own coding assistance, whatever they had access to. So if you had the wherewithal to pay for Claude code yourself, great, do it.

[B]

Which I think just to pause on that for a second, that's an interesting shift from like when this was all first starting to roll out and you would hear from CISOs, I don't even know who's using what and I don't know what they have access to, et cetera. And now you're hearing from at least other parts of the enterprise, maybe not necessarily from the CISO and the CISO Zorg, but now you're hearing go ahead and use those, go ahead and use those because of an effort to control costs. So I Just wanted to highlight because I think that's an interesting dynamic and even shift that now security organizations and leaders have to contend with.

[A]

They've coined a term for it. I mean, late 20, early 2025, I heard one client coined the term bring your own AI. But now I'm starting to see, or rather I'm starting to hear the phrase bring your own LLM as something that is part of, part of the organization's approach to AI adoption within the org. I have another interesting take on shadow AI. This again came from a client engagement where the CISO is faced with a situation where the leadership within the organization goes in, buys licenses for enterprise LLM solutions, but then there's no security review done, even though it's supposed to be done. And those licenses are furnished to the various teams that need access to those licenses. And at the start of the engagement, the way that shadow AI was defined was anything that is licensed is sanctioned, hence not shadow AI. And anything that is not licensed, not enterprise license, is unsanctioned, hence shadow AI. Towards the end of that engagement, the definition changed substantially where the CISO basically said security review was not done if the security team did not give it a thumbs up@Shadow AI, doesn't matter if it's licensed or not licensed. So even their licensed solutions that were not security reviews were not done for that organization at least now means that that falls under the column of shadow AI. And I think that's a really important point to make, which is organizations need to define what shadow AI means for them. This is definitely something that we've seen evolve over a period of time and there's not a standard definition that goes in place. We typically lead the discussions, we help them define what shadow AI is. But I'm not seeing a standard definition for shadow AI, which is fine, but I think that's, that's the, the nature of the beast right now, which is for different organizations that have different risk profiles. In this particular organization's case, there are a lot of enterprise AI solutions that are being used within the org that have not had a security review. And you know, from the perspective of logging visibility, they don't have answers. So that's something that, that is resulting in that definition to say anything that we don't have logging for visibility on is shadow AI. So I think organizations need to take a concerted effort on that to say what does shadow AI mean for us, for our organization, for our use case?

[B]

And I imagine especially on large organizations where you're dealing with orgs that span multiple business units and have their own kind of interests in different kind of pockets, even globally, that can become very challenging. And the assumption that we're all working off the same sheet of music when it comes to defining some of these terms may not actually be the case.

[A]

Exactly right. And I think that also leads to that concept of shadow governance. We've seen that where different business units have. Because either they don't realize that there is a centralized governance in place that has not been socialized properly, it has not been communicated, or it doesn't exist. Different business units, different teams within the Org have set up their own parameters and they're operating within those. What that means is now they've got multiple variations of governance. Like you said, there's no standard approach for that organization. So it becomes harder for those organizations now to say, well, here's a standard framework. Because now all those business units that have already gone down whatever paths they've gone down will have to figure out how to deconflict with what is now a standard governance framework, but also to try and understand how do they fit their current use cases into the overarching governance framework that has been defined after the fact. So yeah, I think it all goes right back to that lack of governance or lack of operationalization of governance as a foundational element for AI adoption.

[B]

One of the things I want to pull the thread on a little bit more that you've mentioned a couple times now is the identity aspect of this. And I think that is something if you know, for most, or maybe I should say most, but so many of the compromises and breaches that we respond to even still today, this year, identity playing a role in that, and obviously going even beyond just targeting an individual employee's credentials, the uptick in utilization of non standard phishing techniques like phishing targeting, things like SaaS integrations, targeting wherever OAuth tokens might live. Identity has just been kind of an area of focus for a significant amount of the activity we've seen over the last couple years to continue this year. And I'm curious now with the expansion of these tools, and you've touched on this a little bit already, but what are some of the issues that crop up that you find when it comes to not properly managing the identities of agents, as these are now starting to proliferate in a lot of enterprise environments.

[A]

So again, speaking to the challenges, or rather the gaps that I'm seeing at the organization level is not having a defined process to onboard agents and not having proper processes defined to offboard agents and Key component of that is the identities for the agents. Having agents that were part of POCs, their identities still exist. They're orphaned IDs, they're, they're highly privileged and again, they're, they're kind of just sitting there at that point, but not really disabling or removing those, those agentic IDs if they're using agentic IDs at all. And like I said before, we've, we've run into a lot of situations where agents are using user accounts, human identities, as what they've been provisioned with. So that's an ongoing challenge for organizations is that part. So you need to have a process through which you are able to enforce that to say agents cannot use user identities. And in the event that they do, we need to be able to detect that and sort of mitigate that if that's happening and remediate it if it's happening. The other part is secrets management that goes with identities. We have run into situations where API keys have been stolen for LLM endpoints and the organization only realized that they'd been stolen when they got the bill for that particular LLM, which unfortunately in some cases have run into a few million dollars. So within that 20 day period, the threat actor or whoever abused that API key was able to run up a substantial bill for that organization. So secrets management is something else that we are running into as a challenge. What does secrets management look like for agents? Having said all of that, I do want to point out that from a security challenge perspective, we aren't necessarily reinventing the wheel to your point. A lot of the, even the areas that we've touched on so far, we can be talking about human identities, we can be talking about user accounts the same way that we're talking about agent accounts and agent identities. We haven't reinvented the wheel when it comes to the security challenges. It's just we're looking at it through a different lens. But the problem statements, the challenges are similar. The tools we might use to mitigate for agents might be a little bit different. So for example, the agent gateway is a good example there where you have a full framework that allows you to onboard and offboard agents manage agent identities. So that is a dedicated platform that helps you strategize your identity management for agents. So yes, there are different platforms, but what you're trying to accomplish is we've got muscle memory around that because we're trying to accomplish the same thing for user identities.

[B]

We've covered a lot of ground here. I Want to be mindful of the time we've covered everything from governance issues to supply chain threats, identity issues obviously and everything in between for organizations, for CISOs and security leaders that are listening to this and maybe they're at the kind of start of their journey or a little bit down the road in terms of AI and agent adoption, what are some of like the key things that you would flag to them as risks to be aware of? Either from a severity standpoint, if you get this wrong, it's going to lead to some really big problems from what you've seen, or just even a commonality, something that's like a very prolific problem that may be minor, but something that's much more extensive in terms of people making organizations making that same mistake.

[A]

I'll try and capture it succinctly. So first off, address your security debt, your legacy security debt. Be honest about it. It's not meant to say don't do it if you have these legacy security debts, but rather be aware of them. So the four that we talked about, vulnerability management, supply chain, your data governance and identity and access management, be aware of what they are and understand how that impacts your AI use cases. Not every single organization is going to be a trainer. What I mean by that is they're not necessarily going to be fine tuning open source models. Majority of our clients are integrators. But even for integrators that are using third party LLMs integrated with data and organizations own tools, you need to understand what that security debt means for those use cases. Understand the risk reality for using AI within your organization. How does that impact your risk for the organization? How does it change the risk profile, address your governance gaps, ensure that you've got defined ownership within the organization. This is something that we find where there's a lot of ambiguity around ownership of AI. If we ask a question, who owns AI within the organization? We have a lot of hands that go up. When we ask who owns the risk for AI at the organization, no hands go up. So you need to get ahead of that, you need to get in front of that. I would also say that you need to try and bridge the gap between compliance and AI use case. That's something else where organizations haven't necessarily connected the dots. What does compliance mean and look like for AI? And that's where the governance for and of AI comes into play. Governance for again is your roadmap and your strategy and your risk definitions. Governance off is how do we operationalize it? How do we ensure that our AI use cases are Adhering to our governance parameters. Governance off is going to be more of a technical solution. That's where your visibility and monitoring capabilities come into play. I would also say get ahead of shadow AI. Define what shadow AI looks like for your organization, what it means for your organization, and then with that, come up with a strategy through which you can mitigate shadow AI as much as possible at a very high level. Those if you want me to go just one layer deeper based on whatever use cases that you have, try and address the visibility gaps. Typically we will recommend to organizations you need to capture telemetry from your AI use cases. The problem with that is sometimes they do get telemetry. These are very verbose applications. They generate a lot of telemetry. So within the telemetry there are specific components that you need to track for transparency, explainability for the agents, for people that are actually interacting with it, for people that have access to the back end of these use cases. But understand what your use cases are based on that. That should give you some inkling as to what type of visibility you need for each of those use cases, whether it's something as simple as just the API calls and the payloads, and whether that means access to the back end. If you are fine tuning open source models, we talked about that earlier, so I won't repeat that too much here, but understanding what the implications are in the event that you are using open source models, how do you define the risks, how do you identify the risks and how do you mitigate those risks?

[B]

Excellent. Well, there's a lot there, I feel. You know, we've been talking for 36 minutes and I feel like we've only scratched the surface of this, but given how rapidly changing this space is, and certainly I think on the adoption side of this, you know, I'm excited to hear more kind of down the road of what you're seeing because I think this is an important piece of the larger landscape and how it's changing and how some of these risks are changing. And it is interesting, I think, as you note, how so many of these trace back to kind of pre existing security issues that organizations have been dealing with. There's obviously new attack services from a technical standpoint and the actual technologies that are being deployed that are changing some of the landscape internally for organizations. But also some of these challenges are ones that we've dealt with and either struggled with as an organization or succeeded in finding and building controls around as well. So it is interesting how that is still continuing to kind of define even with the adoption of these new technologies to define the actual threat posture of an organization.

[A]

Absolutely. I think if we were to do another edition of this podcast in two or three months time, yeah, I don't think the security challenges would change necessarily, but how we're seeing it, I think that's going to continue to evolve for sure.

[B]

Well look forward to continuing that conversation with you, Mohamed. Thanks again for doing this and enjoy the rest of your week.

[A]

Likewise. Thank you very much, Luke.

[B]

Take care.

[A]

Cheers.

[B]

It.