When AI gets a to-do list. [Research Saturday]

Summary

Summary of CyberWire Daily: "When AI Gets a To-Do List. [Research Saturday]"

Release Date: May 3, 2025
Host: Dave Buettner
Guest: Shaked Rayner, Principal Security Researcher at CyberArk

---

Introduction

In this episode of CyberWire Daily, host Dave Buettner welcomes Shaked Rayner, Principal Security Researcher at CyberArk, to discuss the burgeoning field of agentic AI and its profound implications for cybersecurity. The conversation delves into the definition of agentic AI, its unique vulnerabilities, challenges in identity and access management, lifecycle management of AI agents, best practices for deployment, and future trends shaping cybersecurity strategies.

Understanding Agentic AI

Agentic AI refers to artificial intelligence systems that incorporate Large Language Models (LLMs) to autonomously make decisions regarding the control flow of programs. Unlike traditional LLMs or chatbots, agentic AI can perform actions in the real world based on the instructions it autonomously generates.

Shaked Rayner explains:

"Agentic AI is kind of a concept... it basically means any type of system, any type of code that uses LLM in sort of a way that allows the LLM to decide about the control flow of the program."
[01:27]

Security Implications of Agentic AI

Rayner highlights that while traditional security measures focus on preventing unauthorized access and ensuring system integrity, agentic AI introduces new layers of complexity. The ability of agentic AI to perform real-world actions amplifies the potential impact of any security breaches, making the security risks associated with such systems more severe compared to conventional AI applications.

Vulnerabilities in Agentic AI Systems

The research conducted by CyberArk meticulously maps out the threat landscape of agentic AI, categorizing vulnerabilities into two primary areas:

1. Traditional Access Attack Vectors: These include server-level attacks and other conventional threats that have been prevalent in information security for decades. Despite the advanced nature of agentic AI, these traditional vulnerabilities remain relevant and pose significant risks.

2. LLM-Based Attack Surfaces: This new category encompasses vulnerabilities unique to AI systems, such as prompt injections and model manipulations. These attacks can alter the behavior of the AI system, causing it to act in unintended and potentially harmful ways.

Rayner emphasizes:

"We really tried to illustrate and actually demonstrate practically and technically... how those attack vectors manifest in the Agent AI field."
[03:30]

Identity and Access Management Challenges

Agentic AI complicates traditional Identity and Access Management (IAM) frameworks due to the autonomous nature of AI agents. Determining the identity of these agents—whether they are users, machines, or bots—and assigning appropriate permissions presents a significant challenge. Current IAM standards are not fully equipped to handle the dynamic and autonomous actions of AI agents, raising concerns about how to securely grant and manage access tokens and permissions.

Rayner notes:

"Agentic AI is still a beast that we don't know very well as an industry... we need to understand what their identity should be and what access exactly they can have."
[05:30]

Risks of Overprivileging AI Agents and Mitigation Strategies

One of the critical risks associated with agentic AI is overprivileging AI agents—granting them excessive permissions that can be exploited if the AI is manipulated. For instance, an AI agent with broad access can be coerced into accessing multiple databases, even those the initiating user doesn't have access to, through techniques akin to social engineering.

Rayner explains:

"...LLMs can be manipulated... to perform actions that I wouldn't have access to in the beginning."
[06:42]

Mitigation Strategies:

• Principle of Least Privilege: Assign AI agents only the minimum permissions necessary for their functionality.
• Access Control: Carefully manage and restrict access tokens and permissions assigned to agents.
• Monitoring and Auditing: Continuously monitor AI agent activities to detect and respond to unauthorized actions.

Lifecycle Management of AI Agents

Managing the entire lifecycle of AI agents—from deployment to decommissioning—introduces unique challenges:

1. Trustworthiness of LLMs: Organizations must rely on trusted LLM providers, as the underlying models significantly influence AI behavior.

2. Configuration Monitoring: Changes in system prompts or configurations can dramatically alter AI behavior. Monitoring these configurations is essential to prevent unauthorized modifications.

3. Monitoring Agent Actions: Similar to employee oversight, AI agents' actions must be regularly monitored and audited to ensure they operate within defined parameters.

Rayner emphasizes the importance of lifecycle management:

"We need to make sure that we are aware of exactly what are the important parts in the code that we should monitor... the behavior of the agent can be dramatically changed just by changing the instructions."
[10:21]

Best Practices for Deploying Agentic AI

Based on their findings, Rayner offers several best practices for organizations deploying agentic AI:

1. Comprehensive Mapping: Identify and inventory all systems utilizing LLMs and agentic AI within the organization.

2. Never Trust the LLM: Treat outputs from LLMs with skepticism. Validate and sanitize all AI-generated information to prevent malicious exploitation.

3. Task Appropriation: Use AI agents only for tasks that cannot be effectively handled by traditional code. Limit the scope and autonomy of AI agents to reduce risk.

4. Least Privilege Principle: Assign the minimal necessary permissions to AI agents to perform their functions without exposing unnecessary access.

5. Credential Management: Implement robust management and monitoring of credentials assigned to AI agents to prevent unauthorized access.

6. Security Monitoring: Establish continuous monitoring and threat detection mechanisms specifically for AI agents to swiftly identify and mitigate potential compromises.

Rayner summarizes these practices:

"Map out all of the systems using LLMs... never treat your LLM as a security boundary... utilize the old least privilege principle... make sure to have security monitoring and threat detection and response for those agents."
[13:41]

Future Impacts of Agentic AI on Cybersecurity

Looking ahead, Rayner anticipates that the evolution of agentic AI will significantly impact cybersecurity strategies:

• Rapid Technological Advancement: The pace at which agentic AI technology evolves poses a moving target for security measures, necessitating agile and adaptive security frameworks.

• Emergence of Advanced Attack Vectors: As AI capabilities expand, so will the sophistication of attack techniques targeting AI systems, requiring innovative protective measures beyond current practices.

• Unprecedented Functionality: Future agentic AI systems may possess functionalities that are currently unimaginable, further complicating the security landscape.

Rayner shares his foresight:

"The pace that this technology progresses is really, really fast... agentic AI would look entirely different than what it is now... security measures that we will have to create will be different than what we can think of now."
[17:05]

Conclusion

The discussion with Shaked Rayner underscores the critical need for robust security frameworks tailored to the unique challenges posed by agentic AI. As AI systems become more autonomous and integral to organizational operations, understanding their vulnerabilities and implementing best practices is paramount to safeguarding against increasingly sophisticated cyber threats. Organizations must stay vigilant, continuously adapt their security strategies, and foster a deep understanding of agentic AI to navigate the evolving cybersecurity landscape effectively.

---

For more insights and detailed analysis, listeners are encouraged to subscribe to CyberWire Daily and stay ahead in the ever-changing world of cybersecurity.

Transcript

[Dave Buettner]

You're listening to the Cyberwire Network. Powered by N2K, traditional pen testing is resource intensive, slow and expensive, providing only a point in time snapshot of your application's security, leaving it vulnerable between development cycles. Automated scanners alone are unreliable in detecting faults within application logic and critical vulnerabilities. Outpost 24's continuous pen testing as a service solution offers year round protection with recurring manual penetration testing conducted by Crest certified pen testers, allowing you to stay ahead of threats and ensure your web applications are always secure. Hello everyone. Hello everyone. Welcome to the Cyberwires Research Saturday. I'm Dave Buettner and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems and protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

[Shaked Rayner]

So agentic AI is kind of a concept, I should say, and it basically means any type of system, any type of code that uses LLM in sort of way that allows the LLM to decide about the control flow of the program.

[Elliot Peltzman]

That's Shaked Rayner, principal security Researcher at Cyberark, discussing their research agents under attack threat modeling Agentic AI. We'll have a link in the show.

[Shaked Rayner]

Notes.

[Dave Buettner]

What initially prompted you and your colleagues to investigate the security implications of agentic AI?

[Shaked Rayner]

Yeah, so as I'm sure we all know, the buzz around agentic AI is very much present in our industry in the last couple of months. And about two years ago we decided to dive into the whole LLM security train and we did that. Transitioning from traditional security and the security aspects or security risks of agentic AI is much more the security risk is much more severe with Agentic AI because unlike traditional LLMs and traditional chatbots, Agentic AI systems allow the models to actually perform actions in the world. And so having a vulnerability in one of those those systems can actually have much greater implications than just having the LLM spit out some information that it shouldn't. And this is why we decided to go deep into the Agent Aki security and this is what motivated us to start exploring it.

[Dave Buettner]

Well, let's dig into the study together here. What are the primary vulnerabilities associated with agentic AI systems that your research uncovered?

[Shaked Rayner]

Yeah, so we basically focused on systematically mapping out the threat landscape of agentic AI systems. So some of the LLM vulnerabilities or attack vectors we already know, but we really emphasize, put emphasis and focus around how those apply in agentic systems. And moreover, we really tried to illustrate and actually demonstrate practically and technically using a lot of Demos how those attack vectors manifest in the Agent Ki field. So generally speaking, we can divide the threat landscape of agent AI into two categories. One is the traditional access attack vectors. Because agentic AI is built upon normal code, we still have a lot of server level attacks on those systems that we all know and love from the past few decades of information security. And of course it's still relevant and we need to be aware of that because even though the technology LLMs and agents is very different, it's still vulnerable to a lot of traditional stuff as well. In addition to that, those systems also present a completely new attack surface, which is the attack surface that presents a lot of LLM based attacks. There we can see a lot of prompt injections and model manipulations that eventually can manipulate the system to behave differently than what it was intended to.

[Dave Buettner]

Well, let's talk about some of the identity and access management challenges here. I mean, how does agentic AI complicate traditional identity and access management frameworks?

[Shaked Rayner]

That's a good question. I think agentic AI is still a beast that we don't know very well as an industry and there aren't any security standards in it, and we still don't really know how to treat those AI agents. Are there users, are there machines, Are they bots? The question is still open. And of course AI agents, the whole thing about them is that they are able to perform actual actions and for them to do that we have to grant them permissions, we need to give them access tokens, we need to open accounts for them, we need to allow them to access databases and so on. And because of that we really need to understand what their identity should be and what access exactly they can have. And since again, it's a new beast, it's still kind of a challenge that we need to face as an industry.

[Dave Buettner]

Can we talk about some of the risks from over privileging these AI agents and, and how organizations can mitigate these kinds of things?

[Shaked Rayner]

Yeah, of course. So overprivileged is a risk that is not only associated with agentic AI, however, it manifests in a very severe way. It can manifest in a very severe way in agentic AI. So for example, let's say that we have an agentic AI system that needs to access some databases. Of course, it needs to have a token for writing and reading from those databases, and it needs to have it for all of them for it to work. And now let's say we are a user or an attacker and potentially we can only have access to one of those Databases. Just because we can use the system, it doesn't mean we have to be able to access to all of them. So we have a discrepancy between what I or the user or the attacker can access and what the agent can access. And we know that LLMs can be manipulated. And this way, even if I personally, as a user have access only to one database, I can try and manipulate the agent working for me that has access to more of those databases in order to perform actions that I wouldn't have access to in the beginning.

[Dave Buettner]

It's interesting to think about the idea of kind of social engineering. Your AI agent.

[Shaked Rayner]

Exactly. And when we were starting to work on that LLM security field, jailbreaking was just that, trying to convince, to kind of socially engineer, persuade the LLM in any way you can think of in order for it to. To behave in ways that it shouldn't.

[Dave Buettner]

We'll be right back. Cyber threats are evolving every second, and staying ahead is more than just a challenge. It's a necessity. That's why we're thrilled to partner with ThreatLocker, the cybersecurity solution trusted by businesses worldwide. ThreatLocker is a full suite of solutions designed to give you total control, stopping unauthorized applications, securing sensitive data, and ensuring your organization runs smoothly and securely. Visit threatlocker.com today to see how a default deny approach can keep your company safe and compliant.

[Shaked Rayner]

Foreign.

[Dave Buettner]

Secure access is crucial for US public sector missions, ensuring that only authorized users can access certain systems, networks, or data. Are your defenses ready? Cisco's security service Edge delivers comprehensive protection for your network and users. Experience the power of zero trust and secure your workforce wherever they are. Elevate your security Strategy by visiting Cisco.com Go SSE that's Cisco.com Go SSE. Can we talk about life cycle management here? As these AI agents go through, from deployment to decommissioning, how do organizations handle that?

[Shaked Rayner]

Yeah, so in terms of lifecycle, I think AI agents present a few challenges that we haven't seen before in that way. And first of all, we should Talk about the LLMs, the base models that the agents rely on and that they provide most of the functionality of the agents. As we know, LLMs are those huge neural networks that most of us just consume because there aren't a lot of companies that really develop and train those big models. And so as consumers, we don't really know exactly what's going on inside. So this is one aspect we should be aware of. And of course, only use models that we trust to the extent that we can trust them again without knowing exactly what's going on inside. So this is one area. Now, another thing is how do we make sure that our agent is trustworthy? And again, we can. There are a few options here in terms of utilizing or writing agents. We can either download or access an agent in some service, or choose an agent from some store. And then again, we need to trust the developer of this agent, or we can alternatively develop them ourselves. And then we need to make sure that we are aware of exactly what are the important parts in the code that we should monitor. So, for instance, let me give you an example. If we're talking about a traditional software, then we know that when the code itself changes, it may change the behavior of the system dramatically. Now, with AI agents, a lot of times the behavior of the agent can be dramatically changed just by changing the instructions or the system prompts or the configuration of the model that is at the heart of this AI agent. And this is why we need to make sure that we also really monitor and defend those configuration files holding those instructions.

[Dave Buettner]

I mean, I suppose just similarly to how you routinely monitor your employees, just to make sure that they're doing what they're supposed to be doing and nothing has gone astray, you need to monitor and audit your AI agents as well.

[Shaked Rayner]

Exactly. And again, because of the flexibility that LLMs provide to traditional code with AI agents, it's really a good idea to monitor the actions that the agents can do, because again, the nature of them is very dynamic, and if we don't define and design them properly, they can really go rogue.

[Dave Buettner]

So based on your findings here, what are your recommendations? What are best practices for organizations out there who are going to be deploying agentic AI?

[Shaked Rayner]

That's a good question. First of all, I'd like to suggest to map out all of the systems using LLMs and all of the agentic systems in the organization in order to just get a grasp of what we're dealing with, Then I'd like to suggest a few key core principles to go by. The first one, and I like to really emphasize this one, is to never trust the LLM. And I like to say that because a lot of people inherently tend to trust those LLMs because they really have an impressive ability to output intelligent text. However, we know that they can sometimes hallucinate. But for us in the security industry, we know that attackers can very easily manipulate those models to behave in any way they wish. So the idea is to never trust an LLM. And from that stems a lot of security best practices. That we can implement in order to deal with that. So for instance, whenever you consume an output from an LLM, make sure to verify that the information is correct, to validate, to sanitize it, and to never treat your LLM as a security boundary. Next, the other thing I can recommend is to really think about what task you need the LLM to perform and in case it can be performed in traditional code, don't use an LLM for that and really limit the space where the LLM can decide what to do and limit the scope of action. This is the second thing. Next, of course, we can utilize the old least privilege principle. So like we mentioned before, make sure that the LLMs have, or that the agents have the least, have the most minimal set of permissions that they should have in order to be functional for your purpose, so that attackers cannot exploit excessive permissions in those agents. Then, of course, traditional credential management, as we mentioned, those agents will have to be given credentials of all sorts in order to perform actions. So make sure you manage them and monitor them properly. And finally, this is more of a general recommendation, and you mentioned this point again before, make sure to have security monitoring and threat detection and response for those agents. We know that no security measure can be bulletproof, so make sure that you are monitoring those agents and have the appropriate measures to deal with compromises in case they happen.

[Dave Buettner]

You know, I'm curious if you have any insights, looking towards the next few years, how you expect the evolution of agentic AI to impact cybersecurity strategies? How are we going to have to adjust to this new reality?

[Shaked Rayner]

Yeah, that's a great question. And I have to say I don't really have an answer for it. Fair enough. Yeah, I don't have an answer, but I can make some educated guesses. So with this whole AI field, then more specifically with agentic AI, we see that again, the pace that this technology progresses is really, really fast. It's crazy. And in security we see it kind of like a moving target because every day we see new technology running out and it's very, very hard to create security boundaries in this stage of the development of the technology. So I can assume that in about a year or 2, agentic AI would look entirely different than what it is now. Again, we really just see the tip of the iceberg of this technology. Just think about how we were looking at chatgpt a couple of years ago and how usable it was then and how it looks like now. The difference is really, really amazing. And I think that in terms of agentic AI, both from the side of functionality and productivity. It will look completely different and it would probably be able to do stuff that we can't even imagine now. So this is the first thing. And secondly, I'm sure that the security measures that we will have to create will be different than what we can think of now. And one last comment here. As we know the hacker space or the security researchers that are working in this industry, there are some very creative people and I think that we haven't really gotten into the more advanced attack vectors and more advanced techniques that we'll be able to exploit AI agents and AI in general. So we still have some time for it to develop. I'm sure that we will be amazed with what we'll see there. This will require protections that we have a hard time imagining now.

[Elliot Peltzman]

Thanks again to Shaked Rayner and the team at Cyberark for discussing their Research Agents under Attack Threat Modeling Agentic AI. That's Research Saturday we'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity. If you like the show, please share a rating and review in your podcast app. Please also fill out the survey in the show notes or send an email to cyberwire2k.com we're privileged that N2K Cyberwire is part of the daily routine of the most influential leaders and operators in the public and private sector. N2K makes it easy for companies to optimize your biggest investment your people. We make you smarter about your teams while making your teams smart. Learn [e-mail] N2K's senior producer is Alice Carruth. Our Cyberwire producer is Liz Stokes. We're mixed by Trey Hester and Elliot Peltzman. Our executive producer is Jennifer Iban. Peter Kilpe is our publisher and I'm Elliot in for Dave. Thanks for listening.