Dave Buettner (6:19)

Hmm. Well, can we walk through an example here? I mean, suppose an attacker wants to make use of this. Let's go through the process of how they would go about doing that for sure.

Ziv Karliner (6:34)

So in our research we walked through a simple example step by step example. So for instance, let's think about an attacker that wants to compromise any next JS application and how you can do that. So basically the marketplaces for rule files will have directories with basically you can think about it as a directory of every available coding stack and you can actually commit and add suggestions to these marketplaces and basically hubs of rule files that are being shared between developers. Let's take the next JS example. I will go as an attacker to this repo. I will craft a legitimately looking instruction file about next JS best practices and I will embed hidden text into this file using the hidden Unicode characters technique. And we'll commit this. Let's say to GitHub or with some kind of a web form to this marketplace. What we also uncovered during the research is that in GitHub itself, it was invisible. Basically, when you commit code that contains these hidden instructions, a developer that is now going to approve this basically addition request is not going to see anything, is not going to get alerted. This is actually something that was solved by GitHub early this month in one of the vulnerability patches. So now we have this rule file with hidden Instructions live on GitHub. An unsuspecting developer that wants to get better results with his coding project, when using cursor or GitHub, is basically copying this file and adding it to its own project, also sharing it with his team, just in order to improve the quality of code for the full team. And now when it's going to, let's say, request an addition of a simple page to his application, the rule file that contains instructions to add basically malicious JavaScript code to each new HTML file that is being created. It's going to happen only when the agent loads this file, takes in the additional hidden instructions, and generates the additional code on the fly. Now, the interesting thing that we showed on our research paper is basically that in the attack itself, in the malicious instructions, an attacker could also use the agent, I would say, intelligence to its advantage. So what we've shown is that a developer can then ask, hey, why this code snippet was added to the code that was generated. And the AI agent will say, oh, this is the security best practices of our organization. So the attacker instructions could actually be used not only to inject malicious code, it's also being used to trick the user, kind of social engineer it to believe that this was the goal in the first place. So this is utilizing the AI agent intelligence against the end user. This was, I would say, the most interesting finding for us.

Dave Buettner (10:21)

We'll be right back. So the hidden instructions using Unicode can also include instructions to mislead someone who's inquiring as to why things are a certain way.

Ziv Karliner (10:41)

Exactly. So I can add on that some of the most, I would say popular terms these days is human in the loop. So human in the loop is basically when we're talking about responsibility models and how autonomous agents will be part of the future workforce. So human in the loop is the point in, I would say autonomous processes, where an AI agent goes back and asks for approval from the user that tries to achieve some kind of goal. So in this case, most of the coding agent these days, when doing, I would say more Risky actions like changing, deleting a file or creating a web request, they will actually stop and ask the user, are you sure you want to complete the next action? This is like the classic human in the loop flow. One of the things that we've shown here in the blog is basically that if the attack itself is completely hidden to a human, are humans really equipped to be in the loop? That's one of the thoughts that got got us more concerned. I would say a lot of the responsibility is moving to the users, but are we actually equipped to deal with this kind of attacks?

Dave Buettner (12:07)

I mean, it really speaks to that kind of inherent inability to view inside what's really going on. An AI assistant, right?

Ziv Karliner (12:21)

Exactly. And even if you think you are seeing what is going on, the assistants understand, I would say, every language that was ever spoken or written together with hidden Unicode characters, encoded strings, like base 64 for instance, they just understand it as plain English without the need to compute or run any additional processes. So we are kind of not in an even situation between the, I would say the auditor, which is now basically every person that needs to observe and kind of decide if an AI agent is allowed or not allowed to do something, and the agents themselves. So that goes beyond the coding agents, I would say.

Dave Buettner (13:19)

Well, let's talk about mitigation. What sort of steps can developers take to detect and prevent these sorts of things?

Ziv Karliner (13:27)

Of course. So first of all, I would say, as silly as it may sound, sanitation. So think about reducing basically the input options that you have when interacting with the model, even in the language level. I can actually describe another mitigation that, lucky for us as a developer community, was actually taken by GitHub based on this research, which is they actually added a new capability in GitHub itself to alert and basically show a warning message whenever there is a hidden instruction or hidden Unicode text that is now part of a text file that is going to be edited. This is, I would say, a risk reduction effort that's been released for every developer that uses GitHub, which is almost everyone. Another part which is more on the agent builder side is to take into place different guardrails that can be placed around the models when interacting with them. For instance, detection of evasion techniques, detection of malicious instructions, jailbreak attempts, and indirect injection attacks, which are part of these new attack vectors that are really becoming more and more relevant with AI powered applications. There is some great work around uncovering this full attack surface with OWASP top 10 for LLMs and Mitre Atlas and other great initiatives that really talk this new risk language and create the right terminology around it. So I would say awareness is the first step as well.

Dave Buettner (15:34)

What do you suppose this vulnerability reveals about the current state of things when it comes to AI integration and software development, which I think it's fair to say there's a lot of enthusiasm for. It's certainly a powerful tool and yet we have these things. I mean, is it still early enough days that there's lots to be. These things are important to consider as we go forward.

Ziv Karliner (16:07)

For sure. So we're still in the early days, but I would say coming actually myself from experience in the cloud security space and also in the software supply chain security space, we had, I would say, amazing progress with software supply chain security over the last decade with SBOMs becoming a standard and the vulnerability programs, you know, we put a lot of guardrails inside the CI CD pipelines and got, I would say, a lot of awareness around it. And on the other hand, we now have this amazing phenomena of, I would call it like the intelligence age, the AI transformation that doesn't leave any, I would say vertical in the industry or role untouched, but it's moving really fast. So there is kind of a challenge here when both the attack vectors are being discovered as we go, but adoption is moving faster than I ever seen in my career. So it's a combination, I would say, for both. I would say like the security industry in general, you see a lot of awareness, a lot of community efforts to really surface these new emerging threats even before we saw attack vectors being utilized in the wild. I can give an example that one of the accelerators for safer CI CD pipelines or SolarWind that we're all familiar with. So this really didn't happen yet in the AI security space. I guess, as always, it's a matter of time until something becomes more public because we are at a pace of adoption that is only accelerating, I would say, and the opportunities are, I would say that there are great opportunities these days for developer teams to move much faster and build even higher quality code if they utilize these tools in the right ways with the right context. But I would say human supervision is still much needed, especially from the right security expertise. And in order to do that ourselves as a company, we put. Also, one of our main goals is to help increase awareness with this type of research to really also, I would say, put more effort on the responsibility metrics. Right? Who is really responsible for the security issues at hand? Is it on the developers that utilize these two amazing tools? Is it on the tool builders on the model providers. There is a few different players here that are trying to put these new risks under control and I would say walk in progress.

Dave Buettner (19:44)

Our thanks to Ziv Karliner from Pillar Security for joining us. The research is titled new vulnerability in GitHub, copilot and cursor How Hackers Can Weaponize Code Agents. We'll have a link in the show Notes. And that's Research Saturday brought to you by N2K CyberWire. We'd love to hear from you. We're conducting our annual audience survey to learn more about our listeners. We're collecting your insights through August 31st of this year. There's a link in the show notes. We hope you'll check it out. This episode was produced by Liz Stokes. We're mixed by Elliot Peltzman and Trey Hester. Our executive producer is Jennifer Ibin. Peter Kilpe is our publisher. And I'm Dave Buettner. Thanks for listening. We'll see you back here next time.

Ziv Karliner (20:35)

Sam.