CyberWire Daily Podcast Summary: "Hiding in Plain Sight with Vibe Coding"

Release Date: June 14, 2025
Host: Dave Buettner
Guest: Ziv Karliner, Co-Founder and CTO of Pillar Security
Research Discussed: "New Vulnerability in GitHub Copilot and How Hackers Can Weaponize Code Agents"

1. Introduction

In this episode of CyberWire Daily, host Dave Buettner welcomes Ziv Karliner, Co-Founder and CTO of Pillar Security, to discuss groundbreaking research on vulnerabilities within AI-powered coding assistants like GitHub Copilot. The conversation delves into how malicious actors can exploit these tools to embed backdoors in software development processes.

2. Overview of Emerging Attack Vectors in AI Applications

Ziv Karliner opens the discussion by highlighting the extensive research conducted over the past eighteen months on new attack vectors targeting AI-powered applications.

“We spent the last year and a half spending a lot of time with the emerging attack vectors that put AI powered applications at risk...” [01:21]

Key focus areas include:

• Prompt Injection: Directly manipulating AI prompts to produce unintended outcomes.
• Indirect Injections: Subtle manipulations that evade traditional security measures.
• Evasion Techniques: Methods that make attacks invisible to both human oversight and existing security tools.

3. Understanding Rule Files and the Concept of Backdoors

The core of the discussion centers on rule files, which guide AI coding assistants in adhering to project-specific best practices and contexts.

“Rule files are basically a way to onboard the coding agent to your project, to your team...” [03:03]

Rule File Backdoor:

• Definition: A vulnerability where malicious instructions are embedded within rule files, influencing the AI to generate compromised code.
• Mechanism: Attackers inject hidden instructions into rule files using techniques like hidden Unicode characters, making the malicious code appear legitimate to developers and security tools.

4. Demonstrative Example of an Attack

Ziv Karliner provides a step-by-step example to illustrate how an attacker could exploit this vulnerability:

1. Selecting a Target: Choose a widely-used framework, such as Next.js.
2. Crafting the Rule File: Create a legitimate-looking rule file for Next.js best practices, embedding hidden malicious instructions using hidden Unicode characters.
3. Distribution: Commit the compromised rule file to a marketplace like GitHub.
4. Deployment: An unsuspecting developer integrates this rule file into their project, unaware of the hidden backdoor.
5. Execution: When the developer requests code suggestions, the AI injects malicious JavaScript into new HTML files, unaware of the hidden intent.

“...an attacker could also use the agent, I would say, intelligence to its advantage...the AI agent will say, oh, this is the security best practices of our organization.” [06:34]

This method not only injects malicious code but also deceives developers by providing plausible explanations for the injected code, leveraging the AI’s natural language capabilities to mask the intrusion.

5. Hidden Instructions and User Deception

The research underscores how hidden instructions can mislead developers, making it difficult to detect malicious activities:

“...the assistants understand, I would say, every language that was ever spoken or written together with hidden Unicode characters...” [12:21]

Key Points:

• Invisible Manipulations: Hidden Unicode characters render malicious instructions invisible to human inspectors.
• AI Deception: The AI can convincingly explain the presence of malicious code as standard security practices, further misleading developers.

6. Human in the Loop and AI Limitations

The concept of Human in the Loop (HITL) is critically examined, questioning its effectiveness in mitigating such sophisticated attacks:

“...if the attack itself is completely hidden to a human, are humans really equipped to be in the loop?” [10:41]

Challenges:

• Visibility: Humans cannot easily detect hidden manipulations within rule files.
• Responsibility: The burden of security shifts to developers, who may lack the tools or expertise to identify subtle AI-driven threats.
• Trust in AI: Overreliance on AI assistants without adequate oversight can exacerbate security vulnerabilities.

7. Mitigation Strategies

Ziv Karliner proposes several strategies to counteract these threats:

1. Sanitation:

   • Description: Restrict and sanitize inputs interacting with AI models to minimize potential attack vectors.

   “...as silly as it may sound, sanitation...” [13:27]

2. Enhanced Security Features by Platforms:

   • GitHub’s Response: Implementation of warning systems that alert developers when hidden instructions or Unicode characters are detected in rule files.

   “...GitHub actually added a new capability...to show a warning message...” [13:27]

3. Guardrails for AI Models:

   • Definition: Implementing detection mechanisms for evasion techniques, malicious instructions, jailbreak attempts, and indirect injections.
   • Resources: Utilizing frameworks like OWASP Top 10 for LLMs and Mitre Atlas to stay updated on emerging threats.

4. Community Awareness and Responsibility Metrics:

   • Awareness: Increasing community understanding of new threats through research and shared knowledge.
   • Responsibility Metrics: Clarifying accountability among developers, tool builders, and model providers to ensure comprehensive security coverage.

8. Implications for AI Integration and Software Development

The conversation shifts to the broader implications of these vulnerabilities on AI integration within software development:

“...there is a lot of enthusiasm for [AI tools]. It's certainly a powerful tool and yet we have these things...” [15:34]

Key Insights:

• Early Stages: AI integration is still in its nascent phases, with rapid adoption outpacing security measures.
• Supply Chain Security: While progress has been made in areas like Software Bill of Materials (SBOMs) for traditional software supply chains, similar advancements are lagging in AI security.
• Intelligence Age: The exponential growth and integration of AI across industries present both unprecedented opportunities and complex security challenges.
• Human Supervision: Emphasizes the continued necessity for human oversight, especially from security experts, to navigate and mitigate AI-driven risks.

9. Conclusion

In wrapping up, Ziv Karliner emphasizes the critical need for:

• Continued Research: Ongoing investigation into AI vulnerabilities to stay ahead of potential threats.
• Collaborative Efforts: Engaging the broader developer and security communities to establish robust defenses.
• Responsibility Sharing: Defining and distributing security responsibilities across all stakeholders involved in AI tool development and usage.

“...we put more effort on the responsibility metrics. Who is really responsible for the security issues at hand...” [16:07]

Dave Buettner concludes by thanking Ziv Karliner for his insightful contributions and highlights the availability of the research paper in the show notes for listeners seeking more detailed information.

Notable Quotes

• Ziv Karliner [03:03]: “Rule files are basically a way to onboard the coding agent to your project, to your team...”
• Ziv Karliner [06:34]: “The AI agent will say, oh, this is the security best practices of our organization.”
• Ziv Karliner [10:41]: “If the attack itself is completely hidden to a human, are humans really equipped to be in the loop?”
• Ziv Karliner [13:27]: “GitHub actually added a new capability...to show a warning message...”

Stay Informed: For a deeper dive into the discussed research, visit the link provided in the podcast’s show notes.

Produced by Liz Stokes, mixed by Elliot Peltzman and Trey Hester. Executive Producer: Jennifer Ibin. Publisher: Peter Kilpe.