Podcast Summary: Risky Business – "Soap Box: Why AI Can't Fix Bad Security Products"
Episode Information:
- Title: Soap Box: Why AI Can't Fix Bad Security Products
- Host: Patrick Gray
- Guest: Josh Camdu, Co-founder of Sublime Security
- Release Date: August 1, 2025
Introduction to the Episode
In this episode of Risky Business, host Patrick Gray engages in an in-depth discussion with Josh Camdu, co-founder of Sublime Security, about the role of Artificial Intelligence (AI) in the field of information security. The conversation centers around the effectiveness of AI in enhancing security products and the challenges associated with integrating AI into existing security infrastructures.
AI in the Information Security Industry
Patrick Gray initiates the conversation by addressing the proliferation of AI claims among security vendors, particularly the buzz around "agentic AI" and large language models (LLMs). He expresses skepticism about many vendors' AI implementations, suggesting that much of it may be more about marketing than actual functional advancements.
Notable Quote:
"[...] it just seems like every single vendor right now, they're doing some sort of agentic AI LLM based thing. You look at a lot of it and you're like, okay, you've done this engineering work for the press release. This isn't a real thing."
— Patrick Gray [00:50]
Josh Camdu provides a balanced perspective, acknowledging that while some AI integrations may be superficial, many applications of LLMs in the security industry are genuinely useful and enhance workflows for security professionals.
Notable Quote:
"There are problems that you have as an industry [...] and AI agents, LLMs are genuinely good at augmenting your workflows or automating a lot of that work."
— Josh Camdu [02:19]
The Impact of AI on Security Workflows
The discussion delves into how AI, particularly LLMs, are revolutionizing the way security teams operate. Patrick Gray highlights how LLMs have lessened the reliance on scripting languages, making detection engineering more accessible and reducing the learning curve for new security tools.
Notable Quote:
"[...] LLMs have done mercifully is put a bullet into the head of the idea that people need to use scripting languages."
— Patrick Gray [02:50]
Josh Camdu agrees, emphasizing that LLMs can effectively handle the generation of detections and streamline complex tasks without requiring users to engage directly with scripting languages.
Notable Quote:
"If you give an LLM enough context and documentation and tooling and knowledge [...] it can be extremely good at doing that work."
— Josh Camdu [03:21]
Sublime Security’s AI-Driven Approach
Patrick Gray introduces Sublime Security as a modern email security platform that stands out due to its adaptability and efficiency, allowing security teams to delve deeply into their email infrastructure. He raises the question of integrating AI agents into such a platform, prompting Josh Camdu to elaborate on Sublime's innovative solutions.
Notable Quote:
"We have an autonomous security analyst or ASA, and an autonomous detection engineer or ADE. They monitor environments and autonomously improve efficacy."
— Josh Camdu [08:58]
Autonomous Security Analyst (ASA)
Josh Camdu describes ASA as a Tier 1 and Tier 2 security analyst that can investigate, triage, and take action on threats autonomously. ASA leverages a domain-specific language (DSL) tailored to describe complex attacker behaviors and uses extensive context from the customer’s environment to enhance accuracy.
Notable Quote:
"Our autonomous security analyst acts as a Tier 1, Tier 2 analyst to investigate triage attacks in depth and then take actions."
— Josh Camdu [10:00]
ASA can perform tasks such as quarantining malicious messages and communicating with end-users, significantly reducing the manual workload for security teams.
The Security AI Agent Trilemma
Patrick Gray and Josh Camdu explore the concept of the "Security AI Agent Trilemma," a term coined by Camdu to describe the trade-offs between speed, cost, and efficacy in AI-driven security solutions.
Notable Quote:
"If you want something that is really fast and really cheap, then it's not going to be effective basically."
— Josh Camdu [12:33]
This trilemma underscores the challenges in deploying AI agents that are both cost-effective and highly accurate, especially in high-volume, real-time detection systems.
Autonomous Detection Engineer (ADE)
Building on ASA, Josh Camdu introduces ADE, an AI agent designed to autonomously generate and validate new security detections. ADE iterates on misclassifications by creating new rules, backtesting them against historical data, and refining their accuracy before implementation.
Notable Quote:
"ADE will basically be able to take any sort of misclassification and autonomously build a fix for that misclassification within the context of our customer's environment."
— Josh Camdu [16:36]
ADE represents a step towards fully autonomous security operations, allowing detection engineers to focus on more sophisticated threat hunting and incident response tasks.
Multi-Agent Architectures and Future Integrations
The conversation shifts to the future of AI in security, emphasizing the importance of multi-agent systems. Josh Camdu envisions a landscape where different security agents from various companies can communicate and collaborate, enhancing overall threat detection and response capabilities.
Notable Quote:
"We're going to have agents of other companies talking to other agents. [...] your agent ring up another company's agent for additional data."
— Josh Camdu [25:14]
Patrick Gray highlights the need for standardized communication protocols among agents to ensure efficiency and interoperability.
Notable Quote:
"At what point do we develop some sort of standardized method for these agents to exchange information?"
— Patrick Gray [26:01]
Efficacy and Human Oversight
While the AI-driven agents offer substantial automation, both Patrick and Josh acknowledge the necessity of human oversight to handle nuanced and context-dependent scenarios. They emphasize that current AI agents can handle routine tasks efficiently but require human intervention for more complex decisions.
Notable Quote:
"They will lie to you if they think they're going to tell you something you want to hear."
— Patrick Gray [16:29]
Josh Camdu reiterates the importance of transparency and human review in maintaining the efficacy and reliability of AI-driven security solutions.
Real-World Applications and Customer Success
Josh Camdu shares insights into how Sublime Security's AI agents are currently benefiting customers, ranging from large organizations with minimal human intervention to sophisticated companies engaging in active threat hunting. This versatility demonstrates the adaptability and effectiveness of their AI-driven approach.
Notable Quote:
"We're working with one university who is 100,000 mailboxes, one person IT and security team [...] it's just full autopilot."
— Josh Camdu [23:57]
Challenges and Future Directions
Despite the successes, the conversation acknowledges ongoing challenges, such as handling diverse and complex threat campaigns and enhancing agent-to-agent communication. The future direction involves refining AI agents to take on more intricate tasks without compromising on accuracy or efficiency.
Notable Quote:
"We have a fuzzy grouping technology [...] as we see more and more diverse campaigns, that fuzzy grouping problem may get harder."
— Josh Camdu [32:07]
Conclusion: Belief in AI’s Potential
As the episode wraps up, both Patrick Gray and Josh Camdu express a strong belief in the potential of AI to revolutionize information security. They acknowledge the current limitations but remain optimistic about future advancements and the continued integration of AI agents into security operations.
Notable Quote:
"I never thought [...] but I am a believer now."
— Josh Camdu [36:20]
Final Thoughts
This episode of Risky Business provides a comprehensive look into the practical applications and challenges of integrating AI into information security. Josh Camdu’s insights on Sublime Security’s innovative use of autonomous agents highlight both the current successes and the future potential of AI-driven security solutions. The discussion underscores the importance of balancing automation with human oversight to achieve optimal security outcomes.
End of Summary