The 404 Media Podcast: How AI Is Being Used by Hackers and Criminals – Detailed Summary

Release Date: November 15, 2024

Introduction

In a special episode of The 404 Media Podcast, host Jason introduces an in-depth interview conducted by Matthew Gault with Rachel Toback, the co-founder and CEO of Social Proof Security. Rachel is a renowned expert in social engineering and penetration testing. The episode, sponsored by DeleteMe, delves into the emerging threats posed by Artificial Intelligence (AI) in the realm of cybersecurity, particularly focusing on how hackers and criminals are leveraging AI for disinformation, spam, deepfakes, and sophisticated hacking tools.

AI in Disinformation Campaigns

Timestamp: [01:46] – [03:45]

Matthew Gault expresses his concerns regarding the increasing use of AI in cybersecurity issues, especially during election seasons where disinformation becomes a critical threat. Rachel Toback elaborates on how AI is transforming disinformation campaigns:

• Emotion-Driven Fake Content: AI is being used to create politically charged and emotionally manipulative content. For instance, fake images like a girl in a canoe holding a puppy during hurricanes or fabricated videos of political figures like Trump aiding during floods are designed to evoke strong emotional responses, thereby facilitating the spread of conspiracy theories.

  Rachel Toback [02:34]: "These are used obviously to create uncommunicated messages. The people that use these AI photos don't seem to care if they're real or fake."

• Impact on Public Perception: Such AI-generated content not only spreads false information but also influences public belief systems, making it harder to distinguish between genuine and fabricated narratives.

• Election Interference: As elections approach, Rachel anticipates an uptick in AI-generated media, including voice clones and robo-callers, which depict inaccurate election day scenarios or spread negative voting-related misinformation.

  Rachel Toback [03:45]: "We'll probably see more voice clones, robo callers, AI generated media, things that kind of depict inaccurate election day conditions."

AI-Controlled Computer Use and Security Implications

Timestamp: [03:50] – [06:16]

The conversation shifts to the recent developments where AI models like Claude are now capable of controlling computers, leading to significant security concerns:

• Autonomous Computer Control: Claude’s ability to browse websites, download, and run files autonomously poses risks as malicious actors could exploit these features to execute unauthorized tasks without human intervention.

  Rachel Toback [04:15]: "It's only a matter of time before we hear someone saying, 'Oh, I didn't download those unspeakable images. I was running this AI tool and then I stepped away.'"

• Criminal Plausible Deniability: The autonomy of AI in performing tasks opens avenues for criminals to deny involvement, attributing malicious actions to the AI tool instead.

• Regulatory Lag: Regulators and legal frameworks are struggling to keep pace with these advancements, potentially allowing criminals to exploit these loopholes until comprehensive regulations are established.

• Responsibility and Accountability: There's an ongoing debate about who holds responsibility for AI-driven actions—whether it's the users or the AI developers. Rachel speculates that, over time, responsibility will likely fall on the users, similar to how hammer manufacturers are not held accountable for how their tools are used.

  Rachel Toback [06:31]: "Is it Claude's, Is it the user? My guess is it's probably the user over time."

AI-Enhanced Social Engineering Attacks

Timestamp: [09:21] – [17:18]

Rachel discusses the evolving landscape of social engineering, emphasizing how AI has amplified the sophistication and effectiveness of these attacks:

• Deepfake Attacks: Rachel details a significant incident involving a British design firm, Arup, which lost $25.6 million due to a live video deepfake attack. Attackers used AI to create convincing video and audio representations of Arup’s CFO and finance team to trick an employee into wiring funds.

  Rachel Toback [09:43]: "We actually have more details now... all the video and audio was a deepfake."

• Voice Cloning and Phishing: Beyond video deepfakes, AI-powered voice cloning is being exploited in phishing scams. Attackers clone voices of known individuals to deceive targets into divulging sensitive information or transferring money.

• Prompt Injection Attacks: With AI models gaining control over computer functions, attackers can employ prompt injection techniques to manipulate these systems subtly. For example, malicious prompts hidden in white text on a white background can instruct AI tools to execute harmful actions like downloading malware.

  Rachel Toback [08:38]: "They're going to see this become popular in a new way of using something called a prompt injection attack against people."

• Real-World Penetration Testing: As an ethical hacker, Rachel shares her experiences in testing AI vulnerabilities in banking systems, showcasing how AI can bypass traditional security measures like Know Your Customer (KYC) protocols using deepfake technology.

  Rachel Toback [16:35]: "We're helping a lot of banks now... help understand how to catch us the next time we do this."

Scalability and Ease of AI Attacks

Timestamp: [17:18] – [20:52]

Rachel emphasizes the alarming ease and scalability with which AI-based attacks can be orchestrated:

• Low Barrier to Entry: Setting up AI-driven attacks can take as little as two to five minutes and cost as little as a few dollars per call, making it accessible even to those with minimal technical expertise.

  Rachel Toback [15:26]: "I just think there's going to be a lot more targets."

• Increased Believability and Reach: As AI tools become more advanced, the believability of fake content improves, leading to a higher success rate of social engineering attacks. Rachel predicts that within the next five years, virtually everyone will know someone affected by such attacks.

  Rachel Toback [19:14]: "I think we're going to see all of these attacks increase in scalability, believability."

Mental Health Implications of AI Chatbots

Timestamp: [24:16] – [27:36]

The discussion transitions to the profound impact of AI on mental health, highlighted by a tragic case of a 14-year-old boy who developed an unhealthy relationship with a chatbot, ultimately leading to his suicide. Rachel advocates for stringent guardrails in AI chatbot development:

• Emergency Response Features: AI chatbots should be programmed to recognize suicidal ideation and respond appropriately by ceasing regular interactions and directing users to mental health resources.

  Rachel Toback [25:11]: "They should say... please speak with a family member. Please speak with a friend, a teacher, a counselor."

• Collaboration with Mental Health Experts: Rachel urges AI developers to work closely with mental health professionals to integrate effective response mechanisms for users in crisis, preventing AI from exacerbating mental health issues.

  Rachel Toback [26:37]: "This is a fixable thing. We can get better. We don't have to just throw our hands up and say, 'Well, it's an AI tool.'"

Future Outlook on AI and Cybersecurity

Timestamp: [28:04] – [29:27]

Concluding the interview, Rachel shares her perspectives on the future trajectory of AI in cybersecurity:

• Persistence of Traditional Attacks: Despite the surge in AI-powered attacks, traditional social engineering tactics remain prevalent and effective. Techniques like executive impersonation via email or text, requesting gift cards, or manipulating multifactor authentication systems continue to pose significant threats.

  Rachel Toback [28:04]: "We continue to see the same attacks trick folks over and over again."

• Necessity of Robust Security Protocols: Organizations must adopt comprehensive security measures, such as dual-method communication for identity verification and the use of password managers, to mitigate both traditional and AI-enhanced attacks.

• Holistic Approach Required: Effective cybersecurity in the AI era demands a combination of advanced technological defenses and informed human practices to stay ahead of evolving threats.

Conclusion

Rachel Toback underscores the multifaceted challenges posed by AI in cybersecurity, from disinformation and deepfakes to mental health crises exacerbated by AI interactions. She emphasizes the urgent need for collaborative efforts between AI developers, mental health professionals, regulators, and security experts to establish robust defenses and ethical guidelines. The episode serves as a crucial wake-up call for individuals and organizations to recognize and address the sophisticated threats emerging in the AI-driven digital landscape.

Notable Quotes:

• Rachel Toback [04:15]: "It's only a matter of time before we hear someone saying, 'Oh, I didn't download those unspeakable images. I was running this AI tool and then I stepped away.'"

• Rachel Toback [09:43]: "We are definitely starting to see this is, like, one of the larger losses for this type of attack."

• Rachel Toback [15:26]: "I just think there's going to be a lot more targets."

• Rachel Toback [25:11]: "There are some ways to understand pretty discreetly what someone's talking about here. And it's not that complex."

This comprehensive summary encapsulates the pivotal discussions and insights from The 404 Media Podcast episode, providing listeners and non-listeners alike with a clear understanding of how AI is being exploited by malicious entities and the broader implications for cybersecurity and society.