Afternoon Cyber Tea with Ann Johnson Episode: Dawn Song – When AI Becomes the Hacker and the Defender Date: February 17, 2026

Overview

In this episode, Ann Johnson speaks with Dr. Dawn Song, a leading researcher in AI security and privacy and Professor of Computer Science at UC Berkeley. The discussion centers on how rapidly advancing AI technologies are redefining cybersecurity—empowering both attackers and defenders—and what this dual-use dynamic means for enterprises, regulations, and the future of secure systems. Dawn shares insights from her pioneering research on adversarial AI, automation in bug detection and patching, privacy, data governance, and offers her vision for leveraging AI as a transformative tool in building provably secure systems.

Key Discussion Points & Insights

The Growing Importance of Communication & Trust in Cybersecurity

• Storytelling and Transparency as Foundations (01:09–03:21)
  • Ann sets the stage by noting cybersecurity’s shift beyond technicalities, emphasizing communication’s role in shaping risk understanding and trust.
  • Dawn Song: “Communication is really important... especially given how fast AI has been advancing… We hope through this type of communication we can help raise awareness in the broad community to both understand the upcoming risks and also to work together to build resilience and trust.” (01:34)
  • Example: A workshop on frontier AI in cybersecurity drew nearly 4,000 participants, highlighting the appetite for shared understanding.

AI’s Impact at Every Stage of the Cybersecurity Kill Chain

• AI in Detection, Threat Intelligence, and Automation (03:21–07:21)
  • Ann asks about near-term AI impacts in cyber defense.
  • Dawn explains that AI now affects "many different stages" of both attacks and defenses.
  • Her recent work, CyberGym, benchmarks AI’s capabilities to find vulnerabilities in roughly 200 open-source projects, detecting both known and zero-day vulnerabilities.
    • Improvements are rapid: “Even just last year … AI model capabilities weren’t quite there. However, fast forward for this year ... we are really seeing drastic increase.” (03:38)
    • Quantitative leaps: from 18% detection of known bugs with earlier models to 67% when running 30 trials with Anthropic’s most recent release.
  • Quote: “Anthropic has shown that … 30 trials on these benchmarks … the agents were able to identify and generate PoCs for close to 67% of previously known vulnerabilities.” (07:13)
  • Dawn’s team also saw AI uncovering and responsibly disclosing new, previously unknown vulnerabilities: “The agents actually were able to discover 35 zero days ... CVEs have been issued … developers have patched a number … after we reported.” (07:21)

Automation in Patching & the Dual-Use Dilemma

• How Quickly Can Defenders Respond? (08:03–10:38)
  • Ann probes for practical examples of patching automation.
  • Dawn references her projects like BountyBench, demonstrating AI’s advances in generating patches, significantly reducing human effort.
  • Dual-use dynamic: "There’s a key question, who will AI help more both in the near term and in the longer term? Attacker side or the defender side?... AI can help the attacker side more in the near term." (08:13)
    • Example: Patch deployment in hospitals averages nearly 500 days, giving attackers an upper hand even after a patch is available.
  • Looking ahead: Dawn is optimistic AI can eventually help defenders more through automated program verification—eliminating vulnerability classes altogether.

Adversarial AI: Threats, Attacks, and Defensive Strategies

• Adversarial Machine Learning (10:38–13:48)
  • Ann highlights Dawn’s seminal work in this field.
  • Dawn: Her group was among the first to show that “deep learning models ... are very vulnerable to these adversarial attacks ... attackers can just change the inputs ... and this can actually cause the model to give wrong answers.” (10:57)
  • Artifacts from her research are now displayed in the London Science Museum.
  • With the advent of agentic AI—models with privileges to act on data, make decisions, interact with sensitive enterprise resources—attackers can “construct malicious attacks ... jailbreaks, prompt injections, ... data poisoning … they can really cause the agents to misbehave and ... take wrong actions.... leak sensitive information ... even delete databases.” (12:50)

Data Hunger vs. Data Minimization: Privacy and AI

• Risks of Memorization & Extraction (13:48–18:02)
  • Ann discusses the tension between AI’s appetite for data and organizations’ push for minimization and privacy.
  • Dawn’s research was among the first to show large language models are susceptible to memorizing sensitive data—“an attacker that doesn’t even know the details of the trained model ... by just querying ... can actually extract sensitive information in the training data.” (14:28)
  • She details mitigations: differential privacy, synthetic data, reinforcement learning.
  • Quote: "With high quality synthetic data ... it can help both increase the model utility and while mitigating the risks for data privacy." (16:50)

Regulation: The Push for Science- and Evidence-Based Policy

• Fragmentation and the Need for Transparency (18:02–20:25)
  • Ann prompts on the regulatory landscape.
  • Dawn discusses her involvement with a proposal for “A Path for Science and Evidence Based AI Policy,” and emphasizes the value of empirical benchmarks in shaping regulatory approaches.
  • Quote: "The AI policy needs to be grounded in science and evidence…. transparency is the first step." (19:47)
  • Their work helped inform the California bill SB53, which aims to increase transparency in AI’s capabilities and risks.

AI as Cybersecurity Optimist: Towards Secure-by-Design Systems

• Long-term Prospects: Provable Security (20:25–24:38)
  • Ann closes with a call for optimism about AI’s future in security.
  • Dawn: “I do think that AI itself actually can also help us a lot ... to help us to build more secure systems.” (20:46)
  • She outlines a vision for AI-enabled formal verification—moving from today’s “cat and mouse game” of patching bugs to a world where AI helps generate, specify, and prove code is secure by design.
    • “We can now … do what we call verifiable code generation where we can actually generate code that also has provable security.” (23:02)

Notable Quotes & Memorable Moments

• On trust and resilience:
  "We hope through this type of communication we can help raise awareness in the broad community to both understand the upcoming risks and also to work together to build resilience and trust."
  — Dawn Song (01:34)

• On AI outpacing past limits:
  “Even just last year … AI model capabilities weren’t quite there. However, fast forward for this year ... we are really seeing drastic increase.”
  — Dawn Song (03:38)

• AI’s dual-edged nature:
  “AI can help the attacker side more in the near term … deploying a patch even after it becomes available takes close to 500 days.”
  — Dawn Song (08:13)

• Adversarial AI stakes:
  “As we adopt and deploy AI… the consequences of these attacks can be much more severe.”
  — Dawn Song (13:48)

• On science-led policy:
  “The AI policy needs to be grounded in science and evidence…. transparency is the first step.”
  — Dawn Song (19:47)

• Long-term optimism:
  “Instead of just doing this bug finding and patching this cat and mouse game, we can actually develop systems with provable guarantees, with formal verification … Ultimately, using this approach, AI can really help us to create probably secure systems and really help defenders to win over attackers in the longer term.”
  — Dawn Song (23:14)

Key Segment Timestamps

• Communication, Trust & Awareness: 01:09–03:21
• AI Capabilities in Cyber Vulnerability Discovery: 03:21–07:21
• Patch Automation & Dual-Use Risks: 08:03–10:38
• Adversarial Machine Learning: 10:38–13:48
• Data Privacy & Mitigations: 13:48–18:02
• Regulatory Landscape & Science-led Policy: 18:02–20:25
• Secure-by-Design & AI Optimism: 20:25–24:38

Tone & Style

The episode is conversational yet deeply technical, reflecting Ann Johnson’s expertise as an executive and Dawn Song’s leadership in academic and applied research. The speakers balance realism about risks with a focus on tangible, actionable solutions and a notable sense of optimism for AI’s constructive potential in cybersecurity.

Summary for New Listeners

This episode captures the exhilarating, unsettled frontier where AI’s growing powers are reshaping cybersecurity, for both attackers and defenders. Dr. Dawn Song translates cutting-edge research into insights for enterprise leaders: AI is already finding and even fixing vulnerabilities at scale, but its dual-use nature presently helps attackers more than defenders. However, with deliberate focus on automation, privacy-by-design, regulation built on scientific benchmarks, and the pursuit of verifiable security, AI holds the promise to one day tip the balance in favor of defenders. If you want to understand both the urgency and promise of AI in cybersecurity, this episode is essential listening.