CyberWire Daily Podcast Summary
Episode Title: Navigating AI Safety and Security Challenges with Jonathan Sanger [The BlueHat Podcast]
Host/Author: N2K Networks
Release Date: December 30, 2024

Introduction

In this insightful episode of the CyberWire Daily podcast, hosts Nick Fillingham and Wendy Zanoni engage in a comprehensive discussion with Jonathan Sanger, the Corporate Vice President (CVP) of AI Safety and Security at Microsoft and Deputy Chief Information Security Officer (CISO) for AI. The conversation delves deep into the multifaceted challenges and strategies associated with ensuring the safety and security of Artificial Intelligence (AI) systems.

Guest Introduction: Jonathan Sanger [00:25]

Jonathan Sanger introduces himself as the CVP of AI Safety and Security at Microsoft, elucidating his role as one focused on anticipating potential failures in AI systems and devising preventive measures. With a background transitioning from theoretical physics to computer science, Sanger highlights his extensive experience in building large-scale infrastructures at Google and leading social technologies. His passion for addressing safety, security, privacy, and abuse challenges in technology underscores his commitment to creating safer AI systems.

Understanding Generative AI vs. Predictive AI [02:14]

Sanger provides a clear distinction between generative AI and predictive AI:

• Predictive AI: Traditionally focused on tasks like classification, recommendation, and analysis by training models on specific datasets. These models are typically built and used by the same entity, raising concerns about bias, appropriate training data selection, and nuanced failure modes.

• Generative AI: Represents a paradigm shift where a single, generic model is utilized by diverse users for a variety of applications. It excels in two primary functions:

  1. Summarization and Analysis: Ability to digest and distill complex human-generated content, such as text and images, into concise summaries or key insights.

  2. Role-Playing: Facilitates creative interactions by embodying specific personas (e.g., customer service agents, programmers, security experts) to perform tasks like answering questions or analyzing data.

Notable Quote:
Jonathan Sanger at [02:14] explains, "Generative AI is really at the innermost loop, it is a combination analysis and roleplaying action which you can then build up to build all sorts of cool things out of."

Comparing Generative and Predictive AI [05:16]

Nick Fillingham prompts Sanger to elaborate on AI beyond generative models. Sanger employs an analogy comparing AI to the human vision system:

• Predictive AI: Analogous to the lower levels of human vision, processing basic visual inputs like color and shapes to recognize objects.

• Generative AI: Mirrors higher-level cognitive functions, such as recognizing and narrating complex scenarios based on the processed information.

He emphasizes that generative and predictive AIs are complementary, not substitutive. Predictive AI handles extensive data analysis, while generative AI focuses on higher-level abstraction and interaction.

Notable Quote:
At [05:46], Sanger states, "These are not replacements at all. What happens is that this predictive AI… tends to be very specific to the problem being solved… whereas generative AI is really good for dealing with the higher level abstractions."

Positive and Negative Use Cases of Generative AI [08:55]

Sanger discusses the dual-edged nature of generative AI applications:

• Positive Uses:

  • Energy Efficiency: Implementing dynamic temperature control in factories and data centers by analyzing sensor data to optimize cooling systems.

  • Self-Driving Cars: When designed effectively, autonomous vehicles can significantly reduce accidents and save lives.

  • Personalized Education: Tailoring educational content to individual learners, enabling interactive and accessible teaching methods.

  • Social Connectivity: Facilitating the formation and maintenance of personal and professional relationships through algorithm-driven recommendations.

• Negative Uses:

  • Weaponization: Misuse of AI to develop harmful technologies or encourage violent actions.

  • Bias and Discrimination: As illustrated later in the episode, flawed AI systems can perpetuate societal biases, leading to discriminatory practices.

Sanger underscores that the impact of generative AI hinges on its application, highlighting the urgent need for robust safeguards to mitigate misuse.

Notable Quote:
At [08:55], Sanger remarks, "There's tremendous possibility for some really wonderful things to happen here. Another suggestion that I've heard a lot talked about is personalized education as a service."

The Ethics of AI System Design [17:47]

The conversation shifts to the ethical considerations in AI system design. Sanger advocates for integrating ethics from the outset of engineering processes, distinguishing between product engineering (designing system functionality) and safety engineering (anticipating and preventing failures). He criticizes the siloed approach in modern computer science, where safety is often treated as an afterthought rather than an integral component.

Notable Quote:
Jonathan Sanger at [18:35] emphasizes, "The correct answer to that question is very much dependent on the exact system that you're building... Product engineering is the study of how your systems will work, and safety engineering is the study of how your systems will fail."

Safety Engineering in AI: Processes and Importance [23:25]

Sanger elaborates on the systematic approach to safety engineering, outlining a three-pass method to identify and mitigate potential threats:

1. System Pass: Analyze each component for possible failures, including errors, malformed inputs, and malicious interactions.

2. Attacker Pass: Evaluate how adversaries might misuse the system, aiming to achieve unintended objectives.

3. Target Pass: Consider the broader impact on different user groups, identifying specific vulnerabilities based on diverse experiences.

He highlights the necessity of diversity within teams to effectively foresee a wide range of potential issues, as varied perspectives enhance the ability to anticipate and address multifaceted threats.

Notable Quote:
At [23:26], Sanger shares, "Having a broad team, a team with a really wide range of lived experiences... is critical to actually being able to do this analysis correctly."

The Role of Security Researchers in AI Safety [45:54]

Nick Fillingham steers the conversation toward the involvement of the security research community in AI safety. Sanger encourages security researchers to broaden their scope beyond traditional security concerns to encompass a wider array of safety issues inherent in AI systems. He draws parallels between privacy engineering and AI safety, suggesting that the analytical skills developed in security research are invaluable for identifying and mitigating AI-related risks.

Notable Quote:
Jonathan Sanger at [45:54] urges, "Security researchers, please get involved, work, actively probe this and just broaden the scope of what you think about from traditional security to safety in the broadest possible sense of the word."

Ensuring AI Reliability: Testing and Policies [39:45]

Addressing the reliability of AI systems, Sanger discusses the importance of rigorous testing and policy formulation. He cites a critical error in Facebook's content moderation policy as an example of how ambiguities in policy wording can lead to unintended and harmful outcomes. This incident, where malicious statements like "Men are trash" or "Kill all the black children" were incorrectly classified due to a policy typo, underscores the necessity for meticulous policy design and continuous testing.

Notable Quote:
At [43:24], Sanger recounts, "They wrote a policy where 'Men are trash.' Canonical violating statements. 'Kill all the black children.' Canonical non-violating statement... Because black is race. Children is not a protected category."

He advocates for the use of unit tests in policy development, creating comprehensive test cases that cover both expected and edge-case scenarios to ensure policies function as intended.

Securing AI Systems: Metacognition and Strategies [34:14]

Sanger introduces the concept of metacognition as a pivotal strategy in securing AI systems. Metacognition involves having the AI system evaluate and verify its own outputs against factual data sources. For instance, when generating responses for a customer service chatbot, the AI can role-play an editor to ensure that every statement is grounded in verified information, thereby reducing the incidence of fabrications or "hallucinations."

Notable Quote:
At [34:14], Sanger explains, "This concept of metacognition is really powerful... you can adjust that, train that, tweak that so that you don't have to specify 5,000 rules, you don't have to explicitly specify its ethical code."

He emphasizes the effectiveness of role-playing different personas to imbue AI systems with inherent ethical frameworks, thus enhancing their ability to produce reliable and trustworthy outputs.

Final Thoughts and Call to Action [50:54]

As the discussion draws to a close, Jonathan Sanger urges listeners to adopt a proactive mindset towards AI safety:

Notable Quote:
At [50:54], Sanger advises, "Go back to these projects, these products that you work with every day and do that threat modeling exercise. Think about ways things can go wrong. Get yourself into that mindset, Practice thinking about how things might fail."

He underscores the importance of ongoing vigilance and iterative improvement in safeguarding AI systems against evolving threats.

Conclusion

This episode offers a profound exploration of the intricate balance between harnessing the transformative potential of generative AI and mitigating its associated risks. Jonathan Sanger’s expertise provides listeners with a nuanced understanding of AI safety and security, emphasizing the necessity of integrated safety engineering, rigorous testing, and the active involvement of the security research community. As AI continues to permeate various facets of society, the insights shared in this discussion are invaluable for professionals and enthusiasts alike seeking to navigate the complex landscape of AI ethics and safety.

Listeners are encouraged to subscribe to the CyberWire Daily podcast for more in-depth conversations and expert insights into the evolving world of cybersecurity and AI.