Summary of https://arxiv.org/pdf/2401.07836

Examines two contrasting hypotheses regarding existential risks from artificial intelligence. The decisive hypothesis posits that a single catastrophic event, likely caused by advanced AI, will lead to human extinction or irreversible societal collapse.

The accumulative hypothesis, conversely, argues that a series of smaller, interconnected AI-induced disruptions will gradually erode societal resilience, culminating in a catastrophic failure. The paper uses systems analysis to compare these hypotheses, exploring how multiple AI risks could compound over time and proposing a more holistic approach to AI risk governance. Finally, it addresses objections and discusses implications for long-term AI safety.

The provided paper challenges the conventional view of AI existential risk (x-risk) as sudden, decisive events caused by superintelligent AI, proposing instead that AI x-risks can accumulate gradually through interconnected disruptions. This alternative, the "accumulative AI x-risk hypothesis," suggests that seemingly minor AI-driven problems can erode societal resilience, leading to a potential collapse when a critical threshold is crossed. Here are some of the most interesting points:

Two Types of AI Existential Risk: The paper contrasts two hypotheses:
- Decisive AI x-risk is the conventional view where a superintelligent AI causes an abrupt, catastrophic event leading to human extinction or irreversible societal collapse. This is often exemplified by scenarios like the "paperclip maximizer," where an AI with a simple goal causes unintended harm through its pursuit of instrumental sub-goals.
- Accumulative AI x-risk posits that x-risks emerge from the gradual accumulation of smaller AI-induced disruptions. These risks interact and amplify each other over time, weakening critical societal systems until a trigger event causes collapse. This is likened to the slow build-up of greenhouse gasses leading to climate change.

The "Perfect Storm MISTER" Scenario: The paper introduces a thought experiment where multiple AI-driven risks converge. This scenario is meant to illustrate how different types of AI risks (Manipulation, Insecurity threats, Surveillance and erosion of Trust, Economic destabilization, and Rights infringement) can interact and create a catastrophic outcome. It posits a 2040 world with pervasive AI, where vulnerabilities are exploited through manipulation, cyberattacks, and surveillance. This leads to a collapse of critical systems and social order, highlighting how a perfect storm of AI-related issues can cause an existential crisis.
- The MISTER scenario details how AI manipulation erodes public trust and discourse, how IoT device insecurity leads to cyberattacks, how mass surveillance erodes trust and democratic norms, how economic destabilization arises from job losses and market fragmentation, and how rights infringement becomes widespread.

Systems Analysis: The paper uses a systems analysis approach to understand how AI risks propagate. It highlights that systems are defined by their components, their interdependencies, and their boundaries. The analysis traces how initial perturbations, like a software bug or a manipulation campaign, can spread and amplify through networks, leading to catastrophic transitions at critical thresholds. The paper also examines three critical subsystems—economic, political, and military—and how AI impacts these.
Divergent Causal Pathways:
- The decisive pathway assumes a single cause, a misaligned superintelligence, as the source of catastrophic risk. It suggests a unidirectional cascade of effects throughout the interconnected world as the ASI pursues its goals.
- The accumulative pathway describes multiple AI systems causing localized disruptions that interact and amplify through interconnected subsystems, creating a complex causal network.

Reconceptualizing AI Risk Governance: The paper argues that the accumulative risk hypothesis requires a shift in AI governance, moving beyond just focusing on the risks of superintelligent AI. It calls for distributed monitoring systems to track how multiple AI impacts compound across different domains and also calls for centralized oversight for advanced AI development. This suggests a need to unify the governance of social and ethical risks with that of existential risks.
Unifying Risk Frameworks: The paper criticizes the fragmentation of AI risk governance, where different types of risks are addressed separately. It suggests that the accumulative risk perspective can help bridge these fragmented approaches by highlighting how various risks interact. It argues for a more holistic approach that integrates ethical and social risks with existential risk considerations.
Challenges and Future Work: The paper notes that several questions warrant further investigation, such as better methods for identifying when disruptions become critical, structured approaches for analyzing how risks accumulate, and new methods for quantifying accumulative risks. Future work includes developing computational simulations using system dynamics to further explore the accumulative hypothesis.