Researchers Warn of Extinction Risk From Advanced AI, Sparking Policy Debate
Eliezer Yudkowsky and Nate Soares published a book arguing that sufficiently advanced artificial intelligence could cause human extinction within months or a decade. The authors, both prominent in AI safety circles, were among signatories to a 2023 open letter calling for a six-month moratorium on certain AI research.
The book's title - "If Anyone Builds It, Everyone Dies" - signals the authors' assessment of the stakes. Their argument centers on uncertainties about how large language models will behave as they grow more capable and autonomous, a concern that runs through much of the AI safety research community.
The Competing Vision
The extinction-risk position contrasts sharply with accelerationist arguments that advanced AI will cure diseases and increase productivity. This divide shapes competing policy proposals: catastrophists advocate research moratoria and compute limits, while accelerationists push for faster development with lighter regulation.
Yudkowsky has previously recommended extreme enforcement measures, including destroying data centers by air strike to prevent unsafe AI development. The specificity of such proposals underscores how seriously some researchers view the risk.
What This Means for Practice
The debate influences real decisions about funding, red-teaming practices, and model release strategies at major labs. Policymakers and researchers now track formal proposals for international treaties, compute restrictions, and regulatory frameworks.
The technical uncertainties driving this debate - how models scale, how to specify rewards correctly, how capabilities generalize - remain unresolved. The book and broader discussion do not provide new empirical evidence settling these questions, but they do frame them as central to governance decisions.
For research teams, the practical question is how to weigh speed against safety measures. That calculation will likely shape hiring, resource allocation, and publication decisions across the field over the next several years.
Learn more about the technical foundations of this debate through AI Research Courses and Generative AI and LLM Courses.
Your membership also unlocks:
