Anthropic calls for coordinated pause in advanced AI development
Anthropic is proposing that the world's leading AI companies establish a coordinated mechanism to slow or temporarily halt development of advanced systems, citing risks that humans could lose control as the technology accelerates.
The company behind Claude said in a blog post Thursday that AI models are improving so rapidly - particularly in speed and autonomous task execution - that a global pause option would benefit society. Anthropic plans to explore how such a slowdown could work and what systems would verify that companies actually comply.
The proposal directly contradicts OpenAI's position, released Wednesday. OpenAI argues that democratic governments, not private companies, should set the rules and pace of AI development. "Decisions about the pace of AI innovation should not be left to any one lab, company, or special interest group," OpenAI said.
The recursive self-improvement problem
Anthropic's concern centers on a specific capability: AI systems that could design and develop their own successors, known as recursive self-improvement. Given sufficient computing power and current development trends, this milestone could arrive within years.
Such a system would accelerate scientific discovery and healthcare advances, Anthropic acknowledged. But it would also "increase the risks of humans losing control over AI systems."
Jack Clark, Anthropic cofounder, and Marina Favaro, head of its research institute, wrote that a pause would create space for "societal structures and alignment research" to keep pace with technical progress. Alignment refers to ensuring AI systems match human values and intentions.
New security vulnerabilities emerge
Anthropic's call comes as University of Toronto researchers demonstrated how AI tools can create adaptive "worms" that spread across computing networks, evolving their hacking strategy as they move from device to device.
Lead researcher Nicolas Papernot said the findings underscore that "it's not just the biggest, most powerful language models that pose the security concerns." The team built the worm using publicly available AI tools, showing how easily malicious actors could replicate the approach.
Papernot noted that older, overlooked devices now pose significant risk. "That old laptop you have in your basement that you don't check on regularly doesn't seem like a very high-value target, but it can be used as a launch pad to attack these higher-value targets," he said. "Anything connected to the internet is now at risk because of how low the cost has become to mount these cyberattacks."
Regulation lags behind capability
Regulation of advanced AI remains slow, particularly in the US, where most leading labs operate. A Trump administration executive order this week asked AI companies to voluntarily submit their most capable models for government cybersecurity testing before public release - placing responsibility on the labs themselves.
Anthropic has positioned itself as safety-focused. Earlier this year, it refused to allow the US military to use its models for domestic surveillance and fully autonomous weapons, drawing backlash that placed the company on a national security blacklist set to take effect later in 2026.
Previous calls for AI development pauses have gained little traction. Elon Musk backed a 2023 effort by the Future of Life Institute to halt development for six months, which failed to gain industry support.
Coordination challenges
Anthropic's proposed verification mechanism would let labs confirm that competitors have actually slowed work and prevent "a bad actor" from using a coordinated slowdown as cover to advance secretly.
The company argues that without coordination, the "least cautious" players could catch up during a slowdown, pressuring other companies and governments to abandon safety measures.
Papernot said collaboration between companies, government agencies, and academic researchers is essential to develop defenses against AI-powered hacking tools. His team notified Canadian cybersecurity authorities before releasing its research.
Anthropic and OpenAI are both preparing to sell shares in initial public offerings that could value Anthropic at nearly a trillion dollars.
Your membership also unlocks:
