AI news ·
Anthropic calls on AI companies to coordinate a potential pause in advanced AI development
Anthropic is urging top AI companies to build a coordinated system for pausing advanced AI development if risks grow unmanageable. The proposal puts Anthropic at odds with OpenAI, which says governments-not private companies-should set the rules.
Anthropic Proposes Industry Pause Mechanism for Advanced AI Development
Anthropic is calling on the world's top AI companies to establish a coordinated system for pausing advanced AI development if risks become unmanageable. The company, which makes the Claude chatbot, outlined the proposal Thursday in a blog post warning that AI systems are improving faster than the industry can safely manage.
Jack Clark and Marina Favaro, who lead Anthropic's research institute, wrote that as AI models accelerate their ability to perform tasks independently-including writing code-"it would be good for the world to have the option to slow or temporarily pause" development. Anthropic said its research team plans to collaborate with others to build credible systems for such slowdowns, though the company did not specify implementation details.
The Self-Improvement Problem
Anthropic flagged a particular concern: AI systems could eventually design and develop their own successors through what researchers call recursive self-improvement. This milestone would bring benefits in science and healthcare, the company acknowledged, but could also increase the risk of humans losing control over AI systems.
The proposal comes as a separate team at the University of Toronto demonstrated how AI tools could create self-adapting "worms" that spread across computing networks and take over infrastructure. Nicolas Papernot, the lead researcher, said the findings show that security risks extend beyond only the largest language models.
Disagreement on Who Should Decide
Anthropic's call for industry coordination contrasts with OpenAI's position, released Wednesday. OpenAI argued that "democratic governments - not private companies acting alone - must ultimately determine the rules, safeguards and accountability mechanisms" for AI development. The company said decisions about the pace of AI innovation should not rest with any single lab or company.
For IT and development professionals, understanding these competing approaches matters: one emphasizes voluntary industry coordination, while the other delegates control to government bodies. Both acknowledge the need for oversight as generative AI and large language models continue advancing rapidly.
The debate reflects broader questions about how the industry should govern itself as AI capabilities expand and potential risks become harder to predict.