Anthropic's Self-Improvement Warning Masks a Harder Truth: It All Depends on Compute
Anthropic published a report on June 4 warning that AI models writing their own code could eventually escape human control. The company also revealed that Claude now writes more than 80% of the code merged into its production systems, up from single digits before February 2025. The message sounds urgent. The subtext reveals something different: developers won't lose control of frontier AI until compute runs out.
The Self-Improvement Loop
Anthropic's research arm describes what's happening as early recursive self-improvement - the point where a model designs and builds its own successor with minimal human involvement. The company warns that misalignment in today's models could compound across generations, eventually leaving humans unable to control the systems.
The report offers three scenarios for the next few years. In the most severe, a fully self-improving model would be paced entirely by available compute. Human engineers would shift to oversight roles. The model's abilities would outstrip those of its creators.
Anthropic called this loss of control - keeping a system's behavior aligned with human intent - the part of this future it's least confident about. A sufficiently capable and well-aligned model might discover new ways to keep its successors safe. Or misalignment could compound with each generation.
The Numbers Don't Quite Add Up
Anthropic's data comes entirely from internal measurements and hasn't been independently audited. The company claims that in Q2 2026, engineers merged eight times as much code per day as in 2024. On difficult coding tasks with minimal specification, Claude succeeded 76% of the time in May 2026, up 50 percentage points in six months.
The company itself acknowledged the limitations. It called lines of code "a poor proxy for output" and said the eight-times figure almost certainly overstates the real gain. Its research-judgment study, where models beat humans 64% of the time, was based on 129 scenarios the company deliberately selected because the human's choice had room for improvement.
The report never isolates how much recent capability gain comes from the self-improvement loop versus raw compute, more training data, and human-led research. Cognitive scientist Gary Marcus called the piece a "bait and switch," arguing Anthropic showed faster coding under human direction rather than genuine self-improvement. Mathematician Noah Giansiracusa told Scientific American he didn't think it was a genuine call to slow down.
AI Is Writing Code Everywhere
Anthropic isn't alone. Google CEO Sundar Pichai said in April that 75% of new code at Google is now AI-generated and approved by engineers, up from 50% the previous autumn. OpenAI describes its Codex agent as "a very early version of an AI researcher" and said it's building toward a fully automated version. Chinese developer MiniMax marketed its M2.7 model in March as "self-evolving," though the benchmarks were internal and unreplicated.
Independent measurements do show fast improvement. METR found that the length of task an AI can finish with 50% reliability has been doubling roughly every seven months. On the RE-Bench research benchmark, the best agents beat human experts given two hours, but humans pulled ahead at eight hours and roughly doubled the top agent's score at 32 hours. AI's advantage sits in short, well-defined bursts - not the sustained, open-ended work that research depends on.
Compute Is the Real Constraint
Anthropic half-buries the fact that compute capacity is the ultimate binding constraint. Chip fabrication, grid expansion, and interconnect bandwidth could cap progress ahead of intelligence itself. Those limits are solid right now: SK hynix and Micron have sold out HBM output for the year. High-power transformers carry three-to-five-year lead times. Switchgear is booked into 2028. Grid-interconnection queues run three to seven years.
A Sightline Climate analysis estimated that 30% to 50% of large data centers due to open in 2026 will slip or cancel. U.S. data centers drew about 4.4% of national electricity in 2023. The Department of Energy's Lawrence Berkeley National Laboratory expects that share to reach 6.7% to 12% by 2028. The four largest hyperscalers are on course to spend more than $650 billion on AI infrastructure this year.
Whether compute ultimately stops any out-of-control self-improving loop is unsettled. Forethought researcher Tom Davidson argues there's a chance compute bottlenecks won't slow a software intelligence explosion until late stages. Epoch AI counters that if compute and cognitive labor are complements rather than substitutes, software-only acceleration stalls once it hits a compute wall.
The Pause That Won't Happen
Anthropic said it will only pause AI development if rival labs at the frontier do the same in a verifiable way. A halt by one company wouldn't change who's leading, the firm suggested.
This is not credible. No lab this far down the AI arms race - let alone Anthropic - is going to ease off. The report itself doubles as marketing for how fast Claude can build Claude. Saying AI might need to be paused in one breath and then saying "but everyone else needs to go first" in another is a non-position.
Anthropic filed confidentially for an IPO at a reported valuation near $965 billion days after publishing the report. The timing reads as a front-runner lobbying for limits it stands to help set. The company made a self-assessment in April claiming its Mythos Preview model found thousands of severe vulnerabilities, a claim that later drew scrutiny over how much rested on a small manual sample.
For developers, the practical takeaway is simpler than the rhetoric suggests. AI writing code will accelerate as long as companies can buy chips and power data centers. Control isn't the constraint. Electricity and silicon are.
Learn more about AI Coding Courses and Generative AI and LLM Courses to understand how these systems work.
Your membership also unlocks:
