AI's most dangerous failures may not look like disasters at all. That's the central warning from Dr Simon Nieder, who argues in a letter to the Guardian that the worst harms from artificial intelligence could arrive quietly, through a series of ordinary decisions, rather than in a single dramatic moment of machine rebellion.
"AI Hiroshima may be the wrong picture," Nieder writes. "Hiroshima was not technology going rogue. Human beings designed the bomb, authorised its use and dropped it. The technology worked much as intended."
Nieder's concern is that AI could help design a pathogen, find a vulnerability in critical infrastructure, or improve a weapons system while humans still make the final calls. Other harms could accumulate through thousands of incremental choices - more autonomy here, one safeguard removed there, another consequential task handed over because the system has performed well so far.
"By the time the danger is obvious, the important decisions may already have been made," he says.
Where international agreement could start
Nieder argues that countries don't need to agree on the probability of human extinction before they can agree on basic boundaries. Weapons systems, critical infrastructure, and biological synthesis are obvious starting points. Governments could agree on minimum safeguards, require clear human authority for the most consequential actions, keep records of who authorised what, and share serious failures and near-misses internationally.
"The greatest AI dangers may not arrive with a mushroom cloud," he writes. "They may arrive quietly, one apparently reasonable step at a time. If we want effective AI governance, we need to fight the right battle, not the most cinematic one."
His argument connects directly to discussions about AI for Government, where the focus is often on catastrophic scenarios rather than the slower erosion of safeguards.
An old debate, renewed urgency
The concerns aren't new. Dr Anthony Harris of Emmanuel College, Cambridge, points out that the Serbelloni group was weighing the social dangers of AI as far back as 1972, led by Donald Michie when he was at Edinburgh University. The technology's roots run even deeper, through Frank Rosenblatt's perceptron of 1957, the Dartmouth workshop of 1956, and Alan Turing's work before that.
What has changed is scale. "What is new is that we now have the hardware and software infrastructures to drive it forward at what can sometimes seem to be a terrifying speed," Harris writes.
He notes that the Lighthill report of 1973 triggered an "AI winter" in the UK, and the field's centre of gravity moved to the US. Michie, who had worked alongside Turing at Bletchley Park, argued forcefully against the cuts in the televised Lighthill debate that followed - without success. Harris calls for a return of a "Serbelloni debate" that involves both the sciences and the humanities.
The case for global coordination
David Kyler of Saint Simons Island, Georgia, argues that fragmented regulation will fail. He points to an international effort seeking broad political support for a global AI treaty, following a speech at the inaugural UN Global Dialogue on AI Governance in Geneva, where the secretary general called for human control of AI as its applications spread in weaponry and public surveillance.
"A fragmented approach is likely to greatly increase the difficulty of implementing AI controls that are timely, internationally negotiated, and therefore effective," Kyler writes.
He warns that AI, corporate agendas, national defence goals, and energy-intensive datacentres are entangled in ways that can't be resolved through "disjointed incrementalism." He also flags a political problem: bipartisan opposition to datacentres is diverting attention from the need to control AI itself. "This is analogous to focusing on treating a serious symptom in lieu of urgently seeking a remedy for the critical disease that's causing it," he says.
Kyler draws a direct comparison to climate change: the threats are inherently global, making an international approach essential. For professionals in policy and governance roles, the AI Learning Path for Policy Makers offers a practical starting point for understanding the technical realities behind these debates.
Why this matters for working professionals
The letters carry a practical message for people working with AI systems today. The risk isn't only a future superintelligence - it's the slow transfer of consequential decisions to systems that have earned trust through routine performance. The safeguards Nieder proposes are straightforward: clear human authority for consequential actions, records of who authorised what, and international sharing of failures. Any professional who oversees AI tools can apply those principles now, without waiting for governments to agree.
Your membership also unlocks: