A new working paper from the University of Chicago's Harris School of Public Policy finds that competition between AI firms-rather than producing safer outcomes-may push them to sacrifice safety in favor of speed. The study, which models a race to develop artificial general intelligence (AGI), arrives as policymakers scramble to balance innovation incentives against the risk of catastrophic failure.
The Competitive Pressure to Skip Safety
Researchers Ethan Bueno de Mesquita, Wioletta Dziuda, and Mattias Polborn built a model where firms believe huge rewards go to whoever reaches AGI first. Each firm must divide resources between moving fast and building safety measures. As the number of competitors grows, so does the share of resources each pours into speed. More firms, the model shows, mean faster development-and a higher chance of harmful outcomes.
"The more firms you have, the riskier the race becomes as the firms try to outpace one another," said Dziuda, an associate professor at Harris. "Our model challenges the assumption that more competition necessarily produces better outcomes. While competition often benefits consumers and spurs innovation, we show that in a race where being first carries enormous rewards, competition can also create incentives to cut corners on safety."
Why Firms Struggle to Slow Down
The paper describes a classic collective-action problem. Even when individual companies would prefer a slower, safer path, no single firm can ease off without losing ground. The dynamic gets more counterintuitive still: In some scenarios, firms keep racing even when the expected value of achieving AGI turns negative.
"Why would they still race?" Dziuda said. "Because other firms are racing. If a catastrophic outcome occurs, whether they're in the market or not, they're affected anyway. So, they may as well participate and hope to be the one that wins."
The finding illuminates a real-world tension in which AI company leaders issue public warnings about existential risk while at the same time investing billions to advance their own systems.
Policy Levers and Unintended Consequences
The researchers examined several interventions, including limits on computing resources, industry consolidation, public investment, and direct government participation in AI development. Their conclusions are not always intuitive. Restricting key inputs, for instance, doesn't work across the board. In markets with fewer players, giving firms more resources can actually improve safety by letting developers fund safeguards alongside new capabilities.
"We discovered that the answer isn't always to tax or restrict resources," Dziuda explained. "It can be a combination of policies that seek to limit the number of competitors and then offer support so that they have room to pursue both safety and new capability."
Bueno de Mesquita, dean and Sydney Stein Professor at Harris, said the model also re-frames why companies might endorse industry-wide regulation. "In some circumstances, firms may support regulation because common constraints reduce pressure to sacrifice safety in order to keep pace with rivals."
A publicly backed AI effort that prioritizes safety over speed, the paper notes, could produce a safer development trajectory while nudging private competitors toward more cautious behavior-an approach Switzerland is now advancing.
Why this matters for IT and Development
For engineers, architects, and technical leads who integrate AI components into products, the study supplies a clear warning: market dynamics may already be compressing safety margins. Understanding these incentives can help teams advocate for rigorous testing, redundancy, and kill-switch protocols even when business timelines push for acceleration. The growing body of work on safe AI development, covered in resources like AI for IT & Development, gives practitioners frameworks for building those safeguards into real systems.
Your membership also unlocks:
