AI is transforming how organizations operate, but its greatest risk may not be technical failure. The real danger is that it succeeds just enough to make people stop thinking - setting off a compounding loop that degrades both the machines and the humans who depend on them.
When AI systems begin learning primarily from other AI outputs rather than from reality, predictive performance degrades over time. Errors get reinforced rather than corrected. Recent research published in Nature gave this phenomenon a name: model collapse. The 2024 study demonstrated that generative models trained predominantly on AI-produced data begin to "forget" the true underlying data distribution. Outputs become progressively narrower, less accurate, and the rich variability of real-world data disappears over successive generations.
"Indiscriminate use of model-generated content in training causes irreversible defects in the resulting models," the Nature authors warned, leading to an irrecoverable loss of knowledge from original human data. In plain terms, when AI learns mostly from AI, it inherits previous models' mistakes and loses touch with reality.
The recursive loop and competitive lock-in
Why would organizations allow AI systems to primarily talk to each other? Often, it is an unintended consequence of competitive dynamics. In markets where one firm adopts AI for pricing, investment, or trading decisions, rivals face pressure to follow suit or risk being outpaced. The outcome is a Nash equilibrium: no single firm can unilaterally drop its AI without harming its payoff. It becomes an arms race.
This is not hypothetical. By the late 2010s, algorithms executed the majority of trades in many stock markets. The May 6, 2010 Flash Crash illustrated the stakes - a few algorithms misread routine signals and initiated a rapid sell-off. Other algorithms responded in kind, and within minutes the U.S. stock market plunged nearly 1,000 points, erasing roughly $1 trillion in value before human intervention helped correct it.
SEC Chair Gary Gensler later warned that when many actors rely on "the same underlying model or data," a small glitch can "cause a slight downturn to become a rapid collapse." The strategic takeaway is that adopting AI introduces systemic risks no single firm can mitigate alone. Industry-wide coordination or proactive regulatory engagement may be the only rational escape from a race that collectively degrades outcomes for all players.
Cognitive atrophy and the costs of outsourcing thinking
AI over-reliance is not solely a technological problem - it is driven by human psychology. People tend toward the path of least cognitive resistance. Daniel Kahneman described this as the tension between fast, automatic thinking (System 1) and slow, effortful analysis (System 2). AI tools present an enormous temptation for the "lazy" System 2: why struggle through complex analysis when an assistant can deliver a quick answer?
A study from MIT researchers found that participants using AI to write essays showed significantly less brain activity on EEG scans than those writing on their own. Their brain patterns had weaker connectivity in regions tied to creativity and memory. Tellingly, 80% of the AI-assisted group could not quote a single full sentence from the essay they "wrote." The researchers called this the cognitive cost of relying on AI.
The same study found that when people use AI, their solutions start to look the same. Diversity of thinking erodes. One researcher noted that with the language model's help, "you have no divergent opinions being generated… Average everything everywhere all at once." A separate Cornell study observed that writers from different cultural backgrounds produced increasingly similar, Western-norm-aligned responses when using AI auto-complete tools. For businesses, this raises a pressing concern: over-reliance on AI could stifle the outlier ideas and creative sparks that drive innovation.
At the organizational level, employees who stop honing expertise because "the AI will figure it out" accumulate managerial and technical debt. If a supply chain AI breaks or encounters a scenario outside its training, will any humans still understand the intricacies well enough to step in? If executives lean on AI analyses without independently understanding market drivers, the organization may lose the strategic intuition vital in crises or novel situations.
Bonhoeffer's warning and organizational passivity
Long before modern AI, theologian Dietrich Bonhoeffer wrote from prison in 1943 about a form of stupidity that was "a more dangerous enemy of the good than malice." It was not a matter of low IQ but of moral and intellectual passivity - a surrender of independent thought so complete that the person becomes immune to reason. The mechanism of passivity is the same in the AI context: external authority substitutes for internal judgment.
The organizational risk extends beyond the office. Content farms generating AI-written news, social media flooded with bot-generated posts, and machine-curated information feeds are already shaping the opinions of customers, regulators, and markets. A workforce that has grown intellectually passive internally is poorly equipped to recognize or respond to AI-driven narrative manipulation externally.
The remedy is what leaders must cultivate: independent thought and the courage to question. Encouraging teams to ask "Why?" even after the model has spoken, treating AI outputs as propositions to evaluate rather than verdicts to execute, and rewarding the analyst who pushes back - these are not inefficiencies. They are guardrails against the learned helplessness that leaves organizations defenseless against folly.
Reclaiming autonomy through human oversight
Mitigating these risks is an ethical and leadership challenge, not just a technical one. AI models learn from historical data and excel at pattern recognition within familiar domains. But they struggle with novelty, have no innate grasp of causality or context, and cannot weigh right versus wrong the way a human conscience would. An over-reliance on AI can reinforce past biases and blind spots. Real-world cases include AI recruitment systems that had to be scrapped after they were found to unfairly disadvantage certain groups because training data reflected historical hiring imbalances.
Maintaining human-in-the-loop governance is critical. Organizations should set boundaries on AI autonomy: use AI to flag fraud cases but require human review before action; use AI in medical diagnosis as a second opinion, not the sole voice. Requiring a human stamp of approval on important decisions keeps people mentally engaged with the process and preserves accountability. When algorithms trade with algorithms, responsibility can become diffuse - "the AI recommended it" is not an acceptable defense when decisions affect stakeholders' lives.
Technical mitigations are being developed: synthetic data filtering, Reinforcement Learning from Human Feedback (RLHF), constitutional AI approaches, and watermarking techniques. But a model that is technically well-governed can still operate inside an organization that has lost the capacity to question it. Technical guardrails and human ones must be built in parallel.
Why this matters for executives and strategy
The firms that gain the most from AI will be those least dependent on it - not because they use it less, but because they never stopped building the human capacity to question it. In a world where every competitor runs the same models on the same data, AI becomes infrastructure: necessary but not differentiating. What differentiates is the quality of judgment layered on top of it.
Executives should ask not "How much AI should we use?" but "Where does human judgment still have an edge, and are we actively protecting it?" Map your critical decision points, identify where AI currently operates without meaningful human challenge, and treat those as exposure rather than efficiency. For leaders seeking to build this capability systematically, AI for Executives & Strategy resources and a structured AI Learning Path for CEOs can help develop the frameworks needed to govern AI without surrendering the judgment that remains the organization's real competitive advantage.
Your membership also unlocks:
