Law Firms Face Growing AI Hallucination Problem They Can't Verify
Sullivan & Cromwell, one of America's largest law firms, apologized to a federal judge this week for filing briefs containing citations that never existed. The firm blamed inadequate oversight of AI tools, but the real problem runs deeper: the partnership model that makes law firms profitable is fundamentally incompatible with how AI fails.
A database tracking AI errors in legal work contained roughly 90 entries a year ago. It now has 1,333. Most involve solo practitioners and small firms, but Sullivan & Cromwell's mistake shows the problem has reached the top tier.
The Verification Crisis
Law firms operate on a simple economic model. Partners supervise associates. The more associates each partner oversees, the higher the firm's profits per partner. This works because associates know their limits-they flag uncertain work for senior review, signaling where expert eyes are needed.
AI breaks this system. Unlike human associates, AI tools express no uncertainty. A language model trained on benchmarks that reward confident answers will never say "I don't know." It simply invents citations, case names, and legal principles that sound plausible.
Legal AI vendors have claimed to solve this. LexisNexis promised "100% hallucination-free linked legal citations." Reuters said its system "dramatically reduces hallucinations to nearly zero." Stanford University tested them. The three leading products hallucinate between 17% and 33% of the time. Westlaw's AI-Assisted Research invented an entire paragraph of the Federal Rules of Bankruptcy Procedure to support a point the Supreme Court has rejected.
Partners now face a different verification task: check everything instead of checking selectively. The volume of work has exploded. The time available to partners has not.
Three Structural Problems
Incentive misalignment. Partners own their firms. Productivity gains from AI show up immediately in quarterly metrics. Costs from a missed hallucination may not surface for years, if ever. This creates a constant temptation to prioritize speed over caution.
Compensation mismatch. Firms reward visible output. A partner who catches three hallucinations before they reach a client looks less productive than a partner who pushes AI-assisted work through faster. Over time, firms will promote the partners most enthusiastic about AI, not the ones best at managing its risks.
The expertise gap. Partners evaluating AI work are typically less fluent with the tools than the associates using them. The usual quality-assurance intuition-that senior people outperform junior ones at every task-fails here. Partners can't reliably spot what the software got wrong.
First-Mover Disadvantage
The firms that will actually capture AI's productivity gains are the ones willing to look worse in the short term. They'll invest in verification infrastructure. They'll move deliberately. They'll hire more senior staff to handle greater review burdens, which means lower profits per partner.
This pattern has occurred before. When corporations adopted IT systems in the 1990s, the firms that invested in organizational redesign looked sluggish at first. They ended up ahead. The ones that bought technology without rethinking around it spent a decade looking productive before falling behind.
Electricity took about 40 years to produce the factory redesigns necessary for productivity gains. Professional service firms will need to rebuild their verification function from the ground up-likely combining new AI tools, different staffing models, and retraining for both associates and partners.
They can do this proactively or wait for a high-profile failure to force their hand. Sullivan & Cromwell just showed what the latter looks like.
Learn more about AI for Legal work and how professionals are managing these tools, or explore an AI Learning Path for Paralegals to understand verification strategies in practice.
Your membership also unlocks:
