Science Is Drowning in AI Slop - And Peer Review Is Struggling
A respected reviewer opens a manuscript, scans the references, and finds a citation to his own "paper." The title looks plausible. The co-authors look familiar. It doesn't exist.
That's not a fringe issue anymore. Journals you'd proudly publish in are seeing "phantom citations" and text padded with AI-generated filler slip past first checks. The pipes of science are getting clogged.
What's actually happening
Large language models produce confident, made-up citations. Under publish-or-perish pressure, some authors paste without verifying. Others outsource writing to tools that hallucinate sources and facts.
- Phantom citations: plausible titles, real-looking authors, no real paper.
- Boilerplate text: smooth paragraphs with vague claims, thin methods, and generic phrasing.
- Incoherent stats: mismatched Ns, impossible p-values, effect sizes that don't line up.
- Recycled figures/tables: reused visuals with new captions that don't match the data.
Why peer review is buckling
Reviewers are overloaded. Editors triage at speed. Paper mills and careless AI use raise volume and lower signal. Traditional checks miss modern failure modes.
Quick triage: a 60-second screen
- Reference sniff test: Randomly pick three citations; verify DOI and existence. One miss is a yellow flag. Two is a desk reject.
- Methods sanity: Sample sizes, instruments, preregistration, and data access should be specific. Vague = suspect.
- Numbers that add up: Ns match across abstract, methods, tables. CIs align with p-values. If not, stop.
- Scope vs. evidence: Big claims with small, convenience samples? No.
- Stylistic tells: Overly smooth, generic prose; repeated buzz-phrases; inconsistent terminology.
Verification workflow for editors and reviewers
- Automate citation checks: Validate DOIs via Crossref. Flag titles with no record.
- Require provenance: Authors disclose if/where AI tools were used and confirm no AI-generated references or fabricated content.
- Demand artifacts: Data, code, and analysis logs in a trusted repository; if restricted, require a clear access plan and audit trail.
- Enforce reporting standards: Consistent units, versioned datasets, and complete figure legends. Inconsistency is grounds for desk rejection.
- Spot-check claims: Pick one key result and reproduce the stat from shared code or supplied numbers.
- Post-acceptance audit: Random audits with penalties for falsified or phantom references.
Guidance for authors and PIs
- Ban auto-generated references: Use a reference manager with DOI-only imports. Manually verify existence and fit.
- Add an AI-use note: If you used AI for editing, say so. Never for citations, statistical results, or data summaries.
- Pre-submission red team: A lab-mate tries to break your methods and numbers. Fix what they find.
- Artifact first: Prepare your repo and reproducible script before writing the discussion. It forces discipline.
Policy upgrades for journals
- AI disclosure required: No disclosure, no review.
- Data and code by default: Exceptions are documented and reviewed.
- Desk-reject thresholds: >10% reference mismatches; unverifiable primary data; numerical inconsistencies.
- Paper-mill filters: Image forensics, author identity checks, and cross-submission pattern analysis.
What not to do
- Don't rely on AI detectors: High false-positive and false-negative rates. Use evidence-based checks.
- Don't ban all AI: Allow language assistance with transparency. Draw a hard line at generation of sources, data, or results.
The bigger picture
The volume problem isn't new. It started long before chatbots. What's changed is speed and scale. The fix is boring by design: better workflows, stricter artifact rules, and real consequences.
If you improve verification and incentives, signal rises. If you don't, peer review turns into theater.
Useful resources
- Crossref DOI search - fast existence checks for references.
- Committee on Publication Ethics (COPE) - policies and case guidance.
Skill up your team
If your lab or editorial board needs hands-on training to use AI responsibly (and avoid AI slop), consider structured courses grouped by role: AI courses by job.
Your membership also unlocks:
