AI Can Spot Research Errors, But It Won't Fix Social Science's Core Problems
Artificial intelligence can analyze datasets, review code, suggest statistical methods, and even write entire research papers from scratch. Yet recent studies show that current AI models remain unreliable partners for social science-useful for catching some mistakes, but far from solving the field's fundamental issues.
A recent discovery illustrates the gap. Economist Michael Wiebe uncovered technical and coding errors in a 2021 study about tech clusters and innovation that a charity had used to guide funding decisions. The American Economic Review accepted his comment detailing the problems.
When Wiebe tested whether AI chatbots-including two versions of ChatGPT and Refine, a tool designed for academic review-could have caught these errors, the results were mixed. The AI systems flagged several issues, including one key coding error. They also missed many others. Wiebe did not test whether the AI had identified problems that didn't actually exist.
The practical takeaway: researchers should run papers and code through AI systems and investigate any flagged problems. The time investment is minimal. But an AI-vetted paper is not necessarily a reliable one.
Different AI Models Reach Different Conclusions From the Same Data
A more troubling pattern emerges when humans are removed entirely. Researchers gave 150 Claude Code agents the same stock market dataset and asked identical questions: Did daily trading volume, intraday volatility, and price impact change over time?
Some questions produced consistent results across different AI runs. Others produced wildly different answers based on subtle choices. "Trading volume" could mean dollar volume or share volume-yielding opposite conclusions. Volatility changes depend on whether researchers measure raw or proportional shifts.
Different versions of Claude even displayed distinct "empirical styles," favoring particular modeling approaches and measurement methods.
When researchers had one AI peer-review another's work, some revisions occurred but no convergence happened. When they instead showed the AI examples from top papers on similar questions, the models imitated those methods and aligned their results.
The problem: AI systems behave like human researchers. They branch out and do things differently unless steered toward a specific path. That's useful if you already know which path is correct. It's a serious limitation if you want reliable answers without extensive human direction.
AI Inherits the Biases It's Trained On
Social science carries its own ideological baggage. Research analyzing the political orientation of academic work since 1960 found that roughly 90 percent of politically relevant social-science articles leaned left. Every discipline moved further left after 1990.
AI models trained on this existing body of work inherit those biases. Research has documented that AI systems often carry left-leaning ideological priors and exhibit other patterns-such as favoring the first option when given two choices.
AI might help broaden perspectives in social science with careful prompting. But the technology starts from a biased foundation.
The Current State: Useful Tool, Not Substitute
AI excels at generating code and prose quickly and catching errors humans miss. These are genuine advantages worth using.
But the technology still makes frequent mistakes, carries ideological assumptions, and fails to produce consistent results when different models tackle the same question. Without heavy steering from human researchers, AI cannot reliably replace human judgment-which is precisely why many hoped it might.
For now, AI works best as a quality check and productivity tool, not as a solution to social science's deeper methodological and cultural problems.
Your membership also unlocks:
