When an opinion editor tested multiple AI detection tools on her own writing and got results ranging from 17 percent to 100 percent human, she confirmed what many researchers have been saying: these tools are not reliable for identifying AI-generated content. The editor ran samples from her own work, a freelance columnist, and two other writers through different detectors. One tool marked her personally written paragraphs as 75 percent AI; another scored the same paragraphs as 100 percent human.
The incident began when a colleague forwarded an accusation that a guest columnist's submission was "nearly 100 percent AI-written," based on a paid AI detector. The editor, skeptical, tested several tools on multiple writers. The accuser's own writing came back as 51 percent AI. A freelance columnist scored 76 percent. The editor's work ranged from 17 percent to 75 percent, depending on which tool she used.
"It's hard to put stock in AI detectors at the moment," the editor wrote. "Pretty much every AI detector, paid or not, advertises itself as the most accurate. So what gives?"
Why detectors fail
AI large language models are trained on human-generated text, often scraped from the web, including social media. The quality of that training data is uneven. Many news organizations block their content from being scraped, but bots still find sources. The result is that detectors learn patterns that can also appear in human writing - certain sentence lengths, formatting choices, punctuation like em dashes - that existed long before generative AI tools became mainstream.
A December NPR report highlighted the problem in schools: more than 40 percent of surveyed sixth- to 12th-grade teachers used AI detection tools last year, despite research showing the tools are unreliable. "It's now fairly well established in the academic integrity field that these tools are not fit for purpose," said Mike Perkins, a researcher on AI and academic integrity at British University Vietnam.
For writers, the stakes are immediate. Matt Lillywhite of The Daily Draft wrote that "some people are genuinely writing poor-quality and low-effort articles with AI. Many aren't. Yet, the same evidence gets used for both." He added that accusations "spread faster than proof, and entire groups of strangers … end up acting as judge, jury, and executioner without actually knowing what happened."
What this means for writers
The editor in this case knew the accused writer personally and confirmed she did not use AI. Still, a false accusation damaged her reputation temporarily. For those working in content creation, freelance writing, or journalism, relying on an AI detector to prove your work is human is risky - the tools can just as easily flag your original writing as AI-generated. The AI for Writers topic covers how to navigate these issues without leaning on flawed detection tools.
Why this matters for writers
Writers face a growing expectation to prove they didn't use AI, when the tools used to check them are demonstrably unreliable. The burden of proof falls on the accused, not the accuser. Until detectors improve, the safest approach is to save drafts, track revision history, and maintain a clear record of your writing process - not rely on a tool that can't tell the difference between your best work and a bot's filler. Understanding Generative AI and LLM basics can help writers explain why detectors are flawed and defend their work against baseless claims.
Your membership also unlocks:
