Let The Bots Feast: Why Media Should Embrace The Great AI Scrape
The media industry is already losing the battle against AI. Publishers are rushing to block AI crawlers, partnering with companies like Cloudflare to build digital barriers, and filing lawsuits against tech giants for scraping content without compensation. According to Cloudflare, scraping has increased by 18% in the past year. Some publishers, such as Dotdash Meredith, have struck licensing deals with OpenAI while trying to shut down “bad actors.”
But all these efforts are too little, too late. AI isn’t just breaking in—it’s creating something entirely new. The smarter move is to welcome AI inside.
Don’t Get Left Behind
Paywalls are losing their grip. They’ve relied on friction more than genuine loyalty—subscribers often stayed by accident or guilt, not because of sustained interest. Now, generative AI is exposing their weaknesses.
People no longer want to hop between multiple sites for news. They want fast, portable summaries and context all in one place. Large Language Models (LLMs) don’t need to copy your article to reduce your traffic—they just need to replace the need to visit your site.
If ChatGPT can deliver a clear, concise answer to “What happened in Gaza this morning?” most users won’t click through to multiple full-length articles. And if your site blocks AI crawlers, you’ve made yourself invisible in the only newsroom that matters now: the machine’s.
The fight to guard so-called proprietary content misses the point. Content is already a remix culture: news sites summarize other news sites, bloggers paraphrase headlines, analysts repurpose coverage with insights. Even original scoops get sliced, quoted, and shared in multiple versions within hours.
AI simply automates this process. Fighting AI content scraping is like fighting email because it made fax machines obsolete. Moreover, AI-generated summaries often boost the visibility of quality reporting. If a chatbot frequently cites your outlet, that’s increased reach. If it accurately reflects your work, that’s influence.
Blocking AI doesn’t stop your ideas from spreading—it only cuts you out of the credit cycle. It’s like a professor trying to stop their research from being cited because they didn’t approve the footnotes.
A Better Path
Publishers should collaborate with AI platforms instead of building walls. Embed metadata to identify authorship, negotiate deals that prioritize bylines and backlinks, and allow headlines to flow through models with proper attribution.
Become the signal amid the noise. Dotdash Meredith’s licensing deal with OpenAI is a step in this direction. Other smart publishers will follow with partnerships like Google’s AI Overviews.
If an AI model defaults to sources like The New York Times, The Atlantic, or Bloomberg because those outlets made their content indexable and AI-friendly, that’s not a loss. It’s a win for brand equity, reach, and trust. You want machines quoting you, not ghosting you.
The reality is clear: the next generation of news consumers will interact with the world through AI. When a teenager asks their AI assistant why the Supreme Court overturned a precedent or a voter inquires about a candidate’s housing record, they won’t sift through newspaper archives. They’ll get a verbal summary in seconds.
If your journalism isn’t part of that summary, you effectively don’t exist in those conversations. AI isn’t replacing journalism’s value—it’s replacing the path to it. Publishers who treat AI as an enemy risk training their own successors not to remember them.
The future of media isn’t behind paywalls or lawsuits. It’s in training sets, embedded links, smart attribution, and producing work so good that even AI wants to get it right. There’s no dignity in hiding and no sustainability in suing your way back to a failing model.
If survival is the goal, stop stressing over how to keep bots out. Start focusing on making sure they cite you when it counts.
Your membership also unlocks:
