Block, Adapt, Sue or Partner: How Content Creators Are Responding to AI Scraping

Will you snog, marry, or avoid AI?

Companies putting content online today must accept that AI scraping bots will access their media—be it text, images, or video. By 2025, this is the norm most digital marketers have embraced. Instead of fighting AI, many focus on optimizing content to appear favorably in AI-generated answers, like those from Gemini, which often rank above traditional search results.

Blocking AI bots

Some companies resist this trend. Nextdoor’s CEO, Nirav Tolia, for example, blocks AI bots from scraping content entirely. He built Nextdoor without optimizing for Google Search and refuses to allow any AI crawling. While this approach is extreme, it reflects a commitment to controlling how community content is accessed and shared. However, it also means Nextdoor must provide a user experience comparable to AI-powered search without using AI itself.

Suing AI companies

Another route some creators take is legal action. Numerous lawsuits target AI companies like Stability AI, Midjourney, and DeviantArt for allegedly using copyrighted works without permission to train their models. NBCUniversal, for instance, is suing Midjourney over unauthorized AI-generated images based on Disney and Star Wars properties.

AI companies argue that scraping content falls under "fair use," suggesting publicly accessible online materials can be used for training models. While lawsuits vary in scale and resources, from individual artists to Hollywood giants, legal resolutions may take years. Meanwhile, AI bots continue scraping quietly.

Blocking with robots.txt and beyond

Some content owners try to block AI bots via the robots.txt file, which instructs crawlers which parts of a site to avoid. For example:

User-agent: * 
Disallow: /admin/ 
User-agent: GPTBot 
Disallow: / 
User-agent: Googlebot 
Disallow: 
User-agent: * 
Disallow: 

This setup allows Googlebot indexing but blocks GPTBot. However, this method is just a courtesy system, easily ignored by AI bots.

To truly block bots, companies need sophisticated tech measures—constant updates, firewalls, and CAPTCHAs. Still, advanced bots can often bypass these protections, making it an ongoing challenge.

Advanced blocking methods

Thousands of websites use JavaScript puzzles or challenges to differentiate human visitors from bots. Since typical scraping bots can’t process JavaScript, this effectively blocks many of them. Usually, such measures are implemented by tech-savvy creators and organizations with resources to maintain these defenses.

Partnering with AI companies

Large content owners like The New York Times or Reddit often negotiate partnerships with AI firms. These deals grant privileged access to extensive content libraries in exchange for fees. For many publishers, this provides a much-needed revenue stream bridging the gap between traditional and digital media.

Adapting your marketing approach

For most marketers, adapting is the only practical path. This means revising traditional SEO strategies and learning how to craft content that performs well within AI-driven environments.

Ironically, AI tools themselves can assist marketers in this transition, offering insights into how to optimize messaging for AI-powered platforms. Understanding how large language models and multi-modal AIs operate helps create content that stands out.

Assuming competitors have access to the same AI tools, deeper knowledge of these technologies will give marketers an edge in crafting impactful campaigns.

Summary of options for content creators

• Block: Using technical barriers to prevent AI scraping, though resource-intensive and often bypassed.
• Sue: Legal action over copyright infringement, a long and costly process.
• Partner: Negotiating agreements with AI companies for access and revenue sharing, mainly for large-scale publishers.
• Adapt: Evolving marketing strategies to thrive in AI-influenced content discovery, the most feasible choice for most businesses.

Smaller businesses especially should focus on adapting. Blocking or suing may be out of reach, and partnering requires scale and influence. Embracing new AI-driven marketing techniques offers the best chance to connect with audiences effectively.

Explore resources to build your AI skills and stay ahead: Complete AI Training - Courses by Job.