Google, Microsoft, Anthropic and OpenAI back new standard to block AI training crawlers with a single robots.txt entry

Google, Microsoft, Anthropic, and OpenAI announced plans Thursday to support a unified website crawling standard that will let publishers block all AI training scrapers with a single line in their robots.txt file. The proposal, already submitted to the Internet Engineering Task Force, addresses a growing friction point between content creators and AI developers who collect web data for model training.

The new standard targets a practical headache for website operators. Currently, a publisher who wants to prevent AI companies from scraping training data must add separate robots.txt entries for every provider - and sometimes for multiple scrapers used by the same company. The proposed change collapses that into one directive.

How robots.txt works today

Websites have long used the robots.txt file to manage search engine crawlers. The file follows the Robots Exclusion Protocol, a format maintained by the Internet Engineering Task Force. Publishers can block all scraping, restrict access to specific pages, or set crawl rate limits to prevent server overload.

With the rise of generative AI, those same controls became the primary tool for regulating training data collection. The patchwork approach, however, created administrative burden. Each AI provider's scraper needs its own entry, and keeping the list current requires ongoing maintenance as new crawlers appear.

What the new standard changes

The proposal introduces a single instruction that blocks all crawlers collecting data for generative AI training. Models performing "exclusively non-generative tasks" would not be affected, preserving access for search indexing and other traditional crawling functions.

Google, Microsoft, Anthropic, and OpenAI have already submitted a draft to the Internet Engineering Task Force. The group maintains the Robots Exclusion Protocol, giving the proposal a clear path toward formal adoption within the existing technical framework.

Microsoft and OpenAI add granular controls

Microsoft also announced a separate setting called AI Grounding, designed for scenarios where an AI model accesses a webpage to answer a user's question rather than harvest training data. Website operators can block those requests specifically, and the control extends to third-party tools built on Microsoft's AI infrastructure.

OpenAI said it will offer similar controls. Both companies plan to implement the broader crawling standard in the coming months. Google expects to follow sometime next year. Meta expressed support for the idea but has not committed to adopting it.

Why this matters for publishers and content strategists

The single-entry approach reduces the operational burden of managing AI crawler access, but the strategic question remains unchanged: whether to block training data collection at all. Publishers must weigh the long-term value of their content against the visibility that comes from inclusion in AI training sets - a calculation that varies by business model, audience, and competitive landscape.

For legal and communications teams, the standard provides clearer documentation of consent signals. A website's robots.txt becomes a more explicit record of what data access was permitted or denied, which could factor into copyright disputes and licensing negotiations as those conversations intensify across the industry.