AI Crawlers Face a New Toll as Publishers Adopt RSL

RSL: A Clear Licensing Signal for AI Crawlers

Online media brands are backing a new open protocol called RSL (Really Simple Licensing) to set terms and pricing for AI use of their content. The aim: replace vague norms with explicit permissions and payments, and reduce scraping that ignores site rules.

RSL takes its cue from RSS. It's open, decentralized, and can describe licenses for web pages, videos, and datasets. Where robots.txt says "crawl here or don't," RSL tells crawlers "here are the rules and costs."

"RSL builds directly on the legacy of RSS, providing the missing licensing layer for the AI-first Internet," said Tim O'Reilly, CEO of O'Reilly Media. "It ensures that the creators and publishers who fuel AI innovation are not just part of the conversation but fairly compensated for the value they create."

Why PR and Communications Should Care

Publishers report traffic declines up to 25% as AI summaries and chatbots reduce clicks. Google says its AI Overviews send "higher quality clicks," but many newsrooms see fewer visits and lower ad revenue, which complicates partner expectations and sales forecasts.

RSL is both a policy stance and a tech move. Your comms plan needs to frame it as pro-innovation and pro-creator, while preparing for questions from advertisers, platforms, regulators, and your community.

How RSL Fits With Current Controls

• Robots.txt: Acts like a door-allow or disallow crawling. Some AI crawlers ignore it, prompting lawsuits.
• RSL: Acts like the posted terms-license type, usage scope, and pricing that AI crawlers can read.
• Bouncers vs. Rules: Services like Cloudflare can block bots outright; RSL tells compliant crawlers how to proceed. Eckart Walther notes both can work together-charge for crawl access and require royalties for model outputs.
• Billing: Tools like Tollbit aim to meter and charge per crawl or per use. Expect hybrid setups: block unknown bots, license known ones, audit everyone.

Who's Backing It

• Yahoo, Quora, Medium, Reddit, People, Internet Brands, Fastly, wikiHow, O'Reilly, Daily Beast, The MIT Press, Miso, Adweek, Ranker, Evolve Media, Raptive

"If AI is trained on our writers' work, then it needs to pay for that work," said Medium CEO Tony Stubblebine. "Right now, AI runs on stolen content. Adopting this RSL Standard is how we force those AI companies to either pay for what they use, stop using it, or shut down."

Vivek Shah, CEO of Ziff Davis, said, "Widespread adoption of the RSL Standard will protect the integrity of original work and accelerate a mutually beneficial framework for publishers and AI providers."

Action Plan for PR/Comms Teams

• Policy: Align with legal and product on what's permitted (training, retrieval, summarization), pricing models, and enforcement thresholds.
• Implementation: Coordinate with engineering on RSL files, robots.txt updates, and bot fingerprinting. Pair with CDN/WAF blocks for noncompliant crawlers.
• Commercial: Prepare a standard outreach note for AI vendors offering licensed access. Maintain a public stance of "open to deals, firm on terms."
• Governance: Update Terms of Service and Privacy Policy to reflect AI usage rules and penalties for violations.
• Measurement: Baseline traffic, referrals from AI assistants, and content reuse signals. Set monthly reporting for execs and sales.
• Press Kit: Draft a one-pager, FAQ, and quotes from leadership about fair use, licensing, and innovation.
• Crisis Prep: Create response templates for detected scraping, DMCA/takedown steps, and escalation paths to legal.

Suggested Messaging

• We support responsible AI that respects creators and pays for value.
• RSL doesn't shut the door; it clarifies the rules so fair partnerships can scale.
• We're working with security partners to block unlicensed crawlers and welcome licensed access.
• Our goal is a healthy content ecosystem where audiences, advertisers, and innovators all benefit.

Executive FAQ

• Will this hurt SEO? RSL targets AI training/use, not search indexing. Robots.txt and sitemaps still manage search. Monitor organic traffic closely after rollout.
• What if crawlers ignore RSL? Use CDN/WAF blocks, contract enforcement, and legal remedies. Keep logs for evidence.
• What's the revenue angle? Mix of crawl fees, per-output royalties, and broader content licensing. Treat like a new channel with pipeline and forecasts.
• What about user experience? No change for readers. Ensure bot controls don't degrade page speed or accessibility.
• How do we measure success? Reduction in unlicensed hits, licensed partner growth, and net revenue vs. traffic trends.

What to Watch

• Standard adoption across major publishers and platforms.
• Licensing deals between large AI companies and media networks.
• Legal outcomes that set precedents for training and output use.
• Better crawler identification, billing, and audit tooling.

As Eckart Walther put it, think of Cloudflare-style blocking as the bouncer and RSL as the posted rules and cover charge. Both matter if you want compliance and revenue, not just fewer bots.

Next Steps

• Kick off a cross-functional task force (Legal, Product, Security, Revenue, Comms) with a 30-day plan.
• Publish your RSL policy, announce it, and invite licensed partnerships.
• Stand up monitoring, reporting, and an enforcement playbook on day one.

If your team needs to level up on AI policy and workflows, see curated training by job role at Complete AI Training.