IAB Tech Lab Releases Guidance for Managing AI Crawlers and Bots
The IAB Tech Lab has published guidance to help publishers and content owners decide how to manage AI bots and crawlers accessing their sites. The guidance is open for public comment until June 26, 2026.
The move addresses a gap the Tech Lab identified after releasing its CoMP API V1 framework. Many publishers have not developed formal strategies for controlling non-human traffic, leaving content vulnerable to unauthorized access by AI systems.
What the guidance covers
The CoMP Working Group outlined different approaches content owners can take to manage crawler access. Each strategy comes with specific advantages and trade-offs tied to business goals.
The guidance targets non-technical leadership with practical questions to ask about bot management. These questions help organizations understand the value and costs of each decision they make.
Blanket bot-blocking no longer works. With non-human traffic now dominating most of the web, publishers need graduated, standards-based controls to distinguish between bots that drive value and those that drain resources.
Why this matters for management
Publishers face a business problem, not just a technical one. AI bots can extract intellectual property at scale, while legitimate AI partnerships can create value. The difference requires clear decision-making frameworks.
Organizations that establish bot management strategies can reduce operating costs, lower risk, and operate more efficiently. These strategies also enable partnerships with AI systems that support a sustainable content marketplace.
The guidance helps content owners gain visibility into how their material is accessed and used. This transparency is essential for protecting business interests while identifying which AI agents should have access.
Next steps
The IAB Tech Lab invites industry feedback during the public comment period. The guidance, along with the CoMP API framework, is designed to support broader adoption across publishers and content platforms.
For more information on AI for Management and AI for Executives & Strategy, explore how these frameworks apply to your organization's operations.
Your membership also unlocks:
