AI news ·
IAB Tech Lab releases AI bot and crawler management guidance for public comment
IAB Tech Lab released new bot management guidance, open for comment until June 25, 2026. It helps publishers replace blanket blocking with graduated traffic controls.
IAB Tech Lab has released new guidance on managing bot and crawler access, opening the document for public comment until June 25, 2026. The framework helps content owners and publishers evaluate how to handle non-human traffic from AI systems as industry adoption of the CoMP API V1 progresses.
Addressing the strategy gap
The guidance emerged after IAB Tech Lab recognized that many publishers lacked formal strategies for handling bots and crawlers, an area falling outside the CoMP API V1 scope. The document outlines a range of management approaches and highlights the operational and financial trade-offs of each. It is primarily designed for non-technical leadership teams to help them assess traffic management in a way that matches their business requirements. Leaders exploring AI for Management will find this framework directly addresses the need to understand these operational implications.
Anthony Katsur, CEO of IAB Tech Lab, emphasized the necessity of this direction. "Giving content owners clear, practical guidance is key if we want adoption to move forward in a meaningful way," Katsur said. "This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace."
Moving beyond blanket blocking
Industry leaders note that non-human traffic now dominates the open web, making outdated blocking methods ineffective. Blanket restrictions fail to separate useful automation from harmful scraping. As Dave Bellous, VP of Strategy at Metal Toad, said, "Non-human traffic now dominates most of the open web, and AI agents and crawlers are steepening the curve. Blanket bot-blocking is no longer a useful strategy."
He added that distinguishing beneficial automation from invalid traffic through graduated, standards-based controls has become a baseline requirement for publishers and advertisers. This shift requires executives to treat bot management as a core business strategy rather than a technical background task. Organizations focusing on AI for Executives & Strategy can use these frameworks to reduce risk and improve efficiency in their digital partnerships.
Why this matters for management
Managers must now define how their organizations interact with automated systems before bad actors exploit unprotected content. Relying on IT to handle crawler access without a business strategy risks intellectual property loss and inflated infrastructure costs. Reviewing this guidance before the June 25 deadline allows leadership to implement graduated controls that protect assets while preserving legitimate AI partnerships.