Complete AI Training
Sign inGet my AI kit

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

AI news ·

IAB Tech Lab issues guidance to help publishers manage AI crawlers and bots

IAB Tech Lab released guidance helping publishers control AI bots and crawlers, open for public comment until June 26, 2026. It targets non-technical leaders with decision frameworks for managing which AI systems can access their content.

1 min read
Management

IAB Tech Lab Releases Guidance for Managing AI Crawlers and Bots

The IAB Tech Lab has published guidance to help publishers and content owners decide how to manage AI bots and crawlers accessing their sites. The guidance is open for public comment until June 26, 2026.

The move addresses a gap the Tech Lab identified after releasing its CoMP API V1 framework. Many publishers have not developed formal strategies for controlling non-human traffic, leaving content vulnerable to unauthorized access by AI systems.

What the guidance covers

The CoMP Working Group outlined different approaches content owners can take to manage crawler access. Each strategy comes with specific advantages and trade-offs tied to business goals.

The guidance targets non-technical leadership with practical questions to ask about bot management. These questions help organizations understand the value and costs of each decision they make.

Blanket bot-blocking no longer works. With non-human traffic now dominating most of the web, publishers need graduated, standards-based controls to distinguish between bots that drive value and those that drain resources.

Why this matters for management

Publishers face a business problem, not just a technical one. AI bots can extract intellectual property at scale, while legitimate AI partnerships can create value. The difference requires clear decision-making frameworks.

Organizations that establish bot management strategies can reduce operating costs, lower risk, and operate more efficiently. These strategies also enable partnerships with AI systems that support a sustainable content marketplace.

The guidance helps content owners gain visibility into how their material is accessed and used. This transparency is essential for protecting business interests while identifying which AI agents should have access.

Next steps

The IAB Tech Lab invites industry feedback during the public comment period. The guidance, along with the CoMP API framework, is designed to support broader adoption across publishers and content platforms.

For more information on AI for Management and AI for Executives & Strategy, explore how these frameworks apply to your organization's operations.

Share