AI Crawler Access Control: The 2026 Decision Matrix

AI crawler access control is no longer a single switch you flip on or off. Eight major AI crawlers each demand a separate decision, and the distinction that changes everything is purpose: a crawler that harvests your pages for model training is a fundamentally different actor from one that indexes you for AI search answers. Block them as one bucket and you can quietly delete yourself from the fastest-growing referral channel of 2026.

The reason this matters now is that the major AI vendors have split their crawlers in two. OpenAI runs GPTBot for training and OAI-SearchBot for ChatGPT search. Anthropic runs three separate bots. Amazon, Google, and Apple each separate training access from search and assistant access. The robots.txt rule that blocks one no longer blocks the other — which means the old "block all AI bots" advice is now actively harmful to visibility.

This guide gives you the full bot-by-bot decision matrix, the economics behind why blocking training crawlers makes sense, the five control levers ranked by enforcement strength, and a copy-ready 2026 configuration. Every claim is sourced to the operator's own documentation or to Cloudflare's published network research.

Every AI crawler does one of three jobs. It collects pages for model training, it indexes pages for AI search answers, or it fetches a page in real time because a user asked the assistant a question right now. These are different commercial relationships, and as of 2026 the major vendors expose them as different bots with different user-agent strings.

OpenAI's documentation is explicit: GPTBot is the training crawler, and disallowing it "indicates a site's content should not be used in training generative AI foundation models." OAI-SearchBot is a separate crawler that builds the ChatGPT search index. Block OAI-SearchBot and your site will not appear in ChatGPT search answers, even though GPTBot and OAI-SearchBot are run by the same company. Anthropic goes further still, running three distinct bots — ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for real-time user-initiated fetches — each independently controllable in robots.txt.

The practical consequence is that the instinctive move — a blanket Disallow: /for every AI user-agent — now does two things at once: it opts you out of training corpora (often the goal) and it removes you from AI search results (almost never the goal). Search Engine Journal's coverage of Anthropic's granular bot framework reports that roughly 71% of top news publishers block at least one retrieval or search bot, frequently while intending only to block training. That is the exact error this matrix is designed to prevent.

Below is the flagship asset: a bot-by-bot matrix that pairs each crawler's exact user-agent string with its purpose and the recommended 2026 default. The pattern is consistent — block the training bots, allow the search and retrieval bots — with two exceptions worth understanding before you copy anything into production.

The business argument for blocking training crawlers comes down to a single ratio: how many of your pages a bot crawls for every one visitor it sends back. Cloudflare publishes this crawl-to-referral ratio across its network, and the spread between vendors is extraordinary. Traditional Googlebot sits at roughly 5 pages crawled per referral. Anthropic, at its June 2025 peak, was crawling about 70,900 pages for every visitor referred — an asymmetry that reframes ClaudeBot training access as a one-way extraction of value.

Two caveats keep this honest. First, the 70,900:1 figure is Anthropic's peak in the week of June 19-26, 2025; by July 2025 it had improved substantially — reportedly by around 87% to roughly 38,000:1 — after Anthropic shipped web-search features. The direction of travel matters, but even the improved ratio is orders of magnitude worse than Googlebot. Second, Cloudflare's own framing of the broader trend is blunt about where this is heading.

The other half of the economics is the upside you protect by not blocking search crawlers. Cloudflare reports that training now drives roughly 82% of all AI bot activity (up from about 72% a year earlier) while search-based crawling fell to around 15%. That is the macro signal: the volume hammering your servers is overwhelmingly training, not the search indexing that sends traffic back. Meanwhile AI-referred traffic is reportedly growing fast and tends to convert better than generic organic search — so the search crawlers are the cheap, high-value half of the equation that the blanket-block crowd is throwing away.

robots.txt is the polite request layer — well-behaved crawlers honor it, but it has no teeth against bots that choose to ignore it. The second proprietary asset below ranks the five control mechanisms by how hard they actually enforce, because the critical detail is buried in Cloudflare's documentation: a WAF or firewall rule is evaluated before robots.txt is ever read, so a WAF block overrides any robots.txt Allow.

For most teams the right combination is robots.txt for the compliant training bots, Cloudflare AI Crawl Control (or an equivalent managed ruleset) for purpose-level enforcement, and a targeted WAF rule for Bytespider specifically. Cloudflare's one-click "Block AI bots" managed rule, available on all plans including free since July 2024, is a reasonable floor — but verify that it is not silently blocking the search crawlers you want to keep. If you are auditing an existing setup, AI crawler directives belong as a dedicated category in any technical SEO audit checklist.

One of the most persistent misconceptions in this space is that adding an /llms.txtfile gives you control over AI training. It does not. The llms.txt specification defines a Markdown file that helps an LLM efficiently navigate your site's content during a user session — it is an inference-time convenience, the equivalent of a curated sitemap written for a model rather than a search engine. It carries no access or training permissions whatsoever.

Keep the mental model clean: robots.txt governs crawl and access permissions; llms.txt governs how a model finds its way around once it is already reading your pages. Publishing llms.txt is a worthwhile move for AI-search visibility and answer quality — but if your goal is to opt out of training, llms.txt does nothing for you and robots.txt plus a WAF rule does everything. For the file format and how to structure it for inference-time navigation, see our companion guide to the llms.txt specification, and for the foundational access-control mechanics, the robots.txt and meta robots reference.

Common Crawl's CCBot has historically been treated as a passive archiver — a non-profit whose corpus happens to underpin nearly every major LLM, from GPT-class models to LLaMA and Mistral. That framing changed on April 29, 2026, when the News/Media Alliance sent a formal demand letter to Common Crawl's executive director, calling for removal of publisher content, revised terms explicitly prohibiting AI training use, and enforceable opt-out mechanisms. Signatories included NBCUniversal, CNN, McClatchy, Vox Media, Ziff Davis, and USA Today.

The practical takeaway is that blocking CCBot is no longer purely a technical-hygiene decision. For publishers, it has become a precautionary intellectual-property position — increasingly taken on legal advice — because the corpus is now contested ground. If your content has commercial value as licensable IP, disallowing CCBot in robots.txt is the documented opt-out, and doing so early establishes a clear record of intent.

Here is the configuration that follows from the matrix: block the training crawlers in robots.txt, allow the search and retrieval crawlers, and back it with a WAF layer for the bots that do not respect the file. Adjust per-site — a documentation-heavy SaaS may weigh AI-search visibility more heavily than a paywalled publisher guarding licensable IP — but this is the sensible starting point.

A few operational notes. Place the most specific user-agent rules first; some crawlers match the longest applicable directive, others the first. Keep the deprecated Anthropic agents (anthropic-ai, Claude-Web) out of your file — they are no longer active, and citing them gives readers and tools broken instructions. And remember that Pay Per Crawl, Cloudflare's HTTP 402 "pay-to-crawl" model that launched in private beta on June 1, 2025 and reached general availability in August 2025, is now a third path beyond block-or-allow: charge bots a micro-fee for access rather than refusing them outright.

The crawler decision is not one-size-fits-all — it depends on what your content is worth as training data versus how much you stand to gain from AI-search referral. Four common profiles, four different calls.

Whichever profile fits, the mechanics are the same: separate training from search, enforce at the right layer, and measure the AI-referral channel so the decision is data-led rather than reflexive. If you want this configured and monitored as part of a broader technical-SEO program — robots.txt, WAF rules, llms.txt, and AI-search measurement in one engagement — that is exactly the work our agentic SEO service is built around, and crawler governance fits naturally alongside an AI transformation roadmap.