Cloudflare Blocks AI Agents on Ad Pages: Which Bots Are Hit

On September 15, 2026 Cloudflare changed the meaning of its AI crawler settings and the defaults it offers new domains. A site that earns money from ads is now steered, at onboarding, to refuse AI training and to block user-directed agents on pages that carry an ad, while search crawlers stay allowed. Cloudflare fronts a large share of the web, so a default, rather than a publisher's choice, now decides whether an AI agent can read many pages.

This census is for two readers. The site owner on Cloudflare who has to decide whether to keep the default, and the builder whose agent fetches web pages and is about to start seeing refusals. It lists the 20 bots in Cloudflare's bot reference, the category Cloudflare assigns each, what the new ad-page default does to it, and what the operator's own page says the bot is for. Cloudflare's September 15 announcement and its July 1 announcement of the same change are the primaries. All pages were read on September 16.

Cloudflare announced the shape of this change on July 1 and switched it on September 15. The July post said that on September 15 new defaults would apply to every new domain onboarding: the Training and Agent categories blocked by default on pages that display ads, Search allowed. It also said that from that date multi-purpose crawlers, the ones that crawl for search and training at once, would be allowed or blocked according to all of their behaviours, and that existing customers could opt out of that part ahead of time in their security settings.

The September 15 post delivers that, with one refinement the July post did not have. A new Training setting called Disallow AI Training publishes a no-training preference in the site's robots.txt, keeps crawling allowed for what Cloudflare calls Accountable mixed-use crawlers, and blocks every other training crawler. Cloudflare names Apple, Google and Microsoft as Accountable mixed-use operators, and Amazon, Anthropic, Meta and OpenAI as Accountable operators whose training crawlers are separate from their search crawlers and can be blocked without touching search. The post says the controls are "available to all customers, on all plans", and that the ad-supported presets are offered to customers onboarding a new domain.

Two smaller changes matter for the census. The old "Block AI Bots" switch is deprecated in favour of the three category controls, and existing domains that used it are migrated to Disallow AI Training plus Block on pages with ads for agents. And Cloudflare says there is no Disallow setting for agents yet, because the web has no established way to express that preference to an agent; it points at the IETF's AI preferences work as the place that may change.

Cloudflare classifies bots by behaviour and says one bot can show more than one. Its September 15 definitions, quoted: Search is "crawling to build a search index"; Training is "crawling to train or fine-tune a model"; Agent is "user-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents". The bot reference uses older labels for the same idea, AI Search, AI Crawler and AI Assistant, plus Search Engine for Googlebot and Bingbot. The census keeps both, so a reader can see how the reference's label maps to the control that now governs the bot.

The Agent category is the one that changes an agent builder's week. A chat assistant that fetches a page to answer a question, or a browser-use agent that navigates a site to complete a task, is now blocked by default on any ad-bearing page of a new ad-funded Cloudflare domain. That is a class of traffic that was, until this month, treated much like a human visit.

The "default on ad pages" column describes what happens to the bot on a page that serves ads, on a new domain that accepted Cloudflare's ad-supported preset from September 15. A blocked training crawler is blocked everywhere on that domain, not only on ad pages, because the preset uses Disallow AI Training; an agent is blocked only where an ad is detected; a search crawler is allowed. Existing domains keep whatever they had, migrated as described above.

Several operator tokens have no Cloudflare row. Google-Extended and Applebot-Extended are robots.txt tokens with no crawler of their own; they exist so a site can refuse training by the mixed-use crawler without refusing search, which is what Disallow AI Training publishes on your behalf. Amazon documents Amzn-SearchBot and Amzn-User alongside Amazonbot, and Mistral documents MistralAI-Index and MistralAI-Training alongside MistralAI-User. Those are search and training bots the operators separate out, and they are not in the reference read for this census. Whether Cloudflare classifies them elsewhere we could not confirm, because the Radar bots directory the reference points to is an interactive application that returned no bot list at any fetch tier.

If your site is on Cloudflare and runs ads, the preset costs you two things and protects one. It protects your pages from being used as training data by the eight training crawlers in the table, and it does so without touching Google, Bing or Apple search. It costs you the visit an AI assistant makes on a reader's behalf, on ad pages only, and with it whatever citation or link that assistant would have shown. And it may cost you agent traffic you wanted, such as a shopping agent completing a purchase, on the same pages.

Cloudflare's own framing of the trade is worth reading: ad revenue depends on a human seeing the page, training replaces the visit with an answer, and an agent fetches the page with nobody there to see the ad. That is true for a publisher. It is not obviously true for a retailer or a service business, where the agent may be carrying a customer. We set out how to decide that case in our decision matrix for AI crawler access, and the traffic shares behind it in our AI crawler traffic reference. Neither of those is repeated here.

If you build agents that fetch web pages, the practical effect is that a growing set of pages will refuse your fetch, and the refusal will depend on whether an ad was detected on that page, which you cannot see in advance. Three things follow.

Declare what you are. Cloudflare's controls act on verified, declared bots. An agent that fetches with a browser's user agent and no identity is not in the Agent category; it is in the undeclared-bot category, which Cloudflare blocks by default. The July post also narrowed what the Verified label buys: verified bots are no longer default-allowed, and the allowed category now decides what a verified bot may reach. Use an operator token, publish your IP ranges, and expect to be classified.

Respect the block, and log it. A 403 on an ad page is now a policy outcome, not an error. Record it as such so the agent can tell the user the page declined rather than retrying. Meta's documentation says its fetcher may bypass robots.txt because a user requested the fetch; Cloudflare's block is enforced at the network, not in robots.txt, so that argument does not get a fetcher through it.

Measure the refusal rate. The share of pages that refuse your agent is now a number that changes with Cloudflare's defaults and with how many domains onboard under the ad preset. Our 30-day agentic crawler log study shows what that traffic looked like from the site's side before the change; the builder's side is worth logging from now. If your agent depends on the open web and you want that measured before it becomes a support ticket, our agentic SEO practice runs the fetch audit from both sides.