#ai-crawlers

Google plans to honour a shared robots.txt token that opts a site out of AI training across providers from 2027. What it covers, what it does not, what to do.

Meta's revised Commercial Terms take effect October 30 and let its crawlers read sites tied to business accounts. What changed, what didn't and what to do.

From September 15, 2026 new ad-supported Cloudflare domains block AI agents on ad pages and refuse AI training by default. A 20-bot census of who is affected.

User-agent strings are trivial to spoof and many user-triggered fetchers ignore robots.txt. How to verify AI crawlers and enforce access at the edge.

Digiday reports Perplexity blocked Time's markdown ads, copy served to AI crawlers but not to humans, as deceptive. Why this is a cloaking question.

Cloudflare's AEO Visibility Dashboard shipped 6 August in early access with no published price, and it probes only two assistants: Claude and GPT.

Adoption of llms.txt splits hard by cohort, and the best traffic studies found no real AI-bot requests. Who reads the file, and a fifteen-minute setup.

Blocking AI crawlers is now an economic call. Cloudflare's Pay Per Crawl beta lets you charge per request; crawl-to-referral ratios show what each bot returns.

During a pre-launch audit of an independent writer's publication, every archive post returned a 302 to crawlers. How to detect and fix this AI-crawler wall.

DCN sent Common Crawl a cease-and-desist on June 3, 2026. Why the WARC archive makes removal hard and why blocking CCBot alone won't protect your content.

Eight major AI crawlers, eight different calls. A bot-by-bot matrix to block training crawlers like GPTBot and ClaudeBot while keeping AI-search citations.

Log files capture what crawlers actually do — not a simulation. A 2026 guide to crawl-budget analysis and the new training-vs-retrieval AI crawler split.

Why agentic crawlers prefer Markdown, the llms.txt + AGENTS.md pattern, and file-tree spec for agent-ready sites. Reference implementation + audit.