#robots-txt

Google plans to honour a shared robots.txt token that opts a site out of AI training across providers from 2027. What it covers, what it does not, what to do.

Meta's revised Commercial Terms take effect October 30 and let its crawlers read sites tied to business accounts. What changed, what didn't and what to do.

User-agent strings are trivial to spoof and many user-triggered fetchers ignore robots.txt. How to verify AI crawlers and enforce access at the edge.

NotebookLM is now Gemini Notebook, and its fetcher UA changes to Google-GeminiNotebook. Why it ignores robots.txt and how to actually control access.

Google rewrote its StoreBot troubleshooting doc on July 17 — blocked crawls mean unverified local inventory and suspended listings. The fixes to run now.

Bots now generate 57.5% of web requests, a machine-majority first. A sourced reference on AI crawler volume, crawl-to-refer ratios, and blocking.

DCN sent Common Crawl a cease-and-desist on June 3, 2026. Why the WARC archive makes removal hard and why blocking CCBot alone won't protect your content.

Eight major AI crawlers, eight different calls. A bot-by-bot matrix to block training crawlers like GPTBot and ClaudeBot while keeping AI-search citations.

Faceted navigation drives 50% of all Google crawl issues. A facet-by-facet decision matrix mapping every URL type to index, canonical, noindex, or block.

Complete 2026 reference to robots.txt directives and meta robots tags — crawling, indexing, noindex, X-Robots-Tag, and JS rendering pitfalls.