Google's Planned AI-Training Control for robots.txt

On October 30, 2026, Google Search Central announced that Google plans to support a shared AI-training control in robots.txt. The control is a policy token that a site owner writes into an ordinary robots.txt group. When a site disallows that token and says nothing about Google-Extended, Google will treat the site as if it had disallowed Google-Extended. Google aims to support the control, with other AI providers, in 2027.

That is the whole announcement, and it matters for a reason that has nothing to do with Google. For three years, a publisher who wanted to keep content out of AI training has had to learn a separate user-agent token for every company: Google-Extended for Google, one name for OpenAI, another for Anthropic, another for Apple. Each company defined its own token, documented it in its own place, and changed it on its own schedule. The shared token is a proposal, now with a major provider's public commitment, that replaces that list with one line.

This post covers what Google said, how the token works under the IETF draft it is built on, how it interacts with Google-Extended and Gemini grounding, what it cannot do, and what to change in your file now versus when the control goes live. Our existing AI crawler access decision matrix covers the per-vendor tokens this proposal sits beside; this post is about the new layer only.

The post is short and was written by John Mueller, Google's Search Advocate, on the Google Search Central blog. It makes four commitments. Google will support a shared policy token in robots.txt. The token comes from work in the AIPREF working group at the Internet Engineering Task Force, the body that standardises the robots.txt format itself. Existing company-specific controls, Google-Extended among them, keep working. And the target is 2027, with progress updates promised.

The wording is deliberate about tense. Google "plans to interpret" an opt-out through the token as a Google-Extended disallow. Other AI providers "can handle it similarly", which is permission, not a list of who has agreed. The post calls the mechanism a "draft proposal" and says Google will keep refining it through the IETF working group. Nothing in it says the token works today. Google's crawler documentation, read on November 5, 2026, still lists Google-Extended as the working AI-training control and carries only a note that Google intends to support the shared token in early 2027.

The clearest sentence in the post is the one that explains the motive, and it is worth quoting rather than paraphrasing.

A reader who wants the failure case has it already. A site owner who reads the headline, adds the token group today, and deletes the Google-Extended lines they already had has removed a control that works and replaced it with one that no provider yet honours. Until Google documents the token as supported, the old lines stay.

Google's post links to the IETF draft it is adopting: draft-kelley-aipref-training-preference-signal, an individual submission dated October 30, 2026 whose named author is at Anthropic. The design choice that makes it workable is restraint. The draft defines exactly one policy token, __usage_ai-training, and states that it introduces no new directive names and no new grammar. The path language, the longest-match precedence and the group structure of RFC 9309, the robots.txt standard, are untouched.

In practice you write a group whose User-Agent line names the token instead of a crawler, then use Allow and Disallow as usual. The draft's own example expresses a preference against training for the whole site except one directory:

Why does that work without breaking anything? Because every robots.txt parser already ignores groups for user agents it does not recognise. A crawler that has never heard of the token reads the file exactly as before. A consumer that implements the draft looks up the path against the token's group and reads a disallow as a preference against training and an allow as a preference in favour. If no group names the token, no preference has been expressed, which is the draft's phrase for "the site has not decided", not "the site has consented".

The example also shows the second useful property: preferences are per path, not per site. A publisher can keep a press kit or a docs section open to training while closing the archive, which the per-vendor tokens could do too but which no two vendors described the same way. The failure case is a parser that matches tokens loosely; the draft notes some deployed parsers do, and picked the double-underscore prefix so that an accidental match is unlikely.

The new token only makes sense against the control it will sit beside. Google-Extended, which Google has offered since 2023, is not a crawler. Google's crawler documentation states that it has no separate user-agent string, that crawling is done by the existing Google user agents, and that the token is used in robots.txt "in a control capacity". It is a label you address, not a bot you see in your logs.

What the label controls is wider than most site owners assume. Per the same page, Google-Extended governs whether content Google crawls may be used to train future Gemini models that power Gemini Apps and the Vertex AI API for Gemini, and whether it may be used for grounding, which the page defines as providing content from the Search index to the model at prompt time. The page is equally explicit about what it does not touch: Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal.

The grounding clause is where the business decision lives. Google's October 30 post repeats it: sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps. A publisher weighing an opt-out is therefore not only deciding about training. They are deciding whether Gemini may cite them at answer time. Our AI crawl economics analysis covers what that referral traffic is actually worth; the honest answer is that it varies by site and is small for most.

Google's post and the IETF draft describe the same precedence rule from two sides. The draft says specific expressions should take precedence over general ones, and that a consumer may document that rules matching its own proprietary token are more specific than the shared token. Google's post is the documentation: an explicit Google-Extended rule overrides whatever the shared token says, and the shared token fills the gap only when Google-Extended is not mentioned at all.

Four combinations cover every real file. Each row below gives Google's stated outcome for the training question, assuming the control ships as described.

The third row is the one the shared token was designed to make possible. A publisher who wants to be out of every other provider's training set but keep Gemini grounding can say so in two groups rather than a dozen. The failure case is the mirror image: a site with an old Google-Extended allow line, left over from an experiment, that adds the token believing it has opted out of Google. It has not. Audit the explicit lines before you rely on the general one.

The draft is unusually direct about its own boundaries, and a site owner should read them before deciding anything. First, the token is not a crawling directive. The draft says a consumer may use the preference to inform crawling, for example by not fetching pages it only wanted for training, but nothing requires it. If you want a bot to stop fetching, you still write a group for that bot's product token, as our bot verification and edge enforcement playbook describes, and you enforce it at the edge if the bot ignores it.

Second, the vocabulary is narrow by design. The token covers "AI training", which the working group's vocabulary draft defines as using an asset to modify the learned parameters of a generative AI model. It does not cover retrieval at answer time, search indexing or the training of models used only for classification or ranking. Google's own control is broader than that, because Google-Extended also gates grounding; the shared token inherits that breadth only through Google's decision to map one onto the other, not through the draft.

Third, it binds nobody who has not adopted it. The draft says private agreements may override the preference, and Google's post says other providers "can" handle it similarly. A content licence signed with one lab is unaffected. A lab that never implements the token is unaffected. The signal is a statement of preference with a growing list of readers, which is what robots.txt has always been.

Google names its reasons, and they are regulatory as much as technical. The post says the proposal is consistent with the EU Code of Practice for general-purpose AI model providers under the EU AI Act, which it describes as recognising robots.txt as a primary method for web publishers to express machine-readable preferences. A provider that signs that code needs a way to show it honours such preferences; a single token that many providers read the same way is easier to demonstrate than a dozen proprietary ones.

The technical reason is the one the working group has been circling for two years. Every provider invented its own token because there was no shared definition of what "training" meant. The AIPREF vocabulary draft now supplies that definition, and Google's post says the mechanism brings publishers the benefit of that definition "while multistakeholder discussions continue on the rest of the AIPREF vocabulary". In other words, the group agreed on one word and shipped a token for it rather than waiting for the whole dictionary.

There is also a quieter point about who wrote what. The draft's author is listed with an Anthropic affiliation, the working group is chaired by people from Cloudflare and Cisco, and its vocabulary editors are from Mozilla and Open Future. Google has publicly committed to the resulting signal, which is a different thing from Google having designed it. For the scale of crawling this sits on top of, our AI crawler traffic statistics reference carries the measured numbers.

The practical answer splits by date. Nothing in your robots.txt needs to change this week, because no provider has documented support for the token. What is worth doing this week is deciding what you want the file to say when support arrives, because the decision is a business one and the edit is trivial.

One more thing the checklist does not include: a file for AI agents to read. The token is a preference about training, not a description of your site, and it does nothing for the question of whether an answer engine can find and summarise you. That question has its own evidence, which our llms.txt adoption review lays out, and its own service line in our agentic SEO work. Keep the two decisions separate, because the file formats are.