Key takeaways
- OpenAI and Microsoft said on Oct. 30, 2026, that they will honor a shared robots.txt signal for AI training preferences, and Google said it plans to support it [1][2][3]. The signal is the policy token
__usage_ai-training[3][4]. It is defined in an individual IETF Internet-Draft by Kevin Kelley of Anthropic [4]. - The signal covers training only. Google says it plans to treat an opt-out through the token as if the site had disallowed Google-Extended, unless the site specifies Google-Extended itself, and that company-specific controls keep working [1].
- Microsoft says it will honor the preference as it crawls from Feb. 1, 2027 [3]. Google and OpenAI gave no exact date: Google aims for 2027 and OpenAI for “the coming months” [1][2].
Direct answer: Site owners will be able to state one AI training preference in robots.txt that several AI providers say they will read, rather than writing a rule for each provider’s crawler. The signal is a training preference, not a crawl-blocking directive, although providers may use it to inform crawling [4]. How each provider applies it still matters.
What happened: Google announced on its Search Central Blog that it plans “to support a shared AI training control in robots.txt,” based on work from the AI Preferences (AIPREF) working group at the Internet Engineering Task Force (IETF) [1]. OpenAI said the same day that it “will honor a common signal” in robots.txt for “foundational generative AI training” [2]. Microsoft announced support in a Bing blog post that also adds a separate preference for AI grounding [3]. The Verge reported that Anthropic has already adopted the signal and that Meta signaled support without saying whether it will honor it [5]; we found no statement from either company confirming this.
How does the signal work?
The Internet-Draft, published Oct. 30, 2026, defines a “policy token” that site owners write in the User-Agent line of an ordinary robots.txt group [4]. Its example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
According to the draft, this expresses a preference against AI training for all paths except those under /press-kit/ [4]. The draft also says:
- It is not a crawl rule. “This policy token is not a directive for crawling behavior” [4]. Rules for named crawlers still control crawling.
- Wildcards do not count. A
User-Agent: *group must not be read as matching the token. If no group names the token, no preference has been expressed through this mechanism [4]. - Training means generative models. The draft uses the AIPREF working group’s definition: “Using an asset to modify the learned parameters of a generative AI model” [4].
- Its status. It is an individual draft that “is not endorsed by the IETF and has no formal standing in the IETF standards process,” and it expires May 3, 2027 [4].
What each company said
- Google. If a site opts out through the token “(and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended.” Sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps. Google aims to support the control “along with other AI providers in 2027” [1].
- OpenAI. “We aim to implement the signal in the coming months.” It says the signal “will not replace” its existing robots.txt crawler controls [2].
- Microsoft. It will honor AI training preferences in robots.txt as it crawls from Feb. 1, 2027, will let a page-level setting override robots.txt, and will stop reading NoArchive and NoCache as training or grounding preferences after March 1, 2027 [3].
Why it matters
Our view: On Oct. 28 we reported that robots.txt governs crawling, not AI use, and that a shared AI-use rule was only an IETF draft. The proposal is still a draft with no formal IETF standing, but three AI providers now say they will read one training signal. For AI visibility, the line that matters is between training and grounding: the token expresses a training preference, and Microsoft says its training setting does not change its grounding setting [3]. Google’s mapping to Google-Extended shows why each provider’s implementation needs checking.
What we don’t know yet
- When Google and OpenAI will start reading the token. Neither gave an exact date [1][2].
- Whether the IETF working group adopts the draft, and in what form [4].
- How each provider will rank the token against its own proprietary tokens. The draft lets providers treat their own tokens as more specific [4].
- Whether providers that did not announce support will honor the signal.
What to do now
- Decide your training and AI search preferences separately. The token covers training only.
- If you opt out of training, review your crawler rules separately, because the token does not change crawl rules for named crawlers [4].
- Do not remove existing provider-specific rules such as Google-Extended. Google and OpenAI say their company-specific controls keep working [1][2].
Disclosure: AVJ pays OpenAI standard prices for a ChatGPT Pro subscription and API access, receives no credits or special terms, and uses OpenAI and Anthropic models among its editorial tools (see our AI use policy). The AVJ Citation Index will also measure answers from OpenAI’s API. AVJ has no other commercial relationship with OpenAI or Anthropic.
Update log: Oct. 31, 2026 — First published.
Sources
[1] Mueller, J. (2026, Oct. 30). “Supporting a shared AI training control for websites.” Google Search Central Blog. Accessed Oct. 31, 2026. https://developers.google.com/search/blog/2026/10/ai-training-control
[2] OpenAI. (2026, Oct. 30). “Making it easier for websites to express AI training preferences.” Accessed Oct. 31, 2026. https://openai.com/index/helping-websites-express-ai-training-preferences
[3] Madhavan, K. (2026, Oct. 30). “Your content, your choice: clearer preferences for training and grounding.” Microsoft Bing Blogs. Accessed Oct. 31, 2026. https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/
[4] Kelley, K. (2026, Oct. 30). “Machine-Readable Training Preference Signal.” Internet-Draft draft-kelley-aipref-training-preference-signal-00. Work in progress. Accessed Oct. 31, 2026. https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
[5] The Verge. (2026, Oct. 30). “The biggest names in AI are backing a new way to opt out of scraping.” Trade press, context only. Accessed Oct. 31, 2026. https://www.theverge.com/tech/1023763/openai-google-microsoft-ai-scraping-standard-opt-out
How this was made
Prepared with AI assistance from the sources cited above. Editorial review by Nhi Phan (editor) and fact-checking by Jackies continue after publication; any change will be recorded in the update log and in our Corrections Policy and Log.