Key takeaways
- Decide training, search or answer retrieval, and user-requested fetches separately. A documented control for one use does not automatically control the others.
- Robots.txt states crawler preferences; it is not access control. Some operators say their user-requested fetchers may not follow it, so private content needs authentication.
- Google’s Search generative AI control and
Google-Extendedare different controls. The first governs named Google Search AI features; the second covers future Gemini training and grounding in Gemini Apps and Vertex AI.
Reference date: Nov. 11, 2026. Controls were checked against the operators’ pages as published on Nov. 11, 2026. A shared AI-training signal announced Oct. 30, 2026 has begun to be recognized by Anthropic and is announced for later implementation by Google, OpenAI and Microsoft. This guide reports operator documentation, not an AVJ test of how crawlers actually behave.
What kinds of AI access is a site deciding about?
Start with the purpose of the request, not with whether it comes from an AI company.
Training and data use. OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models. Anthropic says restricting ClaudeBot signals that a site’s future materials should be excluded from its training datasets. [2][3] Google-Extended and Applebot-Extended are robots.txt control tokens, not separate crawlers: Google says Google-Extended has no separate HTTP user agent, and Apple says Applebot-Extended does not crawl webpages. [4][21] Google says Google-Extended covers training of future Gemini models and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. [4]
Search and answer retrieval. OpenAI, Anthropic and Perplexity document OAI-SearchBot, Claude-SearchBot and PerplexityBot as search crawlers. [2][3][5] Google does not provide a separate crawler for AI Overviews and AI Mode; it says robots.txt rules for Googlebot are the crawl control for Search. [6] Microsoft says Bing and Copilot search experiences rely on the same core crawling, indexing and ranking foundation as traditional search. [13]
User-requested fetches and agents. ChatGPT-User, Claude-User and Perplexity-User fetch pages in response to users, but their documented robots.txt behavior differs. OpenAI says robots.txt rules “may not apply” to ChatGPT-User. Perplexity says Perplexity-User “generally ignores robots.txt rules.” Anthropic says its bots honor robots.txt directives. Google says its user-triggered fetchers generally ignore robots.txt rules; it does not say this of its automatic crawlers. [2][3][5][7]
Anthropic documents Claude-Agent separately: a browser hosted by Anthropic and driven by a Claude agent at a user’s direction. Anthropic says preferences for its crawlers and fetchers are independent of this traffic, and that anyone can send the same user-agent header, so site owners should check the published IP addresses as well. [18] A user-agent string is not proof of identity. The AVJ AI crawler directory lists operator tokens and verification pointers.
One fetch can also serve two uses. OpenAI says that if a site allows both OAI-SearchBot and GPTBot, it may use the results of one crawl for both. [2] Decide each use deliberately; listing crawler names is not always enough.
What does each control actually change?
| Control | Where it is set | What it changes, per the operator | What it does not do |
|---|---|---|---|
| Robots.txt rule for a named token | Site root | Asks a named crawler not to fetch specified paths | Not access authorization or a guarantee of removal from an index; some user-requested fetchers may not follow it [1][9] |
Google-Extended | Robots.txt | Training of future Gemini models and grounding in Gemini Apps and Vertex AI | Google says it does not affect inclusion or ranking in Google Search, which includes AI Overviews and AI Mode [4][6] |
| Search generative AI control | Search Console | Whether a site’s content can appear in, and be used as input to, AI Overviews, AI Mode and generative AI features in Discover | Not a training control; not a ranking or inclusion signal for other parts of Search [11] |
nosnippet, max-snippet | Page meta tag or HTTP header | Google: snippet display, and direct input to AI Overviews and AI Mode | Not removal from the index [12] |
data-nosnippet | Passage (HTML attribute) | Google: marked text is not used as a snippet. Bing: it prevents captions and may limit Copilot citation quality | Not a page-level exclusion or a confidentiality measure [6][12][13] |
noindex | Page meta tag or HTTP header | Google: removal from Search. Bing: removal from Bing search, Copilot and grounding API results | Has no effect at Google if robots.txt stops the crawler from seeing it [10][13] |
Bing NOARCHIVE, NOCACHE | Page | Today: limits Copilot and grounding use | Microsoft will stop reading them as training or grounding preferences after March 1, 2027 [8][13] |
| Microsoft AI Training and AI Grounding preferences | Training: robots.txt (shared token) or page; grounding: page only | Separate preferences for training and grounding, honored as Microsoft crawls from Feb. 1, 2027 | Not crawl access and not eligibility for traditional Bing results [8] |
__usage_ai-training | Robots.txt | A shared training preference described in an individual Internet-Draft | Not a crawl directive, not an IETF standard and not an AI-search opt-out [14] |
| Authentication | Server | Restricts access to authorized users | Cannot be replaced by robots.txt for confidential content [1][9] |
See also what Google says about AI Overviews and AI Mode and AVJ’s report on Bing’s new AI grounding control for publishers.
The robots.txt and noindex trap
Robots.txt asks crawlers not to fetch URLs; it does not remove pages from an index. Google says a disallowed URL can still be indexed if other pages link to it. [9]
To remove a page from Google Search, the noindex rule must be visible to Google’s crawler. If robots.txt blocks the page, Google never sees the rule. Google also does not support noindex inside robots.txt. [10]
Google’s AI-feature switch is not its training switch
The Search generative AI control in Search Console covers AI Overviews, AI Mode and generative AI features in Discover. Google says that excluding a site means its content won’t be eligible as an input to generate an AI response or preview in those features. The control does not affect AI training and is not a ranking or inclusion signal for other parts of Search. Google says exclusion generally takes a few days, and similar content from other sites can still appear. [11]
At page level, Google says nosnippet prevents content from being used as a direct input for AI Overviews and AI Mode, and max-snippet limits how much can be used. The max-snippet limit does not interrupt uses a publisher has separately permitted, such as in-page structured data or a license agreement with Google. [12] Google documents data-nosnippet as a way to keep marked text out of snippets. [12]
Google-Extended has different uses: training future Gemini models and grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search. [4] Blocking it will not turn off AI Overviews.
Microsoft’s transition has several dates
As of Nov. 11, 2026, Microsoft’s guidelines say NOINDEX removes a URL from Bing search, Copilot experiences and grounding API results. NOARCHIVE prevents content from being used in Copilot responses and grounding results, and NOCACHE limits Copilot to the URL, title and snippet. [13]
Starting Feb. 1, 2027, Microsoft says it will honor two independent preferences as it crawls. The AI Training preference uses the shared __usage_ai-training robots.txt token for a site or path, and a page-level ai-training rule can override it. AI Grounding is set at page level only; Microsoft says there is no site-wide robots.txt grounding preference. A DisallowExtended grounding value limits grounding to Microsoft-operated experiences and excludes grounding services Microsoft provides to third parties. [8]
Microsoft says these settings govern content use, not crawl access: an Allow does not override a robots.txt rule that blocks Bingbot. Disallowing grounding will reduce visibility in its AI experiences, and neither choice guarantees inclusion, traffic or payment. [8]
After March 1, 2027, Microsoft will no longer read NOARCHIVE and NOCACHE as training or grounding preferences. Sites using those tags as of Dec. 15, 2026 can record a temporary grounding preference in Bing Webmaster Tools; it lasts until the page is recrawled or Sept. 30, 2027, whichever comes first. There is no temporary training setting. Microsoft says pages using NOARCHIVE or NOCACHE without a temporary preference recorded by March 1 default to Allow for grounding until recrawled. [8]
What is the shared AI-training signal, and who applies it?
The proposal uses ordinary robots.txt groups keyed to a special policy token:
User-agent: __usage_ai-training
Disallow: /Under the proposal, this expresses a preference against AI-training use; it does not ask crawlers to stop fetching pages. The draft says training preferences and crawl rules are independent. A wildcard User-agent: * group must not be read as matching the token, and a file without the token expresses no preference, which is not the same as consent. [14]
The proposal is an individual Internet-Draft by an Anthropic-affiliated author, dated Oct. 30, 2026. Its IETF record says it is not endorsed by the IETF and has no formal standing; it expires May 3, 2027 unless replaced. It is separate from the AI Preferences working group’s own drafts, which define a Content-Usage attachment and a vocabulary with categories such as train-ai and ai-use. The two syntaxes are not interchangeable. [14][15]
| Operator | Status stated as of Nov. 11, 2026 | Qualification |
|---|---|---|
| Anthropic | Began recognizing the signal Oct. 30, 2026; rollout across its services over time | A robots.txt rule that names ClaudeBot takes priority; otherwise Anthropic follows the general signal [3] |
| Intends to support it in early 2027 | Plans to treat an opt-out as a disallow of Google-Extended when that token is not named [4][16] | |
| OpenAI | Aims to implement it “in the coming months” | No date given; existing robots.txt controls continue to apply [17] |
| Microsoft | Will honor the training preference from Feb. 1, 2027 | Page-level ai-training rules can override robots.txt; grounding is a separate control [8] |
| Perplexity | No statement found in its crawler documentation | Perplexity says PerplexityBot “is not used to crawl content for AI foundation models” [5] |
Anthropic says the way it reads and applies these settings may change as standards evolve. [3] The training controls address future collection or future models: Anthropic refers to a site’s “future materials” and Google to “future generations” of Gemini models. None of the reviewed documents says an opt-out removes content from models already trained. [3][4]
For background, see AVJ’s report on the shared AI-training signal.
How should different kinds of sites decide?
No operator documents a universal “allow all” or “block all” answer, and none reports outcomes by type of site. Start by writing down four goals: eligibility for AI search and citations; whether to permit training use; whether user-requested fetches and agents should reach public pages; and which content must stay private. Then choose the documented control for each goal. The examples below are framed as “if your goal is X, the documented control is Y.”
Publisher
If the goal is to decline training while staying eligible for search features, OpenAI documents independent GPTBot and OAI-SearchBot settings, and Anthropic documents separate ClaudeBot and Claude-SearchBot controls. [2][3] That does not guarantee how an app cites a site, and the reviewed documents do not say whether every API product applies the same settings.
Excluding content from an AI answer feature has stated costs. Google says a site excluded through its Search generative AI control won’t receive traffic or impressions from those features. Microsoft says disallowing grounding reduces visibility in its AI experiences. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. [2][8][11]
Private reporting, paid archives and member-only pages need server-side authorization, not just a disallow rule. RFC 9309 notes that listing paths in robots.txt exposes them publicly. [1]
SaaS
Marketing pages, documentation and logged-in application areas can have different goals. If discovery of public documentation is wanted, keep it crawlable and indexable; protect application, account and customer-data areas with authentication. If user agents or hosted browsers are part of how customers use the product, verify traffic by user agent and the operator’s published IP ranges rather than by the claimed name alone. [5][18]
Ecommerce
Before adding nosnippet or max-snippet to product copy, note that Google says those rules also limit direct input to AI Overviews and AI Mode. [12] Disallow rules can keep crawlers out of cart, checkout or internal-search paths, but they do not make private data safe. Decisions about licensing or access are separate from crawler access.
OpenAI says OAI-AdsBot visits only pages submitted as ChatGPT ads, so it should not be confused with training or search crawling. [2]
Local business
For a small site, blocking Googlebot or Bingbot to avoid AI answers also affects ordinary search. Google has no separate AI-search crawler token, and Microsoft says Copilot relies on the same crawling foundation as Bing search. [6][13] If the goal is to stay out of Google’s AI features only, Google’s documented tool is the Search generative AI control; at Bing, it is the page-level controls described above. [8][11]
No operator source reviewed here says that allowing training earns traffic or that opting out of training preserves visibility. The visibility effects that operators do state concern search and grounding exclusions, not training preferences. [2][3][8][11]
What does a segmented robots.txt look like?
This is a hypothetical example that separates training from search. It is not a recommendation for every site and is not exhaustive; for example, it leaves out Common Crawl’s CCBot, which collects an open dataset.
# Search and answer crawlers: allowed, except technical paths
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /wp-admin/
# Training crawlers and training-control tokens: disallowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# Shared AI-training policy token (individual Internet-Draft, Oct. 30, 2026)
User-agent: __usage_ai-training
Disallow: /
# All other crawlers, including Googlebot and Bingbot
User-agent: *
Disallow: /wp-admin/Four details matter:
- RFC 9309 says groups that match a crawler’s product token are combined, and the wildcard group applies only when no group matches. Within the applicable group, the most specific matching rule must be used. Repeat necessary technical disallows in each named group rather than relying on the wildcard group. [1]
- The shared policy token must be named explicitly;
*does not match it. [14] - Anthropic says a rule naming
ClaudeBottakes priority over the shared signal, and Google plans to apply the signal toGoogle-Extendedonly when that token is not named. Keep the named rules and the shared token consistent. [3][16] - Disallowing
Google-Extendedis not only a training choice. Google says it also covers grounding in Gemini Apps and Vertex AI, and that sites which allow it are eligible to appear in responses and receive clicks via grounding in Gemini Apps. [4][16]
AVJ checked the example with two Python parsers, Python 3.13.5’s urllib.robotparser and Protego 0.7.0, on Nov. 6 and again on Nov. 9, 2026. In both runs, all 15 test cases returned the expected allow or block result in each parser, covering search crawlers, training and control tokens, user-requested fetchers and technical paths. These are parser checks, not tests of Google’s parser, of live crawler behavior or of whether an operator honors a preference. Neither parser implements the shared token’s training-preference meaning; the token was tested only for ordinary path matching. [19]
AVJ’s own robots.txt puts * and 11 named tokens in one group that allows crawling except selected technical paths, and it explicitly allows several training-control tokens. AVJ’s choice reflects its aim to be found, understood and cited accurately by AI systems, accepting that training use may not bring traffic. The file has no __usage_ai-training group; under the draft, that means no preference is expressed through the token. In AVJ’s parser check, a first-match parser and a longest-match parser handled the file’s Allow: / line differently. That is a difference between parsers, not evidence of live crawler behavior. [19]
Which mistakes break these controls?
- Using robots.txt as a password. It is not access authorization or a substitute for content security. Use server-side authentication for private content. [1]
- Blocking the page whose
noindexmust be read. Google cannot act on anoindexrule it cannot crawl, and it does not supportnoindexin robots.txt. [10] - Confusing training use with AI-search eligibility.
Google-Extendeddoes not switch off AI Overviews. Anthropic documents its training signal andClaudeBotseparately fromClaude-SearchBot, which governs search indexing. [3][4] - Assuming every user-requested fetcher obeys, or ignores, robots.txt. Behavior is documented operator by operator. [2][3][5][7]
- Replacing robots.txt with an IP block for Anthropic. Anthropic says blocking its IP addresses may not work reliably as an opt-out because it prevents its crawler from reading the robots.txt file. [3]
- Treating a user-agent string as identity. It can be spoofed. Check published IP lists where operators provide them. [2][5][18]
- Missing subdomains or property inheritance. Anthropic asks site owners to add rules for every subdomain they want to opt out; in Search Console, a property inherits the Search generative AI control from its parent unless configured otherwise. [3][11]
- Letting Microsoft’s legacy controls lapse unnoticed.
NOARCHIVEandNOCACHEstop being read for training or grounding after March 1, 2027. [8] - Expecting changes to apply immediately. OpenAI says search results can take about 24 hours to reflect a robots.txt change; Perplexity says up to 24 hours; Google says its Search control generally takes a few days. These are operator estimates, not guarantees. [2][5][11]
What remains unknown?
The individual draft may change or never become a standard. Google’s and OpenAI’s implementation dates are prospective. Perplexity’s crawler documentation does not address the shared training token. The reviewed documents do not say whether API products respect site opt-outs in the same way as consumer apps. No reviewed operator statement says that a training opt-out removes already-trained material, or predicts its effect on traffic or citations.
The operator statements in this guide describe what each company says its systems do. AVJ has not observed or tested whether crawlers follow them.
EU legal context: what this guide does not decide
Article 4(3) of Directive (EU) 2019/790 makes one text-and-data-mining exception conditional on rightholders not having reserved their rights “in an appropriate manner, such as machine-readable means” for content made publicly available online; Recital 18 discusses machine-readable means. [20] This guide does not say whether any robots.txt token is an appropriate reservation in a particular jurisdiction or case. This is background, not legal advice. For related coverage, see AVJ’s report on an EU Advocate General’s opinion on AI chatbot answers and press publishers’ rights.
About this guide
Sources were checked against the pages as published on Nov. 9, 2026, and checked again on Nov. 11, 2026.
How this was made: AVJ used AI tools to help gather the cited sources, compare each claim with them, run the parser checks and draft and line-edit this guide, as described in our AI Use Policy. The guide was then edited by Nhi Phan and fact-checked by Jackies against the cited sources. A further editorial review follows publication. This is AVJ’s editorial review, not peer review.
Disclosure: AVJ pays standard published rates for basic services from OpenAI (a ChatGPT Pro subscription and API access), Perplexity (API access), Google and Microsoft, and uses OpenAI and Anthropic models among its editorial tools (see our AI Use Policy). AVJ uses these services as an ordinary customer and has received no free credits, promotions or other compensation from OpenAI, Anthropic, Google, Microsoft or Perplexity. AVJ receives no sponsorship and has no partnership with any company named in this guide. The AVJ Citation Index, whose methodology is still under review, plans to measure answers from AI companies’ APIs, including OpenAI’s, Anthropic’s and Perplexity’s. The shared-signal draft discussed above was written by an Anthropic-affiliated author. If AVJ accepts sponsorship in the future, it will disclose this publicly.
Changelog
- Nov. 12, 2026. First published.
Related on AVJ
- AI crawler directory: who crawls your site for AI, and how to control it
- Google: robots.txt governs crawling, not AI use. What does?
- Glossary: grounding, AI Overviews, AI Mode
Sources
[1] Koster, M., G. Illyes, H. Zeller and L. Sassman. Robots Exclusion Protocol. RFC 9309. IETF, September 2022; accessed Nov. 11, 2026. https://www.rfc-editor.org/rfc/rfc9309
[2] OpenAI. Overview of OpenAI Crawlers. OpenAI developer documentation; accessed Nov. 11, 2026. https://developers.openai.com/api/docs/bots
[3] Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler? Claude Help Center. Modified Nov. 4, 2026; accessed Nov. 11, 2026. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
[4] Google. List of Google’s common crawlers. Google Crawling Infrastructure. Updated Oct. 30, 2026; accessed Nov. 11, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
[5] Perplexity. Perplexity Crawlers. Perplexity documentation; accessed Nov. 11, 2026. https://docs.perplexity.ai/docs/resources/perplexity-crawlers
[6] Google. AI features and your website. Google Search Central. Updated Dec. 10, 2025; accessed Nov. 11, 2026. https://developers.google.com/search/docs/appearance/ai-features
[7] Google. List of Google user-triggered fetchers. Google Crawling Infrastructure. Updated Aug. 19, 2026; accessed Nov. 11, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
[8] Microsoft Bing. Your content, your choice: clearer preferences for training and grounding. Bing Blogs, Oct. 30, 2026; accessed Nov. 11, 2026. https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/
[9] Google. Introduction to robots.txt. Google Search Central. Updated Dec. 10, 2025; accessed Nov. 11, 2026. https://developers.google.com/search/docs/crawling-indexing/robots/intro
[10] Google. Block Search indexing with noindex. Google Search Central. Updated Dec. 10, 2025; accessed Nov. 11, 2026. https://developers.google.com/search/docs/crawling-indexing/block-indexing
[11] Google. Search generative AI control. Search Console Help. Rollout noted as of Aug. 31, 2026; accessed Nov. 11, 2026. https://support.google.com/webmasters/answer/16908024?hl=en
[12] Google. Robots meta tag, data-nosnippet, and X-Robots-Tag specifications. Google Search Central. Updated March 24, 2026; accessed Nov. 11, 2026. https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
[13] Microsoft Bing. Bing Webmaster Guidelines. Bing Webmaster Tools help; accessed Nov. 11, 2026. https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a
[14] Kelley, K. (Anthropic). Machine-Readable Training Preference Signal. Individual Internet-Draft draft-kelley-aipref-training-preference-signal-00, Oct. 30, 2026; expires May 3, 2027. Not endorsed by the IETF; no formal standing. Accessed Nov. 11, 2026. https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
[15] IETF AI Preferences Working Group. A Vocabulary For Expressing AI Usage Preferences, draft-ietf-aipref-vocab-08, Sept. 13, 2026, and Associating AI Usage Preferences with Content in HTTP, draft-ietf-aipref-attach-05, Aug. 18, 2026. Working-group drafts, not standards. Accessed Nov. 11, 2026. https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/ ; https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/
[16] Google. Supporting a shared AI training control for websites. Google Search Central Blog, Oct. 30, 2026; accessed Nov. 11, 2026. https://developers.google.com/search/blog/2026/10/ai-training-control
[17] OpenAI. Making it easier for websites to express AI training preferences. Oct. 30, 2026; accessed Nov. 11, 2026. https://openai.com/index/helping-websites-express-ai-training-preferences/
[18] Anthropic. How do Anthropic-hosted browsers identify themselves when they browse the web on a user’s behalf? Claude Help Center. Modified Nov. 4, 2026; accessed Nov. 11, 2026. https://support.claude.com/en/articles/18234133
[19] AVJ. Live robots.txt, retrieved Nov. 11, 2026, and checks of this guide’s hypothetical example with Python 3.13.5 urllib.robotparser and Protego 0.7.0, run Nov. 6 and Nov. 9, 2026. Parser checks, not crawler behavior. https://aivisibilityjournal.com/robots.txt
[20] European Parliament and Council. Directive (EU) 2019/790 on copyright and related rights in the Digital Single Market, Article 4 and Recital 18. EUR-Lex; accessed Nov. 11, 2026. https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32019L0790
[21] Apple. About Applebot. Apple Support; accessed Nov. 11, 2026. https://support.apple.com/en-us/119829