Key takeaways
- OpenAI and Anthropic document separate controls for training and search functions, but that distinction does not describe every entry in this directory.
- Google, OpenAI, Perplexity, Amazon, and Meta document exceptions to robots.txt for certain user-requested fetches.
- A user-agent string is not proof of identity; operators including Google, Microsoft, and Common Crawl document ways to verify requests using additional information.
Reference date: Oct. 29, 2026. Sources were checked against the pages as published on Oct. 29, 2026. The descriptions below report operator documentation, not an AVJ test of actual crawler behavior.
What counts as an AI crawler in this directory?
This directory covers documented crawlers, user-triggered fetchers, and data-use control tokens relevant to AI training or AI-related search. Each row summarizes purpose and robots.txt behavior from the linked operator documentation.
The organizing categories are:
- Training: content collection described by the operator as potentially contributing to model training.
- Search: crawling or indexing described as supporting search or answer retrieval.
- User-initiated fetch: page retrieval associated with a user’s request.
- Other uses and controls: dataset collection, advertising, previews, and controls over already-crawled content.
These are directory categories, not an exhaustive technical standard. The entries can overlap: Meta describes meta-externalagent as supporting both foundation-model training and direct content indexing.
Google-Extended and Applebot-Extended are control tokens, not separate crawlers. Google uses existing crawlers to collect content, while Apple says Applebot-Extended determines how Applebot’s collected data may be used.
Directory of AI crawlers and control tokens
All operator descriptions in this directory are as of Oct. 29, 2026. A documentation gap means the cited page does not explicitly describe that token’s robots.txt behavior.
RFC 9309 requires case-insensitive matching of user-agent product tokens; this is separate from path matching, which the standard says should be case-sensitive.
| Token or HTTP identifier | Documented purpose | Documented robots.txt behavior | Source |
|---|---|---|---|
Googlebot |
Crawls for Google Search; preferences affect Discover and all Google Search features, among other products. | Google’s common crawlers obey robots.txt during automatic crawling. | [2] |
Google-Extended |
Controls use of crawled content for future Gemini model training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It has no separate HTTP user-agent string. | Publishers can disallow relevant content; the setting does not affect Google Search inclusion or ranking. | [2] |
GoogleOther |
A general-purpose crawler for product teams, including occasional internal research and development crawls; preferences do not affect a specific product. | Follows robots.txt as a common crawler. | [2] |
Google-CloudVertexBot |
Handles site-owner-requested crawls for building Vertex AI Agents; preferences do not affect Google Search or other products. | Follows robots.txt as a common crawler. | [2] |
Google-Agent |
An HTTP user-agent identifier for agents on Google infrastructure that navigate websites and perform user-requested actions. | Google says user-triggered fetchers generally ignore robots.txt. | [3] |
Google-GeminiNotebook |
An HTTP user-agent identifier for fetching URLs supplied as sources by Gemini Notebook users; the former Google-NotebookLM agent was supported until August 2026. |
Falls under Google’s user-triggered-fetcher policy, which generally ignores robots.txt. | [3] |
Google separately documents a Search Console setting for inclusion in AI Overviews, AI Mode, and generative AI features in Discover. That control concerns inclusion in those features, not AI training.
OpenAI
OpenAI documents independent settings for OAI-SearchBot and GPTBot, allowing publishers to permit search access while declining training use. Its documentation says search systems may take about 24 hours to adjust after a robots.txt change.
| Token | Documented purpose | Documented robots.txt behavior | Source |
|---|---|---|---|
OAI-SearchBot |
Supports website discovery in ChatGPT search results; opted-out sites may still appear as navigational links. | OpenAI recommends allowing the token and its published IP ranges for search access; opted-out sites are excluded from ChatGPT search answers. | [5] |
GPTBot |
Crawls content that may contribute to generative foundation-model training. | Disallowing it indicates that the site’s content should not be used for that training. | [5] |
ChatGPT-User |
Retrieves pages for certain user actions in ChatGPT and Custom GPTs, rather than automatic web crawling. | Robots.txt may not apply to these user-initiated actions; search opt-outs use OAI-SearchBot instead. |
[5] |
OAI-AdsBot |
Checks pages submitted as ChatGPT ads for safety and may assess landing-page relevance; collected data is not used for foundation-model training. | The page does not specify a robots.txt policy for this token; it limits visits to pages submitted as ads. | [5] |
Anthropic
| Token | Documented purpose | Documented robots.txt behavior | Source |
|---|---|---|---|
ClaudeBot |
Collects web content that could contribute to generative-model training. | Restricting access signals that future site material should be excluded from Anthropic’s training datasets. | [6] |
Claude-SearchBot |
Crawls and analyzes content to improve search responses. | Disabling it prevents Anthropic’s system from indexing the content for search optimization. | [6] |
Claude-User |
Retrieves website content in response to questions from Claude users. | Disabling it prevents retrieval of the site’s content for a user query. | [6] |
Anthropic says its bots honor robots.txt directives and support the nonstandard Crawl-delay extension. Apply opt-out rules to every subdomain you wish to exclude. Anthropic warns that blocking its IP addresses can interfere with reading robots.txt and may not provide a persistent opt-out.
Perplexity
Perplexity says its settings operate independently and may take up to 24 hours to reflect changes.
| Token | Documented purpose | Documented robots.txt behavior | Source |
|---|---|---|---|
PerplexityBot |
Supports linked websites in Perplexity search results, not content collection for AI foundation models. | Perplexity recommends allowing the token in robots.txt and permitting its published IP ranges for search access. | [7] |
Perplexity-User |
Retrieves pages for user questions; it is not used for web crawling or collecting foundation-model training content. | Perplexity says the fetcher generally ignores robots.txt because a user requested the retrieval. | [7] |
Apple
| Token | Documented purpose | Documented robots.txt behavior | Source |
|---|---|---|---|
Applebot |
Supports search features including Spotlight, Siri, and Safari; collected data may also support Apple foundation-model training and AI-answer context. | Respects standard directives in general search crawls; follows Googlebot instructions when Applebot is absent but Googlebot is named; does not follow crawl-delay. |
[8] |
Applebot-Extended |
Controls training use of Applebot-collected content; it does not crawl pages. | Disallowing it opts content out of training Apple’s general-purpose foundation models without excluding those pages from search results. | [8] |
Apple separately documents nosnippet as a way to exclude content from use as additional context for AI-generated output, including broad world knowledge answers in Siri and Search.
Amazon
Amazon says each user-agent setting is independent and changes may take about 24 hours to register. Its automated crawlers respect the Robots Exclusion Protocol’s user-agent and allow/disallow directives, may use a robots.txt copy cached within the preceding 30 days, and do not support crawl-delay. Amazon also documents noarchive as excluding a page from model training.
| Token | Documented purpose | Documented robots.txt behavior | Source |
|---|---|---|---|
Amazonbot |
Collects information to improve Amazon products and services; content may be used to train Amazon AI models. | Covered by Amazon’s automated-crawling robots.txt policy. | [9] |
Amzn-SearchBot |
Supports search experiences such as Alexa, not generative AI model training. | Covered by the automated-crawling policy; if robots.txt omits this token but allows other search bots, it follows directives for those bots. | [9] |
Amzn-User |
Fetches live information for user actions such as Alexa questions; it does not crawl for generative AI model training. | May not follow all robots.txt directives because its actions can be user-initiated. | [9] |
Meta
Meta says robots.txt changes can take up to 24 hours to take effect because its crawlers may cache the file for that period.
| Token | Documented purpose | Documented robots.txt behavior | Source |
|---|---|---|---|
meta-externalagent |
Crawls for foundation-model training or product improvement through direct indexing. | Meta directs publishers to add a disallow rule to block the relevant crawler. | [10] |
meta-webindexer |
Supports Meta AI search-result quality and linking to publisher content in responses. | Meta describes allowing it in robots.txt for access and provides general disallow guidance. | [10] |
meta-externalfetcher |
Fetches individual links at a user’s request and supports agentic AI functions. | May bypass robots.txt. | [10] |
facebookexternalhit |
Collects titles, descriptions, and thumbnails for links shared through Meta apps. | Meta provides general disallow guidance but documents exceptions for security or integrity checks. | [10] |
meta-externalads |
Crawls for advertising and other business-related products and services. | Meta provides disallow guidance and an example rule for this token. | [10] |
Microsoft
| Token | Documented purpose | Documented robots.txt behavior | Source |
|---|---|---|---|
Bingbot |
Bing’s main crawler for its day-to-day crawling needs. | Microsoft documents robots.txt controls and hourly crawl-rate settings in Bing Webmaster Tools. | [11] |
Microsoft’s crawler page also lists AdIdxBot, BingPreview, MicrosoftPreview, and BingVideoPreview for advertising checks or previews.
Bing’s guidelines describe a shared crawling, indexing, and ranking foundation for traditional search and Bing and Copilot search experiences; the guidelines page lists no separate Copilot robots.txt token. The guidelines say NOARCHIVE prevents content use in Copilot responses and grounding results, while NOCACHE limits Copilot to the URL, title, and snippet.
Microsoft’s Sept. 22, 2023, announcement also described training restrictions: NOARCHIVE content would not be used for foundation-model training, while NOCACHE limited training use to URLs, titles, and snippets. That historical announcement explicitly gave NOCACHE precedence when both tags were present and said tagged content could still appear in search results.
Other operators
| Operator | Token | Documented purpose | Documented robots.txt behavior |
|---|---|---|---|
| Common Crawl | CCBot |
Collects data for Common Crawl’s freely available web-crawl repository. | Documents User-agent: CCBot with Disallow: / to prevent crawling and links to an opt-out registry. |
| Mistral AI | MistralAI-Training |
Collects content for datasets used to train Mistral generative AI models, not search indexing or live Vibe answers. | Publishers can disallow it in robots.txt. |
| Mistral AI | MistralAI-Index |
Indexes content for Mistral search in Vibe; collected content is not used for generative AI training. | The page mentions robots.txt controls generally but does not specify this token’s compliance behavior. |
| Mistral AI | MistralAI-User |
Retrieves pages for user actions in Vibe, not automatic crawling or training. | Mistral says the token governs which sites user requests can access, without giving a token-specific allow/disallow example. |
| Diffbot | Diffbot |
Crawls for a general search engine and source discovery through Diffbot’s Knowledge Graph and search services, not AI training. | Follows disallow and crawl-delay by default; specific arrangements can permit overrides. |
| Diffbot | Diffbot-User |
Identifies requests made for a person browsing a URL through Diffbot software. | The page does not separately specify this identifier’s robots.txt behavior. |
| DuckDuckGo | DuckAssistBot |
Retrieves pages in real time for AI-assisted search answers, not AI-model training. | A disallow takes effect after 72 hours; opting out does not affect organic search rankings or ordinary search-result inclusion. |
How do I block training but stay in AI search?
Use separate training and search controls where the operator documents them. OpenAI explicitly supports allowing OAI-SearchBot while disallowing GPTBot, and Anthropic describes separate training and search bots. Apple allows a training opt-out through Applebot-Extended while retaining search eligibility.
Amazon and Mistral also distinguish training-related crawling from search crawling, although Mistral’s page does not explicitly state token-specific robots.txt compliance for MistralAI-Index.
Hypothetical: The example below blocks selected training crawlers, a dataset crawler, and data-use control tokens while allowing selected search crawlers. It is not a site-specific recommendation.
“`text
Training crawlers, dataset crawlers and control tokens: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /
Search crawlers: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
“`
RFC 9309 permits multiple user-agent lines to share a group of rules. We tested this example on Oct. 29, 2026 with Protego 0.7.0 and Python 3.13’s urllib.robotparser: both blocked the six tokens in the first group and allowed the three in the second.
That is a parser check, not evidence of how the operators’ live crawlers behave.
Keep these qualifications in mind:
- The blocked group is not training-only.
CCBotcollects an open web dataset, andGoogle-Extendedcontrols specified grounding uses as well as training. - Some purposes overlap. Meta describes
meta-externalagentas supporting training or direct indexing. - Google Search has a separate inclusion control. Google-Extended does not affect Search inclusion; Search Console’s Search generative AI control manages inclusion in AI Overviews and AI Mode.
- Changes are not necessarily immediate. OpenAI and Amazon describe roughly 24-hour adjustment periods, Perplexity up to 24 hours, Meta up to 24 hours of caching, and DuckDuckGo a 72-hour delay.
Does robots.txt prevent access?
No. RFC 9309 distinguishes robots.txt from access authorization and says it is not a substitute for security measures.
The operators document several user-request exceptions. Google’s user-triggered fetchers generally ignore robots.txt, and OpenAI says its rules may not apply to ChatGPT-User. Perplexity documents a similar exception for Perplexity-User; Amazon says Amzn-User may not follow every directive; Meta says meta-externalfetcher may bypass the file.
Do not assume the same exception applies to every user-requested fetcher: Anthropic says its bots honor robots.txt and describes disabling Claude-User to prevent retrieval.
For private or confidential content, Google recommends password protection so that only authorized users can access it.
How do I verify that a crawler is genuine?
Do not rely on the user-agent string alone. Google and Microsoft warn that these strings can be spoofed, and Common Crawl reports requests falsely identifying themselves as CCBot.
The following verification references reflect the operators’ pages as of Oct. 29, 2026.
| Operator | Published verification information | Evidence |
|---|---|---|
Common-crawler IP ranges in common-crawlers.json and documented reverse DNS masks; user-triggered fetchers have separate IP-range files, including user-triggered-agents.json for Google-Agent. |
||
| OpenAI | Separate published IP lists at https://openai.com/searchbot.json, https://openai.com/gptbot.json, https://openai.com/chatgpt-user.json, and https://openai.com/adsbot.json. |
|
| Anthropic | A linked source-IP list in its crawler article; see the article for the current list. | |
| Perplexity | Published lists at https://www.perplexity.ai/perplexitybot.json and https://www.perplexity.ai/perplexity-user.json (as of Oct. 30, 2026); its WAF guidance combines IP and user-agent checks. |
|
| Apple | Reverse DNS under *.applebot.apple.com and a linked Applebot IP CIDR list. |
|
| Amazon | Separate published IP-address pages for Amazonbot, Amzn-SearchBot, and Amzn-User. | |
| Microsoft | The Verify Bingbot tool and linked verification methods. | |
| Common Crawl | Dedicated IP ranges, reverse DNS except over IPv6, and https://index.commoncrawl.org/ccbot.json. |
|
| Mistral AI | Separate links to published MistralAI-User and MistralAI-Index IP addresses. | |
| DuckDuckGo | An IP list on the crawler page and https://duckduckgo.com/duckassistbot.json. |
|
| Meta | Its page advises allowing FacebookExternalHit by user-agent strings or IP addresses but does not provide an IP list or reverse DNS pattern. |
Google also describes an experiment with Web Bot Auth for Google-Agent using the identity https://agent.bot.goog.
Which crawler names remain unverified?
Do not treat a hypothetical documentation example as an active crawler announcement.
As of Oct. 29, 2026, Cohere’s page says it does not use bots or user agents to crawl or scrape web content for foundation-model training at that time. Its bot-name table contains no named active bot, and Coherebot appears in instructions for a possible future training crawler.
That page therefore does not establish Coherebot as an active training crawler.
What does this directory not establish?
The operator pages describe declared purposes and controls. They do not, by themselves, establish whether every request follows those policies.
The parser check above tests interpretation of the example’s rules. It does not test network access, crawler identity, model-training use, or inclusion in an AI answer.
About this guide
Sources were checked against the pages as published on Oct. 29, 2026.
How this was made: AVJ used AI tools to help compare claims with the cited sources and to line-edit this guide, as described in our AI Use Policy. Nhi Phan reviews the published version.
Update, Oct. 30, 2026: Updated the links to Perplexity’s published IP-address lists, which Perplexity moved to perplexity.ai in its crawler documentation revised Oct. 29, 2026. The lists themselves are unchanged.
Clarification, Oct. 30, 2026: Added the “How this was made” label to the existing disclosure.
Disclosure: AVJ pays OpenAI standard prices for a ChatGPT Pro subscription and for API access, and receives no credits or special terms. AVJ plans to measure answers from OpenAI and other AI companies in its Citation Index.
Related on AVJ
- What Google actually says about AI Overviews and AI Mode
- Glossary: grounding, retrieval-augmented generation, AI visibility
- News: Cloudflare pilot added pages to OpenAI’s search index
Sources
[1] Koster, M., G. Illyes, H. Zeller, and L. Sassman. Robots Exclusion Protocol. RFC 9309. IETF. 2022; accessed Oct. 29, 2026. https://www.rfc-editor.org/rfc/rfc9309
[2] Google. List of Google’s common crawlers. Google Crawling Infrastructure. Updated Oct. 15, 2026; accessed Oct. 29, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
[3] Google. List of Google user-triggered fetchers. Google Crawling Infrastructure. Updated Aug. 19, 2026; accessed Oct. 29, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
[4] Google. Overview of Google crawlers and fetchers (user agents). Google Crawling Infrastructure. Updated June 12, 2026; accessed Oct. 29, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers
[5] OpenAI. Overview of OpenAI Crawlers. OpenAI developer documentation; accessed Oct. 29, 2026. https://developers.openai.com/api/docs/bots
[6] Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler? Claude Help Center. April 7, 2026; accessed Oct. 29, 2026. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
[7] Perplexity. Perplexity Crawlers. Perplexity documentation; accessed Oct. 30, 2026. https://docs.perplexity.ai/docs/resources/perplexity-crawlers
[8] Apple. About Applebot. Apple Support. Published Sept. 4, 2026; accessed Oct. 29, 2026. https://support.apple.com/en-us/119829
[9] Amazon. Amazon Web Search crawlers. Amazon Developer. Updated Oct. 26, 2026; accessed Oct. 29, 2026. https://developer.amazon.com/en-US/amazonbot
[10] Meta. Meta Web Crawlers. Meta for Developers; accessed Oct. 29, 2026. https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
[11] Microsoft. Overview of Bing crawlers (user agents). Bing Webmaster Tools Help; accessed Oct. 29, 2026. https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0
[12] Common Crawl. CCBot; accessed Oct. 29, 2026. https://commoncrawl.org/ccbot
[13] Mistral AI. Mistral crawlers. Mistral Docs; accessed Oct. 29, 2026. https://docs.mistral.ai/robots
[14] DuckDuckGo. Is DuckAssistBot related to DuckDuckGo? DuckDuckGo Help Pages; accessed Oct. 29, 2026. https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
[15] Google. Search generative AI control. Search Console Help; accessed Oct. 29, 2026. https://support.google.com/webmasters/answer/16908024
[16] Google. Control what you share with Google Search. Google Search Central. Updated Oct. 21, 2026; accessed Oct. 29, 2026. https://developers.google.com/search/docs/crawling-indexing/control-what-you-share
[17] Microsoft. Bing Webmaster Guidelines. Bing Webmaster Tools Help; accessed Oct. 29, 2026. https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a
[18] Microsoft Bing. Announcing new options for webmasters to control usage of their content in Bing Chat. Bing Webmaster Blog. Sept. 22, 2023; accessed Oct. 29, 2026. https://blogs.bing.com/webmaster/2023/9/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat/
[19] Common Crawl. Common Crawl — Open Repository of Web Crawl Data. Home page; accessed Oct. 29, 2026. https://commoncrawl.org/
[20] Diffbot. Does Diffbot respect robots.txt? Diffbot documentation; accessed Oct. 29, 2026. https://www.diffbot.com/docs/crawl/faq/robots-txt
[21] Cohere. Cohere Web Crawlers. Cohere documentation; accessed Oct. 29, 2026. https://docs.cohere.com/docs/cohere-web-crawlers
[22] AVJ. Parser test run Oct. 29, 2026. Protego 0.7.0 and Python 3.13.5 urllib.robotparser; results cover the six blocked and three allowed tokens in the displayed example.