Key takeaways
- In a Google Search Central video published Oct. 21, 2026, Google’s Gary Illyes said robots.txt “doesn’t dictate what a system does with the data after it has already crawled it,” according to Search Engine Journal’s account of the episode.
- The same day, Google added its site-wide Search generative AI control and its snippet controls to the “Control what you share with Google Search” page, describing them as existing options.
- A robots.txt rule for stating AI usage preferences, Content-Usage, is an IETF working group draft co-written by Google’s Gary Illyes; as of Oct. 23, 2026, the IETF Datatracker listed its IESG state as “I-D Exists.”
Direct answer: According to Google, robots.txt only says which URLs a crawler may fetch. To keep a site’s links and content out of AI Overviews and AI Mode, Google points to the site-wide Search generative AI control in Search Console. Page-level snippet rules such as nosnippet instead limit how a page’s content is shown and used as a direct input. Gemini training and grounding fall under the separate Google-Extended token.
What happened: On Oct. 21, 2026, Google updated its Control what you share with Google Search page. Its documentation changelog says the update added “existing options such as the robots meta tag, data-nosnippet, X-Robots-Tag, and the Search generative AI control,” and gives the reason as: “To keep the page fresh and reflect the current set of options available to site owners.” Search Engine Land reported the change the same day. The same day, Google Search Central published “What can you actually put in a robots.txt?”, an episode of its Non-Canonical Cabinet video series with Cherry Prommawin and Gary Illyes of Google Search Relations.
What did Google say about robots.txt and AI?
The video’s last segment addresses whether robots.txt can allow crawling but keep content out of generative AI training. Illyes answered, according to Search Engine Journal: “So as defined by RFC 9309, robots.txt is an access preference protocol.” It “doesn’t dictate what a system does with the data after it has already crawled it,” and “disallow bluntly means do not enter the room.” To keep a specific AI crawler out, site owners must “target their specific user agent tokens explicitly in the file.”
That matches the standard itself. RFC 9309 says its rules “are not a form of access authorization.” The video also says Google’s parser recognizes four rules (user-agent, allow, disallow and sitemap) and discards lines it does not understand, so a made-up rule against AI training has no effect.
Which Google control does what?
| Control | What Google’s documentation says it does | Applies to |
|---|---|---|
robots.txt disallow | Tells a crawler which paths not to fetch; Google’s blocking guide lists it as applicable to “images and video” | URL paths, per crawler token |
noindex | Keeps content out of Google Search results | A page |
nosnippet (robots meta tag or X-Robots-Tag) | No text snippet in any search result, including AI Overviews and AI Mode, and “will also prevent the content from being used as a direct input for AI Overviews and AI Mode” | A page |
max-snippet:[number] | Caps snippet length (0 means no snippet; -1 lets Google choose) and “will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode” | A page |
data-nosnippet | Marks text not to be used as a snippet | Part of a page (span, div, section) |
| Search generative AI control | Excludes a site’s links and content from AI Overviews, AI Mode and generative AI features in Google Discover | A Search Console property, site-wide |
Google-Extended | Governs use for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI; does not affect inclusion in Google Search | robots.txt token, no separate user agent |
Google spells out trade-offs for the Search generative AI control. Excluded sites “won’t receive any traffic or impressions from these features,” and the control “doesn’t affect AI training.” Google says exclusion generally takes a few days. Google says it rolled the control out to all websites worldwide as of Aug. 31, 2026.
Is there a robots.txt rule for AI use?
Not yet as a standard. An Internet-Draft by Gary Illyes of Google and Martin Thomson of Mozilla, “Associating AI Usage Preferences with Content in HTTP” (version 05, dated Aug. 19, 2026), would add a Content-Usage rule to robots.txt and a matching HTTP header. It would update RFC 9309 if approved. Under the draft, allow and disallow keep governing crawling, and usage preferences “apply only to those resources that can be crawled.” Its example is Content-Usage: train-ai=n.
As of Oct. 23, 2026, the IETF Datatracker lists the draft as a working group document with IESG state “I-D Exists,” and a working group milestone of August 2026 for sending it to the IESG. The draft expires Feb. 20, 2027. The companion vocabulary draft that defines labels such as train-ai, ai-use and search (version 08, Sept. 14, 2026) states that its contents “DO NOT REFLECT CONSENSUS of the Working Group.”
Why it matters
For publishers in every market AVJ covers, the controls answer different questions. A robots.txt block stops future fetching only by crawlers that follow robots.txt, such as Googlebot. RFC 9309 says the rules are not access authorization. To leave Google’s AI Overviews and AI Mode without leaving Search, Google documents the Search Console control. It removes the site’s links and content from those features, along with their traffic and impressions. Snippet rules instead restrict how content is shown and used as direct input, in all search results. Gemini training and grounding are a separate choice.
Our view: None of these controls is new, but Google’s main guide to limiting what Search shows now names the Search generative AI control next to robots.txt and the snippet rules. That puts the difference between controlling crawling and controlling use in one official place, which makes it easier to cite in site policies and client documents.
What we don’t know yet
- Whether Google would honor a Content-Usage rule if the IETF drafts are approved. Neither the video nor Google’s documentation says.
- How many sites use the Search generative AI control. Google has not published figures.
- Whether
data-nosnippettext is excluded as a direct input to AI Overviews and AI Mode; the specification’sdata-nosnippetsection refers to snippets without naming those features.
What to do now
- Decide which outcome you want (no crawling, no AI Overviews or AI Mode, or no Gemini training and grounding), then match it to the control in the table.
- Check the Search generative AI setting under Settings in Search Console; child properties inherit it by default.
- Name AI crawlers by their exact user agent tokens in robots.txt. Do not rely on custom lines.
Background
Google’s robots.txt specification and its list of common crawlers, both linked in the video’s description, document which rules and tokens Google’s crawlers follow.
Update log: Oct. 28, 2026 — First published.
Oct. 28, 2026 — Clarification: named Google’s Gary Illyes as the speaker of the quoted remarks, as reported by Search Engine Journal; earlier wording attributed them to Google’s Search Relations team.
Sources
[1] Google Search Central. “Control what you share with Google Search.” Google for Developers. Last updated Oct. 21, 2026. Accessed Oct. 23, 2026. https://developers.google.com/search/docs/crawling-indexing/control-what-you-share
[2] Google Search Central. “Latest documentation updates.” Google for Developers. Accessed Oct. 23, 2026. https://developers.google.com/search/updates
[3] Google Search Central. (2026, Oct. 21). “What can you actually put in a robots.txt?” [Video, 9:45]. YouTube. Quotes from the channel’s English caption track, timestamps as shown. Accessed Oct. 23, 2026. https://www.youtube.com/watch?v=-jHONLQgE-s
[4] Koster, M., Illyes, G., Zeller, H., & Sassman, L. (2022). “Robots Exclusion Protocol.” RFC 9309. IETF. https://www.rfc-editor.org/rfc/rfc9309.html
[5] Google Search Central. “Robots meta tag, data-nosnippet, and X-Robots-Tag specifications.” Google for Developers. Last updated March 24, 2026. Accessed Oct. 23, 2026. https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
[6] Google. “Search generative AI control.” Search Console Help. Accessed Oct. 23, 2026. https://support.google.com/webmasters/answer/16908024
[7] Google. “List of Google’s common crawlers.” Google Crawling Infrastructure. Last updated Oct. 15, 2026. Accessed Oct. 23, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
[8] Illyes, G., & Thomson, M. (2026, Aug. 19). “Associating AI Usage Preferences with Content in HTTP.” Internet-Draft draft-ietf-aipref-attach-05. IETF. Work in progress. Accessed Oct. 23, 2026. https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/
[9] Keller, P., & Thomson, M. (Ed.). (2026, Sept. 14). “A Vocabulary For Expressing AI Usage Preferences.” Internet-Draft draft-ietf-aipref-vocab-08. IETF. Work in progress. Accessed Oct. 23, 2026. https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/
[10] Schwartz, B. (2026, Oct. 21). “Google has updated its document on how to block Google Search.” Search Engine Land (trade press). Accessed Oct. 23, 2026. https://searchengineland.com/google-has-updated-its-document-on-how-to-block-google-search-497584
[11] Southern, M. G. (2026, Oct. 22). “Google Explains The Limit Of Robots.txt For AI Content Use.” Search Engine Journal (trade press). Accessed Oct. 23, 2026. https://www.searchenginejournal.com/google-illyes-robots-txt-crawling-ai-training/593957/
How this was made
Prepared with AI assistance from the sources cited above. Editorial review by Nhi Phan (editor) and fact-checking by Jackies continue after publication; any change will be recorded in the update log and in our Corrections Policy and Log at https://aivisibilityjournal.com/corrections/.