robots.txt
-
Bing adds a separate AI grounding control for publishers, with a March 2027 deadline
Microsoft will let publishers set AI training and AI grounding preferences separately. Sites relying on NoArchive or NoCache must act by March 1, 2027, to keep their restrictions.
-
Google, OpenAI and Microsoft back a shared robots.txt signal for AI training
OpenAI and Microsoft say they will honor a shared robots.txt signal for AI training preferences, and Google plans to. It covers training only and rests on an IETF draft with no formal standing.
-
AI crawler directory: who crawls your site for AI, and how to control it
Who runs each AI crawler, what it is for and whether robots.txt controls it, taken only from the operators’ own documentation. Unverified bots are listed separately.
-
What Google actually says about AI Overviews and AI Mode
Google’s own documentation, quoted and dated: who is eligible, how query fan-out works, what Search Console now reports and which controls exist. Where Google is silent, we say so.
-
Google says robots.txt governs crawling, not use, and lists its AI Search controls
A Google video and a documentation update on Oct. 21 separate controlling crawling from controlling use in AI Overviews and AI Mode. A standardized robots.txt rule for AI-use preferences is still only an IETF draft.