AI crawlers
-
Google, OpenAI and Microsoft back a shared robots.txt signal for AI training
OpenAI and Microsoft say they will honor a shared robots.txt signal for AI training preferences, and Google plans to. It covers training only and rests on an IETF draft with no formal standing.
-
AI crawler directory: who crawls your site for AI, and how to control it
Who runs each AI crawler, what it is for and whether robots.txt controls it, taken only from the operators’ own documentation. Unverified bots are listed separately.