crawling
-
Google publishes a guide to allowing Google-Agent through CDN and firewall bot defenses
Google’s Nov. 17 guide is for sites that want to enable Google-Agent. It covers verification, rule order and challenges. It gives no traffic figures and does not name the Google products involved.
-
Cloudflare says its pilot added hundreds of millions of new pages to OpenAI’s search index. Citations are another matter
Cloudflare and OpenAI say signals from about 1% of Cloudflare’s network helped OpenAI index hundreds of millions of new pages for ChatGPT search. Their Oct. 15 post gives no figures for citations, traffic or crawl load.