Key takeaways
- An Oct. 15, 2026, Cloudflare blog post co-written with two OpenAI staff says a research pilot “has enabled OpenAI to discover and index hundreds of millions of new web pages.”
- The pilot used “a participating cohort of ~1% of Cloudflare’s total network,” and OpenAI “independently evaluates and ranks” the pages it finds, so being indexed does not mean being cited.
- The post gives no figures for citations, referral visits or crawl load, and says new pages “can initially increase crawling.”
Direct answer: Cloudflare’s network signals helped OpenAI’s search crawler find pages it did not know about, and OpenAI indexed hundreds of millions of them. That gives ChatGPT search more pages to draw on, but neither the Oct. 15 post nor the July announcement says how many were cited or sent visits.
What happened: Cloudflare announced the research pilot with OpenAI on July 8, 2026, saying it uses “real-time network insights—such as content freshness, traffic quality, and actual page changes” to improve AI search indexing and crawling. On Oct. 15, 2026, Cloudflare’s blog published early results in a post by Matthew Conroy and Jack Galilee, written with Henry Aspegren and James Balfour of OpenAI.
What the companies reported
- Scale. The pilot “used a participating cohort of ~1% of Cloudflare’s total network” and led OpenAI to “discover and index hundreds of millions of new web pages.” No exact count is given.
- Ranking stays with OpenAI. Cloudflare’s signals help the crawler discover pages, and “OpenAI then independently evaluates and ranks how those pages appear.”
- Filtering. The post says the work identifies requests for files that do not exist, sites that redirect “almost any address to their homepage” and duplicate copies, so crawlers can skip them. It also says Cloudflare is testing signals that flag single harmful pages, such as a planted fake login form.
- Crawl load. “Finding new pages can initially increase crawling as those pages are evaluated and indexed,” the post says; change signals are meant to reduce repeat fetches of unchanged pages over time.
- Permission and training. “A signal about a page doesn’t grant permission to access it,” and the research “uses metadata about the web for search and discovery, rather than sharing page content for model training.”
Why it matters
Our view: for AI visibility, discovery comes first, because a search system can rank only pages it has found. The post gives examples such as an appliance manual or a restaurant’s current menu, pages that it says “can be valuable even if few other sites link to them.” If discovery through network signals scales, pages with few inbound links could reach the candidate pool more often; that is our inference, not a reported result, and the post reports no citation outcomes.
Crawler access is still a condition. OpenAI’s crawler documentation says sites that block OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links,” and that OAI-SearchBot and GPTBot settings are independent, so a site can allow search while disallowing training.
“Indexed” and “cited” are different measures, and this pilot reports only the first. Citation and referral data would show whether it helps site owners, and neither company has published them.
What we don’t know yet
- Which sites were in the ~1% cohort and how they were chosen. The post does not say.
- How often the newly indexed pages were cited in ChatGPT answers or sent visits. No figures are given.
- How much extra crawling participating sites saw. The post says new pages “can initially increase crawling” but gives no measured figure.
- Whether and when the program extends beyond the pilot cohort. The post describes “the next phase” (better page selection, faster change signals, and measuring “what happens after content is discovered”) but gives no timetable.
What to do now
- Check your robots.txt for OAI-SearchBot. If it is blocked, your pages will not appear in ChatGPT search answers, though they can still appear as navigational links.
- In your server logs, track OAI-SearchBot request volume. OpenAI says robots.txt changes can take about 24 hours to take effect for search.
Disclosure: AVJ uses Cloudflare as an ordinary customer and has no other commercial relationship with it. AVJ pays OpenAI standard prices for a ChatGPT Pro subscription and for API access, receives no credits or special terms, and uses OpenAI models among its editorial tools (see our AI use policy). The AVJ Citation Index will also measure answers from OpenAI’s API. AVJ has no other commercial relationship with OpenAI.
Update log: Oct. 28, 2026 — First published.
Oct. 28, 2026 — Added disclosure of AVJ’s paid use of OpenAI services.
Sources
[1] Conroy, M., Galilee, J., Aspegren, H., & Balfour, J. (2026, Oct. 15). “Cloudflare and OpenAI: Making AI discovery better for everyone.” The Cloudflare Blog. Accessed Oct. 28, 2026. https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/
[2] Cloudflare. (2026, July 8). “Cloudflare Announces Research Pilot with OpenAI.” Press release. Accessed Oct. 28, 2026. https://www.cloudflare.com/press/press-releases/2026/cloudflare-announces-research-pilot-with-openai/
[3] OpenAI. “Overview of OpenAI Crawlers.” OpenAI developer documentation. Accessed Oct. 28, 2026. https://developers.openai.com/api/docs/bots
How this was made
Prepared with AI assistance from the sources cited above. Editorial review by Nhi Phan (editor) and fact-checking by Jackies continue after publication; any change will be recorded in the update log and in our Corrections Policy and Log at https://aivisibilityjournal.com/corrections/