How AI answer engines choose their sources: what’s documented, what’s observed, what’s unknown

Fast-changing topic. Operator documentation and study versions were checked on Nov. 11, 2026. Microsoft says it will start honoring new AI-training and AI-grounding preferences on Feb. 1, 2027.

Direct answer: When an AI answer engine searches the web for a question, the model cites some of the pages that search returns. The companies document how pages become eligible and, in some cases, how queries are formed. None publishes exact weights for choosing citations. Studies so far report bounded associations, not universal ranking rules.

Does an AI answer come from training data or from a live search?

Both are possible, and they leave different evidence. Knowledge learned in training can produce a brand mention with no linked source. A citation points to a source the answer displays. The difference between AI citations and brand mentions matters when you measure visibility.

The research term retrieval-augmented generation describes combining what a model learned with documents retrieved at answer time. Google says its generative AI features in Search rely on its “core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index” [2]. That is Google’s description of its own system, not a universal pipeline.

Engines do not necessarily search for every question. OpenAI says ChatGPT “may search the web automatically when your question would benefit from current information” [4]. Anthropic’s API documentation says Claude answers without searching when a request draws on stable knowledge [5]. Microsoft says that, for consumer Copilot, “a determination is made if the request requires grounding data from the web” [11]. In OpenAI’s Responses API and Perplexity’s Agent API, the model decides whether to call the search tool; OpenAI’s Chat Completions search models always search [3][16]. None of these companies publishes how often answers come from training versus live retrieval.

A brand can be named without being cited. OpenAI’s API documentation states that the list of consulted sources “is often greater than the number of citations” [3]. Measure mentions, consulted sources and citations separately.

What steps sit between your page and a citation?

Public documents support an analytical sequence. It is not one identical pipeline across products, and stages can repeat, overlap or be skipped:

1. Access: robots.txt rules and network settings allow the operator’s crawler or fetcher to retrieve the page.
2. Index or cache: the operator stores or discovers the page where its search can find it. Being indexed is not a ranking result.
3. Retrieval: the system runs one or more searches for the question, sometimes after rewriting it into related queries.
4. Selection and filtering: some results enter the model’s context or are opened for further reading. Anthropic documents an option in which Claude writes code to filter results before they reach its context [5].
5. Passage use: the answer may draw on a snippet, an extracted passage or a fetched page rather than the whole page. Anthropic’s API returns up to 150 characters of cited_text with each citation [5]. Perplexity says its Search API returns query-relevant text extracted from the page, “not the full page” [14].
6. Citation display: a link appears to the user. Its position and grouping can change independently of which pages were retrieved.

This is AVJ’s own decomposition of public documentation. It does not describe any company’s internal ranking system.

What does Google document for AI Overviews and AI Mode?

Google’s clearest statement concerns eligibility, not selection. To be eligible as a supporting link in AI Overviews or AI Mode, “a page must be indexed and eligible to be shown in Google Search with a snippet” [1]. Google adds that a site must also be included in Search generative AI features through Search Console, which is the default setting [2]. Meeting these requirements does not mean Google will crawl, index or serve a page [1].

Both features may use query fan-out, “issuing multiple related searches across subtopics and data sources.” Google also says AI Mode and AI Overviews “may use different models and techniques,” so the responses and links they show vary [1].

Several controls act at different layers. nosnippet prevents content from being used as a direct input for AI Overviews and AI Mode, and max-snippet limits how much can be used [2]. The Search generative AI control in Search Console decides whether a site’s links and content can appear in, and help ground, Search generative AI features; Google says the control is not “a ranking or inclusion signal” for other parts of Search [2]. Google-Extended is different again: it governs whether crawled content may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI, and Google says it does not affect inclusion in Google Search [2]. These are separate choices, not one “block Google AI” switch.

Google says structured data is not required for generative AI search, that llms.txt files and other special markup are not needed, and that there is no requirement to break content into small chunks. It warns that creating pages for every variation of a query, “primarily to manipulate rankings or generative AI responses,” violates its scaled content abuse policy [2].

Google’s Gemini API, which some studies below use, offers grounding with Google Search as a developer product. Results from that API are not evidence about AI Overviews or AI Mode. Google’s Oct. 15, 2026 announcement that inline links would appear higher in AI Mode and AI Overviews responses more often is a display change, not a disclosure of citation-selection weights [24]. For more detail, see AVJ’s guide to Google’s AI Overviews and AI Mode guidance.

What does OpenAI document for ChatGPT search and the API?

Treat the consumer product and the developer API as separate surfaces. They should not be assumed to behave identically.

For ChatGPT search, OpenAI says OAI-SearchBot is used to surface websites in ChatGPT’s search features. Sites that opt out “will not be shown in ChatGPT search answers, though can still appear as navigational links.” GPTBot, which crawls content that may be used to train OpenAI’s foundation models, is controlled separately. OpenAI also recommends allowing requests from its published IP ranges [6].

OpenAI says ChatGPT “ranks search results using multiple factors intended to help users find relevant, reliable information” and that “placement is not guaranteed.” It does not list those factors or their weights [4]. When ChatGPT works with other search providers, OpenAI says it typically rewrites the question into one or more targeted queries; relevant saved memories and an approximate location derived from the user’s IP address can shape those queries [4]. OpenAI links the privacy statements of Microsoft and Shopify but does not say which provider handles which queries. “ChatGPT uses Bing” is therefore too broad to state as a blanket fact.

OpenAI does operate its own search index. In an Oct. 15, 2026 post co-written by Cloudflare and OpenAI staff, the companies say a pilot helped OpenAI “discover and index hundreds of millions of new web pages,” which OpenAI “independently evaluates and ranks” [7]. Discovery and indexing are not citation guarantees.

In the Responses API, the web_search tool can search, open pages and search within pages. Its sources field returns “the complete list of URLs the model consulted,” while inline citations show “only the most relevant references.” Setting external_web_access to false runs the tool in a cache-only mode, and the search_context_size setting does not “guarantee a specific number of sources or citations” [3]. The same guide says the Chat Completions API gives access to “the fine-tuned models and tool used by Search in ChatGPT” [3]. That does not establish that API answers match the consumer product, and the documents reviewed here do not say whether API search honors an OAI-SearchBot opt-out.

What does Anthropic document for Claude?

Anthropic separates training from search and from user-directed retrieval. ClaudeBot collects content that could contribute to training; Claude-SearchBot supports search indexing; and disallowing Claude-User prevents retrieval of a site’s content in response to a user’s query [8]. Anthropic says that as of Oct. 30, 2026, it has begun recognizing a machine-readable “no AI training” preference signal and is rolling it out across its services over time. Where robots.txt names ClaudeBot, that rule takes priority over the general signal [8]. Both are training choices, not search or citation controls. AVJ tracks these tokens in its AI crawler directory.

In the API, Anthropic documents a loop: Claude decides when to search, the API runs the searches and returns results, and “this process can repeat multiple times” within one request. Citations are always enabled for web search results and include the URL, title and up to 150 characters of cited text. Newer tool versions let Claude filter results with code before they enter its context [5].

Anthropic’s list of subprocessors names Brave Search, Parallel Web Systems and TurboPuffer under web search [25]. A subprocessor list does not say which service performs which function for which query, so it does not establish the blanket claim “Claude uses Brave.” Anthropic documents access controls, tool mechanics and citation structure; it does not publish citation-selection weights.

What does Microsoft document for Copilot and Bing?

Bing says “Bing and Copilot search experiences rely on the same core crawling, indexing, and ranking foundation as traditional search” [10]. The transparency note for consumer Copilot says that when Copilot is grounded in web search results, it “centers its response on high-ranking content” [11]. These describe a direction, not a citation algorithm.

Microsoft’s guidelines contain the most explicit selection guidance in this comparison. Bing says pages are “more likely to be selected for grounding queries and citations when content stands on its own,” that pages focused on one primary topic are more likely to be selected and that “early clarity improves grounding visibility” [10]. That is operator guidance without published weights, not independent proof of causal ranking factors.

Controls are changing. Today, Bing says NOARCHIVE prevents content from being used in Copilot responses and grounding results, while NOCACHE limits Copilot to the URL, title and snippet [10]. Microsoft says it will honor new, separate AI-training and AI-grounding preferences as it crawls from Feb. 1, 2027, and that after March 1, 2027, NoArchive and NoCache will no longer be read as training or grounding preferences. It also says that indexing and ranking pages in traditional search results is not grounding [9].

Keep consumer and enterprise products apart. Microsoft’s documentation for Microsoft 365 Copilot and Copilot Chat says the search query sent to Bing is generated from the prompt and “consists of a few words.” That statement covers the enterprise products; it is not evidence about every consumer Copilot interaction [12].

Bing Webmaster Tools’ AI Performance report shows which pages were cited and grouped “grounding queries,” which are not the same as full user prompts. Microsoft says the report does not show “why a specific page was referenced” and does not measure rankings or authority [23]. It records observed outputs, not their causes.

What does Perplexity document?

Perplexity says PerplexityBot surfaces and links websites in its search results and is not used to crawl content for AI foundation models. User-requested fetches by Perplexity-User “generally ignore” robots.txt, it says, because a user asked for them [13].

Its Search API “returns ranked web results as JSON, not a generated answer.” Each result carries extracted text, and “there is no parameter to re-fetch a page live or to control cache age” [14]. A Perplexity integration page says its web_search tool uses “the same search” that powers Perplexity’s own answer engine [15]. That does not establish that consumer and API answers cite the same pages. In the Agent API, the model decides when to search, result IDs can be referenced in citations, and whether inline markers appear depends on the prompt [16].

We could not retrieve Perplexity’s consumer help article on how answers are produced, or its article on its search architecture, for this review. We therefore do not repeat claims about its index size or reranking stages. The official documents reviewed establish access controls and API mechanics; how consumer answers select citations remains undisclosed.

What have studies observed, and what can’t they show?

The four 2026 studies below were version 1 preprints, not peer-reviewed, when checked on Nov. 11, 2026. Each covers particular models, product surfaces, questions and time windows. None reveals internal ranking weights.

Study and surface	Sample and period	Observation	Limits
Chen and Koudas [17]; public APIs of ChatGPT (GPT-5.4 nano with web search), Gemini (Gemini 3.1 Flash Lite with Google Search grounding) and Perplexity (Sonar)	Two 90-day collections of commercial questions, 158,099 answers and 468,037 citations; main collection for a US internet provider, June 24 to Sept. 21, 2026; plus API replays	In the replay, all 22,315 URLs ChatGPT cited were among pages its own search retrieved. ChatGPT consulted 26 pages per question on average and cited 3; across 4,946 source lists it cited 43.8% (95% CI 42.6% to 45.1%) of the pages it ranked first.	API results can differ from consumer apps. The authors describe associations, not causes. The paper thanks the AI-visibility firm ktau.ai for assistance with the data.
Same study [17]	Variation in citation counts between pages of the same website, ChatGPT	Retrieval by ChatGPT’s own search explained 32% of that variation; page properties explained 2.8%.	A decomposition within one dataset, not a universal ranking-factor share.
Jiang et al. [18]; ChatGPT Web, four model settings	80 English and Chinese queries; two settings ran each query 20 times, two others covered 30 and 32 queries with 4 to 28 runs; March 29 to July 29, 2026	Candidates retrieved more often across runs were more likely to be cited.	Associations; the authors note that changing an individual feature “does not necessarily increase the probability of being cited.” Two sessions silently switched to fallback models.
Walters, Chen and Toni [19]; GPT-5.6 Luna with OpenAI’s native web search, plus six open-weight configurations with fixed search results	225 facts that changed after the model’s training cutoff; 1,747 search-enabled episodes for Luna	Luna answered 53.6% of episodes from result snippets without opening a page. Across configurations, snippet-only episodes had higher rates of stale answers and unfaithful citations.	Tests reading depth and citation support on time-sensitive facts, not how sources are selected.
Chen and Koudas [20]; ChatGPT via the Responses API and Gemini API with Google Search grounding	300 questions, 19 declared locations and five network locations, 31,200 answers	ChatGPT followed the location declared by the application, down to the city; Gemini ignored it and followed the network’s country.	API behavior, not evidence about ChatGPT’s app or Google’s AI Overviews. Same authors as [17].

These are not four independent replications of one finding. They ask different questions, and two share authors. They show where to look, not which tactics are guaranteed to work.

An earlier peer-reviewed KDD 2024 paper on generative engine optimization tested a simulated engine that passed the top five Google results to GPT-3.5, plus a check on Perplexity.ai with the sources supplied as uploaded files. Its reported visibility gain of up to 40% measures how prominently a source appeared in the generated answer, among sources already supplied to the engine, not whether a page gets retrieved by a live engine [21]. A peer-reviewed 2023 audit of the generative search engines of that time found that 51.5% of generated sentences were fully supported by citations and 74.5% of citations supported their sentence [22]. That is historical evidence about citation support, not a current industry average.

For AVJ’s reporting on two of these preprints, see retrieval versus page tactics and location-dependent API citations.

What remains unknown?

No operator reviewed here publishes the query-dependent signals or weights it uses for retrieval, passage selection or citation display. Eligibility documentation does not fill that gap.

Other limits matter as much. OpenAI does not publish a map of which search partner serves which ChatGPT queries. Anthropic’s subprocessor list does not assign functions to providers. API answers need not match their consumer counterparts: one preprint found that Google Search and the OpenAI Responses API together recovered only 28.3% and 26.2% of the URLs cited by two ChatGPT Web settings, compared with 50.3% and 48.7% of cited hostnames. That comes from an 80-query sample, not a universal rate [18].

Google does not show the full set of fan-out queries or explain individual citations. Bing’s grounding queries are not a prompt log. A first-place web ranking, a crawler visit and an indexed page are each insufficient evidence of a citation. Differences across repeated runs may be ordinary variation rather than an algorithm change.

This guide is not a systematic literature review. It uses primary operator documentation and a defined set of peer-reviewed and preprint studies. Where we say a mechanism is not documented, we mean the sources reviewed do not establish it, not that no internal documentation exists.

What can publishers do, based on the evidence?

1. Decide each access type separately. Choose whether training crawlers, search crawlers and user-initiated fetchers should have access, using each operator’s current token names and IP guidance. Blocking one does not control the others. Check both robots.txt and CDN or firewall settings. The AI crawler directory lists the operator-specific tokens.
2. Keep the pages you want cited indexable and snippet-eligible. For Google’s AI Overviews and AI Mode this is a documented requirement, together with the default “include” setting in Search Console’s generative AI control. Before using nosnippet, max-snippet, NOARCHIVE or Microsoft’s new grounding preferences, check which layer each one limits and when it takes effect.
3. Write clear, useful pages, not supposed “AI ranking hacks.” Answer the reader’s question accurately and make the main subject clear. Google says special markup, chunking content and pages for every query variation are not required. Microsoft’s writing guidance is attributed advice for Bing and Copilot, not a universal rule.
4. Check support, not just link presence. A citation can appear while the linked page fails to support the sentence. Read the cited passage and confirm a reader could verify the claim. Citation counts and citation quality are different measures.
5. Measure observable outputs repeatedly. Use Search Console’s generative AI performance report and Bing’s AI Performance report where available. Sample fixed prompts across time, models and markets, keep app and API results apart, and record run counts and uncertainty. AVJ’s AI visibility metrics guide explains how to report these measures without overstating a single observation.
6. Reject guarantees the evidence cannot support. No vendor can infer undisclosed weights from ranking reports alone. Ask for the documented mechanism, the exact product surface, the sample, the time window, the comparison and the uncertainty, rather than a promise that an FAQ block, a schema type, an llms.txt file or a page-production program will secure citations.

Evidence status at a glance

“Documented” means the operator says it; “observed” means a bounded study measured an association; “unknown” means the sources reviewed do not establish it.

Question	Google AI Overviews and AI Mode	ChatGPT search (app)	OpenAI API	Claude (app and API)	Copilot and Bing	Perplexity
Search and access controls	Documented: indexing, snippet rules and a separate generative AI control	Documented: OAI-SearchBot	Unknown whether the OAI-SearchBot opt-out applies	Documented: separate training, search and user fetchers	Documented: Bingbot plus current and new grounding controls	Documented: PerplexityBot; user fetcher treated differently
Query construction	Documented: query fan-out	Documented: query rewriting, with memory and location effects	Documented: search actions	Documented: model-directed searches (API)	Documented at a high level; enterprise query details documented separately	Documented: model-directed API searches
Selection weights	Unknown	Unknown	Unknown	Unknown	Guidance documented; weights unknown	Unknown
Passage handling	Documented: snippet limits	Not established in sources reviewed	Documented: sources versus citations; snippet-only answers observed in one study	Documented: bounded cited text and filtering (API)	Documented: current cache controls	Documented: extracted text in Search API
What publishers can observe	Search Console reports	No causal explanation found	Response fields, not app equivalence	Citation fields, not selection causes	Cited pages and grouped grounding queries	API results, not consumer selection

Disclosure: This guide discusses products from Google, OpenAI, Anthropic, Microsoft, Perplexity and Cloudflare. AVJ pays standard published rates, on standard terms, for basic services from several of these companies, including a ChatGPT Pro subscription and API access from OpenAI, API access from Perplexity, and Cloudflare services for its website. AVJ uses OpenAI and Anthropic models among its editorial tools (see our AI use policy). AVJ uses these services as an ordinary customer and has received no free credits, promotions or other compensation from OpenAI, Anthropic, Google, Microsoft, Perplexity or Cloudflare. AVJ receives no sponsorship and has no partnership with any company discussed in this guide. The AVJ Citation Index, whose methodology is still under review, plans to measure answers from AI companies’ APIs, including OpenAI’s and Perplexity’s.

Sources

[1] Google Search Central. “AI features and your website.” Last updated Dec. 10, 2025. Accessed Nov. 11, 2026. https://developers.google.com/search/docs/appearance/ai-features

[2] Google Search Central. “Google’s guide to optimizing for generative AI features on Google Search,” last updated July 10, 2026; “Robots meta tag, data-nosnippet, and X-Robots-Tag specifications,” last updated March 24, 2026; Search Console Help, “Search generative AI control”; and “Google’s common crawlers” (Google-Extended), last updated Oct. 30, 2026. Accessed Nov. 11, 2026. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag https://support.google.com/webmasters/answer/16908024 https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

[3] OpenAI. “Web search” (API guide). Accessed Nov. 11, 2026. https://developers.openai.com/api/docs/guides/tools-web-search

[4] OpenAI Help Center. “Searching the web with ChatGPT.” Updated about Oct. 20, 2026 (page showed “Updated: 22 days ago” on Nov. 11). Accessed Nov. 11, 2026. https://help.openai.com/en/articles/9237897-chatgpt-search

[5] Anthropic. “Web search tool” (Claude API documentation). Accessed Nov. 11, 2026. https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool

[6] OpenAI. “Overview of OpenAI Crawlers.” Accessed Nov. 11, 2026. https://developers.openai.com/api/docs/bots

[7] Conroy, M., Galilee, J., Aspegren, H. (OpenAI), & Balfour, J. (OpenAI). “Cloudflare and OpenAI: making AI discovery better for everyone.” Cloudflare Blog, Oct. 15, 2026. Accessed Nov. 11, 2026. https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/

[8] Anthropic Help Center. “Does Anthropic crawl data from the web, and how can site owners block the crawler?” Accessed Nov. 11, 2026. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

[9] Microsoft Bing Blogs. “Your content, your choice: clearer preferences for training and grounding.” Oct. 30, 2026. Accessed Nov. 11, 2026. https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/

[10] Microsoft Bing. “Bing Webmaster Guidelines.” Accessed Nov. 11, 2026. https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a

[11] Microsoft Support. “Transparency Note for Microsoft Copilot (for individuals).” Updated Nov. 10, 2026. Accessed Nov. 11, 2026. https://support.microsoft.com/en-us/privacy/microsoft-copilot/transparency-note

[12] Microsoft Learn. “Data, privacy, and security for web search in Microsoft Copilot and Microsoft Copilot Chat.” Oct. 29, 2026. Enterprise products. Accessed Nov. 11, 2026. https://learn.microsoft.com/en-us/copilot/plan/manage-public-web-access

[13] Perplexity. “Perplexity Crawlers.” Accessed Nov. 11, 2026. https://docs.perplexity.ai/docs/resources/perplexity-crawlers

[14] Perplexity. “Search API overview.” Accessed Nov. 11, 2026. https://docs.perplexity.ai/docs/search/overview

[15] Perplexity. “Vercel AI SDK” integration guide. Accessed Nov. 11, 2026. https://docs.perplexity.ai/docs/getting-started/integrations/vercel-ai-sdk

[16] Perplexity. “Web search” (Agent API tool). Accessed Nov. 11, 2026. https://docs.perplexity.ai/docs/agent-api/tools/web-search

[17] Chen, M., & Koudas, N. (2026, Oct. 10). “What Drives AI Citations? Content, Retrieval and Model Knowledge in Generative Engine Optimization (GEO).” arXiv:2610.15033v1. Preprint, not peer-reviewed. https://arxiv.org/abs/2610.15033

[18] Jiang, Y., Zhang, S., Xing, A., Chang, X., & Xiao, X. (2026, Oct. 11). “Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations.” arXiv:2610.15382v1. Preprint, not peer-reviewed. https://arxiv.org/abs/2610.15382

[19] Walters, O., Chen, L., & Toni, F. (2026, Oct. 15). “Retrieved but Not Read: Understanding Why Search-Augmented LLMs Fail to Use Available Evidence.” arXiv:2610.21731v1. Preprint, not peer-reviewed. https://arxiv.org/abs/2610.21731

[20] Chen, M., & Koudas, N. (2026, Oct. 16). “Location, Location, Generation: Where You Ask Matters for Generative Engine Optimization.” arXiv:2610.24081v1. Preprint, not peer-reviewed. https://arxiv.org/abs/2610.24081

[21] Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., & Deshpande, A. (2024). “GEO: Generative Engine Optimization.” In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 5–16. https://doi.org/10.1145/3637528.3671900

[22] Liu, N. F., Zhang, T., & Liang, P. (2023). “Evaluating Verifiability in Generative Search Engines.” In Findings of the Association for Computational Linguistics: EMNLP 2023, 7001–7025. https://doi.org/10.18653/v1/2023.findings-emnlp.467

[23] Microsoft Bing Webmaster Tools Help. “AI Performance.” Accessed Nov. 11, 2026. https://www.bing.com/webmasters/help/ai-performance-9f8e7d6c

[24] Google. “AI Mode and AI Overviews updates to explore the web.” The Keyword, Oct. 15, 2026. Accessed Nov. 11, 2026. https://blog.google/products-and-platforms/products/search/updates-ai-mode-overviews-explore-web/

[25] Anthropic Trust Center. “Subprocessors.” Accessed Nov. 11, 2026. https://trust.anthropic.com/subprocessors

How this was made: researched and drafted with AI assistance from the operator documents and papers listed above. Every quoted passage and number was rechecked against the live documents and the papers’ full text on Nov. 9, 2026, and the live documents were checked again on Nov. 11, 2026. The guide was then edited by Nhi Phan and fact-checked by Jackies. A further editorial review follows publication. It is not peer-reviewed. See our AI use policy and Corrections Policy and Log.

Changelog

• Nov. 11, 2026. First published.