Skip to content

NEWS ITEM

News

Google publishes a guide to allowing Google-Agent through CDN and firewall bot defenses

Google's Nov. 17 guide is for sites that want to enable Google-Agent. It covers verification, rule order and challenges. It gives no traffic figures and does not name the Google products involved.

Key takeaways

  • On Nov. 17, 2026, Google published a guide on allowing Google-Agent, the user agent its agents use to browse sites and take actions for users, through content delivery networks (CDNs), web application firewalls (WAFs) and bot-detection services [1][2].
  • Google says to verify requests by the Google-Agent User-Agent header and its user-triggered-agents.json IP list, and, where present, by Web Bot Auth signatures. It then says “For verified requests, skip bot challenges” [1].
  • Its troubleshooting list says allow rules should take priority over broad cloud or data-center ASN blocks, managed bot rules and “category-level AI agent block policies” [1].

Direct answer: Google’s new guide tells site owners how to let verified Google-Agent requests through CDN, firewall and bot-detection rules. That includes skipping CAPTCHAs and JavaScript challenges and evaluating allow rules before category-level AI agent blocks [1]. The guide does not say which Google products send Google-Agent traffic or how much of it there is [1].

What happened: Google’s crawling documentation changelog lists a Nov. 17, 2026 entry, “Added guidance on allowing Google-Agent,” describing a technical guide “to help publishers that want to enable Google-Agent for their website” [2]. The guide says explicitly allowing Google-Agent traffic “helps prevent legitimate requests from being blocked while preserving your existing security controls” [1]. Google lists Google-Agent among its user-triggered fetchers, and AVJ’s AI crawler directory includes it. This article describes the guide as of Nov. 19, 2026.

What is Google-Agent?

Google describes Google-Agent as the name “used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request” [1][3]. It sits on Google’s list of user-triggered fetchers. Google says that because a user requests these fetches, “these fetchers generally ignore robots.txt rules” [3]. Its requests come from IP ranges listed in user-triggered-agents.json [1][3]. Google also says the user agent string can be spoofed [3].

What does Google tell sites to do?

The guide says some bot-detection services, CDNs and firewalls can verify and allow Google-Agent on their own, and it refers site owners to their provider for exact steps [1]. If a service does not verify Google-Agent automatically, Google suggests three checks [1]:

  • Confirm the User-Agent header contains “Google-Agent.”
  • Confirm the IP address is in user-triggered-agents.json.
  • If the request is signed, verify the Signature and Signature-Input headers for Signature-Agent: g="https://agent.bot.goog" according to Web Bot Auth.

Google says it is experimenting with the Web Bot Auth protocol, using the https://agent.bot.goog identity. In the experiment, it signs Google-Agent requests with HTTP Message Signatures (RFC 9421) and publishes the public keys in a well-known directory on agent.bot.goog [1]. The guide says to skip bot challenges for requests that pass verification [1].

What does the troubleshooting list cover?

If verified requests are still blocked, Google lists six checks [1]:

  • Rule priority: the allow rule should take priority over, and be evaluated before, broad cloud or data-center ASN blocks, managed bot rules or category-level AI agent block policies that could override a verified bot setting.
  • Interactive challenges: the rule should skip interactive CAPTCHAs and JavaScript challenges as well as HTTP blocks, including in-page CAPTCHA widgets triggered on search or form submission.
  • In-journey actions and APIs: allowing should cover more than homepage loads, including on-site search, filtering, detail pages, add-to-cart, booking, and underlying XHR, GraphQL or api.* endpoints.
  • Shared-IP rate limits: per-IP limits should account for or exempt verified Google-Agent ranges, because concurrent requests from multiple users share the same egress pool.
  • IP range updates: IP-based rules should stay synced with the latest ranges in user-triggered-agents.json.
  • Proxy headers: load balancers and proxies should preserve the original client IP, the User-Agent header and, for signed requests, the signature headers.

Why this matters for AI visibility

Our view: The guide points to access controls beyond robots.txt, namely CDN, firewall and bot-detection rules, that sites choosing to allow Google-Agent need to review. Google says user-triggered fetchers generally ignore robots.txt [3]. Google’s rule-priority check warns that category-level AI agent block policies at the CDN or firewall level could override a verified bot setting. Such a policy could therefore block Google-Agent even when a site intends to allow it [1]. Allowing Google-Agent is a choice, not a requirement: Google frames the guide as help for publishers “that want to enable” it [2]. The guide gives no evidence on whether allowing or blocking Google-Agent affects how a site appears in Google Search, AI Overviews or AI Mode, and readers should not assume it does.

What we don’t know yet

  • Which Google products or agents send Google-Agent requests. Google says only “agents hosted on Google infrastructure” [1][3].
  • How much Google-Agent traffic sites receive, or how often it is blocked. The guide gives no figures [1].
  • When the Web Bot Auth experiment might end or become standard for Google-Agent. The guide gives no date [1].
  • Whether blocking Google-Agent has any effect on Search, AI Overviews or AI Mode visibility. The guide does not address this [1].

What to do now

  1. Decide whether you want Google’s user-requested agents to browse and act on your site. Our decision framework for AI crawlers and AI search controls covers the trade-offs.
  2. If you do, check the order of your CDN and WAF rules: a verified Google-Agent allow rule should run before ASN blocks, managed bot rules and category-level AI agent blocks [1].
  3. Test more than the homepage. Google names on-site search, filters, detail pages, add-to-cart, booking and API endpoints as places requests can still be blocked [1].
  4. Base allow rules on verification, not the user agent string alone. Google says the string can be spoofed, and lists the IP file and optional Web Bot Auth signatures for verification [1][3].
  5. If you block Google-Agent, record the decision and its date so you can revisit it if Google’s guidance changes.

Update log: Nov. 19, 2026 — First published.

Sources

[1] Google. “Allow Google-Agent traffic.” Google Crawling Infrastructure documentation. Last updated Nov. 17, 2026 (UTC). Accessed Nov. 19, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/allow-google-agent

[2] Google. “Changelog.” Google Crawling Infrastructure documentation, entry dated Nov. 17, 2026. Accessed Nov. 19, 2026. https://developers.google.com/crawling/docs/changelog

[3] Google. “List of Google user-triggered fetchers.” Google Crawling Infrastructure documentation. Accessed Nov. 19, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers

How this was made

Prepared with AI assistance from the Google documentation cited above, then edited by Nhi Phan. Any change will be recorded in the update log and in our Corrections Policy and Log.

More from AVJ

Get the next one in your inbox.

The Visibility Brief: weekly, evidence-led, free. Every Friday.