# robots.txt — AI Visibility Journal (AVJ)
# Policy: we WANT to be discovered, retrieved, cited and accurately understood by search and AI systems.
# Decision and rationale: website-spec.md §13.2. Review quarterly.
#
# Deploy as a STATIC file in the WordPress web root (next to wp-config.php's public folder, i.e. the
# folder that contains wp-login.php). A physical file overrides WordPress's virtual robots.txt.
# Cloudflare: "Bot Preference Sync" (formerly Managed robots.txt) must be OFF, otherwise Cloudflare
# prepends its own rules. Verify after every Cloudflare change:  curl -s https://www.aivisibilityjournal.com/robots.txt
#
# Note (RFC 9309): a crawler obeys only the most specific group that names it, so every named
# crawler is listed in the same single group as *.
#
# AI search / user-initiated retrieval: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
#   PerplexityBot, Perplexity-User
# Model-training / AI-use tokens (allowed by editorial decision): GPTBot, ClaudeBot,
#   Google-Extended (also governs Gemini Apps & Vertex AI grounding; not Google Search),
#   Applebot-Extended, CCBot
# To opt out of training later: move the training tokens into a separate group with "Disallow: /".

User-agent: *
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Allow: /
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom=
Disallow: /*?preview=true

Sitemap: https://aivisibilityjournal.com/wp-sitemap.xml
Sitemap: https://aivisibilityjournal.com/news-sitemap.xml