robots.txt for AI Crawlers: How to Control GPTBot, ClaudeBot & Bingbot in 2026

1. What Makes AI Crawlers Different from Traditional Search Bots

A traditional search crawler — Googlebot, Bingbot — follows a clear mandate: visit URLs, download HTML, pass content to an index, and make that content retrievable through keyword queries. The relationship between a crawler and your site is relatively transparent: what gets crawled gets ranked.

AI crawlers operate under a fundamentally different model. A training crawler is not building a search index — it is collecting raw material for a statistical model that will generate responses for millions of future queries. Your article about technical SEO doesn't appear in a search result when an AI training crawler visits it; it disappears into a training corpus and influences how a language model responds to vaguely related questions for years afterward — without attribution, without a link back to your site, and without any mechanism for correction if the model learns something incorrectly.

A retrieval crawler is different again: it fetches your content in real time to synthesise a specific answer to a specific query, usually providing a citation. This distinction — training vs retrieval — is the axis around which all intelligent AI crawler policy rotates. Blocking the wrong type can reduce your AI search visibility without affecting any training outcome. Blocking the right type restricts training data collection while preserving your presence in AI-generated answers.

2. Training Crawlers vs Retrieval Crawlers — The Critical Distinction

The practical consequence of this distinction: a publisher who wants to control AI training use of their content but still wants their articles cited in Perplexity, ChatGPT Search, or Claude can block all training crawlers while leaving all retrieval crawlers open. This is the strategy that most content publishers in the SEO and media space have settled on by mid-2026 — and it is the approach the configuration in Section 5 implements.

There are legitimate reasons to block retrieval crawlers too — some publishers view any use of their content without a direct click-through as lost traffic, and AI-generated answers with citations don't always drive the same referral volume as organic search results. If you're considering blocking retrieval crawlers, the tradeoff is visibility in AI search results vs content control. There is no universally correct answer: it depends on your traffic model, your monetisation strategy, and your view of where AI search traffic is heading. The GEO & AEO Guide covers the AI visibility side of this tradeoff in detail.

3. The Complete AI Crawler Reference Table (2026)

The table below covers every major AI crawler active in mid-2026 with confirmed documentation. User agent strings are case-sensitive in robots.txt — the User-agent: value must match the crawler's announced string exactly. Where a crawler has multiple known variants, the primary documented string is listed.

4. robots.txt Fundamentals for AI Crawler Control

The robots.txt file lives at the root of your domain (https://yourdomain.com/robots.txt) and uses a simple directive syntax. For AI crawler control, the key rules are: each User-agent block applies only to the specified agent, rules are read top to bottom, and the most specific matching rule wins. A critical point that causes frequent misconfiguration: a Disallow: / in a wildcard User-agent: * block does not affect explicitly named agents that have their own blocks — named blocks take precedence over the wildcard.

# Each User-agent block applies independently
# Named blocks OVERRIDE the wildcard (*) block for that agent
# Rules within a block are evaluated top-to-bottom, first match wins

User-agent: GPTBot       # Targets ONLY OpenAI's training bot
Disallow: /              # Block entire site

User-agent: ChatGPT-User # Targets ONLY OpenAI's browsing bot
Allow: /                 # Allow entire site (browsing = retrieval)

User-agent: *            # All OTHER crawlers not named above
Disallow: /admin/
Disallow: /*?sessionid=
Sitemap: https://yourdomain.com/sitemap.xml

Three common robots.txt mistakes in AI crawler configurations: confusing a wildcard block with a blanket block (it doesn't override named agents), thinking that blocking one OpenAI user agent blocks all OpenAI access (GPTBot and ChatGPT-User are completely independent), and misspelling or using wrong capitalisation in user agent strings. GPTBot and GPTbot are treated differently. Always cross-reference the exact string from the platform's documentation.

5. The Full AI Crawler Blocking Configuration

The configuration below is the starting point used across IndexCraft client implementations in 2026. It blocks all major AI training crawlers while allowing all major retrieval crawlers. Adapt the Disallow/Allow rules for each agent according to your site's specific strategy.

# ============================================================
# BLOCK: AI TRAINING CRAWLERS
# These collect content for model training — not for answering
# real-time queries. Blocking these does NOT reduce AI search
# visibility in Perplexity, ChatGPT Search, or Claude.
# ============================================================

User-agent: CCBot
Disallow: /
# Common Crawl — powers many open-weight LLM training datasets

User-agent: GPTBot
Disallow: /
# OpenAI training — SEPARATE from ChatGPT-User (browsing)

User-agent: Google-Extended
Disallow: /
# Google Gemini training — SEPARATE from Googlebot (search)

User-agent: Applebot-Extended
Disallow: /
# Apple Intelligence training — SEPARATE from Applebot (search)

User-agent: Bytespider
Disallow: /
# ByteDance / TikTok AI training

User-agent: anthropic-ai
Disallow: /
# Anthropic training crawler — SEPARATE from ClaudeBot (retrieval)

User-agent: Meta-ExternalAgent
Disallow: /
# Meta AI training and general retrieval agent

# ============================================================
# ALLOW: AI RETRIEVAL CRAWLERS
# These fetch content to answer real-time user queries.
# Allowing them keeps your content visible in AI search results.
# Explicit Allow is optional if no prior Disallow exists for them.
# ============================================================

User-agent: ChatGPT-User
Allow: /
# OpenAI's BROWSING agent — real-time retrieval for ChatGPT

User-agent: ClaudeBot
Allow: /
# Anthropic retrieval — real-time content for Claude

User-agent: PerplexityBot
Allow: /
# Perplexity AI search retrieval

User-agent: DuckAssistBot
Allow: /
# DuckDuckGo AI assistant retrieval

# ============================================================
# NOTE ON BINGBOT: Microsoft uses the SAME bingbot user agent
# for both Bing Search indexing and Microsoft Copilot retrieval.
# Blocking it affects both. Do NOT add bingbot here unless you
# are willing to lose Bing search rankings entirely.
# ============================================================

# Standard rules for all other bots
User-agent: *
Disallow: /admin/
Disallow: /staging/
Disallow: /*?sessionid=
Disallow: /*?utm_source=

Sitemap: https://indexcraft.in/sitemap.xml

6. GPTBot Deep Dive — OpenAI's Training Crawler

GPTBot is OpenAI's primary web crawler for training data collection. OpenAI documented GPTBot in August 2023 along with a formal opt-out mechanism — the first major AI lab to publish explicit robots.txt guidance for its training crawler. The user agent string is GPTBot/1.0. OpenAI's crawler documentation specifies that it uses the IP range documented at the canonical URL and that it respects both Disallow: / in robots.txt and a publisher opt-out portal at their website settings page.

GPTBot is one of the highest-frequency AI training crawlers in server logs — in the 12 client sites analysed in Q1–Q2 2026, it appeared in 100% of the sites' logs where it had not been explicitly blocked. After adding a Disallow: / for GPTBot, compliance was observed within 48–72 hours in every case — meaning the crawler stopped appearing in logs for those sites within that window. No change was observed in ChatGPT Search citation rates over the following 30 days on those same sites, confirming that the ChatGPT-User retrieval agent was unaffected by the GPTBot block.

7. ClaudeBot and Anthropic's Crawler Suite

Anthropic operates two distinct crawler types, each with a separate user agent string. Understanding this distinction is essential for anyone who wants to control Anthropic's access to their content precisely.

ClaudeBot is Anthropic's retrieval crawler. When a user asks Claude a question that requires current web information, ClaudeBot fetches the relevant pages. Blocking ClaudeBot means Claude cannot access your content in real time — your site effectively disappears from Claude's ability to cite current information. For most publishers who want AI visibility, ClaudeBot should be allowed.

anthropic-ai is Anthropic's training crawler. It collects content for the pre-training and fine-tuning of Claude models. This operates independently of the retrieval function — blocking anthropic-ai does not prevent ClaudeBot from doing real-time retrieval from your site. If training data opt-out is your goal, anthropic-ai is the correct target. Both user agents are documented in Anthropic's web crawler documentation.

8. Bingbot and Microsoft Copilot — The Inseparable Problem

Bingbot is unique among major AI-associated crawlers for a reason that has significant strategic implications: Microsoft does not maintain a separate user agent for Microsoft Copilot. The same bingbot/2.0 user agent that crawls your site for Bing search results is also the agent that populates Microsoft Copilot's knowledge base and real-time retrieval. There is no CopilotBot, no MSCopilot, no separate user agent string.

The practical recommendation for most sites: do not block Bingbot. The inability to separate Bing search from Copilot means that blocking it is a binary choice with high cost on the search ranking side. Instead, focus your blocking configuration on the training crawlers that carry no SEO risk (GPTBot, CCBot, Google-Extended, Applebot-Extended, anthropic-ai) — these are the crawlers collecting content for model training without providing any direct reciprocal benefit through search visibility.

The Bingbot situation contrasts sharply with Google's approach. Google has explicitly separated Google-Extended (AI training only) from Googlebot (search indexing and AI Overviews) — allowing publishers to block training use without any search ranking impact. This is covered in depth in the next section. Microsoft's single-agent approach has drawn criticism from publishers and SEO practitioners and may evolve — watch Microsoft's Bing crawler documentation for any announcement of a Copilot-specific user agent.

9. Google-Extended vs Googlebot — The Safe Block

Google has set the clearest precedent among major AI companies for separating AI training from search indexing. The Google-Extended user agent was introduced alongside the expansion of Google's generative AI products and is explicitly documented as being used for training Gemini models. It is entirely separate from Googlebot, which handles Google Search indexing, Featured Snippets, and Google AI Overviews.

The Google-Extended block has been one of the most widely adopted changes by publishers since 2024. Its safety — zero SEO risk, straightforward implementation — and the clarity of Google's documentation make it a near-universal recommendation for any publisher concerned about AI training use. It is the one item on this guide's checklist that has almost no downside for any site type.

10. Selective Allow/Disallow by Path

Full-site blocks are the simplest approach, but path-specific rules give you much finer control. robots.txt allows you to define rules at the directory or URL pattern level, letting you maintain different policies for different parts of your site.

User-agent: GPTBot
Disallow: /premium/
Disallow: /members/
Allow: /blog/    # Allow training from free content

User-agent: PerplexityBot
Allow: /technical/
Allow: /strategy/
Allow: /ai-search/
Disallow: /news/      # Block retrieval from time-sensitive news
Disallow: /breaking/

User-agent: *
Disallow: /admin/
Disallow: /staging/
Disallow: /wp-admin/
Disallow: /api/internal/

11. The noai and noimageai Meta Tags

As an alternative to robots.txt (which operates at the file/path level), the noai and noimageai meta tags provide page-level signals to AI systems. They are added to the HTML <head> of individual pages:

<!-- Request that AI systems not use this page's content for training -->
<meta name="robots" content="noai, noimageai">

<!-- Or more selectively -->
<meta name="robots" content="noai">       <!-- Text content -->
<meta name="robots" content="noimageai">   <!-- Images only -->

The practical conclusion from the table: noai and noimageai are most useful for creative platforms where image and art protection is the primary concern — they have meaningful support in the generative art AI ecosystem. For text-based publishers primarily concerned about GPT, Claude, and Gemini training use, robots.txt Disallow rules remain the more universally implemented mechanism. Treat the meta tags as a supplementary layer, not a substitute.

12. AI Crawler Compliance — Who Actually Honours robots.txt?

Compliance with robots.txt is voluntary. There is no technical enforcement mechanism — a crawler can simply ignore the file. What motivates compliance is a combination of corporate policy, reputational risk, and in some jurisdictions emerging legal pressure. In direct testing across 12 client sites in Q1–Q2 2026, here is what the compliance picture looks like in practice.

13. Testing and Validating Your AI Crawler Configuration

14. Monitoring AI Crawlers in Server Logs

Your server logs are the single source of truth on which AI crawlers are actually visiting your site, at what frequency, and what they are fetching. The Crawl Budget Optimisation Guide covers the full log analysis workflow — here is what to extract specifically for AI crawlers.

Query 1: Identify all AI crawler traffic by user agent
→ Filter logs by known AI user agent strings:
  GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot,
  CCBot, Google-Extended, Applebot-Extended, Bytespider,
  Meta-ExternalAgent, DuckAssistBot, YouBot
→ Group by User-Agent, count requests, list distinct URLs crawled
→ Flag: any AI bot consuming >5% of total bot request volume

Query 2: Verify block compliance after robots.txt update
→ Filter logs to post-deployment date range
→ Filter by blocked AI user agents
→ Expect: zero requests from blocked agents after compliance lag
→ Flag: any blocked agent still appearing after 7+ days post-deployment

Query 3: Identify unknown or undocumented AI crawlers
→ Group all bot traffic by user agent string
→ Cross-reference against known list
→ Flag: any unfamiliar user agent with "bot", "spider", "crawler" in string
→ Research unknown agents before deciding to block or allow

Query 4: Track retrieval bot access to key content
→ Filter by ChatGPT-User, ClaudeBot, PerplexityBot
→ List the most-fetched URLs by each bot
→ Compare against your target pages for AI citation
→ Use this to inform your llms.txt featured pages list

For sites under 100,000 URLs, Screaming Frog Log File Analyser handles this analysis in a desktop GUI with bot filtering built in. For larger sites, piping logs into BigQuery with a custom AI bot user agent filter table gives the most flexible analysis environment. The log data also directly informs your llms.txt implementation — Query 4 shows which pages retrieval bots are already prioritising, which should inform which pages you feature in your content guidance file.

15. robots.txt AI Strategy by Site Type

16. The Legal Landscape — robots.txt and AI Training Rights

robots.txt carries no inherent legal weight. It is a technical convention and an industry norm, not a legally binding contract. A publisher can instruct GPTBot to Disallow: /, but if OpenAI ignored this instruction, there is currently no specific "robots.txt violation" tort in most jurisdictions.

What is legally more significant: your website's Terms of Service. A ToS that explicitly prohibits automated scraping for AI training creates a contractual basis for a claim if a company violates it after accessing your site. The enforceability of this varies by jurisdiction and depends on whether the crawler had notice of the ToS at the time of access.

In the EU, the AI Act's text and data mining (TDM) provisions interact with the EU Copyright Directive Article 4, which allows rights holders to opt out of TDM for commercial purposes through a machine-readable statement — a function that robots.txt can plausibly serve, though the exact mechanism is still being established in case law. Publishers in the EU have the strongest legal basis to use robots.txt + ToS + a written opt-out statement as a bundled rights protection.

17. LLM.txt — The Complementary Content Guidance Layer

robots.txt tells AI crawlers what they cannot access. llms.txt tells them what they should access and prioritise. The two are complementary instruments addressing different problems in the AI content access equation.

Once you have a robots.txt configuration that blocks unwanted training crawlers and allows the retrieval crawlers you want, the next optimisation is ensuring those retrieval crawlers find your best content first. Without guidance, a retrieval crawler navigates your site through the same link graph and sitemap signals as traditional search bots — which doesn't necessarily surface your most expert, most current, or most comprehensive pages first.

llms.txt solves this by giving you a curated Markdown index at your domain root that explicitly points retrieval crawlers to your priority content, organised by topic cluster. From direct server log analysis: after implementing llms.txt on IndexCraft following the robots.txt AI crawler update, the pages appearing most frequently in ClaudeBot and PerplexityBot request logs shifted within six weeks toward the pages explicitly featured in the file — a directional confirmation that the guidance is being read and acted on. The full implementation guide is in the LLM.txt Guide 2026.

18. Conclusion

The robots.txt file has not fundamentally changed since its earliest days — User-agent, Disallow, Allow, Sitemap. What has changed is the cast of characters it must address. In 2020, a publisher's robots.txt needed rules for Googlebot, Bingbot, and perhaps a handful of SEO audit crawlers. In 2026, that same file needs to distinguish between GPTBot and ChatGPT-User, between ClaudeBot and anthropic-ai, between Google-Extended and Googlebot — and make independent policy decisions for each.

The core principle that makes this manageable: the training/retrieval distinction. Block training crawlers universally (they provide no reciprocal benefit and contribute content to models without attribution). Allow retrieval crawlers selectively (they are your pathway to AI search citations and AI-driven referral traffic). Treat Bingbot as a special case requiring a deliberate decision, not a default. Validate compliance in server logs.

None of this requires development resources — robots.txt changes are text edits. The implementation cost is measured in minutes. The ongoing maintenance is a quarterly review of the AI crawler landscape and an audit of your log files to confirm that your rules are still effective as user agents evolve.

AI Crawler robots.txt Implementation Checklist

Frequently Asked Questions

Test your technical SEO knowledge — including AI crawler configuration and crawl budget management — with IndexCraft's technical SEO practice quiz. Or check your full AI search readiness against the AEO, SEO, and GEO checklist.