Blog about GEO and SEO

How to allow GPTBot and other AI crawlers in robots.txt – with examples

To be quoted in ChatGPT, Perplexity, Gemini or Copilot, your robots.txt must let those services' crawlers reach your pages. The safest setup is a permissive User-agent: * group plus explicit groups for the main AI bots – GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended – each with Allow: /. Remember that robots.txt only states your intent: a CDN rule, firewall or bot-protection toggle can still block the same crawler at the network level.

Which AI crawlers should I allow?

AI systems use several different bots, and they do different jobs. Some build an index, some fetch a single page while a user is chatting, and some collect data for model training. Blocking one does not block the others.

User-agent tokenOperatorWhat it does
GPTBotOpenAIBroad crawl used for model training and ChatGPT content
OAI-SearchBotOpenAIIndexing for ChatGPT search results and links
ChatGPT-UserOpenAILive fetch when a user or assistant opens your page
PerplexityBotPerplexityIndex that feeds Perplexity's cited answers
Perplexity-UserPerplexityUser-triggered fetch of a specific URL
ClaudeBotAnthropicCrawl for Claude
Claude-User, Claude-SearchBotAnthropicUser-initiated fetch and search indexing
GooglebotGoogleGoogle Search index – also the source for AI Overviews and AI Mode
Google-ExtendedGoogleControl token for Gemini apps and grounding (not a separate crawler)
BingbotMicrosoftBing index, which feeds Copilot
Applebot, Applebot-ExtendedAppleSiri/Spotlight crawl; -Extended controls AI training use
Amazonbot, Meta-ExternalAgent, MistralAI-User, DuckAssistBot, CCBot, BytespiderVariousAssistants, datasets and other AI products

Operators add and rename tokens regularly. Before a big change, check the official documentation pages of OpenAI, Anthropic, Perplexity, Google and Microsoft [check: current token lists at time of publication].

What does a robots.txt that allows all AI crawlers look like?

If you have no robots.txt at all, or an empty one, everything is allowed by default. Explicit groups are useful anyway: they document your policy, and they protect you from an overly broad Disallow in the * group.

# https://example.com/robots.txt

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /search

# --- OpenAI ---
User-agent: GPTBot
Allow: /
Disallow: /wp-admin/
Disallow: /cart/

User-agent: OAI-SearchBot
Allow: /
Disallow: /wp-admin/
Disallow: /cart/

User-agent: ChatGPT-User
Allow: /
Disallow: /wp-admin/
Disallow: /cart/

# --- Perplexity ---
User-agent: PerplexityBot
Allow: /
Disallow: /wp-admin/
Disallow: /cart/

User-agent: Perplexity-User
Allow: /

# --- Anthropic ---
User-agent: ClaudeBot
Allow: /
Disallow: /wp-admin/
Disallow: /cart/

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

# --- Google / Microsoft / Apple ---
User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

Note the repeated Disallow lines. That is not redundancy – it is the single most common mistake, explained below.

Why do I have to repeat Disallow rules in every group?

Under the robots.txt standard (RFC 9309), a crawler obeys one group only: the one whose User-agent line matches it most specifically. Once GPTBot finds a group named GPTBot, it ignores the * group completely.

So this is wrong if you wanted to keep the admin area private:

User-agent: *
Disallow: /wp-admin/

User-agent: GPTBot
Allow: /

GPTBot will happily crawl /wp-admin/, because its own group contains no restriction. Copy every rule you still want to enforce into each named group, or keep the named groups to a plain Allow: / only when the * group has nothing you care about.

How do I allow answer engines but block training crawls?

Many publishers want to be cited without being used as training data. That distinction is partly supported, partly blurry: operators describe their tokens differently, and some use one crawler for several purposes.

A common compromise: allow search/answer tokens, disallow the broad training tokens.

# Allow retrieval for answers and citations
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Opt out of training / dataset collection
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://example.com/sitemap.xml

Be aware of the trade-off. Blocking GPTBot may reduce how often ChatGPT has your content available outside live browsing, and Google-Extended: Disallow affects Gemini-side use while leaving classic Google Search untouched. If visibility in AI answers is your goal, the permissive version earlier in this article is the safer default.

How do I block only the bots I don't want?

Keep the default open and name the exceptions. Order inside the file does not matter – matching is by user-agent specificity, not by line position.

User-agent: *
Allow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

Badly behaved scrapers often ignore robots.txt entirely. For those you need rate limiting, WAF rules or a bot-management product – robots.txt is a request, not a lock.

Where does the file go and what breaks it?

  1. Place it at the root of each host and scheme: https://example.com/robots.txt. A file at /blog/robots.txt is ignored, and www.example.com plus example.com are separate hosts.
  2. Serve it as text/plain with HTTP 200. A 404 is interpreted as "everything allowed"; persistent 5xx errors can be treated as "everything disallowed" by Google, which is far worse.
  3. Keep it under 500 KB – Google ignores content past that limit, and other crawlers have their own caps.
  4. Use one User-agent token per line; no wildcards in the token itself except the single * group. User-agent: GPT* does not work.
  5. Paths are case-sensitive and support * and $: Disallow: /*.pdf$.
  6. Crawl-delay is unsupported by Google and inconsistently handled elsewhere. Don't rely on it.
  7. Never block CSS and JS bundles. Crawlers that parse the DOM need them to understand the page.

How do I verify that an AI crawler can actually reach my page?

robots.txt is only the first gate. In practice, more sites are invisible in AI answers because of CDN rules than because of robots.txt.

  1. Fetch your own robots.txt from outside your network: curl -I https://example.com/robots.txt and confirm 200 plus text/plain.
  2. Simulate a crawler: curl -A "GPTBot" -I https://example.com/your-page and check for 200, not 403 or a challenge page.
  3. Check your CDN. Cloudflare and similar providers ship AI-bot blocking toggles and managed rules that can be on by default – review the bot settings and the WAF event log [check: current default behaviour of your provider].
  4. Grep your access logs for GPTBot, PerplexityBot, ClaudeBot, OAI-SearchBot and look at the status codes they receive.
  5. Confirm the content is in the HTML. Most AI crawlers do not execute JavaScript reliably; curl https://example.com/page | grep "key phrase" tells you whether the text exists server-side.
  6. Use Google Search Console's robots.txt report and Bing Webmaster Tools to confirm the classic crawlers see what you expect.
  7. Verify suspicious traffic: OpenAI, Perplexity, Anthropic and Google publish IP ranges or reverse-DNS methods, because the user-agent string is trivially spoofed.

Do meta tags and llms.txt change anything?

robots.txt controls crawling; meta tags control what happens after the fetch. If you want to be indexed but limit snippet length, use page-level directives:

<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large">

nosnippet and short max-snippet values can suppress the very text an AI answer would quote, so use them deliberately. Google-Extended is documented as a robots.txt control token [check: whether an equivalent meta directive exists at publication time].

llms.txt is a proposed plain-text file at /llms.txt that lists your key pages and context for language models. No major assistant is confirmed to require it, and support is unclear, but it is cheap to publish and harmless:

# Example Inc.
> B2B invoicing software for small agencies.

## Docs
- [Getting started](https://example.com/docs/start): setup in 10 minutes
- [Pricing](https://example.com/pricing): plans and limits

## Policies
- [Terms](https://example.com/terms)

FAQ

Does allowing GPTBot mean my content will be used for training?

OpenAI describes GPTBot as its crawler for content that may be used to improve models. If you want citations without that, allow OAI-SearchBot and ChatGPT-User and disallow GPTBot – accepting that this limits your presence outside live browsing.

Will blocking Google-Extended hurt my Google rankings?

No. Google-Extended is a separate control token and does not affect Googlebot or classic Search ranking. It governs Gemini-side usage. AI Overviews are served from the Google Search index via Googlebot, so blocking Googlebot would remove you from both.

Why is my site missing from AI answers even though robots.txt allows everything?

The usual causes are a CDN or WAF blocking the crawler's IP or user-agent, content rendered only by JavaScript, thin or hard-to-parse pages, or simply no authoritative source signal on the topic. Check the logs for the bot's status codes first.

Do all AI crawlers actually obey robots.txt?

The major named bots from OpenAI, Anthropic, Perplexity, Google, Microsoft and Apple state that they do. Smaller scrapers and unidentified agents often do not, which is why blocking at the edge is the only enforceable option.

Should I use one combined group for several bots?

You can – multiple User-agent lines stacked above a shared set of rules form a single group. It is compact, but harder to adjust per-bot later. One group per token stays easier to maintain.

If you want to see which crawlers your site currently allows, whether answer engines can parse your pages and where you're already being cited, run a free check in Zgadka – it returns GEO and SEO scores plus ready-made robots.txt, llms.txt and schema.org files you can paste straight in.


Fazier badge