Blog about GEO and SEO

How do AI crawlers like GPTBot, ClaudeBot and PerplexityBot read your site?

AI crawlers like GPTBot, ClaudeBot and PerplexityBot work much like classic search bots: they request your URLs over HTTP, read the raw HTML that comes back, strip it down to text, links and structured data, and store that for training or for answering live questions. The key difference is that most of them do little or no JavaScript rendering, so whatever is not in your server-side HTML is effectively invisible to them. They also split into two behaviours – bulk crawling in the background, and real-time fetching triggered by a user's question – and you can control each separately in robots.txt.

What is an AI crawler, and how is it different from a search crawler?

A search crawler builds an index that ranks pages and sends clicks to you. An AI crawler feeds a model or an answer engine: the content either becomes training data or is pulled into a live answer with a citation link.

Because of that, AI crawlers care less about ranking signals and more about clean, extractable text. There is no "position 3" in a chat answer – either your passage is quoted and linked, or it is not there at all.

A second difference is timing. Some AI bots crawl continuously like Googlebot; others fetch a single URL at the moment a user asks a question, read it once, and never come back.

Which AI crawlers should you know about?

Bot (user-agent)OperatorWhat it does
GPTBotOpenAIBulk crawling, mainly for model training data
OAI-SearchBotOpenAIIndexing for ChatGPT search results and citations
ChatGPT-UserOpenAIOn-demand fetch when a user or a tool opens your URL
ClaudeBotAnthropicBulk crawling for Claude
Claude-User / Claude-SearchBotAnthropicUser-triggered fetches and search indexing
PerplexityBotPerplexityIndexing pages for Perplexity answers
Perplexity-UserPerplexityLive fetch when an answer needs the page
Google-ExtendedGoogleNot a crawler – a token that controls whether Googlebot-crawled content may be used by Gemini
BingbotMicrosoftFeeds Bing and, through it, Copilot
CCBotCommon CrawlOpen crawl dataset reused by many model builders
Applebot / Applebot-ExtendedAppleApple search and AI features
Meta-ExternalAgentMetaCrawling for Meta's AI products
BytespiderByteDanceCrawling for ByteDance AI products

Exact user-agent strings and policies change. Always check the operator's own documentation before writing rules, and verify the full user-agent line in your logs rather than trusting a short name.

How does an AI crawler actually read a page?

The pipeline is similar across providers, even though the details are not fully public:

  1. Discovery. The bot finds your URL from a sitemap, from links on other pages, from an existing crawl dataset, or directly from a link a user pasted into a chat.
  2. robots.txt check. Well-behaved bots request /robots.txt first and look for a group matching their user-agent token, then *.
  3. HTTP request. The bot requests the page, following redirects. Slow responses, 403s from your WAF or bot-protection layer, and aggressive rate limiting all cut the crawl short.
  4. HTML parsing. Nav, header, footer, scripts and styles are usually dropped. What remains is the main content: headings, paragraphs, lists, tables, link anchors, image alt text.
  5. Structured data extraction. JSON-LD, meta tags and HTTP headers are read as explicit facts – title, author, date, price, FAQ answers.
  6. Chunking. The text is cut into passages, because models answer from passages, not whole documents. A self-contained paragraph under a clear heading is a good chunk; a sentence that only makes sense after three paragraphs of build-up is not.
  7. Storage or immediate use. Either the chunks go into a training set or search index, or they are injected straight into the model's context to answer the question at hand and then discarded.

Do AI crawlers execute JavaScript?

In general, assume they do not. Most AI crawlers fetch the HTML document and parse it; they do not run a full browser engine the way Googlebot's rendering stage does.

That means a React or Vue app that ships an empty <div id="root"> and loads content client-side may deliver nothing readable. The same applies to tabs, accordions and "load more" blocks that inject text only after a click or an XHR call.

Behaviour differs between services and is not fully documented, and some operators may add rendering over time. The safe engineering choice is server-side rendering or static generation, so the complete main content exists in the first HTML response.

A quick test: run curl -s https://example.com/page | less and read what comes back. If your key paragraphs are not in that output, AI crawlers probably do not see them either.

How do I allow or block AI crawlers in robots.txt?

robots.txt is the main control surface. Each bot needs its own group – blocking GPTBot does not block ChatGPT-User, and blocking Googlebot is not how you opt out of Gemini training.

A permissive setup that welcomes AI answer engines but keeps private areas out:

User-agent: GPTBot
Allow: /
Disallow: /cart/
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

If you want visibility in AI answers but not in training datasets, block the bulk training crawlers and keep the search and user-triggered ones open:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Two caveats. robots.txt is a request, not an enforcement mechanism – the major providers state they honour it, but not every crawler on the internet does. And blocking a crawler does not remove content a model already learned; it only affects future crawls.

For hard blocking, use server-level rules or your CDN's bot controls, and verify requests against the operator's published IP ranges rather than the user-agent string, which anyone can spoof.

What is llms.txt, and does it help?

llms.txt is a proposed convention: a Markdown file at your root that points AI systems to your most useful, clean documents. It is not an official standard and support is not guaranteed by any major provider, but it costs little and is easy to maintain.

# Example Company

> B2B scheduling software for clinics. Founded 2016, based in Berlin.

## Core pages
- [Product overview](https://example.com/product): what the software does
- [Pricing](https://example.com/pricing): plans, limits, billing terms
- [Docs](https://example.com/docs): API reference and setup guides

## About
- [About us](https://example.com/about): team, history, contact
- [Security](https://example.com/security): data handling and compliance

Treat it as a supplement to good HTML, a sitemap and structured data – never a replacement.

What on-page structure makes content easy for an AI to read?

Give every important fact a stable textual home. Prices, dates, names, specifications and answers should exist as plain text in the HTML, not only inside an image, a chart or a PDF.

Use one <h1> and a logical heading hierarchy, with headings phrased the way people ask questions. Put the direct answer in the first two sentences under each heading, then the detail.

Add structured data so machines get unambiguous facts:

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [{
    "@type": "Question",
    "name": "Does the plan include API access?",
    "acceptedAnswer": {
      "@type": "Answer",
      "text": "Yes. API access is included in the Pro and Enterprise plans, with a limit of 10,000 requests per day."
    }
  }]
}

And make sure the basic metadata is present and honest:

<title>Pricing – Example Scheduling Software</title>
<meta name="description" content="Plans from $29/month. Per-clinic pricing, no setup fee, API included on Pro.">
<link rel="canonical" href="https://example.com/pricing">
<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large">

max-snippet:-1 matters: a restrictive snippet limit tells engines to show less of your text, which works against being quoted.

How do I check whether AI bots actually visit my site?

Server logs are the only reliable source. Analytics tools that rely on JavaScript will not see bots at all.

grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|CCBot|Bytespider' \
  /var/log/nginx/access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head -50

Look at three things: which bots come at all, which URLs they hit, and which status codes they get. A wall of 403 or 429 responses usually means your CDN or WAF is blocking crawlers you intended to allow.

Then verify that the requests are genuine by checking the client IP against the operator's published ranges – OpenAI, Anthropic and Perplexity each document this; follow their current pages rather than a copied list. Spoofed user-agents are common.

What are the most common mistakes?

  • Client-side-only content. The page looks fine in a browser and is empty in curl.
  • Blocking by accident. A blanket User-agent: * / Disallow: / left over from staging, or Cloudflare's AI bot toggles set to block while marketing expects citations.
  • Answers buried in PDFs or images. Extraction from these is inconsistent at best.
  • Vague, context-dependent paragraphs. A chunk that starts with "As mentioned above" cannot be quoted standalone.
  • Contradictory facts across pages. Different prices on the pricing page, the blog and the FAQ make a model pick one at random – or skip you.
  • No dates or authorship. Freshness signals help answer engines decide whether your page is still worth citing.

FAQ

Does blocking GPTBot remove my content from ChatGPT?

No. It prevents future crawling for training, but it does not delete what a model has already learned, and it does not affect content that reached the model through third-party datasets or other sites quoting you. It can also reduce your chances of being cited if you block search-oriented bots like OAI-SearchBot.

Should I allow AI crawlers at all?

For most marketing, documentation and e-commerce sites, yes – citations in AI answers are now a discovery channel. Publishers whose business model is paywalled content, and sites with sensitive or licensed material, may reasonably choose to block training crawlers while keeping live-answer bots open.

Is there a difference between a bot that crawls and one that fetches on demand?

Yes, and it matters. Bulk crawlers (GPTBot, ClaudeBot, PerplexityBot) build datasets or indexes in the background. User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) fetch a single URL in response to one person's question, so blocking them can break your page for users who paste your link into a chat.

Do AI crawlers respect crawl-delay or rate limits?

Honouring of Crawl-delay is inconsistent across providers and not guaranteed. If crawl volume is a problem, use server-side rate limiting or CDN rules, and keep your sitemap tight so bots do not wander into faceted URL explosions.

How fast do changes to robots.txt take effect?

Crawlers cache robots.txt, typically for up to around 24 hours [check: current caching guidance per provider]. Expect a delay of a day or so before new rules are fully reflected, and longer before a blocked page stops appearing in answers.

If you want to know which of these issues apply to your own pages, run a free check in Zgadka – it scores your site for both GEO and classic SEO, flags what AI crawlers cannot read, and gives you ready-made robots.txt, llms.txt and schema.org files to fix it.


Fazier badge