How do ChatGPT, Perplexity and Gemini choose which sites to cite?
AI answer engines don't rank sites the way Google does – they retrieve a handful of candidate pages for each question, break them into passages, and cite the passages the model actually used to write the answer. So citation depends on three things: the crawler can access your page, the page matches a specific question, and a self-contained passage on it answers that question clearly. A page can rank in the top 10 of Google and still never be cited, because it has no quotable chunk.
What actually happens between a question and a citation?
Most AI answers with links are built with retrieval-augmented generation (RAG). The system reformulates the user's question into one or more search queries, fetches results from a search index, downloads or pulls cached text from the top candidates, splits that text into passages, and feeds the most relevant passages into the model as context.
The model then writes the answer and attaches citations to the sources whose passages supported each statement. Link selection is therefore a two-stage filter: first retrieval (did your URL enter the candidate set?), then grounding (did a passage from your page end up in the final text?).
This is why "being indexed" is not enough. If your relevant content is buried in a 3,000-word narrative with no clear answer paragraph, the passage that gets scored may belong to a competitor who stated the answer in two sentences.
Do ChatGPT, Perplexity and Gemini crawl the web themselves?
Partly. Each uses some combination of its own crawler, its own index, and third-party search results. The exact mix changes over time and the providers rarely document it in full, so treat the table below as the current public picture rather than a permanent rule.
| Service | Where results come from | Crawlers / user agents to allow | Notes |
|---|---|---|---|
| ChatGPT search | Own search index plus live fetches; third-party search partners | OAI-SearchBot (index), ChatGPT-User (live fetch on user action), GPTBot (model training) | Blocking GPTBot does not block search; blocking OAI-SearchBot removes you from ChatGPT's search index |
| Perplexity | Own index plus live retrieval | PerplexityBot (index), Perplexity-User (user-initiated fetch) | Answers are citation-first – nearly every claim carries a numbered source |
| Gemini app | Google Search grounding | Googlebot for crawling; Google-Extended controls Gemini/Vertex grounding and training use | Google-Extended is not a crawler – it is a usage control token in robots.txt |
| Google AI Overviews | Google Search index | Googlebot | Governed by normal snippet controls (nosnippet, max-snippet, data-nosnippet) rather than Google-Extended |
| Microsoft Copilot | Bing index | Bingbot | Bing Webmaster Tools indexing signals apply |
Two practical takeaways. First, classic SEO still matters, because Google and Bing indexes feed several of these systems. Second, a separate layer of permissions exists on top of indexing, and many sites block it by accident through a CDN bot filter or a blanket robots.txt rule.
How do I check that AI crawlers can reach my site?
- Open
yourdomain.com/robots.txtand look forDisallow: /under any AI user agent or underUser-agent: *. - Check your WAF or CDN bot-protection rules – many managed "AI bot blocking" toggles are on by default.
- Grep your server logs for
OAI-SearchBot,PerplexityBot,ChatGPT-User,Googlebot,Bingbot. No hits in [check: typical window, e.g. 30 days] usually means a block or a reachability problem. - Fetch a key page with
curland no cookies to confirm the main content is in the HTML, not injected only after JavaScript runs and a consent banner is accepted. - Confirm the page returns
200, is in your sitemap, and is notnoindex.
A permissive baseline looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Training-only crawler – allow or disallow by policy
User-agent: GPTBot
Allow: /
Sitemap: https://example.com/sitemap.xml
If you want visibility in AI answers but not in model training, allow the search and user-triggered agents and disallow the training ones. Do not disallow Google-Extended if you want to appear in Gemini-grounded answers – it is a usage opt-out, not a crawl control.
What makes a page likely to be retrieved?
Retrieval favours pages that look like a direct match to a narrow query. AI systems expand one user question into several specific queries, so pages covering a single, well-defined subtopic get pulled in more often than broad hub pages.
Signals that consistently help:
- Query-shaped headings. An
##heading phrased as the question a person asks matches the expanded query almost literally. - Topical depth on one domain. Clusters of related pages linked to each other make a site look like a subject authority rather than a one-off article.
- Freshness where it matters. Prices, versions, regulations and "best X" lists are often re-retrieved; a visible, accurate
dateModifiedhelps. - Being cited elsewhere. Mentions on sites that the engines already trust – industry media, documentation, forums, review platforms – raise the chance your domain is in the candidate set at all.
- Clean, server-rendered HTML. Content that requires JS execution may be missed by lighter fetchers.
What makes a passage likely to be quoted?
Retrieval gets you into the room; chunk quality gets you into the answer. Writing for chunks means every section must survive being cut out of the page and read alone.
- Answer first, then explain. Put the direct answer in the first two sentences under the heading, before caveats.
- Repeat the entity. Don't write "it" or "this tool" – write the product, standard or term by name in each section, so the chunk makes sense without the heading above it.
- Keep paragraphs 2–4 sentences. Long blocks get truncated mid-thought when chunked.
- Use lists and tables for comparisons, steps, specs. Structured data of any kind is easier to extract and reassemble.
- Give concrete numbers, versions and dates – with sources. Vague claims are rarely chosen as grounding for a factual statement.
- Add an FAQ block of real questions with 40–60 word answers. These map almost one-to-one onto user queries.
- Avoid walls of marketing language. Passages that make unverifiable claims are weak grounding material.
Does schema.org markup influence AI citations?
Structured data does not force a citation, but it removes ambiguity about what the page is, who wrote it and when it was updated – and it powers classic rich results that feed the same indexes. Use Article or FAQPage, Organization, and Product/HowTo where they genuinely apply.
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [{
"@type": "Question",
"name": "Do AI answer engines use Google's index?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Some do. Google AI Overviews and Gemini grounding use Google Search. Copilot uses Bing. ChatGPT and Perplexity combine their own indexes with live fetches and search partners."
}
}],
"author": {"@type": "Organization", "name": "Example Inc."},
"dateModified": "2025-01-15"
}
Keep markup consistent with the visible text. Mismatched or inflated structured data is a quality risk, not a shortcut.
Is llms.txt worth adding?
llms.txt is a proposed convention: a Markdown file at your root that lists your most important URLs with one-line descriptions, so an AI client can find canonical content quickly. Adoption by major engines is not confirmed, so treat it as a low-cost addition, not a ranking lever.
# Example Inc.
> B2B invoicing software for small accounting teams.
## Docs
- [Getting started](https://example.com/docs/start): setup in under 10 minutes
- [API reference](https://example.com/docs/api): endpoints, auth, rate limits
## Policies
- [Pricing](https://example.com/pricing): plans, limits, billing cycle
It costs an hour and gives you a clean, machine-readable map of what you consider authoritative. If the convention gains traction, you are already set up.
Why does a top-ranking page get ignored by AI answers?
The most common reasons, in rough order of frequency:
- The page is blocked for AI crawlers at the robots.txt or CDN level while Googlebot is allowed.
- The answer exists on the page but is spread across several paragraphs, so no single chunk is a complete answer.
- The content is behind a consent wall, a login, or rendered entirely client-side.
- The topic is covered better, more recently or more specifically by another page – including an aggregator or forum thread.
- The engine's query expansion produced a question your page never explicitly asks.
The fix for the last three is editorial, not technical: add the exact question as a heading and answer it in the first lines beneath.
How do I measure whether this is working?
AI answers don't appear in Search Console. Build visibility measurement from three sources:
- Server logs – track hits from
OAI-SearchBot,PerplexityBot,ChatGPT-User,Perplexity-User. Rising crawl frequency is the earliest signal. - Referral traffic – segment sessions with referrers such as
chatgpt.com,perplexity.ai,copilot.microsoft.comandgemini.google.comin your analytics. Volumes are typically small but intent is high. - Prompt testing – maintain a fixed list of [check: suggested number, e.g. 20–30] questions your buyers ask, run them monthly across engines, and record whether you are cited, mentioned without a link, or absent.
Track trend, not single runs. These systems are non-deterministic: the same prompt can yield different sources on two consecutive attempts, so one missing citation proves nothing.
FAQ
Does blocking GPTBot remove me from ChatGPT answers?
No. GPTBot is OpenAI's training crawler. ChatGPT's search results rely on OAI-SearchBot and live fetches by ChatGPT-User. You can block training while staying visible in answers – but check your robots.txt actually distinguishes the agents instead of using a blanket rule.
Do backlinks still matter for AI citations?
Indirectly and significantly. Several engines retrieve from Google's or Bing's index, where links remain a ranking factor, and brand mentions on trusted sites increase the odds your domain enters the candidate set. Links alone won't earn a citation if the passage isn't quotable.
How fast do changes show up in AI answers?
It varies by engine and by how often your site is recrawled. Pages fetched live at query time can reflect edits almost immediately; index-based retrieval may lag by days or weeks. [check: typical observed lag per engine]
Should I write a separate version of my pages for AI?
No. Cloaking different content to bots is against search guidelines and risky. Write one version that is clear for humans and structurally clean for machines – short answer-first paragraphs, question headings, lists and tables.
Is AI visibility just SEO with a new name?
There is heavy overlap – crawlability, indexing, authority and freshness matter in both. The difference is the unit of competition: classic SEO competes for a page ranking, AI visibility competes for a passage being chosen as grounding for a sentence.
If you want to see where your own site stands, run a free check in Згадка at /en/audit – it returns GEO and SEO scores, shows which AI crawlers can reach you, and gives a full report with ready-made robots.txt, llms.txt, schema.org and FAQ files plus an action plan.