What is llms.txt and does your website actually need one?
llms.txt is a proposed standard file placed at the root of your domain (/llms.txt) that gives large language models a curated, Markdown-formatted map of your most important content. It is not an official standard, and no major AI provider has publicly confirmed that its crawlers read it. Adding one is cheap and low-risk, but it will not fix an AI visibility problem on its own – crawler access, clean HTML and well-structured content matter far more.
What exactly is llms.txt?
llms.txt is a plain Markdown file served at https://example.com/llms.txt. It lists the pages on your site that an LLM should read first – documentation, product pages, pricing, policies – with a one-line description for each link.
The proposal came from Jeremy Howard of Answer.AI [check: original publication date, late 2024]. The idea is practical: an LLM working with a limited context window cannot crawl and parse an entire website of HTML, navigation menus, cookie banners and JavaScript. A short Markdown index tells it where the substance is.
Despite the name, llms.txt is not a permissions file. It does not block or allow anything. It is closer to a human-written table of contents than to robots.txt.
What does an llms.txt file look like?
The format is deliberately simple: an H1 with the site name, an optional blockquote summary, optional free text, then H2 sections containing bullet lists of links with descriptions.
# Acme Analytics
> Acme Analytics is a self-serve product analytics platform for
> SaaS teams. This file lists the canonical documentation and
> reference pages for the product.
Pricing is per tracked event. All APIs are REST + JSON.
## Docs
- [Quickstart](https://acme.com/docs/quickstart.md): install the SDK and send your first event
- [Event schema](https://acme.com/docs/events.md): required and optional fields for every event type
- [REST API reference](https://acme.com/docs/api.md): endpoints, auth, rate limits
## Product
- [Pricing](https://acme.com/pricing): plan tiers, event limits, annual discounts
- [Integrations](https://acme.com/integrations): supported data sources and destinations
## Policies
- [Security and data residency](https://acme.com/security): SOC 2, EU hosting, retention
- [Terms of service](https://acme.com/terms)
## Optional
- [Engineering blog](https://acme.com/blog): background articles, not product documentation
Two conventions are worth knowing. A section named ## Optional signals content that can be skipped when context is tight. And linking to .md versions of pages, where you can generate them, gives the model clean text instead of HTML.
Who actually reads llms.txt today?
Honestly: unclear, and probably fewer systems than vendors selling "llms.txt services" imply.
No major AI provider – OpenAI, Anthropic, Google, Microsoft or Perplexity – has published documentation saying its crawler fetches /llms.txt as part of indexing or grounding. Google representatives have publicly downplayed it [check: exact statement and date from John Mueller]. Treat any claim that "ChatGPT reads your llms.txt" as unverified.
What does happen in practice:
- Some developer tools, documentation platforms and coding agents fetch llms.txt when a user points them at a domain.
- Some AI assistants, when a user pastes a URL, will follow a discoverable index file to find related pages.
- Third-party scrapers and RAG pipelines built by individual companies use it because it is the cheapest way to ingest a site.
So the realistic value today is for agent-driven and user-initiated retrieval, not for the big crawl-and-index pipelines behind AI Overviews or ChatGPT search.
How is llms.txt different from robots.txt and sitemap.xml?
These three files are often confused. They solve different problems and none replaces another.
| File | Format | Purpose | Who honours it |
|---|---|---|---|
robots.txt | Plain text directives | Allow or block crawlers by user-agent | Widely honoured by Google, Bing, OpenAI, Anthropic, Perplexity |
sitemap.xml | XML | Complete list of indexable URLs plus metadata | Standard, used by search engines |
llms.txt | Markdown | Curated short list of the pages that matter most, with descriptions | Proposal; adoption uncertain |
The practical hierarchy: robots.txt decides whether AI systems can read you at all, sitemap.xml ensures everything is discoverable, llms.txt is an optional hint about priority. If robots.txt blocks AI crawlers, llms.txt is irrelevant.
Here is a robots.txt that allows the main AI crawlers while keeping a sitemap reference:
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://example.com/sitemap.xml
Note that Google-Extended controls Gemini and AI grounding usage, while ordinary Googlebot access governs classic indexing and AI Overviews. Blocking Googlebot removes you from both.
Does your website need an llms.txt file?
It depends on what kind of site you run.
Strong case – add one:
- Documentation sites, APIs and developer tools. Agents and assistants query these constantly, and a Markdown index is genuinely useful.
- Large sites where the important content is buried under thousands of low-value URLs.
- SaaS products where pricing, limits and integration details are frequently asked about by users in AI chats.
Weak case – low priority:
- Small brochure sites of 10–30 pages with clean navigation. A model can read the whole site anyway.
- Local businesses. Your visibility depends on Google Business Profile, reviews and directory consistency, not on a Markdown index.
- Sites whose main problem is JavaScript-rendered content or blocked crawlers. Fix that first.
The honest summary: llms.txt costs maybe an hour to produce and has no downside. Add it. Just do not expect it to move your AI citation rate, and do not let it displace the work that actually does.
How do you create an llms.txt file?
- List your 10–40 most important URLs. Pages that answer real questions: product, pricing, documentation, comparisons, policies, key guides. Not tag archives, not pagination.
- Write one sentence per link describing what a reader gains from it. Be factual, not promotional – "per-event pricing with annual discounts" beats "the best pricing in the industry".
- Group links under H2 headings that reflect intent: Docs, Product, Pricing, Policies, Guides.
- Add an
## Optionalsection for background material that can be skipped. - Write the H1 and blockquote. The blockquote is the single most quotable line in the file – make it a plain, accurate description of what your company does and for whom.
- Save as UTF-8 Markdown and serve it at
/llms.txtwithContent-Type: text/plain(ortext/markdown). It must return HTTP 200, not a redirect chain. - Verify in a browser or with curl.
curl -I https://example.com/llms.txtshould show a 200 and a plain-text content type. - Review it whenever your site structure or pricing changes. A stale llms.txt is worse than none, because it feeds outdated facts to anything that reads it.
What is llms-full.txt?
llms-full.txt is a companion convention: a single large Markdown file containing the full text of your key pages, not just links. It is mainly used by documentation platforms so an agent can ingest everything in one request.
It makes sense for docs of moderate size. For a content-heavy marketing site it becomes a huge file that nobody fetches and that drifts out of date fast. If you generate it, generate it automatically from your source content – never maintain it by hand.
What matters more than llms.txt for AI visibility?
If your goal is being cited in ChatGPT, Perplexity, Gemini or Google AI Overviews, these have far more established impact:
- Crawler access. Check robots.txt, CDN bot rules and WAF settings. Many sites block AI crawlers at the Cloudflare level without realising it.
- Server-rendered content. If the answer only appears after JavaScript executes, many AI crawlers will not see it. Most AI crawlers do not render JavaScript the way Googlebot does.
- Direct answers near the top of the page. A two-sentence answer under a question-shaped heading is what gets extracted and quoted.
- Structured data. Organization, Product, FAQPage and Article schema give machines unambiguous facts.
- Being mentioned elsewhere. AI answers draw heavily on third-party sources – reviews, comparison sites, forums, industry directories. Presence off your own domain is often decisive.
A minimal Organization block that resolves common ambiguities:
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Acme Analytics",
"url": "https://acme.com",
"logo": "https://acme.com/logo.png",
"description": "Self-serve product analytics for SaaS teams.",
"foundingDate": "2019",
"sameAs": [
"https://www.linkedin.com/company/acme-analytics",
"https://github.com/acme-analytics"
]
}
What are the common llms.txt mistakes?
- Treating it as a blocking mechanism. It grants and denies nothing. Use robots.txt for that.
- Dumping the whole sitemap into it. The point is curation. 500 links defeat the purpose.
- Marketing copy instead of facts. Models extract facts; adjectives get dropped or, worse, quoted back at you unflatteringly.
- Letting it go stale. Old prices and discontinued products in llms.txt are a liability.
- Serving it as HTML. It must be raw Markdown text, not a rendered page.
- Paying for it as a standalone "AI SEO" service. It is a one-hour task with uncertain payoff.
FAQ
Is llms.txt an official standard?
No. It is a community proposal with no formal endorsement from any standards body or major AI provider. It may gain adoption, or it may be superseded by something else – treat it as optional today.
Will llms.txt get my site into ChatGPT or Google AI Overviews?
There is no evidence that it will. Those systems rely on their own crawlers and on search indexes. Crawler access, server-rendered content and third-party mentions are what determine whether you appear.
Can llms.txt stop AI companies from training on my content?
No. Use robots.txt directives for crawlers such as GPTBot, Google-Extended, ClaudeBot and Applebot-Extended, and consider contractual or technical blocking at the CDN level if you need stronger control.
Where do I put llms.txt and what should it return?
At the root of each domain – https://example.com/llms.txt – returning HTTP 200 with plain-text or Markdown content type. Subdomains need their own file; it does not inherit.
How long should an llms.txt file be?
Short enough to be useful: typically 10–40 curated links. If you need to expose full content, use a generated llms-full.txt alongside it rather than bloating the index.
Before you spend time on llms.txt, it is worth knowing whether AI crawlers can reach your content at all. You can run a free check of your site in Zgadka to see your GEO and SEO scores, which crawlers are blocked, and get ready-made robots.txt, llms.txt and schema.org files along with a prioritised action plan.