How to fix a site that AI crawlers can't render – JavaScript and blocking
If an AI crawler requests your page and the raw HTML comes back empty, your content does not exist for ChatGPT, Perplexity or Claude – most AI crawlers do not execute JavaScript, and none of them wait for a client-side app to hydrate. The fix is to make sure the important text, links and structured data are present in the first HTML response, and that nothing in robots.txt, your CDN or your firewall returns a 403 to those user agents. Server-side rendering or prerendering plus an open crawl path solves the majority of cases.
Why can't AI crawlers see my JavaScript site?
A classic search engine like Google runs a two-stage pipeline: it fetches the HTML, then queues the page for rendering in a headless browser that executes JavaScript. Most AI crawlers skip the second stage entirely. They fetch, parse the HTML they received, extract text and links, and move on.
That means a single-page application built with React, Vue, Angular or Svelte in client-side mode often returns something like <div id="root"></div> and a script tag. There is nothing to extract. The crawler records an empty or near-empty page, and your brand never becomes a candidate for citation.
There is a second failure mode that looks identical in AI answers but has a different cause: the page renders fine, but the crawler never gets to it. A Disallow rule, a bot-protection challenge, a geo block or an aggressive rate limiter returns 403, 429 or a JavaScript challenge page instead of your content.
How do I test what an AI crawler actually sees?
Test the raw response, not what your browser shows you. The browser runs JavaScript; the crawler usually does not.
# What the crawler receives, with its own user agent
curl -A "GPTBot/1.0 (+https://openai.com/gptbot)" -s -o page.html -w "%{http_code}\n" https://example.com/your-page
# Count visible text characters in the raw HTML
curl -A "PerplexityBot/1.0" -s https://example.com/your-page | sed 's/<[^>]*>//g' | tr -s '[:space:]' ' ' | wc -c
If the status code is not 200, you have a blocking problem. If the status is 200 but the text count is tiny compared with what you see on screen, you have a rendering problem. Repeat the test for ClaudeBot, OAI-SearchBot and Bingbot – some firewalls block selectively.
Three more checks worth running:
- Disable JavaScript in your browser (DevTools → Command Menu → "Disable JavaScript") and reload. What remains is roughly what a non-rendering crawler sees.
- Use "View page source" rather than the Elements panel. The Elements panel shows the DOM after scripts run.
- Grep your server or CDN logs for AI user agents and look at the status codes they received over the last 30 days.
Which crawlers render JavaScript and which don't?
| Crawler | Used for | JavaScript execution |
|---|---|---|
| Googlebot | Google Search, AI Overviews, AI Mode | Yes, renders in a headless browser |
| Bingbot | Bing, Copilot | Renders, but with limits and delays |
| GPTBot | Training data for OpenAI | No rendering documented |
| OAI-SearchBot | ChatGPT search index | No rendering documented |
| ChatGPT-User | Live fetch when a user's prompt needs a page | Limited, fetch-oriented |
| ClaudeBot / Claude-User | Anthropic index and live fetches | No rendering documented |
| PerplexityBot / Perplexity-User | Perplexity index and live fetches | No rendering documented |
| Applebot | Apple services, Siri | Renders JavaScript |
Treat this table as a planning default, not a contract: vendors change crawler behaviour without announcements, and some run limited rendering for specific cases. The safe rule is to assume no rendering for every non-Google crawler.
How do I make the content appear in the raw HTML?
Pick the lightest option your stack allows.
- Server-side rendering (SSR). Next.js, Nuxt, SvelteKit, Remix and similar frameworks render the page on the server and ship complete HTML. This is the most durable fix because it also helps users on slow connections.
- Static generation (SSG) or ISR. For pages that change rarely – articles, product categories, docs, landing pages – pre-build HTML at deploy time or revalidate on a schedule. Fastest and cheapest to serve.
- Prerendering service or edge middleware. If you cannot rewrite the app, run a prerender layer that serves a cached, fully rendered HTML snapshot to bots while users get the SPA. Serve the same content to both: swapping in different text is cloaking.
- Hybrid. Render the main content server-side, hydrate interactive widgets client-side. Filters, carts and configurators can stay in JavaScript; the body text, headings and links cannot.
Whatever you choose, verify with the curl test above. "We added SSR" and "the HTML now contains the article" are different claims.
What must never depend on JavaScript?
- The main body text and all headings (H1–H3).
- Internal links – a crawler cannot click a
divwith an onClick handler. Use real<a href="/path">elements. - Pagination and "load more" content. Give every page a crawlable URL.
- Tabs and accordions that hide answers behind a click. Render the content in the DOM and hide it with CSS instead of injecting it later.
- Structured data. JSON-LD injected by a tag manager after load is unreliable for non-rendering crawlers; put it in the server HTML.
- Canonical tags, meta description, hreflang, title.
How do I fix robots.txt blocking?
Decide deliberately which bots you allow, then write explicit rules. A blanket Disallow: / under a wildcard user agent is the single most common reason a site is invisible in AI answers.
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Applebot
Allow: /
User-agent: *
Allow: /
Disallow: /cart
Disallow: /checkout
Disallow: /*?sort=
Sitemap: https://example.com/sitemap.xml
Two details people get wrong. First, never block /static/, /_next/, /assets/ or your JS and CSS bundles – crawlers that do render need them. Second, if you want to allow search citations but refuse model training, block the training-only agent (GPTBot) while allowing the search agent (OAI-SearchBot); the separation is not symmetrical across vendors, so check each vendor's current documentation before relying on it.
Also check for a page-level block that robots.txt won't reveal:
<!-- Remove or replace this if you want to be cited -->
<meta name="robots" content="noindex, nofollow">
<!-- A reasonable default for public content -->
<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large">
And check the HTTP header version, which overrides nothing but is easy to forget: X-Robots-Tag: noindex.
What if the block is in Cloudflare, a WAF or rate limiting?
This is the hidden killer. Bot-management rules often ship with "block AI crawlers" presets enabled by default, and some hosting providers turn them on for you.
- In your CDN or WAF dashboard, find the bot management or "AI crawlers" setting and review which agents are blocked. Allow the ones you want citations from.
- Check firewall rules for blocks by user agent string, ASN or country. OpenAI, Anthropic and Perplexity publish IP ranges you can allowlist.
- Check rate limits. Crawlers can request many URLs quickly; a 10-requests-per-minute limit turns into a wall of 429s.
- Disable JavaScript challenges and CAPTCHA for paths you want crawled. A challenge page is a 200 response containing no content – it looks like a rendering failure in tests.
- Make sure login walls, cookie banners that block the DOM, and "please verify your age" interstitials are not the only thing in the HTML.
After every change, re-run the curl test with each user agent. Then watch your logs for a week to confirm the bots return.
What should the fixed HTML contain?
Rendered HTML is the minimum. To be quotable, give the crawler a clean structure and explicit facts.
<h1>How much does an air-source heat pump cost to install?</h1>
<p>A typical UK installation costs between £7,000 and £13,000 before grants.
Price depends on property size, radiator upgrades and buffer tank needs.</p>
Add JSON-LD in the server-rendered HTML, not through a tag manager:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [{
"@type": "Question",
"name": "Do AI crawlers execute JavaScript?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Most do not. Googlebot and Applebot render JavaScript; GPTBot, ClaudeBot and PerplexityBot generally parse the raw HTML only."
}
}]
}
</script>
An llms.txt file at your root gives AI systems a plain-text map of your most useful pages. Support is not universal and it does not replace proper HTML, but it costs minutes to publish:
# Example Ltd
> Heat pump installation and servicing in the UK.
## Core pages
- [Pricing](https://example.com/pricing): installation costs by property type
- [Grants guide](https://example.com/grants): eligibility and application steps
- [FAQ](https://example.com/faq): answers to common installation questions
A short fix-it checklist
- Run the curl test on your five most important URLs with four different AI user agents.
- Fix non-200 responses first: robots.txt, meta robots, X-Robots-Tag, WAF, rate limits.
- Move body text, headings, links and JSON-LD into the server response via SSR, SSG or prerendering.
- Replace JavaScript-only navigation with real
<a href>links and crawlable pagination URLs. - Keep JS and CSS assets crawlable.
- Publish an up-to-date XML sitemap and reference it in robots.txt.
- Re-test, then monitor logs for AI user agents over the next 2–4 weeks.
FAQ
Does Google AI Overviews see my JavaScript site?
Usually yes, because AI Overviews draw on Google's regular index and Googlebot renders JavaScript. But rendering is queued and can lag, so a slow or script-heavy page may be indexed late or partially. Server-rendered HTML is still faster and more reliable.
Is prerendering for bots considered cloaking?
Not if the prerendered HTML matches what users see. Cloaking means serving different content to crawlers – extra keywords, hidden text, a different offer. Serving the same content in a different delivery format is accepted practice.
Should I block GPTBot to protect my content?
That is a business decision, not a technical one. Blocking training crawlers reduces the chance your text is used in model training; it may also reduce how often your brand surfaces in AI answers, depending on how each vendor separates training from search crawling. If citations and referral traffic matter to you, keep the search-oriented agents open.
How long until AI answers reflect my fix?
Expect weeks rather than days. Crawlers must revisit, indexes must update, and some assistants fetch live at query time while others rely on cached indexes. Track your log files for crawler visits first – that is the earliest signal the fix worked.
My HTML is fine but I'm still not cited. What else matters?
Rendering is a prerequisite, not a guarantee. After the page is readable, the factors that matter are clear question-style headings, direct answers near the top, specific facts with dates and sources, structured data, and mentions of your brand on third-party sites the models already trust.
Want to know which of these problems your site actually has? Run a free check in Zgadka – it returns GEO and SEO scores, shows what crawlers see on your pages, and gives you a report with fixes and ready-made robots.txt, llms.txt and schema.org files.