Free tools

What AI crawlers see on your page

Many sites look complete in a browser and nearly empty in the HTML the server sends. This tool fetches one URL the way a crawler that does not run JavaScript would, and shows the text, headings and robots.txt access that crawler gets.

By Adem ÇakalPublished 4 sources

Check a page

We fetch raw HTML like a non-rendering crawler: one request, redirects followed up to five times, no JavaScript run. We also read robots.txt. Nothing is stored except the host name.

Short answer

If the HTML your server sends contains your headline and body text, a crawler that does not run JavaScript can read it. If the HTML holds an empty container such as <div id="root"></div> and a script bundle, that crawler gets almost nothing. Vercel's analysis of AI crawler traffic found that the major AI crawlers it measured did not render JavaScript, while Google does, in a separate step.

How to read the result

What the tool showsWhat it meansWhat to do
Fewer than 20 visible wordsAlmost nothing readable in the raw HTMLRender the content on the server or at build time (see below)
20 to 49 visible wordsThin: a header and a sentence or two reach a crawlerCheck whether the main content is loaded by script; if so, move it into the HTML
50 or more visible wordsReal text is present before any script runsCompare the first 600 characters with what you want quoted
Empty app shell foundAn empty root, app or __next container, filled in later by scriptAdd server-side rendering or prerendering for public pages
noscript notice presentThe page tells scriptless visitors to enable JavaScriptThat message is all a non-rendering crawler reads if nothing else is in the body
No H1 or title in raw HTMLThe headings are added by scriptPut the H1 and title in the server response
A bot shows Blockedrobots.txt disallows that crawler for this URLEdit its group; see the AI crawler checker for fix lines

The visible word count follows one rule: count words outside script, style, noscript, template and svg. Under 20 words we call "almost nothing readable" and under 50 "thin". These are our thresholds for a quick triage, not limits set by any search engine.

Which crawlers run JavaScript

Two paths: a non-rendering crawler reads the HTML response only, a rendering crawler also runs scripts in a queue 1. HTML responsewhat this tool shows 2. Render queueheadless browser runs scripts 3. Rendered pagewhat you see in a browser Non-rendering crawlers stop here Google renders after crawling, per Google's docs
Vercel reported that Gemini uses Googlebot's rendering infrastructure and that AppleBot renders pages. Apple's own page says Applebot may render content in a browser.

Vercel's December 2024 analysis lists OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot and PerplexityBot among crawlers that did not render JavaScript. OpenAI's crawler documentation does not state either way. Behaviour can change after any article is published, so use your own server logs as the final check.

Fixing an empty or thin result

  1. Pick the pages that matter first: home, pricing, product and documentation pages.
  2. Move the text into the server response. Next.js, Nuxt, SvelteKit, Astro and Angular can all render pages on the server or at build time. For a Vite or Create React App single-page app, a prerender step produces static HTML for public routes.
  3. Keep links as real anchors. Google can only discover links that are <a> elements with an href.
  4. Run the check again. The word count should be close to what you read in the browser, and the first 600 characters should be your headline and opening paragraph, not a menu.

Do not hide the problem with a noscript paragraph that repeats your pitch. It is not the page, and it will not be the text people or answer engines are shown.

When the content is readable, check access with the AI crawler checker and see what each OpenAI bot does in the OpenAI crawler guide. Our study of new Show HN sites shows how common an empty homepage is.

Questions people ask

Does this tool run JavaScript?

No. It makes one request and reads the HTML the server returns, the same starting point a non-rendering crawler has. A browser then runs scripts and adds more content, which is why the page can look fuller to you than it does here.

Does Google see the same thing?

Not always. Google says Googlebot queues pages with a 200 status for rendering and runs their JavaScript in a headless Chromium, so Google can index content that this tool does not see. Google still recommends server-side or pre-rendering because not all bots can run JavaScript.

My page shows plenty of words here. Is it fine?

For this check, yes: the text is in the HTML response. Also confirm that robots.txt allows the crawlers you care about and that a firewall or CDN rule does not turn them away, which this tool cannot see.

Why does the tool count words outside noscript?

A noscript block is shown only when scripts are off or fail. Text that exists only there is usually a notice such as an enable-JavaScript message, not the page content, so counting it would hide the problem.

Sources

  1. Vercel: The rise of the AI crawler, accessed 2026-11-16.
  2. Google Search Central: Understand JavaScript SEO basics, accessed 2026-11-16.
  3. Apple: About Applebot, accessed 2026-11-16.
  4. OpenAI: Overview of OpenAI crawlers, accessed 2026-11-16.