Fix it

Page indexed without content in Search Console: how to find what Googlebot received and fix it

The URL is in Google's index, but Google could not read what was on it. Your browser shows a full page, so the job is to find out what a crawler receives instead.

By Adem ÇakalPublished 5 sources
Short answer

Google's help page defines the status as a page that appears in the index, but Google could not read its content. It names two causes: the page is cloaked to Google, or it is in a format Google cannot index. It also says robots.txt blocking is not the issue. In practice, check what Googlebot receives when it requests the URL, which can differ from what you see: an empty HTML shell, a bot-protection page, or an empty response. Then fix that response.

What the status means

The URL made it into the index, so Google found and stored it. What failed was reading the content. That makes this different from crawled, currently not indexed, where Google read the page and passed on it, and different from blocked by robots.txt, which Google says is not the cause here. See crawling vs indexing if the distinction is new.

Google does not give a reason per URL, so the table below lists the situations that fit its two documented causes and the JavaScript guidance. Treat each row as a hypothesis to test.

Cause, check, fix

Likely causeHow to checkFix
Content exists only after JavaScript runs (app-shell page)Compare raw HTML from curl with the rendered HTML in URL Inspection. Google says content not visible in the rendered HTML cannot be indexed.Server-side rendering or pre-rendering, so the first response already contains the text. Google recommends both.
Googlebot gets a different page than visitors (firewall, bot protection, CDN challenge)Request the page with a Googlebot user-agent, then check server or CDN logs for the response sent to verified Googlebot IPs.Allow verified Googlebot traffic. Match against Google's published IP ranges, not the user-agent string alone.
Cloaking, on purpose or by a plugin (content for users, something else for crawlers)Run the live test in URL Inspection and compare the rendered page with what you see in a browser.Serve the same content to Google and to users. Remove the rule or plugin that branches on user-agent.
Empty or truncated server response (200 with no body, a crash after the headers)Use curl with a byte count and the status line. Check error logs at the request time.Fix the application error. Return a real error status for a broken page instead of an empty 200.
A format Google cannot index (wrong content type, non-text file served at a page URL)Read the content-type header from curl -I and confirm the URL returns HTML.Serve HTML with a text/html content type, or move the file to a URL meant for it.
Robots meta and header rules interacting with renderingCheck for noindex in the HTML, in the rendered DOM and in the X-Robots-Tag header with the noindex checker. Google says pages with noindex may skip rendering.Make the directives match your intent in both raw and rendered HTML. Do not add noindex in the shell and remove it with a script.

Diagnose it in six steps

  1. Inspect the URL. Open URL Inspection for the exact URL. The Page indexing section reports the status. Google recommends inspecting the page when you see it.
  2. Test the live URL. Run the live test, then open the rendered HTML (the screenshot view, where your account shows one, helps for a quick visual check). Google's JavaScript guidance names the URL Inspection tool and the rendered HTML as the place to check what it can index.
  3. Compare with the raw response. Fetch the page without a browser and look for a sentence from your real content. If it is missing from the raw HTML and present in the rendered view, the page depends on rendering.
  4. Request as Googlebot. Fetch the page with a Googlebot user-agent. A different body or a challenge page points to bot protection or cloaking. A match does not clear you, because protection often keys on IP address too.
  5. Read the logs. Find requests from Google's IPs and look at the status, size and any challenge or block entries. Verify the IP by reverse DNS (the host must end in googlebot.com, google.com or googleusercontent.com, and a forward lookup must return the same IP) or by matching the ranges in Google's common-crawlers.json file.
  6. Retest after each change. Run the live test again. Request indexing once when the rendered HTML shows the content.

Worked example: a page that is empty before JavaScript

A launch site returns this to any client that does not run scripts:

$ curl -s https://example.com/ | head -c 400
<!doctype html>
<html><head><title>Acme</title></head>
<body><div id="root"></div>
<script src="/assets/app.4f2a.js"></script>
</body></html>

$ curl -s -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
    -o /dev/null -w "%{http_code} %{size_download} %{content_type}\n" https://example.com/
200 312 text/html; charset=utf-8

The body is a root element and a script tag, 312 bytes in total. Google says an app shell like this needs JavaScript to reach the content, and content that never appears in the rendered HTML cannot be indexed. In a browser the page looks fine, so nobody notices. The fix is to render the page on the server or pre-render it at build time, so the first response contains the headline and body text. Then rerun the same curl and expect your visible text in the output.

If the same request with the Googlebot user-agent returns a 403, a 503 or a small interstitial instead, the problem sits in front of your application, in a firewall or CDN rule.

The AI crawler angle

Googlebot queues every page with a 200 status for rendering, so a client-rendered page can eventually be read. Other crawlers may not wait. Google's own guidance says server-side rendering and pre-rendering also help because some crawlers cannot run JavaScript. Vercel's research on AI crawlers found that none of the major AI crawlers it tested render JavaScript. They included OpenAI's, Anthropic's, Meta's, ByteDance's and Perplexity's crawlers, and Vercel noted that some fetch script files without running them.

The gap shows up on new sites. In SiteTiller's study of Show HN launches, 24 of 180 sampled sites (13.3%, 95% CI 9.1-19.1) served under 20 readable words in the homepage HTML. A page like that has little for a non-rendering crawler to read. You can check your own robots rules for AI user-agents with the AI crawler checker, and read how Cloudflare settings can block AI crawlers if your site sits behind it.

Fixes in order of effort

  1. Render on the server or at build time. Use the framework's server rendering, static generation or a prerender step so the HTML contains the title, headings, body text and links.
  2. Allow verified Googlebot. In the firewall or bot-management settings, allow requests whose IPs match Google's published ranges. Google notes that the crawler JSON files do not cover every Google IP, such as Apps Script, so use them for Googlebot checks only.
  3. Review CDN bot and challenge settings. Look for rules that issue a challenge, a block or an alternate page to crawlers. Review managed rules separately from your own, since they change without a deploy.
  4. Fix the response itself. Return a full HTML body, the right content type and an honest status code.

Common mistakes