Free tools

Noindex checker: can Google index this URL?

A page can be kept out of Google in four different places: a meta tag, an HTTP header, robots.txt, or a redirect. This checker looks at all four for one URL and tells you which one is in the way.

By Adem ÇakalPublished 3 sources

Check a URL

We request the page once as a normal visitor, follow up to five redirects and read robots.txt. Nothing is stored except the host name.

Short answer

Google can index a URL when it answers with HTTP 200, robots.txt lets Googlebot crawl it, and neither the <meta name="robots"> tag nor the X-Robots-Tag header says noindex or none. If robots.txt blocks the URL, Google never sees the noindex at all, and the URL can still appear in results without a description.

The four gates, in the order Google meets them

Order of indexing checks: robots.txt, HTTP status and redirects, X-Robots-Tag header, meta robots tag 1. robots.txtmay Googlebot fetch it? 2. HTTP status200, redirect or error? 3. X-Robots-Tagheader on the response 4. meta robotstag in the HTML head Blocked: tags 3 and 4 unseen3xx: the target is judgedWorks for PDFs and imagesMost CMS settings land here
If gate 1 says no, Google cannot read gates 3 and 4. That is why "Disallow" plus "noindex" on the same URL backfires.

Reading the result

What the checker showsWhat it meansWhat to do
Meta robots contains noindex or noneThe page asks every search engine not to index itRemove it in your CMS or theme; on WordPress, check Settings → Reading and the SEO plugin's per-page setting
Meta tag with name="googlebot"The rule applies to Google onlySame fix; Bing ignores this one
X-Robots-Tag: noindex headerThe server or CDN adds the rule, often for a whole folder or file typeLook in server config, _headers, middleware or CDN rules; it will not appear in the HTML
robots.txt blocks GooglebotGoogle cannot crawl the page, so any noindex is invisibleTo remove a page from Google: allow crawling and add noindex. To keep it crawlable: remove the Disallow line
HTTP 3xx chainGoogle indexes the final target, not this URLLink to and list the final URL in your sitemap
HTTP 4xx or 5xxNothing to indexFix the page or return 410 if it is gone for good

Where a stray noindex usually comes from

  • A staging setting that went live. WordPress's "Discourage search engines from indexing this site" adds a noindex to every page. It is the first thing to check after a launch or migration.
  • Framework defaults. In Next.js, a robots: { index: false } entry in a layout's metadata applies to every page under that layout. Preview deployments often add X-Robots-Tag: noindex on purpose; make sure the production domain does not.
  • Header rules on a CDN. A rule meant for /admin/ or PDFs that matches more than intended. Header rules never show up in "view source", which is why this check reads the response headers.
  • Plugin conflicts. Two SEO plugins, or a theme and a plugin, each printing a robots tag. The checker lists every robots meta value it finds.

Remove a page from Google the right way: let Googlebot crawl it and serve noindex. Blocking it in robots.txt instead can leave the bare URL in results, because Google may index a URL it knows from links without crawling it.

The tags, exactly

<!-- in the HTML head: all search engines -->
<meta name="robots" content="noindex">

<!-- Google only -->
<meta name="googlebot" content="noindex">

# as an HTTP response header (any file type)
X-Robots-Tag: noindex

After you change any of these on a page that should rank, open URL Inspection in Search Console, run a live test to confirm Google sees the new version, and request indexing. Then run the sitemap checker to make sure no other listed URL carries the same leftover rule.

Questions people ask

I removed noindex. Why is the page still missing from Google?

Google has to recrawl the page and see the change. Google notes that it may take months for Googlebot to revisit a page. Use URL Inspection in Search Console and request indexing to speed up that one URL.

Can I put noindex in robots.txt?

No. Google does not support a noindex rule in robots.txt. Use the meta robots tag or the X-Robots-Tag header, and make sure robots.txt does not block the page.

What is the difference between noindex and nofollow?

noindex keeps the page out of search results. nofollow asks search engines not to follow the links on the page. The value none means both.

Does noindex stop AI tools from using my page?

It keeps the page out of Google Search, including its AI features, and out of Bing. AI crawlers with their own indexes follow their own rules, which are usually set in robots.txt. Check those with the AI crawler checker.

Sources

  1. Google Search Central: Block search indexing with noindex, accessed 2026-11-12.
  2. Google Search Central: Robots meta tag, data-nosnippet and X-Robots-Tag specifications, accessed 2026-11-12.
  3. Google Search Central: How Google interprets the robots.txt specification, accessed 2026-11-12.