Check a URL
We request the page once as a normal visitor, follow up to five redirects and read robots.txt. Nothing is stored except the host name.
Google can index a URL when it answers with HTTP 200, robots.txt lets Googlebot crawl it, and neither the <meta name="robots"> tag nor the X-Robots-Tag header says noindex or none. If robots.txt blocks the URL, Google never sees the noindex at all, and the URL can still appear in results without a description.
The four gates, in the order Google meets them
Reading the result
| What the checker shows | What it means | What to do |
|---|---|---|
Meta robots contains noindex or none | The page asks every search engine not to index it | Remove it in your CMS or theme; on WordPress, check Settings → Reading and the SEO plugin's per-page setting |
Meta tag with name="googlebot" | The rule applies to Google only | Same fix; Bing ignores this one |
X-Robots-Tag: noindex header | The server or CDN adds the rule, often for a whole folder or file type | Look in server config, _headers, middleware or CDN rules; it will not appear in the HTML |
| robots.txt blocks Googlebot | Google cannot crawl the page, so any noindex is invisible | To remove a page from Google: allow crawling and add noindex. To keep it crawlable: remove the Disallow line |
| HTTP 3xx chain | Google indexes the final target, not this URL | Link to and list the final URL in your sitemap |
| HTTP 4xx or 5xx | Nothing to index | Fix the page or return 410 if it is gone for good |
Where a stray noindex usually comes from
- A staging setting that went live. WordPress's "Discourage search engines from indexing this site" adds a noindex to every page. It is the first thing to check after a launch or migration.
- Framework defaults. In Next.js, a
robots: { index: false }entry in a layout's metadata applies to every page under that layout. Preview deployments often addX-Robots-Tag: noindexon purpose; make sure the production domain does not. - Header rules on a CDN. A rule meant for
/admin/or PDFs that matches more than intended. Header rules never show up in "view source", which is why this check reads the response headers. - Plugin conflicts. Two SEO plugins, or a theme and a plugin, each printing a robots tag. The checker lists every robots meta value it finds.
Remove a page from Google the right way: let Googlebot crawl it and serve noindex. Blocking it in robots.txt instead can leave the bare URL in results, because Google may index a URL it knows from links without crawling it.
The tags, exactly
<!-- in the HTML head: all search engines -->
<meta name="robots" content="noindex">
<!-- Google only -->
<meta name="googlebot" content="noindex">
# as an HTTP response header (any file type)
X-Robots-Tag: noindex
After you change any of these on a page that should rank, open URL Inspection in Search Console, run a live test to confirm Google sees the new version, and request indexing. Then run the sitemap checker to make sure no other listed URL carries the same leftover rule.
Questions people ask
I removed noindex. Why is the page still missing from Google?
Google has to recrawl the page and see the change. Google notes that it may take months for Googlebot to revisit a page. Use URL Inspection in Search Console and request indexing to speed up that one URL.
Can I put noindex in robots.txt?
No. Google does not support a noindex rule in robots.txt. Use the meta robots tag or the X-Robots-Tag header, and make sure robots.txt does not block the page.
What is the difference between noindex and nofollow?
noindex keeps the page out of search results. nofollow asks search engines not to follow the links on the page. The value none means both.
Does noindex stop AI tools from using my page?
It keeps the page out of Google Search, including its AI features, and out of Bing. AI crawlers with their own indexes follow their own rules, which are usually set in robots.txt. Check those with the AI crawler checker.
Related
Sources
- Google Search Central: Block search indexing with noindex, accessed 2026-11-12.
- Google Search Central: Robots meta tag, data-nosnippet and X-Robots-Tag specifications, accessed 2026-11-12.
- Google Search Central: How Google interprets the robots.txt specification, accessed 2026-11-12.