Learn

Noindex, robots.txt or canonical: pick by goal

Three controls get mixed up constantly because all three seem to say "leave this page alone". They do different jobs, and using the wrong one often makes things worse. Pick by what you want to happen, using what Google's documentation says.

By Adem ÇakalPublished 3 sources6 min read
Short answer

Use noindex to keep a page out of search results. Use robots.txt to control what gets crawled, not what gets indexed. Use canonical (or a redirect) when several URLs show the same content and you want one to win. Never combine robots.txt blocking with noindex on the same page: Google cannot read the noindex if it cannot fetch the page.

Decision table by goal

Your goalUseWhy, per GoogleDo not use
Remove a page from Google resultsnoindex meta tag or X-Robots-Tag headerGoogle says to use the noindex tag or header, and the page must not be blocked by robots.txtrobots.txt Disallow: Google may still index a disallowed URL linked from elsewhere
Save crawl effort on low-value URLs (filters, internal search)robots.txt DisallowUsed primarily to manage crawler trafficExpecting it to remove URLs already indexed
Merge duplicate or near-duplicate URLsRedirect, rel=canonical, sitemap inclusionRedirect and rel=canonical are strong signals; sitemap is a weak onenoindex or robots.txt for canonicalization
Keep content privatePassword protection or server-side access controlGoogle lists password protection as an alternative to noindexAny of the three alone: they are requests, not locks
Hide a non-HTML file (PDF, image, video)X-Robots-Tag headerGoogle notes the header suits non-HTML filesA meta tag, which these files cannot carry

Crawl control and index control are different layers

robots.txt acts before the fetch, noindex and canonical act after the page is fetched 1. Crawlrobots.txt decideswhether Google fetchesthe URL 2. Indexnoindex is read onlyif the fetch happened 3. Pick canonicalamong indexableduplicates
A block at step 1 hides everything at steps 2 and 3. That is why a blocked page can stay listed with no description.

The exact syntax

<!-- in the page head: keep out of Google (or use name="googlebot" for Google only) -->
<meta name="robots" content="noindex">

# HTTP response header, for PDFs and other files
X-Robots-Tag: noindex

# robots.txt: stop crawling a section (does not de-index)
User-agent: *
Disallow: /search/

<!-- duplicate URL pointing at the preferred one -->
<link rel="canonical" href="https://example.com/product/blue-widget">

These are examples with a placeholder domain. Test your own pages with the noindex checker.

Mistakes that cancel each other

  1. Blocking in robots.txt and adding noindex. Google says the page must not be blocked or the crawler never reads the rule. Remove the Disallow first, wait for the recrawl, and block later only if you still want to. The status in Search Console is explained in Blocked by robots.txt.
  2. Noindex on a duplicate to "help" canonical. Google advises against it within one site; it blocks the page completely. Use rel=canonical.
  3. Canonical pointing to a noindex page. The signals contradict each other. Point canonicals only at pages you want indexed.
  4. Expecting speed. Google says it may take months to revisit a page. Request a recrawl with the URL Inspection tool, as covered in the inspection walkthrough.
  5. Leaving noindex on after launch. It is the most common cause of a vanished page; see noindex stuck after turning it off.

A quick way to choose

Ask three questions in order. First: should anyone be able to reach this page at all? If not, protect it with a login, because the three tags are requests that well-behaved crawlers follow, not security. Second: should it appear in search? If not, use noindex and leave it crawlable. Third: does another URL show the same content? If yes, redirect to it when users do not need the old address, or add rel=canonical when they do. Reach for robots.txt only when the real problem is crawl load on large, low-value URL sets such as faceted filters, and accept that blocked URLs can still be listed without a description if other sites link to them.

When Search Console reports a canonical conflict, go to Duplicate without user-selected canonical.

Questions people ask

Can I put noindex in robots.txt?

Do not rely on it. Google's robots.txt documentation says the file is for managing crawler traffic and is not a mechanism for keeping a page out of Google. Use a noindex meta tag or X-Robots-Tag header on the page itself.

Why is my noindexed page still in Google?

Either Google has not recrawled it yet (Google says it may take months to revisit a page), or robots.txt blocks the page so the noindex is never read.

Should duplicate pages be noindexed?

Within one site, Google does not recommend noindex to steer canonical selection, because it blocks the page from Search completely. It says rel=canonical is the preferred solution.

Is rel=canonical a command?

No. Google describes redirects, rel=canonical and sitemaps as signals of different strength. Google may pick a different canonical than the one you declare.

Sources

  1. Google Search Central: Introduction to robots.txt, accessed 2026-11-12.
  2. Google Search Central: Block Search indexing with noindex, accessed 2026-11-12.
  3. Google Search Central: Consolidate duplicate URLs, accessed 2026-11-12.