Use noindex to keep a page out of search results. Use robots.txt to control what gets crawled, not what gets indexed. Use canonical (or a redirect) when several URLs show the same content and you want one to win. Never combine robots.txt blocking with noindex on the same page: Google cannot read the noindex if it cannot fetch the page.
Decision table by goal
| Your goal | Use | Why, per Google | Do not use |
|---|---|---|---|
| Remove a page from Google results | noindex meta tag or X-Robots-Tag header | Google says to use the noindex tag or header, and the page must not be blocked by robots.txt | robots.txt Disallow: Google may still index a disallowed URL linked from elsewhere |
| Save crawl effort on low-value URLs (filters, internal search) | robots.txt Disallow | Used primarily to manage crawler traffic | Expecting it to remove URLs already indexed |
| Merge duplicate or near-duplicate URLs | Redirect, rel=canonical, sitemap inclusion | Redirect and rel=canonical are strong signals; sitemap is a weak one | noindex or robots.txt for canonicalization |
| Keep content private | Password protection or server-side access control | Google lists password protection as an alternative to noindex | Any of the three alone: they are requests, not locks |
| Hide a non-HTML file (PDF, image, video) | X-Robots-Tag header | Google notes the header suits non-HTML files | A meta tag, which these files cannot carry |
Crawl control and index control are different layers
The exact syntax
<!-- in the page head: keep out of Google (or use name="googlebot" for Google only) -->
<meta name="robots" content="noindex">
# HTTP response header, for PDFs and other files
X-Robots-Tag: noindex
# robots.txt: stop crawling a section (does not de-index)
User-agent: *
Disallow: /search/
<!-- duplicate URL pointing at the preferred one -->
<link rel="canonical" href="https://example.com/product/blue-widget">
These are examples with a placeholder domain. Test your own pages with the noindex checker.
Mistakes that cancel each other
- Blocking in robots.txt and adding noindex. Google says the page must not be blocked or the crawler never reads the rule. Remove the Disallow first, wait for the recrawl, and block later only if you still want to. The status in Search Console is explained in Blocked by robots.txt.
- Noindex on a duplicate to "help" canonical. Google advises against it within one site; it blocks the page completely. Use rel=canonical.
- Canonical pointing to a noindex page. The signals contradict each other. Point canonicals only at pages you want indexed.
- Expecting speed. Google says it may take months to revisit a page. Request a recrawl with the URL Inspection tool, as covered in the inspection walkthrough.
- Leaving noindex on after launch. It is the most common cause of a vanished page; see noindex stuck after turning it off.
A quick way to choose
Ask three questions in order. First: should anyone be able to reach this page at all? If not, protect it with a login, because the three tags are requests that well-behaved crawlers follow, not security. Second: should it appear in search? If not, use noindex and leave it crawlable. Third: does another URL show the same content? If yes, redirect to it when users do not need the old address, or add rel=canonical when they do. Reach for robots.txt only when the real problem is crawl load on large, low-value URL sets such as faceted filters, and accept that blocked URLs can still be listed without a description if other sites link to them.
When Search Console reports a canonical conflict, go to Duplicate without user-selected canonical.
Questions people ask
Can I put noindex in robots.txt?
Do not rely on it. Google's robots.txt documentation says the file is for managing crawler traffic and is not a mechanism for keeping a page out of Google. Use a noindex meta tag or X-Robots-Tag header on the page itself.
Why is my noindexed page still in Google?
Either Google has not recrawled it yet (Google says it may take months to revisit a page), or robots.txt blocks the page so the noindex is never read.
Should duplicate pages be noindexed?
Within one site, Google does not recommend noindex to steer canonical selection, because it blocks the page from Search completely. It says rel=canonical is the preferred solution.
Is rel=canonical a command?
No. Google describes redirects, rel=canonical and sitemaps as signals of different strength. Google may pick a different canonical than the one you declare.
Related
Sources
- Google Search Central: Introduction to robots.txt, accessed 2026-11-12.
- Google Search Central: Block Search indexing with noindex, accessed 2026-11-12.
- Google Search Central: Consolidate duplicate URLs, accessed 2026-11-12.