Robots.txt controls crawling, not indexing. URL blocked by robots.txt means Google did not index the page. Indexed, though blocked by robots.txt is a warning: Google indexed the URL anyway. To keep a page out of Google, remove the block and add noindex. To keep it in, remove the block. In both cases the Disallow line is what goes.
The two statuses side by side
| URL blocked by robots.txt | Indexed, though blocked by robots.txt | |
|---|---|---|
| Category in the report | Not indexed | Warning |
| Google's definition | The page was blocked by your robots.txt file | The page was indexed despite being blocked |
| Caveat | No guarantee the page is kept out by other means | Warnings do not prevent indexing |
| If you want it out of Google | Remove the block, add noindex | Remove the block, add noindex |
| If you want it in Google | Remove the block | Remove the block |
| If the block is intended | Google says these are probably intentional; judge whether to act | Accept the warning, knowing the bare URL may show |
Why a block cannot keep a page out
Google's robots.txt introduction states that the file is not a mechanism for keeping a web page out of Google. A blocked URL can appear with no description. Google also warns that combining crawling and indexing rules can make rules counteract each other. Both statuses come from the same fact: a link can introduce a URL that Google is not allowed to fetch.
Find the rule that blocks the URL
- Open
https://yourdomain/robots.txt. It applies only to that protocol, host and port. A subdomain needs its own file. - Look at the group for your crawler. Google picks the group with the most specific matching user-agent and does not merge it with the
*group. - Read the rule precedence. The longest matching path wins. If Allow and Disallow conflict, Google applies the least restrictive rule.
- Test the URL with the AI crawler checker, which tests robots rules per path, or with a live test in URL Inspection.
# blocks every URL under /shop/ that contains a query string
User-agent: *
Disallow: /shop/*?
# blocks only URLs that end in .pdf
Disallow: /*.pdf$
# a more specific Allow overrides a shorter Disallow
Disallow: /private/
Allow: /private/press/
Google supports * for any run of characters and $ for the end of the URL. Fields such as crawl-delay are not used. Sites with filter or parameter URLs often block them with a wildcard rule like the first one; if those URLs now show as indexed with a warning, the rule is doing what robots.txt does, which is stop crawling, not indexing.
Decide per URL type
| URL type | Goal | Action |
|---|---|---|
| Real page you want ranked | Indexed | Delete the Disallow rule; confirm with a live test |
| Thin or duplicate page, such as parameter variants | Out of results | Allow crawling, add noindex or a canonical to the main page, wait for a recrawl |
| Private area | Not publicly reachable | Password protection; Google names it as an option next to noindex |
| Resources such as images or scripts | Crawl load control | A block may be fine; it is not a way to hide them |
After the change, click Validate fix in the Page indexing report once. Google asks you not to click it again until validation succeeds or fails. Check the leftover URLs with the noindex checker, which reads the robots rules, the header and the meta tag together, and use the sitemap checker to make sure no blocked URL is still listed in your sitemap.
Platform plugins often write robots.txt for you, and a staging copy may carry a blanket Disallow. Open the live file after every launch rather than trusting a settings screen.
Questions people ask
Why would a page blocked by robots.txt be indexed?
Google's documentation says it might still find and index a disallowed URL if it is linked from other places on the web. The result then appears without a description, because Google could not read the page.
Should I add noindex to the robots.txt file?
No. Google's documented robots.txt fields are user-agent, allow, disallow and sitemap. To keep a page out of search, use a noindex meta tag or header on a page that Googlebot is allowed to crawl.
How long does validation take?
Google's Page indexing help says validation typically takes up to about two weeks. It gives no separate timeline for the Indexed, though blocked warning.
I fixed robots.txt. Why does Search Console still show the block?
Google generally caches robots.txt for up to 24 hours, and the report reflects past crawls. Re-run a live URL Inspection to confirm that crawling is now allowed, then validate the fix.
Related
Sources
- Search Console Help: Page indexing report, accessed 2026-11-12.
- Google Search Central: Introduction to robots.txt, accessed 2026-11-12.
- Google Search Central: How Google interprets the robots.txt specification, accessed 2026-11-12.
- Google Search Central: Block search indexing with noindex, accessed 2026-11-12.