Fix it

Blocked by robots.txt in Search Console: what each status means and how to fix it

Search Console has two robots.txt statuses that sound alike and point in opposite directions. One says the page is not indexed. The other says it is indexed anyway. The right fix depends on whether you want the URL in Google.

By Adem ÇakalPublished 4 sources
Short answer

Robots.txt controls crawling, not indexing. URL blocked by robots.txt means Google did not index the page. Indexed, though blocked by robots.txt is a warning: Google indexed the URL anyway. To keep a page out of Google, remove the block and add noindex. To keep it in, remove the block. In both cases the Disallow line is what goes.

The two statuses side by side

URL blocked by robots.txtIndexed, though blocked by robots.txt
Category in the reportNot indexedWarning
Google's definitionThe page was blocked by your robots.txt fileThe page was indexed despite being blocked
CaveatNo guarantee the page is kept out by other meansWarnings do not prevent indexing
If you want it out of GoogleRemove the block, add noindexRemove the block, add noindex
If you want it in GoogleRemove the blockRemove the block
If the block is intendedGoogle says these are probably intentional; judge whether to actAccept the warning, knowing the bare URL may show

Why a block cannot keep a page out

With a Disallow rule Googlebot never reads the page, so noindex is unseen and the URL can be indexed from links; without it, noindex is read and the page drops Disallow plus noindex Googlebot reads robots.txt, stops The page is never fetched noindex is never seen Linked URL may still be indexed Crawlable plus noindex Googlebot fetches the page It reads the noindex rule The page leaves results after a recrawl Documented removal method
Google: if the page is blocked, the crawler never sees the noindex rule, so the page can still appear in results.

Google's robots.txt introduction states that the file is not a mechanism for keeping a web page out of Google. A blocked URL can appear with no description. Google also warns that combining crawling and indexing rules can make rules counteract each other. Both statuses come from the same fact: a link can introduce a URL that Google is not allowed to fetch.

Find the rule that blocks the URL

  1. Open https://yourdomain/robots.txt. It applies only to that protocol, host and port. A subdomain needs its own file.
  2. Look at the group for your crawler. Google picks the group with the most specific matching user-agent and does not merge it with the * group.
  3. Read the rule precedence. The longest matching path wins. If Allow and Disallow conflict, Google applies the least restrictive rule.
  4. Test the URL with the AI crawler checker, which tests robots rules per path, or with a live test in URL Inspection.
# blocks every URL under /shop/ that contains a query string
User-agent: *
Disallow: /shop/*?

# blocks only URLs that end in .pdf
Disallow: /*.pdf$

# a more specific Allow overrides a shorter Disallow
Disallow: /private/
Allow: /private/press/

Google supports * for any run of characters and $ for the end of the URL. Fields such as crawl-delay are not used. Sites with filter or parameter URLs often block them with a wildcard rule like the first one; if those URLs now show as indexed with a warning, the rule is doing what robots.txt does, which is stop crawling, not indexing.

Decide per URL type

URL typeGoalAction
Real page you want rankedIndexedDelete the Disallow rule; confirm with a live test
Thin or duplicate page, such as parameter variantsOut of resultsAllow crawling, add noindex or a canonical to the main page, wait for a recrawl
Private areaNot publicly reachablePassword protection; Google names it as an option next to noindex
Resources such as images or scriptsCrawl load controlA block may be fine; it is not a way to hide them

After the change, click Validate fix in the Page indexing report once. Google asks you not to click it again until validation succeeds or fails. Check the leftover URLs with the noindex checker, which reads the robots rules, the header and the meta tag together, and use the sitemap checker to make sure no blocked URL is still listed in your sitemap.

Platform plugins often write robots.txt for you, and a staging copy may carry a blanket Disallow. Open the live file after every launch rather than trusting a settings screen.

Questions people ask

Why would a page blocked by robots.txt be indexed?

Google's documentation says it might still find and index a disallowed URL if it is linked from other places on the web. The result then appears without a description, because Google could not read the page.

Should I add noindex to the robots.txt file?

No. Google's documented robots.txt fields are user-agent, allow, disallow and sitemap. To keep a page out of search, use a noindex meta tag or header on a page that Googlebot is allowed to crawl.

How long does validation take?

Google's Page indexing help says validation typically takes up to about two weeks. It gives no separate timeline for the Indexed, though blocked warning.

I fixed robots.txt. Why does Search Console still show the block?

Google generally caches robots.txt for up to 24 hours, and the report reflects past crawls. Re-run a live URL Inspection to confirm that crawling is now allowed, then validate the fix.

Sources

  1. Search Console Help: Page indexing report, accessed 2026-11-12.
  2. Google Search Central: Introduction to robots.txt, accessed 2026-11-12.
  3. Google Search Central: How Google interprets the robots.txt specification, accessed 2026-11-12.
  4. Google Search Central: Block search indexing with noindex, accessed 2026-11-12.