Free tools

robots.txt tester

Search Console's robots.txt report shows the files Google fetched and any parsing problems, but it does not test a draft against your URLs. A mistake in this file can hide a whole site. Paste a draft, list the URLs you care about and the crawlers, and see exactly which rule wins for each before you publish anything.

By Adem ÇakalPublished 3 sources

Test a robots.txt draft

Everything runs in this browser tab. Full URLs are reduced to their path and query, which is what robots.txt rules match.

Short answer

A crawler first picks the group with the most specific user-agent that matches it, ignoring all other groups, then applies the longest matching rule in that group, with ties going to Allow. Paste your draft above to see the result for each path and crawler, including which line decided it.

The four rules that decide every result

RuleExampleResult
A named group replaces * for that crawlerUser-agent: * disallows /private/; a GPTBot group only disallows /drafts/GPTBot may crawl /private/
Longest matching path winsDisallow: /docs/ and Allow: /docs/public//docs/public/a is allowed
Equal length: Allow winsAllow: /page and Disallow: /pageAllowed
* and $ wildcardsDisallow: /*.pdf$Blocks /a.pdf, not /a.pdf?x=1

Mistakes worth testing for

Three common robots.txt mistakes and their effect Disallow: /left over from stagingunder User-agent: *Whole site blocked Disallow: /*?meant for search pagesmatches every query stringCampaign URLs blocked Named groupadded for one botwithout copying * rulesPrivate paths opened
Each of these looks harmless in a diff. A test with the right paths shows the effect immediately.

Build your test list from URLs that matter: the homepage, a pricing page, a blog post, a URL with a tracking parameter, a PDF, and any private section. If one of them gives an unexpected answer, the table shows the line number of the rule that decided it.

Testing AI crawlers on purpose

Put the crawlers you actually care about in the crawler box, not only Googlebot. A useful default set covers each job separately: Googlebot and Bingbot for search, OAI-SearchBot and Claude-SearchBot for AI answers, ChatGPT-User for fetches people trigger, and GPTBot or ClaudeBot for training. If your draft blocks a search crawler you meant to allow, the result shows the exact line responsible.

Prefix matching matters here. A group for Claude would also apply to Claude-SearchBot and Claude-User in this tester, because the crawler name starts with it. Name each bot in full when you want different rules for each.

A safe pattern for most sites

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /search

User-agent: GPTBot
User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

This example keeps admin and checkout paths out, blocks two training crawlers entirely, and lets search and answer crawlers read everything else. Change the paths to your own and test them here. After publishing, check what crawlers actually see with the AI crawler checker, and remember that robots.txt does not remove pages from search; for that, use noindex as explained in noindex vs robots.txt vs canonical.

Search Console's robots.txt report shows which robots.txt files Google found for your property, when it last fetched them and any parsing problems. Use it after publishing to confirm Google has the new version.

Questions people ask

Why test a draft instead of the live file?

Because a wrong Disallow on the live file takes effect for every crawler that fetches it. Testing the draft first costs nothing. To test your live file against AI and search crawlers, use the AI crawler checker, which fetches it for you.

Does Allow beat Disallow?

Only when it is at least as long. Google uses the most specific rule, measured by the length of the path, and when an Allow and a Disallow rule are equally specific, it uses the less restrictive Allow.

Are paths case-sensitive?

Yes. Rule paths are matched against the URL path as written, so /Admin/ and /admin/ are different. User-agent names are matched without regard to case.

Do Sitemap or Crawl-delay lines change the result?

No. Only User-agent, Allow and Disallow lines decide access, so the tester ignores the rest. Sitemap lines tell crawlers where your sitemap is, and Google ignores Crawl-delay.

What happens to rules after the first 500 KiB?

Google ignores them. RFC 9309 requires crawlers to parse at least 500 kibibytes, and Google stops there. Large files usually mean generated rules that should be simplified.

Sources

  1. RFC 9309: Robots Exclusion Protocol (IETF, 2022), accessed 2026-11-12.
  2. Google Search Central: How Google interprets the robots.txt specification, accessed 2026-11-12.
  3. Search Console Help: robots.txt report, accessed 2026-11-12.