Free tools

AI crawler checker: which bots can read your site?

Enter an address and see, crawler by crawler, whether your robots.txt lets it in. The result is grouped by what each bot is for: search answers, fetches a user asked for, or model training. Those three need different decisions.

By Adem ÇakalPublished 6 sources

Check a site

Add a path to test one section, for example example.com/docs/. We read the public robots.txt only and change nothing.

Short answer

If you want to show up in ChatGPT, Claude, Perplexity and Copilot answers, the bots that matter are the search ones: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot and Googlebot. Blocking the training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) is a separate choice, and the vendors say it does not remove you from their search features.

The three kinds of AI crawler

Most "block AI" advice treats every AI bot the same. The vendors' own documentation splits them into three jobs, and the right robots.txt rule depends on the job.

Three crawler purposes: search indexing, user-requested fetches, and model training Search and answers Builds the index an answer engine cites and links to OAI-SearchBot · Claude-SearchBot PerplexityBot · Bingbot · Googlebot User-requested fetch Opens one page because a person asked about it ChatGPT-User · Claude-User Perplexity-User Model training Collects text that may train future models GPTBot · ClaudeBot · CCBot Google-Extended · Applebot-Extended Block = you can't be cited robots.txt often not applied Block = opt out of training
Grouping taken from each vendor's crawler page (sources below). Google-Extended and Applebot-Extended are control tokens: they never fetch pages themselves.

What each crawler does, in the vendors' words

TokenOwnerJobIf you block it
OAI-SearchBotOpenAISurfaces sites in ChatGPT searchOpenAI tells you to use this token to manage search opt-outs
GPTBotOpenAICrawls content that may train OpenAI modelsSignals your content should not be used for training
ChatGPT-UserOpenAIUser actions in ChatGPT and custom GPTsOpenAI: robots.txt rules "may not apply"
Claude-SearchBotAnthropicIndexes content for Claude's search resultsMay reduce visibility in Claude's search answers
Claude-UserAnthropicFetches a page when a user asks ClaudeClaude cannot retrieve your page for that user
ClaudeBotAnthropicCollects content that may train Claude modelsFuture content excluded from training data
PerplexityBotPerplexitySurfaces and links sites in Perplexity resultsPerplexity says it is not used for model training
Perplexity-UserPerplexityVisits a page to answer a user's questionPerplexity: "generally ignores robots.txt rules"
Google-ExtendedGoogleToken for Gemini training and grounding useNo effect on Google Search inclusion or ranking
Applebot-ExtendedAppleToken for Apple's generative model trainingPages can still appear in Spotlight, Siri and Safari

Two rows are worth a second look. Google-Extended has no user agent of its own; Google crawls with its normal bots and reads the token as a permission. And the user-triggered fetchers (ChatGPT-User, Perplexity-User) are described by their owners as acting for a person, which is why both say robots.txt may not apply.

Which result do you want?

Your goalAllowBlock
Be cited in AI answers, keep content out of trainingOAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, GooglebotGPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended
Maximum reach in AI toolsEverythingNothing
Stay out of AI products entirelyGooglebot, Bingbot (if you still want classic search)All AI tokens, and expect user-triggered fetchers to get through anyway

A common middle ground, written as robots.txt. Groups are matched by the most specific user agent, so each named bot follows only its own group:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

How the checker decides "allowed" or "blocked"

The checker follows the matching rules Google documents for robots.txt, since most AI crawlers describe the same standard behaviour:

  1. Pick the group. A crawler obeys the group with the most specific user agent that matches it. If GPTBot has its own group, the * group no longer applies to it at all. This is the most common surprise: a site blocks * from /private/, then adds a GPTBot group, and GPTBot can now crawl /private/.
  2. Pick the rule. Inside the group, the longest matching path wins. Allow: /blog/ beats Disallow: / for a blog post.
  3. Break ties toward access. When an allow and a disallow rule match with the same length, Google uses the less restrictive one.
  4. Wildcards. * matches any characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDFs only.

Missing file versus broken file: Google treats a robots.txt that returns 4xx (except 429) as "no rules", so everything is allowed. A 5xx answer is different: Google stops crawling the whole site for the first 12 hours and keeps retrying. If your robots.txt sometimes errors, fix that before anything else.

What this check cannot see

  • Firewall and CDN rules. Bot-management settings can answer a crawler with a 403 or a challenge even when robots.txt allows it. Our page on Cloudflare and AI crawlers shows where that setting lives.
  • Page-level blocks. A noindex meta tag or header stops indexing even when crawling is allowed. Test single URLs with the noindex checker.
  • Whether a bot actually came. Only your server logs show real visits. Allowing a crawler makes a visit possible; it does not schedule one.

Questions people ask

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training data. You can block GPTBot and still allow OAI-SearchBot. OpenAI's crawler page treats them as separate settings.

Does blocking Google-Extended hurt my Google rankings?

Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether content Google crawls may be used for Gemini models and grounding.

My robots.txt allows everything. Why can't an AI tool read my page?

robots.txt is a request, not the only gate. A firewall, CDN bot setting or rate limit can still return a 403 or a challenge page to a crawler that robots.txt allows. This checker reads robots.txt only; check your CDN's bot settings separately.

Do ChatGPT-User and Perplexity-User follow robots.txt?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the action, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.

Sources

  1. OpenAI: Overview of OpenAI crawlers, accessed 2026-11-12.
  2. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?, accessed 2026-11-12.
  3. Perplexity: Perplexity crawlers, accessed 2026-11-12.
  4. Google: Google's common crawlers (Google-Extended), accessed 2026-11-12.
  5. Apple: About Applebot, accessed 2026-11-12.
  6. Google Search Central: How Google interprets the robots.txt specification, accessed 2026-11-12.