Check a site
Add a path to test one section, for example example.com/docs/. We read the public robots.txt only and change nothing.
If you want to show up in ChatGPT, Claude, Perplexity and Copilot answers, the bots that matter are the search ones: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot and Googlebot. Blocking the training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) is a separate choice, and the vendors say it does not remove you from their search features.
The three kinds of AI crawler
Most "block AI" advice treats every AI bot the same. The vendors' own documentation splits them into three jobs, and the right robots.txt rule depends on the job.
What each crawler does, in the vendors' words
| Token | Owner | Job | If you block it |
|---|---|---|---|
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | OpenAI tells you to use this token to manage search opt-outs |
GPTBot | OpenAI | Crawls content that may train OpenAI models | Signals your content should not be used for training |
ChatGPT-User | OpenAI | User actions in ChatGPT and custom GPTs | OpenAI: robots.txt rules "may not apply" |
Claude-SearchBot | Anthropic | Indexes content for Claude's search results | May reduce visibility in Claude's search answers |
Claude-User | Anthropic | Fetches a page when a user asks Claude | Claude cannot retrieve your page for that user |
ClaudeBot | Anthropic | Collects content that may train Claude models | Future content excluded from training data |
PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results | Perplexity says it is not used for model training |
Perplexity-User | Perplexity | Visits a page to answer a user's question | Perplexity: "generally ignores robots.txt rules" |
Google-Extended | Token for Gemini training and grounding use | No effect on Google Search inclusion or ranking | |
Applebot-Extended | Apple | Token for Apple's generative model training | Pages can still appear in Spotlight, Siri and Safari |
Two rows are worth a second look. Google-Extended has no user agent of its own; Google crawls with its normal bots and reads the token as a permission. And the user-triggered fetchers (ChatGPT-User, Perplexity-User) are described by their owners as acting for a person, which is why both say robots.txt may not apply.
Which result do you want?
| Your goal | Allow | Block |
|---|---|---|
| Be cited in AI answers, keep content out of training | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended |
| Maximum reach in AI tools | Everything | Nothing |
| Stay out of AI products entirely | Googlebot, Bingbot (if you still want classic search) | All AI tokens, and expect user-triggered fetchers to get through anyway |
A common middle ground, written as robots.txt. Groups are matched by the most specific user agent, so each named bot follows only its own group:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
How the checker decides "allowed" or "blocked"
The checker follows the matching rules Google documents for robots.txt, since most AI crawlers describe the same standard behaviour:
- Pick the group. A crawler obeys the group with the most specific user agent that matches it. If
GPTBothas its own group, the*group no longer applies to it at all. This is the most common surprise: a site blocks*from/private/, then adds aGPTBotgroup, and GPTBot can now crawl/private/. - Pick the rule. Inside the group, the longest matching path wins.
Allow: /blog/beatsDisallow: /for a blog post. - Break ties toward access. When an allow and a disallow rule match with the same length, Google uses the less restrictive one.
- Wildcards.
*matches any characters and$marks the end of the URL, soDisallow: /*.pdf$blocks PDFs only.
Missing file versus broken file: Google treats a robots.txt that returns 4xx (except 429) as "no rules", so everything is allowed. A 5xx answer is different: Google stops crawling the whole site for the first 12 hours and keeps retrying. If your robots.txt sometimes errors, fix that before anything else.
What this check cannot see
- Firewall and CDN rules. Bot-management settings can answer a crawler with a 403 or a challenge even when robots.txt allows it. Our page on Cloudflare and AI crawlers shows where that setting lives.
- Page-level blocks. A
noindexmeta tag or header stops indexing even when crawling is allowed. Test single URLs with the noindex checker. - Whether a bot actually came. Only your server logs show real visits. Allowing a crawler makes a visit possible; it does not schedule one.
Questions people ask
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training data. You can block GPTBot and still allow OAI-SearchBot. OpenAI's crawler page treats them as separate settings.
Does blocking Google-Extended hurt my Google rankings?
Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether content Google crawls may be used for Gemini models and grounding.
My robots.txt allows everything. Why can't an AI tool read my page?
robots.txt is a request, not the only gate. A firewall, CDN bot setting or rate limit can still return a 403 or a challenge page to a crawler that robots.txt allows. This checker reads robots.txt only; check your CDN's bot settings separately.
Do ChatGPT-User and Perplexity-User follow robots.txt?
Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the action, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.
Related
Sources
- OpenAI: Overview of OpenAI crawlers, accessed 2026-11-12.
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?, accessed 2026-11-12.
- Perplexity: Perplexity crawlers, accessed 2026-11-12.
- Google: Google's common crawlers (Google-Extended), accessed 2026-11-12.
- Apple: About Applebot, accessed 2026-11-12.
- Google Search Central: How Google interprets the robots.txt specification, accessed 2026-11-12.