Beyond the core set in the AI crawler checker, owners document these tokens: Applebot, Amazonbot, Amzn-User, Amzn-SearchBot, Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher, DuckAssistBot, the MistralAI agents, GoogleOther and Google-CloudVertexBot. Most honour robots.txt, but the owners themselves say user-triggered fetchers may not. Verified November 12, 2026.
The reference table
"Robots.txt" is quoted from the owner's page, not from our testing. Purpose is paraphrased.
| Token | Owner | Purpose per owner | robots.txt per owner | Official doc |
|---|---|---|---|---|
Applebot | Apple | Search for Spotlight, Siri and Safari; data may also help train Apple's generative models | Respects robots.txt; follows Googlebot rules if no Applebot rules exist; ignores crawl-delay | support.apple.com/en-us/119829 |
Amazonbot | Amazon | Improve products and services; may train Amazon AI models | Honours user-agent and allow/disallow rules; may use a robots.txt copy up to 30 days old; no crawl-delay | developer.amazon.com/amazonbot |
Amzn-SearchBot | Amazon | Search experiences such as Alexa; not used for generative AI training | If not named but other search bots are allowed, follows those rules | Same page |
Amzn-User | Amazon | User-initiated actions, such as Alexa queries needing fresh information | May not follow all directives | Same page |
Meta-ExternalAgent | Meta | Train foundation AI models or index content to improve products | Controlled through robots.txt Disallow | Meta web crawlers |
Meta-WebIndexer | Meta | Improve Meta AI search quality; allowing it helps Meta AI cite your content | Page does not say it bypasses robots.txt | Same page |
Meta-ExternalFetcher | Meta | Fetch individual links on user request for agentic features | May bypass robots.txt | Same page |
DuckAssistBot | DuckDuckGo | Real-time fetches for cited AI-assisted answers; not used to train AI models | Disallow the token; takes effect after 72 hours | DuckDuckGo help |
MistralAI-User | Mistral AI | User-initiated page visits in Vibe; not for automatic crawling or training | Page does not explicitly state handling | docs.mistral.ai/robots |
MistralAI-Index | Mistral AI | Search indexing for answers in Vibe; not used for training | Not stated on the page | Same page |
MistralAI-Training | Mistral AI | Datasets for training Mistral models | Can be disallowed in robots.txt | Same page |
GoogleOther | Generic crawler for product teams, such as one-off research crawls; variants for images and video | Listed as a common crawler with robots.txt rules; no effect on a specific Google product | Google common crawlers | |
Google-CloudVertexBot | Crawls requested by site owners to build Vertex AI Agents | Listed with robots.txt rules; no effect on Google Search | Same page |
Reading the table
Blocking a training agent does not remove you from an owner's answers if its search agent is still allowed, and the reverse also holds. Write one rule per token and keep a note of which purpose each one serves. For how the OpenAI trio splits, see our OpenAI crawler guide. If a CDN sits in front of your site, the rule may never reach the crawler; see Cloudflare and AI crawlers.
Example robots.txt group
User-agent: Meta-ExternalAgent Disallow: / User-agent: MistralAI-Training Disallow: / User-agent: DuckAssistBot Allow: /
This blocks two training-oriented agents and explicitly allows one answer agent. Test any change in the robots.txt tester.
Left out on purpose
Bytespider, PetalBot and cohere-ai appear in many log reports. We found no owner page we could open that documents their purpose and robots.txt behaviour, so they are not listed. IP verification methods also differ by owner: Apple publishes reverse-DNS and CIDR details, and DuckDuckGo publishes a JSON IP list. We did not test any agent against a live site.
How to use this list safely
Pick a goal before you edit robots.txt. If you want to appear in AI answers, allow the search and index agents of each vendor, and decide separately about training. If you want to stay out of model training, disallow the tokens that owners describe as training or product-improvement crawlers. If your goal is to limit server load, rate controls and caching help more than blocking, and several owners, Apple and Amazon among them, say they ignore crawl-delay.
Then confirm in your server logs that the agent you targeted is what you see. A user agent string can be copied by anyone, so a log line alone does not prove the owner sent the request. Apple documents reverse-DNS verification and an IP list, and DuckDuckGo publishes its IP list as JSON. For agents where the owner publishes no IP list on the page we read, treat the string as a hint.
Questions people ask
Why are Bytespider, PetalBot and cohere-ai not in the table?
We include only agents whose owner documents them on a page we could open on November 12, 2026. For these three we could not open an owner page that states purpose and robots.txt behaviour, so we make no claims about them.
Does a robots.txt rule stop every agent in the table?
No. Meta says Meta-ExternalFetcher may bypass robots.txt, and Amazon says Amzn-User may not follow all directives. User-initiated fetchers are treated differently from crawlers by their owners.
Can I block AI training without leaving search?
Some owners separate the two. Mistral lists a training agent apart from its index agent, and Meta lists Meta-ExternalAgent apart from Meta-WebIndexer. Check each owner's page before you write rules.
How often does this change?
Owners add and rename agents often. Each row carries the verification date, so treat anything older than a few months as a prompt to re-check the linked page.
Related
Sources
- Apple Support: About Applebot, accessed 2026-11-12.
- Amazon: Amazonbot, accessed 2026-11-12.
- Meta for Developers: Web crawlers, accessed 2026-11-12.
- DuckDuckGo Help: DuckAssistBot, accessed 2026-11-12.
- Mistral AI docs: Robots, accessed 2026-11-12.
- Google Search Central: Common crawlers, accessed 2026-11-12.