AI search

AI crawler user agents beyond the big names: a dated reference

The well-known AI crawlers are covered by our checker. This page lists the next tier: user agents that owners document on their own pages, with what each owner says about purpose and robots.txt. Every row links to the owner's page, and we left out agents we could not find official documentation for.

By Adem ÇakalPublished 6 sources6 min read
Short answer

Beyond the core set in the AI crawler checker, owners document these tokens: Applebot, Amazonbot, Amzn-User, Amzn-SearchBot, Meta-ExternalAgent, Meta-WebIndexer, Meta-ExternalFetcher, DuckAssistBot, the MistralAI agents, GoogleOther and Google-CloudVertexBot. Most honour robots.txt, but the owners themselves say user-triggered fetchers may not. Verified November 12, 2026.

The reference table

"Robots.txt" is quoted from the owner's page, not from our testing. Purpose is paraphrased.

TokenOwnerPurpose per ownerrobots.txt per ownerOfficial doc
ApplebotAppleSearch for Spotlight, Siri and Safari; data may also help train Apple's generative modelsRespects robots.txt; follows Googlebot rules if no Applebot rules exist; ignores crawl-delaysupport.apple.com/en-us/119829
AmazonbotAmazonImprove products and services; may train Amazon AI modelsHonours user-agent and allow/disallow rules; may use a robots.txt copy up to 30 days old; no crawl-delaydeveloper.amazon.com/amazonbot
Amzn-SearchBotAmazonSearch experiences such as Alexa; not used for generative AI trainingIf not named but other search bots are allowed, follows those rulesSame page
Amzn-UserAmazonUser-initiated actions, such as Alexa queries needing fresh informationMay not follow all directivesSame page
Meta-ExternalAgentMetaTrain foundation AI models or index content to improve productsControlled through robots.txt DisallowMeta web crawlers
Meta-WebIndexerMetaImprove Meta AI search quality; allowing it helps Meta AI cite your contentPage does not say it bypasses robots.txtSame page
Meta-ExternalFetcherMetaFetch individual links on user request for agentic featuresMay bypass robots.txtSame page
DuckAssistBotDuckDuckGoReal-time fetches for cited AI-assisted answers; not used to train AI modelsDisallow the token; takes effect after 72 hoursDuckDuckGo help
MistralAI-UserMistral AIUser-initiated page visits in Vibe; not for automatic crawling or trainingPage does not explicitly state handlingdocs.mistral.ai/robots
MistralAI-IndexMistral AISearch indexing for answers in Vibe; not used for trainingNot stated on the pageSame page
MistralAI-TrainingMistral AIDatasets for training Mistral modelsCan be disallowed in robots.txtSame page
GoogleOtherGoogleGeneric crawler for product teams, such as one-off research crawls; variants for images and videoListed as a common crawler with robots.txt rules; no effect on a specific Google productGoogle common crawlers
Google-CloudVertexBotGoogleCrawls requested by site owners to build Vertex AI AgentsListed with robots.txt rules; no effect on Google SearchSame page

Reading the table

Three groups of agents: training or indexing crawlers, search crawlers, and user-triggered fetchers Training or product crawl Meta-ExternalAgent MistralAI-Training Amazonbot, Applebot Search and answer index Meta-WebIndexer MistralAI-Index Amzn-SearchBot Fetch on user request Meta-ExternalFetcher Amzn-User, MistralAI-User DuckAssistBot
Grouping is ours, based on each owner's description. Owners say the last group is the one that may ignore robots.txt.

Blocking a training agent does not remove you from an owner's answers if its search agent is still allowed, and the reverse also holds. Write one rule per token and keep a note of which purpose each one serves. For how the OpenAI trio splits, see our OpenAI crawler guide. If a CDN sits in front of your site, the rule may never reach the crawler; see Cloudflare and AI crawlers.

Example robots.txt group

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: MistralAI-Training
Disallow: /

User-agent: DuckAssistBot
Allow: /

This blocks two training-oriented agents and explicitly allows one answer agent. Test any change in the robots.txt tester.

Left out on purpose

Bytespider, PetalBot and cohere-ai appear in many log reports. We found no owner page we could open that documents their purpose and robots.txt behaviour, so they are not listed. IP verification methods also differ by owner: Apple publishes reverse-DNS and CIDR details, and DuckDuckGo publishes a JSON IP list. We did not test any agent against a live site.

How to use this list safely

Pick a goal before you edit robots.txt. If you want to appear in AI answers, allow the search and index agents of each vendor, and decide separately about training. If you want to stay out of model training, disallow the tokens that owners describe as training or product-improvement crawlers. If your goal is to limit server load, rate controls and caching help more than blocking, and several owners, Apple and Amazon among them, say they ignore crawl-delay.

Then confirm in your server logs that the agent you targeted is what you see. A user agent string can be copied by anyone, so a log line alone does not prove the owner sent the request. Apple documents reverse-DNS verification and an IP list, and DuckDuckGo publishes its IP list as JSON. For agents where the owner publishes no IP list on the page we read, treat the string as a hint.

Questions people ask

Why are Bytespider, PetalBot and cohere-ai not in the table?

We include only agents whose owner documents them on a page we could open on November 12, 2026. For these three we could not open an owner page that states purpose and robots.txt behaviour, so we make no claims about them.

Does a robots.txt rule stop every agent in the table?

No. Meta says Meta-ExternalFetcher may bypass robots.txt, and Amazon says Amzn-User may not follow all directives. User-initiated fetchers are treated differently from crawlers by their owners.

Can I block AI training without leaving search?

Some owners separate the two. Mistral lists a training agent apart from its index agent, and Meta lists Meta-ExternalAgent apart from Meta-WebIndexer. Check each owner's page before you write rules.

How often does this change?

Owners add and rename agents often. Each row carries the verification date, so treat anything older than a few months as a prompt to re-check the linked page.

Sources

  1. Apple Support: About Applebot, accessed 2026-11-12.
  2. Amazon: Amazonbot, accessed 2026-11-12.
  3. Meta for Developers: Web crawlers, accessed 2026-11-12.
  4. DuckDuckGo Help: DuckAssistBot, accessed 2026-11-12.
  5. Mistral AI docs: Robots, accessed 2026-11-12.
  6. Google Search Central: Common crawlers, accessed 2026-11-12.