AI search

OpenAI's crawlers: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot

Blocking "ChatGPT" in robots.txt can mean three different things, and only one of them takes you out of ChatGPT search. Here is what each OpenAI bot does according to OpenAI, how the settings interact, and how to check which ones visit you.

By Adem ÇakalPublished 4 sources6 min read
Short answer

OAI-SearchBot decides whether you can appear in ChatGPT search answers. GPTBot collects pages that may train OpenAI's models. ChatGPT-User fetches a page when a person asks ChatGPT to, and robots.txt may not apply to it. OAI-AdsBot only visits pages submitted as ChatGPT ads. Each setting is independent, so "in ChatGPT search, out of training" is one robots.txt group away.

The four bots side by side

OAI-SearchBotGPTBotChatGPT-UserOAI-AdsBot
JobSurfaces sites in ChatGPT searchCrawls content that may train foundation modelsUser actions in ChatGPT and custom GPTsChecks pages submitted as ads
Crawls on its own?YesYesNo, only when a user triggers itOnly submitted ad pages
Follows robots.txt?YesYes"May not apply"Not stated
If you block itNot shown in ChatGPT search answers; can still appear as a navigational linkContent signalled as not for trainingFirewall needed to stop itOnly matters if you run ChatGPT ads
IP listopenai.com/searchbot.jsonopenai.com/gptbot.jsonopenai.com/chatgpt-user.jsonopenai.com/adsbot.json

One detail is easy to miss. When a site allows both OAI-SearchBot and GPTBot, OpenAI says it may use the results of one crawl for both purposes to avoid crawling twice. Your logs may then show fewer GPTBot visits than you expect, even though training access is open.

What "opted out of search" actually removes

Effect of blocking OAI-SearchBot versus GPTBot on ChatGPT search answers and training Block OAI-SearchBot Block GPTBot ✗ Not shown in ChatGPT search answers ✓ Can still appear as a navigational link · Training setting unchanged · Takes about 24 hours to apply ✓ Search answers unchanged ✗ Content signalled as not for training · No statement on content already collected · ChatGPT-User fetches still possible
Based on OpenAI's crawler overview. "Navigational link" means a plain link to your site when someone searches for it by name.

robots.txt for the three common choices

In ChatGPT search, out of training

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Out of ChatGPT entirely (as far as robots.txt reaches)

User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: ChatGPT-User
Disallow: /

ChatGPT-User may still fetch pages a user asks about, so the last line is a request, not a lock.

Open a docs section to search, keep the rest closed

User-agent: OAI-SearchBot
Allow: /docs/
Disallow: /

The longer matching rule wins, so /docs/ pages are allowed and everything else is not. Test any of these against your live file with the AI crawler checker, adding a path such as /docs/ to the address.

Finding OpenAI's bots in your logs

Each bot names itself in the user-agent string, with a version number that OpenAI says may change. Match on the token, not the full string:

# access log lines, simplified
"GET /pricing HTTP/1.1" 200 "... compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
"GET /blog/setup HTTP/1.1" 200 "... compatible; GPTBot/1.4; +https://openai.com/gptbot"
"GET /blog/setup HTTP/1.1" 200 "... compatible; ChatGPT-User/1.0; +https://openai.com/bot"

# count visits per bot (Linux, combined log format)
grep -oE "OAI-SearchBot|GPTBot|ChatGPT-User|OAI-AdsBot" access.log | sort | uniq -c

Anyone can put "GPTBot" in a user agent. To know a visit is really OpenAI, check the request IP against the matching JSON list above. A scraper using OpenAI's name from a residential IP is not OpenAI, and blocking it does not affect ChatGPT.

If your host or CDN shows "verified bot" labels, that check has usually been done for you. If you see ChatGPT-User hits on a page, it often means someone pasted that page into ChatGPT, which is a signal worth noticing on its own.

What none of these settings control

  • Whether you are cited. Allowing OAI-SearchBot makes you eligible. Being chosen depends on whether your page answers the question better than the alternatives.
  • Content already collected. OpenAI's page describes the GPTBot block as a signal about use in training; it does not say what happens to content gathered before you added it.
  • Other companies' crawlers. Claude, Perplexity, Google and Bing each have their own tokens. The checker covers 13 of them in one pass.
  • Firewall blocks. A CDN rule can stop OpenAI's bots even when robots.txt allows them; see Cloudflare and AI crawlers.

Questions people ask

If I block GPTBot, will ChatGPT still cite my site?

It can. OpenAI describes GPTBot (training) and OAI-SearchBot (search) as independent settings. Block GPTBot, allow OAI-SearchBot, and your pages can still be shown in ChatGPT search answers.

How long until a robots.txt change affects ChatGPT search?

OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust.

Why does ChatGPT-User visit pages I disallowed?

ChatGPT-User acts on a person's request inside ChatGPT, for example when they paste your link. OpenAI says robots.txt rules may not apply to these user-initiated actions. If you must stop it, block its published IP ranges at your firewall.

Is a visit from OAI-SearchBot proof that I rank in ChatGPT?

No. It means your page can be considered for ChatGPT search. Whether it is cited depends on the question and the competing pages. Track citations separately.

Sources

  1. OpenAI: Overview of OpenAI crawlers, accessed 2026-11-12.
  2. OpenAI: OAI-SearchBot IP ranges (searchbot.json), accessed 2026-11-12.
  3. OpenAI: GPTBot IP ranges (gptbot.json), accessed 2026-11-12.
  4. OpenAI: ChatGPT-User IP ranges (chatgpt-user.json), accessed 2026-11-12.