Since September 15, 2026, Cloudflare has three AI crawler controls per domain: Search, Training and Agent. Each can be Allow, Block on pages with ads, or Block; Training also has Disallow AI Training. To be cited by AI search engines and keep your content out of training, set Search to Allow and Training to Disallow AI Training. Check Agent too: Block there stops the fetches made when a person asks an AI about your page.
How the defaults changed
| Date | Change | Effect on AI search visibility |
|---|---|---|
| July 1, 2025 | Cloudflare announced it was changing its default to block AI crawlers unless they pay creators, with a single "Block AI bots" switch | One switch covered search, training and agent bots alike, so turning it on could also block AI search bots |
| September 15, 2026 | "Block AI bots" and managed robots.txt deprecated; three controls (Search, Training, Agent) and Bot Preference Sync introduced | Search and training can now be set separately |
What your domain has now if you never changed it
New domains added after September 15 are asked whether the site earns ad revenue. Sites without ads start with everything allowed. Sites with ads start with Search allowed, Training set to Disallow AI Training and Agent blocked on pages with ads.
Which setting gives which result
| Control | Setting | What happens |
|---|---|---|
| Search | Allow | Search crawlers, including AI search bots and Googlebot, can crawl for their index |
| Search | Block | Search crawlers are refused. For mixed-use crawlers such as Googlebot and Bingbot this also affects classic search |
| Training | Disallow AI Training | Cloudflare's way to stay indexed for search while refusing training. Googlebot keeps crawling for search |
| Training | Block | Training crawlers are refused at the edge |
| Agent | Block / on pages with ads | User-directed agents (a person asking an AI about your page) are refused on all pages or on pages with ads |
Block on Search is the setting to avoid if you want to be found anywhere. Cloudflare's post warns that Block now applies to mixed-use crawlers such as Applebot, Bingbot and Googlebot, so it affects normal search results too.
How to check and set it
- Open the Cloudflare dashboard, choose the domain, then Security settings. The AI crawler controls are set per domain on every plan.
- Read the three controls. For most businesses that want AI search traffic: Search Allow, Training your choice, Agent Allow.
- Turn on Bot Preference Sync if it is off, so your robots.txt states the same thing your edge enforces. Crawlers that only read robots.txt then see consistent rules.
- Look for custom WAF rules or bot rules that match "bot" or user agents. They act on top of these controls.
- Run the AI crawler checker on your domain. It reads the robots.txt that Bot Preference Sync writes; it cannot see edge blocks, so step 4 still matters.
Testing whether the edge lets a bot through
A quick request with a crawler's user agent shows whether Cloudflare challenges it. This is only a hint: real crawlers are verified by IP, and Cloudflare may treat a spoofed user agent from your laptop differently from the real bot.
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
https://example.com/pricing
# 200 = served; 403 or a challenge page = blocked at the edge
The reliable confirmation is in your logs or Cloudflare's own AI crawler analytics: real visits from verified OAI-SearchBot, Claude-SearchBot or PerplexityBot addresses, answered with status 200. Details on OpenAI's bots and their IP lists are in our OpenAI crawler guide.
Questions people ask
Does Cloudflare block Googlebot when I block AI bots?
Only if you choose Block for Search, or Block for a category that includes Googlebot. Cloudflare classes Googlebot, Bingbot and Applebot as mixed-use crawlers. With Disallow AI Training, Googlebot keeps crawling for search.
My robots.txt allows OAI-SearchBot but ChatGPT still can't read my site. Why?
A Cloudflare Block setting answers the crawler at the network edge, before robots.txt matters. Check Security settings for the Search and Agent controls, and any custom firewall rule that matches bots.
What does Bot Preference Sync do?
It replaces Cloudflare's older managed robots.txt. Cloudflare writes the choices you make in its AI crawler controls into your robots.txt, so crawlers that read robots.txt see the same preferences your firewall enforces.
I set nothing. What do I have now?
Domains that never touched the controls carried their old setting over on September 15, 2026. If Block AI Bots was off, all three categories are Allow. If it was on, Search is Allow, Training is Disallow AI Training and Agent is Block on pages with ads.
Related
Sources
- Cloudflare blog: Have it both ways: stay discoverable in search while disallowing AI training (Sept 15, 2026), accessed 2026-11-12.
- Cloudflare blog: Content Independence Day (July 1, 2025), accessed 2026-11-12.
- OpenAI: Overview of OpenAI crawlers, accessed 2026-11-12.
- Google: Google's common crawlers (Google-Extended), accessed 2026-11-12.