The short answer
Cloudflare sorts AI bots into Search, Agent and Training groups and can block each one. Since 1 July 2025, new domains are asked about AI crawlers at signup. To be cited by AI search, open Security settings, choose Configure AI bot policies, and set Search to Allow. You can still block Training.
- Cloudflare can block AI crawlers at its network, even when your robots.txt allows them.
- It groups AI bots into Search, Agent and Training, each with its own setting.
- Allowing Search keeps AI answers able to find and cite you; blocking Training keeps content out of model training.
- Bot Preference Sync writes Cloudflare's own section into your robots.txt.
- Check the result with a crawler checker, then watch visits in AI Crawl Control.
Why does Cloudflare matter for AI visibility?
Cloudflare sits in front of millions of websites. It can stop a crawler before the request reaches your server.
On 1 July 2025, Cloudflare said every new domain would start with AI crawler access restricted unless the owner allows it. New domains are asked about AI crawlers during signup.
Cloudflare's documentation also says new domains get updated defaults from 15 September 2026. Training and Agent bots are blocked on pages with ads, and Search is allowed.
Many owners never revisit these settings. If your site launched recently, check them before assuming AI search can read you.
What are Search, Agent and Training bots?
Cloudflare's AI bot policies have three presets. Each can be set to block on all pages, block on pages with ads, or allow.
| Group | What it covers | Suggested setting for visibility |
|---|---|---|
| Search | Crawlers that index pages for AI search products, so they can cite and link you | Allow |
| Agent | AI assistants that visit a page while helping a user | Allow, unless you have a reason not to |
| Training | Crawlers that collect content to train AI models | Your choice. Blocking does not remove you from AI search |
How do you let AI search crawlers in?
These steps follow Cloudflare's documentation. Menu names can change, so look for the same words if yours differ.
- 1
Open security settings
In the Cloudflare dashboard, choose your domain, then go to Security, then Settings.
- 2
Filter to bot traffic
Filter the settings list by Bot traffic, then select Configure AI bot policies.
- 3
Set Search to Allow
Choose Allow for the Search group so AI search products can crawl and cite you.
- 4
Decide on Agent and Training
Allow Agent if you want assistants to open your pages for users. Set Training to block or disallow if you do not want your content used for training.
- 5
Choose whether to sync robots.txt
Turn on Bot Preference Sync if you want Cloudflare to add matching rules to your robots.txt. Leave it off to manage robots.txt yourself.
- 6
Save and check
Select Save. Cloudflare says changes appear within seconds. Then run the free AI crawler checker on your domain.
What does Cloudflare add to your robots.txt?
With Bot Preference Sync on and at least one group blocked, Cloudflare adds a managed section to the top of your robots.txt. If you have no robots.txt, it serves one for you.
The section starts with a Content Signals Policy comment. It is wrapped in the markers shown below and includes a Content-signal line.
Cloudflare's documented example disallows Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. Your own file depends on your settings.
# BEGIN Cloudflare Managed content Content-signal: search=yes, ai-train=no, use=reference User-agent: GPTBot Disallow: / # ...more groups, depending on your settings # END Cloudflare Managed Content
Edit these rules in Cloudflare, not in your site. Changes to your own file will not touch the managed section.
What about the old Block AI bots switch?
Older accounts may still have a single Block AI bots setting under Security settings. Cloudflare says it is being deprecated on 15 September 2026, in favour of the three AI bot policies.
That switch targets bots that crawl for training. Cloudflare says it now also blocks mixed-purpose crawlers used for both search and training, so check what it catches.
How do you see which AI crawlers visit?
Cloudflare's AI Crawl Control, formerly AI Audit, shows which AI services request your pages. Cloudflare says it is available on all plans.
You can set allow or block rules per crawler and see which crawlers follow your robots.txt. Use it to confirm that search crawlers are reaching you after a change.
Questions people ask
Will blocking training crawlers hurt my AI visibility?
Blocking training crawlers keeps your content out of future model training. AI search features use separate search crawlers, such as OAI-SearchBot and PerplexityBot. Keep those allowed if you want to be cited.
Does Cloudflare block AI crawlers by default?
Since 1 July 2025, new domains are asked at signup and start with AI crawler access restricted unless the owner allows it. From 15 September 2026, Cloudflare documents new defaults: Training and Agent blocked on pages with ads, Search allowed. Older domains keep whatever was set.
My robots.txt allows GPTBot but it still cannot read my site. Why?
Cloudflare can block a crawler at its network before robots.txt matters. Check your AI bot policies and any custom firewall or bot rules. A robots.txt checker cannot see those blocks.
Is robots.txt enough to stop AI crawlers?
No. Following robots.txt is voluntary, and Cloudflare itself says so. To enforce a block, use Cloudflare's AI bot policies or AI Crawl Control.