Technical AI SEO · September 2026
How to set Cloudflare for AI search: Search, Agent and Training bots
First published .
Cloudflare now separates AI traffic into Search, Agent and Training. For a brand that wants to be discovered by AI, those three switches should not automatically get the same answer.
If your goal is to get mentioned by ChatGPT, Claude, Perplexity and other assistants, the right crawler policy depends on what you are trying to achieve.
Cloudflare's current controls distinguish three behaviors. Search crawlers collect or index content so it can be used in answers. Agent traffic acts in real time on behalf of a user. Training crawlers collect content that may be used to train or fine-tune models. Cloudflare documents the distinction here.
The simple setting I would use for a new website
For a new site that wants AI visibility, I would start permissive:
| Cloudflare setting | New site | Why |
|---|---|---|
| Search | Allow | You want AI search systems to discover and retrieve your pages. |
| Agent | Allow | You want user-triggered assistants to be able to fetch your site when needed. |
| Training | Allow | A new brand has little stored model knowledge. Giving training crawlers access preserves the possibility of being learned in future model updates. |
This does not guarantee inclusion in a training dataset, and inclusion does not guarantee recommendation. But blocking training before the models know you removes one possible route into future model knowledge.
For an established website, separate memory from retrieval
If the site has been around for years, the question changes. First test whether the model already knows the brand without web search. We published a practical audit for this: check whether your brand is in the weights.
If the brand is already recognized reliably without browsing, there is a reasonable case for keeping Search and Agent open while deciding separately whether you still want to allow Training.
OpenAI itself separates these controls. Its documentation says OAI-SearchBot is used for search, while GPTBot is used to crawl content that may be used to improve generative AI foundation models. The settings are independent. OpenAI's crawler documentation is here.
Important nuance: blocking a training crawler does not force ChatGPT to cite your website. There is no such switch. What it can do is preserve search access while opting out of one training route. When an answer needs fresh or specific evidence, retrieval is then the route you want to be technically available.
Why retrieval matters more than it used to
This is becoming more important because assistants are getting more deliberate about searching specific websites. In our latest piece on the evolution of onsite content, we looked at evidence that ChatGPT is increasingly using domain-restricted searches. That makes the website itself a more important evidence surface.
A site can be known by the model and still fail to appear in a recommendation. It can also be retrieved and still not be selected. Crawler access only solves the first technical gate: can the system reach the evidence at all?
What I would do in Cloudflare today
- Open your domain in Cloudflare and go to Security Settings → Configure AI bot policies.
- Keep Search on Allow (do not block) if AI visibility matters to you.
- Keep Agent allowed unless you have a concrete security or operational reason to block user-triggered fetching.
- For a new or little-known brand, keep Training allowed while you build recognition.
- For an established brand, test no-search recognition first. If you are already well represented in model knowledge, decide whether the incremental training access is still worth it for you.
- Check AI Crawl Control to see which crawlers are actually requesting your pages. Cloudflare lets you monitor crawler activity and set crawler-specific policies.
Also check your robots.txt. Cloudflare notes that robots.txt is a preference signal, not an enforcement mechanism. Its AI Crawl Control can enforce crawler blocks at the network layer. Cloudflare's AI Crawl Control documentation explains the difference.
Crawler settings are not the strategy
Once the bots can reach you, the harder work starts. You need pages that answer the actual questions people ask, contain facts worth retrieving, and make the connection between the buyer's need and your product easy to verify.
That is where we usually find the biggest gains. At Aiso, we use real AI conversations and fan-out searches to find the questions where a brand is missing, then work backward to the content and evidence the assistant needs.
So the practical rule is simple. Do not block the route you are trying to win. For a new site, I would allow all three. For an established site, keep Search and Agent open, test whether you are already in the weights, then make a deliberate choice on Training rather than treating every AI bot as the same thing.