How to set Cloudflare for AI search: Search, Agent and Training bots

For a new brand that welcomes training use, I would start with all three settings on Allow. For an established brand, I would first check what models already know, then review training access separately. That is my starting policy for public marketing content, not a rule for private data or every publisher.

Cloudflare describes Search as crawling or indexing content for later answers. Agent covers activity on a person's behalf. Training covers model training or fine-tuning, including mixed-purpose search-and-training crawlers. Each category has options to block everywhere, block on pages with ads, or add no block. Read Cloudflare's policy definitions and dashboard path.

The setting I would use for a new website

For public content that the business is comfortable making available to training crawlers, this is my starting point:

A new website is not necessarily a new brand. Test rather than assuming that models know nothing about it. OpenAI describes GPTBot content as material that may be used in training. Access is not proof of inclusion. Importantly, its search crawler is independent: a new site can appear through search without first entering training data. OpenAI explains these separate controls.

My reason for allowing Training on a little-known brand is to preserve that option. It is not a claim that this is necessary to earn citations, or that it produces a measured visibility increase. A business protecting valuable licensed material may reasonably choose otherwise.

For an established website, test recognition rather than age

Two years online is not a reliable threshold. An old website can still be unknown to a model, and a newly launched site can belong to a well-known company. Use our no-search brand recognition audit instead.

Use fresh sessions with search and other retrieval tools actually disabled, without saved memory or uploaded company documents. Ask the model to identify the company, then test unbranded buyer questions where the offer fits. Repeat the questions and record the model and date. A company description and a buying recommendation are different outcomes.

Consistent recognition without retrieval is evidence about recall. It does not identify which website, third-party source or development stage supplied that knowledge. It cannot certify that a particular page was in the training corpus.

If recognition is already strong, review whether you still want future training access. That decision should reflect your content and business goals, not an assumption that blocking a crawler creates referral traffic.

Blocking Training does not force a citation

OpenAI explicitly allows website owners to permit OAI-SearchBot while disallowing GPTBot. That supports a search-open, training-restricted policy for its separately controlled crawlers. See the provider documentation.

It does not erase existing model knowledge, require the assistant to search, or make it link to your website. An assistant may answer from stored knowledge, use another source, or retrieve your page without citing it. There is no verified citation-lift estimate attached to this setting.

Check mixed-purpose crawlers before blocking the Training category. Cloudflare includes crawlers used for both search and training in that category. A Training block can therefore also remove a retrieval route, even while Search says Allow. Do not assume every provider separates its bots like OpenAI does. Cloudflare documents this overlap.

Set the policy, then check actual access

1. Open the domain in Cloudflare. Go to Security Settings → Configure AI bot policies.
2. For public pages where AI discovery matters, choose Allow (do not block) for Search and Agent.
3. For a new or little-known brand that welcomes training use, also allow Training. For an established brand, run the recognition audit and review the content-use trade-off.
4. Before restricting Training, inspect the affected bots, especially mixed-purpose crawlers.
5. Review robots.txt and existing firewall or bot rules. Allow means this setting adds no block. It is not a blanket override of other protections. Keep normal security controls in place.
6. Use AI Crawl Control to inspect real crawler requests and the pages they reach. Investigate denied requests, rate limits and challenge pages rather than assuming a green toggle means successful retrieval.

Cloudflare's dashboard supports activity monitoring and individual crawler policies. A robots.txt directive expresses a preference; an enforced access rule can stop a request. Check both layers. See AI Crawl Control's monitoring and access features.

Do not count a crawler visit as a human visitor or a citation. Keep access logs, observed answer citations and qualified enquiries in separate reports. Our AI bot measurement guide explains that distinction.

Give the assistant something useful to retrieve

A query such as site:yourdomain.com pricing needs a useful pricing answer. A hotel comparison may need a precise description of parking or accessibility. Those are examples of evidence to publish, not promises that any particular engine will run those searches.

Our companion article on onsite content and AI retrieval explains why the website you control is worth checking. It does not make third-party coverage irrelevant. The useful question is what information is missing when an assistant evaluates your business.

Publish current facts with dates and a clear source. Where you have original results, explain how you measured them. Link the useful page from relevant parts of your site. A detailed answer to a real buyer question is a better starting task than another general article that repeats what is already everywhere.

At Aiso, we use real AI conversations and the searches behind them to identify those missing answers. Crawler settings keep the route open. The content still has to earn its place in the answer.