Cloudflare AI Bot Blocking: Is It Stopping AI Search Bots?

Cloudflare AI bot blocking refers to the Cloudflare settings that refuse or challenge AI crawlers before they reach your website, whatever your robots.txt says. Cloudflare may be blocking AI bots if you use its AI bot policies, AI Crawl Control, Bot Fight Mode or your own firewall rules, so check each one and allow the search bots you want.

• Cloudflare's older Block AI bots setting was deprecated on 15 September 2026, replaced by AI bot policies with 3 presets: Search, Agent and Training (Cloudflare's AI bot policy documentation).
• From 15 September 2026, new domains block Training and Agent bots on pages with ads by default, while Search stays allowed (Cloudflare's AI bot policy documentation).
• AI Crawl Control can answer a blocked crawler with a 403 or 402 status and counts each crawler's robots.txt violations (Cloudflare's guide to managing AI crawlers).
• Bot Fight Mode cannot be skipped by any of 3 firewall actions, Skip, Bypass and Allow, although IP Access rules are an exception (Cloudflare's Bot Fight Mode documentation).

Where can Cloudflare block AI bots?

Cloudflare can block AI bots in four places: the AI bot policies, AI Crawl Control, Bot Fight Mode or Super Bot Fight Mode, and custom firewall rules.

The AI bot policies are the broad switch. They sit under Security Settings as Configure AI bot policies, and they replace the older Block AI bots toggle, which Cloudflare deprecated on 15 September 2026. Each policy acts on a whole category of bots at once, so one choice can affect several AI companies.

AI Crawl Control is the detailed view. It lists the AI crawlers visiting your domain with an action for each, and blocking one there creates or updates a WAF custom rule on your zone. Bot Fight Mode and Super Bot Fight Mode are general bot defences that challenge traffic matching known bot patterns, and they are not designed with AI search in mind.

The fourth place is easy to forget: rules you or a developer wrote yourselves. A custom rule that challenges unknown user agents, blocks whole countries or limits request rates can catch an AI search bot as easily as a scraper. If an AI bot is refused and none of Cloudflare's AI settings explains it, look through your custom rules next.

What do Cloudflare's AI bot policies block by default?

For domains added from 15 September 2026, Cloudflare blocks Training and Agent bots on pages that show ads and leaves Search allowed. Cloudflare's page does not say that older domains change, so read your own settings.

Cloudflare sorts AI bots by behaviour into three policies. Search covers crawlers that collect or index content to answer questions later, Agent covers automated activity acting in real time for a person, such as chat fetch bots and browser-use agents, and Training covers crawlers that take content to train or fine-tune a model. Each policy can be set to Block on all pages, Block on pages with ads, or Allow, and Cloudflare says each blocking option applies to verified bots classified with that behaviour as well as unverified bots that behave the same way.

The Agent policy deserves attention from any business that wants AI visibility. Chat fetch bots are how assistants open a page a person has asked about, so blocking Agent can stop a customer's assistant reading your price list or opening hours even when your search visibility is fine. On a website with ads, the new default does exactly that on the ad-carrying pages.

Mixed-purpose crawlers need a careful look. Cloudflare's page places crawlers used for both training and search under Training, and its note on the new defaults says every configuration that blocks training also blocks them, while its note on the older toggle says that toggle excluded them. If a crawler you rely on for search is mixed-purpose, check its status in AI Crawl Control rather than assuming.

How is a Cloudflare block different from robots.txt?

A Cloudflare block is enforced at the network edge, so the bot never receives your page, while robots.txt is a request that well-behaved bots choose to follow.

Robots.txt works only through the crawler's cooperation. Cloudflare's own documentation says robots.txt compliance is voluntary, which is why it offers firewall-level blocking for owners who want enforcement. The reverse also holds: a perfectly written robots.txt that welcomes OAI-SearchBot and Claude-SearchBot does nothing if Cloudflare refuses them before the request reaches your server.

Cloudflare can also change your robots.txt. Its managed robots.txt setting prepends Cloudflare's own lines to your file, including a Content-Signal line such as search=yes, ai-train=no and Disallow rules for known AI crawlers such as GPTBot, ClaudeBot and Google-Extended. If your robots.txt contains lines you never wrote, between comments marked as Cloudflare managed content, this setting is the reason.

The two layers should agree. Decide which bots you want using the principles in our guide to robots.txt for AI bots, write those choices into robots.txt, and then make sure Cloudflare enforces the same choices rather than stricter ones. A mismatch in either direction leaves engines confused about your intentions or unable to read you at all.

How do you test whether Cloudflare is blocking an AI bot?

Test from both sides: request your pages as each AI bot from outside, then read Cloudflare's own event logs to see what happened to the real bots.

The outside test shows what a bot receives. You can scan your website with the AI Search Visibility Checker, which checks robots.txt for each engine's search bot and asks for your homepage as each bot to catch firewalls that turn them away. If a bot gets an error or a challenge page instead of your homepage, a rule is refusing that user agent.

The inside view confirms the cause. In the Cloudflare dashboard, AI Crawl Control lists the AI crawlers seen on your domain with the action applied to each, and requests challenged by Bot Fight Mode appear under Security, Analytics, Events with Bot Fight Mode in the Service field. Look for the bot's name, the action taken and the rule responsible.

Keep in mind that the two tests can differ. An outside request using a bot's user agent does not come from the bot's published addresses, so Cloudflare does not treat it as a verified bot, and a rule based on verified identity may treat the real bot differently. A refusal in the outside test still points to a real problem, and Cloudflare's logs show whether the genuine bot meets the same fate.

How do you let AI search bots through Cloudflare?

Let AI search bots through by setting the Search policy to Allow, allowing the named crawlers in AI Crawl Control, and making sure no bot defence challenges them.

Start with the broad settings. Under Security Settings, open Configure AI bot policies and set Search to Allow, then decide on Agent: allowing it lets assistants read pages for people who ask about you. In AI Crawl Control, check that OAI-SearchBot, Claude-SearchBot and PerplexityBot show Allow, and remember that a Block there is a WAF custom rule you can edit.

Custom allow rules work best when they match identity, not just a name. Perplexity's crawler documentation gives a Cloudflare example: a custom rule matching a user agent containing PerplexityBot or Perplexity-User and a source address within Perplexity's published IP ranges, with the action set to Allow. OpenAI publishes its ranges the same way, while Anthropic does not publish IP ranges and asks owners to manage its bots through robots.txt.

Bot Fight Mode needs separate treatment. Cloudflare's documentation says Skip, Bypass and Allow actions in custom rules have no effect on it, so an allow rule will not rescue a bot it challenges. The documented ways round it are an IP Access rule, which Cloudflare says disables Bot Fight Mode when present, moving to Super Bot Fight Mode, which supports Skip rules, or turning Bot Fight Mode off under Security Settings with the Bot traffic filter.

Can other firewalls and hosts block AI bots as well?

Other firewalls, hosts and security plugins can block AI bots in the same way, so the Cloudflare checks apply to any service that sits in front of your website.

Hosting companies often run bot protection across whole servers, and security plugins for website platforms limit or challenge visitors they do not recognise. Neither tends to announce which crawlers it refuses, and an AI search bot is exactly the kind of unfamiliar visitor they are built to slow down. The symptom is the same as with Cloudflare: the website looks normal to people while AI engines cannot read it.

The fix follows the same pattern everywhere. Perplexity's documentation also describes AWS WAF, where owners build IP sets from the published ranges, combine them with user agent matches in Allow rules and give those rules priority over blocking rules. Whatever the tool, allow the search bots by user agent and verified address, and test again after the change while watching your logs, since Perplexity notes that firewall changes can take time to propagate.

Questions owners ask about Cloudflare and AI bots

Did Cloudflare turn on AI bot blocking without my knowledge?

It may have for a new domain. From 15 September 2026, new domains block Training and Agent bots on pages with ads by default, with Search allowed. Cloudflare said all customers could opt out of the new defaults before that date, so check Configure AI bot policies to see your current settings.

Does blocking AI training bots in Cloudflare stop ChatGPT search?

Not if only training is blocked and OAI-SearchBot is allowed. Check mixed-purpose crawlers in AI Crawl Control, because Cloudflare files them under Training and training blocks can include them.

Why does robots.txt say yes but the AI bot is still blocked?

Because Cloudflare decides at the edge whether to serve the page, before robots.txt is consulted. Look in AI Crawl Control and your security events for the rule that refused the bot.

Can I charge AI crawlers instead of blocking them?

Cloudflare offers a Charge action in AI Crawl Control, but its documentation describes pay per crawl as a private beta. Cloudflare also notes that pay per crawl runs after bot solutions, so the older Block AI Bots setting had to be off for it to work.

Will allowing AI bots in Cloudflare let scrapers in?

Not if you allow by identity. Matching each bot's user agent together with its published IP ranges lets the genuine crawler in while a scraper borrowing the name from another address still meets your other defences.

Check your Cloudflare settings in fifteen minutes

Spend fifteen minutes in the Cloudflare dashboard checking the four places that can block AI bots, then test the result from outside.

Open Security Settings and read your AI bot policies: Search should be Allow, and Agent should be Allow unless you have a clear reason. Open AI Crawl Control and confirm OAI-SearchBot, Claude-SearchBot and PerplexityBot show Allow, then check whether Bot Fight Mode is on. Last, read your WAF custom rules for anything that challenges unknown bots, user agents or whole countries.

Finally, run the AI Search Visibility Checker again and compare the result with Cloudflare's event logs. Every search bot should receive a normal homepage, and any refusal should trace back to a rule you can name and change. Repeat the check whenever Cloudflare announces new defaults or you change plan, because a single updated setting can undo the rest of your work.