Block AI Training Bots? What GPTBot and ClaudeBot Blocks Do

Whether to block AI training bots is a business decision rather than a search one: blocking GPTBot, ClaudeBot or Google-Extended keeps your future content out of model training without removing your website from ChatGPT, Claude or Google search answers. Most service businesses and shops gain little from blocking, while publishers whose writing is the product often choose to.

• OpenAI says disallowing GPTBot signals that content should not be used for training, and that its 4 bots each follow their own independent robots.txt setting (OpenAI's crawler documentation).
• Anthropic says restricting ClaudeBot signals that future materials should be excluded from training, and it supports a gentler Crawl-delay: 1 rule (Anthropic's help page on its crawlers).
• Google-Extended has 0 effect on inclusion or ranking in Google Search, but it also controls grounding in Gemini Apps and Vertex AI (Google's list of common crawlers).
• Cloudflare's managed robots.txt example disallows 8 AI crawlers, including GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended (Cloudflare's managed robots.txt documentation).

What does blocking an AI training bot actually do?

Blocking an AI training bot tells that company not to use your pages to train its future models. It is a request in robots.txt that well-behaved crawlers honour, not a technical barrier.

OpenAI describes a GPTBot disallow as a signal that your content should not be used to train its generative AI foundation models. Anthropic uses similar language for ClaudeBot, saying a restriction signals that your future materials should be excluded from its training datasets. Both companies state that their bots respect robots.txt, which is why a single line is usually enough for them.

The key word in Anthropic's wording is future. A block applies to what the bot collects after it reads your new file, and neither company says it removes anything from models that already exist. Treat the decision as a setting for the years ahead rather than a way to undo the past.

Robots.txt also depends on the crawler choosing to comply. Cloudflare's documentation says plainly that robots.txt compliance is voluntary, so the file states your preference without forcing it. The large AI companies publish their bots and honour the file, but a scraper that ignores robots.txt will ignore your training rules too, and only a firewall can stop it.

What does blocking a training bot not change?

Blocking a training bot does not change whether AI search engines can find, read and cite your pages today. Each company runs a separate search bot with its own rule.

OpenAI states that its bot settings are independent, so a site can disallow GPTBot while still allowing OAI-SearchBot, the bot that decides whether pages appear in ChatGPT search answers. Anthropic separates ClaudeBot from Claude-SearchBot and Claude-User in the same way. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal, so AI Overviews and AI Mode, which are part of Google Search, are not touched by it.

Blocking also does not stop models learning about your business from elsewhere. Directories, review sites, press coverage and forum threads that mention you make their own robots.txt choices, and those pages may still be collected. A training block covers only what the company's bot gathers from your own website.

The mistake to avoid runs the other way. Copied lists of AI bots to block often include search bots such as OAI-SearchBot, Claude-SearchBot or PerplexityBot alongside the training ones, and pasting one in removes you from those engines' answers. Block training by naming training bots only.

Why might a business block AI training bots?

A business might block AI training bots when its written content is the product it sells, or when it wants control over how its original work is reused.

Publishers, course creators, researchers and firms that sell reports or detailed guides have the clearest case. If customers pay for your writing, a model that has absorbed it may answer the same questions without sending anyone to you, and a training block is a reasonable way to state that you do not consent. The same applies to original photography, recipes or technical documentation that took real effort to produce.

Server load is sometimes given as a reason, but blocking is a heavy answer to it. Anthropic supports the Crawl-delay rule for ClaudeBot, with an example value of 1, which slows crawling without refusing it. OpenAI also says that when a site allows both GPTBot and OAI-SearchBot it may use one crawl for both purposes, so allowing both need not double the traffic.

Why might a business allow AI training bots?

A business might allow AI training bots so that future models learn its name, services and expertise directly from its own website rather than only from what others say.

For a plumber, an accountant or a shop, the website is marketing rather than the product. Its job is to make the business known, and a model that has read your service pages is more likely to describe you accurately when someone asks about your field or your area. There is little commercial content to protect, because the pages exist to be read.

Allowing is also the simpler choice to maintain. With no training rules, your robots.txt needs only a * group that closes private paths, and there is no risk of a training block being copied into the wrong group later. Every line you add is a line someone must understand when they next edit the file.

The honest position is that nobody outside the AI companies can measure how much training on your pages changes later answers. Allowing is a bet that being well represented in future models helps, and blocking is a choice to keep control of your work. Neither decision affects today's AI search citations, so make it on the merits for your business.

Which training bots exist, and which lines block them?

The main training controls are GPTBot for OpenAI, ClaudeBot for Anthropic and Google-Extended for Google; each is blocked with a robots.txt group naming it and a Disallow: / rule.

Company	Training control	Separate search bot that stays allowed	Notes from the company's documentation
OpenAI	GPTBot	OAI-SearchBot	Settings are independent; IP ranges published
Anthropic	ClaudeBot	Claude-SearchBot	Repeat the rule on every subdomain; Crawl-delay supported
Google	Google-Extended token	Googlebot	No separate crawler; also covers grounding in Gemini Apps and Vertex AI
Microsoft	noarchive or nocache robots meta tags	Bingbot	Controls announced in 2023 for Bing Chat answers and training

Microsoft works differently from the others. Microsoft's 2023 announcement on the Bing Webmaster Blog, written for what was then Bing Chat, introduced two robots meta tags rather than a robots.txt token: noarchive keeps a page out of chat answers and out of training, while nocache limits both uses to the address, snippet and title. Microsoft said pages with either tag still appear in Bing search results, but noarchive removes them from its chat answers, so a business that wants Copilot citations should not use it.

For the robots.txt companies, one group can cover several bots at once. The example below blocks training by the three main companies and leaves every search bot following the * group, which keeps your private paths closed to all of them. Each bot reads only the group that names it, so the training rule never reaches OAI-SearchBot or Claude-SearchBot.

User-agent: GPTBot  # OpenAI training.
User-agent: ClaudeBot  # Anthropic training.
User-agent: Google-Extended  # Gemini training and Gemini grounding.
Disallow: /
User-agent: *  # Search bots and every other crawler follow this group.
Disallow: /admin/

Other crawlers appear in published lists, including CCBot, Applebot-Extended, Bytespider, Amazonbot and meta-externalagent, all of which Cloudflare's managed robots.txt example disallows. Add any of them only after reading the company's own description, because a name on a list does not tell you what the bot is for. Our guide to robots.txt for AI bots explains how groups and fallbacks work if you add more.

After any change, confirm the search bots still get in. You can scan your website with the AI Search Visibility Checker, which checks robots.txt for each engine's search bot, asks for your homepage as each bot, and lists training bots separately so a training block never hides a search problem. If a search bot is refused, fix that first, because it costs citations straight away.

Is Google-Extended only about training?

Google-Extended is not only about training: it also controls whether Google may use your content to ground answers in Gemini Apps and Grounding with Google Search on Vertex AI.

Google's crawler page says Google-Extended governs whether crawled content may train future Gemini models and whether it may be provided to those models at prompt time for grounding. Grounding means the model reads current pages while answering, much like a search engine, so blocking Google-Extended can mean Gemini Apps answer questions about your field without your pages. That is a real visibility cost, even though Google Search itself is unaffected.

Google-Extended is also a token rather than a crawler. Google says it has no separate user agent string, and crawling is done by Google's existing user agents, so you will never see it in your server logs. Block it in robots.txt if you object to Gemini training and grounding, but weigh the grounding effect as well as the training one before you do.

Is there a middle ground between blocking and allowing?

There is a middle ground: block training on some sections only, slow a bot down instead of refusing it, or state a preference through a CDN's managed settings.

Partial blocks suit businesses with a mix of marketing pages and paid or original material. A group for GPTBot with Disallow: /reports/ keeps one directory out of training while the rest of the site stays open, and the same pattern works for ClaudeBot. Under the robots.txt standard the longest matching rule wins, so a narrow Disallow beats a general Allow.

Content delivery networks now offer managed settings too. Cloudflare's managed robots.txt can add a Content-Signal line, such as search=yes, ai-train=no, together with Disallow rules for known AI crawlers, and it places these before your own file's contents. Cloudflare notes that Google Search Console may report its newer lines as syntax it does not understand, and says it has seen no effect on crawling or SEO from those reports.

Questions owners ask about training bots

Will blocking GPTBot or ClaudeBot hurt my search rankings?

No. Neither bot is used by Google or Bing to rank pages, and OpenAI and Anthropic each run a separate search bot for their own answers. Keep those search bots allowed and your visibility is unchanged.

Does blocking training bots remove my content from existing models?

Not according to the companies' own pages. Anthropic describes a ClaudeBot block as excluding future materials, and OpenAI describes a GPTBot block as a signal about training use. Treat a block as applying from now on.

Do all AI bots respect a training block?

The major companies say their bots honour robots.txt, but compliance is voluntary. Bots that ignore the file can only be stopped by a firewall or CDN, and those tools must be set so they do not also block the search bots you want.

Should a local service business block training bots?

Usually there is little reason to. A local business website exists to be read and to make the business known, and allowing training is the simpler setup. Either choice is fine as long as the search bots stay allowed.

Decide on AI training bots and write it down

Make the training decision once, deliberately, and record it so the next person to edit robots.txt does not undo it by accident.

Start by asking whether your written content is something you sell or something that sells you. If it is the product, add a group blocking GPTBot, ClaudeBot and Google-Extended, and consider the Gemini grounding cost before including the last one. If it is marketing, leave training allowed and keep robots.txt to a short * group, or block only the sections that hold paid or original material.

Then write the decision, the date and the reason in a comment at the top of robots.txt or in your website notes. Check the search bots after every change with the checker, because a training block that drifts into a search bot's group costs you citations straight away. Review the choice once a year, or sooner if an AI company changes its bots or publishes new controls.