Robots.txt for AI Bots in 2026: Which to Allow or Block

Robots.txt for AI bots is the set of rules telling each AI company's crawlers what they may fetch from your website. For a business that wants to be cited, the right setup in 2026 is to allow every AI search bot and user fetcher, keep private paths closed, and treat training bots as a separate choice.

This article is part of our complete guide: AI Search Optimization.

• RFC 9309, published in September 2022, says a crawler obeys the group naming it and only falls back to the * group when no group does (RFC 9309, the Robots Exclusion Protocol).
• OpenAI runs 4 bots with separate rules; pages that opt out of OAI-SearchBot are not shown in ChatGPT search answers, and search changes take about 24 hours (OpenAI's crawler documentation).
• Anthropic runs 3 bots, ClaudeBot, Claude-SearchBot and Claude-User, which honour robots.txt, and asks owners to repeat their rules on every subdomain (Anthropic's help page on its crawlers).
• Perplexity says PerplexityBot is not used to crawl content for AI foundation models and that robots.txt changes can take up to 24 hours (Perplexity's crawler documentation).
• Google-Extended has 0 effect on inclusion or ranking in Google Search; it controls Gemini training and grounding only (Google's list of common crawlers).

Which kinds of AI bots visit a website?

AI bots come in three kinds: search bots that index pages for answers, user fetchers that open a page someone asked about, and training bots collecting text for future models.

The difference matters because each kind reads its own line in robots.txt and has a different effect on your business. A search bot decides whether your pages can be found and cited when someone asks ChatGPT, Claude or Perplexity a question today. A user fetcher visits a single page in the moment, usually because a person pasted your address or the assistant needed one more source. A training bot gathers pages that may shape what a model knows months from now, and blocking it has no effect on today's answers.

Company	Search bot	User fetcher	Training bot or token
OpenAI (ChatGPT)	OAI-SearchBot	ChatGPT-User	GPTBot
Anthropic (Claude)	Claude-SearchBot	Claude-User	ClaudeBot
Perplexity	PerplexityBot	Perplexity-User	None listed; PerplexityBot is not used for model training
Google (AI Overviews, AI Mode)	Googlebot	Not a separate bot	Google-Extended, a robots.txt token only
Microsoft (Copilot)	Bingbot	Not documented separately	Robots meta tags rather than a robots.txt token

Two rows work differently from the rest. Google's AI Overviews and AI Mode are part of Google Search, so they rely on Googlebot, and Google-Extended has no user agent of its own: Google's page says crawling is done by its existing user agents and the token exists only for robots.txt. Microsoft's Copilot answers draw on Bing's index, which Bingbot builds, and Microsoft's 2023 controls over use of content in its chat answers and model training were the noarchive and nocache robots meta tags rather than a robots.txt line.

Which AI bots should a business allow?

Allow every AI search bot and every user fetcher, because those are the bots that decide whether you are cited today. Decide on training bots separately; blocking them does not remove you from any AI search answer.

The search bots are OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, apart from navigational links, and recommends allowing the bot. Anthropic says disabling Claude-SearchBot prevents its system indexing your content for search and may reduce your visibility in user search results. Perplexity says owners can allow PerplexityBot in robots.txt so that their pages appear in its results.

The user fetchers are ChatGPT-User, Claude-User and Perplexity-User. They matter when a customer asks an assistant about a specific page, such as your price list or your returns policy, and expects an answer drawn from it. Anthropic says disabling Claude-User prevents its system retrieving your content in response to a user query, so blocking it only stops Claude reading you on behalf of your own customers.

The training bots are GPTBot, ClaudeBot and the Google-Extended token, plus others from companies outside the AI search engines. Whether to allow them is a question about your content and your business, not about search, and either answer is reasonable as long as the search bots stay open. Three narrower questions grow out of this choice and each has its own guide: the difference between GPTBot and OAI-SearchBot, whether a business should block AI training bots such as GPTBot and ClaudeBot, and how to tell whether Cloudflare is blocking AI bots from your website.

What should a robots.txt for AI bots look like?

The robots.txt that suits most businesses is short: one group for every bot that keeps private paths closed, plus an optional group for training bots you choose to refuse.

User-agent: *  # Every bot without its own group, including all AI search bots and fetchers.
Disallow: /admin/
Disallow: /checkout/
User-agent: GPTBot  # Optional training opt-out for OpenAI.
User-agent: ClaudeBot  # Optional training opt-out for Anthropic.
User-agent: Google-Extended  # Optional opt-out for Gemini training and Gemini grounding.
Disallow: /
Sitemap: https://www.example.co.uk/sitemap.xml

Nothing in this file names the search bots, and that is deliberate. Under RFC 9309 a bot with no group of its own follows the * group, so OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot may read everything except the admin and checkout paths. If you want to allow training as well, delete the three training lines and their Disallow line, and the whole file becomes the first three lines plus the sitemap.

The training group shows how several bots can share one set of rules. RFC 9309 lets a group start with more than one user-agent line, so GPTBot, ClaudeBot and Google-Extended all receive the single Disallow rule beneath them. Bots ignore the comments after each hash sign, and the standard lets crawlers read a Sitemap line without it ending the group around it.

Keep the file at the root of each host, because the standard places robots.txt at the top-level path of each service. A shop at shop.example.co.uk needs its own file, which is why Anthropic asks owners to repeat their rules for every subdomain. A rule in the main domain's file does nothing for a booking system or blog on a separate subdomain.

How do AI bots read robots.txt rules?

AI bots read robots.txt the way RFC 9309 describes: each bot finds the group naming it, ignores every other group, and applies the most specific matching rule to each address.

Group selection is the rule that catches most owners out. If a file has a * group with ten careful Disallow lines and then a short group for OAI-SearchBot with only Allow: /, OpenAI's search bot follows its own group and ignores all ten lines. The standard says matching of the bot's name is case-insensitive and that groups naming the same bot are merged, so two separate GPTBot groups in one file act as one. When you add a group for any named bot, copy every private path into it as well.

Within a group, the longest matching rule wins, measured in characters of the path. A file with Allow: / and Disallow: /admin/ blocks /admin/login because the Disallow rule matches more of the address, and when an Allow and a Disallow rule match equally, the standard says the Allow rule should be used. Matching runs from the start of the path and the standard says it should be case-sensitive, so Disallow: /Admin/ may not protect /admin/.

Robots.txt is also not a lock. RFC 9309 states that its rules are not a form of access authorization and that listing a path in the file makes it publicly discoverable. Protect private pages with a login, and use robots.txt only to tell well-behaved crawlers where to spend their time.

What happens when robots.txt is missing, broken or out of date?

A missing robots.txt lets every bot in, while a robots.txt that returns a server error tells compliant bots to stay out of the whole website.

RFC 9309 draws the line by status code. If the file returns a 4xx code such as 404, the crawler may access any resource on the server, which for a business that wants to be cited is perfectly acceptable. If the file returns a 5xx error or the connection fails, the crawler must assume complete disallow, so an overloaded host or a misconfigured plugin can hide a whole website from AI search without any visible error.

Changes also take time to arrive. The standard says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, and OpenAI and Perplexity both describe roughly a day before a change reaches their systems. Crawlers should follow at least five redirects to find the file and must parse at least 500 kibibytes of it, so a sensible robots.txt is never too long to read.

Do user fetchers like ChatGPT-User obey robots.txt?

User fetchers obey robots.txt less consistently than search bots, because a person started the request. OpenAI and Perplexity say their fetchers may not follow it, while Anthropic says Claude-User does.

OpenAI's documentation says robots.txt rules may not apply to ChatGPT-User because these visits are initiated by users, and that ChatGPT-User is not used to decide whether content appears in ChatGPT search. Perplexity says Perplexity-User generally ignores robots.txt for the same reason, and that it is not used for general crawling or for training. Anthropic's page says its bots, Claude-User included, honour robots.txt directives.

The practical lesson is that robots.txt cannot be your only control over these visits, and in most cases you will not want to stop them. A fetcher visit means a real person is asking an assistant about your page right now, which is often a customer comparing you with someone else. If you must refuse these visits, it has to be done at the firewall, and the next section explains how that same firewall can block the bots you want by accident.

Can a firewall block AI bots that robots.txt allows?

A firewall, content delivery network or security plugin can refuse AI bots that robots.txt welcomes, and this second gate is where most hidden blocks sit.

Robots.txt is read before crawling, but whether a page is actually served is decided by your server and anything in front of it. Content delivery networks offer settings that block or challenge AI crawlers, hosts apply bot protection to shared servers, and security plugins limit visitors they do not recognise. Any of these can answer OAI-SearchBot or Claude-SearchBot with an error or a challenge page, and the engine then has nothing to read even though your robots.txt says yes.

The only reliable test is to request your pages the way each bot does. You can scan your website with the AI Search Visibility Checker, which checks robots.txt for each engine's search bot, asks for your homepage as each bot to catch firewalls that turn them away, and lists training bots separately so you can see your training choices without them affecting the result. The checker changes nothing on your website, so it is safe to run before and after every change.

When you find a firewall block, fix it in the tool that made it rather than in robots.txt. OpenAI and Perplexity publish the IP ranges of their bots as JSON files so that firewalls can allow them by verified address, and Perplexity's documentation gives sample allow rules for common firewalls. Anthropic does not publish IP ranges and warns that blocking its bots by IP address can stop them reading your robots.txt, so manage Anthropic's bots with robots.txt rules.

Which robots.txt mistakes hide a website from AI search?

Most hidden blocks come from a few repeat mistakes: a blanket disallow, a named group missing private paths, the wrong bot name, or a file nobody has read for years.

Mistake	What happens	Fix
User-agent: * with Disallow: / left from a staging site	Every compliant bot, search and training alike, is told to stay out	Replace it with Disallow lines for private paths only
Blocking GPTBot to keep out of ChatGPT	ChatGPT search still reads you through OAI-SearchBot, and only training is affected	Decide which result you want and name the right bot
Blocking OAI-SearchBot or Claude-SearchBot with a list copied from elsewhere	Pages drop out of ChatGPT or Claude search answers	Remove the search bot from the copied list
A named group with Allow: / only	That bot ignores every private path in the * group	Copy each private Disallow line into the named group
robots.txt returning a 5xx error	Compliant bots assume the whole site is disallowed	Fix the server or plugin so the file returns 200

Copied block lists deserve special care. Lists of AI bots shared online often mix training bots with search bots, and pasting one in can remove you from ChatGPT, Claude and Perplexity answers in a single edit. Before adding any line, check the bot's name against the company's own documentation and ask whether you are refusing training, search or a user's request.

Questions business owners ask about AI bots

Will blocking GPTBot remove my website from ChatGPT?

No. GPTBot only signals whether your content may be used to train OpenAI's models, while OAI-SearchBot controls whether your pages can be shown in ChatGPT search answers. OpenAI says the two settings are independent.

Does Google use a separate bot for AI Overviews?

No. AI Overviews and AI Mode are part of Google Search and rely on Googlebot, so blocking Googlebot would remove you from Google Search as well. Google-Extended is a robots.txt token for Gemini training and grounding and does not affect Google Search.

Should I list every AI bot by name in robots.txt?

Usually not. Bots without a group of their own follow the group, so a short file that closes private paths already lets every search bot in. Name a bot only when you want it treated differently, and remember that a named group replaces the group for that bot.

How quickly do AI engines notice a robots.txt change?

Usually within a day. OpenAI says search changes take about 24 hours, Perplexity says up to 24 hours, and RFC 9309 tells crawlers not to rely on a cached copy for more than 24 hours. A firewall change can be checked straight away by requesting your homepage as each bot.

Do I need an llms.txt file as well?

No. Robots.txt grants or refuses access, while llms.txt is an optional list of useful pages that none of the major engines documents as a way to choose sources. Get robots.txt and your firewall right first.

Check your robots.txt for AI bots this week

Spend twenty minutes this week reading your robots.txt, deciding on training bots and testing each search bot against your live website.

Start by opening robots.txt on your main domain and on every subdomain, and confirm each returns plain text rather than an error. Remove any blanket Disallow: / under User-agent: *, any copied line that names OAI-SearchBot, Claude-SearchBot or PerplexityBot, and any named group that does not repeat your private paths. Then write down your training decision for GPTBot, ClaudeBot and Google-Extended, add the lines that match it, and note the date so you know when the engines should have picked it up.

Finally, test the result from the outside, because robots.txt is only the first gate. Run the AI Search Visibility Checker to request your homepage as each engine's bot, and if any search bot is turned away, fix the setting in your firewall, CDN or security plugin and run it again the next day. Read the checker's method page to see how bot access counts towards each page's score, then repeat the check whenever your host, theme or security settings change.