GPTBot vs OAI-SearchBot: OpenAI's Training and Search Bots

GPTBot vs OAI-SearchBot is the choice between OpenAI's two separate crawlers: GPTBot collects pages that may be used to train OpenAI's models, while OAI-SearchBot decides whether your pages can appear and be cited in ChatGPT search answers. Blocking GPTBot keeps future content out of training but does not remove a website from ChatGPT search.

• OpenAI lists 4 bots, and states that each robots.txt setting is independent, so a site can allow OAI-SearchBot while disallowing GPTBot (OpenAI's crawler documentation).
• If a site allows both GPTBot and OAI-SearchBot, OpenAI says it may use the results of 1 crawl for both purposes to avoid crawling twice (OpenAI's crawler documentation).
• RFC 9309 says a crawler follows the 1 group that names it and uses the * group only when none does (RFC 9309, the Robots Exclusion Protocol).
• From 15 September 2026, Cloudflare's defaults for new domains block Training and Agent bots on pages with ads while Search stays allowed (Cloudflare's AI bot policy documentation).

What does GPTBot do?

GPTBot is OpenAI's training crawler: it collects web content that may be used to train OpenAI's generative AI foundation models. Disallowing it signals that your content should not be used for that training.

GPTBot identifies itself with a user agent containing GPTBot/1.4 and a link to openai.com/gptbot, and OpenAI publishes its IP ranges as a JSON file at openai.com/gptbot.json. Those two details let a server or firewall confirm that a visitor calling itself GPTBot really comes from OpenAI. Anything claiming the name from another address is an impostor and can be treated like any other unknown bot.

What GPTBot gathers affects what future models know, not what ChatGPT finds today. Training happens long before anyone asks a question, so a page collected now may shape a model released many months later, and the model will not link back to it. That is why the GPTBot decision belongs with your thinking about content rights and reputation rather than with your search plans.

Treat blocking GPTBot as forward-looking. The robots.txt rule is a signal about what OpenAI may use for training, and OpenAI's page does not say it changes models that have already been trained. A business that changes its mind later can simply remove the rule, and GPTBot will read the new file on its next visit.

What does OAI-SearchBot do?

OAI-SearchBot is OpenAI's search crawler: it finds and indexes pages so they can be surfaced, linked and cited in ChatGPT's search features. It is the bot that decides whether ChatGPT can cite you.

OpenAI's documentation is direct about the consequence of blocking it. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and OpenAI recommends allowing the bot and its published IP ranges, listed at openai.com/searchbot.json. Its user agent contains OAI-SearchBot/1.4 and a link to openai.com/searchbot.

Changes to OAI-SearchBot's rules arrive quickly. OpenAI says search-related robots.txt changes can take about 24 hours to take effect, which matches the standard's advice that crawlers should not rely on a cached robots.txt for more than a day. If you discover you have blocked it, the fix shows within a day rather than weeks.

Allowing OAI-SearchBot does not guarantee citations, because ChatGPT still chooses which pages answer each question best. It only makes your pages eligible, and whether they are quoted then depends on whether they answer the question clearly, back it with specifics and come from a business that other websites recognise. Without that eligibility, though, none of the rest matters for ChatGPT.

Where does ChatGPT-User fit in?

ChatGPT-User is a third OpenAI bot that visits a page when a person's request in ChatGPT needs it. It is not used for automatic crawling and does not decide what appears in ChatGPT search.

A typical ChatGPT-User visit happens when someone pastes your page address into ChatGPT, or when ChatGPT or a custom GPT opens a page while answering. OpenAI says that because users start these visits, robots.txt rules may not apply to them. For control over search appearance and automatic crawling, OpenAI points owners to OAI-SearchBot instead.

OpenAI's page lists a fourth bot, OAI-AdsBot, which checks landing pages submitted as ads in ChatGPT. It only visits pages that have been submitted as ads, and OpenAI says its data is not used to train generative AI foundation models. If you do not advertise in ChatGPT you will not see it, and it has nothing to do with organic citations.

Bot	Job	Effect of a robots.txt block	Published IP ranges
GPTBot	Collects content that may train OpenAI's models	Content is not used for training	openai.com/gptbot.json
OAI-SearchBot	Indexes pages for ChatGPT search	Pages are not shown in ChatGPT search answers	openai.com/searchbot.json
ChatGPT-User	Visits pages for a user's request	Robots.txt rules may not apply	openai.com/chatgpt-user.json
OAI-AdsBot	Checks pages submitted as ChatGPT ads	Only visits submitted ad pages	openai.com/adsbot.json

What robots.txt lines control each OpenAI bot?

Each OpenAI bot reads the group in robots.txt that names it, and falls back to the * group when there is none. Most businesses need one line about GPTBot at most.

The most common setup for a business that wants ChatGPT citations but prefers to keep its writing out of training looks like this. GPTBot gets its own group with a full disallow, while OAI-SearchBot and ChatGPT-User have no group of their own and therefore follow the * group. Under RFC 9309 that means they may read everything except the private paths.

User-agent: GPTBot  # Training opt-out only.
Disallow: /
User-agent: *  # OAI-SearchBot and ChatGPT-User follow this group.
Disallow: /admin/  # Private paths stay closed to every bot.

To allow training too, delete the GPTBot group and leave the rest unchanged. To refuse ChatGPT search, which few businesses want, you would add a group for OAI-SearchBot with Disallow: /, and pages would leave ChatGPT search answers within about a day. Avoid adding a group for OAI-SearchBot that only says Allow: /, because a named group replaces the * group for that bot and it would then ignore your private paths.

The same split exists across the industry, so check the matching lines when you set OpenAI's. Anthropic separates ClaudeBot for training from Claude-SearchBot for search, and Google separates the Google-Extended token from Googlebot. The wider choices, including Perplexity, Google and Copilot, are set out in our guide to robots.txt for AI bots.

Why does blocking GPTBot not hide a website from ChatGPT?

Blocking GPTBot does not hide a website from ChatGPT because ChatGPT finds pages for its answers through OAI-SearchBot, which reads its own robots.txt group.

The mistake is easy to make because both bots belong to OpenAI and neither name says what the bot is for. Today OpenAI describes the two bots as independent: one for training, one for search. A website that blocks only GPTBot stays fully eligible for ChatGPT search answers and citations.

There is a second reason a business may still appear after blocking GPTBot. ChatGPT can mention your business from what other websites say about you, such as directories, reviews and press coverage, and those pages make their own choices about bots. Your robots.txt only controls what OpenAI's bots collect from your own website.

If your real aim is to stay out of ChatGPT answers altogether, GPTBot is the wrong lever. You would need to disallow OAI-SearchBot, accept that ChatGPT-User may still fetch a page a person asks about, and accept that others can still describe you. For almost every business that sells to the public, that trade is not worth making.

How can you tell if OAI-SearchBot is blocked by accident?

OAI-SearchBot is usually blocked by accident in one of three places: a copied list of AI bots in robots.txt, a blanket disallow, or a firewall that challenges unfamiliar crawlers.

Copied lists are the first suspect. Lists of "AI bots to block" shared online often mix training bots with search bots, and pasting one in can disallow OAI-SearchBot alongside GPTBot in a single edit. A blanket rule does the same thing more quietly: a file with User-agent: * and Disallow: /, followed by a group that allows only Googlebot, shuts out OAI-SearchBot even though it never names it.

Firewalls are the second gate. Cloudflare now sorts AI bots into Search, Agent and Training policies, and its documentation says each blocking option applies to verified bots classified with that behaviour, so a Search block would reach any bot it classifies as search. A setting chosen to stop training can therefore look harmless and still turn away a search crawler if the wrong policy is selected.

The reliable test is to ask for your pages the way OAI-SearchBot does. You can scan your website with the AI Search Visibility Checker, which reads your robots.txt for each engine's search bot, requests your homepage as each bot to catch firewalls that refuse them, and lists training bots such as GPTBot separately. If OAI-SearchBot is refused, allow it in the firewall by its user agent together with OpenAI's published IP ranges, then test again the next day.

Questions owners ask about OpenAI's bots

Can I allow ChatGPT search but block training?

Yes. Add a robots.txt group that disallows GPTBot and leave OAI-SearchBot under your * group or allowed by name. OpenAI states the two settings are independent.

Does blocking GPTBot hurt my Google rankings?

No. GPTBot is OpenAI's crawler and Google does not read its group in your robots.txt. Googlebot follows its own group, or the * group if it has none.

Will ChatGPT stop knowing about my business if I block GPTBot?

Not necessarily. OpenAI does not say the GPTBot rule changes models already trained, and ChatGPT can still find you through OAI-SearchBot and through what other websites say about you.

How do I know a visitor is really OAI-SearchBot?

Check both its user agent and its address. OpenAI publishes the IP ranges for OAI-SearchBot, GPTBot and ChatGPT-User as JSON files, so a request using the name from an address outside those ranges is not from OpenAI.

Should I block ChatGPT-User?

Usually not. ChatGPT-User visits because a person asked ChatGPT about your page, often a potential customer, and OpenAI says robots.txt rules may not apply to it anyway.

Set your OpenAI bot rules today

Open your robots.txt now, decide your GPTBot answer, and make sure nothing anywhere stops OAI-SearchBot.

Read the file from top to bottom and look for any group that names OAI-SearchBot, any blanket Disallow: / under User-agent: *, and any copied list of AI bots. Remove OAI-SearchBot from anything that blocks it, then add or remove a GPTBot group to match your training decision, and write that decision down with the date. If you keep a named group for OpenAI's bots, copy your private paths into it so they stay closed.

Then check the firewall side, because a robots.txt that says yes is only half the answer. Look at your CDN's AI bot settings and your security plugin's bot rules, run the checker to request your homepage as OAI-SearchBot, and fix any refusal at its source. Repeat the test whenever you change host, theme or security settings, since a new default can undo a careful robots.txt overnight.