Aizii Learn · AI Search
← Back to Learn

GPTBot vs OAI-SearchBot: what's the difference

GPTBot trains models. OAI-SearchBot feeds ChatGPT search answers. That is the whole difference, and it is the difference most robots.txt files get wrong. GPTBot crawls content that may be used to train OpenAI's future foundation models. OAI-SearchBot surfaces websites in search results in ChatGPT's search features. OpenAI says each setting is independent, so you can allow one and block the other, and most brands should: allow OAI-SearchBot if you want to be found, and decide GPTBot on its own terms as a training choice. A file that names only one of them has made only half the decision.

The two bots side by side

| | GPTBot | OAI-SearchBot |

|---|---|---|

| Job | Crawls content that may be used to train future foundation models | Surfaces websites in ChatGPT search answers |

| Token in robots.txt | GPTBot | OAI-SearchBot |

| What a Disallow does | Indicates the content should not be used for training | Sites that opt out will not be shown in ChatGPT search answers |

| What an Allow does | Permits training use of crawled content | Keeps the site eligible to appear in ChatGPT search answers |

| Effect on the other bot | None. The settings are independent | None. The settings are independent |

| Full user-agent string | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot | Mozilla/5.0 (Macintosh; Intel Mac OS X 10157) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot |

| Published IP ranges | openai.com/gptbot.json | openai.com/searchbot.json |

OpenAI notes the version number in each string may change, so match on the token, GPTBot or OAI-SearchBot, not on the full string. When either bot fetches your robots.txt file itself, OpenAI may add a robots.txt marker to the string, so do not be surprised by a variant line for that one request. The version numbers also tell you something about the file's history: the two strings carry different version numbers and different browser prefixes, because they are two crawlers, not two names for one crawler.

What each one costs you

Blocking OAI-SearchBot costs ChatGPT search visibility, and OpenAI states the cost plainly: sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. That last clause matters. An opt-out is not a promise that your domain name will never be visible in ChatGPT. It is a promise that your pages will not be shown as search answers. If a customer asks ChatGPT for a product you sell, and your site is opted out, ChatGPT answers from other sites, and yours is not among them.

Blocking GPTBot costs nothing in search, and OpenAI states that too, by making the settings independent. Disallowing GPTBot indicates your content should not be used for training generative AI foundation models. It does not remove you from ChatGPT search, it does not stop ChatGPT from fetching your pages when a user asks, and it does not reach back into training data already collected. The opt-out works going forward, for future models, which is the same shape as Anthropic's wording for ClaudeBot. If your goal is to disappear from ChatGPT answers, GPTBot is the wrong token to block. If your goal is to keep your content out of future training, OAI-SearchBot is the wrong token to block.

The third bot in the room

There is a third OpenAI agent, and it confuses the comparison because its name sounds like the search crawler. ChatGPT-User fetches a page when a person asks ChatGPT or a Custom GPT a question. OpenAI says it is not used for crawling the web in an automatic fashion, that robots.txt rules may not apply to it because the fetch is user-initiated, and that it is not used to determine whether content may appear in Search. OpenAI tells owners to use OAI-SearchBot in robots.txt for managing search opt-outs and automatic crawling. So the three-way split is: OAI-SearchBot for search, GPTBot for training, ChatGPT-User for live user requests. Blocking ChatGPT-User does not control search. Allowing GPTBot does not create search visibility. Each bot answers a different question, and the file has to answer all three.

The file, written as two decisions

OpenAI gives this exact split as its example: allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep content out of training. The file below is that example, written out with the private paths repeated inside each group, because a named group stands alone and does not inherit the wildcard group's rules.

User-agent: OAI-SearchBot
Allow: /
Disallow: /checkout/
Disallow: /account/

User-agent: GPTBot
Disallow: /

If your training position is the opposite, change the GPTBot group to Allow: / and keep the search group the same. If you allow both bots, OpenAI says it may use the results from one crawl for both uses, to avoid crawling twice, which is a practical detail worth knowing: allowing both does not mean double the crawl load from OpenAI. If you block both, ChatGPT search cannot show your pages and your content is kept out of future training, and the file should say that plainly so the next person who reads it knows the visibility cost was chosen, not accidental. Whichever split you choose, write both groups. A file with one OpenAI group is a file with an undecided half.

How to tell them apart in your logs

Match on the token, not the full string, and verify the IP. A log line with OAI-SearchBot in the user-agent field is a search crawler visit, and the requesting IP should fall inside the ranges published at openai.com/searchbot.json. A log line with GPTBot is a training crawler visit, and the IP should fall inside openai.com/gptbot.json. A log line with ChatGPT-User is a user-initiated fetch, and the IP should fall inside openai.com/chatgpt-user.json. A user-agent string on its own is easy to copy, so a token match without an IP check is a claim, not proof. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust, so check the file again the next day, then check the logs. If OAI-SearchBot still has not visited after a reasonable wait, the likely cause is upstream: a firewall rule, a bot manager, or a hosting setting that blocks it before the file is read. Fix that layer and recheck, rather than rewriting a file that already says allow.

The mistakes that keep repeating

The first mistake is blocking GPTBot to leave ChatGPT search. It does not work that way, because the settings are independent, and the search token is OAI-SearchBot. The second mistake is allowing GPTBot to join ChatGPT search. It does not work that way either, for the same reason. The third mistake is writing one group and assuming it covers OpenAI. It covers one bot, and OpenAI runs three agents with three tokens. The fourth mistake is copying a template from before OAI-SearchBot existed in its current role. Those templates name only GPTBot, because that was the only token in the template's time, and they leave search unaddressed. The fifth mistake is treating the file as the whole control. A firewall or bot manager in front of your site can block OAI-SearchBot while the file says allow, and OpenAI recommends allowing the bot's published IP ranges alongside the robots.txt rule for exactly that reason.

The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

Say you run a bicycle shop

You sell road bikes and repair services, and your robots.txt has a single group for GPTBot with Disallow: /. You wrote it to keep your content out of training, which is a fair choice, and you assumed it also kept you out of ChatGPT answers, which it does not. A customer asks ChatGPT for a bike shop that fits road bikes in their city, and ChatGPT recommends a competitor, because your site is still eligible for search and simply is not being surfaced, for reasons that have nothing to do with the file. You read OpenAI's docs and realize the file never mentioned OAI-SearchBot at all. You add its group with Allow: /, keep the GPTBot Disallow: / group beside it, and allow the search bot's published IP ranges through your firewall. The next day you refetch the file, confirm both groups are live, and grep the logs for the OAI-SearchBot token from its published range. The file now makes the two decisions visible as two groups: search yes, training no. That is the split OpenAI documents, and it is the split your file was missing.

What this guide does not cover

This guide covers two OpenAI tokens. It does not describe your products to the models, it does not make your pages legible once a crawler arrives, and it does not cover the other vendors' tokens, which are separate decisions with separate docs. It also does not cover how ChatGPT chooses which retrieved pages to cite, because OpenAI does not document that choice, and no robots.txt rule can force a citation.

Check your own site now

Run the free scan. One click, no signup, and it shows which AI crawlers your robots.txt actually allows: aizii.ai/aeo-scan

Run the free scan

Sources

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.