Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# How to allow OAI-SearchBot in robots.txt

Add a group for OAI-SearchBot with an Allow rule in \[yourdomain.com\]/robots.txt, replacing yourdomain.com with your actual domain, and place it before any broad disallow rules. OAI-SearchBot is OpenAI's search crawler. OpenAI says it is used to surface websites in search results in ChatGPT's search features, and sites that opt out of it will not be shown in ChatGPT search answers. If you want your brand to be findable when someone asks ChatGPT about your products, this is the token that matters. It is the one that decides search visibility this week, not GPTBot, which is the training crawler.

## Why OAI-SearchBot gets missed or mixed up

1.  Most templates name only GPTBot. Those templates were written before OAI-SearchBot existed in its current role, and a file that allows GPTBot and says nothing about OAI-SearchBot gives training permission while leaving live search unaddressed.
2.  People treat GPTBot as the search switch. It is not. OpenAI says GPTBot crawls content that may be used to train future foundation models, and that the two settings are independent. Blocking GPTBot has no effect on search visibility, and allowing it does not create any.
3.  People treat ChatGPT-User as the search switch. It is not either. OpenAI says ChatGPT-User fetches a page when a person asks ChatGPT or a Custom GPT a question, that it is not used to determine whether content may appear in Search, and that robots.txt rules may not apply to it because the fetch is user-initiated.
4.  A named group stands alone. Once you write a group for OAI-SearchBot, that group is the only set of rules OAI-SearchBot follows. It does not inherit the rules from your wildcard group, so a checkout or account disallow that lives only under User-agent: \* no longer applies to OAI-SearchBot.

## Step 1: Check the file exists and read the wildcard group

Open \[yourdomain.com\]/robots.txt in a browser. If nothing loads, create the file at the root of your site. Read what is already there before you add anything. Look for a User-agent: \* group, and note every Disallow line under it that should still apply to OAI-SearchBot, such as checkout, account, and internal search paths. You will need to repeat those lines inside the OAI-SearchBot group, because a named group does not inherit them. Also check whether any group already names OAI-SearchBot with a Disallow: if one does, that is the reason ChatGPT search cannot show your pages, and changing it to Allow is the fix.

## Step 2: Add the OAI-SearchBot group before any broad disallow

Put the group near the top of the file, before any broad disallow rules, so it is easy to find and easy to audit later.

```
User-agent: OAI-SearchBot
Allow: /
Disallow: /checkout/
Disallow: /account/
```

The Allow line opens the public catalog to OAI-SearchBot. The Disallow lines repeat the private paths from your wildcard group. Change the paths to match your own store. If your wildcard group disallows nothing, the group can be two lines: the User-agent line and Allow: /. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges, so the file rule is the first half and the firewall is the second half, covered in the next step.

## Step 3: Allow its IP ranges through the firewall

A file that says allow means little if the edge blocks the bot before it reads the file. Check your firewall, bot manager, or hosting security settings for a rule that challenges or blocks OAI-SearchBot, and allow the token together with its published IP ranges, not by user agent alone. OpenAI publishes the ranges at openai.com/searchbot.json. Confirm a hit is really OpenAI by checking the requesting IP against that list, because a user-agent string on its own is easy to copy. If you use a managed host or a platform, ask your host where bot rules live before you assume the firewall is yours to change. This step is where most allow rules silently fail: the file is correct, the edge is not, and nobody checks the edge.

## Step 4: Decide GPTBot separately, on its own terms

Do not let the search decision make the training decision for you. GPTBot is the training crawler, and OpenAI says each setting is independent. Its own example is to allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep content out of training. If that split matches your position, add the GPTBot group next to the search group.

```
User-agent: GPTBot
Disallow: /
```

If you allow both bots, OpenAI says it may use the results from one crawl for both uses, to avoid crawling twice. Either way, write the two groups as two decisions, not one. A reader of your file should be able to see that search is allowed and training is not, or that both are allowed, without guessing which line was meant to do which job. Do not add a group for ChatGPT-User expecting it to control search. OpenAI says to use OAI-SearchBot in robots.txt for managing search opt-outs and automatic crawling, and that ChatGPT-User is not the search switch.

## Step 5: Verify the file is live and the token is right

Refetch \[yourdomain.com\]/robots.txt and confirm the OAI-SearchBot group is present, spelled exactly OAI-SearchBot, and placed before any broad disallow. The token is case-insensitive for matching, but spell it the way OpenAI documents it so the next person who reads the file is not guessing. The full user-agent string OpenAI publishes is `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot`, and OpenAI notes the version number may change, so match on the token OAI-SearchBot, not on the full string. When OAI-SearchBot fetches the robots.txt file itself, OpenAI may add a robots.txt marker to the string, so do not be surprised by a variant line for that one request. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust, so check the file again the next day, then check the logs for the token from its published IP range.

### The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

-   •The complete rule reference for every crawler and edge case, see our [robots.txt guide for AI search](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)
-   •What goes in llms.txt and how to publish it, see our [guide to writing a good llms.txt file](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The markup AI engines read to state price and availability in structured form, see our [product schema guide for AI search](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)

### Say you run a pet food brand

You sell limited-ingredient dog food and publish feeding guides next to every product. A customer asks ChatGPT for a dog food brand that ships a sensitive-stomach formula, and ChatGPT recommends a competitor. You check your robots.txt and find a GPTBot group from a template, and nothing else. Your file allowed the training crawler and never mentioned the search crawler, so ChatGPT's live search had no explicit permission from you. You add an OAI-SearchBot group with Allow: / and the checkout and account disallows repeated inside it, and you add a GPTBot group with Disallow: /, because you want search visibility without training use. Then you check your firewall and find a bot rule challenging OAI-SearchBot, so you allow the token together with its published IP ranges. The next day you refetch the file, confirm both groups are live, and grep the logs for the OAI-SearchBot token to confirm a real visit. The file now says what you meant it to say, and the two decisions are visible as two groups.

## What this guide does not cover

This guide covers one token and its two sibling bots. It does not describe your products to the models, it does not make your pages legible once a crawler arrives, and it does not cover how ChatGPT chooses which retrieved pages to cite, because OpenAI does not document that choice, and no robots.txt rule can force a citation. It also does not cover the rest of the OpenAI family in full: OAI-AdsBot validates pages submitted as ads on ChatGPT and its data is not used for training, which matters only if you run ChatGPT ads.

## Check your own site now

Run the free scan. One click, no signup, and it shows which AI crawlers your robots.txt actually allows: aizii.ai/aeo-scan

[Run the free scan](https://aizii.ai/aeo-audit)

## Sources

-   OpenAI bots documentation (specific): [https://platform.openai.com/docs/bots](https://platform.openai.com/docs/bots)
-   Cloudflare robots.txt explainer (general docs): [https://www.cloudflare.com/learning/bots/what-is-robots-txt/](https://www.cloudflare.com/learning/bots/what-is-robots-txt/)
-   RFC 9309, Robots Exclusion Protocol (general standard): [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html)

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
