Add a group for GPTBot with an Allow rule in [yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain, and place it before any broad disallow rules. GPTBot is OpenAI's training crawler. It collects content that may be used to train future foundation models. Allowing it is a training decision only. It does not put your brand into ChatGPT search answers, and blocking it does not take you out of them. That job belongs to a different OpenAI search crawler, OAI-SearchBot, and the two settings are independent.
Why GPTBot gets missed or mixed up
- Most templates name only one OpenAI bot. A file that allows GPTBot and says nothing about OAI-SearchBot gives training permission while leaving live search unaddressed, which is the opposite of what most brands want.
- People treat GPTBot as the ChatGPT switch. It is not. OpenAI says GPTBot crawls content that may be used in training, and that disallowing it indicates the content should not be used for training. Search visibility is a separate setting.
- A named group stands alone. Once you write a group for GPTBot, that group is the only set of rules GPTBot follows. It does not inherit the rules from your wildcard group, so a checkout or account disallow that lives only under User-agent: * no longer applies to GPTBot.
- The file can say allow while the edge says no. A firewall or bot manager in front of your site can challenge or block GPTBot before it ever reads the file, so an allow rule is permission, not a guarantee of a visit.
Step 1: Check the file exists and read the wildcard group
Open [yourdomain.com]/robots.txt in a browser. If nothing loads, create the file at the root of your site. Read what is already there before you add anything. Look for a User-agent: * group, and note every Disallow line under it that should still apply to GPTBot, such as checkout, account, and internal search paths. You will need to repeat those lines inside the GPTBot group, because a named group does not inherit them.
Step 2: Add the GPTBot group before any broad disallow
Put the group near the top of the file, before any broad disallow rules, so it is easy to find and easy to audit later.
User-agent: GPTBot
Allow: /
Disallow: /checkout/
Disallow: /account/
The Allow line opens the public catalog to GPTBot. The Disallow lines repeat the private paths from your wildcard group. Change the paths to match your own store. If your wildcard group disallows nothing, the group can be two lines: the User-agent line and Allow: /. If you want GPTBot kept out of training instead, change Allow: / to Disallow: / in this group and leave the rest of the file alone. That is the training opt-out, and it has no effect on search.
Step 3: Add OAI-SearchBot if ChatGPT search matters to you
This step is the one most files skip. GPTBot does not feed ChatGPT search answers. OAI-SearchBot does. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. If you want your brand to be findable when someone asks ChatGPT about your products, add its group next to the GPTBot group.
User-agent: OAI-SearchBot
Allow: /
OpenAI says each setting is independent, and gives this exact split as its example: allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep content out of training. If you allow both bots, OpenAI says it may use the results from one crawl for both uses, to avoid crawling twice. Do not add a group for ChatGPT-User expecting it to control search. OpenAI says ChatGPT-User fetches a page when a person asks ChatGPT or a Custom GPT a question, that it is not used to determine whether content may appear in Search, and that robots.txt rules may not apply to it because the fetch is user-initiated.
Step 4: Verify the file is live and the token is right
Refetch [yourdomain.com]/robots.txt and confirm the GPTBot group is present, spelled exactly GPTBot, and placed before any broad disallow. The token is case-insensitive for matching, but spell it the way OpenAI documents it so the next person who reads the file is not guessing. Then check your server logs for the token. The full user-agent string OpenAI publishes is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, and OpenAI notes the version number may change, so match on the token GPTBot, not on the full string. When GPTBot fetches the robots.txt file itself, OpenAI may add a robots.txt marker to the string, so do not be surprised by a variant line for that one request. Confirm a hit is really OpenAI by checking the requesting IP against the published list at openai.com/gptbot.json, because a user-agent string on its own is easy to copy. If a log line carries the GPTBot token but the IP falls outside the published range, treat it as untrusted traffic, not as GPTBot. A robots.txt rule will not stop a spoofed token, because the sender is not the vendor, and that traffic needs a firewall rule, not a file edit.
Step 5: Wait before you judge the change
OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. GPTBot is the training crawler, so its visits follow their own schedule, and an allow rule is not a promise of a crawl on any given day. Check the file again the next day, then check the logs. If GPTBot still has not visited after a reasonable wait, the likely cause is upstream: a firewall rule, a bot manager, or a hosting setting that blocks it before the file is read. Fix that layer and recheck, rather than rewriting a file that already says allow.
The fundamentals
These mechanics are shared across every engine and covered in full in their own guides:
- •The complete rule reference for every crawler and edge case, see our robots.txt guide for AI search
- •What goes in llms.txt and how to publish it, see our guide to writing a good llms.txt file
- •The markup AI engines read to state price and availability in structured form, see our product schema guide for AI search
Say you run a candle brand
You sell poured candles and home fragrance, and you paste an allow block from a forum into your robots.txt. It names GPTBot and nothing else. A month later a customer asks ChatGPT for a gift candle, and ChatGPT recommends a competitor. Your file looked complete, but it allowed the training crawler and never mentioned the search crawler. Training permission does not create search visibility. The fix is two groups, not one: GPTBot for the training decision you have already made, and OAI-SearchBot for the search visibility you actually wanted. After you add the second group, you refetch the file, confirm both groups are live, and watch the logs for the OAI-SearchBot token from its published IP range. The file now says what you meant it to say.
What this guide does not cover
This guide covers one token and one decision. It does not describe your products to the models, it does not make your pages legible once a crawler arrives, and it does not cover the edge settings that can overrule the file after you save it. It also does not cover the rest of the OpenAI family in full: OAI-AdsBot validates pages submitted as ads on ChatGPT and its data is not used for training, which matters only if you run ChatGPT ads.
Check your own site now
Run the free scan. One click, no signup, and it shows which AI crawlers your robots.txt actually allows: aizii.ai/aeo-scan
Run the free scanSources
- OpenAI bots documentation (specific): https://platform.openai.com/docs/bots
- Cloudflare robots.txt explainer (general docs): https://www.cloudflare.com/learning/bots/what-is-robots-txt/
- RFC 9309, Robots Exclusion Protocol (general standard): https://www.rfc-editor.org/rfc/rfc9309.html
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
