Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# AEO for ChatGPT: the technical checklist

ChatGPT finds your brand through three separate OpenAI agents, and each one has its own job. OAI-SearchBot crawls pages so they can appear in ChatGPT search answers. GPTBot crawls content that may be used to train future models. ChatGPT-User fetches a page when a person asks ChatGPT a question that needs it. The checklist in this guide is simple to state and easy to get wrong: allow OAI-SearchBot, decide on GPTBot on its own merits, leave ChatGPT-User alone, and make the product facts on your pages true, complete, and readable in the first wave of HTML. No vendor publishes a formula for what ChatGPT recommends, and this guide does not claim one. It covers only the technical ground a brand controls.

This is a developer-led job with marketer checks. The developer owns robots.txt, the server, the firewall, and the markup. The marketer owns the product facts, the page copy, and the decision about training use. If you are both people, agree the training decision first, then change the file, in that order, before you test anything.

## Where it goes wrong

ChatGPT checklists fail in a small number of repeatable ways, and most of them sit in robots.txt or at the edge.

-   OAI-SearchBot is blocked by a blanket rule. A broad Disallow aimed at all crawlers, or a rule copied from a privacy checklist, catches the search crawler too. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. A brand that blocks it and then wonders why ChatGPT never cites it has answered its own question.
-   GPTBot is confused with OAI-SearchBot. The two settings are independent, and OpenAI says so directly. Blocking GPTBot signals that content should not be used for training. It does not block search. Allowing GPTBot does not buy search visibility. A file that names one token and means the other is a file that does neither job.
-   ChatGPT-User is blocked in robots.txt and the team expects the block to hold. OpenAI says ChatGPT-User is not used for automatic crawling, and because its visits are initiated by a user, robots.txt rules may not apply. It is also not used to decide whether content may appear in search. A rule aimed at it does not manage search inclusion. Use OAI-SearchBot for that.
-   The firewall challenges the crawler. A bot rule, a challenge page, or a block on the published IP ranges stops OAI-SearchBot before robots.txt is ever read. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. A 403 or a challenge page in the logs is the warning sign, and the fix lives at the edge, not in the file.
-   The product facts are wrong or late. A price in the page copy that disagrees with the structured data, a size that is gone but marked in stock, a shipping promise the checkout cannot keep. ChatGPT has no authoritative signal to trust when the page disagrees with itself, so it either cites a competitor or answers with a hedge.
-   The content arrives only after JavaScript runs. If the price, the stock state, or the product description loads only in the browser, a crawler that reads the first wave of HTML may never see it. Serve the product facts as server-rendered HTML, and check the raw HTML, not the rendered page in a browser, to confirm they are there.
-   The brand tests once and stops. OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust. A same-day retest proves nothing either way. Test, wait, then test again.

## The steps

**Step 1. Decide the training question, then write it down.** The marketer and the developer agree one sentence: do we want our content used to train future OpenAI models, or not. Blocking GPTBot signals that content should not be used for training. Allowing it does not. There is no right answer for every brand, and the answer is a business decision, not a technical default. Write it down, because the file below has two separate lines for it, and a team that never decided will keep changing the file.

**Step 2. Allow OAI-SearchBot in robots.txt, on its own line.** The developer opens the robots.txt file at the site root and adds a group that names OAI-SearchBot and allows it. Do not hide it inside a wildcard group and hope the wildcard carries it. Name the token. The sample in the machine-readable section below shows the shape. If the file already has a group for OAI-SearchBot that disallows it, change that group. Do not add a second group for the same token and leave the old one in place, because a crawler reads the most specific matching group and a duplicate creates a question with no good answer.

**Step 3. Write the GPTBot line to match the decision from step one.** The developer adds a group for GPTBot that matches what the team decided. Disallow means the content should not be used for training. Allow means it may be. OpenAI says if a site has allowed both bots, it may use the results from just one crawl for both use cases to avoid duplicative crawling. Either way, the GPTBot line is a training decision. It is not a search lever, and it should never be changed to fix a search problem.

**Step 4. Leave ChatGPT-User out of the file, and allow its IP ranges at the edge.** The developer does not add a rule for ChatGPT-User, because OpenAI says robots.txt rules may not apply to it and it is not used to decide search inclusion. Instead, the developer checks the firewall and the CDN rules so requests from OpenAI's published ChatGPT-User IP ranges are not challenged. The published ranges live at the chatgpt-user.json address in Sources. A user who asks ChatGPT about a product and gets a challenge page instead of the product page is a customer lost at the last step, and the cause is the edge, not the file.

**Step 5. Allow the published IP ranges for OAI-SearchBot, and verify by IP, not by user agent alone.** The developer adds the OAI-SearchBot IP ranges from the searchbot.json address in Sources to the allowlist at the edge, and checks the server logs for requests that present themselves as OAI-SearchBot. A user agent string can be faked by anyone, so finding the token in a log does not prove the request came from OpenAI. Check the source IP against the published ranges.

**Step 6. Make the product facts true, complete, and in the first wave of HTML.** The marketer writes the product facts once, in plain words: the name, the price, the stock state, the shipping promise, and the return policy. The developer puts the same facts in Product structured data on the page, with the price, the availability, and the shipping details matching the visible text. Then the developer fetches the raw HTML of a product page, with curl or view source, and confirms the price and the stock state are in it before any JavaScript runs. If they are not, the page needs server-side rendering for those facts.

**Step 7. Check the sitemap, the llms.txt file, and the crawl path, then retest after a day.** The developer confirms the sitemap lists the product pages, the category pages, and the guides at their canonical addresses, and that robots.txt points at the sitemap. The marketer confirms the llms.txt file lists the pages that matter most, with a short note each. Then the team asks ChatGPT the questions a customer would ask, notes whether the brand appears and which pages it cites, and waits. OpenAI says a robots.txt change can take about 24 hours to reach its systems, so a retest the same afternoon is not a verdict. Retest the next day, and fix the gap the comparison shows.

## The machine-readable piece

This robots.txt sample names each OpenAI token on its own line, so the training decision and the search decision are separate and readable. Replace the placeholders with your own sitemap address. The GPTBot group below disallows training use. If your team decided to allow training use, change that group to Allow. The ChatGPT-User group is left out on purpose, for the reason given in step four.

```
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://[yourdomain.com]/sitemap.xml
```

This Product JSON-LD block is the machine-readable version of one product page. It uses only fields from the Product structured data guide in Sources. Replace the placeholders with your real price, your real stock state, and your real shipping details, and make sure every value matches the visible text on the page. The priceValidUntil value is a placeholder shape: replace YYYY-MM-DD with the real date the price is valid until, or remove the field.

```
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "[Product name]",
  "description": "[One plain sentence about the product]",
  "offers": {
    "@type": "Offer",
    "priceCurrency": "USD",
    "price": "[Price]",
    "priceValidUntil": "YYYY-MM-DD",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition"
  }
}
```

Check the file before you rely on it. Open your domain followed by /robots.txt in a browser and confirm the groups load. Fetch a product page as raw HTML and confirm the price and the stock state are in it. Search your server logs for OAI-SearchBot and confirm the source IP sits in the published ranges and the response is 200. If a log line claims to be OAI-SearchBot from an address outside the published ranges, it is not OpenAI, and the fix is at the edge.

## What it looks like when it works

Say you run a handmade ceramics shop with a small catalog, selling mugs, bowls, and plates, with a guide on caring for handmade stoneware. The developer finds a blanket Disallow in robots.txt that a privacy checklist added years ago, and the marketer confirms the team never decided on training use. They decide to keep training out and to welcome search, so the file names OAI-SearchBot with Allow and GPTBot with Disallow, each on its own line. The developer allows the published OAI-SearchBot IP ranges at the edge, where a bot rule had been challenging them. The marketer rewrites the product facts so the price, the stock state, and the shipping promise agree in words and in code, and the developer confirms the price is in the raw HTML before any JavaScript runs. They ask ChatGPT the customer questions, wait a day, and ask again. The brand appears, and the pages it cites are the product pages and the care guide, not a stale listing.

### The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

-   •The llms.txt format and a complete sample, see our [guide to writing a good llms.txt file](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The markup AI systems read to verify facts, see our [product schema guide](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)
-   •The complete crawler rule reference, see our [robots.txt guide for AI search](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

## What this guide does not cover

This guide covers the technical checklist for ChatGPT: the three OpenAI agents, the robots.txt lines for each, the IP verification, and the product facts. It does not cover the full technical audit that checks every crawler and every page, the transaction layer that lets an agent complete a purchase, or the per-engine playbooks for Perplexity, Gemini, and Claude. It does not cover what ChatGPT says about your brand over time, or how to correct a wrong answer once it appears. Those are separate jobs, and this guide is the map, not the whole trip.

## Check your own store now

Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan

[Run the free scan](https://aizii.ai/aeo-audit)

## Sources

-   OpenAI, Overview of OpenAI crawlers (specific): [https://platform.openai.com/docs/bots](https://platform.openai.com/docs/bots)
-   OpenAI Help Center, Shopping with ChatGPT Search (specific): [https://help.openai.com/en/articles/11128490-shopping-with-chatgpt-search](https://help.openai.com/en/articles/11128490-shopping-with-chatgpt-search)
-   Google Search Central, Product structured data (specific): [https://developers.google.com/search/docs/appearance/structured-data/product](https://developers.google.com/search/docs/appearance/structured-data/product)
-   Schema.org, Product type (specific): [https://schema.org/Product](https://schema.org/Product)
-   RFC 9309, Robots Exclusion Protocol (general standard): [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html)
-   llms.txt proposal (specific): [https://llmstxt.org](https://llmstxt.org/)

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
