Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# Technical AEO audit: the complete process

A technical AEO audit is a check of whether AI systems can find your brand, read its pages, verify its facts, and cite it with confidence. It is not a content review, and it is not a ranking report. It is a pass over the machine-readable layer of the store: robots.txt, the sitemap, the llms.txt file, the structured data, the raw HTML, the server logs, and the firewall. This guide covers the complete process, in order, for all four engines at once: ChatGPT, Perplexity, Gemini, and Claude. No vendor publishes a formula for what any engine recommends, and this guide does not claim one. It covers only the technical ground a brand controls, and it says plainly where a vendor publishes no answer.

This is a developer-led job with marketer checks. The developer owns the files, the server, the firewall, and the markup. The marketer owns the product facts and the page copy. If you are both people, run the audit in the order below, because each step depends on the one before it: a crawler that cannot reach the site makes every later check meaningless.

## Where it goes wrong

Audits fail in a small number of repeatable ways, and most of them are about what the audit never looked at.

-   The audit reads the rendered page, not the raw HTML.
-   The audit checks robots.txt and stops. A file that allows a crawler means nothing if the firewall challenges it, the CDN blocks it, or the server returns an error.
-   The audit checks one engine and calls it done. ChatGPT, Perplexity, Gemini, and Claude each have their own crawlers, their own tokens, and their own rules. A site that is open to one can be closed to another by a single line. The audit must name each token, for each engine, and check each one.
-   The audit checks the homepage and a product page, and never the rest. Category pages, guides, the about page, and the policy pages are the pages an assistant cites when it explains a choice, and they are the pages that go stale first.
-   The audit trusts the user agent string. A user agent string can be faked by anyone, so finding a token in a log does not prove the request came from that vendor.
-   The audit finds a mismatch and fixes the page, not the process.
-   The audit is run once and never repeated.

## The steps

**Step 1. Inventory the pages that matter.** The marketer lists the pages worth finding: the product pages, the category pages, the brand pages, the guides, the about page, and the policy pages shoppers ask about. Leave out the cart, the checkout, the account pages, the site search results, and filtered faceted addresses. Pick a sample that covers every page type, including one product that recently sold out and one that recently came back, and one guide that was recently rewritten. The audit runs on this list, not on the homepage alone.

**Step 2. Check robots.txt for every engine, token by token.** The developer opens the robots.txt file at the site root and checks each token in turn. For ChatGPT: OAI-SearchBot allowed, GPTBot set to match the team's training decision, ChatGPT-User left out of the file. For Perplexity: PerplexityBot allowed, Perplexity-User left out of the file. For Gemini: Googlebot allowed, Google-Extended allowed unless the team means to refuse training and grounding use. For Claude: Claude-SearchBot and Claude-User allowed, ClaudeBot set to match the team's training decision, on every subdomain. A blanket Disallow that catches any of these is a finding, not a policy. Record each token as allowed, blocked, or missing, and do not guess.

**Step 3. Check the edge: the firewall, the CDN, and the server.** The developer checks the firewall and the CDN rules for each engine's published IP ranges, and confirms genuine requests are not challenged. For Perplexity, the allow rule matches the user agent and the published IP ranges together. For OpenAI, the ranges live at the searchbot.json, gptbot.json, and chatgpt-user.json addresses in Sources. For Anthropic, the list lives at the bots.json address in Sources. For Googlebot, verify by reverse DNS: the hostname must resolve to a googlebot.com address and forward-resolve back to the same IP. Then the developer checks the server logs for each token, confirms the source IP against the published ranges, and records the response code. A run of genuine hits returning 200 is a pass. A run of 403s or challenge pages is the fix list.

**Step 4. Fetch the raw HTML and check the product facts.** The developer fetches the raw HTML of every page in the sample, with curl or view source, and confirms the price, the stock state, and the description are in it before any JavaScript runs. The marketer then compares the facts in the raw HTML with the visible text and with the structured data, page by page. One true price, one true stock state, and one true shipping promise, in words and in code, at the same time. A mismatch is a finding, and the finding names the page, the field, and the two values that disagree. Do not fix the page yet. First record every mismatch, because a pattern of mismatches points at the routine, not at one page.

**Step 5. Check the structured data, the sitemap, and the llms.txt file.** The developer validates the Product structured data on the sample pages against the Product guide in Sources, and confirms the price, the availability, and the shipping details match the visible text. Then the developer confirms the sitemap lists the sample pages at their canonical addresses, with honest change dates, and that robots.txt points at the sitemap. Then the marketer confirms the llms.txt file lists the pages that matter most, with a short note each, and that it loads at the site root. The llms.txt proposal defines llms.txt only. It does not define llms-full.txt, which is a companion convention, not part of the proposal, not a vendor requirement, and not documented as read by any engine.

**Step 6. Ask each engine the customer questions, and record what it cites.** The marketer asks ChatGPT, Perplexity, Gemini, and Claude the questions a customer would ask about the category, the products, and the location, and records whether the brand appears, which pages each engine cites, and what facts it gets wrong. No vendor publishes a refresh schedule, so a fix is not instant: OpenAI says a robots.txt change can take about 24 hours to reach its systems, Perplexity says up to 24 hours, and Google says crawling can take anywhere from several days to several months. A retest the same afternoon is not a verdict. Record the answers, fix the gap the comparison shows, and retest after the engines have had time to return.

**Step 7. Write the fix list as routines, not as pages.** The developer and the marketer turn the findings into routines: the price change runs through one record that updates the page, the structured data, and the feed in the same pass; the discontinued product leaves the catalog, the sitemap, and the llms.txt file the same day; the robots.txt change is checked in a browser and in the logs before the team moves on. Each routine names the person who runs it and the check that proves it ran. Then set the recheck: run the audit again when the catalog changes, when the platform updates, or when a firewall rule is added, and on a regular cadence the team agrees in advance. An audit with no recheck is a snapshot, and a snapshot goes stale the day the catalog changes.

## The machine-readable piece

This robots.txt sample names every engine token in one file, so the training decisions and the search decisions are separate and readable. Replace the placeholder with your own sitemap address. The GPTBot and ClaudeBot groups below disallow training use. If your team decided to allow training use for either engine, change that group to Allow. The ChatGPT-User and Perplexity-User groups are left out on purpose, because their vendors say robots.txt rules may not apply to user-triggered fetches.

```
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

Sitemap: https://[yourdomain.com]/sitemap.xml
```

This Product JSON-LD block is the machine-readable version of one product page. It uses only fields from the Product structured data guide in Sources. Replace the placeholders with your real price, your real stock state, and your real shipping details, and make sure every value matches the visible text on the page. The priceValidUntil value is a placeholder shape: replace YYYY-MM-DD with the real date the price is valid until, or remove the field.

```
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "[Product name]",
  "description": "[One plain sentence about the product]",
  "offers": {
    "@type": "Offer",
    "priceCurrency": "USD",
    "price": "[Price]",
    "priceValidUntil": "YYYY-MM-DD",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition"
  }
}
```

Check the file before you rely on it. Open your domain followed by /robots.txt in a browser and confirm the groups load, on every subdomain. Fetch a product page as raw HTML and confirm the price and the stock state are in it. Search your server logs for each token and confirm the source IP sits in that vendor's published ranges and the response is 200. If a log line claims to be a vendor bot from an address outside the published ranges, it is not that vendor, and the fix is at the edge.

## What it looks like when it works

Say you run a running shoe store with a large catalog, selling road shoes, trail shoes, and racing shoes, with a guide on choosing a shoe for a first marathon. The developer runs the audit on a sample that covers every page type, including a shoe that recently sold out and a guide that was recently rewritten. The file check finds a blanket Disallow catching OAI-SearchBot and Claude-SearchBot, and a Google-Extended block a privacy checklist added years ago. The edge check finds a firewall rule challenging PerplexityBot. The raw HTML check finds the price on three pages only after JavaScript runs, and the fact check finds two pages where the structured data disagrees with the visible text.

### The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

-   •The llms.txt format and a complete sample, see our [guide to writing a good llms.txt file](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The markup AI systems read to verify facts, see our [product schema guide](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)
-   •The complete crawler rule reference, see our [robots.txt guide for AI search](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

## What this guide does not cover

This guide covers the complete technical audit process: the inventory, the robots.txt check for every engine, the edge check, the raw HTML and fact check, the structured data and sitemap check, and the engine retest. It does not cover the transaction layer that lets an agent complete a purchase, the per-engine playbooks in detail, or what each engine says about your brand over time. It does not cover monitoring and drift, or how to correct a wrong answer once it appears. Those are separate jobs, and this guide is the map, not the whole trip.

## Check your own store now

Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan

[Run the free scan](https://aizii.ai/aeo-audit)

## Sources

-   OpenAI, Overview of OpenAI crawlers (specific): [https://platform.openai.com/docs/bots](https://platform.openai.com/docs/bots)
-   Perplexity, Perplexity crawlers (specific): [https://docs.perplexity.ai/guides/bots](https://docs.perplexity.ai/guides/bots)
-   Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler (specific): [https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
-   Google, Google's common crawlers (specific): [https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
-   Google Search Central, AI features and your website (specific): [https://developers.google.com/search/docs/appearance/ai-features](https://developers.google.com/search/docs/appearance/ai-features)
-   Google Search Central, Product structured data (specific): [https://developers.google.com/search/docs/appearance/structured-data/product](https://developers.google.com/search/docs/appearance/structured-data/product)
-   Schema.org, Product type (specific): [https://schema.org/Product](https://schema.org/Product)
-   RFC 9309, Robots Exclusion Protocol (general standard): [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html)
-   llms.txt proposal (specific): [https://llmstxt.org](https://llmstxt.org/)

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
