Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# AEO for Claude: the technical checklist

Claude reads your brand through three Anthropic bots, and each one has its own job. ClaudeBot collects web content that could contribute to training Anthropic's models. Claude-SearchBot navigates the web to improve the quality of Claude's search results. Claude-User fetches a page when a person asks Claude a question that needs it. Anthropic says all three honour robots.txt. The checklist in this guide is short: allow Claude-SearchBot and Claude-User, decide on ClaudeBot on its own merits, set the rules on every subdomain, and make the product facts true, complete, and readable in the first wave of HTML. No vendor publishes a formula for what Claude recommends, and this guide does not claim one. It covers only the technical ground a brand controls.

This is a developer-led job with marketer checks. The developer owns robots.txt, the server, the firewall, and the markup. The marketer owns the product facts and the page copy. Anthropic's crawler page lists three bots. It does not list a crawler for Claude Code, which is a developer tool, not the assistant that answers shoppers. This checklist is about the assistant, not the developer tool.

## Where it goes wrong

Claude checklists fail in a small number of repeatable ways, and most of them sit in robots.txt or in how the page is rendered.

-   Claude-SearchBot is blocked by a blanket rule. A broad Disallow aimed at all crawlers catches the search bot too. Anthropic says disabling Claude-SearchBot prevents its system from indexing content for search, which may reduce a site's visibility and accuracy in user search results. A brand that blocks it and then wonders why Claude never cites it has answered its own question.
-   Claude-User is blocked, and user-directed answers go quiet. Anthropic says disabling Claude-User prevents its system from retrieving content in response to a user query, which may reduce a site's visibility for user-directed web search. Unlike some other engines, Anthropic says its bots honour robots.txt, including the user-initiated one, so a block here does bite. Allow it if you want Claude to fetch your pages when a person asks about them.
-   ClaudeBot is confused with Claude-SearchBot. The two settings are separate. Restricting ClaudeBot signals that the site's future materials should be excluded from training datasets. It does not block search. A file that names one token and means the other is a file that does neither job.
-   The rules are set on one subdomain and forgotten on the rest. Anthropic says to set the robots.txt rules for every subdomain you wish to opt out from. A store on one subdomain and a guide library on another need the same treatment, or the guide library stays open while the store is closed, and the team wonders why the answers are half right.
-   The IP list is used to block, and the opt-out breaks. Anthropic says blocking IP addresses may not work correctly or persistently guarantee an opt-out, because doing so impedes its ability to read the robots.txt file. Its IP list, at the bots.json address in Sources, is for verifying that a request came from Anthropic, not for blocking. Keep the blocking in robots.txt.
-   The content arrives only after JavaScript runs. If the price, the stock state, or the product description loads only in the browser, a bot that reads the first wave of HTML may never see it. Serve the product facts as server-rendered HTML, and check the raw HTML, not the rendered page in a browser, to confirm they are there.
-   The product facts are wrong or late. A price in the page copy that disagrees with the structured data, a size that is gone but marked in stock, a shipping promise the checkout cannot keep. Claude cites pages it can verify, and a page that disagrees with itself gives it nothing to cite with confidence.

## The steps

**Step 1. Decide the training question, then write it down.** The marketer and the developer agree one sentence: do we want our content used to train future Anthropic models, or not. Restricting ClaudeBot signals that future materials should be excluded from training datasets. Allowing it does not. There is no right answer for every brand, and the answer is a business decision, not a technical default. Write it down, because the file below has a separate line for it, and a team that never decided will keep changing the file.

**Step 2. Allow Claude-SearchBot and Claude-User in robots.txt, on their own lines.** The developer opens the robots.txt file at the site root and adds a group for each token, naming it and allowing it. Do not hide either inside a wildcard group and hope the wildcard carries it. Name the tokens. The sample in the machine-readable section below shows the shape. If the file already has a group for either token that disallows it, change that group. Do not add a second group for the same token and leave the old one in place.

**Step 3. Write the ClaudeBot line to match the decision from step one.** The developer adds a group for ClaudeBot that matches what the team decided. Disallow means future materials should be excluded from training datasets. Allow means they may be used. The ClaudeBot line is a training decision. It is not a search lever, and it should never be changed to fix a search problem.

**Step 4. Set the same rules on every subdomain.** The developer repeats the file on every subdomain the brand runs, the store, the guide library, and any regional or staging subdomain that serves live content. Anthropic says to do this for every subdomain you wish to opt out from, and the same logic holds for the allow lines: a subdomain with no file at all is a subdomain with no stated preference. Check each one in a browser and confirm the groups load.

**Step 5. Verify by IP, and use Crawl-delay if the crawl load is the problem.** The developer checks the server logs for requests that present themselves as ClaudeBot, Claude-SearchBot, or Claude-User, and confirms the source IP sits on Anthropic's published list at the bots.json address in Sources. Use it to slow the crawl, not to block it, and do not use IP blocking as the opt-out, because Anthropic says that may not work correctly or persistently guarantee it.

**Step 6. Make the product facts true, complete, and in the first wave of HTML.** The marketer writes the product facts once, in plain words: the name, the price, the stock state, the shipping promise, and the return policy. The developer puts the same facts in Product structured data on the page, with the price, the availability, and the shipping details matching the visible text. Then the developer fetches the raw HTML of a product page, with curl or view source, and confirms the price and the stock state are in it before any JavaScript runs. If they are not, the page needs server-side rendering for those facts.

**Step 7. Check the sitemap and the llms.txt file, then retest.** The developer confirms the sitemap lists the product pages, the category pages, and the guides at their canonical addresses, and that robots.txt points at the sitemap. The marketer confirms the llms.txt file lists the pages that matter most, with a short note each. Then the team asks Claude the questions a customer would ask, notes whether the brand appears and which pages it cites, and fixes the gap the comparison shows. Anthropic publishes no fixed time for a robots.txt change to take effect, so a same-day retest is not a verdict. Retest after a day, and again after the next crawl.

## The machine-readable piece

This robots.txt sample names each Anthropic token on its own line, so the training decision and the search decision are separate and readable. Replace the placeholder with your own sitemap address. The ClaudeBot group below disallows training use. If your team decided to allow training use, change that group to Allow. The Crawl-delay line is an example only, not a recommendation: Anthropic supports the extension, and the value shown is a placeholder. Remove it if crawl load is not a problem.

```
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /
Crawl-delay: 10

Sitemap: https://[yourdomain.com]/sitemap.xml
```

This Product JSON-LD block is the machine-readable version of one product page. It uses only fields from the Product structured data guide in Sources. Replace the placeholders with your real price, your real stock state, and your real shipping details, and make sure every value matches the visible text on the page. The priceValidUntil value is a placeholder shape: replace YYYY-MM-DD with the real date the price is valid until, or remove the field.

```
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "[Product name]",
  "description": "[One plain sentence about the product]",
  "offers": {
    "@type": "Offer",
    "priceCurrency": "USD",
    "price": "[Price]",
    "priceValidUntil": "YYYY-MM-DD",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition"
  }
}
```

Check the file before you rely on it. Open your domain followed by /robots.txt in a browser and confirm the groups load, on every subdomain. Fetch a product page as raw HTML and confirm the price and the stock state are in it. Search your server logs for Claude-SearchBot and confirm the source IP sits on Anthropic's published list and the response is 200. If a log line claims to be Claude-SearchBot from an address outside the published list, it is not Anthropic, and the fix is at the edge.

## What it looks like when it works

Say you run a coffee roaster with an online shop, selling beans, brewing gear, and subscriptions, with a guide on brewing pour over at home. The developer finds a blanket Disallow in robots.txt that a privacy checklist added years ago, and the marketer confirms the team never decided on training use. They decide to keep training out and to welcome search, so the file names Claude-SearchBot and Claude-User with Allow and ClaudeBot with Disallow, each on its own line, on the store subdomain and on the guide subdomain. The developer verifies the requests against Anthropic's published IP list, where a bot rule had been challenging them. The marketer rewrites the product facts so the price, the stock state, and the shipping promise agree in words and in code, and the developer confirms the price is in the raw HTML before any JavaScript runs. They ask Claude the questions a customer would ask about pour over coffee, wait a day, and ask again. The brand appears, and the pages it cites are the product pages and the brewing guide, not a stale listing. The fix was a file, a verification check, and true facts, in that order.

### The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

-   •The llms.txt format and a complete sample, see our [guide to writing a good llms.txt file](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The markup AI systems read to verify facts, see our [product schema guide](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)
-   •The complete crawler rule reference, see our [robots.txt guide for AI search](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

## What this guide does not cover

This guide covers the technical checklist for Claude: the three Anthropic bots, the robots.txt lines for each, the IP verification, and the product facts. It does not cover the full technical audit that checks every crawler and every page, the transaction layer that lets an agent complete a purchase, or the per-engine playbooks for ChatGPT, Perplexity, and Gemini. It does not cover what Claude says about your brand over time, or how to correct a wrong answer once it appears. Those are separate jobs, and this guide is the map, not the whole trip.

## Check your own store now

Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan

[Run the free scan](https://aizii.ai/aeo-audit)

## Sources

-   Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler (specific): [https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
-   Anthropic, Claude bots IP list (specific): [https://claude.com/crawling/bots.json](https://claude.com/crawling/bots.json)
-   Google Search Central, Product structured data (specific): [https://developers.google.com/search/docs/appearance/structured-data/product](https://developers.google.com/search/docs/appearance/structured-data/product)
-   Schema.org, Product type (specific): [https://schema.org/Product](https://schema.org/Product)
-   RFC 9309, Robots Exclusion Protocol (general standard): [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html)
-   llms.txt proposal (specific): [https://llmstxt.org](https://llmstxt.org/)

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
