Claude recommends brands it can read. Its live answers pull from the Brave search index, which reads pages as server-rendered HTML. If your product content only appears after JavaScript runs, Brave may return only the page shell and Claude recommends a competitor with readable pages instead. The fix is making your site legible as plain HTML.
Why Claude skips your brand
Claude's failure modes are its own:
- JavaScript shells are invisible. Headless storefronts and heavy client-side rendering are a common reason Claude misses a brand. The crawler is not blocked; there is simply nothing to read.
- Three crawlers, three jobs. ClaudeBot feeds training data, Claude-SearchBot builds the search index, and Claude-User fetches pages live when someone asks a question. Most guides conflate them. Blocking ClaudeBot alone only opts you out of training; it is Claude-SearchBot and Claude-User that control whether you appear in answers.
- Thin pages lose. Clear heading structure and text facts help here. Specs trapped in images, one-line product descriptions, and missing H1s all read as nothing.
Step 1: Allow all three crawlers
Check [yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain. If nothing loads, create the file at the root of your site. Then add near the top, before any broad disallow rules:
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
All three respect robots.txt, and they do different jobs: ClaudeBot trains the model, Claude-SearchBot builds the search index, Claude-User fetches your page at answer time. If you only allowlist the first two, Claude can know about you but can never re-check your current prices. Then check the edge, because the file is what you declare and the edge is what enforces: Cloudflare Bot Fight Mode and WAF bot rules can block ClaudeBot even when robots.txt allows it. After a few days, verify in your logs:
grep -i "claudebot\|claude-searchbot\|claude-user" /var/log/nginx/access.log
Step 2: Serve real HTML
This is the step that matters most for Claude and the one the ranking guides skip. View source on a product page: if the price, description, and specs are in the HTML, you are fine. If they only appear after JavaScript executes, Brave may index only the page shell.
- One clear H1 per page, with H2s that match the questions buyers ask. Headings that mirror likely questions can help matching; headings that mirror those questions get matched.
- Price, materials, dimensions, and availability as text, never trapped in images.
- Stable product URLs. Frequently changing URLs fragment your index presence.
You do not need to rebuild the site. Pre-rendered HTML is an option where your platform supports it to crawlers without changing what shoppers see.
A hypothetical, to make it concrete
Say you run a home-goods store on a headless storefront. The site looks beautiful and ranks on Google, which renders JavaScript. But Brave may not render the full page, so Claude's index may hold only the page shell of your product pages. A scan would flag it in one pass: ClaudeBot allowed, pages unreadable. The fix is server-side rendering on the product templates, not a redesign. After that, the same pages Claude could not see become citable.
The fundamentals
These mechanics are shared across every engine and covered in full in their own guides:
- •llms.txt: the file format, a complete sample, and per-platform publish steps, see our guide to writing a good llms.txt file
- •Product schema: the JSON-LD markup that states price and stock in structured form, see our product schema guide for AI search
- •Every AI crawler in one reference table, see our robots.txt guide for AI search
What this guide does not cover
Readability gets you into Claude's answers. Staying there takes monitoring: Brave re-indexes on its own schedule and your catalog drifts. And answers are not orders; the transaction layer is separate work.
For the ChatGPT version of this playbook, see our guide to getting ChatGPT to recommend your brand.
Check your own brand now
Run the free scan. It takes seconds, no signup, and shows exactly where Claude stands with your site: aizii.ai/aeo-scan
Run the free scanSources
- Anthropic support: crawling and blocking (specific): https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Claude bots list, bots.json (specific): https://claude.com/crawling/bots.json
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
