Aizii Learn · AI Search
← Back to Learn

Why does Claude recommend my competitors

Claude recommends your competitors because it can read more about them, in pages it can reach, than it can read about your brand. There is no submission form for Claude recommendations and no vendor formula for the shortlist, so nobody can buy or file for a place in the answer. Anthropic publishes three crawlers with three different jobs, and it says blocking the search crawler keeps your pages out of its search index while blocking the answer-time fetcher keeps it from re-checking your pages when a person asks. If your competitor's pages are reachable by those two crawlers, written as plain text, and backed by pages other sites have written about them, Claude has material to recommend. If yours are not, it recommends the brand it can verify.

Why Claude picks the competitor

  1. The competitor's pages are in the index Claude searches, and yours are thin there. Anthropic lists Brave Search among its web-search subprocessors, which makes Brave Search a practical place to check whether your pages show up at all. Anthropic has not named a search provider in its own product documentation, so treat that check as a working test, not a vendor promise. A page that does not appear in Brave Search is a page Claude is unlikely to cite.
  2. The competitor allowed the two crawlers that matter for answers, and you allowed only the training one. ClaudeBot collects content that could go into training. Claude-SearchBot indexes content for search. Claude-User fetches a page when a person asks a question. Blocking ClaudeBot only keeps future pages out of training. Blocking the other two is what costs you citations.
  3. The competitor's product facts are plain text in the first HTML. If your price, specs, and availability only appear after JavaScript runs, a crawler that does not render the page may index only the shell. The competitor's pages give Claude a price and a spec to quote. Yours give it a heading and a blank.
  4. The competitor answers the comparison question. A buyer asks Claude for the best option for a need, not for a brand name. The competitor has a comparison page, a guide, and a page that names who the product suits. You have a product page that says what the product is and stops there.
  5. Other sites write about the competitor. Review sites, roundups, and forums name them, and Claude reads pages across the web, not only brand sites. A brand that no other site mentions gives Claude only its own pages to work from, and its own pages are the weakest source for a recommendation.
  6. The competitor's facts agree everywhere. One brand name, one price, one description, repeated the same way on the site, on listings, and on reseller pages. Your facts disagree from one place to the next, so Claude has no clean version to repeat.

Step 1: Run the same prompts and log who gets named

Write ten to twenty prompts your buyers actually ask, including prompts that never name a brand, and run the same set in Claude more than once. Log three things for every prompt: whether your brand was named, whether a competitor was named, and which sources were cited. Then run the cited pages through Brave Search by hand. If a competitor page appears there and yours does not, the gap starts at the index, not at the wording of your page.

Step 2: Check the three crawlers, then check the edge

Open [yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain. If nothing loads, create the file at the root of your site. Add near the top, before any broad disallow rules:

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

All three respect robots.txt, and they do different jobs: ClaudeBot trains the model, Claude-SearchBot builds the search index, Claude-User fetches your page at answer time. If you only allow the first two, Claude can know about you but can never re-check your current prices. Then check the edge, because the file states your intent and the edge decides what really happens: a bot rule at your CDN or firewall can block a crawler before it reads a page, even while robots.txt says allow. After a few days, verify in your logs:

grep -i "claudebot\|claude-searchbot\|claude-user" /var/log/nginx/access.log

Step 3: Put the facts in the HTML Claude reads

View source on a product page. If the price, description, and specs are in the HTML, you are fine. If they only appear after JavaScript executes, the page may be indexed as a shell. Give each page one clear H1, with H2 headings that match the questions buyers ask. Put price, materials, dimensions, and availability in text, never trapped in images. Keep product URLs stable, since URLs that change often break up your presence in the index. You do not need to rebuild the site. Pre-rendered HTML is an option where your platform supports it, without changing what shoppers see.

Step 4: Publish the comparison page the prompt needs

For each prompt where Claude named a competitor, open the cited page and ask what it answers that yours does not. Then publish or improve a page that answers that question directly, in plain text, with the specific facts Claude needs: the price, the specs, who it suits, and how it compares. Write from what you know first hand, with details a summary of other sites cannot offer. A comparison page and a use-case page are the two most often missing. Do not build a thin page for every wording of the same question. Build pages that answer real buyer questions, and let one good page serve several related prompts.

Step 5: Earn the mentions you cannot write yourself

Ask your customers for reviews on the sites your buyers actually read, and make sure those reviews are real and visible on the page. If a roundup lists your competitor and not you, contact the publisher with your product details and a link to your product page. You cannot buy a recommendation, and no vendor sells placement in answers, so the work is the listing itself, not a payment. A model that reads review sites will only mention you if public reviews about you exist.

Step 6: Make the facts agree, then mark them up

Pick one canonical brand name, one description, and one price per product, and make every property you control match: site, listings, resellers, directories. A new page built on top of conflicting facts inherits the conflict. Where a reseller or directory lists an old fact, ask the owner to correct it and give them the link to your corrected page. Then add Product schema that states the same price and availability in structured form, so the facts Claude reads as text and the facts it reads as markup agree. For the full crawler reference, see our robots.txt guide for AI search.

Say you run a skincare brand

Say you run a skincare brand that sells a fragrance-free moisturizer. You run your prompt set and find that Claude names a competitor on nearly every buyer prompt that never names a brand, while you appear only when the prompt names you. You check Brave Search and find the competitor's comparison page and ingredient guide there, while your product page is the only page of yours that appears. Your product page is correct, but it answers none of the comparison questions, and your price lives in an image, not in text. You publish a comparison page and an ingredient guide with your current price and specs as text, you ask two review sites that listed the competitor to list you as well, and you check that Claude-SearchBot and Claude-User can reach the new pages. You rerun the prompt set and log the citations. The competitor does not disappear. Your brand starts appearing alongside them on the prompts your new pages answer.

The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

What this guide does not cover

This guide covers why Claude recommends a competitor and what to do about it. It does not cover correcting a wrong answer about your brand, which is a separate guide, or the transaction layer that turns a recommendation into a sale. It also does not cover monitoring, which is how you track whether the gap is closing across runs instead of guessing from a single check.

Check your own brand now

Run the free scan. It takes seconds, no signup, and shows exactly where your site stands with every AI crawler today: aizii.ai/aeo-scan

Run the free scan

Sources

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.