Aizii Learn · AI Search
← Back to Learn

How to allow ClaudeBot in robots.txt

Add a group for ClaudeBot with an Allow rule in [yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain, and place it before any broad disallow rules. ClaudeBot is Anthropic's training crawler. It collects web content that could contribute to training Anthropic's generative AI models. Allowing it is a training decision only. It does not put your brand into Claude search answers, and blocking it does not take you out of them. Those jobs belong to Claude-SearchBot and Claude-User, and Anthropic treats each of its three bots as a separate setting.

Why ClaudeBot gets missed or mixed up

  1. People write one rule for Claude and stop. Anthropic runs three bots, and a rule that names only ClaudeBot says nothing about the search crawler or the user fetcher, which are the two that carry your brand into answers.
  2. Old names still circulate. Claude-Web and anthropic-ai appear in forum lists and third-party bot directories, but they are retired names that match nothing Anthropic runs now, so a rule for either one is dead text in your file.
  3. A named group stands alone. Once you write a group for ClaudeBot, that group is the only set of rules ClaudeBot follows. It does not inherit the rules from your wildcard group, so a checkout or account disallow that lives only under User-agent: * no longer applies to ClaudeBot.
  4. The wrong block breaks the opt-out. Anthropic warns against blocking its crawler IP addresses to opt out, because that stops its bots from reading your robots.txt file. An IP block can leave the file unread and the opt-out unreliable, which is the opposite of what the block was meant to do.

Step 1: Check the file exists and read the wildcard group

Open [yourdomain.com]/robots.txt in a browser. If nothing loads, create the file at the root of your site. Anthropic says the file belongs in the top-level directory, and it says to repeat the rules for every subdomain you want covered, because a file on one subdomain says nothing about another. Read what is already there before you add anything. Note every Disallow line under User-agent: * that should still apply to ClaudeBot, such as checkout, account, and internal search paths. You will need to repeat those lines inside the ClaudeBot group.

Step 2: Add the ClaudeBot group before any broad disallow

Put the group near the top of the file, before any broad disallow rules, so it is easy to find and easy to audit later.

User-agent: ClaudeBot
Allow: /
Disallow: /checkout/
Disallow: /account/

The Allow line opens the public catalog to ClaudeBot. The Disallow lines repeat the private paths from your wildcard group. Change the paths to match your own store. If your wildcard group disallows nothing, the group can be two lines: the User-agent line and Allow: /. If you want ClaudeBot kept out of training instead, change Allow: / to Disallow: / in this group and leave the rest of the file alone. Anthropic says restricting ClaudeBot signals that the site's future materials should be excluded from its training datasets. Note the word future: the opt-out works going forward, and it does not reach back into material already collected.

Step 3: Add the search and user bots if Claude answers matter to you

This step is the one most files skip. ClaudeBot does not feed Claude search answers. Claude-SearchBot does the indexing: Anthropic says it navigates the web to improve search result quality, and disabling it prevents the system from indexing your content for search, which may reduce your visibility and accuracy in search results. Claude-User does the live fetching: Anthropic says it supports Claude users, and when someone asks Claude a question it may access a website using a Claude-User agent, and disabling it prevents the system from retrieving your content in response to a user query. Add both groups next to the ClaudeBot group.

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Anthropic says its bots honor robots.txt, including Claude-User, which is a real difference from some other vendors whose user fetchers generally ignore the file. That makes the Claude-User allow rule worth writing: it is a rule Anthropic says it follows.

Step 4: Slow it down instead of blocking it, if load is the problem

If the reason you are editing the file is server load rather than principle, Anthropic offers a middle option. It supports the non-standard Crawl-delay extension to robots.txt, which asks the bot to wait between requests. Its example sets a delay of one second for ClaudeBot.

User-agent: ClaudeBot
Crawl-delay: 1
Allow: /

Crawl-delay is not part of the robots.txt standard, so support varies by crawler, and Google does not recognize it at all. For Anthropic's bots it is documented, so it is the right first lever when ClaudeBot is the bot adding load. If the load continues after you set it, check the logs to confirm the traffic really comes from Anthropic before you change anything else, and confirm the requesting IP against the list Anthropic publishes at claude.com/crawling/bots.json. Do not answer load with an IP block: Anthropic warns that blocking its IP addresses may not work correctly or persistently guarantee an opt-out, because it stops the bots from reading your robots.txt file.

Step 5: Verify the file is live and the token is right

Refetch [yourdomain.com]/robots.txt and confirm the ClaudeBot group is present, spelled exactly ClaudeBot, and placed before any broad disallow. Anthropic does not publish a full user-agent string for its bots, so in your logs match on the token ClaudeBot inside the user-agent field, not on a full string you found in a forum. Check the same spelling for the other two tokens, Claude-SearchBot and Claude-User, and remove any group for Claude-Web or anthropic-ai, since those names match nothing. Then watch the logs over the next day or two. If the bots still have not visited after a reasonable wait, the likely cause is upstream: a firewall rule, a bot manager, or a hosting setting that blocks them before the file is read. Fix that layer and recheck, rather than rewriting a file that already says allow.

The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

Say you run a coffee roaster

You roast small-batch coffee and sell beans and brewing gear online. Your robots.txt has a single line for ClaudeBot from a template, and nothing else. A customer asks Claude for a roaster that ships fresh beans to their city, and Claude recommends a competitor. Your file allowed the training crawler and never mentioned the search crawler or the user fetcher, so Claude's search index and its live fetching had no explicit permission from you. The fix is three groups, not one: ClaudeBot for the training decision, Claude-SearchBot for the search index, and Claude-User for live answers. You also notice ClaudeBot has been visiting heavily, so you add Crawl-delay: 1 to its group instead of blocking it outright, because the problem is pace, not principle. After you save, you refetch the file, confirm all three groups are live on every subdomain you run, and check the next day's logs for the three tokens. The file now says what you meant it to say.

What this guide does not cover

This guide covers one token and its two sibling bots. It does not describe your products to the models, it does not make your pages legible once a crawler arrives, and it does not cover the edge settings that can overrule the file after you save it. It also does not cover how Claude chooses which retrieved pages to cite, because Anthropic does not document that choice, and no robots.txt rule can force a citation.

Check your own site now

Run the free scan. One click, no signup, and it shows which AI crawlers your robots.txt actually allows: aizii.ai/aeo-scan

Run the free scan

Sources

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.