Aizii Learn · AI Search
← Back to Learn

Should you block AI crawlers or let them in

Let the search crawlers in, and decide the training crawlers on their own terms. That is the short answer for most ecommerce brands. The search crawlers are OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and bingbot: they are the ones that carry your brand into answers that cite your pages. The training crawlers are GPTBot, ClaudeBot, and meta-externalagent, plus the Google-Extended policy token: they govern whether your content may be used to train future models, and blocking them does not remove you from search. The user fetchers are ChatGPT-User, Claude-User, Perplexity-User, and meta-externalfetcher: they visit when a person asks a question, and several of them generally ignore robots.txt, so a block rule is not a reliable way to stop them. One blanket decision for all of them is the mistake. Make three decisions, one per job, and write each one down in the file.

What blocking actually costs

Blocking a search crawler costs visibility, and the cost is specific to that crawler. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says disabling Claude-SearchBot prevents its system from indexing your content for search, which may reduce your visibility and accuracy in search results. Perplexity says PerplexityBot is designed to surface and link websites in its search results, and it recommends allowing it if you want to appear there. Google says its Search inclusion runs through Googlebot, not through Google-Extended, so blocking the training token does not cost you Search, while blocking Googlebot does.

What blocking does not buy

Blocking a training crawler does not buy search privacy, because training and search are separate settings at every vendor that documents both. OpenAI says the OAI-SearchBot and GPTBot settings are independent. Anthropic runs ClaudeBot for training and Claude-SearchBot for search as separate bots with separate rules. Blocking GPTBot or ClaudeBot opts you out of future training use, and it does not remove you from answers. Blocking a crawler also does not remove content that is already collected or already indexed. A robots.txt change governs future crawling, not the past, and Anthropic's wording makes the point explicit: restricting ClaudeBot signals that future materials should be excluded from training datasets.

The file is a request, not a lock

The robots.txt standard, RFC 9309, says plainly that its rules are not a form of access authorization. The file asks well-behaved crawlers to follow your rules, and it does not enforce anything against a crawler that ignores it. Cloudflare makes the same point in its own docs: a robots.txt file helps manage well-behaved bot activity, and it will not do much for malicious bot traffic. That matters twice over. First, the fetchers that ignore the file will keep arriving: Perplexity says Perplexity-User generally ignores robots.txt because a user requested the fetch, OpenAI says the same about ChatGPT-User, and Meta says meta-externalfetcher may bypass robots.txt. Second, the crawlers that no vendor documents cannot be gated by a token at all. No vendor publishes a crawler token for Grok, and no vendor publishes one for DeepSeek, so a rule for GrokBot or DeepSeekBot matches nothing official, and their live fetches arrive as ordinary browsers. A block list built only from robots.txt is a partial control, and a brand that needs enforcement needs a firewall rule as well.

The decision, token by token

Work through the list and write a group for each token you name, because a named group stands alone and does not inherit your wildcard group's rules.

Allow the search crawlers. OAI-SearchBot, Claude-SearchBot, and PerplexityBot are the retrieval bots for ChatGPT search, Claude search, and Perplexity search. Googlebot feeds Google Search, including its AI features, and bingbot feeds the Bing index behind Copilot. If you want to be recommended, these are the doors to leave open.

Decide the training crawlers as a business choice, not a visibility choice. GPTBot, ClaudeBot, meta-externalagent, and Google-Extended govern training use. Allowing them lets vendors train on your content. Blocking them opts you out of training. Neither choice changes whether you appear in answers. Make the choice with whoever owns your content and your brand position, and write it in the file so the next person who reads it can see it was deliberate.

Allow the user fetchers, and know their limit. ChatGPT-User, Claude-User, and Perplexity-User visit when a person asks about your products. Blocking them means the model cannot check your live page at the moment a citation is earned. Anthropic says its bots honor robots.txt, including Claude-User, so a Claude-User rule there is followed. OpenAI and Perplexity say their user fetchers generally ignore the file, so a rule for those two is a statement of preference, not a lock. If you need a user fetcher kept out, a firewall rule keyed to its user agent together with its published IP ranges is the enforceable route.

Keep the private paths closed to everyone. Checkout, account, and internal search paths should stay disallowed in every group, including the named ones, because a named group does not inherit the wildcard group's disallows. Copy those lines into each group you write.

A starting file for a brand that wants to be found

If your position is to be visible in answers and to stay out of training, the file below is the shape of that decision. It allows the search crawlers and the user fetchers, and it disallows the training crawlers.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Two notes on this file. First, Google-Extended is a policy token, not a crawler, so its group controls Gemini training and grounding, not Search, and it never appears in server logs. Second, the training disallows above are a choice, not a recommendation for every brand. A brand that is comfortable with training use can allow those tokens instead, and the search allows stay the same either way.

Check the edge before you declare the decision made

A file that says allow while the firewall says block is the most common way this decision fails. After you save the file, refetch it to confirm the groups are live, then check your firewall, bot manager, and hosting security settings for rules that challenge or block the tokens you just allowed. Perplexity publishes firewall guidance for Cloudflare and AWS because its agents get blocked on sites whose robots.txt permits them. OpenAI recommends allowing OAI-SearchBot and its published IP ranges together. Confirm real visits by checking the requesting IP against each vendor's published list, because a user-agent string on its own is easy to copy. OpenAI, Perplexity, and Meta all say robots.txt changes take about 24 hours to take effect, so check the file again the next day, then check the logs.

The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

Say you run a furniture store

You sell sofas and dining tables, and a consultant tells you to block all AI crawlers to protect your product photos. You write a blanket Disallow under User-agent: * and move on. A month later a customer asks ChatGPT for a solid oak dining table that seats eight, and ChatGPT recommends two competitors and never mentions you. The blanket block did what you asked it to do: it kept every crawler out, including the search crawlers that would have carried your brand into that answer. It also did not do what you hoped it would do: it did not remove your photos from training data already collected, and it did not stop user fetchers from visiting when a person asked about you, because several of them generally ignore the file. You go back and replace the blanket block with three decisions. The search crawlers get Allow groups, because being recommended is how new customers find you. The training crawlers get Disallow groups, because that is the protection you actually wanted. The user fetchers get Allow groups, because a person asking about a sofa is the moment a citation is earned. The file now protects what you meant to protect, and it stops blocking what you never meant to block.

What this guide does not cover

This guide covers the allow-or-block decision. It does not describe your products to the models, it does not make your pages legible once a crawler arrives, and it does not cover server load, which is a separate problem with separate levers: slowing a crawler down is different from refusing it, and the load guide covers that in full. It also does not cover enforcement against crawlers that ignore the file, which needs firewall and bot management rules, not more lines in robots.txt.

Check your own site now

Run the free scan. One click, no signup, and it shows which AI crawlers your robots.txt actually allows: aizii.ai/aeo-scan

Run the free scan

Sources

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.