To block AI crawlers in robots.txt, name each crawler's token in its own group and give that group a Disallow rule for the whole site. One group per crawler, because each vendor runs several crawlers with different jobs, and a rule for one does not cover the others. Blocking the training crawler opts your content out of training. Blocking the search crawler takes you out of that assistant's answers. Decide which of those two outcomes you want before you write a line, because the file will do exactly what you write and nothing you meant.
Two limits matter before you start. First, robots.txt is a request, not a lock. RFC 9309, the standard behind the file, states that these rules are not a form of access authorization. Second, some fetchers never read the file at all. When a person asks an assistant a question and the assistant fetches your page to answer it, several vendors state that robots.txt rules may not apply, or that the fetcher generally ignores them.
How it works
The file lives at the root of a host, at an address like [yourdomain.com]/robots.txt, and it is written in groups. Each group starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow rules. To block a crawler from the whole site, the group is two lines: the User-agent line with the token, and Disallow: / with nothing after the slash.
RFC 9309 sets the matching rules, and two of them decide whether a block works the way you expect. First, the most specific rule wins, so a narrow Allow inside a group can open a path that a broad Disallow in the same group closed. Second, a group that names a crawler replaces the User-agent: * group for that crawler. Once you name a crawler, the broad group stops applying to it, and the named group has to carry the whole policy for that crawler on its own. A file that blocks a crawler in the broad group and then names it in an allow group has unblocked it, because the named group wins.
The tokens below are the ones each vendor documents on its own pages. Match on the token, not the full user agent string, because the version number in the string changes. Check the vendor docs in Sources before you copy a rule from any article, including this one.
OpenAI documents four tokens. GPTBot is the training crawler, and disallowing it signals that the site's content should not be used in training OpenAI's models. OAI-SearchBot is the search crawler, and sites that opt out of it will not be shown in ChatGPT search answers. ChatGPT-User fetches a page when a person asks ChatGPT about it, and OpenAI states that robots.txt rules may not apply to it because the fetch was started by a user. OAI-AdsBot only visits pages submitted as ads. OpenAI states that each setting is independent, so blocking GPTBot does not block search.
Anthropic documents three. ClaudeBot collects web content that could contribute to training, and restricting it signals that the site's future materials should be excluded from training datasets. Claude-SearchBot navigates the web to improve search result quality, and disabling it prevents indexing for search. Claude-User fetches a page when a person asks a question that needs it. Anthropic states that its bots honor robots.txt, and it warns against blocking by IP address, because blocking the addresses stops the bots from reading the file that tells them to stay away.
Perplexity documents two. PerplexityBot is designed to surface and link sites in Perplexity search results, and Perplexity states it is not used to crawl content for training. Blocking it takes the site out of Perplexity results. Perplexity-User visits a page when a question needs it, and Perplexity states that since a user requested the fetch, this fetcher generally ignores robots.txt rules.
Google works differently. Googlebot is the search crawler, and it always respects robots.txt rules for automatic crawls, so blocking Googlebot takes the site out of Google Search. Google-Extended is not a crawler. It is a product token with no user agent string of its own, used in robots.txt to manage whether content Google already crawls may be used to train future Gemini models and for grounding, which is providing content from the search index to the model at prompt time. Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.
Meta documents five crawlers. meta-externalagent crawls for uses such as training foundation AI models, and blocking it is the training opt out. meta-webindexer crawls to improve Meta AI search results, and blocking it takes the site out of Meta AI citations. meta-externalfetcher fetches individual links at a user's request, and Meta states it may bypass robots.txt rules. facebookexternalhit fetches shared links to build the title, description, and thumbnail preview, and blocking it breaks link previews when pages are shared on Facebook, Instagram, or Messenger. meta-externalads crawls pages submitted for ads.
One more caution before the steps. Do not write rules for crawler tokens you cannot verify in a vendor's own documentation. Third party bot lists name tokens that no vendor documents, and a rule for an undocumented token is a guess. The tokens in this guide are the documented ones.
The steps
Step 1. Decide the goal in one sentence. Write down what you want the block to achieve: no training, no AI search visibility, or no AI access at all. The sentence decides the tokens.
Step 2. List the tokens for that goal. For no training, the list is GPTBot, ClaudeBot, meta-externalagent, and Google-Extended. For no AI search visibility, add OAI-SearchBot, Claude-SearchBot, PerplexityBot, and meta-webindexer. For no AI access at all, add the user fetchers too, ChatGPT-User, Claude-User, and Perplexity-User, and meta-externalfetcher, knowing the vendors state that several of these may ignore the file.
Step 3. Write one group per token. Each group is two lines: the User-agent line with the token, and Disallow: / with nothing after the slash. Write the groups together near the top of the file, before any broad rules, so they are easy to find and audit. Make each group complete on its own, because a named group replaces the broad group for that crawler.
Step 4. Decide what happens to the rest of the web. A file that blocks only the AI tokens leaves every other crawler, including Googlebot, free to crawl. That is usually what a store wants: out of AI training, still in search. If you want everything blocked, add a User-agent: * group with Disallow: /, and understand that it blocks the search engines too.
Step 5. Keep link previews working unless you mean to break them. facebookexternalhit builds the preview when a page is shared on Facebook, Instagram, or Messenger. Blocking it does not protect the page. It only makes shared links show up bare, with no title, description, or image. Leave it allowed unless the goal is to stop previews as well as crawling.
Step 6. Do not enforce the block by IP address. Anthropic warns that blocking its IP addresses may not work correctly or persistently guarantee an opt out, because it stops the bots from reading the file that tells them to stay away. The same logic applies to the other vendors, which publish IP ranges so you can verify a visitor, not so you can firewall the opt out. Put the policy in the file, and verify visitors against the published ranges instead.
Step 7. Enforce at the edge if the goal is a real block. A robots.txt block is a request, and the user triggered fetchers are documented as ignoring it. If the goal is no AI access at all, the file is not enough. The enforceable control sits in front of the site, in the firewall or bot manager, matching the user agent token and the published IP ranges together. Perplexity publishes its own WAF configuration guidance for this, and Cloudflare documents its verified bot controls. Check the vendor docs in Sources for the current path.
Step 8. Wait, then verify in the logs. OpenAI, Perplexity, and Meta all state that changes can take about a day to reach their systems. Do not judge the block on the same day you write it. After a day, check the server logs for the blocked tokens, and confirm they have stopped arriving with successful responses.
The machine-readable piece
This is a robots.txt file for a hypothetical store that wants out of AI training but wants to stay visible in AI search answers and in Google Search. It blocks the training crawlers, leaves the search crawlers and Googlebot alone, and keeps the private paths closed to every crawler. Replace the placeholders in square brackets with your own domain.
# Opt out of AI training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Google-Extended
Disallow: /
# Search crawlers stay allowed, so the store can still
# appear in AI answers and in Google Search
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: Googlebot
Allow: /
# Private paths, closed to every crawler
User-agent: *
Disallow: /checkout/
Disallow: /account/
Disallow: /search?
Sitemap: https://[yourdomain.com]/sitemap.xml
Check the file after you publish it: fetch it from a browser and confirm every group is present.
What it looks like when it works
On the page, nothing changes for a shopper. Under the surface, the robots.txt file answers each training crawler with a Disallow, and the server logs show those crawlers stop arriving, while the search crawlers keep arriving with successful responses. When a shopper asks an assistant about a product the store sells, the assistant can still retrieve the page and cite it, because the search crawlers were left alone.
A hypothetical example
Say you run a hypothetical handmade soap brand. You sell small batch bars and gift sets, and your product descriptions carry the recipes and the sourcing notes you spent years writing. You want shoppers to find you in AI answers, but you do not want those descriptions used to train models. Your file blocks GPTBot, ClaudeBot, meta-externalagent, and Google-Extended, and allows OAI-SearchBot, Claude-SearchBot, PerplexityBot, and meta-webindexer. A shopper asks an assistant for a handmade soap gift set under 40 dollars. The assistant retrieves your product page, verifies the price, and recommends your gift set with a link, because the search crawlers were allowed in.
The fundamentals
These mechanics are shared across every store and covered in full in their own guides:
- •The llms.txt format and a complete sample, see our guide to writing a good llms.txt file
- •The markup AI systems read to verify facts, see our product schema guide
- •The complete crawler rule reference, see our robots.txt guide for AI search
What this guide does not cover
This guide covers blocking AI crawlers in robots.txt: which tokens to name, what blocking each one costs, and where the file stops being enough. It does not cover the edge enforcement in depth, which is the subject of the guide on why an allow rule still fails, and it does not cover testing the file, which has its own guide. It also does not cover the legal questions around training data, which are a matter for the brand's own counsel, not a robots.txt file. This guide is the map for the block, not the whole trip.
Check your own store now
Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan
Run the free scanSources
- RFC 9309, Robots Exclusion Protocol (specific): https://www.rfc-editor.org/rfc/rfc9309.html
- OpenAI, Overview of OpenAI crawlers (specific): https://platform.openai.com/docs/bots
- Perplexity, Perplexity crawlers (specific): https://docs.perplexity.ai/guides/bots
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler (specific): https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Google, Overview of Google crawlers and fetchers (specific): https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
- Meta, Meta web crawlers (specific): https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
- Cloudflare, Verified bots (general docs): https://developers.cloudflare.com/bots/concepts/bot/verified-bots/
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
