robots.txt is a small plain text file at the root of your site that tells crawlers which pages they may request. It is the first file an AI crawler reads, and it is the only control most stores ever set for AI access. The file has two jobs: let the search crawlers in, and keep the training crawlers out if that is your choice. One warning first: the file is a request, not a lock. It tells a crawler what you want. It cannot force a crawler to obey, it cannot stop a fetch that a user asked for, and it cannot reach past the firewall in front of your site.
The most common mistake is treating all AI crawlers as one thing. They are not. Each vendor runs several crawlers with different jobs, and each one takes its own line in the file. Block the training crawler and you have opted out of training. Block the search crawler and you have opted out of being recommended.
How it works
The file lives at the root of a host, at an address like [yourdomain.com]/robots.txt, and it is written in groups. Each group starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow rules for that crawler. The rules are path patterns matched from the start of the address. The format is defined by RFC 9309, the Robots Exclusion Protocol, which states plainly that these rules are not a form of access authorization. A crawler that ignores the file breaks no lock. It breaks a convention.
RFC 9309 sets the matching rules. The most specific rule wins, meaning the rule whose pattern matches the most characters. If an Allow rule and a Disallow rule match the same address with the same length, the Allow rule should win. If no rule matches, the address is allowed. If a crawler has no group of its own, it follows the group for User-agent: *, if one exists. And if the file itself cannot be fetched because the server returns an error in the 500 range, crawlers must assume everything is disallowed.
Two lines people expect to find in the standard are not there. Crawl-delay, which asks a crawler to wait between requests, is not defined in RFC 9309. It is a vendor extension. Anthropic documents support for it, and other vendors handle it their own way, so do not write it expecting every crawler to honor it. The Sitemap line, which points crawlers at your sitemap, is also an extension rather than part of the core grammar, but it is widely supported.
The tokens are the stable part. A crawler's full user agent string carries a version number that changes, but the token stays the same, so match on the token in your file and in your logs. Check the vendor docs in Sources before you copy a rule from any article, including this one, because vendors add and rename crawlers.
OpenAI documents four. OAI-SearchBot is the search crawler, used to surface sites in ChatGPT search, and sites that opt out of it will not be shown in ChatGPT search answers. GPTBot is the training crawler, and disallowing it signals that the site's content should not be used in training. OpenAI states that each setting is independent, so a store can allow search while disallowing training, and it says changes can take about a day to reach its search systems. ChatGPT-User is not a crawler. It fetches a page when a person asks ChatGPT about it, and OpenAI states that robots.txt rules may not apply to it because the fetch was started by a user. OAI-AdsBot only visits pages submitted as ads on ChatGPT.
Anthropic documents three. ClaudeBot collects web content that could contribute to training, and restricting it signals that the site's future materials should be excluded from training datasets. Claude-SearchBot navigates the web to improve search result quality, and disabling it prevents indexing for search. Claude-User fetches a page when a person asks a question that needs it. Anthropic states that its bots honor robots.txt, and it warns against blocking by IP address, because blocking the addresses stops the bots from reading the file that tells them to stay away.
Perplexity documents two. PerplexityBot is designed to surface and link sites in Perplexity search results, and Perplexity states it is not used to crawl content for training. Perplexity-User supports user actions inside Perplexity, visiting a page when a question needs it, and Perplexity states that since a user requested the fetch, this fetcher generally ignores robots.txt rules. Each setting works independently, and Perplexity states it can take up to a day for its systems to reflect a change.
Google works differently. Googlebot is the search crawler, and it always respects robots.txt rules for automatic crawls. Google-Extended is not a crawler at all. It is a product token with no user agent string of its own, used in robots.txt to manage whether content Google already crawls may be used to train future Gemini models and for grounding, which is providing content from the search index to the model at prompt time. Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.
Meta documents five crawlers. meta-externalagent crawls for uses such as training foundation AI models or improving products by indexing content directly. meta-webindexer crawls to improve Meta AI search results, and allowing it helps Meta cite and link to a site's content in Meta AI responses. meta-externalfetcher fetches individual links at a user's request, and Meta states it may bypass robots.txt rules. facebookexternalhit fetches shared links to build the title, description, and thumbnail preview, and Meta states it may bypass the file for security or integrity checks. meta-externalads crawls pages submitted for ads. Meta states that changes can take up to a day to take effect, because its crawlers cache the file.
The steps
Step 1. Read the file you have today. Open [yourdomain.com]/robots.txt in a browser and read every group. Note which tokens are named, what each group disallows, and whether a broad User-agent: * group blocks paths you want crawled. If the file returns an error, fix that before anything else, because a server error tells crawlers to assume a full disallow.
Step 2. Decide training and search separately for each vendor. For each vendor, answer two questions: do you want to appear in that assistant's answers, and do you want your content used to train that vendor's models? Most stores answer yes to search and make their own call on training. Write the answers down before you touch the file.
Step 3. Write a group per crawler, and make each group complete. A group that names a crawler replaces the User-agent: * group for that crawler, so a named group with no Disallow lines allows everything. Put the search crawlers in allow groups and the training crawlers you want to opt out of in disallow groups, keeping the tokens exactly as the vendor writes them.
Step 4. Keep the paths you block narrow. Disallow the paths that should stay private, such as checkout, account, and search result pages, and allow the product and content pages. A Disallow pattern matches from the start of the address, so Disallow: /search blocks both a search page and any address that starts with those letters.
Step 5. Add the sitemap line. Put a Sitemap line at the end of the file pointing at your sitemap index, using the full address. If you run several sitemaps, list each one on its own line.
Step 6. Publish the file at the root of every host it should cover. A robots.txt file covers only the host, protocol, and port where it lives. A file on the main domain does not cover a subdomain, and a file served over plain http does not cover the https site. Check each host separately after you publish.
Step 7. Wait, then check the logs. OpenAI and Perplexity state that changes can take about a day to reach their systems, and Meta states the same for its cache. Do not judge the change on the same day you make it. After a day, check the server logs for the tokens you allowed, and confirm they arrive with successful responses.
The machine-readable piece
This is a complete robots.txt file for a hypothetical store that wants to appear in AI search answers but does not want its content used for training. It allows the search crawlers and the user fetchers, disallows the training crawlers, keeps checkout and account paths closed to all crawlers, and points at the sitemap. Replace the placeholders in square brackets with your own domain. Note that the user fetchers are allowed here, though some vendors state that robots.txt rules may not apply to user-requested fetches.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /checkout/
Disallow: /account/
Disallow: /search?
Sitemap: https://[yourdomain.com]/sitemap.xml
Check the file after you publish it: fetch it from a browser, confirm every group is present, and confirm the sitemap address loads.
What it looks like when it works
On the page, nothing changes for a shopper. Under the surface, the robots.txt file answers every crawler with a clear rule, the sitemap lists the pages, and the server logs show the search crawlers arriving and receiving successful responses. When a shopper asks an assistant about a product the store sells, the assistant can retrieve the product page and cite it, because the search crawler was allowed in. The training crawlers stay out, and the checkout and account paths stay closed to all crawlers.
A hypothetical example
Say you run a hypothetical specialty coffee roaster. You sell whole bean coffee by subscription, and your best seller is a medium roast from a single origin. A shopper asks an assistant for a medium roast whole bean coffee subscription under 25 dollars a bag. Your robots.txt allows OAI-SearchBot and PerplexityBot, and disallows GPTBot and ClaudeBot, because you want to be recommended but do not want your tasting notes used for training. The assistant retrieves the page, verifies the price, and recommends your roast with a link. Your competitor sells a similar roast for the same price, but their file blocks everything with a broad Disallow, written years ago when someone wanted to keep a staging site out of search, so no search crawler has ever read their product pages and the assistant never sees them.
The fundamentals
These mechanics are shared across every store and covered in full in their own guides:
- •The llms.txt format and a complete sample, see our guide to writing a good llms.txt file
- •The markup AI systems read to verify facts, see our product schema guide
- •The complete crawler rule reference, see our robots.txt guide for AI search
What this guide does not cover
This guide covers the robots.txt file itself: the format, the rules, the tokens for each vendor, and the checks that confirm the file works. It does not cover the edge in front of your site, which is the subject of the guide on why an allow rule still fails, and it does not cover testing the file in depth, which has its own guide. It also does not cover what happens after a crawler reads your pages, which is the job of structured data and the llms.txt file.
Check your own store now
Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan
Run the free scanSources
- RFC 9309, Robots Exclusion Protocol (specific): https://www.rfc-editor.org/rfc/rfc9309.html
- OpenAI, Overview of OpenAI crawlers (specific): https://platform.openai.com/docs/bots
- Perplexity, Perplexity crawlers (specific): https://docs.perplexity.ai/guides/bots
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler (specific): https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Google, Overview of Google crawlers and fetchers (specific): https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
- Meta, Meta web crawlers (specific): https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
