Start by finding out which crawler is actually causing the load, then slow that crawler down at the server, and block it only as a last resort. Blocking a search crawler to fix load also removes your brand from that engine's answers, and for most stores that trade is far worse than the load. The levers, in order, are: cut the low-value paths crawlers waste requests on, slow the named crawlers that document a slow-down signal, rate limit at the server or firewall with a proper 429 response, and cache the pages crawlers ask for most. Robots.txt is the weakest of these levers, not the strongest, because it is a request that only well-behaved crawlers follow, and it has no rate limit in the standard at all.
Step 1: Prove which bot is doing it
Do not guess from a slow site. Open your server logs and grep for the crawler tokens, case-insensitive, one family at a time.
grep -i "gptbot\|oai-searchbot\|chatgpt-user" /var/log/nginx/access.log
grep -i "claudebot\|claude-searchbot\|claude-user" /var/log/nginx/access.log
grep -i "perplexitybot\|perplexity-user" /var/log/nginx/access.log
grep -i "googlebot\|bingbot" /var/log/nginx/access.log
Count the hits per token, and look at which paths they request. A bot that hammers faceted search URLs, filtered category pages, and internal search results is a different problem from a bot that crawls your product pages once each. Also check the timing: bursts that line up with a sale or a new catalog drop are a different problem from steady background crawling. Then verify the hits are real. A user-agent string on its own is easy to copy, so check the requesting IP against the vendor's published list: OpenAI publishes per-bot IP ranges, Perplexity publishes per-bot ranges, and Anthropic publishes its crawler ranges at claude.com/crawling/bots.json. If the IPs do not match, the load is not coming from the vendor whose name is on the string, and a robots.txt rule for that vendor will not stop it. That traffic needs a firewall rule, not a file edit.
Step 2: Cut the low-value paths first
Most crawler load on a store is wasted on URLs no customer and no model needs: faceted navigation with every filter combination, internal search result pages, sort and pagination variants of the same category, and preview or staging paths. Disallow those paths in robots.txt, and repeat the disallows inside each named crawler group, because a named group stands alone and does not inherit the wildcard group's rules.
User-agent: *
Disallow: /search?
Disallow: /collections/*?*filter
Disallow: /account/
Disallow: /checkout/
User-agent: GPTBot
Allow: /
Disallow: /search?
Disallow: /collections/*?*filter
Disallow: /account/
Disallow: /checkout/
Change the paths to match your own store, and keep the product and category pages open. This step alone usually removes the bulk of the load, because it stops crawlers from spending requests on near-duplicate pages. It also helps the crawlers: fewer wasted fetches means more of their attention goes to the pages that carry your products. Check your sitemap at the same time, and keep it to the pages you want crawled, because a sitemap that lists every filtered variant is an invitation to crawl every filtered variant.
Step 3: Slow the named crawlers that document a slow-down signal
Where the vendor documents a slow-down signal, use it before you reach for a block. Anthropic supports the non-standard Crawl-delay extension to robots.txt, which asks the bot to wait between requests, and its example sets a delay of one second for ClaudeBot.
User-agent: ClaudeBot
Crawl-delay: 1
Allow: /
Crawl-delay is not part of the robots.txt standard, RFC 9309, so support varies by crawler and you cannot assume it works everywhere. Cloudflare's docs say Google does not recognize the command, and Google manages its own crawl pace instead. That means a Crawl-delay line will not slow Googlebot, and it will not slow a crawler that ignores the file. Treat it as one lever for one vendor, not as a general throttle. If the load comes from Anthropic's bots, it is the right first lever. If the load comes from Googlebot, the lever is server-side, covered in the next step.
Step 4: Rate limit at the server or firewall, with a proper 429
When a polite signal is not enough, enforce the limit yourself. Cloudflare's docs list rate limiting through a web application firewall as one of the steps for limiting AI crawlers, alongside updating robots.txt, using meta tags, and separating good bots from bad ones. The shape of the rule is: match the crawler's user agent together with its published IP ranges, set a rate that real crawling stays under and abusive bursts exceed, and return HTTP 429, too many requests, with a Retry-After header that tells the crawler how long to wait. A 429 is a signal a well-behaved crawler understands, and it is different from a block page: it says slow down, not go away. Keep the limit generous at first, watch the logs, and tighten it only if the load continues. Do not answer load with an IP block for Anthropic's bots: Anthropic warns that blocking its IP addresses may not work correctly or persistently guarantee an opt-out, because it stops the bots from reading your robots.txt file. A rate limit that still lets the bot read the file is the safer shape.
Step 5: Cache the pages crawlers ask for most
Crawlers ask for the same pages repeatedly, and every one of those requests costs your server a fresh render if nothing is cached. Cache the product and category pages at the edge or in front of your origin, so a crawler request is answered from cache instead of rendered from scratch. Check that your pages do not force a fresh render for a bot that a human would get from cache. This step attacks the cost per request instead of the count of requests. It also helps your human visitors.
What not to do
Do not block the search crawlers as a first step. Blocking OAI-SearchBot removes you from ChatGPT search answers, blocking Claude-SearchBot stops Anthropic from indexing your content for search, and blocking PerplexityBot keeps you out of Perplexity results. Those are visibility decisions, not load decisions, and they should be made as visibility decisions, with the cost understood. Do not block by IP address as a first step, for the Anthropic reason above and because IP ranges change. Do not return a block page with a 200 status and expect a crawler to learn from it: a 200 says the page is fine, so the crawler has no reason to slow down. Do not add a Crawl-delay line and assume every crawler honors it, because the standard does not define it and Google does not recognize it. And do not treat the absence of a token as proof of absence of load: crawlers that no vendor documents, and fetches that arrive as ordinary browsers, will not show up under any token you grep for, and they need firewall and bot management rules, not more lines in robots.txt.
The fundamentals
These mechanics are shared across every engine and covered in full in their own guides:
- •The complete rule reference for every crawler and edge case, see our robots.txt guide for AI search
- •What goes in llms.txt and how to publish it, see our guide to writing a good llms.txt file
- •The markup AI engines read to state price and availability in structured form, see our product schema guide for AI search
Say you run a toy store
You sell toys and games, and your site slows to a crawl every afternoon. Your host blames AI crawlers, and your first instinct is to block them all. Instead you grep the logs. The bulk of the requests are not product pages at all: they are filtered category URLs, one for every combination of age range, brand, and price sort, plus your internal search pages. A smaller share is ClaudeBot, visiting steadily. You disallow the filtered and search paths in every crawler group, add Crawl-delay: 1 to the ClaudeBot group, and set a firewall rate limit that returns 429 with a Retry-After header for bursts over a generous threshold. You also cache the category pages at the edge. The next day the load is down, the product pages are still open to every search crawler, and the logs show the crawlers spending their requests on pages that matter. Nothing was blocked, and the site is faster for customers too. That is the order that works: diagnose, cut the waste, slow the named bot, enforce the rate, cache the rest.
What this guide does not cover
This guide covers load from declared crawlers that identify themselves. It does not cover scrapers that ignore robots.txt and arrive as ordinary browsers, which need bot management and firewall rules rather than file edits. It does not cover the allow-or-block decision itself, which is a visibility and training choice with its own guide, and it does not cover monitoring over time: a fix that works this week can drift as your catalog grows, your filters multiply, and new crawlers appear, so the log check needs repeating.
Check your own site now
Run the free scan. One click, no signup, and it shows which AI crawlers your robots.txt actually allows: aizii.ai/aeo-scan
Run the free scanSources
- Cloudflare robots.txt explainer (general docs): https://www.cloudflare.com/learning/bots/what-is-robots-txt/
- Cloudflare how to block AI crawlers (general docs): https://www.cloudflare.com/en-in/learning/ai/how-to-block-ai-crawlers/
- Anthropic Help Center, does Anthropic crawl data from the web (specific): https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- OpenAI bots documentation (specific): https://platform.openai.com/docs/bots
- Perplexity bots documentation (specific): https://docs.perplexity.ai/guides/bots
- RFC 9309, Robots Exclusion Protocol (general standard): https://www.rfc-editor.org/rfc/rfc9309.html
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
