Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# How to rate-limit AI crawlers without blocking them

Rate limiting means slowing a client down when it asks for too much, too fast, while still letting it in. Blocking means shutting the door. For AI crawlers, the difference matters: a crawler that is blocked cannot read your pages at all, while a crawler that is slowed can still read them, just at a pace your server can handle. The tools for the job are the edge rule that counts requests and answers the excess with a 429 status, and the robots.txt line that asks a polite crawler to wait between requests. The plan in this guide uses both, and checks what actually happened in the logs.

Say this first, because it shapes every step. Robots.txt rules are a request, not a lock. The standard behind robots.txt, RFC 9309, says plainly that its rules are not a form of access authorization. A crawler can ignore them. Rate limiting at the edge is different: it is enforced by your own server or CDN, so it works whether the crawler reads your robots file or not. Use robots.txt to ask, and use the edge to enforce, and do not confuse the two.

Also say this plainly about the robots.txt slow-down line. Crawl-delay is not part of RFC 9309. Google ignores it: Google supports only the user-agent, allow, disallow, and sitemap fields, and its advice for an overloaded server is to return a 500, 503, or 429 status for a short period instead. Bing documents support for crawl-delay. So a crawl-delay line is worth adding for the crawlers that document support, and worthless as a way to slow Google down. Do not put it in the catch-all group and expect it to work everywhere.

## Where it goes wrong

Rate limiting fails in a few specific ways, and most of them come from limiting the wrong thing, or limiting everyone at once.

-   The rule blocks instead of slowing. A rule that returns a block page to every request over the limit, with no retry signal, looks the same as a ban to the crawler. Use a 429 status with a Retry-After header, which tells the client when it can try again. Cloudflare uses 429 as the default response code for a blocked rate limit rule, and the response includes a Retry-After header.
-   The rule counts the wrong requests. Count only the requests that reach the origin, and scope the rule to the paths or the user-agents that are actually causing the load. A rule that counts every request from every visitor will throttle real customers during a sale. Cloudflare lets you disable counting cached assets so only requests that reach the origin count toward the rate.
-   The rule treats all crawlers as one client. A shared limit for every crawler means one busy crawler eats the allowance for all of them, and a quiet crawler gets throttled for a neighbor's traffic. Count per crawler where the tool allows it, by user-agent or by the characteristics the rule groups on, so each crawler gets its own counter.
-   The crawl-delay line sits in the wrong group. A crawl-delay line in the catch-all group is ignored by Google and may be ignored by crawlers that only read their own group. Put the line in the group for the crawler that documents support for it, and keep it out of groups for crawlers that do not.
-   The edge blocks the crawler before the rule can allow it. A bot management rule that challenges every automated client, or a managed robots file that blocks AI crawlers, will stop a crawler that your rate limit rule was meant to slow. Check the whole chain, in the order a request meets them: the AI crawler settings, the bot rules, and the rate limit rules.
-   Nobody watches the logs after the rule goes live. A rule with no monitoring is a guess that never gets tested. Confirm the crawler is still visiting and still reaching the pages that matter. A crawler that disappears from the logs after a rule goes live is a crawler you blocked, not one you slowed.

## The steps

**Step 1. Find out which crawler is actually causing the load.** The developer opens the server logs or the CDN analytics and lists the automated clients by request count, over a busy day, not a quiet one. Look for the AI crawler names and the paths they hit most: the search pages, the filtered category pages, and the feed addresses are the usual suspects. Do not write a rule until you can name the client and the paths.

**Step 2. Decide the limit from your own numbers.** The developer takes the busiest crawler from step one, counts its requests per minute at its busiest hour, and sets the limit above that peak with room to spare, so normal traffic never trips it. The limit is a ceiling for abuse, not a target for normal traffic. Write the number down with the log window it came from, so the next person to touch the rule knows why it is what it is.

**Step 3. Add the crawl-delay line for the crawlers that document support.** The developer adds a crawl-delay line to the robots.txt group for each crawler that documents support for it, asking for a pause between requests. Keep the line out of the catch-all group and out of the groups for crawlers that ignore it, including Google. Check the vendor docs in Sources before relying on a claim about which crawler supports it.

**Step 4. Build the edge rule that enforces the limit.** The developer creates a rate limiting rule in the CDN or firewall, scoped to the crawler and the paths from step one. The rule has three parts: the matching criteria that pick the requests, the rate that sets the maximum requests per period, and the action that applies when the rate is exceeded. In Cloudflare, open the Security rules page, create a rate limiting rule, set the match, the counting characteristics, the rate, and the action, and deploy. Set the action to block with a 429 status and a Retry-After header, and never use a challenge on a machine client that cannot pass one. Check the vendor docs in Sources for the current path, since platform screens change.

**Step 5. Let verified crawlers through and keep the challenge for the rest.** The developer checks the bot management settings and the AI crawler settings before the rule goes live. Allow the verified search and agent crawlers you want, keep the limit for the traffic that is actually overloading the server, and keep the challenge for unknown automated clients. A rule that treats a verified crawler like an unknown scraper will cost you the visits you wanted.

**Step 6. Watch the logs and adjust.** The developer watches the logs for a full busy day after the rule goes live: the request counts per crawler, the 429 responses served, and whether the crawler still visits the pages that matter. If the crawler disappears, the rule is too tight. If the server is still overloaded, the problem is elsewhere. Put a review date on the rule and revisit it, so a temporary limit does not become a permanent throttle.

## The machine-readable piece

This is the robots.txt part. The crawl-delay line sits in the group for a crawler that documents support for it, and stays out of the catch-all group and out of the Google group, since Google ignores it. The catch-all group allows the paths the store wants read. Replace the placeholders with your own paths.

```
User-agent: Bingbot
Allow: /
Crawl-delay: 10

User-agent: *
Allow: /
```

Do not copy the number above as your own setting. It is an example of the shape, not a recommendation. Your number comes from your own logs in step two.

This is the edge rule, written as a settings table in the shape Cloudflare documents for a rate limiting rule: matching criteria, rate, and action. Fill it with the client and the paths from step one, and the limit from step two. The values below are an example, not a recommendation.

```
Rule name: Slow AI crawler on search pages
Match: User-Agent contains the crawler name, and URI Path starts with /search
Count by: IP address
Rate: 100 requests per 10 minutes
Action: Block with a 429 status and a Retry-After header
Duration: 10 minutes
```

Check the rule after you deploy it. Open the logs and confirm the crawler still visits, the 429 responses only appear above the limit, and shoppers are not caught by the rule. If the rule fires during normal traffic, the limit is too low or the match is too broad.

## What it looks like when it works

On the page, nothing looks different to a shopper. The store loads at the same speed, the search works, and the checkout works. Under the surface, the busiest crawler still visits, still reads the pages that matter, and still appears in the logs, but its bursts are cut off at the limit and the excess requests get a 429 answer with a time to try again. A shopper who asks an assistant about a product still gets an answer drawn from the current pages, because the crawler that reads those pages was slowed, not shut out.

Say you run a pet supplies store, selling dog food, cat food, treats, and toys, with a search page that lets shoppers filter by brand, flavor, and size. The developer checks the logs and finds one AI crawler hitting the filtered search addresses hundreds of times an hour, while the product pages get a steady, normal pace. The developer counts the crawler's peak rate, sets a limit above it, adds a crawl-delay line to that crawler's robots.txt group, and builds an edge rule that answers the excess with a 429 status and a Retry-After header. The next day the logs show the crawler still visiting the product pages at its normal pace, the search bursts cut off at the limit, and the server load back to normal. A shopper asks an assistant for a grain-free dog food for a sensitive stomach. The assistant can find the current category page and the current product pages, and cite the store instead of a stale listing.

### The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

-   •The llms.txt format and a complete sample, see our [guide to writing a good llms.txt file](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The markup AI systems read to verify facts, see our [product schema guide](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)
-   •The complete crawler rule reference, see our [robots.txt guide for AI search](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

## What this guide does not cover

This guide covers slowing AI crawlers down without shutting them out. It does not cover the full technical audit that checks every crawler and every page, the transaction layer that lets an agent complete a purchase, or the per-engine playbooks for getting each assistant to find you. It does not cover blocking a crawler outright, which has its own guide, and it does not cover fixing a slow server, which is a hosting and caching job. Those are separate jobs, and this guide is the map, not the whole trip.

## Check your own store now

Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan

[Run the free scan](https://aizii.ai/aeo-audit)

## Sources

-   RFC 9309, Robots Exclusion Protocol (specific): [https://www.rfc-editor.org/rfc/rfc9309.txt](https://www.rfc-editor.org/rfc/rfc9309.txt)
-   Google Search Central, How Google interprets the robots.txt specification (specific): [https://developers.google.com/search/docs/crawling-indexing/robots/robots\_txt](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
-   Cloudflare, Create a rate limiting rule in the dashboard (specific): [https://developers.cloudflare.com/waf/rate-limiting-rules/create-zone-dashboard/](https://developers.cloudflare.com/waf/rate-limiting-rules/create-zone-dashboard/)
-   Cloudflare, Verified bots (specific): [https://developers.cloudflare.com/bots/concepts/bot/verified-bots/](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/)
-   Google Search Central, Optimizing your website for generative AI features on Google Search (specific): [https://developers.google.com/search/docs/fundamentals/ai-optimization-guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
