Your robots.txt file allows the AI crawlers, and they still never show up. The file is not lying, and it is not broken. It is simply not the thing standing between the crawler and your site. Between them sits the edge: the firewall, the bot manager, and the content delivery network in front of your store. If any of those blocks the crawler, the request dies before your server ever sees it, and the allow rule in your file never gets a vote.
This is the most common reason a correct robots.txt file fails, and it is also the most confusing, because every check you run from your own browser says the site is fine. Your browser is not a crawler. It carries a different user agent, comes from a different address, and passes every bot check without noticing. The crawler carries a user agent that says it is a bot, and the edge treats it like one.
How it works
Picture the request as a journey with four gates. The crawler asks for a page. The first gate is the edge: the content delivery network, the web application firewall, and the bot manager. The second gate is the server itself, which can refuse requests by user agent, by address, or by rate. The third gate is the robots.txt file, which the crawler reads and honors if it is a well behaved crawler. The fourth gate is the page, which can itself refuse the request or return an error. Your allow rule lives at the third gate.
The edge is where most blocks happen, and it blocks for reasons that have nothing to do with your file. A bot manager scores every visitor and challenges or blocks the ones that look automated, and AI crawlers look automated by definition, because they are. A firewall rule written to stop scrapers matches the same signals the AI crawlers send: a declared bot user agent, a burst of requests, an address range the vendor publishes. The edge does not read your robots.txt file before deciding.
Three edge setups cause most of the trouble. The first is a bot manager with a blanket rule against automated traffic, the kind that ships turned on by default or was switched on during a scraper attack and never revisited. The second is a managed firewall rule group for bot control, which scores requests and blocks the ones that look like bots, by user agent token or by address reputation, before the request reaches your origin. The third is a managed edge run by an agency or a platform, with its own bot policy that the store owner never sees.
The server itself can block too. Some hosts block by user agent, some block by country, and some rate limit any visitor that requests pages quickly. A server that returns a 403 to a crawler and a 200 to a browser is enforcing a rule somewhere, and the rule is not in the robots.txt file.
Two more causes sit outside the edge entirely. The first is the wrong file: a robots.txt file covers only the host, protocol, and port where it lives, so a file on the main domain does not cover a subdomain, and a file served over plain http does not cover the https site. If the crawler reads a different host than the one you checked, it reads a different file. The second is the wait: OpenAI states that it can take about a day from a robots.txt update for its search systems to adjust, Perplexity states it can take up to a day for its systems to reflect a change, and Meta states the same for its cache. A file changed this morning has not reached the vendors yet, and no amount of edge work changes that.
The last cause is the named group trap inside the file itself. RFC 9309 states that a group naming a crawler replaces the User-agent: * group for that crawler. If you allow a crawler in the broad group and then name it in a group that disallows a path, the named group wins for that crawler, and the broad allow no longer applies to it. The file looks like an allow and behaves like a block for that one crawler. Read the groups, not just the first line.
The steps
Step 1. Confirm what the file actually says, on the right host. Open [yourdomain.com]/robots.txt in a browser, replacing yourdomain.com with your actual domain, and read every group. Confirm the tokens are spelled exactly as the vendor writes them, confirm each group carries the rules you meant, and confirm the file returns a 200. Then check the other hosts: the www version, the apex domain, and any subdomain the store uses. A file that is correct on one host and missing on another is a file that works for one crawler path and fails for the other.
Step 2. Test from outside with a real bot user agent. Run the curl commands from the machine-readable piece below, from any machine outside your own network. Request the robots.txt file and a product page twice: once with a bot user agent, and once with an ordinary browser user agent. If the browser gets a 200 and the bot gets a 403, a 429, or a challenge page, the edge is blocking declared crawlers regardless of what the file says. Test a product page as well as the file, because some setups allow the file itself and block the crawler on content pages.
Step 3. Check the bot manager and firewall settings. Look at the bot management settings in front of your site for anything that challenges or blocks automated traffic, and check whether it distinguishes a declared search crawler from a scraper. Add explicit allowances for the AI search crawlers you want, using the tokens and the published IP ranges together, not the token alone. OpenAI publishes per bot IP ranges as JSON files, Anthropic publishes its crawler prefixes, and Perplexity publishes its bot IP lists and its own WAF configuration guidance. Check the vendor docs in Sources for the current path, because dashboard menus change.
Step 4. Check the server itself. If the edge is clean and the curl test still fails, look at the server. Check for rules that block by user agent, for country blocks, and for rate limits that trigger on bursts of requests. Check the server error log for the time of the failed test. A 403 from the server and a 403 from the edge look the same in curl, and only the logs tell you which layer answered.
Step 5. Confirm in the server logs whether the crawler ever arrives. The logs are the ground truth. Search the access logs for the tokens you allow, such as OAI-SearchBot, PerplexityBot, Claude-SearchBot, and meta-webindexer. If the logs show hits arriving with 200 responses, the path is clear and the problem was never access. If the file allows them, the edge allows them, and the logs still show no hits after a day or two, something further upstream is still filtering, or the crawlers simply have not come back yet.
Step 6. Wait a day and test again. Changes to the file take about a day to reach the vendors' systems, and changes to the edge take effect on their own schedule. Do not judge either change on the same day you make it. Run the curl test again the next day, and check the logs again. A fix that works shows up as 200s in the logs, not as a feeling that the site should work now.
The machine-readable piece
This is a robots.txt file that demonstrates the named group trap, followed by the curl test that catches an edge block. In the file below, the broad group allows everything, but the named group for OAI-SearchBot disallows the product pages. Because a named group replaces the broad group for that crawler, OAI-SearchBot is blocked from the product pages even though the file looks like an allow. The fix is to remove the Disallow line from the named group, or to add an Allow line for the product path in that group. The curl commands test the same product page twice, once as the crawler and once as a browser, and the comparison between the two results is the test.
# The trap: the broad group allows everything, but the
# named group below replaces it for OAI-SearchBot
User-agent: *
Allow: /
User-agent: OAI-SearchBot
Disallow: /products/
# Request a product page as OAI-SearchBot
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" -I https://[yourdomain.com]/products/[your-product]
# Request the same page as an ordinary browser, for comparison
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36" -I https://[yourdomain.com]/products/[your-product]
Read the results like this: a 200 for the browser and a 403 for the crawler means the edge or the server is blocking the crawler, not the file. A 200 for both means the path is open. A 404 for both means the product address is wrong. Check the vendor docs in Sources for the current user agent strings, because the version numbers change.
What it looks like when it works
On the page, nothing changes for a shopper. Under the surface, the robots.txt file allows the search crawlers, the edge allows them too, and the server logs show them arriving on product pages with 200 responses. When a shopper asks an assistant about a product the store sells, the assistant can retrieve the page and cite it, because every gate on the journey is open. The bot manager still blocks the scrapers it was set up to block, because the allowances are narrow: they name the declared crawlers and their published address ranges, not every visitor that claims to be a bot.
A hypothetical example
Say you run a hypothetical independent bookstore. Your robots.txt allows OAI-SearchBot and PerplexityBot, and a shopper asks an assistant for a literary fiction gift under 30 dollars. The assistant names three competitors and never names you. You open the file and it is correct, so you run the curl test from the machine-readable piece. Your browser gets a 200 on the product page, and OAI-SearchBot gets a 403. You open the bot manager in front of the site and find a rule that challenges any visitor whose user agent contains the word bot, switched on during a scraper attack and never revisited. You add an allowance for the AI search crawlers, using their tokens and their published address ranges, and run the curl test again. The page returns a 200 for OAI-SearchBot.
The fundamentals
These mechanics are shared across every store and covered in full in their own guides:
- •The llms.txt format and a complete sample, see our guide to writing a good llms.txt file
- •The markup AI systems read to verify facts, see our product schema guide
- •The complete crawler rule reference, see our robots.txt guide for AI search
What this guide does not cover
This guide covers the layers between the crawler and your pages: the edge, the server, the file, and the wait. It does not cover the fetchers that ignore the file entirely, such as the user triggered fetchers that several vendors state may bypass robots.txt rules, because no allowlist reaches those. It also does not cover monitoring: crawler behavior and firewall rules drift, and a setup that works today can silently close again after a vendor update. And access is not recommendation: once crawlers can reach your pages, they still need verifiable facts to cite, which is the product schema guide's job.
Check your own store now
Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan
Run the free scanSources
- OpenAI, Overview of OpenAI crawlers (specific): https://platform.openai.com/docs/bots
- Perplexity, Perplexity crawlers (specific): https://docs.perplexity.ai/guides/bots
- Meta, Meta web crawlers (specific): https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
- Cloudflare, Verified bots (general docs): https://developers.cloudflare.com/bots/concepts/bot/verified-bots/
- RFC 9309, Robots Exclusion Protocol (specific): https://www.rfc-editor.org/rfc/rfc9309.html
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
