You test a robots.txt file in three moves. First, fetch the file itself and read the rules. Second, request a product page while pretending to be each AI crawler, and watch the response code. Third, check the server logs to see whether the real crawlers are arriving and what response they get. If all three agree, the file works. If they disagree, the disagreement tells you where the problem sits: in the file, in the firewall in front of the site, or in the wait for the vendors to pick up the change.
The test matters because a file can look right and still fail. A typo in a token name means the crawler never matches the group. A rule in the wrong group means the crawler follows a different group than the one you wrote. A firewall in front of the site can block the crawler before the file is ever read. And a change made this morning may not reach the vendors' systems until tomorrow.
How it works
A robots.txt file lives at the root of a host, at an address like [yourdomain.com]/robots.txt, and it is written in groups. Each group starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow rules. RFC 9309, the standard behind the file, sets the matching rules. The most specific rule wins. If an Allow rule and a Disallow rule match the same address with the same length, the Allow rule should win. If no rule matches, the address is allowed. If a crawler has no group of its own, it follows the group for User-agent: *, if one exists.
The response code is the second thing to understand. When you request a page, the server answers with a status code. A 200 means the request succeeded and the page was served. A 403 means the server understood the request and refused it, which is what a firewall block looks like. A 404 means the address does not exist. A code in the 500 range means the server failed, and RFC 9309 states that when the robots.txt file itself cannot be fetched because of a server error, crawlers must assume everything is disallowed.
The third thing to understand is the wait. OpenAI states that it can take about a day from a robots.txt update for its search systems to adjust. Perplexity states it can take up to a day for its systems to reflect a change. Meta states that changes can take up to a day to take effect, because its crawlers cache the file. A test run on the same day you change the file proves nothing about the new rules. Run the tests below, then run them again the next day, and judge the change on the second run.
The tokens you test with are the ones each vendor documents. OpenAI documents OAI-SearchBot for search, GPTBot for training, and ChatGPT-User for user triggered fetches, and states that robots.txt rules may not apply to ChatGPT-User because the fetch was started by a user. Perplexity documents PerplexityBot for search and Perplexity-User for user triggered fetches, and states that Perplexity-User generally ignores robots.txt rules. Anthropic documents ClaudeBot for training, Claude-SearchBot for search, and Claude-User for user triggered fetches. Google documents Googlebot for search and Google-Extended as a training and grounding token with no crawler of its own. Meta documents meta-externalagent for training, meta-webindexer for Meta AI search, and meta-externalfetcher for user requested fetches, which Meta states may bypass robots.txt rules.
The steps
Step 1. Fetch the file in a browser. Open [yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain, and read every group. Confirm the tokens are spelled exactly as the vendor writes them, confirm each group carries the rules you meant it to carry, and confirm the file returns a 200 response. If the file returns an error, stop and fix the file first. Do the same for every host the store uses: a file on the main domain does not cover a subdomain, and a file served over plain http does not cover the https site.
Step 2. Request the file as each crawler. Use the curl command from the machine-readable piece below to fetch the robots.txt file with each crawler's user agent string. The file should return a 200 for every crawler you allow, and the content should be the same file you read in step 1. If a crawler gets a different file, or an error, the site is serving different content by user agent, and the file you read in your browser is not the file that crawler reads.
Step 3. Request a product page as each crawler. Use curl again, this time on a product page, with each crawler's user agent string. A page the crawler is allowed to read should return a 200. Compare the result for the crawler with the result for your own browser. If the browser gets a 200 and the crawler gets a 403 on a page the file allows, the block sits in front of the site, in the firewall or bot manager, not in the file.
Step 4. Check the groups against the matching rules. Look at the group for each crawler and ask two questions. Does the crawler have a group of its own, or does it fall through to the User-agent: * group? And does any Allow rule in its group conflict with a Disallow rule for the same path? A named group replaces the broad group for that crawler, so a named group with no Disallow lines allows everything, even if the broad group blocks the same paths.
Step 5. Check the server logs for the real crawlers. Search the access logs for the tokens you allow, such as OAI-SearchBot, PerplexityBot, Claude-SearchBot, and meta-webindexer. Confirm they appear, confirm they request product pages rather than only the robots.txt file, and confirm the response codes are 200s. Confirm the blocked tokens do not appear with 200s on pages they should not read. OpenAI and Perplexity publish IP ranges for their crawlers, and Anthropic publishes its crawler prefixes, so check a suspicious hit against the published ranges as well as the token, because a user agent string on its own can be copied by anyone.
Step 6. Check Google separately. Googlebot is the crawler behind Google Search and AI Overviews, and Google provides its own check. In Search Console, use the URL inspection tool to see what Googlebot receives for a product page, including the rendered page. Google's guide to generative AI features states that a page must be indexed and eligible to be shown in search with a snippet to appear in its generative features. Check the vendor docs in Sources for the current path to the tool.
Step 7. Wait a day and run the tests again. Changes take about a day to reach the vendors' systems, so the first run tells you whether the file and the edge are set up right, and the second run tells you whether the vendors have picked the change up.
The machine-readable piece
This is a set of curl commands for testing a robots.txt file and a product page as two different AI crawlers. Curl is a command line tool that requests a web address and shows the response. The -A flag sets the user agent string the request carries, and the -I flag asks for the headers only, which is where the status code appears. Replace the placeholders in square brackets with your own domain and product address. The user agent strings below are the ones OpenAI and Perplexity publish in their own documentation, and the version numbers may change, so check the vendor docs in Sources for the current strings.
# Fetch the robots.txt file as OAI-SearchBot
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" -I https://[yourdomain.com]/robots.txt
# Fetch a product page as OAI-SearchBot
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" -I https://[yourdomain.com]/products/[your-product]
# Fetch the same product page as PerplexityBot
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" -I https://[yourdomain.com]/products/[your-product]
# Fetch the same product page as an ordinary browser, for comparison
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36" -I https://[yourdomain.com]/products/[your-product]
Read the results like this: a 200 on the robots.txt file means the file loads. A 200 on the product page for a crawler means the page is reachable by that crawler. A 403 for a crawler where the browser gets a 200 means something in front of the site is blocking that crawler. A 500 range code on the robots.txt file means crawlers must assume a full disallow, so fix the server error before anything else. A 404 on the product page means the address is wrong, so check the address and run the command again.
What it looks like when it works
On the page, nothing changes for a shopper. Under the surface, the robots.txt file loads with a 200, the curl tests return 200s for the crawlers you allow, and the browser comparison returns a 200 as well. The server logs show the allowed crawlers arriving on product pages with 200 responses, and the blocked crawlers either absent or arriving and being refused. The next day, the same tests return the same results, and the logs show the crawlers returning on their own schedule. When a shopper asks an assistant about a product the store sells, the assistant can retrieve the page, because the path from the file to the page is open and tested.
A hypothetical example
Say you run a hypothetical camping gear store. Your robots.txt allows OAI-SearchBot and PerplexityBot, and a shopper asks an assistant for a two person tent under 200 dollars. The assistant never names your tent. You run the curl test from the machine-readable piece, and the product page returns a 200 for your browser and a 403 for OAI-SearchBot. Your robots.txt file is correct, so the file is not the problem. You check the bot manager in front of the site and find a rule that challenges any visitor whose user agent contains the word bot. You add an exception for the AI search crawlers, run the curl test again, and the page returns a 200 for OAI-SearchBot. The next day, the logs show OAI-SearchBot fetching product pages with 200s, and the assistant starts naming your tent.
The fundamentals
These mechanics are shared across every store and covered in full in their own guides:
- •The llms.txt format and a complete sample, see our guide to writing a good llms.txt file
- •The markup AI systems read to verify facts, see our product schema guide
- •The complete crawler rule reference, see our robots.txt guide for AI search
What this guide does not cover
This guide covers testing the robots.txt file: fetching the file, requesting pages as each crawler, reading the response codes, and checking the logs. It does not cover fixing the edge in front of your site, which is the subject of the guide on why an allow rule still fails, and it does not cover writing the file itself, which has its own guide. It also does not cover what happens after a crawler reads your pages, which is the job of structured data and the llms.txt file, and it does not cover the transaction layer that lets an agent complete a purchase. This guide is the map for the test, not the whole trip.
Check your own store now
Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan
Run the free scanSources
- RFC 9309, Robots Exclusion Protocol (specific): https://www.rfc-editor.org/rfc/rfc9309.html
- OpenAI, Overview of OpenAI crawlers (specific): https://platform.openai.com/docs/bots
- Perplexity, Perplexity crawlers (specific): https://docs.perplexity.ai/guides/bots
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler (general docs): https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Google, Overview of Google crawlers and fetchers (general docs): https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
- Google Search Central, Optimizing your website for generative AI features on Google Search (specific): https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- Meta, Meta web crawlers (general docs): https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
