Aizii Learn · AI Search
← Back to Learn

Why your AI readiness score gets capped when the edge blocks bots

Your AI readiness score measures what a scan can read. When the edge of your site blocks or challenges the bots that do the reading, the scan cannot see the pages, the files, or the markup it is scoring, and every check it cannot run counts as missing. That is the cap. It is not a penalty for bad content. It is a ceiling set by what the scan was allowed to see. Fix the access, and the score can move. Leave the block in place, and no amount of better copy will lift it, because the better copy is behind the same wall.

Say this plainly too: no vendor publishes a formula for the AI readiness score, and no vendor doc cited here says a blocked crawler costs a fixed number of points. The mechanism below is enough to act on. A scan reads your site the way an AI crawler reads it. If the edge stops the crawler, it stops the scan the same way, and the score reflects the unreadable site, not the real one.

The edge is the layer in front of your site: your content delivery network, your web application firewall, and your bot management settings. It answers requests before your server ever sees them. Your robots.txt file lives behind it. That order is the whole problem. A crawler that is blocked at the edge never reaches robots.txt, so an allow rule in robots.txt cannot rescue it.

Where it goes wrong

The cap shows up through a handful of edge settings, and most of them were switched on for a good reason, then forgotten.

  • An AI bot policy blocks search crawlers along with training crawlers. Cloudflare groups AI traffic by behavior: Search, Agent, and Training. Search crawlers collect content so it can answer questions later. Agent traffic acts in real time on a person's behalf. Training crawlers take content to train a model. A brand that wants to be cited needs the Search and Agent traffic allowed, and only the Training traffic is a separate policy choice. Blocking all three at once shuts the citation door while trying to shut the training door.
  • A managed robots.txt prepends disallow lines you never wrote. Some edge platforms can write AI disallow directives into the robots.txt that crawlers receive, even though the file on your server says allow. The file you check on your server and the file a crawler receives are then two different files, and the crawler obeys the one it receives.
  • Bot Fight Mode or a challenge page greets every non-browser visitor. A managed challenge asks the visitor to prove it is human, usually with a browser check that a crawler cannot pass. The crawler gets a challenge page instead of your product page, and the scan records the page as unreadable. Some challenge modes cannot be bypassed by firewall rules, so an allow rule you add later has no effect on them.
  • A broad firewall rule catches crawler user-agents by accident. A rule written to stop one abusive scraper, a country block, or a rate limit can catch the search crawlers too, because they arrive from data center addresses with bot user-agents. The block was aimed at abuse and hit the readers you want.
  • Under Attack Mode or a high security level challenges everything. These modes are meant for an active attack, and they challenge every visitor that looks automated. Left on after the attack ends, they challenge the crawlers every day, and the score stays capped until someone turns them off.
  • The origin log looks clean, so nobody looks further. A block at the edge never forwards the request to your server, so your server log shows nothing and your analytics shows nothing. The only trace is in the edge platform's own security dashboard, which the marketing team rarely checks. The site looks fine from the inside while being unreadable from the outside.

The steps

Step 1. Read what the edge is actually serving. The developer opens the site's robots.txt in a browser, then fetches the same file with a bot user-agent using a command line request. Compare the two. If the bot fetch returns disallow lines the browser fetch does not, a managed robots.txt setting is prepending them at the edge. Check the edge platform's docs in Sources for the current setting name and turn the managed file off or set it to match your own file.

Step 2. Check the AI bot policies one behavior at a time. The developer opens the edge dashboard and finds the AI crawler and agent access policies. Cloudflare documents these under its bot settings, with separate actions for Search, Agent, and Training. Set Search to Allow and Agent to Allow if you want to appear in AI answers and let assistants fetch your pages for shoppers. Set Training to whatever your policy is, knowing it is a separate choice. Do not set all three the same way by reflex. The three behaviors are three decisions, and the score only needs the first two.

Step 3. Fetch a product page as each crawler. The developer runs a request for a product page with the user-agent of each search crawler your brand cares about: OAI-SearchBot and ChatGPT-User for ChatGPT search, Claude-SearchBot and Claude-User for Claude, PerplexityBot for Perplexity, and Googlebot for Google. Compare each response with a normal browser request. A 403, a 429, a 503, or a 200 whose body is a challenge page instead of your content is a block. A 200 with the real page is a pass. Test each crawler separately, because one may be allowed while another is blocked.

Step 4. Turn off the challenge modes that cannot be bypassed. The developer checks whether Bot Fight Mode or a similar free bot challenge is on. Cloudflare documents that Bot Fight Mode cannot be bypassed with firewall rules, so if it is challenging the crawlers, switch it off and replace it with a targeted firewall rule that blocks the specific abusive traffic while leaving verified bots alone. The same applies to Under Attack Mode left on after an incident: turn it off, then confirm with a fresh crawler fetch.

Step 5. Fix the firewall rules that catch crawlers. The developer opens the firewall rules and looks for country blocks, user-agent blocks, and rate limits that could catch crawler traffic. Add an explicit exception for verified bots before any broad block, using the edge platform's verified bot field where it offers one. Cloudflare documents a verified bot field that is true for verified crawlers, so a rule can be limited to traffic that is not a verified bot. Check the vendor docs in Sources for the current field name on your platform.

Step 6. Confirm the fix in the edge log, not the origin log. The developer opens the edge platform's security dashboard and searches for the crawler user-agents. If the requests appear with 200 responses, the crawlers are reaching you. If they do not appear at all, that is not proof they never came: an edge block never forwards the request, so the origin log stays silent either way. The edge log is the only log that can confirm a block or a pass. Re-run the scan after the edge log shows clean 200 responses for the crawlers.

Step 7. Re-check after every platform change. The marketer adds one line to the launch checklist for any change to hosting, CDN, firewall, or security settings: fetch a product page as OAI-SearchBot and confirm a 200 with real content. Edge settings change when a plan changes, when a migration happens, and when a teammate turns on a new protection. A score that was fine can cap again overnight, and the re-check catches it early.

The machine-readable piece

This is the robots.txt allow block for the search crawlers, written the way the vendors document their own tokens. It belongs in your robots.txt file, and it only works if the edge lets the crawlers reach the file. Replace nothing in the tokens.

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

The last block is deliberate. Google-Extended is the token for training use, and it is separate from Googlebot, which is the search crawler. Blocking the training token does not block search, and allowing the search crawler does not permit training. Keep the two decisions apart in the file the same way you keep them apart in the edge dashboard.

And this is the test command the developer runs for each crawler, replacing yourdomain.com with the actual domain. It fetches a product page as OAI-SearchBot and shows the status code and the first lines of the body, so a challenge page is visible at once.

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://[yourdomain.com]/products/[your-product]

A 200 with real product content is a pass. A 403, a 429, or a 200 whose body is a challenge page is a block, and the fix is at the edge, not in robots.txt.

What it looks like when it works

When the edge is open to the search crawlers, a scan reads the same site a shopper sees: product pages with prices and stock in the markup, llms.txt at the root, robots.txt saying allow, and the guides linked and loading. The score then reflects the content itself. When the edge is closed, the scan lists the same access failure on every page, and the score cannot climb past it no matter how good the pages are.

Say you run a skincare brand with a strong catalog and a guide library on routines. Your robots.txt allows every AI crawler by name, and your llms.txt is current. But the score comes back capped, and the scan flags every page as unreadable. The developer fetches a product page as OAI-SearchBot and gets a challenge page. The edge dashboard shows an AI bot policy set to block all three behaviors, switched on during a migration months ago and never revisited. The developer sets Search and Agent to Allow, leaves Training blocked, and switches off a leftover challenge mode. The re-run scan reads the catalog for the first time, and the fixes it lists are about the pages themselves: a missing size guide link, a product with no availability markup. The cap was never about the products. It was about the door.

The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

What this guide does not cover

This guide covers the edge settings that cap your AI readiness score by blocking the readers. It does not cover the full technical audit that checks every crawler and every page, the transaction layer that lets an agent complete a purchase, or the per-engine playbooks for getting each assistant to find you. It does not cover crawler behavior on platforms other than the edge layer, and it does not cover what to do about scrapers that steal your content, which is a security decision for your security provider, not a visibility setting. Those are separate jobs, and this guide is the map, not the whole trip.

Check your own store now

Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan

Run the free scan

Sources

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.