Yes, site speed affects how much of your site a crawler can cover, but not in the way most speed claims suggest. The effect runs through your server, not through a single speed score. Google's crawl budget guidance says a site's crawl budget is the set of URLs Google can and wants to crawl, and that it is set by two things: crawl capacity limit and crawl demand. Crawl capacity limit is how much crawling your server can take without slowing down. When your server answers fast and stays stable, the limit rises and the crawler can cover more pages. When your server slows down, returns server errors, or returns rate-limit signals, the limit falls and the crawler covers less. There is no vendor formula that turns a speed score into a fixed number of pages crawled, and no vendor publishes a fixed speed threshold for AI crawler coverage, so this guide does not invent one. It presents the vendor docs as they are, and shows what to fix first.
Start with the two sides of crawl budget, because confusing them is the usual mistake. Crawl demand is how much a crawler wants to crawl your site, based on your site's size, how often pages change, page quality, and popularity. Crawl capacity limit is how much crawling your server can handle, based on response time, stability, and errors. Speed affects the capacity side. A fast server lets a crawler use more connections and hold them for less time, so it can cover more pages in the same window.
Where speed hurts coverage
Speed hurts crawler coverage in specific ways, and each one has a different fix.
- The server answers slowly. Every page takes a long time to return its first bytes, so the crawler spends its time waiting on connections instead of reading pages, and covers fewer pages per visit.
- The server fails under load. The site returns server errors when traffic rises, so the crawler sees errors, lowers its crawl capacity limit, and comes back less often.
- The site returns rate-limit signals. The server or a security layer returns too-many-requests responses to the crawler, which Google's guidance lists alongside server errors as a reason the capacity limit falls.
- The catalog is huge and mostly duplicate. Faceted navigation, sorted copies, and parameter versions of the same page create thousands of near-identical URLs, so the crawler spends its coverage on duplicates and never reaches the new or changed pages.
- Removed pages still return a normal page. A deleted product returns a soft error page that looks like a real page, so the crawler keeps coming back to it and wastes coverage on a page that no longer exists.
- The sitemap is stale. New pages are not in the sitemap, and removed pages still are, so the crawler has no clean list of what to cover and spends time on the wrong URLs.
- The page is heavy for the reader, not the server. Large images, heavy scripts, and layout shifts make the page feel slow to a shopper, and while that matters for the shopper, it is a different problem from server response time, which is what sets crawl capacity.
The steps
Step 1. Check whether crawl budget matters for your site at all. The marketer or developer opens Google's crawl budget guidance and reads who it is for. Google says the guide is for large sites with one million or more unique pages that change moderately often, medium or larger sites with ten thousand or more unique pages that change daily, and sites with a large share of URLs classified as discovered and currently not indexed. Google also says that if your site does not have a large number of pages that change rapidly, or if your pages seem to be crawled the same day they are published, you do not need the guide, and keeping your sitemap up to date and checking the Page Indexing report is adequate. Most stores fall in that second group. Fix speed for shoppers first, and treat crawl coverage as a live problem only when the signs in Step 2 show it is one.
Step 2. Look at what the crawler actually sees. The developer opens Search Console, then Settings, then Crawl stats, and reads the average response time, the total crawl requests, and the host status. A rising average response time, a run of server errors, or host availability problems are signs the server is limiting coverage. The developer also opens the Page Indexing report and looks for a large share of URLs marked as discovered and currently not indexed. Those two reports, not a speed score from a third-party tool, are the evidence that coverage is short.
Step 3. Fix server response time before page weight. The developer checks hosting first: the server, the database, and any layer that builds the page before it is sent. Google's crawl budget guidance says that if the site responds consistently and its response times remain stable or improve, the limit goes up, and that if the site slows down or responds with server errors or rate-limit signals, the limit goes down. Ask the host or the developer to measure time to first byte for the product and category pages, and to fix slow database queries, uncached pages, and an overloaded server before spending time on smaller page tweaks. A fast server is the lever that moves crawl capacity.
Step 4. Cut the duplicate URLs that waste coverage. The developer lists the URLs the crawler is spending time on, and consolidates duplicate content, blocks unimportant pages in robots.txt where they cannot be consolidated, and avoids long redirect chains. Google's guidance lists managing the URL inventory as a best practice, and says that if Google spends too much time crawling URLs it should not, its crawlers might not explore the rest of the site. Faceted navigation, sorted copies, and infinite scroll pages that duplicate linked pages are the usual source of waste on a store. Block the copies, keep the canonical page.
Step 5. Return the right status for removed pages. The developer makes sure a permanently removed product returns a not-found status, not a soft error page that looks like a real page. Google's guidance says a not-found status is a strong signal not to crawl that URL again, and that soft error pages will continue to be crawled and waste budget. Check the Page Indexing report for soft errors, fix the pages that cause them, and remove the deleted URLs from the sitemap.
Step 6. Keep the sitemap current and support caching. The developer keeps the sitemap up to date with every page the brand wants crawled, and removes pages that no longer exist. Google's guidance says to include all the content you want Google to crawl, and to include a last modification tag for updated content. The developer also supports conditional requests, so an unchanged page can return a not-modified status instead of the full page, which Google's guidance says saves server bandwidth and resources. A current sitemap and a working not-modified response let the crawler spend its time on pages that actually changed.
Step 7. Measure page experience separately, with the published thresholds. The marketer opens PageSpeed Insights and reads the Core Web Vitals for the product and category pages: Largest Contentful Paint for loading, Interaction to Next Paint for responsiveness, and Cumulative Layout Shift for visual stability. Google's published good thresholds are Largest Contentful Paint at 2.5 seconds or less, Interaction to Next Paint at 200 milliseconds or less, and Cumulative Layout Shift at 0.1 or less, judged at the 75th percentile of real visits. Treat those as page experience targets for shoppers, not as a crawl coverage formula. A page can pass all three and still have thin coverage if the server is slow or the catalog is full of duplicates, and a page can fail one and still be crawled when the server is fast and the page is in demand.
The machine-readable piece
This is a sitemap snippet for a hypothetical store, showing the pages the brand wants crawled. Swap the placeholders in square brackets for your own values. The loc entry is the page address. The lastmod placeholder must be replaced with the real last modification time in ISO 8601 format, with timezone information, for that page only. Only list pages that exist, that return a normal status, and that the brand wants crawled. Do not list removed pages, blocked pages, or duplicate copies of the same page. A sitemap does not guarantee a page will be crawled, and it does not raise a speed score. It gives the crawler a clean list to work from, so the coverage it has goes to the right pages.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://[yourdomain.com]/products/[product-slug]</loc>
<lastmod>[last modification time in ISO 8601 format, with timezone]</lastmod>
</url>
<url>
<loc>https://[yourdomain.com]/collections/[collection-slug]</loc>
<lastmod>[last modification time in ISO 8601 format, with timezone]</lastmod>
</url>
</urlset>
Google's crawl budget guidance says to keep sitemaps up to date, to include all the content you want Google to crawl, and to include a last modification tag for updated content. It also says that making pages efficient to load can let Google read more content from a site, and that supporting a not-modified status saves server resources. None of that is a fixed threshold. It is the vendor guidance, stated as the vendor states it.
What it looks like on the page
On the site, a shopper sees product and category pages that load fast, with the main image and headline appearing promptly, buttons that respond when tapped, and no layout jump while the page loads. Under the surface, the server answers quickly and stays stable under load, the sitemap lists the pages that exist and no others, removed products return a not-found status, and duplicate copies are blocked or consolidated. A crawler that visits can cover the new and changed pages without spending its time on duplicates or dead ends.
Say you run a hypothetical furniture brand with a large catalog. New products appear each week, and older products are removed each season. You check Crawl stats and see the average response time rising, and the Page Indexing report shows many URLs marked as discovered and currently not indexed. You fix the slow server response first, then remove the deleted products from the sitemap and make them return a not-found status, then block the sorted copies of category pages in robots.txt. The next crawl covers more of the new products, because the crawler is no longer spending its time waiting on a slow server or re-reading pages that no longer exist.
The fundamentals
These mechanics are shared across every site and written in full in their own guides, so this guide points to them instead of repeating them:
- •How to write the llms.txt file for your site, in our llms.txt guide
- •How to add the Product markup AI systems read, in our product schema guide
- •How to write the crawler rules for AI systems, in our robots.txt guide for AI search
What this guide does not cover
This guide covers how site speed affects crawler coverage through server response time and crawl budget, as Google documents it. It does not cover a full performance audit, hosting migration, or image and script optimization in detail, and it does not cover the AI crawlers run by other vendors, for which no vendor publishes a crawl budget formula. It does not cover the transaction layer that lets an agent complete a purchase. Those belong to other guides. This one covers speed and coverage only.
Check your own site
Run the free scan at aizii.ai/aeo-scan. One click, no signup, and it checks whether AI crawlers can read your site today.
Run the free scanSources
- Google Search Central, Managing crawl budget for large sites (specific): https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget
- Google Search Central, Optimizing your website for generative AI features on Google Search (specific): https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- Google Search Central, AI features and your website (specific): https://developers.google.com/search/docs/appearance/ai-features
- Web.dev, Web Vitals (specific): https://web.dev/articles/vitals
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
