Faceted navigation is the set of filters on a category page that lets a shopper narrow the products shown: color, size, brand, price, material, and sort order. Each filter choice usually adds a parameter to the address, and each combination of choices creates a new address. A category with a few filters can produce thousands of near-identical addresses, and crawlers cannot tell which ones are useful without fetching them first. Google's faceted navigation guidance says this creates two harms: overcrawling, where crawlers fetch a very large number of filtered addresses before they work out the addresses are useless, and slower discovery of new pages, because the time spent on filtered addresses is time not spent on new products. Handling faceted navigation for AI crawlers means letting shoppers filter freely while keeping crawlers on the pages that matter: the product pages and one unfiltered page per category. This guide covers the decision per filter, the robots rules, the fragment option, and the canonical fallback. It does not promise a fixed recrawl time or a guaranteed ranking change, and no vendor publishes a formula for AI visibility. This guide does not invent one.
This is a developer-led job with one marketer check: the developer owns the filter addresses, the robots file, and the canonical tags, and the marketer owns the decision about which filtered pages are worth finding. Agree the filter list first, then change the code, before you submit anything.
Where it goes wrong
Faceted navigation fails in ways that look normal to a shopper and break for a crawler.
- Every filter combination is a crawlable address. A shopper clicks red, then size 9, then a price band, and each click creates a new address that crawlers can follow and fetch. A category with five colors, six sizes, and four brands can produce hundreds of combinations, and a catalog with many categories can produce far more.
- The same filtered page has many addresses. The filters appear in a different order, or the same filter is written two ways, and each order is a different address showing the same products. Crawlers fetch each version before they can tell the versions are the same.
- Filtered pages sit in the sitemap. The sitemap lists filtered addresses beside the product pages and the category pages, and crawlers spend their attention on addresses no shopper should land on from search.
- Empty combinations return a live page. A filter combination with no products returns a normal page with an empty list, and crawlers keep fetching combinations that lead nowhere. Google's faceted navigation guidance says to return a 404 status code when a filter combination returns no results.
- The canonical tag points the wrong way. A filtered page carries a canonical tag that points to itself, or to another filtered page, instead of to the unfiltered category page. The signal that should consolidate the versions instead confirms each one as its own page.
The steps
Step 1. List every filter and decide its fate. The marketer and the developer list every filter on the site: color, size, brand, price, material, rating, sort order, and any other. For each filter, ask one question: do shoppers search for this filtered page by name, or do they only use it to narrow a category they already found? A filter people search for, such as a color or a size in a popular category, can become a real page. A filter people only use to narrow, such as a sort order or a second and third refinement, should not become a crawlable page. Write the decision down per filter, and keep the list where the team can find it.
Step 2. Block the filtered addresses you do not need indexed. The developer adds robots.txt rules that disallow crawling of the filtered addresses the team decided not to index, while keeping the product pages and one unfiltered listing page per category crawlable. Google's faceted navigation guidance describes robots.txt as the way to prevent crawling of faceted navigation addresses you do not need in search, and says there is often no good reason to allow crawling of filtered items. Use one rule per filter parameter, and test each rule before you publish it. A rule that is too broad can block the category page itself, and a rule that is too narrow blocks nothing.
Step 3. Use fragments where the platform supports them. The developer checks whether the platform can carry the filter state after a hash sign in the address, instead of in a parameter. Google says it generally does not support URL fragments in crawling and indexing, so a filter carried in a fragment has no effect on crawling: the crawler sees the unfiltered category address. Where the platform supports it, this is the cleanest option, because shoppers can still filter and share the address, and crawlers never see a new address at all. Where the platform does not support it, use the robots rules from step two instead. Do not mix the two approaches on the same filter, or the team will not know which rule is doing the work.
Step 4. Make the filtered pages you keep worth finding. The marketer picks the small number of filtered pages that shoppers search for, and the developer makes each one a real page: one address, one canonical tag pointing at itself, a short paragraph in words that says what the filtered list holds, and links a crawler can follow. A page for red running shoes should say so in words, and name the sizes and brands it covers. Do not keep a filtered page that has no words of its own. If a filtered page cannot carry its own paragraph, it is not a page worth indexing, and it belongs behind the robots rules from step two.
Step 5. Fix the address shape on the pages you keep. The developer makes sure the filtered addresses that stay crawlable use the industry standard parameter separator, the ampersand, with parameters in one fixed order, and that each parameter is written one way only. Google's faceted navigation guidance says to use the ampersand separator, because other characters are hard for crawlers to detect as separators, and to keep the parameter order fixed. Return a 404 status code when a filter combination returns no results, so crawlers stop fetching combinations that lead nowhere. Do not offer further filtering on a page with zero results.
Step 6. Point the rest back to the category. The developer adds a canonical link tag on each filtered address that stays crawlable but is not a page worth indexing on its own, pointing to the unfiltered category page. Google's canonical guidance says a rel canonical link annotation is a strong signal that the specified address should become canonical, and that sitemap inclusion is a weak signal that helps the included addresses become canonical. Use the canonical tag as a fallback, not as the main fix: a canonical tag does not stop a crawler from fetching the filtered address, so it does not save the crawl time the robots rules save.
Step 7. Check the crawl after the change. The developer checks the server logs and the Search Console crawl reports after the rules go live, and the marketer checks that shoppers can still filter and buy. Confirm the filtered addresses you blocked are no longer fetched, the product pages and the unfiltered category pages are still fetched, and the filtered pages you kept are still reachable. Confirm the sitemap lists only the product pages, the category pages, and the filtered pages you kept, and no blocked address. If a blocked address still appears in the crawl, check the rule pattern against the real address shape, and fix the pattern.
The machine-readable piece
This is a robots.txt block for a hypothetical shoe store, followed by the canonical link tag for a filtered page. Replace the placeholders in square brackets with your own parameter names and addresses, and check the parameter names against your real filter addresses before you publish the file. The rules disallow crawling of addresses carrying color, size, and sort parameters, while leaving the product pages and the unfiltered category pages crawlable. The canonical tag sits in the head of a filtered page and points to the unfiltered category page. The Sitemap line points crawlers at the sitemap index. Only use parameter names that match the addresses your site actually serves.
User-agent: *
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sort=
Sitemap: https://[yourdomain.com]/sitemap-index.xml
<link rel="canonical" href="https://[yourdomain.com]/collections/running-shoes" />
Check the file after you publish it: open the robots.txt address and confirm the rules read as you wrote them, open a blocked filtered address and confirm the rule pattern matches it, and open an unfiltered category page and confirm no rule blocks it.
What it looks like when it works
On the page, a shopper sees the same filters and the same products, and filtering feels no different. Under the surface, the filter addresses that do not matter are closed to crawlers, the product pages and one unfiltered page per category stay open, and the small number of filtered pages worth finding carry their own words and their own canonical tags. The sitemap lists only the pages worth finding. The server serves fewer wasted fetches, and new products are discovered sooner, because the crawl time that went to filtered addresses now goes to new pages. A model asked about a product finds the product page, not a filtered copy of the category.
Say you run a hypothetical shoe store selling running shoes, trainers, and boots, with filters for color, size, brand, price, and sort order. The marketer decides that only two filtered pages are worth finding: red running shoes and waterproof boots, because shoppers search for those by name. The developer adds robots rules that disallow crawling of the color, size, and sort parameters, and gives the two kept pages their own paragraphs and self-referential canonical tags. A shopper asks an assistant for a red running shoe in size 9 under a set price. The assistant can find the red running shoes page, check the size 9 products in stock, and cite your store instead of a competitor's category page buried under filtered copies.
The fundamentals
These mechanics are shared across every store and covered in full in their own guides:
- •The llms.txt format and a complete sample, see our guide to writing a good llms.txt file
- •The markup AI systems read to verify facts, see our product schema guide
- •The complete crawler rule reference, see our robots.txt guide for AI search
What this guide does not cover
This guide covers handling faceted navigation for AI crawlers: the per-filter decision, the robots rules, the fragment option, the canonical fallback, and the crawl check. It does not cover the full technical audit that checks every crawler and every page, the sitemap strategy for a large catalog, the product feed build, or the transaction layer that lets an agent complete a purchase. It does not cover redesigning the filter experience for shoppers, or the platform-specific steps for changing how a platform builds filter addresses. Those are separate jobs, and this guide is the map for the crawler side, not the whole trip.
Check your own store now
Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan
Run the free scanSources
- Google Search Central, Managing crawling of faceted navigation URLs (specific): https://developers.google.com/crawling/docs/faceted-navigation
- Google Search Central, How to specify a canonical with rel="canonical" and other methods (specific): https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls?hl=en&visit_id=637782960147834858-654044296&rd=1
- Google Search Central, Build and submit a sitemap (specific): https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
- IETF, RFC 9309 Robots Exclusion Protocol (specific): https://datatracker.ietf.org/doc/rfc9309/
- Google Search Central, Optimizing your website for generative AI features on Google Search (specific): https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
