Aizii Learn · AI Search
← Back to Learn

Sitemap strategy for large catalogs

A sitemap strategy for a large catalog is a plan for splitting your pages across several sitemap files, grouped under one sitemap index, so every page worth finding has a place in a file that stays within the size limits and stays current. A single sitemap holds up to 50,000 URLs and no more than 50MB uncompressed, under the sitemaps protocol, and a catalog past that limit in one file breaks the format. A large catalog also changes every day, with products added, removed, and repriced, and a sitemap that lags the catalog teaches the model last season's store. This guide covers how to split the files, how to name them, how to keep the change dates honest, and how to check the whole set. A sitemap does not guarantee a page gets crawled or indexed. Google says submitting a sitemap is a hint, not a guarantee, and this guide does not claim otherwise.

A sitemap strategy is a developer-led job with one marketer check: the developer owns the files, the split, the format, and the submission, and the marketer owns the decision about which pages belong in the set. Agree the page list and the split first, then build the files, before you submit anything.

Where it goes wrong

Large catalog sitemaps fail in ways a small store never sees, and they fail quietly.

  • One file holds the whole catalog. A catalog past 50,000 URLs or 50MB uncompressed sits in a single file that breaks the protocol limit, and the pages past the limit never get listed. The file loads, the submission says success, and a slice of the catalog is missing.
  • The split follows the database, not the store. The files are split by an internal table or a warehouse code, and no file matches a page type a person can check. When a product type goes wrong, no one can tell which file holds it.
  • The files list everything, including the pages that should never be cited. Cart, checkout, account, site search results, and filtered faceted addresses all end up in the files, and crawlers spend their attention on addresses no shopper should land on.
  • The change dates are fake. Every page carries the same new date, stamped when the file was generated, not when the page changed. Google says it uses the lastmod value only if it is consistently and verifiably accurate, and that a significant update means a change to the main content, structured data, or links on the page. A set that stamps every page with the same date teaches the crawler to ignore the signal on every page.
  • The index lists files that no longer exist. A child sitemap is renamed or removed, and the index still points at the old address. A crawler that follows the index finds a dead end, and the pages in that file lose their route to discovery.
  • The sitemap and the robots file disagree. A sitemap lists a page that robots.txt blocks, or robots.txt allows a section no sitemap lists. A crawler given two answers follows the block, and the sitemap entry is wasted.
  • The set is never rechecked. Products are added, removed, and renamed, and the sitemap set is rebuilt only when someone remembers. A new product waits weeks for a sitemap entry, and a removed product stays listed long after it leaves the catalog.

The steps

Step 1. Decide which pages belong in the set. The marketer lists the pages worth finding: product pages, category pages, brand pages, guides, the about page, and the policy pages shoppers ask about. Leave out the cart, the checkout, the account pages, the site search results, and filtered faceted addresses that only rearrange a category. Check the list against the live site. Every address in the set should load, show the right page, and match the canonical address for that page. A sitemap set that lists everything says nothing about what matters.

Step 2. Split by page type, not by database table. The developer splits the catalog into files by page type: one file for products, one for categories, one for guides, and one for the remaining pages, with further splits inside products when one type alone passes the limit. Split products by category or by brand where the catalog needs it, and keep each file under 50,000 URLs and under 50MB uncompressed. Name each file so a person can tell what it holds from the name alone, such as a products file, a categories file, and a guides file. A split a person can read is a split a person can check.

Step 3. Group the files under one sitemap index. The developer builds a sitemap index file that lists each child sitemap with its own lastmod value, set to the most recent change inside that child file. A sitemap index may list up to 50,000 sitemaps and must also stay under 50MB. Submit the index, not the individual files, and each child sitemap is discovered from it. Keep the index at a stable address, and keep each child file at a stable address. Moving a file breaks the index entry and the submission at the same time.

Step 4. Keep the change dates honest in every file. The developer sets the lastmod value on each page from the real date the page last changed, with time and timezone where the system stores them, and changes it only when the main content, structured data, or links on the page change. Do not bump the date for a cosmetic edit, a footer change, or a new copyright line. Do not stamp every page with the generation date. Set the lastmod on each child sitemap in the index to the most recent change inside that file. A trustworthy lastmod is the highest-value field in the set, because it tells a crawler which pages are worth re-fetching.

Step 5. Keep the files within the limits with room to spare. The developer checks the count and the size of each file before it is published, and splits a file before it reaches 50,000 URLs or 50MB uncompressed. Split a file before it reaches the limit, because a file past the limit may not be read in full. The sitemaps protocol sets the limits, and the same limits apply to every file in the set, including the index.

Step 6. Submit the index and point to it. The developer submits the sitemap index in Google Search Console using the Sitemaps report, and adds a sitemap line to the robots.txt file that points at the index. Check the Sitemaps report after submission, and open each child file to confirm it loads as XML and lists the pages it should. Fix any errors the report shows, such as addresses that fail to load or a file that fails to parse. If the platform generates the sitemaps for you, confirm the generated files list the pages from step one before you submit them, and submit the address the platform serves.

Step 7. Recheck the set when the catalog changes. The marketer and the developer recheck the sitemap set when products are added, removed, or renamed. A removed product should leave its file the same day it leaves the catalog. A renamed product should appear under its new address with its new lastmod value. A new guide should appear in the guides file when it is published, not weeks later. Build the recheck into the same routine that publishes a product, so the set stays current without a separate project. Check the Sitemaps report on a regular routine, and fix any file the report flags.

The machine-readable piece

This is a sitemap index for a large store, grouping three child sitemaps by page type, followed by the start of one child sitemap for products. Replace the placeholders with your own addresses and the real dates each page last changed. The loc value must be the full absolute address, and the lastmod value uses a placeholder shape: replace YYYY-MM-DD with the real date. Each child sitemap in the index carries its own lastmod value, set to the most recent change inside it. The limits come from the sitemaps protocol: 50,000 URLs and 50MB uncompressed per file, and up to 50,000 sitemaps in one index.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://[yourdomain.com]/sitemap-products.xml</loc>
    <lastmod>YYYY-MM-DD</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://[yourdomain.com]/sitemap-categories.xml</loc>
    <lastmod>YYYY-MM-DD</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://[yourdomain.com]/sitemap-guides.xml</loc>
    <lastmod>YYYY-MM-DD</lastmod>
  </sitemap>
</sitemapindex>
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://[yourdomain.com]/products/example-product-one</loc>
    <lastmod>YYYY-MM-DD</lastmod>
  </url>
  <url>
    <loc>https://[yourdomain.com]/products/example-product-two</loc>
    <lastmod>YYYY-MM-DD</lastmod>
  </url>
</urlset>

Check the set before you submit it: open the index and each child sitemap, confirm each loads as XML and lists the pages it should, and open a few listed addresses to confirm each loads the right page. Submit the index in Search Console, and fix any file the report flags.

What it looks like when it works

On the page, nothing looks different to a human reader. A sitemap is not a page shoppers see. The change is under the surface: every product page, category page, and guide has one canonical address, the set lists exactly those addresses with honest change dates, the files split cleanly by page type, and the robots file points at the index. Each file sits comfortably under the size limits, and each child file in the index carries a lastmod value that matches the newest change inside it. A crawler that reads the index finds the pages worth finding, skips the pages that are not in it, and re-fetches the pages whose change dates moved.

Say you run a hypothetical electronics store with a large catalog, selling headphones, speakers, cables, and accessories across thousands of products. The marketer lists the pages worth finding and cuts the cart, checkout, account, search results, and filtered faceted addresses from the list. The developer splits the catalog by page type, with one file for products, one for categories, and one for guides, grouped under an index, because the catalog passes the size where one file is enough. The developer submits the index in Search Console and adds the sitemap line to the robots file. A shopper asks an assistant for a wireless headphone under a set price with a named feature. The assistant can find the current category page, the current product pages, and the buying guide, and it cites the store instead of a stale listing.

The fundamentals

These mechanics are shared across every store and covered in full in their own guides:

What this guide does not cover

This guide covers the sitemap strategy for a large catalog: the split, the index, the limits, the change dates, and the recheck routine. It does not cover the full technical audit that checks every crawler and every page, the product feed build, faceted navigation design, image or video sitemaps, or international sitemaps for multi-language stores, or the transaction layer that lets an agent complete a purchase. Those are separate jobs, and this guide is the map for the sitemap set, not the whole trip.

Check your own store now

Run the free scan. One click, no signup, and it shows whether AI crawlers can read your store today: aizii.ai/aeo-scan

Run the free scan

Sources

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.