Aizii Learn · AI Search
← Back to Learn

How to edit robots.txt on BigCommerce for AI crawlers

BigCommerce puts a plain-text robots.txt editor in the store control panel. Go to Settings, then Website, scroll down to the Search Engine Robots section, and add your AI crawler groups below the defaults. Save.

Platform screens change over time. If a menu name in this guide does not match what you see, check the vendor docs linked in Sources for the current path.

Why BigCommerce is different

No template language, no plugin, no code. What you type in the text box is what gets served. That makes it the simplest edit of any hosted platform, and it means syntax errors are entirely yours to make.

BigCommerce's default file is long and specific. It disallows account, cart, checkout, login, order-status, search, and wishlist paths, and it declares your sitemap at the bottom. Keep every default line. Your job is to append AI crawler rules, not to rebuild the file.

The default file contains two separate User-agent: * groups with a Sitemap line between them. Put your AI crawler groups in their own groups. Robots.txt matches by specificity, not by order, so specific groups win over broad groups regardless of where they sit, and leave the Sitemap line where it is.

The steps, in order

  1. Check what is served today. Open https://[yourdomain.com]/robots.txt in a browser, replacing yourdomain.com with your actual domain. Read the full default file so you know what must stay.
  1. Open the editor. In the BigCommerce control panel go to Settings, then Website, and scroll to the Search Engine Robots section.
  1. Add the AI retrieval crawler groups above the first User-agent: * group:
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /
Note: ChatGPT-User is a user-initiated, per-request fetch. An allow rule grants permission only. Robots.txt rules may not apply to fetchers that ignore the file, so an allow rule does not control those fetchers.

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: bingbot
Allow: /
  1. Save and verify. Click Save, then reload https://[yourdomain.com]/robots.txt and confirm your groups appear at the top with all the defaults intact below them.

One note for multi-storefront setups: BigCommerce also exposes this setting through its API (PUT on the storefront robotstxt settings endpoint), which supports per-channel values. If you run more than one storefront, that is the route to different rules per storefront.

The fundamentals

Three things work the same on every platform:

  • •What goes in an llms.txt file and where to publish it on BigCommerce, see our llms.txt guide
  • •The product markup AI tools read on the page, see our product schema guide
  • •The full crawler token reference and what each allow or block costs, see our robots.txt guide

A hypothetical example

Say you run an outdoor-gear store on BigCommerce. Wanting a "clean" file, you replace the whole editor contents with just the AI allowlist and save. Two weeks later your filtered category URLs are getting crawled aggressively and your cart pages start showing up in places they should not. The default disallows you wiped out were doing real work. The fix is to restore BigCommerce's default lines and append the AI groups instead. On BigCommerce, the defaults are the file; your rules are the addition.

What this guide does not cover

The file is only the declaration. If a bot manager or CDN blocks an AI crawler at the edge before it reads robots.txt, the file's rules do not matter. That edge check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.

Check your own brand now

Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.

Run the free scan

Sources

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.