In Salesforce Commerce Cloud, robots.txt is developer-managed and instance-specific. In Business Manager go to Merchant Tools, then SEO & Discoverability, then Robots. Select the instance type, choose the Custom robots.txt Definition option, and write the complete file content there.
Platform screens change over time. If a menu name in this guide does not match what you see, check the vendor docs linked in Sources for the current path.
Why Salesforce Commerce Cloud is different
There are two ways to serve the file, and they do not mix. You can type a custom definition into Business Manager (Salesforce's recommended route), or you can place a robots.txt file in the cartridge/static/default directory of your storefront cartridge. There can only be one robots.txt per cartridge, so pick one mechanism and stay with it.
The definition is per instance type. Staging and production want different rules: staging should block everything, production should welcome the retrieval crawlers. Salesforce's own docs recommend authoring the file on a staging instance and replicating the site preferences to production.
The hostname alias must be configured, or the file is not served under your domain at all. A correct definition with a missing alias produces a 404 at /robots.txt.
Storefront Password Protection auto-generates a block-all robots.txt that overrides your definition. If password protection is on, your custom file never takes effect, no matter how carefully you wrote it.
Salesforce's own documentation names GPTBot, ClaudeBot, and PerplexityBot as crawlers the platform serves the file to. The AI angle here is not theoretical: the platform expects these bots to request the file.
The steps, in order
- Check what is served today, on the right instance. Open https://[yourdomain.com]/robots.txt in a browser, replacing yourdomain.com with your actual domain, on staging and on production separately. The two should look different.
- Choose your mechanism. For most teams, the Custom robots.txt Definition in Business Manager is the right call. Use the deployed-cartridge file only if you need the same file shared across multiple sites.
- Write the definition. In Business Manager go to Merchant Tools, then SEO & Discoverability, then Robots. Select the instance type. For staging, a full block is the correct content:
User-agent: *
Disallow: /
For production, write the full file with your defaults plus the AI retrieval crawler groups near the top, before any broad disallow rules:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
Note: ChatGPT-User is a user-initiated, per-request fetch. An allow rule grants permission only. Robots.txt rules may not apply to fetchers that ignore the file, so an allow rule does not control those fetchers.
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: bingbot
Allow: /
User-agent: *
Disallow: /account*
Disallow: /cart*
Disallow: /checkout*
- Replicate staging site preferences to production. Authoring on production directly skips the staging check and risks pushing staging's block-all rules live.
- Verify on the production hostname. If the old file persists, invalidate the static content cache and reload /robots.txt.
The fundamentals
Three things work the same on every platform:
- •What goes in an llms.txt file and where to publish it on Salesforce Commerce Cloud, see our llms.txt guide
- •The product markup AI tools read on the page, see our product schema guide
- •The full crawler token reference and what each allow or block costs, see our robots.txt guide
A hypothetical example
Say you run a fashion brand on Salesforce Commerce Cloud. You write the production definition in Business Manager, verify it on your staging hostname, and call it done. A scan of the production domain still shows a block-all file. The definition was fine; the site preferences were never replicated from staging to production, and the hostname alias check was never run. On this platform the file is never "just" a file. Instance, replication, and alias are part of the edit.
What this guide does not cover
The file is only the declaration. If a bot manager or the Commerce Cloud edge blocks an AI crawler before it reads robots.txt, the file's rules do not matter. That edge check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.
Check your own brand now
Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.
Run the free scanSources
- Salesforce B2C Commerce developer docs, custom APIs guide (general docs): https://developer.salesforce.com/docs/commerce/commerce-api/guide/custom-apis.html
- Salesforce B2C Commerce developer docs, on-demand sandbox guide (general docs): https://developer.salesforce.com/docs/commerce/b2c-commerce/guide/b2c-develop-with-on-demand-sandbox.html
- RFC 9309, Robots Exclusion Protocol (general standard): https://www.rfc-editor.org/rfc/rfc9309.html
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
