Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# How to edit robots.txt on WooCommerce for AI crawlers

WooCommerce runs on WordPress, and WordPress generates robots.txt on the fly. There is no physical file by default. You edit it through your SEO plugin's file editor, or you place a physical robots.txt in the site root, which overrides the virtual one completely.

Platform screens change over time. If a menu name in this guide does not match what you see, check the vendor docs linked in Sources for the current path.

## Why WooCommerce is different

Two files can exist and only one wins. A physical robots.txt sitting in your site's root directory (usually public\_html) overrides the virtual one entirely. Editing the plugin editor while a forgotten physical file sits in the root is the classic reason changes appear to do nothing.

WordPress's virtual defaults only block wp-admin and wp-includes. WooCommerce adds no crawler rules of its own. Cart, checkout, and account pages stay out of indexes through other means, not through this file. That makes your AI crawler groups purely additive: you are adding allow rules to a file that never blocked them.

Plugin editors are not interchangeable. Yoast SEO and Rank Math each keep their own editor, and each writes the file its own way. Know which plugin is authoritative on your site before you edit.

## The steps, in order

1.  Check what is served today. Open [https://\[yourdomain.com\]/robots.txt](https://[yourdomain.com]/robots.txt) in a browser, replacing yourdomain.com with your actual domain. Note whether the rules look like WordPress defaults or something custom.

1.  Check for a physical file first. In your hosting file manager or over FTP, look in the site root (public\_html) for a file named robots.txt. If one exists, that file wins over everything else. Edit that file directly, or delete it if you want the virtual file (and your plugin editor) to take over.

1.  If there is no physical file, use your SEO plugin's editor:

-   With Yoast SEO: in the WordPress dashboard go to SEO, then Tools, then File editor. If no robots.txt exists, Yoast offers to create one. Edit the rules and click Save changes to robots.txt.
-   With Rank Math: go to Rank Math, then General Settings, then Edit robots.txt. Make your changes and save.

1.  Add the AI retrieval crawler groups near the top, before any broad disallow rules:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /
Note: ChatGPT-User is a user-initiated, per-request fetch. An allow rule grants permission only. Robots.txt rules may not apply to fetchers that ignore the file, so an allow rule does not control those fetchers.

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: bingbot
Allow: /
```

Keep WordPress's default blocks on wp-admin and wp-includes. Do not disallow wp-content wholesale: Google needs the CSS and JavaScript inside it to render your product pages. Other AI crawlers effectively do not render, so scope this rendering point to Google.

1.  Save and verify. Reload [https://\[yourdomain.com\]/robots.txt](https://[yourdomain.com]/robots.txt) and confirm your groups are present.

### The fundamentals

Three things work the same on every platform:

-   •What goes in an llms.txt file and where to publish it on WooCommerce, see our [llms.txt guide](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The product markup AI tools read on the page, see our [product schema guide](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)
-   •The full crawler token reference and what each allow or block costs, see our [robots.txt guide](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

### A hypothetical example

Say you run a bike-parts store on WooCommerce with Yoast installed. You add the AI crawler allowlist through Yoast's file editor, save, and check the URL. Nothing changed. The file still shows old rules. The reason: a physical robots.txt left over from a site migration was sitting in public\_html, and it overrode everything Yoast wrote. Deleting the stale physical file let the plugin's version take over, and the new rules appeared immediately. On WooCommerce, always check the root directory before assuming the editor is broken.

## What this guide does not cover

The file is only the declaration. If a security plugin or the hosting firewall blocks an AI crawler at the edge before it reads robots.txt, the file's rules do not matter. That edge check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.

## Check your own brand now

Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.

[Run the free scan](https://aizii.ai/aeo-audit)

## Sources

-   WooCommerce documentation, core products (general docs): [https://woocommerce.com/documentation/woocommerce/core-products/](https://woocommerce.com/documentation/woocommerce/core-products/)
-   Yoast help, editing robots.txt through Yoast SEO (general docs): [https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/](https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/)
-   Rank Math knowledge base, blocking web crawlers (general docs): [https://rankmath.com/kb/how-to-block-web-crawlers/](https://rankmath.com/kb/how-to-block-web-crawlers/)
-   WordPress developer docs, robots\_txt hook (general docs): [https://developer.wordpress.org/reference/hooks/robots\_txt/](https://developer.wordpress.org/reference/hooks/robots_txt/)

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
