Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# How to edit robots.txt on a custom website for AI crawlers

"Custom website" describes how the site was built, not where it runs, so the editing method follows your hosting. Identify the hosting layer first: a static host, a platform-as-a-service, or a server you manage. The rule is the same in all three: find the file that is actually served at /robots.txt and edit that one.

## Why a custom website is different

There is no single path, because there is no single platform. The method is determined by the hosting layer, and hand-built sites hide an extra step: confirming which file the server actually serves. A robots.txt in your repo means nothing if your router or build pipeline never deploys it.

On static hosts (Netlify, Vercel, Cloudflare Pages), robots.txt is a static file in the repo's publish directory, the same directory index.html is served from. It deploys with the site. Change the file, redeploy, and the new rules go live. No redeploy, no change.

Single-page apps are the classic trap. A catch-all route that rewrites every request to index.html can swallow /robots.txt requests, handing crawlers your app shell instead of rules. The file must be served before the catch-all in your routing order.

On a platform-as-a-service (Render, Fly.io, Heroku), there is no shared static directory. Serve robots.txt from your app code or your static middleware, and treat it like any other route you own.

On a VPS or dedicated box that you manage yourself, this is a self-hosted site. The file lives in the document root. See our guide to editing robots.txt on a self-hosted site.

## The steps, in order

1.  Check what is served today. Open [https://\[yourdomain.com\]/robots.txt](https://[yourdomain.com]/robots.txt) in a browser, replacing yourdomain.com with your actual domain. If you get your app shell, a 404, or an HTML page, the request is not reaching a robots.txt file at all.

1.  Identify your hosting layer. Static host, platform-as-a-service, or your own server. The steps below branch from here.

1.  Static host: add or edit robots.txt in the publish directory of your repo (where index.html is served from). Commit, redeploy, and reload the URL to confirm.

1.  Single-page app: check your routing order. Make sure /robots.txt is served as a static file before the catch-all rewrite to index.html. A rewrite rule that fires first will shadow the file no matter how correct it is.

1.  Platform-as-a-service: add a route or static middleware in your app that serves the file at /robots.txt with a text/plain content type. Deploy and verify.

1.  Your own server: edit the file in the document root. See our guide to editing robots.txt on a self-hosted site for the full method.

1.  Add the AI retrieval crawler groups near the top, before any broad disallow rules:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /
Note: ChatGPT-User is a user-initiated, per-request fetch. An allow rule grants permission only. Robots.txt rules may not apply to fetchers that ignore the file, so an allow rule does not control those fetchers.

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: bingbot
Allow: /
```

1.  Verify from the outside. Reload [https://\[yourdomain.com\]/robots.txt](https://[yourdomain.com]/robots.txt) and confirm the groups are present with the right content type. What your repo contains and what the URL serves are two different facts.

### The fundamentals

Three things work the same on every platform:

-   •What goes in an llms.txt file and where to publish it on a custom website, see our [llms.txt guide](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The product markup AI tools read on the page, see our [product schema guide](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)
-   •The full crawler token reference and what each allow or block costs, see our [robots.txt guide](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

### A hypothetical example

Say you run a SaaS marketing site built with a static generator and deployed on a static host. Your developer adds robots.txt to the repo root, where the source files live, and pushes. The URL still 404s. The publish directory is a build output folder, and the file never made it into the build. Moving it next to the built index.html and redeploying fixed it in one deploy. On a custom site, the question is never "did I write the file" but "did the file reach the URL".

## What this guide does not cover

The file is only the declaration. If a bot manager or CDN blocks an AI crawler at the edge before it reads robots.txt, the file's rules do not matter. That edge check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.

## Check your own brand now

Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.

[Run the free scan](https://aizii.ai/aeo-audit)

## Sources

-   RFC 9309, Robots Exclusion Protocol (specific): [https://www.rfc-editor.org/rfc/rfc9309.html](https://www.rfc-editor.org/rfc/rfc9309.html)
-   Cloudflare bot solutions docs, verified bots (general docs): [https://developers.cloudflare.com/bots/concepts/bot/verified-bots/](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/)

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
