Aizii Learn · AI Search
← Back to Learn

How to edit robots.txt on a self-hosted site for AI crawlers

You own the server, so you own the file. Create or edit robots.txt in the web server's document root as plain text, and it is served at https://[yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain, on the next request.

Why self-hosted is different

No panel, no template, no plugin. The file is the configuration. That removes every platform quirk and replaces it with one responsibility: the file on disk is exactly what crawlers read.

You can verify end to end in a way hosted platforms never allow. Fetch the file with curl, request a page with a bot user-agent to confirm the edge lets it through, grep your access logs for the crawler tokens, and validate the hits against the vendors' published IP ranges. The pillar guide has the exact log command and the validation sources.

The server config can shadow the file. An nginx location block or an Apache rewrite that handles /robots.txt, or a CDN caching an old copy, means your edits do not take effect until you fix the config or purge the cache. The file being correct and the file being served are two different facts. Check both.

One file per server. Behind a load balancer, keep it identical on every node, or better, serve it from your deploy pipeline so it cannot drift.

The steps, in order

  1. Check what is served today. In a terminal, run:
curl https://[yourdomain.com]/robots.txt

replacing yourdomain.com with your actual domain. Read the current rules.

  1. Edit the file in the document root. Open robots.txt in the web server's document root (for example, /var/www/html/robots.txt on a typical nginx or Apache setup) in a plain-text editor. If no file exists, create one. Save as plain text with UTF-8 encoding.
  1. Put the AI retrieval crawler groups near the top, before any broad disallow rules:
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /
Note: ChatGPT-User is a user-initiated, per-request fetch. An allow rule grants permission only. Robots.txt rules may not apply to fetchers that ignore the file, so an allow rule does not control those fetchers.

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: bingbot
Allow: /
  1. Verify the serve, not just the file. Fetch the URL again with curl and confirm the new groups appear. Check the content type is text/plain. If you run a CDN, purge the cache for /robots.txt or wait out the cache TTL.
  1. Confirm crawlers actually arrive. After a week, grep your access logs for the tokens. No hits can mean something upstream is still blocking, or that crawlers have not returned yet.

The fundamentals

Three things work the same on every platform:

  • •What goes in an llms.txt file and where to publish it on a self-hosted site, see our llms.txt guide
  • •The product markup AI tools read on the page, see our product schema guide
  • •The full crawler token reference and what each allow or block costs, see our robots.txt guide

A hypothetical example

Say you run software documentation on a self-hosted server behind a CDN. You write the file perfectly, verify it on disk, and move on. A scan a month later shows the AI crawlers still seeing the old rules. The CDN had cached /robots.txt with a long TTL and never picked up the change. One cache purge fixed it. On a self-hosted setup the file is the easy part; the cache in front of it is the part that bites.

What this guide does not cover

The file is only the declaration. If your firewall or CDN blocks an AI crawler at the edge before it reads robots.txt, the file's rules do not matter, and you are the one who configured that edge. That check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.

Check your own brand now

Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.

Run the free scan

Sources

Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.