Add a group for PerplexityBot with an Allow rule in [yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain, and place it before any broad disallow rules. PerplexityBot is Perplexity's search crawler. Perplexity says it is designed to surface and link websites in search results on Perplexity, and it recommends allowing it if you want your site to appear in those results. Perplexity also says PerplexityBot is not used to crawl content for AI foundation models, so this allow rule is a search decision, not a training decision, and there is no separate Perplexity training crawler to split it from.
Why PerplexityBot gets missed or mixed up
- People confuse the two Perplexity agents. PerplexityBot builds the search results. Perplexity-User fetches a page when a person asks Perplexity a question. A rule for one says nothing about the other, and the two follow different rules.
- People assume a PerplexityBot allow also controls live answers. It does not. Perplexity says Perplexity-User generally ignores robots.txt rules, because a user requested the fetch, so an allow or a disallow for PerplexityBot has no reliable effect on live fetches.
- A named group stands alone. Once you write a group for PerplexityBot, that group is the only set of rules PerplexityBot follows. It does not inherit the rules from your wildcard group, so a checkout or account disallow that lives only under User-agent: * no longer applies to PerplexityBot.
- The file can say allow while the firewall says no. Perplexity publishes its own firewall configuration guidance for Cloudflare and AWS, which only makes sense because firewalls block its agents on sites whose robots.txt permits them. An allow rule is permission, not a guarantee of a visit.
Step 1: Check the file exists and read the wildcard group
Open [yourdomain.com]/robots.txt in a browser. If nothing loads, create the file at the root of your site. Read what is already there before you add anything. Look for a User-agent: * group, and note every Disallow line under it that should still apply to PerplexityBot, such as checkout, account, and internal search paths. You will need to repeat those lines inside the PerplexityBot group, because a named group does not inherit them. Also check for a blanket Disallow: / under the wildcard group: if one is there, PerplexityBot is already blocked, and adding a named allow group is the fix, not a second wildcard line.
Step 2: Add the PerplexityBot group before any broad disallow
Put the group near the top of the file, before any broad disallow rules, so it is easy to find and easy to audit later.
User-agent: PerplexityBot
Allow: /
Disallow: /checkout/
Disallow: /account/
The Allow line opens the public catalog to PerplexityBot. The Disallow lines repeat the private paths from your wildcard group. Change the paths to match your own store. If your wildcard group disallows nothing, the group can be two lines: the User-agent line and Allow: /. If you want PerplexityBot kept out of Perplexity results instead, change Allow: / to Disallow: / in this group and leave the rest of the file alone. Perplexity says each setting works independently, so a rule for PerplexityBot does nothing to Perplexity-User, and the reverse is also true.
Step 3: Allow its IP ranges through the firewall
Perplexity recommends allowing PerplexityBot in robots.txt and permitting requests from its published IP ranges. The robots.txt rule is only half the job if a firewall sits in front of your site. Check your firewall or bot manager for a rule that challenges or blocks PerplexityBot, and allow the token by user agent together with the published IP ranges, not by user agent alone. Perplexity publishes its IP ranges at perplexity.com/perplexitybot.json, and it publishes firewall configuration guidance for Cloudflare and AWS in its crawler docs. Follow the vendor docs in Sources for the current path in your firewall's settings, because menu names change and a guessed click path goes stale. If you use a managed host or a platform, ask your host where bot rules live before you assume the firewall is yours to change.
Step 4: Record your Perplexity-User preference, and know its limit
Perplexity-User is the second agent, and it deserves its own group even though the group cannot enforce much. Perplexity says Perplexity-User supports user actions within Perplexity: when someone asks a question, it might visit a page to help provide an accurate answer and include a link to the page in its response. Perplexity also says it is not used for web crawling or to collect content for training AI foundation models. Add the group so your preference is on the record.
User-agent: Perplexity-User
Allow: /
Perplexity says plainly that since a user requested the fetch, this fetcher generally ignores robots.txt rules. Treat the group as a statement of preference, not as a lock. If you need Perplexity-User kept out of a page, a firewall rule keyed to its user agent together with its published IP ranges is the enforceable route, and Perplexity publishes those ranges at perplexity.com/perplexity-user.json. Think through the trade-off before you build that rule: blocking it means Perplexity cannot check your live page when a person asks about your product, which is the moment a citation is earned.
Step 5: Verify the file is live and the token is right
Refetch [yourdomain.com]/robots.txt and confirm the PerplexityBot group is present, spelled exactly PerplexityBot, and placed before any broad disallow. The full user-agent string Perplexity publishes is Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot), and Perplexity notes the version number may change, so match on the token PerplexityBot, not on the full string. Confirm a hit is really Perplexity by checking the requesting IP against the published list, because a user-agent string on its own is easy to copy. Perplexity says it may take up to 24 hours for its systems to reflect a robots.txt change, so check the file again the next day, then check the logs. If PerplexityBot still has not visited after a reasonable wait, the likely cause is upstream: a firewall rule or a bot manager that blocks it before the file is read.
The fundamentals
These mechanics are shared across every engine and covered in full in their own guides:
- •The complete rule reference for every crawler and edge case, see our robots.txt guide for AI search
- •What goes in llms.txt and how to publish it, see our guide to writing a good llms.txt file
- •The markup AI engines read to state price and availability in structured form, see our product schema guide for AI search
Say you run an outdoor gear shop
You sell tents, packs, and trail layers, and your best pages are detailed buying guides next to the product pages. You check your robots.txt and find no mention of Perplexity at all, and your wildcard group disallows the site search paths your guides link through. PerplexityBot follows the wildcard group when it has no group of its own, so it is already limited by rules written for a different purpose. You add a named PerplexityBot group with Allow: / and the checkout and account disallows repeated inside it, and you add a Perplexity-User group so your preference for live fetches is on the record. Then you check your firewall and find a bot rule challenging PerplexityBot, so you allow the token together with its published IP ranges. The next day you refetch the file, confirm both groups are live, and grep the logs for the PerplexityBot token to confirm a real visit. The file and the firewall now agree, which is the part the file alone could not do.
What this guide does not cover
This guide covers one crawler and its sibling fetcher. It does not describe your products to the models, it does not make your pages legible once a crawler arrives, and it does not cover how Perplexity chooses which retrieved pages to cite, because Perplexity does not document that choice, and no robots.txt rule can force a citation. It also does not cover the rest of the Perplexity surface in full: getting your products into Perplexity shopping is a separate job from being surfaced in Perplexity search results.
Check your own site now
Run the free scan. One click, no signup, and it shows which AI crawlers your robots.txt actually allows: aizii.ai/aeo-scan
Run the free scanSources
- Perplexity bots documentation (specific): https://docs.perplexity.ai/guides/bots
- Cloudflare robots.txt explainer (general docs): https://www.cloudflare.com/learning/bots/what-is-robots-txt/
- RFC 9309, Robots Exclusion Protocol (general standard): https://www.rfc-editor.org/rfc/rfc9309.html
Note: links marked specific point to the page that supports the claim in this guide. Links marked general docs point to the vendor's general docs section for that topic, not to a page that proves every step. Check the vendor docs for the current path before relying on a menu name or click path in this guide.
