Quick answer
PerplexityBot indexes pages so Perplexity can cite them in answers. Allow it with "User-agent: PerplexityBot" and "Allow: /", or block it with "Disallow: /". Perplexity-User, which fetches pages when a user asks, generally ignores robots.txt according to Perplexity, so use firewall rules if you need a hard block.
Perplexity cites its sources prominently, often listing them before the answer, which makes it one of the AI products most likely to send visible referral traffic. Whether it can cite you depends on PerplexityBot, and on understanding the one Perplexity fetcher robots.txt doesn't control.
Perplexity's two user agents
| User agent | What it does | Follows robots.txt? |
|---|---|---|
| PerplexityBot | Crawls and indexes pages for Perplexity's answers and citations | Yes; Perplexity says it won't index content from sites that disallow it |
| Perplexity-User | Fetches a page live when a user's question needs it | Generally no, per Perplexity's documentation, because a user requested it |
Perplexity's documentation notes that robots.txt changes can take up to 24 hours to be reflected. It also publishes the IP ranges both agents use, which helps if you manage access at the firewall.
Copy-paste robots.txt rules
Allow Perplexity to index and cite your site:
User-agent: PerplexityBot
Allow: /Block Perplexity indexing:
User-agent: PerplexityBot
Disallow: /Allow everything except a private section:
User-agent: PerplexityBot
Disallow: /members/
Allow: /If you give PerplexityBot its own group, it stops reading your User-agent: * rules, so include any paths you still want blocked.
When robots.txt isn't enough
Because Perplexity-User generally ignores robots.txt, a Disallow for it won't reliably stop live fetches when someone asks Perplexity about a URL. If you need content kept away from Perplexity entirely, robots.txt alone won't do it:
- Put private content behind authentication. No crawler, rule-following or not, should reach it.
- Use firewall rules that block Perplexity's published IP ranges or user agents, at your CDN or host.
- Use your CDN's AI bot controls. Cloudflare, for example, lets you block named AI crawlers at the edge.
In August 2025 Cloudflare published research alleging that Perplexity used undeclared crawlers to fetch pages from sites that had blocked it, which Perplexity disputed. Whatever the merits, it is a reminder that robots.txt is a request, and enforcement happens at the network level.
How to test the rules
- Publish robots.txt and confirm it's live at
/robots.txt. - Run the AI crawler access checker and check PerplexityBot and Perplexity-User show the status you expect.
- Check your CDN or firewall isn't blocking PerplexityBot if you want to be cited.
- Ask Perplexity a question your page answers and see whether it's cited, allowing a day or two after changes.
Should you allow PerplexityBot?
If you want to be cited in Perplexity's answers, yes. Perplexity is one of the more consistent sources of AI referral traffic because citations are prominent and clickable. Blocking it has no effect on Google rankings.
The equivalent rules for other AI companies are in GPTBot robots.txt and ClaudeBot robots.txt, and the full picture is in the AI search readiness checker guide.
Common PerplexityBot robots.txt mistakes
Copying a "block AI" template. Wrong: pasting a list of 30 AI user agents set to Disallow: / to stop training, which includes PerplexityBot and removes you from Perplexity's citations. Right: PerplexityBot is a search-style crawler; block the training crawlers you object to and leave it allowed.
Expecting Disallow to stop Perplexity-User. Wrong: adding a Perplexity-User group and assuming live fetches stop. Right: treat that rule as a stated preference; use authentication or firewall rules where you need enforcement.
Forgetting the firewall. Wrong: allowing PerplexityBot in robots.txt while a Cloudflare setting blocks AI crawlers at the edge, so the crawler gets 403s. Right: check the CDN's AI bot settings match your robots.txt, and confirm in logs that PerplexityBot receives 200 responses.
Testing once on the homepage. Wrong: running one check on / and assuming the whole site is open. Right: testing one URL per section, because a rule like Disallow: /guides/ only shows up on those pages.
Not watching referrals. After allowing PerplexityBot, add perplexity.ai as a segment in your analytics. The landing pages it sends visitors to show which of your pages Perplexity is citing, which is the clearest feedback you'll get on what to write more of.
Frequently asked questions
What is PerplexityBot?
PerplexityBot is Perplexity's crawler. It indexes web pages so Perplexity can find and cite them in its answers. It identifies itself as PerplexityBot and follows robots.txt rules.
Does PerplexityBot respect robots.txt?
Perplexity says PerplexityBot does and won't index content from sites that disallow it. Its user-triggered fetcher, Perplexity-User, generally doesn't follow robots.txt, because the fetch is requested by a person.
How do I completely block Perplexity?
Disallow PerplexityBot in robots.txt and add firewall or CDN rules for Perplexity's published IP ranges and user agents. For content that must stay private, use authentication, which no crawler can bypass.
Does Perplexity send traffic to websites?
Yes. Perplexity lists its sources with links alongside answers, and many sites see referral visits from perplexity.ai in their analytics. Allowing PerplexityBot is a prerequisite for being one of those sources.
Check your own page
Seotific audits a URL against the keyword it should rank for and lists the fixes in order of impact. Free plan, no card.
Run a free audit