PerplexityBot Explained: User Agents, Verification, and the robots.txt Controversy

PerplexityBot is Perplexity's search crawler that indexes public pages so they can be surfaced and linked in Perplexity answers, and it identifies itself as PerplexityBot/1.0 in your server logs.

PerplexityBot is Perplexity's crawler for search, not for model training. It sends the user agent PerplexityBot/1.0, reads server-delivered HTML, and indexes pages so Perplexity can cite and link them in answers. A second agent, Perplexity-User, fetches a page live when a user asks about it. You control PerplexityBot with robots.txt, and because the user agent can be forged, a genuine PerplexityBot request has to be verified rather than trusted on its name.

Quick answer: Perplexity runs two named agents. PerplexityBot (user agent PerplexityBot/1.0) crawls and indexes pages for Perplexity search and links, and Perplexity states it respects robots.txt. Perplexity-User (Perplexity-User/1.0) fetches a page in real time when a person asks a question, and Perplexity says this user-driven agent generally does not follow robots.txt. In August 2025, Cloudflare accused Perplexity of stealth crawling pages that had blocked it, using undeclared user agents and rotating IP addresses; Perplexity disputed the framing, arguing a user-driven assistant is not the same as a crawler. Because these agents read raw HTML and skip JavaScript, browser analytics miss them and your server log is the only reliable record.

What PerplexityBot is and what it does

PerplexityBot is the crawler Perplexity uses to discover and index public web pages. Its job is search: it reads your content so Perplexity can surface your page as a source and link to it inside an answer. Perplexity states plainly that PerplexityBot does not crawl for AI model training. That makes it closer to a search-engine spider than to a training bot like GPTBot or ClaudeBot. It is one of the major AI crawlers you can expect to see reading your pages.

If your goal is visibility in Perplexity answers, PerplexityBot is the agent you want reading your pages. Blocking it tends to reduce how often you appear as a cited source, which is usually the opposite of an SEO team's aim.

The exact Perplexity user agent strings

Perplexity publishes a fixed token for each agent. At the time of writing the full strings are:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Match on the product tokens PerplexityBot and Perplexity-User. The URL inside each string points back to Perplexity's documentation for that agent.

PerplexityBot vs Perplexity-User

Perplexity splits its activity into two agents with different rules.

PerplexityBot/1.0 -> indexing for search and links, follows robots.txt Perplexity-User/1.0 -> live fetch when a user asks, generally ignores robots.txt

PerplexityBot is the systematic crawler. Perplexity says it honors robots.txt, so a disallow rule for PerplexityBot should stop it from indexing the disallowed paths.

Perplexity-User fires when a person asks Perplexity a question that requires visiting a specific page. Perplexity describes this as a user action and says the agent generally does not apply robots.txt, on the reasoning that a human requested the fetch. This is the same logic OpenAI and Anthropic apply to their user-triggered agents.

Why a PerplexityBot user agent is not proof

A user agent header can be faked by anyone, and a request calling itself PerplexityBot inherits the trust the real crawler has earned. The log line shows a claimed identity, nothing more.

citAEOtion verifies every claimed PerplexityBot and Perplexity-User request against the operator behind the name and returns a verdict on each hit: verified, impostor, or unverifiable. That is also the only clean way to settle the robots.txt argument on your own site, because the question is never just whether Perplexity followed the rules but whether the request was Perplexity at all. Across the sites citAEOtion monitors, 23,614 requests claiming a trusted crawler name have failed verification; the breakdown is in our report on fake AI crawlers.

How to control PerplexityBot in robots.txt

Place your rules at the root of your domain. To allow PerplexityBot so you stay eligible to appear in Perplexity answers:

User-agent: PerplexityBot Allow: /

To block PerplexityBot from indexing your site:

User-agent: PerplexityBot Disallow: /

To attempt to block the user-driven agent as well:

User-agent: Perplexity-User Disallow: /

Keep in mind Perplexity's own statement that Perplexity-User generally does not follow robots.txt, so a disallow on that token may not stop a fetch a user triggered. If you need a hard stop, enforce it at the server or firewall level, and verify that what you blocked was actually Perplexity rather than something wearing its name.

The Cloudflare robots.txt dispute, stated neutrally

In August 2025, Cloudflare published research accusing Perplexity of what it called stealth crawling. Cloudflare said that when sites blocked the declared PerplexityBot user agent, it observed requests it attributed to Perplexity arriving from undeclared user agents and rotating IP addresses that were not on Perplexity's published list, reaching pages the sites had tried to block. Cloudflare responded by removing Perplexity from its list of verified bots and adding rules to block the activity.

Perplexity disputed Cloudflare's framing. Its position was that a user-driven assistant fetching a page on a person's behalf is not the same as an automated crawler, and that such user actions are not obligated to honor robots.txt in the way a bulk crawler is. Multiple outlets, including Search Engine Journal, Computing, and AppleInsider, covered both the Cloudflare allegation and Perplexity's rebuttal. The two companies did not reach a shared account of events. The practical lesson for a site owner is simple: robots.txt is a request, not a wall, and the only way to know what actually reached your pages is to read your own server logs and verify who was really behind each request.

How to see PerplexityBot on your site

PerplexityBot does not run JavaScript. It reads the HTML your server returns and stops. Google Analytics and other browser-tag tools only record visitors whose browsers run their script, which a bot never does, so GA shows zero Perplexity activity even during active crawling. The requests happen. The tracking tag never fires.

Your web server access log is the only record. That is what citAEOtion reads to track AI crawlers on WordPress. It reads real traffic, classifies each hit by its actual user agent, and separates PerplexityBot and Perplexity-User into categories such as AI Search and AI Assistant. You can see every Perplexity hit on your site, which pages were pulled, how often, and the HTTP status returned, which is exactly the data you need to check whether a block is actually holding. Given the Cloudflare dispute, that verification is worth having.

A single site is $34.99 a month, with agency plans and a Founders Club discount. Review the options on the citAEOtion pricing page, or book a live demo to see Perplexity traffic classified on a real WordPress site.

Perplexity's current agent reference and IP files are documented in Perplexity's official crawler documentation.

Frequently Asked Questions

Does PerplexityBot train AI models?

No. Perplexity states PerplexityBot crawls to index pages for search and links, not to train models. That sets it apart from training crawlers like GPTBot and ClaudeBot.

Does PerplexityBot respect robots.txt?

Perplexity says PerplexityBot follows robots.txt. It also says the separate Perplexity-User agent, which fetches pages on a user's request, generally does not. Cloudflare has disputed how consistently the rules are followed in practice.

How do I verify a real PerplexityBot request?

You cannot trust the user agent, since anyone can send it. The request has to be verified against the operator behind the name. citAEOtion does that automatically on every request and returns a verdict: verified, impostor, or unverifiable.

What was the Cloudflare accusation about?

In August 2025 Cloudflare said it saw traffic it attributed to Perplexity reaching blocked pages through undeclared user agents and rotating IPs. Perplexity argued its user-driven assistant is not a crawler bound by robots.txt. The two sides did not agree on the facts.

Why does Google Analytics miss PerplexityBot?

PerplexityBot does not run JavaScript, and Google Analytics only counts visitors whose browsers run its script. The bot never runs it, so GA cannot see it. A server-log tool like citAEOtion reads the raw request and does.

Verify what PerplexityBot actually fetched. Book a free demo