PerplexityBot

What PerplexityBot does

PerplexityBot is the crawler Perplexity operates to fetch web pages that support its answer engine, an assistant built around live search rather than a fixed, pre-trained knowledge base alone. It carries its own documented user-agent token, PerplexityBot, so a site owner can write rules targeting it specifically in robots.txt, separate from any other crawler touching the site. Because Perplexity's product leans heavily on live retrieval, answering a query by fetching current web content rather than relying only on what a model learned during training, PerplexityBot's access has a fairly direct line to what shows up in a Perplexity answer, more so than a training-only crawler whose effect is delayed and gradual. This makes PerplexityBot access worth checking specifically rather than lumping it in with a generic AI-crawler decision, since blocking it can have a faster, more visible effect on citation than blocking a crawler used purely for model training. Perplexity publishes documentation describing the crawler's intended behavior, including that it is meant to respect robots.txt rules, the baseline every other major AI crawler is also expected to follow.

How it works, and the compliance question around it

PerplexityBot is documented to check robots.txt before requesting a page and to follow the rules written under its own user-agent name, the same baseline behavior expected of GPTBot, ClaudeBot and other established AI crawlers. In 2024, however, public reporting raised cases where PerplexityBot appeared to access content on sites whose robots.txt disallowed it, and Perplexity disputed that characterization, contesting the tests and conclusions behind those reports. As of 2026, the question has not settled into a clean, universally agreed record on either side, which is why the accurate way to describe it is that practitioners have reported cases of access despite robots.txt rules, not that the crawler is confirmed to ignore the standard outright. For a site owner, the practical response is the same regardless of where that dispute lands: write the robots.txt rule you want, and separately verify through server logs whether requests matching Perplexity's crawler are showing up on pages you intended to block. A mismatch between the rule and the logs is worth investigating and, if it persists, worth raising with Perplexity directly rather than assuming the rule alone has settled the matter. This is one of the few AI crawlers where checking logs, not just the robots.txt file, is a genuinely necessary step rather than a belt-and-suspenders extra.

Why it matters for AI visibility

Perplexity is one of the AI assistants buyers now use to research a category directly, and because its answers lean on live retrieval, PerplexityBot's access has a more immediate connection to whether a brand gets cited than a training-only crawler does. A brand that wants to appear in Perplexity's answers needs PerplexityBot to actually reach its pages, not just a robots.txt rule that states it should be allowed. Given the reported compliance questions, this is also one of the few crawler checks worth verifying twice, once in the rule and once in the logs, rather than treating an Allow rule as the end of the check. Getting this wrong in either direction carries a real cost: blocking PerplexityBot removes a brand from one of the faster-moving AI answer surfaces, while assuming access is working without checking the logs risks missing a mismatch that already has real, documented precedent behind it.

Good practices

• Add an explicit "User-agent: PerplexityBot" rule in robots.txt rather than relying on a generic wildcard.
• Check server logs for PerplexityBot requests, not just the robots.txt rule, given the reported compliance questions.
• Investigate any mismatch between what your rule states and what the logs show, and document it.
• Re-check the rule after any CMS or CDN change, since these can silently reset it.
• Treat PerplexityBot access as higher-priority to verify than a training-only crawler, since Perplexity's answers lean more directly on live retrieval.
• Fold this check into the wider crawler and schema audit in Truffle's LLM SEO guide.

Common mistakes

• Assuming a robots.txt rule alone guarantees PerplexityBot's behavior, without checking server logs to confirm it.
• Treating the 2024 reporting as settled proof the crawler ignores robots.txt outright, when Perplexity disputes that characterization.
• Blocking PerplexityBot without weighing that Perplexity's answers depend more directly on live crawl access than a training-only crawler does.
• Never revisiting the rule after a site migration, when a reset robots.txt file is a common, easy-to-miss cause of unexpected access.

Related terms

• GPTBot: OpenAI's crawler, generally treated as compliant with robots.txt in contrast to the disputed reports around PerplexityBot.
• ClaudeBot: Anthropic's crawler, another AI bot worth auditing alongside PerplexityBot.
• OAI-SearchBot: OpenAI's crawler for ChatGPT's search feature, a closer functional match to PerplexityBot's live-retrieval role than GPTBot is.
• robots.txt for AI Crawlers: the standard PerplexityBot is documented to check, and the file where its access is controlled.

Frequently asked questions