robots.txt for AI Crawlers

What the file does, and what it does not control

robots.txt is a plain text file placed at a domain's root, at example.com/robots.txt, and it has worked the same way since 1994: a crawler checks the file before requesting pages and follows whatever Allow or Disallow rules apply to its own name. What makes this relevant to AI visibility now is that each AI company runs its own named crawler, and a site owner can permit one while blocking another in the same file. GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot each get their own User-agent block, so allowing OpenAI's crawler does not automatically allow Anthropic's, and blocking one training crawler does not block a separate search crawler from the same company. The file only ever controls crawling, the act of a bot requesting and reading a page. It does not control indexing directly, and it does not guarantee a blocked page stays invisible everywhere: a URL disallowed in robots.txt can still be referenced by another system if it appears linked or described on a page the crawler was allowed to read. That distinction, access versus visibility, is easy to state and easy to get backwards when auditing a site.

How a site owner writes and tests the rules

A robots.txt entry names a crawler with User-agent, then lists paths under Allow or Disallow. A block written as "User-agent: GPTBot" followed by "Disallow: /" blocks OpenAI's crawler from the entire site, while the same two lines under "User-agent: ClaudeBot" would leave Anthropic's crawler untouched. A generic block, "User-agent: *", applies to every crawler that has no more specific rule of its own, which is why a specific AI-crawler rule placed after a general one can accidentally undo an intended allowance, or the reverse. The standard is voluntary: nothing in the file itself stops a bot from requesting a page anyway, it only states the site owner's intent, and a compliant crawler is one that checks the file first and honors it. Most established AI crawlers document that they respect robots.txt, and treating the file as a request rather than a lock is part of why disputes over whether a specific crawler actually honored it do come up in practice. Because each AI company runs crawlers for different jobs, training a model versus live search versus answering a query in the moment, a site owner can reasonably allow one and block another for the same company without inconsistency. Testing means fetching the live file at the root URL after any change, since a CMS migration or a CDN configuration can silently rewrite or reset it, and a rule that looked correct in staging is not confirmed until it is checked on the production domain.

The last row is the one that catches people out: a specific block placed after a general one overrides it for that crawler only. Blocking a company's training crawler also leaves its separate search crawler untouched, which is why OAI-SearchBot needs its own line.

Why it matters for AI visibility

If an AI crawler cannot fetch a page, nothing downstream can happen: no citation, no mention, no summary drawing on that content, regardless of how well the page is written or structured. A blocked crawler is not a ranking problem to fix later, it is a visibility problem that makes every other optimization moot for that specific AI system. This is also where an old privacy-era habit causes damage without anyone noticing: a blanket "Disallow: /" left over from a prior audit, meant to keep a different kind of bot out, can block GPTBot, ClaudeBot and PerplexityBot at the same time, with no error message anywhere to flag it. Checking robots.txt is one of the first things worth confirming before assuming a visibility gap has anything to do with content quality. It also has to be rechecked, not confirmed once: new AI crawlers appear, and a rule set that was complete a year ago can already be missing an entry for a crawler that did not exist yet.

Good practices

• Fetch your live robots.txt directly (yourdomain.com/robots.txt) after any CMS or CDN change, since deployments can silently reset it.
• List GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot by name instead of relying only on a generic wildcard rule.
• Place crawler-specific rules where they will not be overridden by a broader wildcard block placed after them.
• Re-check the file whenever a new AI crawler becomes publicly known, since the list of relevant bots changes over time.
• Treat an Allow rule as a starting point, not a guarantee, and confirm separately whether the content is actually being cited. Truffle's LLM SEO guide covers where crawler permissions fit among the checks worth automating.
• Keep a single source of truth for the file rather than environment-specific copies that can drift out of sync.

Common mistakes

• Leaving a blanket Disallow rule from an earlier, unrelated audit in place, which blocks every AI crawler without anyone intending it.
• Assuming a page blocked in robots.txt is fully invisible everywhere, when the file only ever governs crawling, not every path to visibility.
• Confusing separate crawlers from the same company, allowing GPTBot while assuming that also covers OAI-SearchBot, or the reverse.
• Checking robots.txt once at launch and never again, missing new crawlers that appear later.

Related terms

• GPTBot: OpenAI's crawler for training data, one of the crawlers robots.txt rules can allow or block by name.
• ClaudeBot: Anthropic's crawler, controlled the same way as any other named AI crawler in robots.txt.
• PerplexityBot: Perplexity's crawler, worth checking separately since it has raised its own compliance questions.
• OAI-SearchBot: OpenAI's separate crawler for ChatGPT's search feature, not the same bot as GPTBot and not covered by the same rule.
• llms.txt: a newer, unconfirmed proposal for summarizing a site for AI, often confused with the established robots.txt standard.

Frequently asked questions