OAI-SearchBot

What OAI-SearchBot is, and why OpenAI runs two crawlers

OAI-SearchBot is a crawler OpenAI operates specifically to support ChatGPT's search capability, fetching current web pages so ChatGPT can answer a query using live information rather than relying only on what a model learned during training. It carries its own documented user-agent token, OAI-SearchBot, distinct from GPTBot, the crawler OpenAI uses separately to gather content for training its models. The two exist because they serve genuinely different jobs: GPTBot's fetches become part of a training dataset, processed on OpenAI's own schedule and reflected only once a newer model version ships; OAI-SearchBot's fetches support a live answer, closer in spirit to what a search engine's crawler does, retrieving current content at or near the moment a user asks a question. A site owner who confuses the two and writes a robots.txt rule for only one of them ends up with an incomplete setup: blocking GPTBot without also addressing OAI-SearchBot still leaves ChatGPT's search feature able to fetch the site, and the reverse is also true, each rule only ever governs the crawler it names.

How it works, and how it differs from GPTBot in practice

OAI-SearchBot checks robots.txt before requesting a page, following whatever Allow or Disallow rules are written under "User-agent: OAI-SearchBot" specifically, the same baseline behavior documented for GPTBot and other established AI crawlers, but governed by its own separate rule. Because its purpose is live search rather than training, its access has a more direct line to what a user might see in a ChatGPT search answer relatively soon after a page becomes reachable, compared to GPTBot, where the effect is delayed until a future model is trained and released. This means the two crawlers can reasonably be treated differently in a robots.txt strategy: a site owner might, for instance, allow OAI-SearchBot to support appearing in ChatGPT's live search answers while blocking GPTBot to opt out of having content used for model training, a distinction that only makes sense once the two crawlers are understood as separate tools with separate effects. Verifying OAI-SearchBot's access follows the same pattern as any other AI crawler: confirm the robots.txt rule under its specific user-agent name, then check server logs for actual requests carrying that user-agent string, since the rule alone only states intent and does not confirm the crawler is actually behaving as expected on the live site.

Why it matters for AI visibility

ChatGPT's search feature is one of the more direct paths from a live web page to a cited answer, closer in mechanism to how a traditional search engine surfaces a result than to how a trained model recalls something it learned months earlier. That makes OAI-SearchBot's access a meaningful, separate lever from the GPTBot decision, and treating the two as one setting is one of the more common, avoidable gaps in an AI-visibility setup. A brand that carefully manages GPTBot but never checks OAI-SearchBot specifically can end up blocked from ChatGPT's search answers without realizing it, simply because a generic AI-crawler rule did not happen to name this particular token. Checking both crawlers by name, not as a single combined decision, is a small step with an outsized effect on whether a brand shows up in this specific, increasingly used ChatGPT feature that buyers rely on for direct, sourced answers.

Good practices

• Write a robots.txt rule that names "User-agent: OAI-SearchBot" specifically, separate from any rule covering GPTBot.
• Do not assume a GPTBot rule, allow or block, applies to OAI-SearchBot as well; verify each independently.
• Check server logs for OAI-SearchBot requests to confirm the crawler is actually reaching the pages you expect.
• Decide the training question, GPTBot, and the live-search question, OAI-SearchBot, as two separate choices, not one.
• Re-check both rules after any CMS or CDN change, since deployments can silently reset robots.txt.

Common mistakes

• Confusing OAI-SearchBot with GPTBot, and assuming one rule covers both.
• Blocking OAI-SearchBot by accident through an overly broad rule meant to target only GPTBot.
• Never checking server logs, so a misconfigured rule for either crawler goes unnoticed.
• Treating ChatGPT search visibility and ChatGPT training visibility as the same problem with the same fix.

Related terms

• GPTBot: OpenAI's separate crawler for training data, easy to confuse with OAI-SearchBot but governed by its own rule.
• ClaudeBot: Anthropic's crawler, an example of how each AI company documents its own crawler tokens.
• PerplexityBot: Perplexity's crawler, a closer functional match to OAI-SearchBot's live-retrieval role than a training-only crawler is.
• robots.txt for AI Crawlers: the file and standard that governs OAI-SearchBot's access, separately from any other AI crawler.
• Passage Ranking: the retrieval step that can determine which part of a page a live search feature like ChatGPT's actually surfaces.

Frequently asked questions