OAI-SearchBot

What OpenAI documents

OpenAI describes its crawlers on one page of its developer documentation, "Overview of OpenAI Crawlers". The page shows no date, so the quotes here are as retrieved on 15 September 2026. The entry for OAI-SearchBot begins:

"OAI-SearchBot is for search. OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features."

OpenAI introduced ChatGPT search on 31 October 2024. An update to that announcement says it has been "available to everyone in regions where ChatGPT is available" since 5 February 2025, with no signup required.

OpenAI's publishers FAQ names the condition for appearing there: "For your site content to be included in summaries and snippets in ChatGPT, make sure you aren't blocking OAI-SearchBot."

Four user agents, four jobs

OpenAI lists four user agents. Each robots.txt setting "is independent of the others", so a rule for one says nothing about the rest.

• OAI-SearchBot crawls for the search features of ChatGPT.
• GPTBot "is used to crawl content that may be used in training our generative AI foundation models." Disallowing it indicates that a site's content should not be used for that training.
• ChatGPT-User visits a page when a request in ChatGPT or a Custom GPT needs it. OpenAI says it "is not used for crawling the web in an automatic fashion" and that for these user-initiated actions "robots.txt rules may not apply".
• OAI-AdsBot checks landing pages of ads submitted to ChatGPT. It "only visits pages submitted as ads", and OpenAI says its data is not used to train generative AI foundation models.

The documentation gives the combination most site owners ask about as its own example. A site "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models." In robots.txt:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

If a site allows both, OpenAI notes that it "may use the results from just one crawl for both use cases to avoid duplicative crawling."

What blocking it does

"Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

A block removes a site from the answers, not necessarily from ChatGPT altogether. The publishers FAQ describes a similar case for ChatGPT Atlas: if OpenAI learns the URL of a disallowed page from a third-party search provider or from other pages, it "may surface just the link and page title". To prevent that, OpenAI points to the noindex meta tag, with a condition that is easy to miss: "in order for our crawler to read a meta tag, it must be allowed to crawl the relevant page(s)."

Changes take time: "For search results, please note it can take ~24 hours from a site’s robots.txt update for our systems to adjust."

A rule for ChatGPT-User does not change search visibility. OpenAI states that ChatGPT-User "is not used to determine whether content may appear in Search" and tells site owners to "use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl."

Confirming that a request is OAI-SearchBot

Anyone can copy a user-agent string. OpenAI publishes an example and notes that the version number may change:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

When the bot fetches robots.txt itself, the string can carry an additional robots.txt; marker, which separates those requests from others even in logs without paths. The address is the stronger check. OpenAI publishes the IP ranges for OAI-SearchBot at https://openai.com/searchbot.json. On 15 September 2026 the file listed 39 ranges and a creation time of 2 January 2026. A request that names OAI-SearchBot but comes from outside those ranges is not confirmed as OpenAI.

The help article on ChatGPT search adds the other side: the host or content delivery network has to let traffic from these addresses through. An Allow rule in robots.txt does not help if a firewall blocks the bot first.

Measuring the traffic

According to the publishers FAQ, "ChatGPT automatically includes the UTM parameter utm_source=chatgpt.com in referral URLs". Visits from ChatGPT can therefore be filtered by that source in an analytics tool. The parameter marks the visit. It does not say which answer the link appeared in.

Common mistakes

• Writing one rule for GPTBot and assuming it covers OAI-SearchBot. OpenAI documents each setting as independent.
• Blocking OAI-SearchBot to keep content out of training. Training is the GPTBot setting; blocking OAI-SearchBot removes the site from ChatGPT search answers.
• Adding noindex to a page that robots.txt disallows. The crawler has to be allowed on the page to read the tag.
• Trusting the user-agent string without checking the address against searchbot.json.
• Allowing the bot in robots.txt while the CDN or firewall blocks the published addresses.
• Expecting a robots.txt change to show in ChatGPT within minutes. OpenAI gives about 24 hours.

Related terms

• GPTBot: OpenAI's crawler for content that may be used in model training, controlled by its own rule.
• robots.txt for AI Crawlers: the file in which OAI-SearchBot and GPTBot are allowed or disallowed.
• PerplexityBot: Perplexity's crawler for its search results, documented with a similar split between crawler and user-triggered visits.
• ClaudeBot: Anthropic's crawler, controlled by its own robots.txt token.
• Answer Engine: a search system built to answer a question directly rather than list links.