OAI-SearchBot is OpenAI's web crawler for search. It visits websites so they can be surfaced in ChatGPT's search features, including the answers and source links people see when ChatGPT searches the web. If a plumbing company blocks OAI-SearchBot in its robots.txt file, its pages generally won't be used in ChatGPT search answers, even if a customer asks ChatGPT for a plumber in that exact town.
For businesses, OAI-SearchBot is one of the simplest technical settings that affects AI visibility. It is separate from OpenAI's training crawler, so you can let your content appear in ChatGPT search without agreeing to have it used for model training. Getting this setting wrong, often by accident, can quietly remove your site from one of the largest AI search tools.
What is OAI-SearchBot?
OAI-SearchBot is one of several automated agents OpenAI uses on the web. OpenAI's crawler documentation lists its purpose as search: it is used to surface websites in ChatGPT search features. It identifies itself with a user agent string that includes OAI-SearchBot and a link to openai.com/searchbot. OpenAI notes that the version number in the string may change.
OpenAI publishes the IP addresses OAI-SearchBot uses at openai.com/searchbot.json, so site owners and firewall providers can verify that a visit really comes from OpenAI and not from an impostor using the same name.
OpenAI's main web agents are:
- OAI-SearchBot: crawls to surface sites in ChatGPT search.
- GPTBot: crawls content that may be used to train OpenAI's foundation models.
- ChatGPT-User: visits pages when a user's request in ChatGPT or a custom GPT needs it. It is not used for automatic crawling.
- OAI-AdsBot: checks landing pages submitted as ads on ChatGPT.
Why OAI-SearchBot matters
It controls whether ChatGPT search can use your content. OpenAI says sites that have opted out of OAI-SearchBot won't appear in ChatGPT search answers, though they can still show up as navigational links. OpenAI's publisher FAQ warns that blocking it may mean your content isn't included in summaries and snippets. For a business that wants AI visibility, that is a big loss.
Search and training are separate choices. OpenAI says each robots.txt setting is independent. You can allow OAI-SearchBot and disallow GPTBot. Many businesses want exactly that: visibility in answers without contributing to training. If you allow both, OpenAI says it may use the results of one crawl for both purposes.
It can drive measurable traffic. OpenAI's publisher FAQ says referral traffic from ChatGPT includes the UTM parameter utm_source=chatgpt.com. That makes it easier to track AI referral traffic from ChatGPT in your analytics.
How OAI-SearchBot works
Crawling and robots.txt
Like other well-behaved crawlers, OAI-SearchBot checks your robots.txt file to see what it is allowed to fetch. You control it with a user-agent group for OAI-SearchBot. OpenAI says changes to robots.txt can take about 24 hours to affect search.
Opting out vs. noindex
Blocking a page in robots.txt doesn't guarantee it disappears entirely. OpenAI's publisher FAQ says disallowed pages may still appear as a link and title if OpenAI finds the URL elsewhere and it seems relevant. To keep a page out, OpenAI recommends a noindex meta tag, and the crawler must be allowed to crawl the page to see that tag.
Verifying real visits
Any bot can claim to be OAI-SearchBot in its user agent. To confirm a visit is genuine, compare the requesting IP address with the ranges OpenAI publishes. This is useful during log file analysis and when setting firewall rules.
How to allow OAI-SearchBot
- Check your robots.txt. Visit yourdomain.com/robots.txt and look for any rule that blocks OAI-SearchBot, or a blanket rule that blocks all bots.
- Add a clear rule. A group with "User-agent: OAI-SearchBot" followed by "Allow: /" permits crawling of your whole site. Disallow only the folders you truly want excluded.
- Decide on GPTBot separately. Add its own group if you want to block training while keeping search access.
- Check your firewall and CDN. Bot protection tools can block AI crawlers even when robots.txt allows them. OpenAI recommends allowing requests from its published IP ranges.
- Use noindex for pages that must stay out. For pages you never want shown, use noindex and let the crawler reach the page.
- Wait and measure. Allow about a day for changes, then watch for OAI-SearchBot in server logs and utm_source=chatgpt.com in analytics.
Common mistakes and misconceptions
- "Blocking GPTBot keeps me out of ChatGPT search." GPTBot is for training. ChatGPT search depends on OAI-SearchBot. Blocking only GPTBot does not remove you from search answers.
- "Blocking OAI-SearchBot protects my content from training." Training is controlled by GPTBot. Blocking OAI-SearchBot mostly costs you search visibility.
- "robots.txt is enough to hide a page." Disallowed URLs can still appear as a link and title. Use noindex for pages that must stay out.
- "If robots.txt allows it, it can reach my site." Firewalls, CDNs and security plugins may block AI crawlers on their own. Check those settings too.
- "Anything named OAI-SearchBot is OpenAI." User agents can be faked. Verify with OpenAI's published IP list.
OAI-SearchBot vs. GPTBot
Both are OpenAI crawlers, but they do different jobs. OAI-SearchBot gathers pages so ChatGPT search can find, summarize and link to them. GPTBot gathers content that may be used to train OpenAI's models. Allowing OAI-SearchBot supports visibility in ChatGPT answers. Blocking GPTBot signals that you don't want your content used for training. OpenAI treats the two settings independently, so you can choose one without the other.
How Butter approaches OAI-SearchBot
At Butter, checking AI crawler access is one of the first steps in our GEO work. We review a client's robots.txt, firewall and CDN settings to confirm that search crawlers like OAI-SearchBot can reach the pages that matter, and we talk through the training question separately so the client can make an informed choice about GPTBot.
From there, our indexing strategy makes sure key pages are reachable and indexable, and our monthly AI prompt testing shows whether those pages are showing up in ChatGPT answers. We can't promise ChatGPT will cite you, but we can make sure a crawler setting isn't the reason it doesn't.
Frequently asked questions
Should I allow OAI-SearchBot?
If you want your site to appear in ChatGPT search answers, yes. OpenAI says sites that opt out of OAI-SearchBot won't appear in ChatGPT search answers, though they can still appear as navigational links.
Is OAI-SearchBot the same as GPTBot?
No. OAI-SearchBot is used for ChatGPT search, while GPTBot crawls content that may be used to train OpenAI's models. You can allow one and block the other in robots.txt.
How long do robots.txt changes take to apply?
OpenAI says it can take about 24 hours for its systems to adjust after you update robots.txt. Check your server logs afterward to confirm the crawler is visiting.
How can I tell if OAI-SearchBot is crawling my site?
Look for OAI-SearchBot in the user agent field of your server logs. Then confirm the IP address matches the list OpenAI publishes at openai.com/searchbot.json.