GPTBot is the web crawler OpenAI uses to collect publicly available content that may be used to train its generative AI foundation models, the models behind ChatGPT. Like Googlebot, it identifies itself with a user agent string, and site owners can manage its access through their robots.txt file.
For businesses, GPTBot is a decision point. Allowing it means your public content may help shape what future OpenAI models know about your brand. Blocking it signals that your content shouldn't be used for that training. Neither choice, on its own, controls whether you appear in ChatGPT search, which uses a different crawler.
What is GPTBot?
OpenAI first published GPTBot's documentation in August 2023. According to OpenAI's current crawler documentation, GPTBot crawls content that may be used in training its generative AI foundation models, and disallowing it tells OpenAI that a site's content should not be used for training.
Its user agent string includes the token GPTBot and a link to openai.com/gptbot, along with a version number that can change. OpenAI also publishes the IP addresses GPTBot uses, at openai.com/gptbot.json, so you can check whether a request claiming to be GPTBot is genuine.
GPTBot is one of several AI crawlers OpenAI runs. Each has a different job:
- GPTBot: collects content that may be used for model training.
- OAI-SearchBot: surfaces websites in ChatGPT search results. OpenAI says sites that opt out won't be shown in ChatGPT search answers, though they can still appear as navigational links.
- ChatGPT-User: visits pages when a user's action in ChatGPT or a custom GPT calls for it. Because these visits are user-initiated, OpenAI says robots.txt rules may not apply.
- OAI-AdsBot: checks the safety of landing pages submitted as ads on ChatGPT.
Why GPTBot matters
It shapes what models know about you. Content collected for training data can influence how future models describe your brand, your category and your competitors, even when they answer without searching the web. Blocking GPTBot removes your public pages from that pool going forward.
It's easy to block by accident. Many sites added blanket AI bot blocks over the past few years, and some CDN and security providers now offer AI bot blocking at the network level. Cloudflare, for example, announced in July 2025 that it was changing its default to block AI crawlers unless they pay for content. If you never made a deliberate choice, check what your current setup is doing.
It's separate from search visibility. OpenAI states that each robots.txt setting is independent of the others. You can block training while still allowing OAI-SearchBot, so your pages stay eligible for ChatGPT Search.
How GPTBot works
Crawling public pages
GPTBot fetches publicly accessible pages, much like a search engine crawler. It may not see content that depends on JavaScript. Vercel's December 2024 analysis of crawler traffic found that ChatGPT's crawlers fetched JavaScript files but didn't execute them. If your key content only appears after scripts run, server-side rendering is the safer choice.
Controlling GPTBot with robots.txt
You manage GPTBot in your robots.txt file using its user agent token. To block it from your whole site, add a group with User-agent: GPTBot followed by Disallow: /. To block only certain folders, list those paths instead. To allow it, you can leave it out, since crawlers are allowed by default, or add an explicit Allow rule.
OpenAI notes that if a site allows both GPTBot and OAI-SearchBot, it may use the results from just one crawl for both purposes. For search results, it says robots.txt changes can take about 24 hours to take effect.
Verifying real GPTBot traffic
Any bot can claim to be GPTBot in its user agent. To check, compare the request's IP address against OpenAI's published list. Log file analysis is the most reliable way to see how often GPTBot actually visits and which pages it requests.
How to decide whether to allow GPTBot
- Start with your goals. If broad brand awareness in AI tools matters, allowing training crawlers can help models learn accurate facts about you. If your content is your product, like paid research or proprietary data, blocking may make sense.
- Separate training from search. Decide on GPTBot and OAI-SearchBot independently. Many brands allow both; some block training only.
- Check every layer. Look at robots.txt, your CDN or firewall bot settings, and any security plugins. A block at any layer counts.
- Use folder-level rules. You can allow GPTBot on marketing pages and block it from areas like gated resources or customer portals.
- Review the decision regularly. Crawler names and options change. Revisit your rules when OpenAI or your hosting provider updates its documentation.
Common mistakes and misconceptions
"Blocking GPTBot removes me from ChatGPT." Not exactly. ChatGPT search relies on OAI-SearchBot, and user-triggered visits use ChatGPT-User. Blocking GPTBot only signals that your content shouldn't be used for training.
"Blocking GPTBot erases what models already learned." A robots.txt rule applies to future crawling. It doesn't remove content from models that have already been trained.
"robots.txt is a security tool." robots.txt is a set of instructions that well-behaved crawlers choose to follow. It doesn't stop bad actors or hide private content. Protect sensitive pages with authentication.
"One AI rule covers every AI company." Each company uses its own tokens. Blocking GPTBot does nothing about Anthropic's crawlers, Perplexity's crawlers or Google's Google-Extended control.
GPTBot vs. OAI-SearchBot
GPTBot is about training. OAI-SearchBot is about search. Blocking GPTBot signals that your content shouldn't be used in future training, while blocking OAI-SearchBot keeps your site out of ChatGPT search answers. For most businesses that want AI visibility, OAI-SearchBot is the one to keep open, whatever they decide about GPTBot.
How Butter approaches GPTBot
At Butter, AI crawler access is one of the first things we check, because no amount of great content helps if the bots that power AI search can't read it. In our GEO plan, monthly AI prompt testing and indexing strategy include reviewing which AI user agents your robots.txt and hosting setup allow, and flagging pages that rely on client-side rendering.
We don't make the training opt-out decision for you. Allowing search bots while restricting training bots is a business decision, so we explain the trade-offs and let you choose. On the GEO+SEO plan, robots.txt problems fall under the technical fixes we handle. You can see how this fits our broader GEO service.
Frequently asked questions
What is GPTBot used for?
OpenAI uses GPTBot to crawl public web content that may be used to train its generative AI foundation models. It is separate from OAI-SearchBot, which OpenAI uses to surface websites in ChatGPT search results.
Should I block GPTBot?
It depends on your goals. Allowing it can help future models learn accurate information about your brand. Blocking it keeps your content out of future training. Many businesses allow both, while some block training and keep search access open.
Does blocking GPTBot affect my Google rankings?
No. GPTBot is OpenAI's crawler and has no role in Google Search. Google's crawlers and controls, such as Googlebot and Google-Extended, are managed separately in robots.txt.
How do I know if GPTBot is visiting my site?
Check your server logs for requests with GPTBot in the user agent string. To confirm they are genuine, compare the IP addresses against the list OpenAI publishes at openai.com/gptbot.json.