Quick answer
GPTBot is OpenAI's training crawler. To block it, add "User-agent: GPTBot" and "Disallow: /" to robots.txt. Blocking GPTBot doesn't remove you from ChatGPT search, which uses OAI-SearchBot; allow that separately if you want to be cited. ChatGPT-User fetches pages on user request, and OpenAI says robots.txt may not apply to it.
"Block GPTBot" became standard advice for publishers in 2023, when OpenAI first documented the crawler. Since then OpenAI has split its crawling into three user agents, and blocking the wrong one can remove you from ChatGPT search, which is usually not what people want.
OpenAI's three user agents
| User agent | What it does | Effect of blocking |
|---|---|---|
| GPTBot | Crawls content that may be used to train OpenAI's models | Your content isn't used for training |
| OAI-SearchBot | Indexes pages for ChatGPT's search features | Your pages aren't shown in ChatGPT search results |
| ChatGPT-User | Fetches a page when a ChatGPT user's request needs it | OpenAI says robots.txt rules may not apply |
OpenAI says the settings are independent: you can allow OAI-SearchBot and disallow GPTBot. It also says it can take about 24 hours for its search systems to reflect a robots.txt change.
Copy-paste robots.txt rules
Block training, keep ChatGPT search (the most common choice):
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Allow everything (maximum visibility):
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Allow: /Block training on one section only, for example premium articles:
User-agent: GPTBot
Disallow: /premium/Block all OpenAI crawling (also leaves ChatGPT search):
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /Remember that a crawler follows only the most specific group that names it. If you give OAI-SearchBot its own group, it ignores your User-agent: * rules, so repeat any Disallow lines (such as /admin/ or /cart/) that should still apply.
How to test the rules
- Publish the new robots.txt and open
https://yoursite.com/robots.txtto confirm it's live. - Run the AI crawler access checker on a page and confirm GPTBot and OAI-SearchBot show the status you intended.
- Check your CDN or firewall (for example Cloudflare's AI Crawl Control) isn't blocking the crawler you allowed.
- After a few days, search your server logs for the user agents and confirm the allowed ones receive status 200.
Logs can contain fake GPTBot requests from scrapers using its name. OpenAI publishes the IP ranges its crawlers use, so you can tell real requests from impostors.
Should you block GPTBot?
It depends on how you feel about your content training AI models. Blocking GPTBot has no effect on Google rankings and, as long as OAI-SearchBot is allowed, no effect on appearing in ChatGPT search. Many news publishers block it; many businesses that want brand awareness in AI answers allow it. There's no SEO penalty either way.
What you should avoid is blocking OAI-SearchBot by accident. Copy-paste "block AI" templates often list every OpenAI user agent together. If being cited in ChatGPT matters to you, see how to get cited by ChatGPT, and for Anthropic's equivalent crawlers see ClaudeBot robots.txt. The full picture across AI products is in the AI search readiness checker guide.
GPTBot robots.txt mistakes to avoid
Blocking with User-agent: * only. Wrong: relying on a wildcard group to cover OpenAI's crawlers, then adding Disallow: / to it for a staging reason and forgetting it on launch. Right: give the OpenAI user agents their own named group, so their access is a deliberate, visible choice.
Listing GPTBot under the wrong rule. Wrong: User-agent: GPTBot followed by Allow: / in a file whose next group, meant for OAI-SearchBot, ends up empty because of a blank line in the wrong place. Right: one blank line between groups and every group with at least one rule; then test each user agent with the checker.
Blocking the CDN path but not the page. Wrong: disallowing /wp-content/ or /_next/ for GPTBot to save bandwidth, which also blocks the CSS and images a renderer would need. Right: block whole sections by URL path, not shared asset folders, unless you're sure nothing needs them.
Assuming robots.txt is retroactive. Blocking GPTBot today stops future crawling. It doesn't remove content already collected; OpenAI's documentation covers training opt-outs separately from robots.txt.
Frequently asked questions
What is GPTBot?
GPTBot is OpenAI's web crawler that collects content which may be used to train its AI models. It identifies itself with "GPTBot" in its user agent string and follows robots.txt rules.
Does blocking GPTBot remove my site from ChatGPT?
No. ChatGPT's search results come from OAI-SearchBot crawling and search partners, not GPTBot. Block GPTBot to opt out of training, and keep OAI-SearchBot allowed if you want your pages to appear in ChatGPT search.
Does GPTBot respect robots.txt?
OpenAI says GPTBot follows robots.txt. Changes take time to reach its systems; OpenAI cites about 24 hours for search. If you see GPTBot ignoring your rules, check the IP addresses, as impostors often use its name.
Is blocking GPTBot bad for SEO?
No. GPTBot has no role in Google Search, so blocking it doesn't affect Google rankings. It only affects whether your content may be used to train OpenAI's models.
Check your own page
Seotific audits a URL against the keyword it should rank for and lists the fixes in order of impact. Free plan, no card.
Run a free audit