Quick answer
ClaudeBot is Anthropic's training crawler. Block it with "User-agent: ClaudeBot" and "Disallow: /". Anthropic also runs Claude-SearchBot, which indexes pages for Claude's search results, and Claude-User, which fetches pages when someone asks Claude. All three follow robots.txt, so you can block training and still be cited.
Anthropic's crawler setup changed in early 2026. Instead of one bot, its documentation now describes three, each for a different job, and each controlled separately in robots.txt. Older robots.txt files that only mention ClaudeBot, or the retired names Claude-Web and anthropic-ai, no longer describe what Anthropic actually runs.
Anthropic's three bots
| User agent | What it does | Effect of blocking |
|---|---|---|
| ClaudeBot | Collects web content that may be used to train Claude | Your content isn't used for training |
| Claude-SearchBot | Navigates the web to improve the quality of Claude's search results | Anthropic warns it may reduce your visibility in Claude's answers |
| Claude-User | Fetches pages when a Claude user asks a question that needs them | Claude can't retrieve your page for users |
Anthropic's documentation, updated in February 2026, says all three honour robots.txt. That's a difference from OpenAI and Perplexity, which say their user-triggered fetchers may not follow it.
Copy-paste robots.txt rules
Block training, keep Claude search and user fetches:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /Allow all Anthropic bots:
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /Block everything from Anthropic:
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /As with any named group, these bots ignore your User-agent: * rules once they have their own group, so copy over any private paths you still want blocked, such as /admin/ or /checkout/.
Old user agent names
Robots.txt files written in 2023 and 2024 often block anthropic-ai and Claude-Web. Anthropic no longer documents those names for its current crawlers. Leaving them in place does no harm, but on their own they don't control what Anthropic crawls today. Add the three current names explicitly.
How to test the rules
- Publish the updated robots.txt and confirm it loads at
/robots.txt. - Run the AI crawler access checker and check ClaudeBot, Claude-SearchBot and Claude-User each show the intended status.
- Check Cloudflare or your firewall isn't blocking Anthropic's crawlers at the network level.
- Watch your logs for the three user agents over the following days, and confirm each allowed bot receives status 200 rather than 403. If ClaudeBot keeps visiting after you block it, give it a day: crawlers cache robots.txt and pick up changes on their next fetch of the file.
Should you block ClaudeBot?
Blocking ClaudeBot is a decision about training, and it has no effect on Google rankings. If you want to appear in Claude's answers, allow Claude-SearchBot and Claude-User, because those are what make your pages available when Claude searches or fetches. Anthropic's own documentation warns that blocking Claude-SearchBot may reduce your visibility.
The equivalent rules for OpenAI are in GPTBot robots.txt and for Perplexity in PerplexityBot robots.txt. The broader guide to AI visibility is the AI search readiness checker.
ClaudeBot robots.txt decisions by site type
The right setup depends on what you publish and why.
Publishers and paywalled content. Block ClaudeBot to keep articles out of training. Allow Claude-SearchBot only if you're comfortable with Claude summarising and linking to your public pages, and keep paywalled paths disallowed for all three bots. Robots.txt isn't access control, so the paywall itself must stop unauthenticated fetches.
Businesses that want to be recommended. Allow all three. When someone asks Claude for a supplier, a tool or a service in your area, Claude-SearchBot's index and Claude-User's live fetches are how your pages get considered. Blocking either removes you from those answers.
Documentation and developer tools. Allow all three, and consider publishing an llms.txt summary. Developers frequently ask Claude to read documentation pages, which is exactly what Claude-User does.
Sites with heavy crawl load. If Anthropic's crawlers are using too much bandwidth, use a Crawl-delay line or rate-limit at the CDN rather than blocking outright. Anthropic's documentation says ClaudeBot respects Crawl-delay; Google ignores it, so it won't affect Googlebot.
Whatever you choose, write it down in a comment at the top of robots.txt (lines starting with # are ignored by crawlers) so the next person to edit the file knows the rules are deliberate.
Frequently asked questions
What is ClaudeBot?
ClaudeBot is Anthropic's web crawler that collects content that may be used to train Claude models. Anthropic also runs Claude-SearchBot for search and Claude-User for user-requested fetches, each controlled separately in robots.txt.
Does blocking ClaudeBot stop Claude from citing my site?
Not on its own. Citations depend on Claude-SearchBot and Claude-User. If you block only ClaudeBot, your content is excluded from training but can still appear in Claude's search answers and user fetches.
Does ClaudeBot respect robots.txt?
Anthropic says all three of its bots honour robots.txt, including Claude-User. If you see "ClaudeBot" requests ignoring your rules, check that they come from Anthropic, since scrapers sometimes borrow well-known bot names.
Should I still block anthropic-ai and Claude-Web?
You can leave those lines, but they don't cover Anthropic's current crawlers. Add ClaudeBot, Claude-SearchBot and Claude-User explicitly with the rules you want.
Check your own page
Seotific audits a URL against the keyword it should rank for and lists the fixes in order of impact. Free plan, no card.
Run a free audit