The short answer
ClaudeBot is Anthropic's web crawler for collecting public content that may be used to train its Claude AI models. Anthropic also runs Claude-SearchBot, which supports search results in Claude, and Claude-User, which fetches pages when a user asks. Each one can be allowed or blocked separately in your robots.txt file.
- ClaudeBot collects training data; Claude-SearchBot and Claude-User do other jobs.
- Blocking ClaudeBot does not stop Claude-SearchBot unless you block that too.
- Anthropic says its bots follow robots.txt and has said ClaudeBot honors Crawl-delay.
- Older names like anthropic-ai and Claude-Web still appear in some blocklists.
What does ClaudeBot do?
ClaudeBot visits public web pages and collects content that may help train Anthropic's Claude models.
Anthropic says its crawlers follow robots.txt. It also says they do not access password-protected or login-gated pages.
ClaudeBot does not control what Claude says about you today. It affects what future Claude models may learn from your public pages.
As with any crawler, details can change. Check Anthropic's help center for its current crawler documentation.
Should you block ClaudeBot?
Like any training bot, this is a business choice. Weigh what you gain against what you want to protect.
- Allow it if you want future Claude models to know your brand, services and facts.
- Block it if you sell content, have licensing concerns or prefer not to contribute to training.
- Block only certain folders if just part of your site is sensitive.
- Whatever you choose for ClaudeBot, decide separately about Claude-SearchBot and Claude-User.
ClaudeBot vs Claude-SearchBot vs Claude-User
Anthropic splits its bots by purpose, much like OpenAI. That lets you opt out of one job without losing the others.
Each agent reads only the robots.txt group that names it. If none names it, it follows the general User-agent: * group.
| User agent | Job | If you block it |
|---|---|---|
| ClaudeBot | Collects training data | Future content is not collected for training |
| Claude-SearchBot | Supports search results in Claude | Your pages may be less likely to appear in Claude's search answers |
| Claude-User | Fetches a page when a user asks | Claude may not open your page during a conversation |
How to block ClaudeBot but stay visible in Claude
This sample opts out of training while keeping search and user fetches open. Flip the rules if you want the opposite.
Changes are not instant. Crawlers re-read robots.txt from time to time, so allow a little while before checking your logs.
Blocking also only works going forward. It does not remove content that was collected before the rule existed.
User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
Can you slow ClaudeBot down instead of blocking it?
Sometimes the worry is server load, not training. Crawl-delay is an unofficial robots.txt rule that asks a bot to wait between requests.
Google ignores Crawl-delay, but Anthropic has said ClaudeBot honors it. Check the current docs before relying on this.
The number is the wait in seconds between requests. A small delay is usually enough to ease load on a modest server. Caching and a CDN help with all bots, not just this one.
User-agent: ClaudeBot Crawl-delay: 5
Old Anthropic user agents you may still see
Older robots.txt templates often list anthropic-ai or Claude-Web. These were earlier names, and Anthropic's current docs list the three agents above.
Old entries do no harm, but they can give a false sense of control. Make sure your rules name ClaudeBot, Claude-SearchBot and Claude-User directly.
How to check your Claude crawler settings
WebValet's free check reads your robots.txt rules for the main AI bots. Claude answer tracking is available as a paid add-on to WebValet Pro.
- 1
Search your robots.txt
Look for ClaudeBot, Claude-SearchBot, Claude-User and any old Anthropic names.
- 2
Check broad rules
Make sure a User-agent: * block is not shutting out Claude's search bot by accident.
- 3
Review CDN and firewall settings
Some services block AI bots with one switch. Confirm it matches your robots.txt choices.
- 4
Read your server logs
Search for Claude in the user-agent field to see which agents actually visit.
Questions people ask
Does blocking ClaudeBot stop Claude mentioning my brand?
No. Claude can still know about you from other sources and from content collected earlier. Its search features use Claude-SearchBot, which you control separately. Third-party pages that mention you also remain readable.
Does ClaudeBot follow robots.txt?
Anthropic says it does. If traffic claiming to be ClaudeBot ignores your rules, it may be an impostor using the name. Check your rule spelling too, since a typo means the rule never applies.
Will ClaudeBot read pages behind a login?
Anthropic says its crawlers do not access password-protected or login-gated pages. Keep private content behind a login rather than relying on robots.txt.
Does WebValet track Claude answers?
Yes, as a paid add-on to Pro. Pro itself tracks ChatGPT, Gemini and Perplexity.
How do I know a visit really came from ClaudeBot?
Check the user agent and the source of the request. Impostors often copy well-known bot names. If a visit ignores your robots.txt rules, treat it with suspicion and check Anthropic's docs for verification guidance.