Free scanTools…

ClaudeBot user agent

Operated by Anthropic · Model training · Learning from you

robots.txt token
ClaudeBot
Full User-Agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com

ClaudeBot identifies itself with the token ClaudeBot. That is the case-insensitive substring to grep for in access logs, and the exact name to put after User-agent: in robots.txt. It collects pages as training data for a model.

Allow or block ClaudeBot in robots.txt

Robots.txt applies the most specific matching group, so a group naming ClaudeBot overrides your User-agent: * rules for this bot alone.

Allow
User-agent: ClaudeBot
Allow: /
Block
User-agent: ClaudeBot
Disallow: /

What blocking costs you: Your content is excluded from Claude training data.

To see which AI crawlers your robots.txt allows right now, run your domain through the AI crawler access checker. To check whether AI answers actually cite you, use the AI citation checker.

Anthropic's other crawlers

Blocking one Anthropic bot does not block the others — each token is a separate group.

Anthropic's own documentation: https://support.anthropic.com/en/articles/8896518

FAQ

What is the ClaudeBot user agent?
ClaudeBot announces itself with the User-Agent token "ClaudeBot". That token is the case-insensitive substring to match in server logs and the exact name to put after "User-agent:" in robots.txt. The full line Anthropic publishes is: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com
Who operates ClaudeBot and what does it collect?
ClaudeBot is run by Anthropic and collects pages as training data for a model. In the AI picture that puts it under "Learning from you": Feeding the models that power future answers.
How do I block ClaudeBot?
Add a robots.txt group naming the token exactly: User-agent: ClaudeBot followed by Disallow: /. Robots.txt matches the most specific group, so a ClaudeBot group overrides whatever your User-agent: * group says. Your content is excluded from Claude training data.
Should I block ClaudeBot?
That depends on what you lose. Your content is excluded from Claude training data. Sites chasing AI citations usually keep the live-answer and AI-search bots open and make a separate decision about the training crawlers. Run your domain through the AI crawler access checker to see which ones your robots.txt lets in today.
Can ClaudeBot be spoofed?
Yes. A User-Agent header is self-reported, so any scraper can claim to be ClaudeBot. The token is fine for reporting and for robots.txt, but if you are rate-limiting or firewalling on it, verify the request IP against Anthropic's published ranges before you trust the name.

Paste a full User-Agent line and identify any crawler with the AI bot user agent list and lookup, or see every token side by side.