Free scanTools…

Diffbot user agent

Operated by Diffbot · Research · Other crawlers

robots.txt token
Diffbot

Diffbot identifies itself with the token Diffbot. That is the case-insensitive substring to grep for in access logs, and the exact name to put after User-agent: in robots.txt. It crawls for research and dataset work rather than a consumer product.

Allow or block Diffbot in robots.txt

Robots.txt applies the most specific matching group, so a group naming Diffbot overrides your User-agent: * rules for this bot alone.

Allow
User-agent: Diffbot
Allow: /
Block
User-agent: Diffbot
Disallow: /

What blocking costs you: Knowledge-graph tools that buy Diffbot data won't include your content.

To see which AI crawlers your robots.txt allows right now, run your domain through the AI crawler access checker. To check whether AI answers actually cite you, use the AI citation checker.

FAQ

What is the Diffbot user agent?
Diffbot announces itself with the User-Agent token "Diffbot". That token is the case-insensitive substring to match in server logs and the exact name to put after "User-agent:" in robots.txt. Diffbot does not publish one fixed full string, so match on the token rather than the whole line.
Who operates Diffbot and what does it collect?
Diffbot is run by Diffbot and crawls for research and dataset work rather than a consumer product. In the AI picture that puts it under "Other crawlers": Non-AI crawl infrastructure — excluded from your AI picture.
How do I block Diffbot?
Add a robots.txt group naming the token exactly: User-agent: Diffbot followed by Disallow: /. Robots.txt matches the most specific group, so a Diffbot group overrides whatever your User-agent: * group says. Knowledge-graph tools that buy Diffbot data won't include your content.
Should I block Diffbot?
That depends on what you lose. Knowledge-graph tools that buy Diffbot data won't include your content. Sites chasing AI citations usually keep the live-answer and AI-search bots open and make a separate decision about the training crawlers. Run your domain through the AI crawler access checker to see which ones your robots.txt lets in today.
Can Diffbot be spoofed?
Yes. A User-Agent header is self-reported, so any scraper can claim to be Diffbot. The token is fine for reporting and for robots.txt, but if you are rate-limiting or firewalling on it, verify the request IP against Diffbot's published ranges before you trust the name.

Paste a full User-Agent line and identify any crawler with the AI bot user agent list and lookup, or see every token side by side.