Free scanTools…

AI bot user agent list & lookup

Paste a User-Agent string and we'll identify the crawler, search bot, SEO tool, social fetcher, or browser behind it.

GPTBot
Controlled by OpenAI
AI training crawler

Your content won't train future GPT models. ChatGPT browsing may still reach you via ChatGPT-User.

robots.txt: Publicly stated to honor robots.txt.
Official docs ↗

AI user agents list (21 crawlers)

Every AI crawler we track: the exact user-agent token it announces, who runs it, and what it collects. These tokens are also the User-agent names to use in robots.txt. Each bot links to its own page with the full published User-Agent string and the robots.txt lines to allow or block it.

BotUA tokenOperatorPurposeDocs
GPTBotGPTBotOpenAIModel trainingdocs
ChatGPT-UserChatGPT-UserOpenAILive answersdocs
OAI-SearchBotOAI-SearchBotOpenAIAI search indexdocs
ClaudeBotClaudeBotAnthropicModel trainingdocs
Claude-WebClaude-WebAnthropicLive answersdocs
anthropic-aianthropic-aiAnthropicModel training—
PerplexityBotPerplexityBotPerplexityLive answersdocs
Google-ExtendedGoogle-ExtendedGoogleModel trainingdocs
GoogleOtherGoogleOtherGoogleResearchdocs
Applebot-ExtendedApplebot-ExtendedAppleModel trainingdocs
ApplebotApplebotAppleAI search indexdocs
BytespiderBytespiderByteDanceModel training—
AmazonbotAmazonbotAmazonLive answersdocs
Meta-ExternalAgentMeta-ExternalAgentMetaModel training—
Meta-ExternalFetcherMeta-ExternalFetcherMetaLive answers—
CCBotCCBotCommon CrawlModel trainingdocs
DiffbotDiffbotDiffbotResearch—
YouBotYouBotYou.comLive answers—
MistralAI-UserMistralAI-UserMistralLive answers—
cohere-aicohere-aiCohereModel training—
DuckAssistBotDuckAssistBotDuckDuckGoLive answers—

Want to know which of these your robots.txt actually lets in? Run your domain through the AI crawler access checker.

FAQ

What is the ChatGPT user agent string?
OpenAI's crawlers identify themselves with three User-Agent tokens: GPTBot (model training), ChatGPT-User (a live fetch when a ChatGPT user asks about your page) and OAI-SearchBot (the index behind ChatGPT Search). Each token appears verbatim in the User-Agent header, so those three are what to grep for in your logs and what to name in robots.txt. They honor robots.txt separately, so you can allow live answers and search while opting out of training.
Where can I get a full bot user agent list?
The table on this page lists every AI crawler we track — 21 at the moment — with its User-Agent token, who operates it and what it collects, each linking to a page with that bot's full published User-Agent string. The lookup at the top classifies any full User-Agent line you paste, AI bot or not: search crawlers, SEO tools, social fetchers and browsers.
How do I identify an AI crawler in my server logs?
Grep your access logs for the user-agent tokens in the table above. GPTBot, ClaudeBot, PerplexityBot and the rest all announce themselves in the User-Agent header. Paste any full log line into the lookup on this page to classify it. Be aware some scrapers spoof these names; serious verification means checking the request IP against the vendor's published ranges.
Which AI user agents should I allow or block?
Most sites that want AI visibility allow the search and live-answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-Web) since those produce citations and referrals, and make a separate call on the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) based on how they feel about model training. Check what your robots.txt currently allows with the AI crawler access checker.

Related free tools

AI Crawler Access Checker

Check whether ChatGPT, Claude, Perplexity, and other AI crawlers can crawl your site.

Robots.txt tester

Test whether a URL is crawlable under a robots.txt for a given User-Agent.

AI citation & brand mention checker

See whether AI models would cite your domain, and which off-site brand signals you're missing.