IndexerNow
Free scanTools…

User-Agent lookup

Paste a User-Agent string and we'll identify the crawler, search bot, SEO tool, social fetcher, or browser behind it.

GPTBot
Controlled by OpenAI
AI training crawler

Your content won't train future GPT models. ChatGPT browsing may still reach you via ChatGPT-User.

robots.txt: Publicly stated to honor robots.txt.
Official docs ↗

AI user agents list (21 crawlers)

Every AI crawler we track — the exact user-agent token it announces, who runs it, and what it collects. These tokens are also the User-agent names to use in robots.txt.

BotUA tokenOperatorPurposeDocs
GPTBotGPTBotOpenAIModel trainingdocs
ChatGPT-UserChatGPT-UserOpenAILive answersdocs
OAI-SearchBotOAI-SearchBotOpenAIAI search indexdocs
ClaudeBotClaudeBotAnthropicModel trainingdocs
Claude-WebClaude-WebAnthropicLive answersdocs
anthropic-aianthropic-aiAnthropicModel training—
PerplexityBotPerplexityBotPerplexityLive answersdocs
Google-ExtendedGoogle-ExtendedGoogleModel trainingdocs
GoogleOtherGoogleOtherGoogleResearchdocs
Applebot-ExtendedApplebot-ExtendedAppleModel trainingdocs
ApplebotApplebotAppleAI search indexdocs
BytespiderBytespiderByteDanceModel training—
AmazonbotAmazonbotAmazonLive answersdocs
Meta-ExternalAgentMeta-ExternalAgentMetaModel training—
Meta-ExternalFetcherMeta-ExternalFetcherMetaLive answers—
CCBotCCBotCommon CrawlModel trainingdocs
DiffbotDiffbotDiffbotResearch—
YouBotYouBotYou.comLive answers—
MistralAI-UserMistralAI-UserMistralLive answers—
cohere-aicohere-aiCohereModel training—
DuckAssistBotDuckAssistBotDuckDuckGoLive answers—

Want to know which of these your robots.txt actually lets in? Run your domain through the AI crawler access checker.

FAQ

What user agent does ChatGPT use?
OpenAI runs three: GPTBot crawls the web for model training; ChatGPT-User fetches a page live when a ChatGPT user asks about it; and OAI-SearchBot builds the index behind ChatGPT Search. They honor robots.txt separately, so you can allow live answers and search while opting out of training.
How do I identify an AI crawler in my server logs?
Grep your access logs for the user-agent tokens in the table above — GPTBot, ClaudeBot, PerplexityBot, and the rest all announce themselves in the User-Agent header. Paste any full log line into the lookup on this page to classify it. Be aware some scrapers spoof these names; serious verification means checking the request IP against the vendor's published ranges.
Which AI user agents should I allow or block?
Most sites that want AI visibility allow the search and live-answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-Web) since those produce citations and referrals, and make a separate call on the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) based on how they feel about model training. Check what your robots.txt currently allows with the AI crawler access checker.

Related free tools

AI Crawler Access Checker

Check whether ChatGPT, Claude, Perplexity, and other AI crawlers can crawl your site.

Robots.txt tester

Test whether a URL is crawlable under a robots.txt for a given User-Agent.

AI citation & brand mention checker

See whether AI models would cite your domain, and which off-site brand signals you're missing.