AI bot auditor
Enter a domain — we read its robots.txt and show which AI crawlers are allowed or blocked, and what blocking each costs.
Frequently asked questions
What user agent does GPTBot use?
OpenAI's training crawler identifies itself as GPTBot. To control it, add a User-agent: GPTBot block to your robots.txt. ChatGPT browsing uses a separate agent, ChatGPT-User, and search uses OAI-SearchBot.
How do I block AI crawlers from my site?
Add a User-agent block per bot in robots.txt with Disallow: / — e.g. for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended. This tool shows which AI bots your robots.txt currently allows or blocks.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended only governs whether your content trains Gemini and feeds AI Overviews. Googlebot crawls for Search independently, so blocking Google-Extended does not affect your normal Search ranking.
Why allow AI crawlers at all?
Allowing AI crawlers lets your content appear in ChatGPT, Claude, Perplexity, and Google AI Overviews answers — an emerging discovery channel. Blocking them opts you out of that visibility.