AI Crawler Access Checker
Paste your domain — we read its robots.txt and show whether ChatGPT, Claude, Perplexity, and every other AI crawler are allowed or blocked, and what blocking each one costs.
Frequently asked questions
Can ChatGPT crawl my site?
Only if your robots.txt allows OpenAI's crawlers. GPTBot collects training data, ChatGPT-User fetches pages when a user asks about them, and OAI-SearchBot powers ChatGPT Search. This checker shows whether each is allowed or blocked on your domain.
What user agent does GPTBot use?
OpenAI's training crawler identifies itself as GPTBot. To control it, add a User-agent: GPTBot block to your robots.txt. ChatGPT browsing uses a separate agent, ChatGPT-User, and search uses OAI-SearchBot.
How do I block AI crawlers from my site?
Add a User-agent block per bot in robots.txt with Disallow: / — e.g. for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and Applebot-Extended. This tool shows which AI bots your robots.txt currently allows or blocks.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended only governs whether your content trains Gemini and feeds AI Overviews. Googlebot crawls for Search independently, so blocking Google-Extended does not affect your normal Search ranking.
If AI crawlers are allowed, are they actually crawling me?
Not necessarily. robots.txt grants permission — it doesn't prove a bot fetched your pages. To confirm real visits, track AI-crawler hits with the IndexerNow AI-crawler dashboard: drop a lightweight pixel in your <head> and watch which bots actually land, and on which pages.