Robots.txt tester

Paste a robots.txt, a URL, and a User-Agent — we report which rule wins and whether the URL is crawlable. Everything runs in your browser.

URL to testUser-agent

Allowed

/admin/public/whitepaper.pdf

Matched Allow: / in the [googlebot] group.

Sitemaps declared

robots.txt contentSupports user-agent groups, Allow/Disallow, * wildcards and $ anchors. Longest match wins; Allow beats Disallow on ties.

Related free tools

Check whether a live URL can be indexed by Google — status, X-Robots-Tag, meta robots, canonical, robots.txt.

Identify AI crawlers, bots, and browsers behind a User-Agent string.

Check whether ChatGPT, Claude, Perplexity, and other AI crawlers can crawl your site.

Verify your IndexNow key file is hosted correctly for Bing, Yandex, Seznam, and Naver.