Fix "Blocked by robots.txt" in Google Search Console

"Blocked by robots.txt" is one of the clearer Search Console statuses: Google found your URL, tried to crawl it, and your own robots.txt told it not to. The page can't be indexed with fresh content while that block stands. The good news is it's almost always a one-line fix.

Step 1: confirm which rule is blocking it

Open your robots.txt (yourdomain.com/robots.txt) and look for a Disallow line that matches the path. Remember that robots.txt is matched by prefix: Disallow: /blog blocks /blog, /blog/, and everything under it. A stray Disallow: / blocks your entire site — a classic staging-config-leaked-to-production disaster.

Step 2: decide if the block is intentional

• Intentional: admin areas, cart/checkout, internal search results, faceted-navigation traps, staging subdomains. Leave these blocked.
• Accidental: your blog, product pages, or whole site blocked by a leftover rule, a CMS default, or a migrated staging robots.txt. Fix these.

Step 3: remove or narrow the rule

Delete the offending Disallow line, or narrow it so it only matches what you actually want blocked. If you need to block a section but still let Google read one URL inside it, add a more specific Allow line (Allow lines win over Disallow lines for longer-matching paths in Google's implementation).

Step 4: re-crawl and confirm

1. Deploy the updated robots.txt and load it in a browser to confirm it's live.
2. Re-run the URL through a status check — the block should clear once Google re-fetches robots.txt (usually within a day).
3. Push the URL through the Indexing API to prompt a fresh crawl now that it's allowed.