Crawl budget explained: when it actually matters (and when it doesn't)

Crawl budget is the amount of crawling Google is willing to do on your site in a given window. It's shaped by two things: crawl rate limit (how fast Google can crawl without hurting your server) and crawl demand (how much Google wants to crawl your URLs). For most sites it's a non-issue. For a few, it's the whole ballgame.

If you have a small site, stop worrying

Under ~10,000 URLs, crawl budget is almost never your problem. Google can comfortably crawl a site that size. If your pages aren't getting indexed, the cause is quality, discoverability, or technical blockers — not crawl budget. Chasing crawl-budget optimizations on a 200-page blog is wasted effort.

Who actually needs to manage crawl budget

• Large e-commerce sites with tens of thousands of products plus faceted-navigation URL explosions.
• Programmatic SEO sites generating thousands of templated pages.
• News and listing sites with constant new URLs and high update frequency.
• Any site where Search Console's crawl stats show Google spending most of its budget on junk URLs.

How crawl budget gets wasted

• Faceted navigation: every filter combination is a crawlable URL, generating millions of near-duplicates.
• Infinite spaces: calendars, paginated archives with no end, session-ID parameters.
• Soft 404s and redirect chains: Google burns crawls fetching pages that lead nowhere.
• Low-value URLs in your sitemap diluting Google's attention away from the pages that matter.

What to actually do

1. Block crawl traps (filter parameters, internal search) in robots.txt so Google doesn't waste budget on them.
2. Keep your sitemap to canonical, indexable URLs only — it's a priority signal, not a dumping ground.
3. Fix redirect chains and soft 404s so every crawl lands on a real 200 page.
4. Use the Indexing API to point Google straight at your highest-value new and updated URLs, instead of hoping it finds them in a crowded crawl queue.