Blog
Indexing, crawling & AI discoverability
73 guides on getting found by Google, Bing, and AI search.
AI discoverability
All AI discoverability →What is llms.txt and do you actually need one?
A practical guide to the llms.txt proposal: what it is, what it isn't, who's adopting it, and a copy-pasteable starter you can ship today.
Should you block GPTBot, ClaudeBot, and PerplexityBot? A 2026 decision matrix
Every site owner is making this call. Here's the actual tradeoff per bot: who's training, who's citing, who's sending traffic, and what each block costs you.
How to get cited by ChatGPT and Perplexity (and what counts as a citation)
Citation-driven traffic from AI assistants is a real channel now. Here's what the major LLMs actually look at when they decide whose URL to surface.
Google AI Overviews: how to get your site cited, and what changes about indexing
AI Overviews now sit above the blue links for a growing share of queries. Here's how Google picks the URLs it cites and what to change in your indexing playbook.
AI search is not killing SEO — it's adding a parallel game
Generative engine optimization and traditional SEO use mostly the same primitives. Here's where they diverge, and where one playbook works for both.
How to track AI bot traffic on your site (without a third-party pixel)
GPTBot, ClaudeBot, and PerplexityBot are crawling your site right now. Here's how to count them in your existing analytics — and why it matters.
Agencies
All Agencies →Debugging
All Debugging →Debug "Discovered, currently not indexed" — the actual root causes
The most frustrating Search Console status. Here's how to use the URL Inspection API to diagnose it, fix it, and get the page indexed.
Diagnose "Crawled, currently not indexed" without guessing
Google saw your page and decided to skip it. Here's the audit checklist for figuring out exactly why — and what to change.
Fix canonical issues, then resubmit: the right sequence for indexability bugs
Pushing a URL with a broken canonical tag accomplishes nothing. Fix first, then push — here's the order of operations.
Fix thin pages, then push: the content-first indexing workflow
Most indexability problems are content problems wearing a technical mask. Improve the page first, then use the Indexing API to get Google to re-evaluate.
How to force Google to recrawl a page after you update it
Updated a page and Google still shows the old title? Here's exactly how to nudge Google's crawler to re-fetch — and how to confirm it worked.
"Discovered, currently not indexed" in 2026: what changed and what fixes it now
Google's coverage state hasn't changed names but its meaning has shifted as the index gets more selective. Here's what works now to flip URLs out of the discovered-not-indexed limbo.
Why isn't your website showing up on Google? 12 reasons and the fix for each
Searched your own site and got nothing? It's almost always one of twelve specific, fixable causes — not a ban. Here's how to find yours, fastest diagnostic first.
How to check if a page is indexed by Google (4 reliable methods)
site: searches lie more often than people think. Here are four ways to check whether a page is actually indexed by Google — and which one to trust when they disagree.
Fix "Blocked by robots.txt" in Google Search Console
The "Blocked by robots.txt" status means Google can't crawl your URL at all. Here's how to find the offending rule, fix it, and get the page indexed.
Fix "Page with redirect" excluded status in Search Console
"Page with redirect" means Google indexed the destination, not the URL you submitted. Here's when that's fine, when it's a problem, and how to fix redirect chains.
Fix "Duplicate without user-selected canonical" in Search Console
Google found multiple near-identical URLs and picked the canonical for you — maybe not the one you wanted. Here's how to take back control and consolidate.
Fix "Excluded by 'noindex' tag" when you didn't mean to block the page
Google obeyed a noindex directive on your page — but you never intended to add one. Here's how to find the accidental noindex and get the page back in the index.
Fix "Soft 404" errors and get the page indexed
A soft 404 is a page that returns 200 OK but looks empty or error-like to Google. Here's how to tell why Google flagged it and how to make it index.
E-commerce
All E-commerce →Shopify: get new product pages indexed within hours, not weeks
Shopify stores live or die on product page visibility. Use the Indexing API to push new SKUs to Google the same day they go live.
WooCommerce: index seasonal collections before peak traffic hits
Black Friday, holiday gift guides, summer sales — get your seasonal collections indexed weeks before the traffic shows up.
Indexing headless commerce sites: Saleor, Medusa, Vendure, and friends
Headless commerce stacks decouple the storefront from the catalog API. That creates indexing gaps the legacy SEO plugins don't catch. Here's the modern workflow.
How to get e-commerce category pages indexed by Google
Category pages are an online store's biggest ranking opportunity and its messiest indexing problem. Here's how to get the right ones indexed and keep the junk out.
Frameworks
All Frameworks →Next.js: automate Indexing API submissions from your CI pipeline
Tie your deploy step to a script that pushes changed URLs through the Indexing API — so every release is automatically known to Google.
Documentation sites: keep Google's index in sync with your docs
Docusaurus, Mintlify, Fern — modern doc sites publish frequently. Here's how to make sure Google has the latest version of every page.
Astro, Hugo, and other static sites: post-build indexing automation
Static site generators are fast to build, fast to deploy — and have no built-in way to tell Google about new pages. Here's the missing piece.
How to get a JavaScript / SPA site indexed by Google
Single-page apps and JS-heavy sites have a hidden indexing tax: Google has to render before it can index. Here's how to make sure your content actually makes it into the index.
Publishers
All Publishers →How to get a new blog post indexed by Google the same day you publish
Step-by-step workflow to push a freshly published blog post through Google's Indexing API and confirm it's crawled within minutes — not weeks.
WordPress: bulk-submit your archive to Google after a redesign
Re-pointed permalinks, swapped themes, or migrated to a new domain? Here's how to push every WordPress URL through Google's Indexing API in one batch.
News publishers: get articles into Google within minutes of publish
News indexing is a speed race. The Indexing API + URL Inspection gives publishers a fast lane that doesn't depend on Google News partnership status.
Podcast episode pages: get show notes indexed alongside the audio
Your podcast episode pages are the long-tail SEO play. Push each new episode's page through the Indexing API as part of release day.
SaaS Product Hunt launch: get every landing page indexed before launch day
Launch day traffic is wasted if your launch landing page isn't indexed. Here's the pre-launch indexing checklist.
Substack: get your custom-domain newsletter pages indexed
Substack hosts your newsletter, but the SEO win is the indexed archive. Here's how to push every post to Google.
Ghost CMS: indexing tips for posts, tags, and authors
Ghost generates pages for every post, tag, and author. Here's the indexing playbook that gets the right ones in and the wrong ones out.
SEO workflows
All SEO workflows →Sitemap-based indexing triage: find every unindexed URL in 10 minutes
Use the Sitemap picker plus bulk URL Inspection to spot every URL on your site that isn't indexed yet — and decide which ones to push.
The pre-launch indexability audit: catch issues before Google does
Before you flip DNS to a new site, run the indexability audit on every key URL. Here's the checklist of signals that matter.
Run an indexability audit before you spend a dollar on backlinks
A $500 backlink to a noindexed page is $500 wasted. Audit the target URL before you buy.
Affiliate marketers: bulk-index large URL sets without burning your quota
Programmatic affiliate sites generate thousands of URLs. Here's the playbook for getting them indexed without wasting credits on dead pages.
Mass-recheck indexing status after a Google core update
Core updates shuffle the index. Use bulk URL Inspection to spot which of your pages got demoted, indexed for the first time, or quietly dropped.
Local SEO: get city and service pages indexed without duplicate-content penalties
Service-area businesses live and die on city-page indexing. Here's how to push them without tripping Google's duplicate-content classifier.
Multi-region sites: hreflang, locale URLs, and getting all variants indexed
International sites generate URL variants Google often fails to index correctly. Here's the workflow for pushing every locale.
Post-migration: bulk-check whether Google has followed your 301s
After a domain or URL-structure migration, bulk URL Inspection tells you exactly which old URLs Google has rolled over to the new ones.
Indexing API vs Search Console "Request indexing": when each one wins
Two ways to ask Google to crawl a URL. Here's what each one actually does, when to use it, and why batching matters.
IndexNow explained: the open indexing protocol Bing, Yandex, and AI search use
IndexNow is the open-protocol sibling of Google's Indexing API — supported by Bing, Yandex, Seznam, and Naver. Here's how it works and why it matters more in 2026.
Google's Indexing API compliance: the real story on JobPosting and BroadcastEvent
Google's docs say the Indexing API is only for JobPosting and BroadcastEvent. Everyone uses it for everything. Here's what's actually OK, what's risky, and what John Mueller has said publicly.
Indexing after a content migration: the 30-day playbook
Migrations are where rankings go to die. Here's a day-by-day plan for keeping every URL in the index when you change CMS, URLs, or both.
10 quick and easy ways to index your website on Google
A no-fluff checklist of the ten things that actually get a site crawled and indexed by Google — from sitemaps and Search Console to the Indexing API fast lane.
How long does Google take to index a new website? (and 7 ways to speed it up)
Realistic indexing timelines for a new site — from hours to weeks — what actually controls the speed, how to check your status, and seven ways to make it faster.
The simplest way to get your own pages indexed by Google
A plain-English tour of the Google indexing tool landscape — manual Search Console, the raw Indexing API, plugins, and the black-box paid services — and the easiest path if you own the pages you want indexed.
IndexerNow vs IndexMeNow: the cheaper way to index your own pages
IndexMeNow charges roughly $0.47–$0.98 per URL to index backlinks with off-Google methods. If you own the pages you want indexed, IndexerNow uses Google's official Indexing API at about $0.20 per URL — with a free daily tier on top.
IndexerNow vs Rapid URL Indexer: which indexing tool fits your job?
Rapid URL Indexer is a cheap, pay-per-success backlink indexer (~$0.05/URL). IndexerNow pushes your own pages through Google's official Indexing API. They solve different problems — here's how to choose without overpaying.
IndexerNow vs Omega Indexer: skip the $60-a-month minimum
Omega Indexer moved to subscriptions starting at $60/month, with credits that vanish if you don't use them. IndexerNow is free to start, then $9.99 for 50 credits you keep for 30 days — through Google's official API.
The best (and cheapest) Google indexing tools in 2026, compared
A no-spin comparison of how to get URLs indexed by Google in 2026 — Search Console, the official Indexing API, Rank Math, IndexMeNow, Rapid URL Indexer, Omega Indexer and SpeedyIndex — with real prices and the right pick for each situation.
How to remove or deindex a URL from Google (the right way)
Need a page gone from Google? Here's how to remove a URL fast, how to deindex it permanently, and the common mistakes that leave pages stuck in the index.
Crawl budget explained: when it actually matters (and when it doesn't)
Crawl budget is the most over-worried-about SEO concept for small sites and the most under-managed for big ones. Here's who needs to care and what to do about it.
How to get a brand-new domain indexed by Google fast
New domains start with zero trust and a slow crawl. Here's the launch-day checklist to get your first pages discovered and indexed in days, not weeks.
Google Indexing API limits and quota, explained
The Indexing API has a 200-requests-per-day cap and rules about what it's officially for. Here's exactly how the quota works and how to spend it wisely.
How to get backlinks indexed by Google (and why the official API can't help here)
An unindexed backlink passes no value. Here's how backlink indexing actually works, why Google's Indexing API doesn't apply, and the honest options for getting links crawled.
How to get your site indexed without a sitemap
No sitemap? You can still get indexed. Here's how Google discovers pages without one — and why you should probably add a sitemap anyway.
Mobile-first indexing: the 2026 checklist
Google indexes the mobile version of your site, full stop. Here's the checklist to make sure the mobile page Google sees has everything your rankings depend on.
How to get your images indexed by Google Images
Google Images is a major traffic source most sites ignore. Here's how image indexing works and the checklist to get your images discovered and ranked.
Is Google's Indexing API real? Read the receipt.
Skeptical that a paste-box can actually talk to Google? Here's the literal response Google returns for every URL you submit — and how to read it yourself.
Why your URLs say "crawl hint only" — and why you should still submit
That yellow banner about "not officially supported" content scares people off. Here's what it actually means and why pushing your pages anyway is still the right move.
Site builders
All Site builders →Webflow CMS: push new items to Google as soon as you hit publish
Webflow's auto-publish is great for layouts, but it doesn't notify Google. Here's how to wire Webflow CMS publishes into the Indexing API.
Framer marketing pages: get crawled and indexed the day you ship
Framer makes shipping landing pages effortless — but Google's crawl cadence doesn't care. Push every new page through the Indexing API to cut the delay.
Squarespace: get your pages indexed by Google faster
Squarespace handles the SEO basics, but it won't tell Google when you publish. Here's how to get Squarespace pages crawled and indexed without the usual wait.
Wix SEO: how to get your pages indexed by Google
Wix has shaken its old 'bad for SEO' reputation, but indexing still needs a nudge. Here's the practical checklist to get Wix pages crawled and indexed.
Get a Notion site indexed by Google (Notion, Super.so & friends)
Publishing a site from Notion is fast, but raw Notion pages are nearly invisible to Google. Here's how to make a Notion-based site actually index and rank.
Special content
All Special content →Index job postings on Google: the only officially supported content type
Job postings are one of two content types Google officially supports through the Indexing API. Here's the right way to push them.
Index live event pages (BroadcastEvent) with the Indexing API
Live streams, sports broadcasts, online conferences — the other officially supported Indexing API content type and how to use it.
How to get a PDF indexed by Google
Yes, Google indexes PDFs — they can rank like any page. Here's how PDF indexing works and how to make your documents discoverable.
How to get video pages indexed by Google (video SEO basics)
Google indexes the page your video lives on, not the raw file. Here's how video indexing works, what structured data you need, and how to get video pages crawled fast.