Blog
Indexing, crawling & AI discoverability
75 guides on getting found by Google, Bing, and AI search.
AI discoverability
All AI discoverability →What is llms.txt and do you actually need one?
A practical guide to the llms.txt proposal: what it is, what it isn't, who's adopting it, and a copy-pasteable starter you can ship today.
Should you block GPTBot, ClaudeBot, and PerplexityBot? A 2026 decision matrix
Every site owner is making this call. Here's the actual tradeoff per bot: who's training, who's citing, who's sending traffic, and what each block costs you.
How to get cited by ChatGPT and Perplexity (and what counts as a citation)
Citation-driven traffic from AI assistants is a real channel now. Here's what the major LLMs actually look at when they decide whose URL to surface.
Google AI Overviews: how to get your site cited, and what changes about indexing
AI Overviews now sit above the blue links for a growing share of queries. Here's how Google picks the URLs it cites and what to change in your indexing playbook.
AI search is not killing SEO — it's adding a parallel game
Generative engine optimization and traditional SEO use mostly the same primitives. Here's where they diverge, and where one playbook works for both.
How to track AI bot traffic on your site (without a third-party pixel)
GPTBot, ClaudeBot, and PerplexityBot are crawling your site right now. Here's how to count them in your existing analytics — and why it matters.
Why “Best X” listicles are the most-cited format in AI search
Ahrefs analysed over a billion data points across AI chatbots: “Best X” listicles are the single most prominent format cited by AI Overviews and ChatGPT. Here's how to write one that gets pulled into answers.
AI Overviews change every 2.15 days — what that means for your visibility
Google's AI Overviews rewrite themselves roughly every two days, but their semantic meaning barely moves. Here's why volatility ≠ instability, and how to track your AI visibility without panicking at every wording change.
YouTube and brand mentions beat backlinks for AI visibility: the data
Across Ahrefs' AI search studies, YouTube mentions had the highest correlation (0.737) with AI brand visibility — above backlinks, Domain Rating, and traffic. Here's how to build the off-site signals that get you cited.
Crawled but not cited: closing the gap between AI bots reading you and engines quoting you
AI bots fetch your pages thousands of times a month, yet ChatGPT and Perplexity still don't cite you. That's the crawl-to-cite gap — here's why it happens and the exact playbook to close it.
Agencies
All Agencies →Debugging
All Debugging →Debug "Discovered, currently not indexed" — the actual root causes
The most frustrating Search Console status. Here's how to use the URL Inspection API to diagnose it, fix it, and get the page indexed.
Diagnose "Crawled, currently not indexed" without guessing
Google saw your page and decided to skip it. Here's the audit checklist for figuring out exactly why — and what to change.
Fix canonical issues, then resubmit: the right sequence for indexability bugs
Pushing a URL with a broken canonical tag accomplishes nothing. Fix first, then push — here's the order of operations.
Fix thin pages, then push: the content-first indexing workflow
Most indexability problems are content problems wearing a technical mask. Improve the page first, then use the Indexing API to get Google to re-evaluate.
How to force Google to recrawl a page after you update it
Updated a page and Google still shows the old title? Here's exactly how to nudge Google's crawler to re-fetch — and how to confirm it worked.
"Discovered, currently not indexed" in 2026: what changed and what fixes it now
Google's coverage state hasn't changed names but its meaning has shifted as the index gets more selective. Here's what works now to flip URLs out of the discovered-not-indexed limbo.
Why isn't your website showing up on Google? 12 reasons and the fix for each
Searched your own site and got nothing? It's almost always one of twelve specific, fixable causes — not a ban. Here's how to find yours, fastest diagnostic first.
How to check if a page is indexed by Google (4 reliable methods)
site: searches lie more often than people think. Here are four ways to check whether a page is actually indexed by Google — and which one to trust when they disagree.
Fix "Page with redirect" excluded status in Search Console
"Page with redirect" means Google indexed the destination, not the URL you submitted. Here's when that's fine, when it's a problem, and how to fix redirect chains.
Fix "Soft 404" errors and get the page indexed
A soft 404 is a page that returns 200 OK but looks empty or error-like to Google. Here's how to tell why Google flagged it and how to make it index.
E-commerce
All E-commerce →Shopify: get new product pages indexed within hours, not weeks
Shopify stores live or die on product page visibility. Use the Indexing API to push new SKUs to Google the same day they go live.
WooCommerce: index seasonal collections before peak traffic hits
Black Friday, holiday gift guides, summer sales — get your seasonal collections indexed weeks before the traffic shows up.
Indexing headless commerce sites: Saleor, Medusa, Vendure, and friends
Headless commerce stacks decouple the storefront from the catalog API. That creates indexing gaps the legacy SEO plugins don't catch. Here's the modern workflow.
How to get e-commerce category pages indexed by Google
Category pages are an online store's biggest ranking opportunity and its messiest indexing problem. Here's how to get the right ones indexed and keep the junk out.
Frameworks
All Frameworks →Next.js: automate Indexing API submissions from your CI pipeline
Tie your deploy step to a script that pushes changed URLs through the Indexing API — so every release is automatically known to Google.
Documentation sites: keep Google's index in sync with your docs
Docusaurus, Mintlify, Fern — modern doc sites publish frequently. Here's how to make sure Google has the latest version of every page.
Astro, Hugo, and other static sites: post-build indexing automation
Static site generators are fast to build, fast to deploy — and have no built-in way to tell Google about new pages. Here's the missing piece.
How to get a JavaScript / SPA site indexed by Google
Single-page apps and JS-heavy sites have a hidden indexing tax: Google has to render before it can index. Here's how to make sure your content actually makes it into the index.
Publishers
All Publishers →How to get a new blog post indexed by Google the same day you publish
Step-by-step workflow to push a freshly published blog post through Google's Indexing API and confirm it's crawled within minutes — not weeks.
WordPress: bulk-submit your archive to Google after a redesign
Re-pointed permalinks, swapped themes, or migrated to a new domain? Here's how to push every WordPress URL through Google's Indexing API in one batch.
News publishers: get articles into Google within minutes of publish
News indexing is a speed race. The Indexing API + URL Inspection gives publishers a fast lane that doesn't depend on Google News partnership status.
Podcast episode pages: get show notes indexed alongside the audio
Your podcast episode pages are the long-tail SEO play. Push each new episode's page through the Indexing API as part of release day.
SaaS Product Hunt launch: get every landing page indexed before launch day
Launch day traffic is wasted if your launch landing page isn't indexed. Here's the pre-launch indexing checklist.
Substack: get your custom-domain newsletter pages indexed
Substack hosts your newsletter, but the SEO win is the indexed archive. Here's how to push every post to Google.
Ghost CMS: indexing tips for posts, tags, and authors
Ghost generates pages for every post, tag, and author. Here's the indexing playbook that gets the right ones in and the wrong ones out.
SEO workflows
All SEO workflows →Sitemap-based indexing triage: find every unindexed URL in 10 minutes
Use the Sitemap picker plus bulk URL Inspection to spot every URL on your site that isn't indexed yet — and decide which ones to push.
The pre-launch indexability audit: catch issues before Google does
Before you flip DNS to a new site, run the indexability audit on every key URL. Here's the checklist of signals that matter.
Run an indexability audit before you spend a dollar on backlinks
A $500 backlink to a noindexed page is $500 wasted. Audit the target URL before you buy.
Affiliate marketers: bulk-index large URL sets without burning your quota
Programmatic affiliate sites generate thousands of URLs. Here's the playbook for getting them indexed without wasting credits on dead pages.
Mass-recheck indexing status after a Google core update
Core updates shuffle the index. Use bulk URL Inspection to spot which of your pages got demoted, indexed for the first time, or quietly dropped.
Local SEO: get city and service pages indexed without duplicate-content penalties
Service-area businesses live and die on city-page indexing. Here's how to push them without tripping Google's duplicate-content classifier.
Multi-region sites: hreflang, locale URLs, and getting all variants indexed
International sites generate URL variants Google often fails to index correctly. Here's the workflow for pushing every locale.
Post-migration: bulk-check whether Google has followed your 301s
After a domain or URL-structure migration, bulk URL Inspection tells you exactly which old URLs Google has rolled over to the new ones.
Indexing API vs Search Console "Request indexing": when each one wins
Two ways to ask Google to crawl a URL. Here's what each one actually does, when to use it, and why batching matters.
Bing IndexNow explained: the instant URL indexing protocol
Bing IndexNow is the open protocol that instantly pushes new URLs to Bing, Yandex, Seznam, and Naver. How it works, the key file, and why it matters in 2026.
Indexing API content types: JobPosting & BroadcastEvent only
The Google Indexing API officially supports only 2 content types: JobPosting and BroadcastEvent. See exactly what's OK for other URLs, what's risky, and what Google said.
Indexing after a content migration: the 30-day playbook
Migrations are where rankings go to die. Here's a day-by-day plan for keeping every URL in the index when you change CMS, URLs, or both.
10 quick and easy ways to index your website on Google
A no-fluff checklist of the ten things that actually get a site crawled and indexed by Google — from sitemaps and Search Console to the Indexing API fast lane.
How long does Google take to index a new website?
Realistic indexing timelines for a new site — from hours to weeks — what actually controls the speed, how to check your status, and seven ways to make it faster.
The simplest way to get your own pages indexed by Google
A plain-English tour of the Google indexing tool landscape — manual Search Console, the raw Indexing API, plugins, and the black-box paid services — and the easiest path if you own the pages you want indexed.
IndexerNow vs IndexMeNow: $0.20 vs $0.47–$0.98/URL
IndexMeNow runs $0.47–$0.98/URL for backlinks via off-Google methods. For pages you own, IndexerNow uses Google's official API at ~$0.20/URL, plus a free tier.
IndexerNow vs Rapid URL Indexer: which indexing tool fits your job?
Rapid URL Indexer is a cheap, pay-per-success backlink indexer (~$0.05/URL). IndexerNow pushes your own pages through Google's official Indexing API. They solve different problems — here's how to choose without overpaying.
IndexerNow vs Omega Indexer: skip the $60-a-month minimum
Omega Indexer moved to subscriptions starting at $60/month, with credits that vanish if you don't use them. IndexerNow is free to start, then $9.99 for 50 credits you keep for 30 days — through Google's official API.
The best (and cheapest) Google indexing tools in 2026, compared
A no-spin comparison of how to get URLs indexed by Google in 2026 — Search Console, the official Indexing API, Rank Math, IndexMeNow, Rapid URL Indexer, Omega Indexer and SpeedyIndex — with real prices and the right pick for each situation.
Crawl budget explained: when it actually matters (and when it doesn't)
Crawl budget is the most over-worried-about SEO concept for small sites and the most under-managed for big ones. Here's who needs to care and what to do about it.
How to get a brand-new domain indexed by Google fast
New domains start with zero trust and a slow crawl. Here's the launch-day checklist to get your first pages discovered and indexed in days, not weeks.
Google Indexing API Quota Limits (2026): the 200/day cap explained
What counts against your 200/day Indexing API cap, the separate 2,000/day URL Inspection limit, 4 rules to avoid wasting quota, and a free calculator.
How to get backlinks indexed by Google (and why the official API can't help here)
An unindexed backlink passes no value. Here's how backlink indexing actually works, why Google's Indexing API doesn't apply, and the honest options for getting links crawled.
How to get your site indexed without a sitemap
No sitemap? You can still get indexed. Here's how Google discovers pages without one — and why you should probably add a sitemap anyway.
Mobile-first indexing: the 2026 checklist
Google indexes the mobile version of your site, full stop. Here's the checklist to make sure the mobile page Google sees has everything your rankings depend on.
How to get your images indexed by Google Images
Google Images is a major traffic source most sites ignore. Here's how image indexing works and the checklist to get your images discovered and ranked.
Is Google's Indexing API real? Read the receipt.
Skeptical that a paste-box can actually talk to Google? Here's the literal response Google returns for every URL you submit — and how to read it yourself.
Why your URLs say "crawl hint only" — and why you should still submit
That yellow banner about "not officially supported" content scares people off. Here's what it actually means and why pushing your pages anyway is still the right move.
Tag Parrot shut down: the best alternatives for auto-indexing in 2026
Tag Parrot, the popular auto-indexing tool, has gone dark. Here's what it did, why losing it hurts, and the alternatives that keep your new pages flowing into Google's Indexing API.
Migrating from Tag Parrot to IndexerNow: a step-by-step guide
A practical migration playbook for Tag Parrot users — reconnect Search Console, rebuild your auto-indexing schedule, and confirm everything is flowing again, in about ten minutes.
Site builders
All Site builders →Webflow CMS: push new items to Google as soon as you hit publish
Webflow's auto-publish is great for layouts, but it doesn't notify Google. Here's how to wire Webflow CMS publishes into the Indexing API.
Framer marketing pages: get crawled and indexed the day you ship
Framer makes shipping landing pages effortless — but Google's crawl cadence doesn't care. Push every new page through the Indexing API to cut the delay.
Squarespace: get your pages indexed by Google faster
Squarespace handles the SEO basics, but it won't tell Google when you publish. Here's how to get Squarespace pages crawled and indexed without the usual wait.
How to get Wix pages indexed on Google (2026 guide)
Wix has shaken its old 'bad for SEO' reputation, but indexing still needs a nudge. Here's the practical checklist to get Wix pages crawled and indexed.
Get a Notion site indexed by Google (Notion, Super.so & friends)
Publishing a site from Notion is fast, but raw Notion pages are nearly invisible to Google. Here's how to make a Notion-based site actually index and rank.
Special content
All Special content →Index job postings on Google: the only officially supported content type
Job postings are one of two content types Google officially supports through the Indexing API. Here's the right way to push them.
Index live event pages (BroadcastEvent) with the Indexing API
Live streams, sports broadcasts, online conferences — the other officially supported Indexing API content type and how to use it.
How to get a PDF indexed by Google
Yes, Google indexes PDFs — they can rank like any page. Here's how PDF indexing works and how to make your documents discoverable.
How to get video pages indexed by Google (video SEO basics)
Google indexes the page your video lives on, not the raw file. Here's how video indexing works, what structured data you need, and how to get video pages crawled fast.