Free scanTools…

URL list cleaner

Paste a messy list of URLs. We dedupe, normalize, strip tracking parameters, and validate them into a clean list ready to submit. Everything runs in your browser.

3 clean2 duplicates removed1 invalid6 input lines

Skipped — not valid URLs

  • not a url
The short answer

To clean a list of URLs, paste it above one per line. Duplicates are removed, tracking parameters (utm_*, gclid, fbclid, msclkid) are stripped, the hostname is lowercased, fragments are dropped, and anything that is not a valid URL is flagged rather than passed through. There is no row limit and no sign-in, and the list never leaves your browser.

What the cleaner does

Removes duplicates

Exact repeats go, and so do the ones a plain de-dupe misses: the same URL with its query parameters in a different order, or separated only by a #fragment, or by the campaign parameters that were just stripped off both.

Strips tracking parameters

utm_source, utm_medium, utm_campaign, utm_term, utm_content, plus gclid, fbclid, msclkid and the other click identifiers. What is left is the canonical address of the page rather than the address of one campaign.

Normalizes what it keeps

Whitespace trimmed, the hostname lowercased, fragments dropped, and a bare domain given an https:// scheme. Two lines that pointed at the same page end up identical, which is what makes the de-dupe correct rather than approximate.

Flags what is not a URL

Blank lines, stray quotes, text pasted out of a spreadsheet cell and anything that will not parse are separated out instead of silently passing through into whatever you submit next.

Clean the list before you spend quota on it

Every indexing submission is metered. Google's Indexing API counts each URL against your Cloud project's daily publishing quota, and IndexNow submissions are rate-limited per site. A list exported from a crawler, an analytics report or a client spreadsheet almost always carries the same page several times over under different campaign parameters, so a de-dupe first is the cheapest quota you will ever save.

Once the list is clean: check which URLs Google has already indexed so you only submit the ones that need it, confirm the rest are actually indexable, then submit the batch. To work out how far a day's quota goes, the quota calculator does the arithmetic.

FAQ

How do I clean a list of URLs?
Paste the list into the box above, one URL per line. The cleaner removes duplicates, strips tracking parameters, lowercases the hostname, drops fragments and separates out anything that is not a valid URL. Two URLs that differ only in the order of their query parameters count as one. Copy the cleaned list back out. There is no file upload and no sign-in.
Does this remove UTM parameters?
Yes. utm_source, utm_medium, utm_campaign, utm_term and utm_content are stripped, along with the common click identifiers (gclid, fbclid, msclkid). Parameters your pages actually need, such as a product ID or a pagination parameter, are kept — only known tracking parameters are removed.
How many URLs can I paste?
There is no row limit. The whole thing runs in your browser on your own machine, so the practical ceiling is your device rather than a quota we set. Nothing is uploaded to a server.
Are my URLs sent anywhere?
No. The cleaner is client-side JavaScript: the list never leaves the page. That matters if you are cleaning a client's URLs or a list pulled from an internal crawl.
Why clean a URL list before submitting it for indexing?
Because every duplicate costs a submission. Google's Indexing API and IndexNow both meter requests, and a list exported from a crawler or a spreadsheet routinely carries the same page three or four times over under different tracking parameters. De-duping first is the difference between spending a day's quota on your site and spending it on utm_campaign variants of one page.
What is the difference between a clean URL and a canonical URL?
A clean URL is one with the noise removed — no tracking parameters, no fragment, a consistent hostname. A canonical URL is the address a page itself declares as its real one, in a link rel="canonical" tag. Cleaning gets you close to the canonical in most cases, but only the page can tell you for certain. The canonical tag finder reads the declared one.

Related free tools

Sitemap diff

Compare two sitemaps or URL lists: added, removed, unchanged.

Google index checker

Bulk-check which of your URLs Google has indexed, via the URL Inspection API on your own Search Console.

Credit calculator

Work out free pushes, credit packs, and total cost for a set of URLs.

IndexNow checker

Verify your IndexNow key file is hosted correctly for Bing, Yandex, Seznam, and Naver.