URL List Cleaner

When a URL list needs cleaning

Exports from crawlers, analytics, log files and backlink tools contain the same page many times: with and without a trailing slash, with UTM parameters from newsletters, with #fragments from in-page links, in http and https. Before you use such a list for a sitemap, a redirect map or an indexing check, reduce it to one line per real page.

What each option does

• Remove duplicates compares URLs after all other rules have been applied, so http://YourSite.com/a#top and https://yoursite.com/a collapse into one line when the matching options are on.
• Query strings: keep them, remove only tracking parameters, or cut everything after the ?. Removing all parameters is wrong for sites where a parameter selects the content, such as ?id=123 or ?page=2.
• Trailing slash: "Add" skips URLs that end in a file name like /file.pdf. Use the version your server answers with 200, not the one it redirects.
• Filters: keep only a section (/blog/) or drop patterns such as /tag/. With the regex switch on you can write expressions like \.(jpg|png|webp)$.
• URLs per host shows at a glance whether a list mixes www and non-www, subdomains or foreign domains.

Typical workflows

Build a sitemap from a crawl: remove tracking parameters and fragments, dedupe, filter out /cart and /search, then paste the result into the sitemap generator.

Prepare a migration: clean the old URL list, check what the URLs return today with the bulk HTTP status checker, and create the rules with the redirect rule generator.

Check indexing: reduce the list to canonical URLs first, so you do not spend time on duplicates. TurboIndex then tracks the index status of each URL for you.