Where duplicate lines come from
Merge two mailing lists and the people on both appear twice. Export a report every week and combine the files, and everything that did not change is repeated. Keyword research pulled from several tools, product codes gathered from different spreadsheets, log files, references pasted from multiple drafts: any workflow that stitches lists together breeds duplicates, and spotting them by eye in a few hundred lines is slow and unreliable. Duplicates are not just untidy: they mean double-sent emails, inflated counts and skewed reports.
This tool does the job in one paste. Every line is compared against the ones above it; the first time a line appears it stays, and every later repeat is deleted. The counters show exactly how many duplicates went and how many lines survive, updating live as you type or paste. Because the surviving lines keep their original order, the list you get back is your list, just without the repeats.
A worked example
The first apple (line 1) and the first banana (line 2) survive; the repeated apples on lines 3 and 6 and the repeated banana on line 5 are removed. Cherry only appears once, so it is untouched. Three kept plus three removed accounts for all six original lines, which is a quick sanity check worth doing on your own lists: kept plus removed always equals the line count you pasted.
Ignoring case
By default, capitalisation matters: Leeds and leeds are different strings, so both survive. Tick Ignore case and lines that differ only in capitalisation are treated as the same entry, with the first version keeping its place and its own capitalisation. This is usually what you want for names, email addresses and keywords, where LEEDS, Leeds and leeds are clearly the same thing typed by different people. Email addresses are a particularly good example: the standard treats the part after the @ as case-insensitive, and in practice the whole address almost always is, so a merged sign-up list should nearly always be deduplicated with the box ticked. Leave it unticked only when capitalisation genuinely distinguishes entries, such as case-sensitive codes or passwords.
When lines refuse to match
If two lines look identical but the tool keeps both, invisible characters are almost certainly the culprit. A trailing space, a stray tab or a non-breaking space copied from a web page makes "John Smith" and "John Smith " different strings, even though your eye cannot tell them apart. The fix is to clean the list first with the remove extra spaces tool, which trims line ends and normalises the invisible characters, and then dedupe it here. If you also want the finished list in order, the alphabetical order sorter takes the same one-item-per-line format. Everything runs in your browser, so a customer list never leaves your device.
Frequently asked questions
Which copy of a duplicate line is kept?
The first one. The tool reads your list from top to bottom, keeps a line the first time it appears and deletes every later repeat, so the surviving lines stay in their original order. If you want the list sorted afterwards, the alphabetical order sorter will do that.
What does the ignore case option do?
With the box ticked, lines that differ only in capitalisation count as the same line, so Apple, apple and APPLE become one entry. The first version to appear is the one that survives, keeping its own capitalisation. Untick the box to treat differently capitalised lines as different entries.
Why are two lines that look identical not being matched?
Almost always invisible characters. A trailing space, a tab, or a non-breaking space copied from a web page makes two lines different even though they look the same on screen. Run the list through the remove extra spaces tool first, then dedupe it.
Are blank lines treated as duplicates?
Yes. Blank lines are compared like any other line, so the first blank line is kept and the rest are removed. If you want no blank lines at all, tidy the list with the remove extra spaces tool and its remove blank lines option first.
Is my list uploaded anywhere?
No. The deduplication happens in your browser using JavaScript, so email lists, customer names and anything else sensitive never leave your device. Refreshing the page clears everything.