Remove Duplicate Lines

Clean up a list

Paste a list — email addresses, keywords, URLs, product codes, log lines — and the repeats disappear. The counts at the top tell you how many lines went in, how many were removed and how many are left, so you can see the damage before you copy the result.

Everything runs in your browser. Nothing is uploaded, stored or logged, which matters here more than on most tools: the lists people deduplicate are usually customer emails, subscriber exports or internal data.

The only real question: what counts as the same line?

Every option in this tool answers that one question, and none of them touch the text itself.

Ignore capitalization treats Apple and apple as one line.

Ignore spaces at the start and end treats apple and   apple   as one line. This catches the single most common source of invisible duplicates: a copy-paste from a table or a PDF that brought trailing spaces with it.

Both change matching only. The line that survives comes out exactly as it went in — with its own capitals, its own indentation, its own trailing space. That is deliberate: ticking a box to say two lines are equivalent is not the same as asking to have your data rewritten, and indented text would be destroyed with no way to get it back.

Which copy survives

Keep the first copy is the default and the one you want almost always. Repeats are dropped and everything else keeps its original position.

Keep the last copy keeps the final occurrence instead — and keeps it where it was, at the bottom. Reach for this when later entries supersede earlier ones: an export where the most recent row is the current state, or a log where the last line for an ID is the one that matters.

Remove every copy does something different, and it is the option people miss. Rather than collapsing repeats to one, it deletes every line that appears more than once, leaving only the entries that occur exactly once. That answers “what is in this list only a single time?” — which is how you find the odd one out in a pair of exports, or the records that failed to import twice.

On the command line, the three correspond to awk '!seen[$0]++', tac | awk '!seen[$0]++' | tac, and sort | uniq -u.

Blank lines

Left alone, a blank line is just another line: repeated blanks collapse to one, and a single blank survives wherever it is. That keeps paragraph breaks intact when you deduplicate prose.

Tick Remove blank lines to strip them all. Lines containing only spaces or tabs count as blank, which they usually should — they are invisible on screen and are almost always leftovers from an export.

Sorting

Sorting is optional and runs after the duplicates are gone. Leave it off to preserve the order you pasted, which is what you want for anything where sequence carries meaning.

When you do sort, numbers are ordered the way you read them. item2 comes before item10. Plain alphabetical ordering compares one character at a time and puts item10 first, because “1” sorts before “2” — the behavior that makes sorted filenames come out in the wrong order in every file manager that has not been fixed.

Sorting also respects the matching options. With “Ignore spaces at the start and end” on, an indented line sorts by its text rather than by its indentation, so lines you have just declared equivalent do not end up in different places.

What people use this for

Mailing lists. Merging two exports produces duplicates immediately, and sending twice to the same address is the fastest way to a spam complaint. Turn on “Ignore capitalization” — the local part of an email address is technically case-sensitive, but in practice no mail provider treats it that way, and the same person often appears twice with different capitals.

Keyword research. Exports from different tools overlap heavily. Deduplicate, then sort, and the list is ready to group.

Log files. “Remove every copy” is the quick way to find the events that happened only once in a run.

Spreadsheet columns. Copy a column, paste it here, clean it, paste it back. Faster than the Remove Duplicates dialog when you also want to sort or drop blanks in the same pass, and you can see the result before committing to it.

Reconciling two lists. Paste both one after the other and choose “Remove every copy”: what is left appeared in exactly one of them. To find what they share instead, deduplicate each list separately first, then combine them and keep only the lines that repeat — the text diff tool is usually a better fit for that job.

What it does not do

It compares whole lines, not parts of lines. Two rows of a CSV that differ only in a timestamp are different lines, and both survive. If you need to deduplicate on one field, extract that column first.

It also does not find similar lines — a typo, a different spelling, an extra middle initial. Those are genuinely different lines, and deciding whether they mean the same thing needs judgement rather than a rule.

Nothing leaves your browser

The list is processed on your own device. It is never uploaded, never stored, never logged, and is deliberately never written into the page address, so it cannot end up in your browser history or in the referrer of a link you click afterwards.

Closing or reloading the tab clears it. For a list of customer emails or internal records, that is the property that matters.

Frequently asked questions

Does it change the lines it keeps?

Never. Every option here changes what counts as a match — capitalization, spaces at the ends — and none of them rewrite the text. A line that survives comes out exactly as it went in, indentation and capitals intact.

Which copy of a repeated line is kept?

The first, by default, and the rest of the list stays in its original order. Choose "Keep the last copy" to keep the final occurrence instead — useful when later entries in a log or export are the more recent ones.

How do I find the lines that appear only once?

Choose "Remove every copy". Instead of collapsing repeats down to one, it drops every line that appears more than once, leaving only the genuinely unique entries. That is the equivalent of uniq -u on the command line.

Why did "Apple" and "apple" both survive?

Because they are different lines. Tick "Ignore capitalization" to treat them as the same, and the first one you pasted is the one that stays — with its original capitals.

Does it remove blank lines?

Only if you ask. Left alone, blank lines are treated like any other line, so repeated blanks collapse to one. Tick "Remove blank lines" to strip them all, including lines that contain nothing but spaces or tabs.

How does the sorting handle numbers?

The way a person reads them. "item2" comes before "item10", not after — plain alphabetical ordering compares character by character and puts every number starting with 1 ahead of every number starting with 2.

How long a list can I paste?

Tens of thousands of lines are no problem; everything runs in your browser, so the limit is your device rather than a server. Nothing is uploaded, stored or logged.