ToolsAbout

FASTA deduplicator

Drop repeated records from a multi-FASTA file and see exactly which ones went.

Result
What counts as a duplicate?
That depends on what went wrong upstream, so the tool offers three definitions:
  • By sequence. Two records are duplicates when their sequences are identical, even if the headers differ. This is the usual case: the same gene pulled from two databases, or a set of hits that overlap.
  • By header. Two records are duplicates when the header line is identical. Use it to find repeated identifiers, which break many downstream tools even when the sequences differ.
  • By both. A record is dropped only when its header and its sequence have both been seen together before, which is what happens when files are concatenated twice.
In every mode the first occurrence is kept, the rest are dropped, and the order of the surviving records is the order they had in the input.
Worked example
The example file holds five records: sample A, sample B, a second copy of sample A, sample B written on the other strand, and sample A again in lower case. By sequence, with ignore case on and the reverse complement option off, 3 of 5 records are kept. Dropped areseq3 and seq5, both duplicates of seq1, the second because upper and lower case are treated as the same letter.
Switch the reverse complement option on and seq4 goes too, leaving 2 records: its sequence is the reverse complement of seq2, so on double-stranded DNA it is the same molecule. Switch ignore case off and seq5 survives, because its sequence is written in lower case.
How the comparison works
Each record is reduced to a single key, and the first record with a given key wins:
key = sequence (upper-cased when ignore case is on)
With the reverse complement option on, the key of a DNA sequence is instead whichever of the sequence and its reverse complement sorts first, so a record and its reverse complement always produce the same key. Sequences containing anything other than A, C, G, T, U and N keep their own key, which leaves RNA with U, protein and ambiguous codes untouched by that option.
Frequently asked questions
How do I remove duplicate sequences from a FASTA file?
Paste the file and keep the default mode, by sequence. Records whose sequence has been seen before are dropped, whatever their header says, and the first record of each group is kept. The table under the result names every dropped record and the record it duplicates.
Which copy is kept?
The first one in the file, and the surviving records stay in their original order. That makes the result reproducible: sorting or re-running the tool cannot change which copy you get. If you want a particular copy kept, move it above the others before pasting.
Should reverse complements count as duplicates?
For double-stranded DNA, yes: a contig and its reverse complement are the same molecule, and assemblers and PCR product lists often contain both. The option is off by default because it is wrong for RNA, for protein and for single-stranded designs such as primers or oligo pools, where direction matters.
Why are two records with the same header not removed by default?
Because the default compares sequences, not names. Two records can share a header and hold different sequences, for example after a badly merged assembly. Switch to the by header tab to find repeated identifiers, or to by both to drop only records that repeat the header and the sequence together.
Does the comparison ignore line breaks and formatting?
Yes. Each record is read into one continuous string before comparison, so line width, spaces and position numbers make no difference. Gap characters do count as characters, so an aligned sequence and the same sequence without gaps are not duplicates.