VIM ---- Removing duplicate lines

STEP 1:

sort all lines 

:sort u

STEP 2:

remove duplicates , the first leaves only the last line, the second leaves only the first line.

g/^\(.*\)$\n\1$/d
g/\%(^\1$\n\)\@<=\(.*\)$/d

Breakdown of the second version:

g//d <-- Delete the lines matching the regexp
\@<= <-- If the bit following matches, make sure the bit preceding this symbol directly precedes the match
\(.*\)$ <-- Match the line into subst register 1
\%( ) <--- Group without placing in a subst register.
^\1$\n <--- Match subst register 1 followed by end of line and the new line between the 2 lines