How to remove duplicate lines with awk whilst keeping all empty lines?

Question

Below awk command removes all duplicate lines as explained here:

awk '!seen[$0]++'

If the text contains empty lines, all but one empty line will be deleted.

How can I keep all empty lines whilst deleting all non-empty duplicate lines, using only awk? Please, also include a brief explanation.

Answer 1

Another option is to check NF, eg:

awk '!NF || !seen[$0]++'

Or equivalently:

awk '!(NF && seen[$0]++)'

Answer 2

Alternatively

awk '!/./ || !seen[$0]++' file

The main trick is the same, seen[$0]++ creates an entry in the seen associative array whose key is the current line ($0). Therefore, !seen[$0]++ will be false if this line has already been seen. The /./ is checking whether the line contains any non-blank characters, so !/./ matches non blank lines. Combined with || !seen[$0]++ it will ignore all duplicate lines except blank ones and print the rest.

Answer 3

Here is another awk solution, similar to @Thor's answer, less concise but more efficient:

awk '!NF {print;next}; !($0 in a) {a[$0];print}' file

With this, we only check a[$0] has existed or not. If not, initializing it then print. In this case, we don't have any reference, assignment to a[$0] if it existed.

Answer 4

awk '/^[[:blank:]]*$/ { print; next; }; !seen[$0]++'

All you have to do is check for an empty (really empty or just blank) line first.

Answer 5

Logical operator NOТ move outside of parentheses:

awk '!(NF && seen[$0]++)'

Answer 6

awk '/^$/ || !seen[$0]++' filename

Answer 7

This answer is similar to cuognim's answer, but is much shorter, and easier to read (at least for me). All credit goes to AdminBee:

awk '!($0 in a) {if (NF) a[$0]; print}' filename

By ensuring the $0 is only stored if the line is non-empty, ($0 in a) will always be false for empty (blank) lines.