I’ve lost count of how many times uniq has saved me from writing an overly complicated script, usually paired right alongside sort in the same pipeline. It looks trivial on the surface, but there’s one detail about how it works that trips up almost everyone the first time they use it — and once you understand that detail, a whole category of “why isn’t this working” confusion disappears. Let me walk through the full picture.
What uniq Does
uniq filters out adjacent duplicate lines from its input, leaving only the first occurrence of each run of identical lines. That word “adjacent” is the single most important thing to understand about this command, and I’ll get into exactly why in a moment.
Basic Syntax
uniq [options] [input_file [output_file]]
uniq can also read from standard input, which is how it’s used in the overwhelming majority of real-world cases.
The Critical Detail: uniq Only Removes Adjacent Duplicates
Here’s a test that demonstrates exactly why this matters:
$ printf "apple\napple\nbanana\nbanana\nbanana\ncherry\napple\n" > fruits.txt
$ cat fruits.txt
apple
apple
banana
banana
banana
cherry
apple
Running uniq directly on this:
$ uniq fruits.txt
apple
banana
cherry
apple
Notice: apple still appears twice in the output — once at the top, once at the bottom — because those two occurrences aren’t adjacent to each other; there’s a run of banana and cherry sitting between them. uniq has no memory of lines it’s already seen elsewhere in the file; it only looks at whether the current line matches the immediately preceding one.
This is why uniq is almost always paired with sort first, since sorting brings all identical lines together into contiguous runs:
$ sort fruits.txt | uniq
apple
banana
cherry
Now every distinct value appears exactly once, because sorting guaranteed that all instances of each fruit ended up adjacent before uniq ever saw them.
Parameters and Options
| Option | Description |
|---|---|
-c, --count | Prefix each output line with the number of times it occurred consecutively |
-d, --repeated | Print only lines that appeared more than once (duplicates) |
-u, --unique | Print only lines that appeared exactly once (true singletons) |
-i, --ignore-case | Treat uppercase and lowercase as equivalent when comparing |
-f N, --skip-fields=N | Ignore the first N whitespace-separated fields when comparing |
-s N, --skip-chars=N | Ignore the first N characters when comparing |
-w N, --check-chars=N | Only compare the first N characters of each line |
-z, --zero-terminated | Use NUL instead of newline as the line delimiter |
I tested the counting and filtering options directly:
$ sort fruits.txt | uniq -c
2 apple
3 banana
1 cherry
$ uniq -d fruits.txt
apple
banana
$ uniq -u fruits.txt
cherry
apple
Notice something important in these last two results: I ran -d and -u on the unsorted file directly, and the results reflect adjacency, not global counts. -d found apple and banana because each had at least one adjacent repeat somewhere in the file, while -u found cherry and the final trailing apple because those specific occurrences had no adjacent duplicate right next to them — even though apple overall appears twice in the file. This is a perfect illustration of why sorting first is almost always the right move unless you specifically want adjacency-based (not global) duplicate detection.
How uniq Works Internally
uniq‘s algorithm is genuinely simple: read a line, compare it to the previous line kept, and either suppress it (if identical) or emit it and update the “previous line” reference (if different). This single-pass, constant-memory design is precisely why uniq can’t detect non-adjacent duplicates — doing so would require holding every previously seen line in memory (essentially building a hash set), which is a fundamentally different algorithm with different memory characteristics. uniq deliberately avoids that overhead, which is part of why it’s fast and safe to use even on very large files, as long as you’ve already sorted them.
The comparison behavior can be narrowed using -f, -s, and -w, which is useful when duplicate detection needs to ignore certain leading data — for instance, ignoring a timestamp prefix on log lines while still comparing the rest of the line content.
Real-World Use Cases
Removing duplicate lines from a sorted list of unique IP addresses, hostnames, or usernames:
sort access.log | awk '{print $1}' | sort | uniq
Finding which values appear more than once — a very common troubleshooting need:
sort user_ids.txt | uniq -d
I use this exact pattern constantly to detect accidental duplicate entries in configuration lists, CSV exports, or generated ID lists.
Counting occurrences of each distinct value, a classic frequency-analysis pattern:
sort access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10
This gives me the top 10 most frequent IP addresses hitting a web server — uniq -c produces the counts, and the second sort -rn orders them from most to least frequent.
Finding truly unique (singleton) values in a dataset — useful for spotting one-off anomalies:
sort transactions.csv | uniq -u
Case-insensitive deduplication of a list that has inconsistent capitalization:
sort -f names.txt | uniq -i
uniq vs sort -u vs awk ‘!seen[$0]++’
These three approaches often get compared, and each has a real use case:
- sort | uniq is the classic pairing, and it’s what I default to since it’s readable and composes naturally with additional pipeline stages like
uniq -c. - sort -u does sorting and deduplication in a single pass, slightly more efficient than piping to a separate
uniqwhen you only need the plain deduplicated, sorted result and don’t needuniq‘s counting or filtering options. - awk ‘!seen[$0]++’ deduplicates lines without sorting, preserving original input order — genuinely useful when order matters and you can’t afford to lose it by sorting first. This uses an associative array to track every line seen so far, trading
uniq‘s low memory footprint for the ability to detect duplicates regardless of position.
# Order-preserving deduplication, unlike sort | uniq
awk '!seen[$0]++' original_order.txt
I reach for this AWK one-liner specifically when I need to deduplicate a log or list but the original ordering genuinely matters — a case where sort | uniq would be the wrong tool entirely.
Troubleshooting Common Problems
“uniq isn’t removing all my duplicates” — nearly always means the input wasn’t sorted first; this is the single most common uniq mistake, and it’s exactly the scenario demonstrated above with the fruits.txt file. Add sort before uniq in the pipeline.
Case differences preventing expected matches — “Apple” and “apple” are treated as different lines by default; use -i if case shouldn’t matter for your comparison.
Leading whitespace or timestamp prefixes causing false “duplicates” to be missed — if lines differ only in a leading timestamp but are otherwise identical, plain uniq will treat them as distinct. Use -f to skip leading fields or -s to skip leading characters so the comparison ignores that varying prefix.
Unexpected results with -w — remember -w N only compares the first N characters; two lines can be reported as duplicates even if they differ significantly beyond that character count, which is exactly the intended (if occasionally surprising) behavior when you specifically want partial-line comparison.
Performance Optimization
uniq itself is extremely lightweight — a single linear pass with constant memory use regardless of input size, since it only ever needs to remember the immediately preceding line. The real performance cost in sort | uniq pipelines almost always comes from the sort stage, since sorting large files requires either enough memory to hold the data or disk-based external merge sort for very large datasets. If you’re processing genuinely huge files, sort‘s own performance flags (--parallel, -S for buffer size, -T for temp directory placement) matter far more than anything about uniq‘s own performance characteristics.
Security Implications
uniq has essentially no security surface of its own — it doesn’t execute code, doesn’t follow untrusted paths beyond explicit arguments, and processes input in a straightforward, predictable way. As with any text-processing tool used in scripts, standard shell hygiene (quoting variables, being careful with filenames containing unusual characters) is the relevant concern, rather than anything specific to uniq itself.
Compatibility Across Distributions
uniq is part of GNU coreutils, present with identical behavior across Debian, Ubuntu, RHEL, Fedora, Arch, and openSUSE. As with wc, macOS and BSD ship a different, non-GNU implementation of uniq that supports the core POSIX flags (-c, -d, -u, -i) consistently, but may lack some GNU-specific long-option names or -z NUL-delimited support. For maximum portability across Linux and macOS/BSD in shared scripts, sticking to the short POSIX-standard flags is the safer choice.
Using uniq With Field and Character Skipping in Practice
The -f and -s options genuinely earn their keep once you’re working with structured or semi-structured log data. Suppose each line begins with a timestamp, and you want to find duplicate log messages regardless of when they occurred:
$ cat events.log
2026-07-31 10:00:01 disk usage warning
2026-07-31 10:00:05 disk usage warning
2026-07-31 10:00:09 backup completed
Running plain uniq here won’t merge the two “disk usage warning” lines, since the timestamps differ and make each line technically unique. Skipping the first two whitespace-separated fields (the date and time) fixes this:
$ uniq -f 2 events.log
2026-07-31 10:00:01 disk usage warning
2026-07-31 10:00:09 backup completed
Now the comparison ignores the timestamp fields entirely and correctly merges the two structurally-identical warning lines, keeping only the first occurrence’s full line (timestamp included) in the output.
Combining uniq With cut or awk for Column-Based Deduplication
Sometimes you need uniqueness based on a specific column rather than the entire line. Since uniq itself doesn’t understand column semantics beyond simple field-skipping, I typically extract the column of interest first:
awk -F',' '{print $3}' sales.csv | sort | uniq -c | sort -rn
This extracts the third CSV column, sorts it, counts occurrences of each distinct value, and orders the results from most to least common — a pattern I use constantly for quick frequency analysis on any delimited data export without needing a full spreadsheet or database tool.
uniq’s Exit Status and Scripting Reliability
Like most well-behaved Unix utilities, uniq returns a zero exit status on success and non-zero on genuine errors (like an unreadable input file), which makes it safe to chain inside conditional logic:
if sort data.txt | uniq -d | grep -q .; then
echo "Duplicates found!" >&2
exit 1
fi
Here, uniq -d prints only duplicated lines, and grep -q . checks whether any output was produced at all — a clean way to turn “are there duplicates?” into a boolean check usable directly in a script’s control flow, without needing to count lines or parse output manually.
A Historical Note on uniq’s Design
uniq dates back to the earliest versions of Unix, originally written to complement sort in the classic pipeline-oriented philosophy the whole system was built around: small tools, each doing one narrow job well, composed together through pipes rather than any single tool trying to do everything itself. This is exactly why uniq never gained its own sorting logic even decades later — the Unix design philosophy consistently favored composing sort | uniq over building sorting directly into a deduplication tool, keeping each utility focused, predictable, and easy to reason about in isolation.
Summary
uniq does exactly one job, and it does it with an intentionally simple, memory-efficient algorithm — but that simplicity comes with the crucial caveat that it only ever compares a line to the one immediately before it. Once that clicks, the near-universal sort | uniq pattern stops being a rule you memorize and becomes something you understand: you’re deliberately grouping identical values together first, specifically so uniq‘s adjacency-based comparison actually finds every duplicate rather than just the ones that happened to already be next to each other.
References
- GNU Coreutils Manual, uniq section: https://www.gnu.org/software/coreutils/manual/html_node/uniq-invocation.html
man uniq- POSIX specification for uniq: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/uniq.html