sort, uniq, cut, head/tail, wc, and Archives
Six small, single-purpose utilities that combine into real, useful pipelines — plus tar's real relationship to compression, a genuinely common point of confusion.
What you'll learn
- Combine sort, uniq, cut, head/tail, and wc into a real, useful pipeline
- Explain why uniq only removes ADJACENT duplicates, and why this makes sort-then-uniq a necessary pattern
- Explain tar's actual, separate relationship to compression
Prerequisites
Explanation
Every real command below runs only in your own terminal — this lesson's exercises model these utilities as JavaScript array/string logic and never execute anything.
sort orders lines (alphabetically by default; -n for numeric order, -r reversed). uniq removes duplicate lines — but with one precise, genuinely important limitation: uniq only removes adjacent duplicate lines, comparing each line only to the one immediately before it, not the whole file. This is exactly why sort file.txt | uniq is such a common, near-idiomatic pipeline: sorting first guarantees every duplicate line ends up adjacent to its other copies, which is the only situation uniq alone can actually deduplicate correctly — running uniq on unsorted input silently leaves non-adjacent duplicates untouched, a real, easy-to-miss mistake. uniq -c prefixes each line with a count of how many times it appeared consecutively, genuinely useful combined with sort for a quick "most common lines" report.
cut extracts a specific column or character range: cut -d, -f2 (delimiter comma, field 2) is a lighter-weight alternative to awk specifically for simple, single-field extraction from clearly-delimited data, without awk's broader field-processing capability. head/tail show the first/last N lines (-n 20); tail -f specifically follows a growing file continuously, printing new lines as they're appended — the standard way to watch a live log file update in real time. wc counts: wc -l (lines), wc -w (words), wc -c (bytes) — this is the exact tool behind grep pattern file | wc -l's line-counting from an earlier lesson.
tar (tape archive, a name reflecting genuinely old history) bundles multiple files into one archive file — and critically, on its own, tar does not compress anything at all; it only concatenates. tar -czf archive.tar.gz folder/ bundles and compresses in one command specifically because the -z flag tells tar to additionally pipe its output through gzip compression — -c creates, -z compresses (gzip), -f names the output file. This is a genuinely common point of confusion worth stating precisely: "tar" and "compression" are two separate, composable operations that happen to be combined in one command by convention, not one single feature.
Example
Modeling why sort-then-uniq is necessary (uniq only removes ADJACENT duplicates), and word-counting, as pure data operations.
function uniqAdjacent(lines) {
// Models the REAL uniq behavior: only removes a duplicate if it's immediately
// adjacent to its previous occurrence -- NOT a general "remove all duplicates."
const result = [];
for (const line of lines) {
if (result.length === 0 || result[result.length - 1] !== line) {
result.push(line);
}
}
return result;
}
const unsorted = ["b", "a", "b", "a"];
console.log(uniqAdjacent(unsorted)); // ["b", "a", "b", "a"] -- UNCHANGED! No adjacent duplicates existed.
const sorted = [...unsorted].sort();
console.log(sorted); // ["a", "a", "b", "b"] -- now duplicates ARE adjacent
console.log(uniqAdjacent(sorted)); // ["a", "b"] -- NOW uniq can actually deduplicate
function wordCount(text) {
return text.trim().split(/\s+/).filter(Boolean).length;
}
console.log(wordCount("the quick brown fox")); // 4Try it yourself
Call uniqAdjacent on an array that's ALREADY sorted, and confirm it correctly deduplicates without needing a separate sort step.
Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.
Guided exercise
Guided exercise
This models uniq's real, adjacent-only behavior -- no real tool is invoked. Write uniqAdjacent(lines) removing a line only when it's IMMEDIATELY the same as the previous line kept, exactly matching real uniq's limitation (not a general duplicate-removal across the whole array).
Checks: correctly removes adjacent duplicates · correctly leaves non-adjacent duplicates alone, matching real uniq's actual behavior
Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.
Stuck? Get a hint.
Independent exercise
Independent exercise
This models a sort-uniq-count pipeline only -- no real tools are invoked. Write mostCommonLines(lines) that returns an array of {line, count} objects (one per DISTINCT line, count = how many times it appeared anywhere in the array, not just adjacently), sorted by count descending -- modeling `sort | uniq -c | sort -rn`.
Checks: correctly ranks the most frequent line first · reports exactly one entry per distinct line · handles empty input
Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.
Stuck? Get a hint.
Common mistakes
- Running `uniq` directly on unsorted input, expecting it to remove all duplicates -- uniq only removes ADJACENT duplicates; non-adjacent repeated lines are silently left in place, which is exactly why `sort file | uniq` is the standard, necessary pattern.
- Assuming `tar -cf archive.tar folder/` produces a compressed archive -- plain tar only bundles files together; compression requires an explicit additional flag like -z (gzip) or -j (bzip2), or a separate compression step entirely.
- Reaching for awk when a simple `cut -d, -f2` would do the whole job more directly -- cut is the lighter, more direct tool specifically for extracting one clearly-delimited field, not requiring awk's broader capability.
Knowledge check
Takeaway
uniq only removes adjacent duplicates, which is exactly why sort-then-uniq is the standard pipeline for genuine deduplication; cut, head/tail, and wc are small, single-purpose tools that combine well into real pipelines; and tar's bundling and compression are two genuinely separate operations conventionally combined in one command, not one single feature.
Summary
sort orders lines; uniq removes only adjacent duplicates (sort first for full deduplication); uniq -c counts occurrences. cut extracts a delimited field simply; head/tail show the start/end of input (tail -f follows a growing file live); wc counts lines/words/bytes. tar bundles files; compression (-z for gzip) is a separate, additional, commonly-combined step.
References
Your notes
Notes save automatically.
Finished this lesson?
Mark it complete to track your progress and schedule a future review.
AI tutor
The optional AI tutor isn't enabled in this deployment. All lessons, exercises, quizzes, and search work fully without it.