How to Pipe Commands in Bash

If there’s one feature that captures the spirit of the Unix philosophy — small tools that do one thing well, combined to do something powerful — it’s the pipe. The first time I chained three or four commands together with pipes and watched a messy log file turn into a clean, sorted summary in one line, I understood why so many people describe the shell as addictive once it clicks. In this article, I’ll cover how piping works, common patterns, and how to build genuinely useful one-liners and scripts around it.

What Is a Pipe?

A pipe, written as |, takes the standard output of one command and feeds it directly into the standard input of the next command.

command1 | command2

Here, whatever command1 prints to the screen instead gets sent directly to command2 as its input.

A Simple Example

ls -l | grep ".txt"

This lists files in the current directory (ls -l) and pipes that output into grep, which filters the lines to only those containing “.txt”.

Chaining Multiple Pipes

You’re not limited to a single pipe — you can chain as many as you need:

cat access.log | grep "ERROR" | sort | uniq -c | sort -nr

Let’s break this down:

1. cat access.log — outputs the full contents of the log file.
2. grep "ERROR" — filters to only lines containing “ERROR”.
3. sort — sorts those lines alphabetically (important for the next step).
4. uniq -c — collapses duplicate lines and counts how many times each appeared.
5. sort -nr — sorts numerically in reverse order, so the most frequent errors appear first.

This kind of pipeline is one of the most common patterns in day-to-day command-line work — take raw text, filter it, transform it, and summarize it, all in one line.

Common Commands Used in Pipelines

• grep — filters lines matching a pattern
• sort — sorts lines
• uniq — removes or counts duplicate lines
• wc — counts lines, words, or characters
• awk — processes and reformats structured text
• sed — performs find-and-replace or other text transformations
• cut — extracts specific columns or fields
• head / tail — shows the first or last N lines
• xargs — converts piped input into arguments for another command

Counting Lines, Words, and Characters with wc

cat file.txt | wc -l    # counts lines

Technically, in this case, you could skip the cat and just run wc -l file.txt directly, since wc can read files on its own. This is a common pattern people notice once they get more comfortable — the “useless use of cat” — but it’s not wrong, just occasionally redundant. I still use cat at the start of a pipeline sometimes because it makes the flow of data easier to read left to right, especially in longer pipelines.

Extracting Columns with cut and awk

echo "alex:x:1000:1000:Alex:/home/alex:/bin/bash" | cut -d: -f1

This outputs alex, extracting the first field from a colon-delimited string. awk can do similar things with more power:

ps aux | awk '{print $1, $11}'

This prints the first and eleventh columns (typically user and command) from the output of ps aux.

Using xargs to Pass Piped Data as Arguments

Pipes send data through standard input, but some commands expect their input as arguments rather than reading from stdin. That’s where xargs comes in:

find . -name "*.tmp" | xargs rm

This finds all .tmp files and passes each one as an argument to rm, effectively deleting them all. Without xargs, rm wouldn’t know what to do with data piped into it, since rm doesn’t read from standard input.

Redirecting Both Pipe and File Output with tee

Sometimes you want to see command output on screen and save it to a file at the same time:

ls -l | tee output.txt

tee reads from standard input, writes it to the specified file, and also passes it along to standard output — so it still works within a longer pipeline:

cat access.log | grep "ERROR" | tee errors_only.log | wc -l

This saves matching lines to errors_only.log while also counting them.

How Pipes Work Internally

When Bash encounters a pipe, it creates a pipe using the pipe() system call, which returns two file descriptors: one for reading and one for writing. Bash then forks a new process for each command in the pipeline, redirecting the write end of the pipe to the first process’s standard output and the read end to the second process’s standard input, using dup2() to make the substitution. Crucially, all commands in a pipeline start running roughly simultaneously — the kernel manages a buffer between them, so command2 can start consuming data as soon as command1 produces it, rather than waiting for command1 to finish completely. This is why pipelines can process huge files without needing to hold the entire output in memory at once. It’s also worth knowing that by default, only the exit status of the last command in a pipeline is what a script sees in $? — if an earlier command in the pipeline fails, that failure can be silently swallowed unless you enable set -o pipefail.

Real-World Use Cases

Finding the top memory-consuming processes:

ps aux --sort=-%mem | head -n 10

Counting how many times each IP address appears in a web server log:

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -n 20

Searching for a specific process and killing it:

ps aux | grep "myapp" | grep -v grep | awk '{print $2}' | xargs kill

This finds processes matching “myapp”, excludes the grep command itself from the results (a classic trick), extracts the process ID, and passes it to kill.

Automation Example: Disk Usage Report

#!/bin/bash

echo "Top 10 largest directories under /var:"
du -sh /var/* 2>/dev/null | sort -rh | head -n 10

echo ""
echo "Files modified in the last 24 hours:"
find /var/log -type f -mtime -1 | wc -l

This script combines du, sort, and head in a pipeline to produce a quick disk usage summary, along with a separate pipeline counting recently modified files.

Best Practices

• Keep pipelines readable — if a chain grows past four or five stages, consider breaking it into a script with intermediate variables or comments explaining each step.
• Use set -o pipefail in scripts where you need to detect failures anywhere in a pipeline, not just the last command.
• Prefer awk over long chains of cut, grep, and sed when the transformation logic gets complex, since a single awk script is often clearer and faster than five chained commands.
• Use tee when you need to both inspect and save intermediate pipeline output, rather than running the same command twice.

Security Considerations

• Be cautious piping output into xargs when the input might contain unexpected characters like spaces or special shell characters — use xargs -0 combined with find -print0 for filenames that might contain spaces or newlines, to avoid misinterpretation.
• Never pipe untrusted or unvalidated data directly into commands like sh, bash, or eval, since this is a classic pattern for command injection if the source of that data isn’t fully trusted (for example, piping the output of curl directly into bash without inspecting it first).
• When using sudo within a pipeline, remember that only the specific command you prefix with sudo runs with elevated privileges — the rest of the pipeline still runs as your regular user, which can lead to permission errors that are easy to misdiagnose.

Optimization Tips

• Since pipeline stages run concurrently, well-designed pipelines can process very large files efficiently without waiting for each stage to fully complete before the next begins.
• Avoid unnecessary intermediate commands (the “useless use of cat” pattern) when a command can read directly from a file, since each additional process in a pipeline adds a small amount of overhead.
• For very large datasets, tools like awk that combine filtering and transformation in a single pass are often more efficient than chaining multiple single-purpose commands together.

Troubleshooting

A pipeline seems to hang with no output: Check if an earlier command is waiting for input it’s not receiving — for example, forgetting to specify a filename to cat will cause it to wait on standard input from your keyboard.

$? shows success even though a command in the middle of the pipeline failed: By default, Bash only reports the exit status of the last command. Add set -o pipefail at the top of your script to make the pipeline’s exit status reflect any failure in the chain.

xargs mishandles filenames with spaces: Use find ... -print0 | xargs -0 ... to null-delimit filenames, which safely handles spaces and other unusual characters.

Common Mistakes

1. Forgetting that pipeline exit status defaults to the last command only, hiding failures earlier in the chain.
2. Overusing cut, grep, and sed in long chains when a single awk command would be clearer and faster.
3. Piping filenames into xargs without null-delimiting, causing issues with filenames containing spaces.
4. Piping untrusted data directly into an interpreter like bash or sh without reviewing it first.

FAQs

What’s the difference between a pipe (|) and redirection (>)? A pipe connects the output of one command directly to the input of another command. Redirection sends output to (or reads input from) a file rather than another command.

Can I pipe the output of a loop? Yes. For example: for i in {1..5}; do echo "$i"; done | sort -nr pipes the loop’s combined output into sort.

How do I see the exit status of every command in a pipeline, not just the last one? Use the PIPESTATUS array immediately after running the pipeline: echo "${PIPESTATUS[@]}" shows the exit status of each stage.

Does the order of commands in a pipeline matter for performance? Generally yes — filtering out unnecessary data as early as possible (for example, running grep before sort) reduces the amount of data later stages need to process.

Summary

Pipes are one of the most powerful and elegant features of Bash, letting you compose small, focused commands into pipelines that can filter, transform, and summarize data in a single line. Understanding how data flows between commands, knowing tools like xargs and tee for the edge cases pipes alone can’t handle, and being aware of pipefail for proper error detection will let you build pipelines that are both expressive and reliable. Once you’re fluent in piping, a huge amount of everyday text processing stops requiring a “real” script at all.

References

• Bash Reference Manual (Pipelines): https://www.gnu.org/software/bash/manual/bash.html#Pipelines
• GNU Coreutils Manual: https://www.gnu.org/software/coreutils/manual/coreutils.html
• xargs(1) man page: https://man7.org/linux/man-pages/man1/xargs.1.html