Offensive Security Researcher & Quantum Cryptography Analyst

How to Pipe Commands in Bash

Abdul Wahab Junaid
How to Pipe Commands in Bash

How to Pipe Commands in Bash

If there’s one feature that captures the spirit of the Unix philosophy — small tools that do one thing well, combined to do something powerful — it’s the pipe. The first time I chained three or four commands together with pipes and watched a messy log file turn into a clean, sorted summary in one line, I understood why so many people describe the shell as addictive once it clicks. In this article, I’ll cover how piping works, common patterns, and how to build genuinely useful one-liners and scripts around it.

What Is a Pipe?

A pipe, written as |, takes the standard output of one command and feeds it directly into the standard input of the next command.

command1 | command2

Here, whatever command1 prints to the screen instead gets sent directly to command2 as its input.

A Simple Example

ls -l | grep ".txt"

This lists files in the current directory (ls -l) and pipes that output into grep, which filters the lines to only those containing “.txt”.

Chaining Multiple Pipes

You’re not limited to a single pipe — you can chain as many as you need:

cat access.log | grep "ERROR" | sort | uniq -c | sort -nr

Let’s break this down:

  1. cat access.log — outputs the full contents of the log file.
  2. grep "ERROR" — filters to only lines containing “ERROR”.
  3. sort — sorts those lines alphabetically (important for the next step).
  4. uniq -c — collapses duplicate lines and counts how many times each appeared.
  5. sort -nr — sorts numerically in reverse order, so the most frequent errors appear first.

This kind of pipeline is one of the most common patterns in day-to-day command-line work — take raw text, filter it, transform it, and summarize it, all in one line.

Common Commands Used in Pipelines

Counting Lines, Words, and Characters with wc

cat file.txt | wc -l    # counts lines

Technically, in this case, you could skip the cat and just run wc -l file.txt directly, since wc can read files on its own. This is a common pattern people notice once they get more comfortable — the “useless use of cat” — but it’s not wrong, just occasionally redundant. I still use cat at the start of a pipeline sometimes because it makes the flow of data easier to read left to right, especially in longer pipelines.

Extracting Columns with cut and awk

echo "alex:x:1000:1000:Alex:/home/alex:/bin/bash" | cut -d: -f1

This outputs alex, extracting the first field from a colon-delimited string. awk can do similar things with more power:

ps aux | awk '{print $1, $11}'

This prints the first and eleventh columns (typically user and command) from the output of ps aux.

Using xargs to Pass Piped Data as Arguments

Pipes send data through standard input, but some commands expect their input as arguments rather than reading from stdin. That’s where xargs comes in:

find . -name "*.tmp" | xargs rm

This finds all .tmp files and passes each one as an argument to rm, effectively deleting them all. Without xargs, rm wouldn’t know what to do with data piped into it, since rm doesn’t read from standard input.

Redirecting Both Pipe and File Output with tee

Sometimes you want to see command output on screen and save it to a file at the same time:

ls -l | tee output.txt

tee reads from standard input, writes it to the specified file, and also passes it along to standard output — so it still works within a longer pipeline:

cat access.log | grep "ERROR" | tee errors_only.log | wc -l

This saves matching lines to errors_only.log while also counting them.

How Pipes Work Internally

When Bash encounters a pipe, it creates a pipe using the pipe() system call, which returns two file descriptors: one for reading and one for writing. Bash then forks a new process for each command in the pipeline, redirecting the write end of the pipe to the first process’s standard output and the read end to the second process’s standard input, using dup2() to make the substitution. Crucially, all commands in a pipeline start running roughly simultaneously — the kernel manages a buffer between them, so command2 can start consuming data as soon as command1 produces it, rather than waiting for command1 to finish completely. This is why pipelines can process huge files without needing to hold the entire output in memory at once. It’s also worth knowing that by default, only the exit status of the last command in a pipeline is what a script sees in $? — if an earlier command in the pipeline fails, that failure can be silently swallowed unless you enable set -o pipefail.

Real-World Use Cases

Finding the top memory-consuming processes:

ps aux --sort=-%mem | head -n 10

Counting how many times each IP address appears in a web server log:

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head -n 20

Searching for a specific process and killing it:

ps aux | grep "myapp" | grep -v grep | awk '{print $2}' | xargs kill

This finds processes matching “myapp”, excludes the grep command itself from the results (a classic trick), extracts the process ID, and passes it to kill.

Automation Example: Disk Usage Report

#!/bin/bash

echo "Top 10 largest directories under /var:"
du -sh /var/* 2>/dev/null | sort -rh | head -n 10

echo ""
echo "Files modified in the last 24 hours:"
find /var/log -type f -mtime -1 | wc -l

This script combines du, sort, and head in a pipeline to produce a quick disk usage summary, along with a separate pipeline counting recently modified files.

Best Practices

Security Considerations

Optimization Tips

Troubleshooting

A pipeline seems to hang with no output: Check if an earlier command is waiting for input it’s not receiving — for example, forgetting to specify a filename to cat will cause it to wait on standard input from your keyboard.

$? shows success even though a command in the middle of the pipeline failed: By default, Bash only reports the exit status of the last command. Add set -o pipefail at the top of your script to make the pipeline’s exit status reflect any failure in the chain.

xargs mishandles filenames with spaces: Use find ... -print0 | xargs -0 ... to null-delimit filenames, which safely handles spaces and other unusual characters.

Common Mistakes

  1. Forgetting that pipeline exit status defaults to the last command only, hiding failures earlier in the chain.
  2. Overusing cut, grep, and sed in long chains when a single awk command would be clearer and faster.
  3. Piping filenames into xargs without null-delimiting, causing issues with filenames containing spaces.
  4. Piping untrusted data directly into an interpreter like bash or sh without reviewing it first.

FAQs

What’s the difference between a pipe (|) and redirection (>)? A pipe connects the output of one command directly to the input of another command. Redirection sends output to (or reads input from) a file rather than another command.

Can I pipe the output of a loop? Yes. For example: for i in {1..5}; do echo "$i"; done | sort -nr pipes the loop’s combined output into sort.

How do I see the exit status of every command in a pipeline, not just the last one? Use the PIPESTATUS array immediately after running the pipeline: echo "${PIPESTATUS[@]}" shows the exit status of each stage.

Does the order of commands in a pipeline matter for performance? Generally yes — filtering out unnecessary data as early as possible (for example, running grep before sort) reduces the amount of data later stages need to process.

Summary

Pipes are one of the most powerful and elegant features of Bash, letting you compose small, focused commands into pipelines that can filter, transform, and summarize data in a single line. Understanding how data flows between commands, knowing tools like xargs and tee for the edge cases pipes alone can’t handle, and being aware of pipefail for proper error detection will let you build pipelines that are both expressive and reliable. Once you’re fluent in piping, a huge amount of everyday text processing stops requiring a “real” script at all.

References

Exit mobile version