Text manipulation is where Bash genuinely shines. Long before I got comfortable with Python for data processing, I was already parsing log files, reformatting CSVs, and extracting fields from messy output using nothing but a handful of Bash tools. There’s something satisfying about chaining a few commands together and watching a wall of raw text transform into exactly the structured output you need.
This guide covers the core text manipulation tools in Bash — grep, sed, awk, cut, sort, tr, and parameter expansion — from basic usage to advanced, real-world scripting patterns.
Searching Text with grep
grep is the tool you reach for first when you need to find lines matching a pattern.
grep "error" application.log
This prints every line containing the word “error”. Some genuinely essential flags:
grep -i "error" application.log # case-insensitive
grep -v "debug" application.log # invert match — show lines NOT containing "debug"
grep -c "error" application.log # count matching lines instead of printing them
grep -n "error" application.log # show line numbers alongside matches
grep -r "TODO" ./src # recursively search all files in a directory
For regex-based searching, use -E (extended regex):
grep -E "error|warning|critical" application.log
This matches lines containing any of the three words, using the | (OR) operator in extended regex mode.
Editing Text with sed
sed (stream editor) is built for search-and-replace and line-based transformations, applied without needing to open a file in an editor.
The most common use is substitution:
sed 's/foo/bar/' file.txt
This replaces the first occurrence of “foo” with “bar” on each line. To replace every occurrence on every line, add the global flag:
sed 's/foo/bar/g' file.txt
By default, sed prints the modified text to standard output without touching the original file. To actually edit the file in place:
sed -i 's/foo/bar/g' file.txt
On macOS (BSD sed), -i requires an explicit backup extension argument (even if empty): sed -i '' 's/foo/bar/g' file.txt. This is a common cross-platform gotcha worth remembering.
Deleting lines matching a pattern:
sed '/^#/d' config.txt
This deletes any line starting with # (typically comments), which is handy for stripping comments before processing a config file further.
Printing only a specific range of lines:
sed -n '10,20p' file.txt
-nsuppresses automatic printing of every line.10,20ptellssedto print only lines 10 through 20.
Field Extraction and Processing with awk
awk is a full pattern-scanning and processing language, and it’s the tool I reach for whenever I need to work with columns of data rather than whole lines.
Given a file like this:
john,25,engineer
jane,30,designer
mike,28,manager
Extracting the second column (comma-separated):
awk -F',' '{print $2}' people.csv
-F','sets the field separator to a comma instead of the default whitespace.$2refers to the second field on each line,$1the first, and so on.$0refers to the entire line.
Filtering rows based on a condition:
awk -F',' '$2 > 27 {print $1}' people.csv
This prints the name ($1) of anyone whose age ($2) is greater than 27.
Computing sums or aggregates:
awk -F',' '{ sum += $2 } END { print "Total age:", sum }' people.csv
{ sum += $2 }runs for every line, accumulating the value of the second field.END { print ... }runs once, after all lines have been processed, printing the final total.
Formatting output with printf-style precision:
awk -F',' '{ printf "%-10s %s\n", $1, $3 }' people.csv
%-10s left-aligns the first field in a 10-character-wide column, producing neatly aligned output — extremely useful for generating readable reports directly from raw data.
Extracting Columns with cut
For simpler column extraction where you don’t need awk‘s full processing power, cut gets the job done with less overhead:
cut -d',' -f1,3 people.csv
-d','sets the delimiter to a comma.-f1,3selects the first and third fields.
Output:
john,engineer
jane,designer
mike,manager
cut is also useful for fixed-width text, using character positions instead of a delimiter:
cut -c1-10 file.txt
This extracts characters 1 through 10 from each line, regardless of delimiters — handy for legacy fixed-width data formats.
Sorting and Deduplicating with sort and uniq
sort names.txt
Sorts lines alphabetically. Useful flags:
sort -n numbers.txt # numeric sort instead of lexicographic
sort -r names.txt # reverse order
sort -k2 -t',' people.csv # sort by the 2nd field, comma-delimited
To remove duplicate lines, uniq only works correctly on sorted input, since it only removes consecutive duplicates:
sort names.txt | uniq
To count occurrences of each unique line:
sort names.txt | uniq -c
This is an extremely common pattern for quick log analysis — for example, counting how many times each IP address appears in an access log:
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
Breaking this pipeline down: extract the first field (typically the IP address) from each log line, sort the resulting list so identical IPs are adjacent, count occurrences of each with uniq -c, sort those counts numerically in reverse order (sort -rn), and show just the top 10 — giving you the ten most frequent visitors in a single line.
Character Translation with tr
tr operates on individual characters rather than whole patterns, making it ideal for simple transformations:
echo "Hello World" | tr '[:lower:]' '[:upper:]'
Output: HELLO WORLD
Removing specific characters:
echo "Hello, World!" | tr -d ','
Output: Hello World!
Squeezing repeated characters (collapsing consecutive duplicates into one):
echo "aaa bbb ccc" | tr -s ' '
Output: aaa bbb ccc
This is particularly useful for cleaning up inconsistent whitespace in messy text before further processing.
Bash’s Built-In Parameter Expansion
For simple string operations, you don’t always need an external tool — Bash’s own parameter expansion syntax handles a surprising amount natively, and it’s faster since it avoids spawning a subprocess.
filename="report.tar.gz"
echo "${filename%.gz}" # removes shortest match of .gz from the end -> report.tar
echo "${filename%%.*}" # removes longest match from first dot -> report
echo "${filename#*.}" # removes shortest match from the start -> tar.gz
echo "${filename##*.}" # removes longest match from the start -> gz
echo "${filename/tar/zip}" # replaces first match -> report.zip.gz
echo "${#filename}" # length of the string -> 15
These four operators (%, %%, #, ##) are genuinely worth memorizing:
%and%%trim from the end of the string (shortest and longest match respectively).#and##trim from the start of the string (shortest and longest match respectively).
This trick is extremely common for extracting file extensions or base filenames without spawning an external process.
Combining Tools in a Real Pipeline
Here’s a practical example combining several tools to analyze a web server access log:
#!/bin/bash
LOG_FILE="/var/log/nginx/access.log"
echo "=== Top 10 IP addresses ==="
awk '{print $1}' "$LOG_FILE" | sort | uniq -c | sort -rn | head -10
echo ""
echo "=== HTTP status code breakdown ==="
awk '{print $9}' "$LOG_FILE" | sort | uniq -c | sort -rn
echo ""
echo "=== Requests containing 'error' or '5xx' responses ==="
grep -E ' (5[0-9]{2}) ' "$LOG_FILE" | awk '{print $1, $9, $7}' | sort | uniq -c | sort -rn | head -20
This script produces a quick, readable summary of traffic patterns and error rates directly from a raw access log — the kind of ad-hoc analysis that would take much longer to set up in a full programming language.
Real-World Use Cases
1. Log analysis. Extracting error patterns, counting occurrences of specific events, and summarizing traffic by IP or status code, as shown above.
2. Data cleaning. Stripping whitespace, normalizing case, removing unwanted characters, and reformatting delimited files before importing them into a database or spreadsheet.
3. Configuration file management. Using sed to update configuration values programmatically as part of a deployment script, rather than manually editing files on each server.
4. Report generation. Combining awk and printf formatting to turn raw CSV or log data into neatly aligned, human-readable reports.
Best Practices
- Prefer
awkwhen you need column-based logic or calculations; prefersedfor line-based search-and-replace; prefergreppurely for filtering matching lines. - Always test destructive operations (
sed -i) on a copy of the file first, or use a backup extension (sed -i.bak) so you can revert if something goes wrong. - Use Bash’s built-in parameter expansion for simple string operations instead of spawning external processes like
sedorawk, which is noticeably faster in tight loops. - Chain tools with pipes rather than trying to do everything in one complex command — smaller, composable steps are easier to debug.
- Quote variables inside these commands to avoid word-splitting issues, especially when dealing with filenames or data containing spaces.
Security Considerations
- Be cautious with
sed -iand similar in-place editing tools on files you don’t fully control — a poorly crafted pattern could corrupt configuration files in unexpected ways. - Validate any data used to build
sedorawkpatterns dynamically from user input, since a malicious pattern could be used to inject unexpected behavior into your text processing pipeline. - When processing log files that might contain sensitive data (passwords, tokens accidentally logged), be mindful of where your processed output ends up — don’t inadvertently expose sensitive data in a public report or dashboard.
Troubleshooting Common Issues
sed -i fails on macOS but works on Linux: This is the classic BSD vs. GNU sed difference. On macOS, provide an empty string argument for the backup suffix: sed -i '' 's/foo/bar/' file.txt.
awk prints nothing: Double-check your field separator (-F) matches the actual delimiter in your file, and confirm the file doesn’t have Windows-style line endings (\r\n) which can throw off field parsing — dos2unix can help clean this up.
uniq -c isn’t actually deduplicating: Remember uniq only removes consecutive duplicate lines. Always sort your input first: sort file.txt | uniq -c.
Pipeline is slow on very large files: Consider whether awk alone can replace a chain of grep | cut | sort, since combining logic into a single awk script avoids the overhead of multiple subprocess invocations and repeated data passes.
Common Mistakes to Avoid
- Using
uniqwithout sorting first, leading to incorrect duplicate counts. - Forgetting that
sed‘s in-place editing (-i) behaves differently between GNU and BSD versions. - Overcomplicating a task with multiple piped tools when a single well-written
awkcommand could do it all in one pass. - Not quoting variables when building dynamic
sedorgreppatterns, risking word-splitting or glob expansion issues.
FAQs
Q: When should I use awk instead of sed? Use sed for straightforward line-based search-and-replace or deletion. Use awk when you need to work with specific columns/fields, perform calculations, or apply conditional logic based on field values.
Q: How do I edit a file in place safely? Always create a backup first: sed -i.bak 's/old/new/g' file.txt creates file.txt.bak before applying changes, so you can recover the original if needed.
Q: What’s the fastest way to count occurrences of a word across many files? grep -c "word" *.txt gives per-file counts, or grep -o "word" *.txt | wc -l gives a total count across all matches.
Q: Can Bash parameter expansion fully replace sed and awk? For simple, single-string operations (trimming, replacing a fixed substring, extracting an extension), yes — and it’s faster. For anything involving patterns across multiple lines, complex regex, or column-based logic, you’ll still want sed or awk.
Q: How do I handle CSV files with quoted fields containing commas? Basic tools like cut and simple awk -F',' calls break on quoted commas within fields. For genuinely complex CSV parsing, a dedicated tool or a proper CSV-aware library (in Python, for example) is usually more reliable than a pure Bash approach.
Summary
Bash’s text manipulation toolkit — grep, sed, awk, cut, sort, uniq, tr, and built-in parameter expansion — covers an enormous range of real-world tasks, from quick one-liners to full log analysis pipelines. Understanding which tool fits which job, and how to chain them together with pipes, turns raw, messy text into exactly the structured information you need, often faster than reaching for a full scripting language.