There was a period early in my Linux career where I avoided sed entirely and did everything with a text editor and manual find-and-replace, even on remote servers over SSH. That was slow and, honestly, embarrassing in retrospect. The turning point was a task that would have taken hours by hand: renaming a config key across 300 files on a production box, with zero downtime tolerance for mistakes. One sed -i command, tested first on a copy, did the whole job in seconds. I’ve never gone back.
sed — short for stream editor — is one of the most powerful and most misunderstood tools in the Linux world. It’s not just “find and replace.” It’s a full, if minimal, scripting language for transforming text line by line, with pattern matching, addressing, branching, and hold-space manipulation that can do things most people never realize it’s capable of.
What sed Does
sed reads input line by line (from a file or STDIN), applies a set of editing commands to each line, and writes the result to standard output (or, with -i, back into the file).
sed [OPTIONS] 'SCRIPT' [FILE]...
The most basic and most common form is substitution:
$ echo "hello world" | sed 's/world/linux/'
hello linux
I tested this directly and it produced exactly what you’d expect — world replaced with linux.
How sed Works Internally
sed operates on a concept called the pattern space. For each input line, sed loads that line into the pattern space, applies every command in the script to it, and — unless told otherwise — prints the pattern space at the end of the cycle before moving to the next line. There’s also a secondary buffer called the hold space, which persists across cycles and lets you stash content from one line to use later — this is what enables advanced multi-line operations like reversing a file’s lines or joining multiple lines together.
This line-by-line, pattern-space model is why sed is described as a “stream editor” rather than a text editor: it never loads the whole file into memory the way a text editor does (in the default execution model), and it processes input as a continuous stream, one line at a time — which is exactly why it can handle files far larger than available RAM without trouble.
Basic Syntax: The s Command
The substitution command has the form:
s/PATTERN/REPLACEMENT/FLAGS
PATTERN— a regular expression (basic regex by default, extended with-E).REPLACEMENT— the text to substitute in.FLAGS— modifiers controlling the substitution behavior.
g — Global (Replace All Matches on the Line)
By default, sed only replaces the first match per line. Add g to replace every match:
$ echo "aa bb aa cc aa" | sed 's/aa/XX/g'
XX bb XX cc XX
p — Print (Used with -n)
Combine -n (suppress automatic printing) with the p flag or command to print only specific lines:
$ seq 1 5 | sed -n '2p;4p'
2
4
Numeric Flag — Replace Only the Nth Occurrence
echo "a a a a" | sed 's/a/X/2'
This replaces only the second occurrence of a on each line.
i — Case-Insensitive Matching
echo "Hello HELLO hello" | sed 's/hello/hi/gi'
Line Addressing
One of sed‘s most useful features is targeting specific lines or line ranges before applying a command.
By Line Number
$ printf "1\n2\n3\n4\n" | sed '2d'
1
3
4
I tested this: line 2 was deleted (d command), everything else printed normally.
By Range
$ seq 1 10 | sed '3,5d'
1
2
6
7
8
9
10
This deletes lines 3 through 5 inclusive — confirmed by the tested output above, which skips straight from 2 to 6.
By Regular Expression
$ printf "apple\nbanana\ncherry\n" | sed '/banana/d'
apple
cherry
Any line matching the pattern between the slashes gets the command applied — here, deleted.
Printing a Line Range
seq 1 10 | sed -n '3,5p'
This is functionally similar to head -5 file | tail -3 but expressed as a single, more explicit command.
$ — Last Line
sed '$d' file.txt
Deletes only the last line of the file — useful for stripping trailing footer lines from generated reports.
Multiple Commands with -e
You can chain multiple edit commands in a single invocation using -e:
$ echo "hello world" | sed -e 's/hello/hi/' -e 's/world/earth/'
hi earth
Or, equivalently, separate commands with semicolons inside a single script string: sed 's/hello/hi/; s/world/earth/'.
In-Place Editing with -i
This is the option that makes sed genuinely dangerous if used carelessly and genuinely powerful when used correctly:
sed -i 's/test/TEST/' file.txt
I tested this on a copy of a sample file:
$ sed -i 's/test/TEST/' sample_copy.txt
$ cat sample_copy.txt
TEST line 1
TEST LINE 2
Test Line 3
The file was modified directly — no output printed to the terminal, and no > redirection needed.
Always create a backup when using -i on anything important. GNU sed supports an optional backup suffix:
sed -i.bak 's/old/new/' config.conf
This edits config.conf in place but first saves the original as config.conf.bak. I make this a non-negotiable habit on any production config file edit.
Backreferences and Capture Groups
& — The Whole Matched Text
$ echo "foo" | sed 's/foo/[&]/'& in the replacement represents whatever was matched by the pattern — here, wrapping the matched text in brackets without needing to retype it.
\1, \2, … — Captured Groups
With basic regex, capture groups need escaped parentheses \(...\). With extended regex (-E or -r), you use plain parentheses:
$ echo "John Smith" | sed -E 's/([A-Za-z]+) ([A-Za-z]+)/\2 \1/'
Smith John
I tested this and confirmed it correctly swapped first and last name using two capture groups and backreferences in the replacement.
Insert, Append, and Change Commands
a — Append Text After a Line
$ printf "line1\nline2\n" | sed '1a\
inserted after 1'
line1
inserted after 1
line2
i — Insert Text Before a Line
sed '3i\
--- new section ---' file.txt
c — Change (Replace) a Whole Line
sed '2c\
This line replaces line 2 entirely' file.txt
Note: in GNU sed, you can often use these commands more compactly on one line (sed '1a inserted text'), but the backslash-newline form shown above is the traditional, most portable syntax across sed implementations.
The y Command — Character Transliteration
$ echo "hello" | sed 'y/el/ip/'
hippo
This is sed‘s equivalent of tr for simple one-to-one character mapping — e→i, l→p in this example.
Practical, Real-World Examples
1. Replacing a Config Value Across Many Files
find /etc/myapp -name '*.conf' -exec sed -i 's/^debug=false/debug=true/' {} \;
2. Removing Comments and Blank Lines from a Config File
sed -e '/^#/d' -e '/^$/d' nginx.conf
3. Extracting a Block Between Two Markers
sed -n '/BEGIN/,/END/p' file.txt
4. Stripping Trailing Whitespace
sed -i 's/[[:space:]]*$//' file.txt
5. Converting Windows Line Endings
sed -i 's/\r$//' file.txt
6. Numbering Lines Manually (Without nl)
sed = file.txt | sed 'N;s/\n/\t/'
7. Double-Spacing a File
sed 'G' file.txt
The G command appends the (empty, by default) hold space after the pattern space, effectively inserting a blank line after every line.
8. Reversing a File’s Lines (a tac Substitute)
sed '1!G;h;$!d' file.txt
This is the classic sed one-liner for reversing a file when tac isn’t available (e.g., on BSD/macOS) — it builds up the hold space in reverse order and dumps it at the end.
sed in Shell Scripting and Automation
A pattern I use often in deployment scripts is templating config files by replacing placeholders with actual environment-specific values:
#!/bin/bash
sed -e "s/{{DB_HOST}}/$DB_HOST/g" \
-e "s/{{DB_PORT}}/$DB_PORT/g" \
-e "s/{{APP_ENV}}/$APP_ENV/g" \
config.template > config.conf
This is a lightweight alternative to a full templating engine like Jinja or Consul-template when the number of variables is small and you want zero extra dependencies.
Another common automation pattern: bumping a version number in a file as part of a release script:
sed -i -E "s/^version = \".*\"/version = \"$NEW_VERSION\"/" pyproject.toml
Comparing sed to Related Commands
| Task | Best Tool |
|---|---|
| Simple character substitution | tr |
| Pattern-based line editing, regex substitution | sed |
| Field/column-based processing, computation | awk |
| Multi-line, programmatic transformations | awk or a scripting language |
| Applying a pre-generated diff to a file | patch |
The general rule I follow: if it’s a single substitution or deletion based on a pattern, sed is usually the fastest thing to write. If I need to reason about columns, do arithmetic, or maintain state across many fields, I reach for awk instead — trying to force complex logic into sed scripts usually turns into an unreadable mess of hold-space juggling.
Troubleshooting Common sed Issues
sed: -e expression #1, char 1: unterminated 's' command — usually a missing closing delimiter (/) or an unescaped delimiter character inside the pattern or replacement. If your pattern contains /, switch the delimiter: sed 's#/old/path#/new/path#'.
In-place edit produces no output and no error, but nothing seems to have changed — check whether the pattern actually matched; sed silently does nothing on non-matching lines by design, which is correct behavior but easy to mistake for a bug.
Regex behaves differently than expected — remember GNU sed uses Basic Regular Expressions (BRE) by default, where +, ?, |, and grouping parentheses need to be escaped (\+, \?, \|, \(...\)). Use -E (or -r on some systems) for Extended Regular Expressions (ERE), where those characters work unescaped.
Backup files (.bak) cluttering directories after -i.bak — this is expected behavior; clean them up explicitly with find . -name '*.bak' -delete once you’ve verified the changes.
Performance Optimization
sed streams input and doesn’t hold the whole file in memory (aside from hold-space usage in multi-line scripts), so it scales well to very large files. For maximum performance on huge files with many substitution commands, combining multiple -e expressions into a single sed invocation is faster than piping several separate sed processes together, since each pipe stage adds process-spawning and I/O overhead.
Security Implications
Be extremely careful with sed -i combined with dynamically constructed patterns from untrusted input — improperly escaped regex metacharacters or delimiter characters from user input can produce unintended matches or, in scripts that build the whole sed command as a string and eval it, even command injection. Always validate or escape untrusted input before interpolating it into a sed script, and avoid eval entirely where possible.
Compatibility Across Distributions
GNU sed (tested here at version 4.9) is standard on Ubuntu, Debian, Fedora, RHEL/CentOS, Arch, and openSUSE. macOS ships BSD sed, which has notably different syntax for in-place editing — BSD sed requires an explicit (even if empty) backup suffix argument: sed -i '' 's/old/new/' file.txt versus GNU’s sed -i 's/old/new/' file.txt. This difference trips up a huge number of scripts that were only ever tested on Linux; if portability to macOS matters, either detect the platform and branch, or install GNU sed via Homebrew as gsed.
A Closer Look at Basic vs Extended Regular Expressions
Since regex flavor differences are one of the most common sources of confusion for people moving between sed and other tools, it’s worth spending more time on exactly what changes between Basic Regular Expressions (BRE) and Extended Regular Expressions (ERE). In BRE, the default mode for sed, metacharacters like + (one or more), ? (zero or one), | (alternation), and grouping parentheses (...) all lose their special meaning unless escaped with a backslash. So matching “one or more digits” in BRE looks like:
echo "abc123" | sed 's/[0-9]\+/NUM/'
While in ERE (-E or -r), the same pattern drops the backslash:
echo "abc123" | sed -E 's/[0-9]+/NUM/'
This inversion — plain characters are special in ERE but literal in BRE, and vice versa when escaped — trips up almost everyone who’s used to Perl-compatible or JavaScript-style regex, where +, ?, and | are always special without escaping. My personal habit is to default to -E for anything beyond the simplest substitution, since it matches what most people’s regex intuition already expects from other languages and tools, and it avoids the visual noise of backslash-escaping common quantifiers.
Understanding sed’s Exit Status and Error Handling in Scripts
For anything beyond one-off interactive use, it’s worth knowing how sed reports failure so scripts can respond appropriately. sed exits with a non-zero status if it encounters a genuine syntax error in the script itself, or if it can’t open a specified input file. It does not exit with an error just because a pattern failed to match any line — that’s considered normal, successful operation, since “no lines matched” is a perfectly valid outcome for a substitution or deletion command. If your script needs to know whether a substitution actually changed anything (as opposed to running successfully but matching nothing), you generally need an explicit check, such as comparing file content before and after, or using the w flag combined with a sentinel file:
sed 's/^debug=false/debug=true/w /tmp/sed_changed_marker' config.conf
if [ -s /tmp/sed_changed_marker ]; then
echo "Config was updated"
else
echo "Pattern not found — config unchanged" >&2
fi
sed’s Hold Space in More Depth
The hold space is the feature that separates sed‘s truly advanced usage from everyday substitution work, and it’s worth walking through a concrete example rather than just describing it abstractly. Consider joining every pair of lines in a file into one line — a task that requires remembering the previous line while processing the current one, which is exactly what the hold space is for:
sed 'N;s/\n/ /' file.txt
Here, N appends the next input line to the current pattern space (rather than replacing it), joining two lines into a single pattern space separated by an embedded newline. The subsequent s/\n/ / then replaces that embedded newline with a space, effectively merging each pair of lines into one. This pattern — using N to pull in additional lines before applying a transformation — is the foundation for a huge range of multi-line sed scripts, from paragraph reflowing to HTML tag stripping across line boundaries, well beyond what a purely line-at-a-time tool could achieve.
Using sed for Quick Data Extraction Without Full Regex Mastery
Not every use of sed needs deep regex knowledge. A genuinely common, low-complexity pattern is using sed -n with address ranges to extract structured sections from semi-structured text, like pulling a specific section out of an INI-style config file:
sed -n '/^\[database\]/,/^\[/p' config.ini
This prints everything from the [database] section header up to (and including) the next section header, giving you just that one config block — useful for quickly inspecting a specific section of a large config file without needing to open it in an editor or write a full parser.
Summary
sed rewards the investment of actually learning its model — pattern space, hold space, addressing, and the small set of core commands (s, d, p, a, i, c, y) cover the overwhelming majority of real-world text transformation needs. The s command with g, capture groups, and -i alone will handle most day-to-day tasks; the rest becomes useful once you start writing more elaborate one-liners or multi-line scripts for genuinely complex text surgery.
References
- GNU sed Manual: https://www.gnu.org/software/sed/manual/sed.html
- POSIX Specification for
sed: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html man sed(local manual page)