How to Create a Bash File Compression Tool

Compression is one of those things that feels simple until you actually need to do it well. Sure, everyone knows tar -czf archive.tar.gz folder/, but there’s a lot more nuance once you start caring about compression ratio, speed, format compatibility, and automation. I ended up building my own Bash compression tool after getting tired of manually picking flags every time, and in this article I’ll walk you through the entire process — from the simplest possible version to a genuinely useful, production-ready script.

Why Build a Custom Compression Tool

The honest answer: convenience and consistency. I wanted one command that could:

• Compress a file or folder with sensible defaults
• Let me choose the compression format when I care about it
• Show me the resulting compression ratio
• Exclude files I never want archived (like .git or node_modules)

Let’s build it step by step.

Prerequisites

sudo apt install tar gzip bzip2 xz-utils zip p7zip-full

Step 1: The Basic Compression Script

#!/usr/bin/env bash
set -euo pipefail

SOURCE="${1:-}"
FORMAT="${2:-gz}"

if [[ -z "$SOURCE" ]]; then
    echo "Usage: $0 <file-or-folder> [gz|bz2|xz|zip]"
    exit 1
fi

if [[ ! -e "$SOURCE" ]]; then
    echo "Error: '$SOURCE' does not exist."
    exit 1
fi

BASENAME=$(basename -- "$SOURCE")

case "$FORMAT" in
    gz)
        tar -czf "${BASENAME}.tar.gz" "$SOURCE"
        OUTPUT="${BASENAME}.tar.gz"
        ;;
    bz2)
        tar -cjf "${BASENAME}.tar.bz2" "$SOURCE"
        OUTPUT="${BASENAME}.tar.bz2"
        ;;
    xz)
        tar -cJf "${BASENAME}.tar.xz" "$SOURCE"
        OUTPUT="${BASENAME}.tar.xz"
        ;;
    zip)
        zip -qr "${BASENAME}.zip" "$SOURCE"
        OUTPUT="${BASENAME}.zip"
        ;;
    *)
        echo "Error: Unsupported format '$FORMAT'. Choose gz, bz2, xz, or zip."
        exit 1
        ;;
esac

echo "Created: $OUTPUT"

How This Works

• BASENAME=$(basename -- "$SOURCE") strips any trailing slash and directory path, giving us a clean name to use for the output archive.
• The case statement maps a human-friendly format name (gz, bz2, xz, zip) to the correct tar or zip invocation. This is the same pattern-matching approach we used in the decompression tool, just running in reverse.
• tar -czf — -c creates a new archive, -z compresses with gzip, -f specifies the output filename. Swap -z for -j (bzip2) or -J (xz) to change compression algorithm while keeping the same tar container format.
• zip -qr — -q runs quietly, -r recurses into directories.

Step 2: Showing Compression Ratio

Knowing how much space you actually saved is genuinely useful, so let’s add that:

ORIGINAL_SIZE=$(du -sb "$SOURCE" | cut -f1)
COMPRESSED_SIZE=$(stat -c "%s" "$OUTPUT" 2>/dev/null || stat -f "%z" "$OUTPUT")

RATIO=$(awk "BEGIN { printf \"%.1f\", ($ORIGINAL_SIZE - $COMPRESSED_SIZE) / $ORIGINAL_SIZE * 100 }")

echo "Original size:   $(numfmt --to=iec-i --suffix=B "$ORIGINAL_SIZE")"
echo "Compressed size: $(numfmt --to=iec-i --suffix=B "$COMPRESSED_SIZE")"
echo "Space saved:     ${RATIO}%"

Explaining This Section

• du -sb "$SOURCE" | cut -f1 gets the total size of the source in bytes (-s for summary, -b for bytes), then cut -f1 grabs just the numeric field, discarding the filename that du also prints.
• We use awk for the percentage calculation because Bash’s built-in arithmetic ($(( ))) only handles integers, and we want a decimal result. awk is a lightweight, universally available tool for this kind of quick math.
• numfmt --to=iec-i --suffix=B converts raw byte counts into human-readable units like 4.2MiB instead of 4404019, which is much easier to read at a glance.

Step 3: Excluding Unwanted Files and Folders

This is where a custom tool really starts to shine over typing raw tar commands. Let’s add an exclude list:

EXCLUDES=(".git" "node_modules" "*.log" "__pycache__")

EXCLUDE_ARGS=()
for pattern in "${EXCLUDES[@]}"; do
    EXCLUDE_ARGS+=(--exclude="$pattern")
done

tar -czf "${BASENAME}.tar.gz" "${EXCLUDE_ARGS[@]}" "$SOURCE"

The Mechanics

• EXCLUDES=(...) declares a Bash array holding our exclusion patterns.
• The for loop builds a second array, EXCLUDE_ARGS, where each pattern is turned into a properly formatted --exclude="pattern" flag.
• "${EXCLUDE_ARGS[@]}" expands the array into separate, correctly quoted arguments when passed to tar — this is the safe way to build a dynamic list of command-line flags in Bash. Using a plain string here instead of an array would break on patterns containing spaces.

Step 4: Putting It All Together

Here’s the complete, polished script:

#!/usr/bin/env bash
set -euo pipefail

SOURCE="${1:-}"
FORMAT="${2:-gz}"
EXCLUDES=(".git" "node_modules" "*.log" "__pycache__")

if [[ -z "$SOURCE" || ! -e "$SOURCE" ]]; then
    echo "Usage: $0 <file-or-folder> [gz|bz2|xz|zip]"
    exit 1
fi

BASENAME=$(basename -- "$SOURCE")
EXCLUDE_ARGS=()
for pattern in "${EXCLUDES[@]}"; do
    EXCLUDE_ARGS+=(--exclude="$pattern")
done

case "$FORMAT" in
    gz)  OUTPUT="${BASENAME}.tar.gz"; tar -czf "$OUTPUT" "${EXCLUDE_ARGS[@]}" "$SOURCE" ;;
    bz2) OUTPUT="${BASENAME}.tar.bz2"; tar -cjf "$OUTPUT" "${EXCLUDE_ARGS[@]}" "$SOURCE" ;;
    xz)  OUTPUT="${BASENAME}.tar.xz"; tar -cJf "$OUTPUT" "${EXCLUDE_ARGS[@]}" "$SOURCE" ;;
    zip) OUTPUT="${BASENAME}.zip"; zip -qr "$OUTPUT" "$SOURCE" -x ".git/*" "node_modules/*" ;;
    *) echo "Unsupported format: $FORMAT"; exit 1 ;;
esac

ORIGINAL_SIZE=$(du -sb "$SOURCE" | cut -f1)
COMPRESSED_SIZE=$(stat -c "%s" "$OUTPUT" 2>/dev/null || stat -f "%z" "$OUTPUT")
RATIO=$(awk "BEGIN { printf \"%.1f\", ($ORIGINAL_SIZE - $COMPRESSED_SIZE) / $ORIGINAL_SIZE * 100 }")

echo "Created: $OUTPUT"
echo "Original size:   $(numfmt --to=iec-i --suffix=B "$ORIGINAL_SIZE")"
echo "Compressed size: $(numfmt --to=iec-i --suffix=B "$COMPRESSED_SIZE")"
echo "Space saved:     ${RATIO}%"

Sample output:

Created: myproject.tar.gz
Original size:   128MiB
Compressed size: 31MiB
Space saved:     75.8%

Real-World Use Cases

• Pre-deployment packaging — compressing a build folder before shipping it to a server, excluding development artifacts.
• Log rotation — periodically compressing old log files to save disk space (paired with the exclude logic to skip active logs).
• Email attachments — quickly zipping a folder of documents to send as one file.
• Backup staging — this tool pairs naturally with the Bash File Backup Tool script, acting as the compression step before files are moved offsite.

Automation Example

Compress and timestamp a project folder nightly:

0 1 * * * /usr/local/bin/compress.sh /var/www/myproject gz && mv myproject.tar.gz /backups/myproject-$(date +\%F).tar.gz

This runs at 1 AM, compresses the project directory, and renames the output with the current date for easy identification later.

Security Considerations

• Never compress directories containing secrets (like .env files) into archives that will be shared publicly. Add sensitive patterns to your EXCLUDES array as a safety habit.
• Set restrictive permissions on archives containing sensitive data: chmod 600 archive.tar.gz ensures only your user can read it.
• Be cautious compressing symlinks. By default tar follows symlinks into their targets unless you pass -h/--dereference deliberately or avoid it — know which behavior you want, since accidentally archiving symlink targets can leak files outside the intended folder.
• Checksum your archives after creation (sha256sum archive.tar.gz > archive.tar.gz.sha256) so recipients — or future you — can verify integrity.

Optimization Tips

• xz typically achieves the best compression ratio but is the slowest; gzip is fastest but compresses less; bzip2 sits in between. Choose based on whether you’re optimizing for speed or size.
• For huge datasets, use pigz (parallel gzip) or pxz (parallel xz) to use multiple CPU cores: tar --use-compress-program=pigz -cf archive.tar.gz folder/.
• Compression level matters: gzip -9 maximizes compression at the cost of speed, while gzip -1 is fastest with less compression. Pass this through tar via GZIP=-9 tar -czf ....

Troubleshooting

• “tar: Removing leading ‘/’ from member names” — this is just a warning, not an error; tar is converting absolute paths to relative ones for safety. It’s expected behavior, not a bug.
• Archive is unexpectedly large — check whether your exclude patterns are actually matching; tar --exclude patterns are matched against the path as stored in the archive, which can behave differently than you expect with nested paths.
• “zip: command not found” — install it explicitly; unlike tar, zip/unzip aren’t always preinstalled: sudo apt install zip.
• Compression seems to hang on large folders — this is often disk I/O bound rather than actually stuck; check with iostat or simply be patient with very large datasets.

Common Mistakes to Avoid

• Compressing a folder without checking its contents first, potentially bundling gigabytes of cache files or logs unintentionally.
• Using zip on Linux systems where tar.gz would be more appropriate (and vice versa on Windows-heavy environments) — pick the format your audience can actually open.
• Forgetting to test that an archive actually extracts correctly before relying on it for backup purposes.
• Not accounting for extremely long file paths, which can cause issues with older archive formats like classic zip.

Frequently Asked Questions

Which compression format should I use by default? For general Linux-to-Linux use, tar.gz is a safe, fast, widely-compatible default. Use .zip when sharing with Windows users, and .xz when you need maximum compression and don’t mind the extra time.

Can I encrypt the archive at the same time as compressing it? Yes — pipe the output through gpg: tar -czf - folder/ | gpg -c -o archive.tar.gz.gpg. This compresses and symmetrically encrypts in one pipeline.

How do I compress without including the parent directory structure? cd into the folder first, then compress its contents with . as the source, rather than compressing the folder from outside.

Summary

We built a flexible Bash compression tool that supports multiple formats, reports compression ratios, and intelligently excludes files you never want archived. The underlying lesson is that tar‘s consistent flag structure (-c, -z/-j/-J, -f) makes it easy to build a single script that adapts to different compression algorithms, and that small usability touches — like human-readable size output — make a world of difference in a tool you’ll use daily.

References

• GNU Tar Manual
• GNU Gzip Manual
• Bash Reference Manual — GNU Project