Offensive Security Researcher & Quantum Cryptography Analyst

How to Download Files with Bash

Abdul Wahab Junaid
How to Download Files with Bash

How to Download Files with Bash

I download files from the command line more times a day than I’d like to admit — installers, configuration files, release archives, container images, datasets. Over the years I’ve settled into a reliable set of Bash patterns using curl and wget that handle retries, resuming, verification, and error handling properly. In this article I’ll share everything I use, from a basic single download to a resilient batch downloader with checksum verification.

curl vs wget: Which One Should You Use?

Both tools ship on most Linux distributions, and macOS has curl pre-installed (though wget needs Homebrew). I use both depending on the task:

Basic Download with curl

curl -O https://example.com/file.zip

The -O flag tells curl to save the file using the same name it has on the server, rather than printing the contents to stdout.

To save it under a custom name:

curl -o myfile.zip https://example.com/file.zip

Basic Download with wget

wget https://example.com/file.zip

wget defaults to saving with the original filename and shows a progress bar automatically, which is one reason I reach for it during interactive downloads.

Step-by-Step: A Robust Download Script

Here’s a script I use whenever a download absolutely needs to succeed, with retries and proper error handling:

#!/usr/bin/env bash
set -euo pipefail

URL="$1"
OUTPUT="${2:-$(basename "$URL")}"
MAX_RETRIES=5
RETRY_DELAY=3

download_file() {
    local attempt=1
    while [ "$attempt" -le "$MAX_RETRIES" ]; do
        echo "Attempt $attempt of $MAX_RETRIES: downloading $URL"
        if curl -fSL --connect-timeout 10 -o "$OUTPUT" "$URL"; then
            echo "Download succeeded: $OUTPUT"
            return 0
        fi
        echo "Download failed, retrying in ${RETRY_DELAY}s..."
        sleep "$RETRY_DELAY"
        attempt=$((attempt + 1))
    done
    echo "Failed to download after $MAX_RETRIES attempts" >&2
    return 1
}

download_file

Run it like this:

./download.sh https://example.com/bigfile.tar.gz

Explaining the Script Internally

Resuming Interrupted Downloads

For large files, an interrupted connection means starting over unless you explicitly support resuming.

With curl:

curl -C - -O https://example.com/largefile.iso

The -C - flag tells curl to automatically detect how much of the file already exists locally and resume from that byte offset.

With wget:

wget -c https://example.com/largefile.iso

-c (continue) does the same thing — it checks the existing partial file size and resumes the download from there, as long as the server supports HTTP range requests.

Verifying File Integrity After Download

Downloading a file is only half the job — I always verify the checksum when one is provided, especially for anything I’m going to execute or install.

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com/tool.tar.gz"
EXPECTED_SHA256="a1b2c3d4e5f6..."
OUTPUT="tool.tar.gz"

curl -fSL -o "$OUTPUT" "$URL"

ACTUAL_SHA256=$(sha256sum "$OUTPUT" | awk '{print $1}')

if [ "$ACTUAL_SHA256" != "$EXPECTED_SHA256" ]; then
    echo "Checksum mismatch! Expected $EXPECTED_SHA256 but got $ACTUAL_SHA256" >&2
    rm -f "$OUTPUT"
    exit 1
fi

echo "Checksum verified successfully."

sha256sum computes the SHA-256 hash of the downloaded file, and awk '{print $1}' extracts just the hash portion (since sha256sum‘s output includes both the hash and the filename).

Real-World Use Case: Batch Downloading a List of URLs

I often need to download dozens of files listed in a text file, one URL per line:

#!/usr/bin/env bash
set -euo pipefail

URL_LIST="urls.txt"
DEST_DIR="downloads"

mkdir -p "$DEST_DIR"

while IFS= read -r url; do
    [ -z "$url" ] && continue
    filename=$(basename "$url")
    echo "Downloading $filename..."
    curl -fSL -o "${DEST_DIR}/${filename}" "$url" || echo "Failed: $url" >> failed_downloads.txt
done < "$URL_LIST"

echo "Batch download complete."

This pattern is useful for mirroring assets, pulling multiple dataset files, or grabbing a set of release artifacts across platforms.

Automation Example: Nightly Download and Extraction

Here’s a cron-driven script I use to pull the latest nightly build of a tool and extract it automatically:

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com/nightly/latest.tar.gz"
DEST="/opt/nightly-build"

TMP_FILE=$(mktemp)
curl -fSL -o "$TMP_FILE" "$URL"

mkdir -p "$DEST"
tar -xzf "$TMP_FILE" -C "$DEST" --strip-components=1
rm -f "$TMP_FILE"

echo "Nightly build updated in $DEST"

Scheduled nightly via:

0 2 * * * /usr/local/bin/update_nightly.sh >> /var/log/nightly_update.log 2>&1

Security Considerations

Optimization Tips

Troubleshooting

Common Mistakes to Avoid

FAQs

Which is better for scripting, curl or wget? Both are fine; curl tends to be preferred when you need more control over requests (headers, methods, auth), while wget is often simpler for straightforward file downloads and has built-in recursive support.

How do I download a file only if it’s newer than my local copy? With wget, use -N (timestamping): wget -N https://example.com/file.zip. With curl, use -z with a reference date: curl -z file.zip -O https://example.com/file.zip.

How do I download through a proxy? Both tools respect the http_proxy and https_proxy environment variables, or you can pass --proxy explicitly to curl.

Can I show a progress bar with curl? Yes, curl shows a progress meter by default unless you use -s (silent). Use --progress-bar for a simpler, cleaner bar.

Summary

Downloading files reliably from Bash comes down to a handful of habits: prefer HTTPS, always check for HTTP errors with -f, add timeouts so scripts never hang indefinitely, verify checksums for anything important, and support resuming for large files. Once these patterns became muscle memory, my download scripts went from “usually works” to something I can genuinely trust in unattended automation.

References

Exit mobile version