How to Download Files with Bash

I download files from the command line more times a day than I’d like to admit — installers, configuration files, release archives, container images, datasets. Over the years I’ve settled into a reliable set of Bash patterns using curl and wget that handle retries, resuming, verification, and error handling properly. In this article I’ll share everything I use, from a basic single download to a resilient batch downloader with checksum verification.

curl vs wget: Which One Should You Use?

Both tools ship on most Linux distributions, and macOS has curl pre-installed (though wget needs Homebrew). I use both depending on the task:

• curl is better when I need fine control over headers, authentication, or when the download is part of a larger scripted interaction with an API.
• wget is better for straightforward, unattended downloads, especially when I want built-in recursive downloading or simple resume support with minimal flags.

Basic Download with curl

curl -O https://example.com/file.zip

The -O flag tells curl to save the file using the same name it has on the server, rather than printing the contents to stdout.

To save it under a custom name:

curl -o myfile.zip https://example.com/file.zip

Basic Download with wget

wget https://example.com/file.zip

wget defaults to saving with the original filename and shows a progress bar automatically, which is one reason I reach for it during interactive downloads.

Step-by-Step: A Robust Download Script

Here’s a script I use whenever a download absolutely needs to succeed, with retries and proper error handling:

#!/usr/bin/env bash
set -euo pipefail

URL="$1"
OUTPUT="${2:-$(basename "$URL")}"
MAX_RETRIES=5
RETRY_DELAY=3

download_file() {
    local attempt=1
    while [ "$attempt" -le "$MAX_RETRIES" ]; do
        echo "Attempt $attempt of $MAX_RETRIES: downloading $URL"
        if curl -fSL --connect-timeout 10 -o "$OUTPUT" "$URL"; then
            echo "Download succeeded: $OUTPUT"
            return 0
        fi
        echo "Download failed, retrying in ${RETRY_DELAY}s..."
        sleep "$RETRY_DELAY"
        attempt=$((attempt + 1))
    done
    echo "Failed to download after $MAX_RETRIES attempts" >&2
    return 1
}

download_file

Run it like this:

./download.sh https://example.com/bigfile.tar.gz

Explaining the Script Internally

• "${2:-$(basename "$URL")}" uses Bash parameter expansion: if a second argument (custom output filename) wasn’t provided, it defaults to the filename extracted from the URL using basename.
• curl -fSL: -f makes curl fail silently (return a non-zero exit code) on HTTP errors like 404 instead of saving an error page as if it were the file; -S shows the error message even with -s silent mode; -L follows redirects, which matters a lot since many download links redirect through CDNs.
• --connect-timeout 10 prevents the script from hanging indefinitely if the server never responds.
• The while loop with a manually incremented attempt counter implements a simple retry mechanism, pausing between attempts with sleep to avoid hammering a flaky server.

Resuming Interrupted Downloads

For large files, an interrupted connection means starting over unless you explicitly support resuming.

With curl:

curl -C - -O https://example.com/largefile.iso

The -C - flag tells curl to automatically detect how much of the file already exists locally and resume from that byte offset.

With wget:

wget -c https://example.com/largefile.iso

-c (continue) does the same thing — it checks the existing partial file size and resumes the download from there, as long as the server supports HTTP range requests.

Verifying File Integrity After Download

Downloading a file is only half the job — I always verify the checksum when one is provided, especially for anything I’m going to execute or install.

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com/tool.tar.gz"
EXPECTED_SHA256="a1b2c3d4e5f6..."
OUTPUT="tool.tar.gz"

curl -fSL -o "$OUTPUT" "$URL"

ACTUAL_SHA256=$(sha256sum "$OUTPUT" | awk '{print $1}')

if [ "$ACTUAL_SHA256" != "$EXPECTED_SHA256" ]; then
    echo "Checksum mismatch! Expected $EXPECTED_SHA256 but got $ACTUAL_SHA256" >&2
    rm -f "$OUTPUT"
    exit 1
fi

echo "Checksum verified successfully."

sha256sum computes the SHA-256 hash of the downloaded file, and awk '{print $1}' extracts just the hash portion (since sha256sum‘s output includes both the hash and the filename).

Real-World Use Case: Batch Downloading a List of URLs

I often need to download dozens of files listed in a text file, one URL per line:

#!/usr/bin/env bash
set -euo pipefail

URL_LIST="urls.txt"
DEST_DIR="downloads"

mkdir -p "$DEST_DIR"

while IFS= read -r url; do
    [ -z "$url" ] && continue
    filename=$(basename "$url")
    echo "Downloading $filename..."
    curl -fSL -o "${DEST_DIR}/${filename}" "$url" || echo "Failed: $url" >> failed_downloads.txt
done < "$URL_LIST"

echo "Batch download complete."

This pattern is useful for mirroring assets, pulling multiple dataset files, or grabbing a set of release artifacts across platforms.

Automation Example: Nightly Download and Extraction

Here’s a cron-driven script I use to pull the latest nightly build of a tool and extract it automatically:

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com/nightly/latest.tar.gz"
DEST="/opt/nightly-build"

TMP_FILE=$(mktemp)
curl -fSL -o "$TMP_FILE" "$URL"

mkdir -p "$DEST"
tar -xzf "$TMP_FILE" -C "$DEST" --strip-components=1
rm -f "$TMP_FILE"

echo "Nightly build updated in $DEST"

Scheduled nightly via:

0 2 * * * /usr/local/bin/update_nightly.sh >> /var/log/nightly_update.log 2>&1

Security Considerations

• Always use HTTPS URLs where possible. Downloading over plain HTTP means the content can be tampered with in transit.
• Verify checksums or signatures whenever the source provides them, especially for anything executable. A checksum confirms integrity but not authenticity, so for critical software prefer GPG signature verification if it’s offered.
• Avoid piping downloads directly into a shell, a pattern like curl https://example.com/install.sh | bash. This executes remote code with no chance to inspect it first. When I must use this pattern, I download the script first, read through it, then execute it manually.
• Set explicit timeouts (--connect-timeout, --max-time) so a malicious or broken server can’t hang your script indefinitely.
• Be careful with -L (follow redirects) combined with credentials in headers — curl will resend the same headers to the redirected host unless you take steps to prevent that, potentially leaking secrets to a different domain.

Optimization Tips

• Use curl --parallel (available in curl 7.66+) to download multiple files concurrently instead of looping sequentially: curl --parallel --parallel-max 5 -O url1 -O url2 -O url3
• For very large files, add --limit-rate if you need to avoid saturating your network link during business hours.
• Use wget --mirror for recursive site mirroring instead of writing custom recursive logic yourself.

Troubleshooting

• “curl: (6) Could not resolve host”: DNS resolution failed; check your network connection or /etc/resolv.conf.
• “curl: (28) Connection timed out”: the server didn’t respond within the timeout window; increase --connect-timeout or check firewall rules.
• Downloaded file is HTML instead of the expected binary: this usually means the server returned an error page (like a 404 or login redirect) that wasn’t caught because -f wasn’t used. Always add -f so curl treats HTTP error codes as failures.
• Resume doesn’t work and it restarts from zero: some servers don’t support HTTP range requests; in that case resuming isn’t possible and a fresh download is required.

Common Mistakes to Avoid

• Forgetting -f with curl, causing error pages to be saved as if they were valid files.
• Not setting a timeout, leading to scripts that hang forever on unresponsive servers.
• Skipping checksum verification for downloaded executables or installer scripts.
• Piping an unreviewed remote script directly into bash or sh.

FAQs

Which is better for scripting, curl or wget? Both are fine; curl tends to be preferred when you need more control over requests (headers, methods, auth), while wget is often simpler for straightforward file downloads and has built-in recursive support.

How do I download a file only if it’s newer than my local copy? With wget, use -N (timestamping): wget -N https://example.com/file.zip. With curl, use -z with a reference date: curl -z file.zip -O https://example.com/file.zip.

How do I download through a proxy? Both tools respect the http_proxy and https_proxy environment variables, or you can pass --proxy explicitly to curl.

Can I show a progress bar with curl? Yes, curl shows a progress meter by default unless you use -s (silent). Use --progress-bar for a simpler, cleaner bar.

Summary

Downloading files reliably from Bash comes down to a handful of habits: prefer HTTPS, always check for HTTP errors with -f, add timeouts so scripts never hang indefinitely, verify checksums for anything important, and support resuming for large files. Once these patterns became muscle memory, my download scripts went from “usually works” to something I can genuinely trust in unattended automation.

References

• curl documentation: https://curl.se/docs/manpage.html
• GNU Wget manual: https://www.gnu.org/software/wget/manual/wget.html
• GNU Coreutils sha256sum documentation: https://www.gnu.org/software/coreutils/manual/html_node/sha2-utilities.html