I’ve relied on wget for so long that it’s basically muscle memory at this point — mirror a site, grab a tarball for a build, resume a download that died halfway through a bad connection. It’s one of those tools that seems simple on the surface but has a genuinely deep feature set once you start digging into its options. Let me walk you through what I’ve learned using it across countless servers and scripts.
What wget Is and Why It Still Matters
wget is a non-interactive network downloader, meaning it’s designed to run without a human sitting there watching it — perfect for scripts, cron jobs, and unattended automation. It supports HTTP, HTTPS, and FTP, handles redirects, retries on failure, can resume partial downloads, and can even recursively mirror entire websites. Despite the rise of curl as its more flexible sibling, wget remains extremely common specifically because its defaults are so well suited to “just download this file reliably,” and its recursive mirroring capability has no direct equivalent in curl without extra tooling.
I verified the version on my own system while writing this:
wget --version
GNU Wget 1.21.4 built on linux-gnu.
Basic Syntax
wget [options] [URL...]
The simplest possible use is just:
wget https://example.com/file.tar.gz
This downloads the file into your current directory, preserving the original filename, and prints a progress bar with transfer speed, ETA, and completion percentage as it goes.
I tested this directly:
wget -q https://raw.githubusercontent.com/git/git/master/README.md -O test.md
That completed successfully and pulled down a real file, confirming basic functionality works as expected.
Core Parameters
Here are the options I use constantly, grouped by what they actually accomplish.
Output Control
wget -O newname.tar.gz https://example.com/file.tar.gz
-O (--output-document) saves the downloaded content under a specific filename rather than the one implied by the URL. This is essential when a URL doesn’t end in a sensible filename, or when you want a predictable name for scripting purposes.
wget -P /opt/downloads https://example.com/file.tar.gz
-P (--directory-prefix) sets the directory to save into, without renaming the file itself.
Quiet and Verbose Modes
wget -q https://example.com/file.tar.gz # completely silent
wget -v https://example.com/file.tar.gz # verbose (default level, actually)
wget --no-verbose https://example.com/file.tar.gz # brief, single-line summary
I use -q constantly inside scripts where I don’t want progress bar noise cluttering log output, often combined with explicit error checking on the exit code instead.
Resuming Interrupted Downloads
wget -c https://example.com/largefile.iso
-c (--continue) is one of the most valuable flags in the entire tool. If a download was interrupted partway, this resumes from where it left off rather than starting over, which matters enormously for large files over unreliable connections.
Retry Behavior
wget --tries=10 --wait=5 https://example.com/file.tar.gz
--tries sets how many attempts to make before giving up, and --wait sets the delay between retries in seconds. I combine these two constantly for downloads over flaky connections, or when hitting a server that occasionally throttles requests.
wget --tries=0 https://example.com/file.tar.gz
Setting --tries=0 means retry indefinitely, which I use in long-running unattended download scripts where I’d rather wait than fail.
Limiting Bandwidth
wget --limit-rate=500k https://example.com/file.tar.gz
Caps the download speed, useful when I don’t want a big download to saturate a shared network link during business hours.
Background Downloads
wget -b https://example.com/file.tar.gz
-b (--background) forks the process into the background immediately and logs output to wget-log in the current directory, letting you close the terminal session without killing the download.
User Agent and Headers
Some servers reject requests that don’t look like they’re coming from a real browser:
wget --user-agent="Mozilla/5.0 (X11; Linux x86_64)" https://example.com/file.zip
You can also add arbitrary custom headers:
wget --header="Authorization: Bearer mytoken123" https://api.example.com/protected-file
Authentication
For HTTP basic authentication:
wget --user=myuser --password=mypass https://example.com/protected/file.zip
For FTP:
wget --ftp-user=myuser --ftp-password=mypass ftp://ftp.example.com/file.zip
I generally avoid putting passwords directly on the command line in shared environments, since command-line arguments are visible to other users via ps. Instead, I use a .wgetrc or .netrc file with restricted permissions, or pass credentials via environment variables read by a wrapper script.
Recursive Downloading / Mirroring
This is where wget really distinguishes itself from curl. To download an entire directory structure or mirror a site:
wget -r -np -k https://example.com/docs/
-r(--recursive) follows links recursively-np(--no-parent) prevents it from climbing up to parent directories outside the target path-k(--convert-links) rewrites links in downloaded HTML to point to the local copies, making the mirrored site browsable offline
For a full site mirror suitable for offline browsing:
wget --mirror --convert-links --page-requisites --no-parent https://example.com/
--page-requisites additionally pulls in everything needed to properly display the page — images, CSS, JavaScript — even if those individual assets wouldn’t otherwise be picked up by the link-following logic.
I limit recursion depth when I don’t want to accidentally pull down an entire massive site:
wget -r -l 2 https://example.com/
-l (--level) sets maximum recursion depth; -l 2 means “follow links two levels deep and stop.”
Restricting File Types During Recursive Downloads
wget -r -A "pdf,zip" https://example.com/files/
-A (--accept) restricts recursive downloads to matching file extensions, useful when mirroring a directory but only caring about certain file types. The inverse, -R (--reject), excludes specific extensions instead.
Downloading Multiple URLs from a List
wget -i urls.txt
-i reads a list of URLs from a file, one per line, and downloads each in turn — extremely useful for batch downloading a set of files gathered from another script or generated list.
Checking Without Downloading
wget --spider https://example.com/file.tar.gz
--spider mode checks that a URL exists and is reachable without actually downloading the content, printing the HTTP response headers. I use this in monitoring scripts to check that a resource is still available without wasting bandwidth pulling the whole file.
SSL/TLS Options
wget --no-check-certificate https://self-signed.example.com/file.zip
I use --no-check-certificate sparingly and only against internal servers I control with self-signed certificates I trust — never against public internet resources, since it defeats the entire point of certificate validation and opens the door to man-in-the-middle tampering.
How wget Handles Redirects and Protocols Internally
wget follows HTTP redirects (3xx status codes) automatically by default, up to a limit controlled by --max-redirect (default 20). It reads the Location header from the redirect response and issues a new request to that target, which is why a single wget https://short.url/xyz command can transparently follow through several hops to the final file, printing each hop if run without -q.
For HTTPS, wget performs a standard TLS handshake, validating the server’s certificate chain against the system’s trusted CA bundle (typically /etc/ssl/certs/ca-certificates.crt on Debian-family systems) unless certificate checking is explicitly disabled.
Configuration File: .wgetrc
Rather than repeating the same flags on every invocation, I keep persistent defaults in ~/.wgetrc:
tries = 5
wait = 3
timeout = 30
user_agent = Mozilla/5.0 (X11; Linux x86_64)
System-wide defaults live in /etc/wgetrc, applied before the user’s own file, letting you set organization-wide download policies (like default proxy settings) that individual users can still override.
Practical Real-World Examples
Downloading and verifying a checksum in one script:
#!/bin/bash
URL="https://example.com/software-1.0.tar.gz"
EXPECTED_SHA256="abc123..."
wget -q "$URL" -O software.tar.gz
ACTUAL_SHA256=$(sha256sum software.tar.gz | awk '{print $1}')
if [[ "$ACTUAL_SHA256" == "$EXPECTED_SHA256" ]]; then
echo "Checksum verified, proceeding with install"
else
echo "Checksum mismatch! Aborting." >&2
exit 1
fi
Downloading through a proxy:
export http_proxy="http://proxy.example.com:3128"
export https_proxy="http://proxy.example.com:3128"
wget https://example.com/file.tar.gz
Scheduled nightly mirror via cron:
# crontab entry
0 2 * * * /usr/bin/wget --mirror --no-parent -q -P /backup/mirror https://internal-docs.example.com/
Comparison with curl
I get asked constantly which one to use, so here’s my honest take. curl is more of a general-purpose data transfer tool, supports more protocols out of the box, has a more scriptable API-friendly design (particularly with -d for POST data and easy header manipulation), and is the better choice when you’re interacting with REST APIs. wget wins clearly for straightforward recursive site mirroring, resumable downloads by default with simpler syntax, and unattended background downloading with -b. In practice I use wget for “just get me this file/directory reliably” and curl for “I’m scripting an interaction with an API.”
Troubleshooting Common Issues
“Unable to establish SSL connection” — usually a CA certificate bundle issue; verify with wget --ca-certificate=/path/to/cert.pem pointing at a known-good bundle, or check if your system’s certificate store needs updating via your package manager.
Downloads stall or time out on large files — adjust --timeout and --read-timeout, and enable -c so a subsequent retry resumes rather than restarting.
Recursive mirror pulls way more than expected — double check -np is set and consider adding -l to cap recursion depth; also verify robots.txt isn’t being silently obeyed in a way you don’t want (disable with -e robots=off if you have legitimate reason to ignore it).
403 Forbidden despite the file being publicly accessible in a browser — often a User-Agent block; try setting --user-agent to mimic a standard browser string.
Security Considerations
Never disable certificate verification against public servers. Be cautious with -i urls.txt files sourced from untrusted input, since a malicious list could redirect downloads to unexpected destinations or exhaust disk space via extremely large files — I always add --quota when processing untrusted URL lists:
wget --quota=1g -i urls.txt
This stops the whole batch once total downloaded data crosses the specified limit, preventing a runaway download from filling a disk.
Compatibility Across Distributions
wget is part of the GNU project and ships essentially identically across all major Linux distributions — Debian, Ubuntu, RHEL, Fedora, CentOS, Arch, openSUSE — all provide the same GNU Wget implementation with the same flag set, differing only in packaged version number. It’s also available on macOS via Homebrew and on Windows via WSL or standalone builds, though it’s not installed by default on macOS or Windows the way it typically is on Linux server distributions.
Timestamping and Conditional Downloads
wget supports checking whether a remote file has actually changed before re-downloading it, useful for periodic sync scripts that shouldn’t waste bandwidth re-pulling something unchanged:
wget -N https://example.com/data.csv
-N (--timestamping) compares the remote file’s last-modified timestamp against the local copy, and only downloads if the remote version is genuinely newer. I use this in cron-driven scripts that periodically sync a dataset or configuration file from a remote source, where re-downloading an unchanged multi-gigabyte file every run would be wasteful.
Mirroring with Timestamps and Incremental Updates
Combined with recursive mirroring, timestamping makes for efficient incremental site mirrors that only pull what’s actually changed since the last run:
wget --mirror -N --no-parent https://example.com/releases/
This is meaningfully different from a full re-mirror each time — subsequent runs only download files that are new or have been updated since the previous mirror pass, dramatically reducing bandwidth and time for a scheduled, repeated mirroring job.
Setting Timeouts Explicitly
By default, wget will wait a fairly long time on a stalled connection before giving up, which isn’t always what you want in an automated context where a hung download should fail fast rather than block a script indefinitely:
wget --timeout=15 --tries=3 https://example.com/file.tar.gz
--timeout sets a combined value for DNS lookup, connection, and read timeouts in one shot (or you can control each independently with --dns-timeout, --connect-timeout, and --read-timeout respectively). I set explicit, fairly aggressive timeouts in any unattended script specifically so a single unresponsive server doesn’t stall an entire automated pipeline waiting on a default timeout that might be minutes long.
Summary
wget remains one of the most dependable tools in the Linux toolkit precisely because its defaults handle the common cases so well — resumable, retryable, quiet-capable downloads that work great in scripts and cron jobs, plus a genuinely powerful recursive mirroring mode that has no simple equivalent in most alternatives. Once you know the handful of flags that matter most — -O, -c, -q, --tries, -r with -np — you can handle the overwhelming majority of real-world download automation tasks without ever needing to reach for anything heavier.
References
- GNU Wget official manual:
man 1 wget - GNU Wget project documentation (gnu.org/software/wget/manual/)
.wgetrcconfiguration reference in the GNU Wget manual- Distribution package documentation for
wget(Debian, RHEL package repositories)