How to Create a Bash File Transfer Utility

A few years back I was constantly moving files between three or four servers I managed — a staging box, a backup server, and a couple of client environments. I got tired of typing out long scp commands with different hostnames, ports, and paths every time, so I built myself a small file transfer utility in Bash. It’s saved me an enormous amount of typing and, more importantly, mistakes. In this article, I’ll show you exactly how to build one yourself, from a simple wrapper around scp to a more capable tool with retries, logging, and integrity checks.

What This Tool Actually Does

At its core, this is a Bash script that wraps existing transfer tools — scp, rsync, and optionally curl for HTTP-based transfers — into one consistent command-line interface. Instead of remembering different syntax for each tool, I type one command, and the script figures out the right method underneath.

Prerequisites

• Bash 4+ on Linux or macOS (or WSL on Windows).
• scp and rsync installed (usually present by default on most distros).
• SSH access configured between the machines you want to transfer files between, ideally with key-based authentication rather than passwords.

Check what you have installed:

which scp rsync curl

Understanding the Underlying Tools

Before building the wrapper, it helps to know what each tool actually does:

• scp copies files over SSH. It’s simple and universally available, but doesn’t resume interrupted transfers.
• rsync synchronizes files and directories, and crucially, it can resume partial transfers and only copy the differences between source and destination, which makes it far more efficient for repeated transfers.
• curl is useful when transferring to or from an HTTP/HTTPS/FTP endpoint rather than another server you control via SSH.

Step 1: A Basic Wrapper Script

Let’s start with something simple. Save this as filexfer.sh:

#!/usr/bin/env bash
set -euo pipefail

usage() {
    echo "Usage: $0 <source> <user@host:destination>"
    exit 1
}

if [ "$#" -ne 2 ]; then
    usage
fi

SOURCE="$1"
DEST="$2"

echo "Transferring '$SOURCE' to '$DEST' using rsync..."
rsync -avz --progress "$SOURCE" "$DEST"
echo "Transfer complete."

Make it executable and test it:

chmod +x filexfer.sh
./filexfer.sh ./report.pdf user@192.168.1.50:/home/user/documents/

Explaining the rsync Flags

• -a (archive mode) preserves permissions, timestamps, symbolic links, and ownership where possible, which is what you want for most file transfers.
• -v (verbose) prints details about what’s being transferred.
• -z compresses data during transfer, which speeds things up significantly over slower connections.
• --progress shows a live progress bar for each file, so you’re not staring at a blank terminal wondering if anything is happening.

Step 2: Adding Retry Logic

Networks are unreliable, and I got tired of transfers failing halfway through a large file with no automatic recovery. Here’s an improved version with retries:

#!/usr/bin/env bash
set -euo pipefail

SOURCE="$1"
DEST="$2"
MAX_RETRIES=3
ATTEMPT=1

while [ "$ATTEMPT" -le "$MAX_RETRIES" ]; do
    echo "Attempt $ATTEMPT of $MAX_RETRIES..."
    if rsync -avz --partial --progress "$SOURCE" "$DEST"; then
        echo "Transfer succeeded."
        exit 0
    else
        echo "Attempt $ATTEMPT failed. Retrying in 5 seconds..."
        ATTEMPT=$((ATTEMPT + 1))
        sleep 5
    fi
done

echo "Transfer failed after $MAX_RETRIES attempts."
exit 1

The --partial flag is the key addition here — it tells rsync to keep partially transferred files instead of deleting them on failure, so the next retry attempt can resume from where it left off rather than starting over.

Step 3: Verifying File Integrity After Transfer

I don’t fully trust any transfer until I’ve verified the destination file matches the source exactly. Checksums make this trivial:

#!/usr/bin/env bash
set -euo pipefail

SOURCE="$1"
DEST_HOST="$2"
DEST_PATH="$3"

LOCAL_HASH=$(sha256sum "$SOURCE" | awk '{print $1}')

rsync -avz "$SOURCE" "${DEST_HOST}:${DEST_PATH}"

REMOTE_HASH=$(ssh "$DEST_HOST" "sha256sum ${DEST_PATH}$(basename "$SOURCE")" | awk '{print $1}')

if [ "$LOCAL_HASH" == "$REMOTE_HASH" ]; then
    echo "Integrity check passed. Hashes match."
else
    echo "WARNING: Hash mismatch! Transfer may be corrupted."
    exit 1
fi

This script computes a SHA-256 hash locally, transfers the file, then computes the hash again on the remote machine over SSH and compares the two. If they don’t match, something went wrong during transfer, and I’d rather know immediately than discover a corrupted file weeks later.

Step 4: Logging Every Transfer

For anything running unattended, I always add logging:

#!/usr/bin/env bash
set -euo pipefail

LOGFILE="/var/log/filexfer.log"
SOURCE="$1"
DEST="$2"

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOGFILE"
}

log "Starting transfer: $SOURCE -> $DEST"

if rsync -avz --partial "$SOURCE" "$DEST" >> "$LOGFILE" 2>&1; then
    log "Transfer succeeded."
else
    log "Transfer FAILED."
    exit 1
fi

The tee -a command writes output to both the terminal and the log file simultaneously, which is handy when you’re watching a transfer live but still want a permanent record.

Step 5: Adding a Progress Notification

For long transfers I sometimes want a desktop or terminal bell notification when it’s done:

if rsync -avz "$SOURCE" "$DEST"; then
    echo -e "\a"  # terminal bell
    echo "Transfer complete: $(date)"
fi

How It All Works Internally

When you call rsync -avz source dest, rsync doesn’t just blindly copy bytes. It builds a list of files on both ends, compares metadata (size, modification time), and only transfers the parts that have actually changed using a rolling checksum algorithm. This is why re-running the same rsync command on a large directory is fast the second time — it skips files that are already identical. scp, by contrast, always transfers the full file regardless of whether it’s already present at the destination, which is why I default to rsync in most of my scripts now.

Real-World Use Cases

• Nightly backup sync from a production server to an off-site backup machine, only transferring the changed files each night.
• Deploying build artifacts from a CI pipeline to a staging or production server.
• Client file delivery, where I need integrity verification to guarantee a client receives an uncorrupted deliverable.
• Syncing configuration files across multiple servers in a small fleet without a full configuration management system.

Automation Example: Cron-Based Nightly Sync

0 1 * * * /home/user/scripts/filexfer.sh /home/user/data/ backupuser@backup.example.com:/backups/data/ >> /var/log/nightly_sync.log 2>&1

This runs the sync every night at 1 AM and appends both stdout and stderr to a log file for later review.

Best Practices

• Always use SSH key-based authentication instead of passwords for automated transfers — this avoids storing plaintext credentials anywhere in the script.
• Use rsync over scp whenever you expect to re-run the same transfer, since it avoids re-copying unchanged data.
• Log every transfer, especially for anything running unattended via cron.
• Validate the destination path exists before transferring, to avoid rsync silently creating unexpected directory structures.
• Test your retry logic by intentionally killing the network mid-transfer, so you know it behaves as expected under real failure conditions.

Security Considerations

• Never hardcode SSH passwords into a script. Use SSH keys, and restrict the key’s permissions on the remote side using authorized_keys options like command= and no-port-forwarding if the key is single-purpose.
• Restrict permissions on any log files that might contain file paths or hostnames that reveal sensitive infrastructure details.
• If transferring over an untrusted network, always use SSH-based tools (scp/rsync over SSH) rather than plain FTP, which transmits credentials and data unencrypted.
• Consider using rsync‘s --chmod option to explicitly control permissions on transferred files, rather than relying on defaults that might be too permissive.

Optimization Tips

• For transferring many small files, consider archiving them into a single tarball first (tar czf) since the overhead of establishing a connection per file can outweigh the benefit of individual file transfer.
• Use rsync‘s --bwlimit flag to cap bandwidth usage during business hours, so large transfers don’t saturate your network link.
• For transfers between machines on the same local network, disabling compression (-z) can actually speed things up, since compression overhead outweighs the benefit when bandwidth isn’t the bottleneck.

Troubleshooting Common Issues

“Connection refused” errors — Verify SSH is running on the destination and that the port matches (-p flag for rsync/scp if using a non-default SSH port).

Transfer hangs indefinitely — Check for firewall rules blocking the connection, or add -e "ssh -o ConnectTimeout=10" to rsync to force a timeout instead of hanging forever.

Permission denied on destination — Confirm the remote user has write access to the destination directory, and double check SSH key permissions (chmod 600 on your private key).

Partial or corrupted files after interruption — Make sure you’re using --partial with rsync, and always run an integrity check (as shown in Step 3) after any transfer you can’t afford to have fail silently.

Common Mistakes to Avoid

• Using scp for repeated large transfers instead of rsync, wasting bandwidth and time re-copying unchanged data.
• Forgetting trailing slashes on rsync source/destination paths, which changes whether the directory itself or just its contents get copied.
• Not testing what happens on a failed or interrupted transfer before relying on the script for anything important.
• Storing passwords in plaintext within the script instead of using SSH keys.

Frequently Asked Questions

Should I use rsync or scp for a one-off transfer? For a single small file, either works fine. For anything repeated, large, or important enough to need resumability, rsync is the better choice.

Can this utility transfer files between two remote machines, not just to/from my local machine? Yes, as long as one of the two remote machines has SSH access to the other, you can run the script from a third machine with rsync -avz user1@hostA:/path user2@hostB:/path, though this routes data through wherever the script itself is executed unless you use --rsync-path tricks.

Does this work over the public internet, or only local networks? It works over the public internet as long as SSH access is properly configured and the firewall allows the connection, though you may want to add bandwidth limiting for large transfers.

What if I need to transfer files to a cloud storage bucket instead of a server? For services like Amazon S3, you’d typically use a dedicated CLI tool like aws s3 cp rather than rsync/scp, since object storage doesn’t expose an SSH interface.

Summary

Building your own Bash file transfer utility isn’t about replacing dedicated tools — it’s about creating a consistent, reliable interface around them so you stop worrying about remembering flags and syntax differences. Starting from a simple rsync wrapper, I’ve shown you how to add retry logic for unreliable networks, integrity verification with checksums, logging for unattended runs, and the security practices that keep the whole thing safe to automate. Once you have this in place, moving files between machines becomes one command instead of a small research project every time.

References

• rsync Official Documentation
• OpenSSH Manual Pages
• GNU Bash Reference Manual