Offensive Security Researcher & Quantum Cryptography Analyst

How to Create a Bash File Transfer Utility

Abdul Wahab Junaid
How to Create a Bash File Transfer Utility

How to Create a Bash File Transfer Utility

A few years back I was constantly moving files between three or four servers I managed — a staging box, a backup server, and a couple of client environments. I got tired of typing out long scp commands with different hostnames, ports, and paths every time, so I built myself a small file transfer utility in Bash. It’s saved me an enormous amount of typing and, more importantly, mistakes. In this article, I’ll show you exactly how to build one yourself, from a simple wrapper around scp to a more capable tool with retries, logging, and integrity checks.

What This Tool Actually Does

At its core, this is a Bash script that wraps existing transfer tools — scp, rsync, and optionally curl for HTTP-based transfers — into one consistent command-line interface. Instead of remembering different syntax for each tool, I type one command, and the script figures out the right method underneath.

Prerequisites

Check what you have installed:

which scp rsync curl

Understanding the Underlying Tools

Before building the wrapper, it helps to know what each tool actually does:

Step 1: A Basic Wrapper Script

Let’s start with something simple. Save this as filexfer.sh:

#!/usr/bin/env bash
set -euo pipefail

usage() {
    echo "Usage: $0 <source> <user@host:destination>"
    exit 1
}

if [ "$#" -ne 2 ]; then
    usage
fi

SOURCE="$1"
DEST="$2"

echo "Transferring '$SOURCE' to '$DEST' using rsync..."
rsync -avz --progress "$SOURCE" "$DEST"
echo "Transfer complete."

Make it executable and test it:

chmod +x filexfer.sh
./filexfer.sh ./report.pdf user@192.168.1.50:/home/user/documents/

Explaining the rsync Flags

Step 2: Adding Retry Logic

Networks are unreliable, and I got tired of transfers failing halfway through a large file with no automatic recovery. Here’s an improved version with retries:

#!/usr/bin/env bash
set -euo pipefail

SOURCE="$1"
DEST="$2"
MAX_RETRIES=3
ATTEMPT=1

while [ "$ATTEMPT" -le "$MAX_RETRIES" ]; do
    echo "Attempt $ATTEMPT of $MAX_RETRIES..."
    if rsync -avz --partial --progress "$SOURCE" "$DEST"; then
        echo "Transfer succeeded."
        exit 0
    else
        echo "Attempt $ATTEMPT failed. Retrying in 5 seconds..."
        ATTEMPT=$((ATTEMPT + 1))
        sleep 5
    fi
done

echo "Transfer failed after $MAX_RETRIES attempts."
exit 1

The --partial flag is the key addition here — it tells rsync to keep partially transferred files instead of deleting them on failure, so the next retry attempt can resume from where it left off rather than starting over.

Step 3: Verifying File Integrity After Transfer

I don’t fully trust any transfer until I’ve verified the destination file matches the source exactly. Checksums make this trivial:

#!/usr/bin/env bash
set -euo pipefail

SOURCE="$1"
DEST_HOST="$2"
DEST_PATH="$3"

LOCAL_HASH=$(sha256sum "$SOURCE" | awk '{print $1}')

rsync -avz "$SOURCE" "${DEST_HOST}:${DEST_PATH}"

REMOTE_HASH=$(ssh "$DEST_HOST" "sha256sum ${DEST_PATH}$(basename "$SOURCE")" | awk '{print $1}')

if [ "$LOCAL_HASH" == "$REMOTE_HASH" ]; then
    echo "Integrity check passed. Hashes match."
else
    echo "WARNING: Hash mismatch! Transfer may be corrupted."
    exit 1
fi

This script computes a SHA-256 hash locally, transfers the file, then computes the hash again on the remote machine over SSH and compares the two. If they don’t match, something went wrong during transfer, and I’d rather know immediately than discover a corrupted file weeks later.

Step 4: Logging Every Transfer

For anything running unattended, I always add logging:

#!/usr/bin/env bash
set -euo pipefail

LOGFILE="/var/log/filexfer.log"
SOURCE="$1"
DEST="$2"

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOGFILE"
}

log "Starting transfer: $SOURCE -> $DEST"

if rsync -avz --partial "$SOURCE" "$DEST" >> "$LOGFILE" 2>&1; then
    log "Transfer succeeded."
else
    log "Transfer FAILED."
    exit 1
fi

The tee -a command writes output to both the terminal and the log file simultaneously, which is handy when you’re watching a transfer live but still want a permanent record.

Step 5: Adding a Progress Notification

For long transfers I sometimes want a desktop or terminal bell notification when it’s done:

if rsync -avz "$SOURCE" "$DEST"; then
    echo -e "\a"  # terminal bell
    echo "Transfer complete: $(date)"
fi

How It All Works Internally

When you call rsync -avz source dest, rsync doesn’t just blindly copy bytes. It builds a list of files on both ends, compares metadata (size, modification time), and only transfers the parts that have actually changed using a rolling checksum algorithm. This is why re-running the same rsync command on a large directory is fast the second time — it skips files that are already identical. scp, by contrast, always transfers the full file regardless of whether it’s already present at the destination, which is why I default to rsync in most of my scripts now.

Real-World Use Cases

Automation Example: Cron-Based Nightly Sync

0 1 * * * /home/user/scripts/filexfer.sh /home/user/data/ backupuser@backup.example.com:/backups/data/ >> /var/log/nightly_sync.log 2>&1

This runs the sync every night at 1 AM and appends both stdout and stderr to a log file for later review.

Best Practices

Security Considerations

Optimization Tips

Troubleshooting Common Issues

“Connection refused” errors — Verify SSH is running on the destination and that the port matches (-p flag for rsync/scp if using a non-default SSH port).

Transfer hangs indefinitely — Check for firewall rules blocking the connection, or add -e "ssh -o ConnectTimeout=10" to rsync to force a timeout instead of hanging forever.

Permission denied on destination — Confirm the remote user has write access to the destination directory, and double check SSH key permissions (chmod 600 on your private key).

Partial or corrupted files after interruption — Make sure you’re using --partial with rsync, and always run an integrity check (as shown in Step 3) after any transfer you can’t afford to have fail silently.

Common Mistakes to Avoid

Frequently Asked Questions

Should I use rsync or scp for a one-off transfer? For a single small file, either works fine. For anything repeated, large, or important enough to need resumability, rsync is the better choice.

Can this utility transfer files between two remote machines, not just to/from my local machine? Yes, as long as one of the two remote machines has SSH access to the other, you can run the script from a third machine with rsync -avz user1@hostA:/path user2@hostB:/path, though this routes data through wherever the script itself is executed unless you use --rsync-path tricks.

Does this work over the public internet, or only local networks? It works over the public internet as long as SSH access is properly configured and the firewall allows the connection, though you may want to add bandwidth limiting for large transfers.

What if I need to transfer files to a cloud storage bucket instead of a server? For services like Amazon S3, you’d typically use a dedicated CLI tool like aws s3 cp rather than rsync/scp, since object storage doesn’t expose an SSH interface.

Summary

Building your own Bash file transfer utility isn’t about replacing dedicated tools — it’s about creating a consistent, reliable interface around them so you stop worrying about remembering flags and syntax differences. Starting from a simple rsync wrapper, I’ve shown you how to add retry logic for unreliable networks, integrity verification with checksums, logging for unattended runs, and the security practices that keep the whole thing safe to automate. Once you have this in place, moving files between machines becomes one command instead of a small research project every time.

References

Exit mobile version