Offensive Security Researcher & Quantum Cryptography Analyst

How to Create a Bash File Sync Tool

Abdul Wahab Junaid
How to Create a Bash File Sync Tool

How to Create a Bash File Sync Tool

I manage a handful of machines — a home server, a laptop, a couple of VPS instances — and keeping certain folders in sync between them used to be a manual, error-prone chore. Eventually I got fed up and built a Bash file sync tool on top of rsync that handles local syncing, remote syncing over SSH, dry runs, and logging. In this article I’ll take you through the whole build, explaining exactly what each piece does so you can adapt it to your own setup.

Why rsync Is the Right Foundation

Bash itself doesn’t need to reimplement file syncing — rsync already does the hard part brilliantly: it only transfers the differences between files (using a clever delta algorithm), preserves permissions and timestamps, and supports both local and remote (SSH) destinations. Our job is to wrap it in a script that’s easy to invoke, safe by default, and repeatable.

Prerequisites

sudo apt install rsync openssh-client

For remote syncing, make sure SSH key-based authentication is set up so the script can run non-interactively:

ssh-keygen -t ed25519 -C "sync-tool"
ssh-copy-id user@remote-host

Step 1: A Basic Local Sync Script

#!/usr/bin/env bash
set -euo pipefail

SOURCE="${1:-}"
DEST="${2:-}"

if [[ -z "$SOURCE" || -z "$DEST" ]]; then
    echo "Usage: $0 <source-dir> <destination-dir>"
    exit 1
fi

rsync -av --progress "$SOURCE"/ "$DEST"/

Understanding the Core Command

Step 2: Adding a Dry-Run Mode

Before syncing anything for real, I like to preview exactly what will change:

DRY_RUN="${3:-}"

if [[ "$DRY_RUN" == "--dry-run" ]]; then
    rsync -av --dry-run "$SOURCE"/ "$DEST"/
    echo "Dry run complete. No files were changed."
else
    rsync -av --progress "$SOURCE"/ "$DEST"/
fi

--dry-run tells rsync to calculate and display everything it would do without actually touching any files — it’s one of the most valuable safety features available for any sync or deletion workflow.

Step 3: Supporting Remote Sync Over SSH

Now let’s extend the script to support remote destinations, detected automatically by the presence of a colon (:) in the destination argument, which is the standard rsync/scp convention for remote paths:

#!/usr/bin/env bash
set -euo pipefail

SOURCE="${1:-}"
DEST="${2:-}"
DRY_RUN="${3:-}"

if [[ -z "$SOURCE" || -z "$DEST" ]]; then
    echo "Usage: $0 <source-dir> <destination-dir-or-user@host:path> [--dry-run]"
    exit 1
fi

RSYNC_OPTS=(-av --progress --human-readable)

if [[ "$DRY_RUN" == "--dry-run" ]]; then
    RSYNC_OPTS+=(--dry-run)
fi

if [[ "$DEST" == *:* ]]; then
    echo "Detected remote destination. Syncing over SSH..."
    rsync "${RSYNC_OPTS[@]}" -e ssh "$SOURCE"/ "$DEST"/
else
    rsync "${RSYNC_OPTS[@]}" "$SOURCE"/ "$DEST"/
fi

Breaking This Down

Step 4: Adding Exclusions and Deletion Sync

Real syncing setups usually need to ignore certain files and, sometimes, mirror deletions too:

EXCLUDES=(".git/" "node_modules/" "*.tmp" ".DS_Store")
EXCLUDE_ARGS=()
for pattern in "${EXCLUDES[@]}"; do
    EXCLUDE_ARGS+=(--exclude="$pattern")
done

MIRROR="${4:-}"
if [[ "$MIRROR" == "--mirror" ]]; then
    RSYNC_OPTS+=(--delete)
fi

rsync "${RSYNC_OPTS[@]}" "${EXCLUDE_ARGS[@]}" "$SOURCE"/ "$DEST"/

--delete makes rsync remove files from the destination that no longer exist in the source, turning a one-way sync into a true mirror. This is powerful but also the single most dangerous flag in rsync — always test with --dry-run --delete together first.

Step 5: Logging Every Sync

For anything running unattended (like a cron job), you want a record of what happened:

LOG_DIR="$HOME/.local/share/sync-tool/logs"
mkdir -p "$LOG_DIR"
LOG_FILE="$LOG_DIR/sync-$(date +%Y%m%d-%H%M%S).log"

rsync "${RSYNC_OPTS[@]}" "${EXCLUDE_ARGS[@]}" "$SOURCE"/ "$DEST"/ | tee "$LOG_FILE"

echo "Log saved to $LOG_FILE"

tee "$LOG_FILE" is a genuinely useful Bash idiom — it takes standard input and writes it to both the terminal and a file simultaneously, so you get live feedback while also keeping a permanent record.

Full Combined Script

#!/usr/bin/env bash
set -euo pipefail

SOURCE="${1:-}"
DEST="${2:-}"
shift 2 || true

DRY_RUN=false
MIRROR=false
for arg in "$@"; do
    case "$arg" in
        --dry-run) DRY_RUN=true ;;
        --mirror) MIRROR=true ;;
    esac
done

if [[ -z "$SOURCE" || -z "$DEST" ]]; then
    echo "Usage: $0 <source> <dest> [--dry-run] [--mirror]"
    exit 1
fi

EXCLUDES=(".git/" "node_modules/" "*.tmp" ".DS_Store")
EXCLUDE_ARGS=()
for pattern in "${EXCLUDES[@]}"; do
    EXCLUDE_ARGS+=(--exclude="$pattern")
done

RSYNC_OPTS=(-av --human-readable --progress)
$DRY_RUN && RSYNC_OPTS+=(--dry-run)
$MIRROR && RSYNC_OPTS+=(--delete)

LOG_DIR="$HOME/.local/share/sync-tool/logs"
mkdir -p "$LOG_DIR"
LOG_FILE="$LOG_DIR/sync-$(date +%Y%m%d-%H%M%S).log"

if [[ "$DEST" == *:* ]]; then
    rsync "${RSYNC_OPTS[@]}" "${EXCLUDE_ARGS[@]}" -e ssh "$SOURCE"/ "$DEST"/ | tee "$LOG_FILE"
else
    rsync "${RSYNC_OPTS[@]}" "${EXCLUDE_ARGS[@]}" "$SOURCE"/ "$DEST"/ | tee "$LOG_FILE"
fi

echo "Sync complete. Log: $LOG_FILE"

shift 2 || true removes the first two positional arguments (source and dest) so we can loop through any remaining flags cleanly; the || true prevents set -e from killing the script if there happen to be fewer than two extra arguments.

Real-World Use Cases

Automating Sync with Cron

*/30 * * * * /usr/local/bin/sync.sh /home/user/projects user@backup-host:/backups/projects >> /var/log/sync-cron.log 2>&1

This runs the sync every 30 minutes and appends both standard output and errors to a persistent log file.

Security Considerations

Optimization Tips

Troubleshooting

Common Mistakes to Avoid

Frequently Asked Questions

Is rsync safe to interrupt mid-transfer? Yes, especially with --partial — a subsequent run will resume rather than starting over, since rsync only transfers differences.

Can I sync in both directions (bidirectional sync)? Plain rsync is one-directional. For true bidirectional sync with conflict resolution, look into tools like unison, though you can approximate simple two-way sync by running this script in both directions carefully.

How is this different from just using scp? scp copies everything every time; rsync only transfers what’s changed, making repeated syncs dramatically faster, especially on large directory trees.

Summary

We built a flexible Bash sync tool on top of rsync that supports local and remote destinations, dry runs, exclusions, mirroring with deletion, and logging — all through a single consistent interface. The core insight is that Bash doesn’t need to reinvent syncing logic; it just needs to wrap a powerful tool like rsync in sensible defaults and safety rails so you can trust it to run unattended.

References

Exit mobile version