How to Create a Bash File Splitter

I remember the first time I needed to upload a 10GB backup file to a service that capped uploads at 2GB per file. Rather than downloading some sketchy third-party tool, I wrote a Bash file splitter in about ten minutes, and I’ve been reusing and refining that script ever since. In this article, I’ll show you exactly how I approach building a reliable file splitter in Bash, from the basics all the way to a production-grade tool.

What a File Splitter Does

A file splitter takes one large file and breaks it into smaller chunks of a defined size (or count), so the pieces are easier to store, transfer, or upload. Later, those chunks can be joined back together to reconstruct the original file exactly.

Bash is a natural choice here because it ships with split, a purpose-built GNU coreutils tool designed exactly for this job. I don’t need to reinvent byte-copying logic — I just need to wrap it intelligently.

The Core Tool: split

The heart of any Bash file splitter is the split command:

split -b 100M largefile.zip part_

This breaks largefile.zip into 100MB chunks named part_aa, part_ab, part_ac, and so on.

Let’s break down what’s happening:

• -b 100M tells split to create chunks of 100 megabytes each.
• largefile.zip is the source file.
• part_ is the prefix used for naming each output chunk.
• By default, split uses alphabetical suffixes (aa, ab, ac…), but you can switch to numeric suffixes.

A Basic Bash Splitter Script

#!/bin/bash

input_file="$1"
chunk_size="$2"

split -b "$chunk_size" "$input_file" "${input_file}.part_"

echo "Split $input_file into chunks of $chunk_size"

Usage:

./splitter.sh largefile.iso 500M

This produces files like largefile.iso.part_aa, largefile.iso.part_ab, etc.

Adding Numeric Suffixes and Validation

Alphabetical suffixes work fine for a small number of parts, but once you exceed 676 chunks (26×26), you run out of two-letter combinations. I prefer numeric suffixes for anything beyond a handful of parts.

#!/bin/bash

set -euo pipefail

usage() {
    echo "Usage: $0 <input_file> <chunk_size e.g. 100M> <output_prefix>"
    exit 1
}

if [ "$#" -ne 3 ]; then
    usage
fi

input_file="$1"
chunk_size="$2"
prefix="$3"

if [ ! -f "$input_file" ]; then
    echo "Error: input file '$input_file' does not exist." >&2
    exit 1
fi

split -d -b "$chunk_size" "$input_file" "$prefix"

echo "Split complete. Parts:"
ls -lh "${prefix}"*

Internals Explained

• -d tells split to use numeric suffixes (00, 01, 02…) instead of letters.
• -b "$chunk_size" accepts human-readable sizes like 10K, 100M, 1G.
• set -euo pipefail ensures the script halts on any unexpected error rather than silently continuing.
• The final ls -lh gives immediate visual confirmation of what was created and each part’s size.

Splitting by Number of Parts Instead of Size

Sometimes you don’t care about chunk size — you just want the file divided into, say, exactly 5 equal parts. split supports this too:

split -n 5 largefile.zip part_

This divides the file into 5 roughly equal pieces regardless of the original file’s size.

Splitting by Line Count (Text Files)

If you’re working with text files, CSVs, or logs rather than binaries, splitting by line count is often more useful than splitting by byte size:

split -l 10000 access.log log_chunk_

This creates files each containing 10,000 lines, which is great for distributing text processing across multiple workers.

Generating a Manifest for Safe Reassembly

One thing I learned the hard way: if you don’t record metadata about the split, reassembling it later can be error-prone, especially across different machines. I now always generate a manifest file alongside the split:

#!/bin/bash

set -euo pipefail

input_file="$1"
chunk_size="$2"
prefix="$3"

split -d -b "$chunk_size" "$input_file" "$prefix"

sha256sum "$input_file" > "${prefix}.manifest"
echo "original_name=$input_file" >> "${prefix}.manifest"
ls "${prefix}"[0-9]* >> "${prefix}.manifest"

echo "Manifest written to ${prefix}.manifest"

This manifest records the original file’s checksum, its name, and the list of generated parts — everything needed to verify a correct reassembly later.

Real-World Use Cases

• Cloud upload limits: Splitting large files to fit under provider-imposed size caps (email attachments, some cloud storage free tiers).
• Removable media constraints: Copying a large file across FAT32-formatted USB drives that cap individual files at 4GB.
• Parallel processing: Splitting large log files or datasets by line count so multiple workers can process chunks concurrently.
• Bandwidth-limited transfers: Splitting a file so it can be resumed chunk-by-chunk if a transfer drops midway, instead of restarting from zero.
• Backup rotation: Splitting backup archives into fixed-size chunks to fit onto a rotation of physical backup disks.

Automation Example

Here’s a script I use to automatically split any file dropped into a “to-upload” folder, useful in combination with a cron job:

#!/bin/bash

set -euo pipefail

watch_dir="/data/to-upload"
chunk_size="200M"

for file in "$watch_dir"/*; do
    [ -f "$file" ] || continue
    base=$(basename "$file")
    split -d -b "$chunk_size" "$file" "${watch_dir}/${base}.part_"
    sha256sum "$file" > "${watch_dir}/${base}.manifest"
    mv "$file" "${watch_dir}/processed_${base}"
done

This keeps an “inbox” style folder clean by splitting new arrivals automatically and preserving a checksum manifest for verification.

Best Practices

• Always generate a checksum of the original file before splitting, so you can verify successful reassembly later.
• Use numeric suffixes (-d) rather than alphabetical ones for anything beyond a small number of chunks.
• Pick chunk sizes based on your actual constraint (upload limit, media size, bandwidth window), not arbitrary round numbers.
• Store a manifest listing filenames and their intended order.
• Clean up temporary chunk files once you’ve confirmed successful reassembly and upload.

Security Considerations

• Sensitive data exposure: Splitting a sensitive file into multiple pieces doesn’t provide any encryption — each part is just as readable as the original. If confidentiality matters, encrypt the file (e.g., with gpg) before splitting.
• Manifest tampering: If manifests are transferred alongside chunks over an untrusted channel, sign or checksum the manifest itself to detect tampering.
• Disk space: Splitting temporarily requires roughly double the disk space (original plus all parts) unless you delete the original immediately after — plan storage accordingly.
• Filename injection: If chunk prefixes come from user input, sanitize them to avoid unintended path traversal or command injection when used in later scripts.

Optimization Tips

• split is I/O-bound; using an SSD noticeably speeds up large splits compared to spinning disks.
• Avoid unnecessarily small chunk sizes (like 1MB) for multi-gigabyte files — this creates thousands of files, which slows down filesystem operations and directory listings.
• If you need compression as well as splitting, pipe through gzip first: gzip -c largefile.tar | split -b 100M - part_.

Troubleshooting Common Issues

Problem: “split: too many arguments” or unexpected suffix behavior. This usually happens with extremely large numbers of parts and alphabetical suffixes running out of combinations. Switch to -d and increase suffix length with -a <length>.

Problem: Reassembled file doesn’t match the original. Verify checksum of the original before splitting and after joining. Check that no parts were skipped, renamed, or corrupted during transfer.

Problem: Chunks are unexpectedly small or large. Double check the unit suffix in your -b argument — 100M and 100m behave differently depending on your split version (uppercase generally means MiB-based binary units in GNU coreutils).

Common Mistakes to Avoid

• Forgetting to record the original file’s checksum before splitting.
• Using alphabetical suffixes for very large numbers of chunks and running out of combinations.
• Not accounting for the extra disk space required during the split process.
• Splitting sensitive files without encrypting them first.

Frequently Asked Questions

Does splitting compress the file at all? No. split only divides bytes; it doesn’t compress. Combine it with gzip or xz beforehand if you want smaller output.

Can I split a file into an exact number of equal parts? Yes, use split -n <number> file prefix_.

Will splitting corrupt binary files like videos or ZIPs? No, as long as you rejoin the parts in the correct order using cat, the result is byte-for-byte identical to the original.

What’s the difference between -b and -n? -b splits by a fixed byte size per chunk (resulting in a variable number of chunks), while -n splits into a fixed number of chunks (resulting in variable chunk sizes).

Can I split text files by line instead of by byte? Yes, use split -l <lines> file prefix_ to split based on line count instead of byte size.

Summary

A Bash file splitter built around the split command is simple to set up but incredibly useful once integrated into a real workflow — whether that’s getting around upload limits, distributing processing across workers, or archiving to fixed-size media. The details that matter most are choosing the right suffix strategy, recording checksums before splitting, and picking a chunking method (-b, -n, or -l) that fits your actual use case.

References

• GNU Coreutils split manual: https://www.gnu.org/software/coreutils/manual/html_node/split-invocation.html
• Bash Reference Manual: https://www.gnu.org/software/bash/manual/bash.html
• sha256sum documentation: https://www.gnu.org/software/coreutils/manual/html_node/sha2-utilities.html
• gzip manual: https://www.gnu.org/software/gzip/manual/gzip.html