Offensive Security Researcher & Quantum Cryptography Analyst

How to Create a Bash File Filtering Tool

Abdul Wahab Junaid
How to Create a Bash File Filtering Tool

How to Create a Bash File Filtering Tool

Filtering is the task of pulling relevant lines, records, or files out of a larger set based on some condition — log entries from the last hour, CSV rows matching a value, files above a certain size, text lines matching a pattern while excluding another. grep, awk, and find already do most of the heavy lifting individually; the value of a dedicated filtering tool is combining them into a consistent interface that covers the common filtering shapes people actually need.

This article builds a flexible Bash file filtering tool covering content filtering (grep/awk-based) and file-level filtering (find-based), with attention to performance and correctness on large inputs.

Two Kinds of Filtering

It’s worth separating this into two distinct problems up front:

  1. Content filtering — selecting lines within a file that match a condition.
  2. File filtering — selecting which files to operate on, based on metadata (size, age, name, type).

A capable tool handles both, since real tasks often combine them (“find all .log files modified in the last day, then filter for ERROR lines within them”).

Step 1: Basic Content Filtering with Grep

#!/usr/bin/env bash
set -euo pipefail

FILE="$1"
PATTERN="$2"

grep -n "$PATTERN" "$FILE"

-n prefixes matching lines with their line number, which is almost always useful context when filtering. Running:

./filter.sh app.log "ERROR"

42:2026-07-28 10:15:03 ERROR failed to connect to database
187:2026-07-28 10:22:41 ERROR timeout on request

Step 2: Inverting the Match (Exclusion)

Sometimes the goal is to filter out noise rather than filter in matches:

grep -v "$EXCLUDE_PATTERN" "$FILE"

-v inverts the match, printing every line that does not match. Combining inclusion and exclusion:

grep "$INCLUDE_PATTERN" "$FILE" | grep -v "$EXCLUDE_PATTERN"

This chains two grep calls: first keep lines matching the pattern of interest, then drop any of those that also match a known-noisy pattern (e.g., keep ERROR lines but exclude ones from a known-flaky health check).

Step 3: Case-Insensitive and Regex Filtering

grep -i "$PATTERN" "$FILE"          # case-insensitive
grep -E "$EXTENDED_REGEX" "$FILE"   # extended regex support
grep -P "$PERL_REGEX" "$FILE"       # Perl-compatible regex (lookaheads, etc.)

Step 4: Field-Based Filtering with Awk

Grep matches whole lines; awk filters based on specific fields, which is essential for structured data like CSVs or space-delimited logs:

awk -F',' '$3 == "active"' users.csv

Filtering by numeric comparison:

awk -F',' '$4 > 100' sales.csv

This keeps only rows where the fourth column, interpreted numerically, exceeds 100.

Step 5: Filtering Files by Metadata with Find

find . -name "*.log" -size +10M

Finds every .log file larger than 10 megabytes. Common metadata filters:

find . -mtime -1          # modified in the last 1 day
find . -mtime +30         # modified more than 30 days ago
find . -size +100k        # larger than 100 KB
find . -type f -empty     # empty files
find . -newer reference.txt  # modified more recently than reference.txt

Step 6: A Unified Filtering Script

#!/usr/bin/env bash
set -euo pipefail

usage() {
    cat <<EOF
Usage: $0 <mode> [options]

Modes:
  content <file> -i <pattern> [-x exclude_pattern] [-c (case-insensitive)]
  field <file> -d delimiter -f field_num -v value
  files <dir> [-n name_glob] [-s size] [-m mtime_days]
EOF
    exit 1
}

MODE="${1:-}"
[[ -z "$MODE" ]] && usage
shift

case "$MODE" in
  content)
    FILE="$1"; shift
    INCLUDE=""; EXCLUDE=""; CASE_FLAG=""
    while getopts ":i:x:c" opt; do
        case "$opt" in
            i) INCLUDE="$OPTARG" ;;
            x) EXCLUDE="$OPTARG" ;;
            c) CASE_FLAG="-i" ;;
            *) usage ;;
        esac
    done
    RESULT=$(grep $CASE_FLAG "$INCLUDE" "$FILE")
    if [[ -n "$EXCLUDE" ]]; then
        RESULT=$(echo "$RESULT" | grep -v "$EXCLUDE")
    fi
    echo "$RESULT"
    ;;

  field)
    FILE="$1"; shift
    DELIM=","; FIELD=1; VALUE=""
    while getopts ":d:f:v:" opt; do
        case "$opt" in
            d) DELIM="$OPTARG" ;;
            f) FIELD="$OPTARG" ;;
            v) VALUE="$OPTARG" ;;
            *) usage ;;
        esac
    done
    awk -F"$DELIM" -v field="$FIELD" -v val="$VALUE" '$field == val' "$FILE"
    ;;

  files)
    DIR="$1"; shift
    NAME="*"; SIZE=""; MTIME=""
    while getopts ":n:s:m:" opt; do
        case "$opt" in
            n) NAME="$OPTARG" ;;
            s) SIZE="$OPTARG" ;;
            m) MTIME="$OPTARG" ;;
            *) usage ;;
        esac
    done
    FIND_ARGS=(-name "$NAME")
    [[ -n "$SIZE" ]] && FIND_ARGS+=(-size "$SIZE")
    [[ -n "$MTIME" ]] && FIND_ARGS+=(-mtime "$MTIME")
    find "$DIR" -type f "${FIND_ARGS[@]}"
    ;;

  *)
    usage
    ;;
esac

Example invocations:

./filter.sh content app.log -i "ERROR" -x "healthcheck"
./filter.sh field users.csv -d ',' -f 3 -v "active"
./filter.sh files ./logs -n "*.log" -s +10M -m -1

The field mode passes both the field number and comparison value into awk as external variables (-v field="$FIELD" -v val="$VALUE") rather than interpolating them directly into the awk program string — this avoids quoting headaches and keeps the awk script itself static and safe regardless of what values are passed in.

Real-World Use Cases

Automation Example: Daily Error Report

#!/usr/bin/env bash
set -euo pipefail

LOG="/var/log/app/current.log"
REPORT="/tmp/error-report-$(date +%F).txt"

grep -E "ERROR|CRITICAL" "$LOG" | grep -v "known-flaky-check" > "$REPORT"

if [[ -s "$REPORT" ]]; then
    echo "Errors found, see $REPORT"
    mail -s "Daily Error Report" [email protected] < "$REPORT"
else
    echo "No errors today."
fi

[[ -s "$REPORT" ]] checks whether the report file has non-zero size — i.e., whether any errors were actually captured — before deciding to send a notification.

Best Practices

Security Considerations

Optimization Tips

Troubleshooting

Common Mistakes

FAQs

Can this filter binary files? grep can search binary files with -a (treat as text) or report matches without printing binary garbage using its default binary-detection behavior; generally, filtering is intended for text-based content.

How do I filter based on multiple conditions combined with AND/OR logic? Chain grep calls for AND logic (each stage narrows further), or use grep -E "pattern1|pattern2" for OR logic within a single pass.

Is awk overkill for simple line filtering? For simple whole-line pattern matching, grep is simpler and typically faster. awk earns its place once the filter depends on specific fields or requires arithmetic/conditional logic per line.

Summary

A filtering tool built around grep, awk, and find covers both major filtering needs — narrowing down file contents and narrowing down which files to look at in the first place — using nothing beyond standard, universally available Unix utilities. Structuring the wrapper around clear modes (content, field, files) keeps a small set of powerful primitives accessible through one consistent, memorable interface.

References

Exit mobile version