How to Monitor Log Files in Bash

Long before I had a proper monitoring stack in place on any project, I was watching log files directly from the terminal — tailing them live during a deploy, grepping through them after an incident, writing small scripts to alert me the moment an error pattern showed up. Bash turns out to be genuinely excellent at this, largely thanks to tail -f, grep, and awk working together. This article covers everything I use for log monitoring in Bash, from watching a file live to building a full alerting script.

The Foundation: tail -f

The single most useful command for log monitoring is:

tail -f /var/log/syslog

-f (follow) keeps the command running, printing new lines to the terminal as they’re appended to the file — exactly what you want when watching a live deployment or debugging an issue in real time. Press Ctrl+C to stop.

For log files that get rotated (renamed and replaced with a fresh empty file, which is standard practice with tools like logrotate), use -F instead:

tail -F /var/log/syslog

-F is smarter than -f — it detects when the file has been rotated and automatically reopens the new file by the same name, rather than continuing to follow the old, now-renamed file that’s no longer being written to.

Filtering a Live Log Stream

Combining tail -f with grep is the pattern I use constantly:

tail -f /var/log/nginx/access.log | grep "ERROR"

For case-insensitive matching:

tail -f /var/log/app.log | grep -i "error"

To highlight matches in color while still seeing everything (using grep --color with a pattern that matches everything, .*, combined with the specific pattern):

tail -f /var/log/app.log | grep --color=always -E "ERROR|WARN|$"

This colors ERROR and WARN lines while still passing every line through, since |$ (end of line) matches every line and lets it through unfiltered.

Step-by-Step: A Real-Time Error Watcher

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="/var/log/app.log"
PATTERN="ERROR|CRITICAL|FATAL"

echo "Watching $LOG_FILE for: $PATTERN"

tail -F "$LOG_FILE" | grep --line-buffered -E "$PATTERN" | while IFS= read -r line; do
    echo "[ALERT] $(date '+%Y-%m-%d %H:%M:%S') - $line"
done

Explaining the Script Internally

• --line-buffered on grep is essential here. Normally, when grep‘s output is piped into another command rather than a terminal, it buffers output in larger chunks for efficiency, which means alerts would appear in delayed bursts instead of immediately. --line-buffered forces grep to flush its output after every matching line.
• The while IFS= read -r line loop processes each matching line as it arrives, letting me attach a timestamp or trigger further actions (like sending an email) for every match, one at a time.
• tail -F (capital F) ensures the watcher survives log rotation without needing to be restarted manually.

Real-World Use Case: Alerting on Error Spikes

Simply catching every error can be noisy. A pattern I actually use in production is counting errors over a time window and only alerting if the rate crosses a threshold:

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="/var/log/app.log"
WINDOW_SECONDS=60
THRESHOLD=10

error_count=0
window_start=$(date +%s)

tail -F "$LOG_FILE" | grep --line-buffered "ERROR" | while IFS= read -r line; do
    now=$(date +%s)
    elapsed=$((now - window_start))

    if [ "$elapsed" -ge "$WINDOW_SECONDS" ]; then
        error_count=0
        window_start=$now
    fi

    error_count=$((error_count + 1))

    if [ "$error_count" -ge "$THRESHOLD" ]; then
        echo "ALERT: $error_count errors in the last ${WINDOW_SECONDS}s!"
        # trigger notification here
        error_count=0
        window_start=$now
    fi
done

Note: because the final while loop runs in a subshell (due to the pipe), variables like error_count reset each time the script itself restarts, but persist correctly within this run since the whole counting logic lives inside that same subshell.

Real-World Use Case: Searching Historical Logs for a Pattern

Live monitoring is only half the picture — I also frequently need to search through existing, potentially huge log files after the fact:

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="/var/log/app.log"
SEARCH_TERM="$1"
START_DATE="$2"

grep "$SEARCH_TERM" "$LOG_FILE" | awk -v start="$START_DATE" '$1 >= start'

Assuming log lines start with an ISO-format date like 2026-07-28 ..., this filters lines matching the search term and then further filters by date using awk‘s string comparison, which works correctly for ISO 8601 dates since they sort lexicographically in chronological order.

For very large log files, grep is far faster than reading the file line by line in a Bash loop, since grep is a highly optimized compiled tool, while Bash’s own while read loop has meaningfully more per-line overhead.

Automation Example: Daily Log Summary Report

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="/var/log/app.log"
TODAY=$(date +%Y-%m-%d)
REPORT_FILE="/tmp/log_summary_${TODAY}.txt"

{
    echo "Log Summary for $TODAY"
    echo "========================"
    echo ""
    echo "Total lines: $(grep -c "^${TODAY}" "$LOG_FILE")"
    echo "Errors: $(grep -c "^${TODAY}.*ERROR" "$LOG_FILE")"
    echo "Warnings: $(grep -c "^${TODAY}.*WARN" "$LOG_FILE")"
    echo ""
    echo "Top 5 error messages:"
    grep "^${TODAY}.*ERROR" "$LOG_FILE" | awk -F'ERROR' '{print $2}' | sort | uniq -c | sort -rn | head -5
} > "$REPORT_FILE"

cat "$REPORT_FILE"

I run this nightly via cron and feed the resulting report into the email script covered elsewhere in this series, giving me a daily digest without having to manually dig through logs each morning.

Scheduled via:

0 23 * * * /usr/local/bin/daily_log_summary.sh

Multi-File Monitoring

To watch several log files at once and know which file each line came from:

tail -F /var/log/app1.log /var/log/app2.log /var/log/app3.log

When following multiple files, tail automatically prefixes each block of output with a header showing which file it came from whenever the source switches, making it easy to distinguish interleaved output.

Security Considerations

• Log files often contain sensitive data — IP addresses, session tokens, sometimes even accidentally logged passwords or personal information. Restrict read permissions on log files and any monitoring scripts’ output to only those who need it (chmod 640, appropriate group ownership).
• Be careful about what you forward to external services. If a monitoring script sends log excerpts to a third-party alerting tool (like a Slack webhook), make sure you’re not inadvertently leaking sensitive data captured in those log lines.
• Sanitize log content before displaying it in a web-based dashboard. Log lines can contain content that, if rendered unescaped in HTML, could lead to stored XSS if an attacker manages to get malicious content logged (for example, through a crafted request that gets logged verbatim).
• Watch for log injection. If your own application logs raw user input without sanitization, an attacker could inject fake log lines (including fake timestamps or fake “SUCCESS” markers) to confuse anyone — or any script — parsing those logs later.

Optimization Tips

• Use grep --line-buffered (or stdbuf -oL as a more general-purpose alternative for other commands) whenever you’re piping a live stream through several tools, or you’ll see confusing output delays.
• For searching very large historical log files, grep is dramatically faster than a Bash while read loop; reserve line-by-line Bash processing for when you genuinely need per-line logic that grep/awk/sed can’t express.
• Rotate and compress old logs (logrotate is the standard tool for this) so that both live monitoring and historical searches stay fast — a single log file that’s grown to tens of gigabytes will slow down every tool that touches it.
• When counting or aggregating patterns across huge files, prefer awk over multiple chained grep | wc -l calls, since awk can do the counting and filtering in a single pass over the data.

Troubleshooting

• tail -f stops receiving updates after a log rotation: switch to tail -F (capital), which reopens rotated files automatically.
• Piped output through grep appears delayed in bursts: add --line-buffered to grep, since default buffering behavior changes when output isn’t going directly to a terminal.
• Script consumes high CPU while monitoring: check you’re not accidentally polling the file in a tight loop instead of using tail -f‘s efficient blocking read; tail -f is efficient because it waits for new data rather than repeatedly re-reading the file.
• Permission denied reading a system log file: many logs under /var/log require root or membership in a specific group (like adm on Debian-based systems); use sudo or add your user to the appropriate group rather than loosening file permissions system-wide.

Common Mistakes to Avoid

• Using tail -f instead of tail -F on logs that get rotated, silently losing track of new log entries after rotation.
• Forgetting --line-buffered on grep when building a real-time alerting pipeline, leading to alerts arriving in delayed clumps instead of immediately.
• Writing a manual while read polling loop to check for new lines instead of just using tail -f, which is both simpler and far more efficient.
• Not accounting for multi-line log entries (like stack traces) when pattern-matching line by line, which can cause grep to only catch the first line of a multi-line error.

FAQs

How do I monitor logs across multiple servers at once? Combine ssh with tail -f in a loop, or better, forward logs to a centralized system like rsyslog, journald with a central collector, or a dedicated log aggregation tool once you outgrow single-server Bash monitoring.

Can I monitor journald logs (systemd) the same way? Not with tail, since journald stores logs in a binary format; use journalctl -f instead, which provides the same “follow” behavior for systemd’s journal.

How do I handle multi-line log entries like stack traces in my pattern matching? Use awk with a record separator trick, or a tool like sed with pattern-range matching, to treat a stack trace as one logical block rather than matching line by line.

Is Bash log monitoring a replacement for a real monitoring/alerting system? For small setups or quick diagnostics, yes it’s genuinely sufficient. For larger production systems, dedicated tools (Prometheus, ELK/Elastic stack, Datadog, etc.) offer far more robust aggregation, retention, and alerting — but the Bash patterns here remain useful even alongside those tools for quick manual investigation.

Summary

Log monitoring in Bash comes down to a small number of tools working together well: tail -f/-F for live streaming, grep --line-buffered for real-time filtering, and awk for structured parsing and aggregation. Whether I’m watching a deploy in real time, building a threshold-based alerting script, or generating a daily summary report, these same few building blocks cover the vast majority of what I need — no dedicated log monitoring platform required for smaller-scale needs.

References

• GNU Coreutils tail documentation: https://www.gnu.org/software/coreutils/manual/html_node/tail-invocation.html
• GNU grep manual: https://www.gnu.org/software/grep/manual/grep.html
• GNU awk user’s guide: https://www.gnu.org/software/gawk/manual/gawk.html
• logrotate documentation: https://github.com/logrotate/logrotate
• journalctl man page: https://man7.org/linux/man-pages/man1/journalctl.1.html