Offensive Security Researcher & Quantum Cryptography Analyst

How to Create a Bash Web Scraping Tool

Abdul Wahab Junaid
How to Create a Bash Web Scraping Tool

How to Create a Bash Web Scraping Tool

I’ll admit upfront: Bash is not the tool most people reach for when they think “web scraping.” Python with BeautifulSoup or Scrapy is the more common choice. But for quick, lightweight scraping tasks — checking if a price changed, pulling headlines from a page, monitoring for new listings — I’ve found that a combination of curl and a few text-processing tools gets the job done without installing a single extra dependency. In this article, I’ll walk through how I build web scraping tools entirely in Bash.

Why Scrape with Bash At All?

That said, I want to be upfront about the limits: Bash scraping works best on relatively simple, static HTML. If a site relies heavily on JavaScript to render content, you’ll need a real browser automation tool like Selenium or Playwright instead — no amount of grep will execute JavaScript for you.

Tools I Rely On

Step One: Fetching the Page

curl -s -A "Mozilla/5.0" https://example.com > page.html

Basic Example: Extracting Text with grep and sed

Let’s say I want to scrape the <title> tag from a page:

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com"

html=$(curl -s -A "Mozilla/5.0" "$URL")

title=$(echo "$html" | grep -oP '(?<=<title>).*(?=</title>)')

echo "Page title: $title"

A More Reliable Approach: Using pup

Regex-based HTML parsing is fragile — HTML isn’t a regular language, and pages with nested or malformed tags will break naive patterns quickly. For anything beyond trivial extraction, I use pup, a command-line HTML parser that understands CSS selectors, similar to how you’d use document.querySelector() in JavaScript.

Install it:

# Requires Go, or download a prebuilt binary from the pup GitHub releases page
go install github.com/ericchiang/pup@latest

Example: extracting all the headline links from a news page assuming headlines are in <h2 class="headline"> tags with an <a> inside:

#!/usr/bin/env bash
set -euo pipefail

URL="https://news.example.com"

curl -s -A "Mozilla/5.0" "$URL" | pup 'h2.headline a text{}'

This grabs the text content of every anchor tag inside an h2.headline element — something that would be extremely fragile to do with regex alone, especially if headlines span multiple lines or contain nested tags.

Real-World Use Case: Price Monitoring Script

Here’s a script I use to track a product’s price and alert me when it drops below a threshold:

#!/usr/bin/env bash
set -euo pipefail

URL="https://shop.example.com/product/123"
THRESHOLD=49.99

html=$(curl -s -A "Mozilla/5.0" "$URL")

# Assumes the price appears like: <span class="price">$54.99</span>
price=$(echo "$html" | grep -oP '(?<=class="price">\$)[0-9]+\.[0-9]{2}')

echo "Current price: \$${price}"

if (( $(echo "$price < $THRESHOLD" | bc -l) )); then
    echo "Price dropped below threshold! Sending alert..."
    # mail -s "Price Alert" me@example.com <<< "Price is now \$${price}"
fi

I use bc -l here because Bash’s built-in arithmetic doesn’t support floating-point comparisons natively; bc handles the decimal comparison and returns 1 or 0, which (( )) then evaluates as true/false.

Real-World Use Case: Scraping a Table of Data

For a page containing a simple HTML table, I combine pup with jq since pup can output JSON directly:

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com/stats-table"

curl -s -A "Mozilla/5.0" "$URL" \
  | pup 'table tr json{}' \
  | jq -r '.[] | .children[].text'

This converts the scraped table rows into JSON via pup, then uses jq to walk through and print out each cell’s text — combining two of the tools covered elsewhere in this series into one pipeline.

Automation Example: Scheduled Scraping with Change Detection

A pattern I use constantly: scrape a page, compare it to the last saved version, and only alert if something changed.

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com/jobs"
CURRENT_FILE="/tmp/jobs_current.txt"
PREVIOUS_FILE="/tmp/jobs_previous.txt"

curl -s -A "Mozilla/5.0" "$URL" | pup 'a.job-title text{}' > "$CURRENT_FILE"

if [ -f "$PREVIOUS_FILE" ]; then
    if ! diff -q "$PREVIOUS_FILE" "$CURRENT_FILE" > /dev/null; then
        echo "New job listings detected:"
        diff "$PREVIOUS_FILE" "$CURRENT_FILE"
    fi
fi

cp "$CURRENT_FILE" "$PREVIOUS_FILE"

Scheduled via cron to run every hour:

0 * * * * /usr/local/bin/scrape_jobs.sh >> /var/log/job_scraper.log 2>&1

Security Considerations

Optimization Tips

Troubleshooting

Common Mistakes to Avoid

FAQs

Is web scraping with Bash legal? It depends on the site’s terms of service, the jurisdiction, and what data you’re collecting. Publicly available, non-personal data scraped respectfully (rate-limited, following robots.txt) is generally lower risk, but I’m not a lawyer — check the specific site’s terms if you’re unsure.

Can Bash handle JavaScript-rendered pages? No. curl only retrieves the initial HTML response; it cannot execute JavaScript. For dynamic pages, you need a headless browser tool.

Is pup still maintained? pup has had periods of limited maintenance; check its GitHub repository for current status. xmllint --html (part of libxml2) is a solid, well-maintained alternative available on nearly every Linux system by default.

How do I scrape a page that requires login? You can use curl with cookies (-b cookies.txt -c cookies.txt) after performing a login request, but be extra cautious with credentials in scripts — never hardcode passwords directly in a script file.

Summary

Building a web scraping tool in Bash is a great fit for quick, lightweight tasks: monitoring a price, checking for new listings, or pulling a single piece of text from a page on a schedule. curl handles the fetching, and tools like grep, sed, and especially pup handle the extraction. For anything involving JavaScript-heavy sites or large-scale scraping, it’s worth graduating to a dedicated tool, but for the 80% of small scraping tasks I run into, Bash has never let me down.

References

Exit mobile version