How to Create a Bash Web Scraping Tool

I’ll admit upfront: Bash is not the tool most people reach for when they think “web scraping.” Python with BeautifulSoup or Scrapy is the more common choice. But for quick, lightweight scraping tasks — checking if a price changed, pulling headlines from a page, monitoring for new listings — I’ve found that a combination of curl and a few text-processing tools gets the job done without installing a single extra dependency. In this article, I’ll walk through how I build web scraping tools entirely in Bash.

Why Scrape with Bash At All?

• No need to set up a Python virtual environment or install libraries just to check one thing.
• Easy to drop into a cron job on any server that already has Bash.
• Great for quick, disposable scripts I don’t need to maintain long-term.
• Combines naturally with other Bash tools I already use for downloading, parsing, and notifications.

That said, I want to be upfront about the limits: Bash scraping works best on relatively simple, static HTML. If a site relies heavily on JavaScript to render content, you’ll need a real browser automation tool like Selenium or Playwright instead — no amount of grep will execute JavaScript for you.

Tools I Rely On

• curl: to fetch the raw HTML.
• grep: to search for patterns.
• sed: to strip out HTML tags or reformat text.
• awk: for column-based text extraction.
• pup or xmllint --html: for real HTML/CSS-selector-based parsing when regex isn’t reliable enough.

Step One: Fetching the Page

curl -s -A "Mozilla/5.0" https://example.com > page.html

• -s runs silently, hiding the progress meter.
• -A "Mozilla/5.0" sets a custom User-Agent header. Many sites block requests that don’t look like they’re coming from a real browser, so I always set this.

Basic Example: Extracting Text with grep and sed

Let’s say I want to scrape the <title> tag from a page:

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com"

html=$(curl -s -A "Mozilla/5.0" "$URL")

title=$(echo "$html" | grep -oP '(?<=<title>).*(?=</title>)')

echo "Page title: $title"

• grep -oP enables Perl-compatible regex (-P) and outputs only the matched portion (-o).
• (?<=<title>) and (?=</title>) are lookbehind/lookahead assertions, meaning “match the text between these tags without including the tags themselves in the output.”

A More Reliable Approach: Using pup

Regex-based HTML parsing is fragile — HTML isn’t a regular language, and pages with nested or malformed tags will break naive patterns quickly. For anything beyond trivial extraction, I use pup, a command-line HTML parser that understands CSS selectors, similar to how you’d use document.querySelector() in JavaScript.

Install it:

# Requires Go, or download a prebuilt binary from the pup GitHub releases page
go install github.com/ericchiang/pup@latest

Example: extracting all the headline links from a news page assuming headlines are in <h2 class="headline"> tags with an <a> inside:

#!/usr/bin/env bash
set -euo pipefail

URL="https://news.example.com"

curl -s -A "Mozilla/5.0" "$URL" | pup 'h2.headline a text{}'

This grabs the text content of every anchor tag inside an h2.headline element — something that would be extremely fragile to do with regex alone, especially if headlines span multiple lines or contain nested tags.

Real-World Use Case: Price Monitoring Script

Here’s a script I use to track a product’s price and alert me when it drops below a threshold:

#!/usr/bin/env bash
set -euo pipefail

URL="https://shop.example.com/product/123"
THRESHOLD=49.99

html=$(curl -s -A "Mozilla/5.0" "$URL")

# Assumes the price appears like: <span class="price">$54.99</span>
price=$(echo "$html" | grep -oP '(?<=class="price">\$)[0-9]+\.[0-9]{2}')

echo "Current price: \$${price}"

if (( $(echo "$price < $THRESHOLD" | bc -l) )); then
    echo "Price dropped below threshold! Sending alert..."
    # mail -s "Price Alert" me@example.com <<< "Price is now \$${price}"
fi

I use bc -l here because Bash’s built-in arithmetic doesn’t support floating-point comparisons natively; bc handles the decimal comparison and returns 1 or 0, which (( )) then evaluates as true/false.

Real-World Use Case: Scraping a Table of Data

For a page containing a simple HTML table, I combine pup with jq since pup can output JSON directly:

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com/stats-table"

curl -s -A "Mozilla/5.0" "$URL" \
  | pup 'table tr json{}' \
  | jq -r '.[] | .children[].text' 

This converts the scraped table rows into JSON via pup, then uses jq to walk through and print out each cell’s text — combining two of the tools covered elsewhere in this series into one pipeline.

Automation Example: Scheduled Scraping with Change Detection

A pattern I use constantly: scrape a page, compare it to the last saved version, and only alert if something changed.

#!/usr/bin/env bash
set -euo pipefail

URL="https://example.com/jobs"
CURRENT_FILE="/tmp/jobs_current.txt"
PREVIOUS_FILE="/tmp/jobs_previous.txt"

curl -s -A "Mozilla/5.0" "$URL" | pup 'a.job-title text{}' > "$CURRENT_FILE"

if [ -f "$PREVIOUS_FILE" ]; then
    if ! diff -q "$PREVIOUS_FILE" "$CURRENT_FILE" > /dev/null; then
        echo "New job listings detected:"
        diff "$PREVIOUS_FILE" "$CURRENT_FILE"
    fi
fi

cp "$CURRENT_FILE" "$PREVIOUS_FILE"

Scheduled via cron to run every hour:

0 * * * * /usr/local/bin/scrape_jobs.sh >> /var/log/job_scraper.log 2>&1

Security Considerations

• Respect robots.txt and the site’s terms of service. Scraping isn’t inherently illegal, but ignoring a site’s stated policies can get your IP blocked or, in some jurisdictions, create legal exposure. I check robots.txt before writing any scraper.
• Rate-limit your requests. Hammering a site with rapid requests in a loop can look like a denial-of-service attack. I add sleep between requests in any loop that hits the same domain repeatedly.
• Never scrape and store personal data you don’t have a legitimate reason to hold. Scraping usernames, emails, or other personal information carries real privacy and legal obligations (like GDPR, depending on jurisdiction).
• Validate scraped content before using it in further commands. Never pipe scraped HTML content directly into eval or execute it as a command — treat all scraped text as untrusted input.
• Use a realistic but honest User-Agent. Spoofing a User-Agent to appear as a browser is common and generally fine for basic scraping, but I avoid impersonating specific bots (like Googlebot) since that can violate a site’s terms.

Optimization Tips

• Cache the raw HTML fetch when developing/debugging your extraction logic, so you’re not re-requesting the page every time you tweak a grep pattern.
• Batch multiple pages with curl‘s --parallel flag rather than looping through URLs one at a time when scraping many pages from the same site (while still respecting rate limits).
• Prefer pup/xmllint over chained sed/grep regex once your extraction logic grows past two or three patterns — it will be more maintainable and less fragile against markup changes.

Troubleshooting

• Empty output despite the content clearly being on the page: the content is likely rendered by JavaScript after the initial page load. curl only fetches the raw HTML returned by the server, not what a browser renders after running scripts. You’ll need a headless browser tool for JS-rendered content.
• Getting blocked or receiving a CAPTCHA page: the site has detected automated traffic. Slow down your request rate, ensure your User-Agent looks legitimate, and check if the site offers an official API instead.
• grep -P not supported: some minimal environments (like Alpine Linux with BusyBox grep) don’t support Perl-compatible regex. Install GNU grep, or switch the pattern to a POSIX-compatible one.
• Encoding issues (garbled text): check the page’s actual charset with curl -sI URL | grep -i content-type and convert with iconv if it’s not UTF-8.

Common Mistakes to Avoid

• Relying purely on regex for deeply nested or inconsistent HTML instead of a real HTML parser like pup.
• Forgetting to set a User-Agent and getting blocked by basic bot detection.
• Scraping at a rate that could be mistaken for an attack.
• Assuming static HTML output always matches what you see in a browser — dynamic, JavaScript-rendered pages will not match.

FAQs

Is web scraping with Bash legal? It depends on the site’s terms of service, the jurisdiction, and what data you’re collecting. Publicly available, non-personal data scraped respectfully (rate-limited, following robots.txt) is generally lower risk, but I’m not a lawyer — check the specific site’s terms if you’re unsure.

Can Bash handle JavaScript-rendered pages? No. curl only retrieves the initial HTML response; it cannot execute JavaScript. For dynamic pages, you need a headless browser tool.

Is pup still maintained? pup has had periods of limited maintenance; check its GitHub repository for current status. xmllint --html (part of libxml2) is a solid, well-maintained alternative available on nearly every Linux system by default.

How do I scrape a page that requires login? You can use curl with cookies (-b cookies.txt -c cookies.txt) after performing a login request, but be extra cautious with credentials in scripts — never hardcode passwords directly in a script file.

Summary

Building a web scraping tool in Bash is a great fit for quick, lightweight tasks: monitoring a price, checking for new listings, or pulling a single piece of text from a page on a schedule. curl handles the fetching, and tools like grep, sed, and especially pup handle the extraction. For anything involving JavaScript-heavy sites or large-scale scraping, it’s worth graduating to a dedicated tool, but for the 80% of small scraping tasks I run into, Bash has never let me down.

References

• curl manual: https://curl.se/docs/manpage.html
• GNU grep manual (Perl-compatible regex support): https://www.gnu.org/software/grep/manual/grep.html
• pup GitHub repository: https://github.com/ericchiang/pup
• libxml2 xmllint documentation: https://gitlab.gnome.org/GNOME/libxml2/-/wikis/home
• The Web Robots Pages (robots.txt standard): https://www.robotstxt.org/