hakrawler: Complete Guide to Web Crawling and URL Discovery Using Kali Linux

hakrawler is a fast, lightweight web crawler written in Go by Luke Stephens (hakluke), designed for quickly discovering endpoints, URLs, JavaScript files, forms, and subdomains from a target website. It is built for speed and simplicity — it is meant to be piped together with other recon tools in a Unix-style pipeline rather than used as a full standalone crawler like Burp or ZAP’s spider. hakrawler can crawl both statically-linked HTML content and, with additional flags, discover URLs referenced inside JavaScript files, making it a staple in bug bounty and fast recon workflows.

How to Install

Not pre-installed on Kali by default; install via Go:

# Requires Go installed
sudo apt update
sudo apt install golang-go -y

go install github.com/hakluke/hakrawler@latest

# Ensure GOPATH/bin is in your PATH
export PATH=$PATH:$(go env GOPATH)/bin
hakrawler -h

Syntax

echo <url> | hakrawler [options]
cat urls.txt | hakrawler [options]

All Command-Line Options

Note: hakrawler’s flags have shifted slightly across versions; run hakrawler -h to confirm the exact flag set for your installed build.

Basic Usage (Expected Output in Bash)

$ echo "https://testphp.vulnweb.com" | hakrawler

Output:

[href] https://testphp.vulnweb.com/index.php
[href] https://testphp.vulnweb.com/categories.php
[href] https://testphp.vulnweb.com/login.php
[script] https://testphp.vulnweb.com/js/jquery.min.js
[form] https://testphp.vulnweb.com/search.php?test=query

Practical Examples with Output

Example 1 — Crawl with increased depth

$ echo "https://testphp.vulnweb.com" | hakrawler -depth 3

Output:

[href] https://testphp.vulnweb.com/index.php
[href] https://testphp.vulnweb.com/artists.php?artist=1
[href] https://testphp.vulnweb.com/artists.php?artist=2
[href] https://testphp.vulnweb.com/comment.php?aid=1

Example 2 — Include subdomains in scope

$ echo "https://example.com" | hakrawler -subs

Output:

[href] https://example.com/about
[href] https://blog.example.com/post-1
[href] https://api.example.com/v1/users

Example 3 — Output unique URLs only, sorted

$ echo "https://testphp.vulnweb.com" | hakrawler -unique | sort

Output:

https://testphp.vulnweb.com/artists.php
https://testphp.vulnweb.com/categories.php
https://testphp.vulnweb.com/index.php
https://testphp.vulnweb.com/login.php

Example 4 — Crawl multiple targets from a file

$ cat targets.txt | hakrawler -t 20 -depth 2

Output:

[href] https://target1.com/page1
[href] https://target2.com/dashboard
[href] https://target3.com/api/v1/status

Example 5 — JSON output for automation

$ echo "https://testphp.vulnweb.com" | hakrawler -json | jq -r '.URL'

Output:

https://testphp.vulnweb.com/login.php
https://testphp.vulnweb.com/search.php

Example 6 — Crawl through Burp proxy

$ echo "https://testphp.vulnweb.com" | hakrawler -proxy http://127.0.0.1:8080

Output:

[+] Routing through proxy 127.0.0.1:8080
[href] https://testphp.vulnweb.com/index.php

Example 7 — Include Wayback Machine historical URLs

$ echo "https://testphp.vulnweb.com" | hakrawler -wayback

Output:

[href] https://testphp.vulnweb.com/index.php
[wayback] https://testphp.vulnweb.com/old-login.php
[wayback] https://testphp.vulnweb.com/backup.sql

Example 8 — Extract forms for injection testing

$ echo "https://testphp.vulnweb.com" | hakrawler -forms

Output:

[form] action=/search.php method=POST params=[searchFor,goButton]
[form] action=/login.php method=POST params=[uname,pass]

Example 9 — Chain with httpx and Nuclei

$ echo "https://testphp.vulnweb.com" | hakrawler -depth 2 -unique | httpx -silent | nuclei -silent

Output:

[http-missing-security-headers] [http] [info] https://testphp.vulnweb.com/login.php

Common Use Cases

• Fast endpoint discovery as a first pass before running deeper crawlers like katana.
• Building URL lists to feed directly into Nuclei, httpx, or manual Burp analysis.
• Extracting form parameters as candidates for injection-based testing.
• Discovering JavaScript-embedded endpoints for API reconnaissance.
• Quick single-command crawl during time-constrained bug bounty recon.

Automation with Bash

#!/bin/bash
# hakrawler-recon.sh — crawl a domain and prepare a clean, deduplicated URL list

DOMAIN="$1"
OUTFILE="hakrawler-urls-$(echo $DOMAIN | tr '.' '_').txt"

echo "https://$DOMAIN" | hakrawler -subs -depth 3 -unique -forms > "$OUTFILE"

echo "[*] Deduplicating and sorting..."
sort -u "$OUTFILE" -o "$OUTFILE"

echo "[+] $(wc -l < "$OUTFILE") unique URLs saved to $OUTFILE"

Tips and Best Practices

• Use -depth conservatively on large sites — depth 3+ can generate an enormous number of requests quickly.
• Combine -subs with scope restriction carefully to avoid crawling unrelated third-party domains linked from the page.
• Pipe hakrawler directly into httpx and nuclei for a compact one-liner recon-to-scan chain.
• Use -forms output specifically when preparing an SQLi/XSS injection test plan.
• hakrawler is intentionally minimal — for JavaScript-heavy SPAs, prefer katana with headless rendering for better coverage.

Troubleshooting

References

• Official GitHub repository: https://github.com/hakluke/hakrawler
• Author’s blog/tools: https://hakluke.com/
• Kali/Go install guide: https://github.com/hakluke/hakrawler#installation