hakrawler: Complete Guide to Web Crawling and URL Discovery Using Kali Linux

hakrawler is a fast, lightweight web crawler written in Go by Luke Stephens (hakluke), designed for quickly discovering endpoints, URLs, JavaScript files, forms, and subdomains from a target website. It is built for speed and simplicity — it is meant to be piped together with other recon tools in a Unix-style pipeline rather than used as a full standalone crawler like Burp or ZAP’s spider. hakrawler can crawl both statically-linked HTML content and, with additional flags, discover URLs referenced inside JavaScript files, making it a staple in bug bounty and fast recon workflows.

How to Install

Not pre-installed on Kali by default; install via Go:

# Requires Go installed
sudo apt update
sudo apt install golang-go -y

go install github.com/hakluke/hakrawler@latest

# Ensure GOPATH/bin is in your PATH
export PATH=$PATH:$(go env GOPATH)/bin
hakrawler -h

Syntax

echo <url> | hakrawler [options]
cat urls.txt | hakrawler [options]

All Command-Line Options

Option	Description
-d, -depth <n>	Maximum crawl depth (default 2)
-h, -headers	Include custom headers (read from stdin, format Header: Value)
-i, -insecure	Disable TLS certificate verification
-json	Output results as JSON
-plain	Output plain URLs only (no source labels)
-proxy <url>	Route requests through a proxy
-s, -subs	Include subdomains found during crawl
-t, -threads <n>	Number of concurrent threads (default 8)
-timeout <secs>	Request timeout (default 3s)
-u, -unique	Output only unique URLs
-w, -wayback	Include URLs from the Wayback Machine
-usewayback	Alias for wayback inclusion
-scope <toplevel|subs>	Restrict crawl scope
-outdir <dir>	Output directory for results
-forms	Include HTML form details in output
-size <bytes>	Max response size to process
-auth <token>	Add an Authorization header value
-sitemap	Parse sitemap.xml for additional URLs
-robots	Parse robots.txt for additional URLs
-linkfinder	Extract endpoints from JavaScript files

Note: hakrawler’s flags have shifted slightly across versions; run hakrawler -h to confirm the exact flag set for your installed build.

Basic Usage (Expected Output in Bash)

$ echo "https://testphp.vulnweb.com" | hakrawler

Output:

[href] https://testphp.vulnweb.com/index.php
[href] https://testphp.vulnweb.com/categories.php
[href] https://testphp.vulnweb.com/login.php
[script] https://testphp.vulnweb.com/js/jquery.min.js
[form] https://testphp.vulnweb.com/search.php?test=query

Practical Examples with Output

Example 1 — Crawl with increased depth

$ echo "https://testphp.vulnweb.com" | hakrawler -depth 3

Output:

[href] https://testphp.vulnweb.com/index.php
[href] https://testphp.vulnweb.com/artists.php?artist=1
[href] https://testphp.vulnweb.com/artists.php?artist=2
[href] https://testphp.vulnweb.com/comment.php?aid=1

Example 2 — Include subdomains in scope

$ echo "https://example.com" | hakrawler -subs

Output:

[href] https://example.com/about
[href] https://blog.example.com/post-1
[href] https://api.example.com/v1/users

Example 3 — Output unique URLs only, sorted

$ echo "https://testphp.vulnweb.com" | hakrawler -unique | sort

Output:

https://testphp.vulnweb.com/artists.php
https://testphp.vulnweb.com/categories.php
https://testphp.vulnweb.com/index.php
https://testphp.vulnweb.com/login.php

Example 4 — Crawl multiple targets from a file

$ cat targets.txt | hakrawler -t 20 -depth 2

Output:

[href] https://target1.com/page1
[href] https://target2.com/dashboard
[href] https://target3.com/api/v1/status

Example 5 — JSON output for automation

$ echo "https://testphp.vulnweb.com" | hakrawler -json | jq -r '.URL'

Output:

https://testphp.vulnweb.com/login.php
https://testphp.vulnweb.com/search.php

Example 6 — Crawl through Burp proxy

$ echo "https://testphp.vulnweb.com" | hakrawler -proxy http://127.0.0.1:8080

Output:

[+] Routing through proxy 127.0.0.1:8080
[href] https://testphp.vulnweb.com/index.php

Example 7 — Include Wayback Machine historical URLs

$ echo "https://testphp.vulnweb.com" | hakrawler -wayback

Output:

[href] https://testphp.vulnweb.com/index.php
[wayback] https://testphp.vulnweb.com/old-login.php
[wayback] https://testphp.vulnweb.com/backup.sql

Example 8 — Extract forms for injection testing

$ echo "https://testphp.vulnweb.com" | hakrawler -forms

Output:

[form] action=/search.php method=POST params=[searchFor,goButton]
[form] action=/login.php method=POST params=[uname,pass]

Example 9 — Chain with httpx and Nuclei

$ echo "https://testphp.vulnweb.com" | hakrawler -depth 2 -unique | httpx -silent | nuclei -silent

Output:

[http-missing-security-headers] [http] [info] https://testphp.vulnweb.com/login.php

Common Use Cases

• Fast endpoint discovery as a first pass before running deeper crawlers like katana.
• Building URL lists to feed directly into Nuclei, httpx, or manual Burp analysis.
• Extracting form parameters as candidates for injection-based testing.
• Discovering JavaScript-embedded endpoints for API reconnaissance.
• Quick single-command crawl during time-constrained bug bounty recon.

Automation with Bash

#!/bin/bash
# hakrawler-recon.sh — crawl a domain and prepare a clean, deduplicated URL list

DOMAIN="$1"
OUTFILE="hakrawler-urls-$(echo $DOMAIN | tr '.' '_').txt"

echo "https://$DOMAIN" | hakrawler -subs -depth 3 -unique -forms > "$OUTFILE"

echo "[*] Deduplicating and sorting..."
sort -u "$OUTFILE" -o "$OUTFILE"

echo "[+] $(wc -l < "$OUTFILE") unique URLs saved to $OUTFILE"

Tips and Best Practices

• Use -depth conservatively on large sites — depth 3+ can generate an enormous number of requests quickly.
• Combine -subs with scope restriction carefully to avoid crawling unrelated third-party domains linked from the page.
• Pipe hakrawler directly into httpx and nuclei for a compact one-liner recon-to-scan chain.
• Use -forms output specifically when preparing an SQLi/XSS injection test plan.
• hakrawler is intentionally minimal — for JavaScript-heavy SPAs, prefer katana with headless rendering for better coverage.

Troubleshooting

Issue	Cause	Fix
No output produced	Target blocks default crawler User-Agent	Add custom headers via -h or -headers
Command not found	Go bin path not exported	Add $(go env GOPATH)/bin to PATH
Very few URLs discovered on SPA	JavaScript not executed during crawl	Use katana with -headless instead
TLS handshake errors	Self-signed/internal cert	Add -insecure flag
Crawl takes too long	Deep recursion on large site	Lower -depth or add -timeout

References

• Official GitHub repository: https://github.com/hakluke/hakrawler
• Author’s blog/tools: https://hakluke.com/
• Kali/Go install guide: https://github.com/hakluke/hakrawler#installation