gau: Complete Guide to Historical URL Collection and Web Asset Discovery Using Kali Linux

gau (Get All URLs) is a Go-based tool created by lc (lc/0) that fetches known URLs for a given domain from multiple passive sources: the Wayback Machine, Common Crawl, OTX (AlienVault), and URLScan.io. Rather than actively crawling a site (which generates traffic against the live target), gau queries these third-party historical/passive archives to surface URLs that were ever indexed for the domain — including old parameters, deprecated endpoints, backup files, and forgotten API routes that may still be live and vulnerable, even if no longer linked from the current site.

How to Install

# Kali Linux
sudo apt update
sudo apt install gau -y
gau --version

# Or via Go
go install github.com/lc/gau/v2/cmd/gau@latest

Syntax

echo <domain> | gau [options]
gau [options] <domain>
cat domains.txt | gau [options]

All Command-Line Options

Option	Description
--blacklist <ext>	Exclude specific file extensions (e.g., png,jpg,gif)
--fc, --filter-code <codes>	Filter results by status code (requires --providers-timeout/verification)
--from <date>	Fetch URLs from a specific date (format: YYYYMM)
--to <date>	Fetch URLs up to a specific date
--json	Output results as JSON
--mc, --match-code <codes>	Match only specific status codes
--mt, --match-type <types>	Match specific content types
--o, --output <file>	Write output to a file
--providers <list>	Specify providers to query: wayback, commoncrawl, otx, urlscan
--proxy <url>	Route requests through a proxy
--retries <n>	Number of retries per request
--subs	Include subdomains in the query
--threads <n>	Number of concurrent threads
--timeout <secs>	Timeout per request
--verbose	Verbose logging
--config <file>	Path to a custom config file
--ip-blacklist <cidr>	Exclude IP ranges from output

Basic Usage (Expected Output in Bash)

$ echo "testphp.vulnweb.com" | gau

Output:

http://testphp.vulnweb.com/
http://testphp.vulnweb.com/login.php
http://testphp.vulnweb.com/search.php?test=query
http://testphp.vulnweb.com/artists.php?artist=1
http://testphp.vulnweb.com/Mod_Rewrite_Shop/
http://testphp.vulnweb.com/comment.php?aid=1

Practical Examples with Output

Example 1 — Include subdomains in the search

$ echo "example.com" | gau --subs

Output:

https://example.com/
https://blog.example.com/2023/post-title
https://api.example.com/v1/legacy-endpoint

Example 2 — Filter out image/static file noise

$ echo "example.com" | gau --blacklist png,jpg,gif,css,svg,woff

Output:

https://example.com/index.html
https://example.com/api/v1/users?id=5
https://example.com/download.php?file=report.pdf

Example 3 — Query only Wayback Machine and Common Crawl

$ echo "example.com" | gau --providers wayback,commoncrawl

Output:

https://example.com/archive/2019/promo
https://example.com/old-api/v0/status

Example 4 — Date-restricted historical search

$ echo "example.com" | gau --from 202001 --to 202112

Output:

https://example.com/covid-updates
https://example.com/legacy-portal/login

Example 5 — Output as JSON with metadata

$ echo "testphp.vulnweb.com" | gau --json | jq '.url' | head -3

Output:

"http://testphp.vulnweb.com/login.php"
"http://testphp.vulnweb.com/search.php?test=query"
"http://testphp.vulnweb.com/artists.php?artist=1"

Example 6 — Bulk scan a list of domains

$ cat domains.txt | gau --threads 5 -o all-urls.txt

Output:

[+] Fetching URLs for 20 domains...
[+] 4,382 URLs written to all-urls.txt

Example 7 — Match only specific content types (JS files)

$ echo "example.com" | gau --mt application/javascript

Output:

https://example.com/static/main.js
https://example.com/static/legacy-app.js

Example 8 — Chain with httpx to filter for currently-live URLs

$ echo "testphp.vulnweb.com" | gau --blacklist png,jpg,gif | httpx -silent -mc 200

Output:

http://testphp.vulnweb.com/login.php
http://testphp.vulnweb.com/search.php?test=query

Example 9 — Extract URLs with parameters for injection testing

$ echo "testphp.vulnweb.com" | gau | grep '?' | sort -u

Output:

http://testphp.vulnweb.com/artists.php?artist=1
http://testphp.vulnweb.com/comment.php?aid=1
http://testphp.vulnweb.com/listproducts.php?cat=2
http://testphp.vulnweb.com/search.php?test=query

Common Use Cases

• Discovering forgotten/deprecated endpoints and parameters not linked from the current live site.
• Finding historically indexed backup files, config files, or debug endpoints (e.g., .bak, .sql, .env).
• Building large parameter lists for automated injection testing (SQLi, XSS, SSRF).
• Passive reconnaissance that avoids generating direct traffic against the live target.
• Complementing active crawlers (katana/hakrawler) with historical data they cannot see.

Automation with Bash

#!/bin/bash
# gau-recon.sh — passive URL discovery, cleanup, and live-host filtering

DOMAIN="$1"
OUTDIR="gau-recon-$(date +%Y%m%d)"
mkdir -p "$OUTDIR"

echo "[*] Gathering historical URLs for $DOMAIN..."
echo "$DOMAIN" | gau --subs --blacklist png,jpg,jpeg,gif,css,svg,woff,ttf -o "$OUTDIR/raw-urls.txt"

echo "[*] Deduplicating..."
sort -u "$OUTDIR/raw-urls.txt" -o "$OUTDIR/urls-dedup.txt"

echo "[*] Filtering for live URLs..."
cat "$OUTDIR/urls-dedup.txt" | httpx -silent -mc 200 -o "$OUTDIR/live-urls.txt"

echo "[*] Extracting parameterized URLs for testing..."
grep '?' "$OUTDIR/live-urls.txt" > "$OUTDIR/param-urls.txt"

echo "[+] Done. $(wc -l < "$OUTDIR/param-urls.txt") parameterized live URLs saved."

Tips and Best Practices

• Always pair gau with a live-host filter (httpx) since many archived URLs will be dead or redirected.
• Use --blacklist aggressively to strip static asset noise (images, fonts, CSS) that clutters results.
• Combine gau with waybackurls for slightly different data source coverage — results are not always identical.
• Use --subs on large organizations to surface forgotten subdomains along with their historical URLs.
• Extract parameterized URLs (grep '?') as a fast-start seed list for automated fuzzing tools like ffuf or sqlmap.

Troubleshooting

Issue	Cause	Fix
Very few/no results	Domain has little historical index presence	Try --providers individually to isolate which source responds
Rate-limited by a provider	Too many requests too fast	Reduce --threads, add --retries
Output dominated by static files	No blacklist applied	Add --blacklist png,jpg,gif,css,svg,woff
Command not found	Go bin path not in PATH	Add $(go env GOPATH)/bin to PATH
Stale/duplicate entries	Multiple providers overlap	Pipe through sort -u after collection

References

• Official GitHub repository: https://github.com/lc/gau
• Wayback Machine API: https://archive.org/help/wayback_api.php
• Common Crawl: https://commoncrawl.org/
• Kali tool page: https://www.kali.org/tools/gau/