Katana: Complete Guide to Web Crawling and Attack Surface Discovery Using Kali Linux

Abdul Wahab Junaid

4 weeks ago

Katana: Complete Guide to Web Crawling and Attack Surface Discovery Using Kali Linux

katana is a next-generation crawling and spidering framework developed by ProjectDiscovery, written in Go, designed to handle both traditional server-rendered HTML sites and modern JavaScript-heavy Single Page Applications (SPAs). Unlike simpler crawlers, katana supports a headless browsing mode (via chromedp) that fully renders JavaScript before extracting links, form actions, and API calls — making it far more effective against React/Angular/Vue applications. It also supports scope control, output filtering, passive-source integration, and direct piping into other ProjectDiscovery tools like httpx and Nuclei.

How to Install

# Kali Linux
sudo apt update
sudo apt install katana -y
katana -version

# Or via Go
go install github.com/projectdiscovery/katana/cmd/katana@latest

Headless mode requires Chromium:

sudo apt install chromium -y

Syntax

katana -u <target> [options]
katana -list <targets_file> [options]

All Command-Line Options (Key Flags)

Option	Description
`-u, -list <url/file>`	Target URL or file with list of URLs
`-d, -depth <n>`	Maximum crawl depth (default 3)
`-jc, -js-crawl`	Enable crawling/parsing of JavaScript files
`-hl, -headless`	Enable headless browser-based crawling
`-sc, -system-chrome`	Use local installed Chrome instead of downloading one
`-xhr, -xhr-extraction`	Extract XHR/API requests during headless crawl
`-jsluice`	Use jsluice for deeper JS endpoint extraction
`-c, -concurrency <n>`	Concurrent workers (default 10)
`-p, -parallelism <n>`	Concurrent parallel crawls
`-rd, -rate-limit-domain <n>`	Rate limit per domain
`-rl, -rate-limit <n>`	Global rate limit (requests/sec)
`-kf, -known-files <type>`	Fetch known files (`robotstxt`, `sitemapxml`, `all`)
`-fs, -field-scope <scope>`	Scope: `rdn` (root domain), `fqdn`, `dn`
`-cs, -crawl-scope <regex>`	Regex-based crawl scope
`-cos, -crawl-out-scope <regex>`	Exclude regex from crawl scope
`-mdc, -match-condition <cond>`	Match based on custom DSL condition
`-f, -field <field>`	Extract specific field: url, qurl, path, fqdn, rdn, rurl
`-em, -extension-match <ext>`	Match specific extensions
`-ef, -extension-filter <ext>`	Filter out specific extensions
`-o, -output <file>`	Write output to file
`-j, -jsonl`	Output as JSON Lines
`-silent`	Suppress banner, show only results
`-nc, -no-color`	Disable colored output
`-timeout <secs>`	Request timeout
`-retry <n>`	Retry count on failure
`-proxy <url>`	Route requests through proxy
`-H, -headers <header>`	Custom HTTP headers
`-form-extraction`	Extract HTML form details
`-tls-impersonate`	TLS fingerprint randomization/impersonation
`-aff, -automatic-form-fill`	Automatically fill and submit discovered forms

Basic Usage (Expected Output in Bash)

$ katana -u https://testphp.vulnweb.com

Output:

   __        __                 
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/  v1.1.2

		projectdiscovery.io

[INF] Current katana version: v1.1.2
https://testphp.vulnweb.com/
https://testphp.vulnweb.com/login.php
https://testphp.vulnweb.com/search.php?test=query
https://testphp.vulnweb.com/categories.php

Practical Examples with Output

Example 1 — Headless crawl of a JavaScript SPA

$ katana -u https://example-spa.com -headless -jc

Output:

[INF] Starting headless crawl...
https://example-spa.com/
https://example-spa.com/api/v1/products
https://example-spa.com/api/v1/user/profile
https://example-spa.com/static/app.bundle.js

Example 2 — Extract XHR/API requests specifically

$ katana -u https://example-spa.com -headless -xhr -jsonl | jq '.request.endpoint'

Output:

"https://example-spa.com/api/v1/login"
"https://example-spa.com/api/v1/cart"

Example 3 — Crawl with scope restricted to root domain

$ katana -u https://example.com -fs rdn -d 4

Output:

https://example.com/
https://example.com/blog/post-1
https://sub.example.com/dashboard

Example 4 — Extract only unique query-string URLs

$ katana -u https://testphp.vulnweb.com -f qurl

Output:

https://testphp.vulnweb.com/search.php?test=query
https://testphp.vulnweb.com/artists.php?artist=1
https://testphp.vulnweb.com/listproducts.php?cat=1

Example 5 — Filter by file extension (only JS files)

$ katana -u https://example.com -em js

Output:

https://example.com/static/main.js
https://example.com/static/vendor.js

Example 6 — Crawl a list of targets and save JSONL

$ katana -list targets.txt -jsonl -o katana-results.jsonl
$ jq -r '.request.endpoint' katana-results.jsonl | head -3

Output:

https://target1.com/
https://target1.com/about
https://target2.com/login

Example 7 — Known-files discovery (robots.txt, sitemap.xml)

$ katana -u https://example.com -kf all

Output:

[INF] Found robots.txt with 12 disallowed paths
[INF] Found sitemap.xml with 84 URLs
https://example.com/admin (from robots.txt)
https://example.com/archive/2024 (from sitemap.xml)

Example 8 — Route through Burp for combined manual/automated recon

$ katana -u https://testphp.vulnweb.com -proxy http://127.0.0.1:8080

Output:

[INF] Proxy configured: http://127.0.0.1:8080
https://testphp.vulnweb.com/login.php

Example 9 — Chain with httpx and Nuclei

$ katana -u https://testphp.vulnweb.com -silent | httpx -silent | nuclei -silent -tags xss,sqli

Output:

[sqli-error-based] [http] [high] https://testphp.vulnweb.com/artists.php?artist=1

Example 10 — Automatic form filling and submission

$ katana -u https://testphp.vulnweb.com -aff -form-extraction

Output:

[INF] Discovered form: /search.php (method=GET, field=searchFor)
[INF] Auto-filled and submitted form
https://testphp.vulnweb.com/search.php?searchFor=test&goButton=go

Common Use Cases

Deep crawling of modern JavaScript SPAs where traditional crawlers fail to find links.
API endpoint discovery via XHR/fetch interception during headless rendering.
Building comprehensive, scope-restricted URL inventories for large bug bounty targets.
Extracting and testing HTML forms automatically for injection points.
Feeding structured JSONL crawl data into custom analysis or reporting pipelines.

Automation with Bash

#!/bin/bash
# katana-full-crawl.sh — headless crawl + API extraction + Nuclei scan pipeline

TARGET="$1"
OUTDIR="katana-scan-$(date +%Y%m%d)"
mkdir -p "$OUTDIR"

echo "[*] Running headless crawl with XHR extraction..."
katana -u "$TARGET" -headless -jc -xhr -jsonl -o "$OUTDIR/crawl.jsonl" -silent

echo "[*] Extracting unique endpoints..."
jq -r '.request.endpoint' "$OUTDIR/crawl.jsonl" | sort -u > "$OUTDIR/endpoints.txt"

echo "[*] Probing endpoints with httpx..."
cat "$OUTDIR/endpoints.txt" | httpx -silent -o "$OUTDIR/live-endpoints.txt"

echo "[*] Running Nuclei against live endpoints..."
nuclei -l "$OUTDIR/live-endpoints.txt" -silent -o "$OUTDIR/nuclei-findings.txt"

echo "[+] Pipeline complete. Findings: $OUTDIR/nuclei-findings.txt"

Tips and Best Practices

Use -headless and -jc together for full JavaScript-rendered coverage of modern web apps.
Restrict scope with -fs rdn or -cs/-cos to prevent crawling unrelated third-party domains.
Use -kf all early in a crawl to quickly pick up robots.txt/sitemap.xml disclosed paths.
Combine -xhr with headless mode specifically when targeting API-driven SPA back ends.
Pipe results directly into httpx and Nuclei for a compact, efficient end-to-end pipeline.

Troubleshooting

Issue	Cause	Fix
Headless mode fails to launch	Missing Chromium	`sudo apt install chromium` and retry, or use `-sc` for system Chrome
Crawl misses SPA routes	JS not executed	Ensure `-headless -jc` are both enabled
Too many out-of-scope results	No scope restriction set	Use `-fs rdn` or `-cs` with a regex
Extremely slow crawl	High depth + headless overhead	Reduce `-d` depth or increase `-c` concurrency carefully
Rate-limited/blocked by target	No rate limiting configured	Set `-rl` and `-rd` to reasonable values

References

Official GitHub repository: https://github.com/projectdiscovery/katana
Documentation: https://docs.projectdiscovery.io/tools/katana/overview
Kali tool page: https://www.kali.org/tools/katana/