httpx: Complete Guide to HTTP Probing and Web Service Discovery Using Kali Linux

httpx is a fast and multi-purpose HTTP toolkit developed by ProjectDiscovery, written in Go, designed to probe large lists of hosts and confirm which are running live HTTP/S services. Beyond simple liveness checks, httpx extracts a wealth of metadata per host: status codes, page titles, content length, technology stack (via built-in Wappalyzer-style fingerprinting), TLS certificate details, response headers, redirect chains, and more. It is built for speed at scale — capable of probing tens of thousands of hosts per minute — and is the standard “second stage” tool in most ProjectDiscovery-based recon pipelines, sitting between subdomain enumeration and vulnerability scanning (Nuclei).

How to Install

# Kali Linux
sudo apt update
sudo apt install httpx-toolkit -y   # Kali packages it as httpx-toolkit to avoid conflict with python httpx
httpx -version

# Or install via Go
go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest

Note: On Kali, the binary may be installed as httpx-toolkit to avoid a naming collision with the unrelated Python httpx HTTP client package. Verify with which httpx-toolkit or which httpx.

Syntax

httpx -u <target> [options]
cat targets.txt | httpx [options]

All Command-Line Options (Key Flags)

Option	Description
-u, -target <url>	Single target URL/host
-l, -list <file>	File containing list of hosts
-silent	Show only results, suppress banner
-title	Display page title
-status-code	Display HTTP status code
-content-length	Display response content length
-tech-detect	Display detected technologies
-web-server	Display server header
-location	Display redirect location
-follow-redirects	Follow HTTP redirects
-follow-host-redirects	Follow redirects only within the same host
-tls-grab	Grab TLS certificate details
-cdn	Detect CDN/WAF provider
-ip	Display resolved IP address
-cname	Display CNAME record
-asn	Display ASN information
-ports <list>	Probe specific ports (e.g., 80,443,8080)
-threads <n>	Number of concurrent threads (default 50)
-rate-limit <n>	Max requests per second
-timeout <secs>	Request timeout
-retries <n>	Number of retries
-mc, -match-code <codes>	Match only specific status codes
-fc, -filter-code <codes>	Filter out specific status codes
-mr, -match-regex <regex>	Match responses via regex
-o, -output <file>	Write output to file
-json	Output results in JSON format
-screenshot	Capture screenshot (headless mode)
-probe	Display probe status (successful/failed)
-random-agent	Use a random User-Agent per request
-H, -header <header>	Custom HTTP header
-x, -method <verb>	HTTP method (GET, POST, HEAD, etc.)
-proxy <url>	Route requests through a proxy

Basic Usage (Expected Output in Bash)

$ echo "testphp.vulnweb.com" | httpx -silent

Output:

http://testphp.vulnweb.com

Practical Examples with Output

Example 1 — Probe with status code, title, and tech detection

$ echo "testphp.vulnweb.com" | httpx -silent -status-code -title -tech-detect

Output:

http://testphp.vulnweb.com [200] [Home of Acunetix Art] [Apache,PHP]

Example 2 — Bulk probing a subdomain list

$ cat subdomains.txt | httpx -silent -o live-hosts.txt

Output:

http://api.example.com
https://www.example.com
https://mail.example.com
[+] 3/50 hosts responded, saved to live-hosts.txt

Example 3 — Filter only 200 OK responses

$ cat subdomains.txt | httpx -silent -mc 200

Output:

https://www.example.com
https://blog.example.com

Example 4 — Probe specific ports across hosts

$ cat hosts.txt | httpx -silent -ports 80,443,8080,8443

Output:

http://192.168.1.10:8080
https://192.168.1.11:8443

Example 5 — JSON output with full metadata

$ echo "example.com" | httpx -silent -json | jq .

Output:

{
  "url": "https://example.com",
  "status_code": 200,
  "title": "Example Domain",
  "webserver": "cloudflare",
  "tech": ["Cloudflare", "HSTS"],
  "content_length": 1256
}

Example 6 — TLS certificate grabbing

$ echo "example.com" | httpx -silent -tls-grab -json | jq '.tls'

Output:

{
  "issuer_cn": "R3",
  "subject_cn": "example.com",
  "not_after": "2026-10-12T00:00:00Z"
}

Example 7 — Detect CDN/WAF provider

$ echo "example.com" | httpx -silent -cdn

Output:

https://example.com [cloudflare]

Example 8 — Screenshot capture

$ echo "example.com" | httpx -silent -screenshot -srd screenshots/

Output:

[+] Screenshot saved: screenshots/example.com.png

Example 9 — Chain with subfinder and Nuclei (full pipeline)

$ subfinder -d example.com -silent | httpx -silent | nuclei -silent -severity high,critical

Output:

[CVE-2023-XXXX] [http] [high] https://legacy.example.com

Example 10 — Custom threading and rate limiting for large scans

$ cat large-list.txt | httpx -silent -threads 100 -rate-limit 200 -o results.txt

Output:

[INF] Probing 10000 hosts with 100 threads at 200 req/s
[+] 843 live hosts found, saved to results.txt

Common Use Cases

• Confirming which subdomains/hosts from a large enumeration list are actually live.
• Enriching host lists with titles, status codes, and technology fingerprints before deeper scanning.
• Detecting CDN/WAF presence at scale prior to running active scanners.
• Screenshotting large numbers of live hosts for visual triage.
• Serving as the connective “liveness + enrichment” stage between subdomain discovery and vulnerability scanning tools like Nuclei.

Automation with Bash

#!/bin/bash
# httpx-enrich.sh — probe a subdomain list and produce an enriched CSV report

INPUT="subdomains.txt"
OUTPUT="httpx-enriched.json"

cat "$INPUT" | httpx -silent -status-code -title -tech-detect -content-length \
    -tls-grab -cdn -json -o "$OUTPUT"

echo "[*] Building CSV summary..."
jq -r '[.url, .status_code, .title, (.tech // [] | join(";"))] | @csv' "$OUTPUT" > httpx-summary.csv

echo "[+] Done. Summary saved to httpx-summary.csv"
column -t -s, httpx-summary.csv | head -20

Tips and Best Practices

• Always pipe httpx output into Nuclei/other tools rather than re-resolving hosts manually — it’s designed for pipeline chaining.
• Use -mc/-fc to filter noise (e.g., excluding 404s) before passing results downstream.
• Enable -tech-detect early to help decide which Nuclei template tags to run next.
• Tune -threads and -rate-limit based on target sensitivity — internal assets can handle higher concurrency than public production systems.
• Use -json output for all automation; human-readable output is for manual review only.

Troubleshooting

Issue	Cause	Fix
No output at all	Hosts unreachable or all filtered by -mc/-fc	Remove filters and re-test connectivity
Command not found	Kali package name mismatch	Try httpx-toolkit instead of httpx
Screenshot feature fails	Missing headless browser dependency	Install Chromium: sudo apt install chromium
Very slow on large lists	Low thread count	Increase -threads (test target tolerance first)
False tech-detect results	Fingerprint DB outdated	Update httpx binary to latest release

References

• Official GitHub repository: https://github.com/projectdiscovery/httpx
• Documentation: https://docs.projectdiscovery.io/tools/httpx/overview
• Kali tool page: https://www.kali.org/tools/httpx/