httpx: Complete Guide to HTTP Probing and Web Service Discovery Using Kali Linux

Abdul Wahab Junaid

4 weeks ago

httpx: Complete Guide to HTTP Probing and Web Service Discovery Using Kali Linux

httpx is a fast and multi-purpose HTTP toolkit developed by ProjectDiscovery, written in Go, designed to probe large lists of hosts and confirm which are running live HTTP/S services. Beyond simple liveness checks, httpx extracts a wealth of metadata per host: status codes, page titles, content length, technology stack (via built-in Wappalyzer-style fingerprinting), TLS certificate details, response headers, redirect chains, and more. It is built for speed at scale — capable of probing tens of thousands of hosts per minute — and is the standard “second stage” tool in most ProjectDiscovery-based recon pipelines, sitting between subdomain enumeration and vulnerability scanning (Nuclei).

How to Install

# Kali Linux
sudo apt update
sudo apt install httpx-toolkit -y   # Kali packages it as httpx-toolkit to avoid conflict with python httpx
httpx -version

# Or install via Go
go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest

Note: On Kali, the binary may be installed as httpx-toolkit to avoid a naming collision with the unrelated Python httpx HTTP client package. Verify with which httpx-toolkit or which httpx.

Syntax

httpx -u <target> [options]
cat targets.txt | httpx [options]

All Command-Line Options (Key Flags)

Option	Description
`-u, -target <url>`	Single target URL/host
`-l, -list <file>`	File containing list of hosts
`-silent`	Show only results, suppress banner
`-title`	Display page title
`-status-code`	Display HTTP status code
`-content-length`	Display response content length
`-tech-detect`	Display detected technologies
`-web-server`	Display server header
`-location`	Display redirect location
`-follow-redirects`	Follow HTTP redirects
`-follow-host-redirects`	Follow redirects only within the same host
`-tls-grab`	Grab TLS certificate details
`-cdn`	Detect CDN/WAF provider
`-ip`	Display resolved IP address
`-cname`	Display CNAME record
`-asn`	Display ASN information
`-ports <list>`	Probe specific ports (e.g., `80,443,8080`)
`-threads <n>`	Number of concurrent threads (default 50)
`-rate-limit <n>`	Max requests per second
`-timeout <secs>`	Request timeout
`-retries <n>`	Number of retries
`-mc, -match-code <codes>`	Match only specific status codes
`-fc, -filter-code <codes>`	Filter out specific status codes
`-mr, -match-regex <regex>`	Match responses via regex
`-o, -output <file>`	Write output to file
`-json`	Output results in JSON format
`-screenshot`	Capture screenshot (headless mode)
`-probe`	Display probe status (successful/failed)
`-random-agent`	Use a random User-Agent per request
`-H, -header <header>`	Custom HTTP header
`-x, -method <verb>`	HTTP method (GET, POST, HEAD, etc.)
`-proxy <url>`	Route requests through a proxy

Basic Usage (Expected Output in Bash)

$ echo "testphp.vulnweb.com" | httpx -silent

Output:

http://testphp.vulnweb.com

Practical Examples with Output

Example 1 — Probe with status code, title, and tech detection

$ echo "testphp.vulnweb.com" | httpx -silent -status-code -title -tech-detect

Output:

http://testphp.vulnweb.com [200] [Home of Acunetix Art] [Apache,PHP]

Example 2 — Bulk probing a subdomain list

$ cat subdomains.txt | httpx -silent -o live-hosts.txt

Output:

http://api.example.com
https://www.example.com
https://mail.example.com
[+] 3/50 hosts responded, saved to live-hosts.txt

Example 3 — Filter only 200 OK responses

$ cat subdomains.txt | httpx -silent -mc 200

Output:

https://www.example.com
https://blog.example.com

Example 4 — Probe specific ports across hosts

$ cat hosts.txt | httpx -silent -ports 80,443,8080,8443

Output:

http://192.168.1.10:8080
https://192.168.1.11:8443

Example 5 — JSON output with full metadata

$ echo "example.com" | httpx -silent -json | jq .

Output:

{
  "url": "https://example.com",
  "status_code": 200,
  "title": "Example Domain",
  "webserver": "cloudflare",
  "tech": ["Cloudflare", "HSTS"],
  "content_length": 1256
}

Example 6 — TLS certificate grabbing

$ echo "example.com" | httpx -silent -tls-grab -json | jq '.tls'

Output:

{
  "issuer_cn": "R3",
  "subject_cn": "example.com",
  "not_after": "2026-10-12T00:00:00Z"
}

Example 7 — Detect CDN/WAF provider

$ echo "example.com" | httpx -silent -cdn

Output:

https://example.com [cloudflare]

Example 8 — Screenshot capture

$ echo "example.com" | httpx -silent -screenshot -srd screenshots/

Output:

[+] Screenshot saved: screenshots/example.com.png

Example 9 — Chain with subfinder and Nuclei (full pipeline)

$ subfinder -d example.com -silent | httpx -silent | nuclei -silent -severity high,critical

Output:

[CVE-2023-XXXX] [http] [high] https://legacy.example.com

Example 10 — Custom threading and rate limiting for large scans

$ cat large-list.txt | httpx -silent -threads 100 -rate-limit 200 -o results.txt

Output:

[INF] Probing 10000 hosts with 100 threads at 200 req/s
[+] 843 live hosts found, saved to results.txt

Common Use Cases

Confirming which subdomains/hosts from a large enumeration list are actually live.
Enriching host lists with titles, status codes, and technology fingerprints before deeper scanning.
Detecting CDN/WAF presence at scale prior to running active scanners.
Screenshotting large numbers of live hosts for visual triage.
Serving as the connective “liveness + enrichment” stage between subdomain discovery and vulnerability scanning tools like Nuclei.

Automation with Bash

#!/bin/bash
# httpx-enrich.sh — probe a subdomain list and produce an enriched CSV report

INPUT="subdomains.txt"
OUTPUT="httpx-enriched.json"

cat "$INPUT" | httpx -silent -status-code -title -tech-detect -content-length \
    -tls-grab -cdn -json -o "$OUTPUT"

echo "[*] Building CSV summary..."
jq -r '[.url, .status_code, .title, (.tech // [] | join(";"))] | @csv' "$OUTPUT" > httpx-summary.csv

echo "[+] Done. Summary saved to httpx-summary.csv"
column -t -s, httpx-summary.csv | head -20

Tips and Best Practices

Always pipe httpx output into Nuclei/other tools rather than re-resolving hosts manually — it’s designed for pipeline chaining.
Use -mc/-fc to filter noise (e.g., excluding 404s) before passing results downstream.
Enable -tech-detect early to help decide which Nuclei template tags to run next.
Tune -threads and -rate-limit based on target sensitivity — internal assets can handle higher concurrency than public production systems.
Use -json output for all automation; human-readable output is for manual review only.

Troubleshooting

Issue	Cause	Fix
No output at all	Hosts unreachable or all filtered by `-mc`/`-fc`	Remove filters and re-test connectivity
Command not found	Kali package name mismatch	Try `httpx-toolkit` instead of `httpx`
Screenshot feature fails	Missing headless browser dependency	Install Chromium: `sudo apt install chromium`
Very slow on large lists	Low thread count	Increase `-threads` (test target tolerance first)
False tech-detect results	Fingerprint DB outdated	Update httpx binary to latest release

References

Official GitHub repository: https://github.com/projectdiscovery/httpx
Documentation: https://docs.projectdiscovery.io/tools/httpx/overview
Kali tool page: https://www.kali.org/tools/httpx/