spiderfoot: Automated OSINT tool for threat intelligence

Abdul Wahab Junaid

1 year ago

spiderfoot: Automated OSINT tool for threat intelligence

SpiderFoot is an automated OSINT reconnaissance tool that integrates with well over 200 data sources — including DNS, WHOIS, Shodan, Have I Been Pwned, social media, dark web indices, and threat intelligence feeds — to build a correlated intelligence picture of a target. It can be run as a command-line tool or, more commonly, as a locally hosted web application with an interactive graphical dashboard, module configuration screen, and visual entity-relationship graphs. SpiderFoot is designed to automate what would otherwise be dozens of manual individual lookups, making it a powerful tool for both attack-surface mapping and blue-team threat intelligence/brand monitoring.

Installation

# Kali/Debian (pre-installed, or reinstall)
sudo apt update && sudo apt install spiderfoot -y

# From source
git clone https://github.com/smicallef/spiderfoot.git
cd spiderfoot
pip3 install -r requirements.txt --break-system-packages
python3 sf.py -l 127.0.0.1:5001

Verify installation:

sf.py --help

Once installed, the web UI is started with:

spiderfoot -l 127.0.0.1:5001

Then browse to http://127.0.0.1:5001 in a local web browser.

Syntax

Command-line (sf.py):

sf.py [OPTIONS]

Web UI launch:

spiderfoot -l ADDRESS:PORT

Command-Line Options

Flag	Description
`-l ADDRESS:PORT`	Launch the SpiderFoot web UI listening on the given address/port
`-s TARGET`	Target to scan (domain, IP, email, username, etc.) in CLI scan mode
`-m MODULE1,MODULE2`	Comma-separated list of specific modules to run
`-t TYPE1,TYPE2`	Restrict scan to specific data element types (e.g., `EMAILADDR`, `IP_ADDRESS`)
`-u CASE`	Use case shortcut: `all`, `footprint`, `investigate`, `passive`
`-o FORMAT`	Output format: `tab`, `csv`, `json`
`-H`	Suppress column headers in tabular output
`-n`	Strip newlines from output
`-r`	Include the source data element that triggered each result
`-x`	Restrict to only export events that resolve strictly within scope of the target
`-q`	Quiet mode — suppress banner/logging noise
`-d`	Enable debug output
`-M`	List all available modules
`-T`	List all available data element types
`-C SCANID`	Resume/reference a previously run scan by its scan ID

Basic Usage

sf.py -s example.com -u footprint -o tab

Expected output:

[*] Starting scan for example.com (footprint use case)
Source            Type            Data
example.com        DNS_NS          a.iana-servers.net
example.com        DNS_TXT         v=spf1 -all
example.com        IP_ADDRESS      93.184.216.34
crt.sh             INTERNET_NAME   dev.example.com

Practical Examples

Example 1 — Launch the web UI

spiderfoot -l 127.0.0.1:5001

[*] Starting SpiderFoot 5.1 web server at 127.0.0.1:5001
[*] Open http://127.0.0.1:5001 in your browser to begin

Example 2 — CLI passive footprint scan

sf.py -s example.com -u passive -o tab

Source          Type            Data
crt.sh          INTERNET_NAME   www.example.com
virustotal      INTERNET_NAME   mail.example.com

Example 3 — Scan restricted to specific modules

sf.py -s example.com -m sfp_dnsresolve,sfp_crtsh -o tab

Source          Type            Data
DNS Resolver    IP_ADDRESS      93.184.216.34
crt.sh          INTERNET_NAME   dev.example.com

Example 4 — Scan restricted to a specific data type

sf.py -s example.com -t EMAILADDR -o tab

Source          Type            Data
Hunter.io       EMAILADDR       info@example.com
Hunter.io       EMAILADDR       support@example.com

Example 5 — Export scan results as JSON

sf.py -s example.com -u footprint -o json > results.json
cat results.json | head -5

[
  {"type": "IP_ADDRESS", "data": "93.184.216.34", "source": "sfp_dnsresolve"},
  {"type": "INTERNET_NAME", "data": "dev.example.com", "source": "sfp_crtsh"}
]

Example 6 — Investigate use case (deeper, more intrusive checks)

sf.py -s example.com -u investigate -o tab

Source              Type                Data
HaveIBeenPwned      EMAILADDR_COMPROMISED   admin@example.com (breach: Collection#1)
Shodan              TCP_PORT_OPEN       93.184.216.34:22

Example 7 — List all available modules

sf.py -M | head -10

sfp_abstractapi         Abstract API
sfp_alienvault          AlienVault OTX
sfp_binaryedge          BinaryEdge
sfp_crtsh               Certificate Transparency (crt.sh)
sfp_dnsresolve          DNS Resolver
sfp_dnsraw              Raw DNS Records
sfp_googlesearch        Google Search
sfp_hunter              Hunter.io

Example 8 — CSV output for spreadsheet analysis

sf.py -s example.com -u footprint -o csv > footprint.csv

"type","data","source"
"IP_ADDRESS","93.184.216.34","sfp_dnsresolve"
"INTERNET_NAME","dev.example.com","sfp_crtsh"

Common Use Cases

Automated, wide-coverage OSINT scanning combining hundreds of data sources in a single scan rather than manually querying each one.
Brand/executive monitoring for organizations wanting to detect data leaks, typosquatting domains, or exposed credentials tied to their brand.
Attack-surface discovery during the reconnaissance phase of a penetration test, especially useful for large or unfamiliar targets.
Threat intelligence correlation — linking IPs, domains, and email addresses associated with a known malicious actor or campaign.
Visualizing entity relationships (via the web UI’s graph view) to understand how disparate pieces of gathered intelligence connect.

Automation with Bash

Run a scheduled footprint scan and archive results with a timestamp:

#!/bin/bash
# spiderfoot_scheduled.sh
DOMAIN=$1
DATE=$(date +%F)
sf.py -s "$DOMAIN" -u footprint -o json > "spiderfoot_${DOMAIN}_${DATE}.json"
echo "[+] Scan complete: spiderfoot_${DOMAIN}_${DATE}.json"

Batch scan multiple targets from a list using the passive use case:

#!/bin/bash
while IFS= read -r target; do
    echo "[*] Scanning $target"
    sf.py -s "$target" -u passive -o csv > "sf_$(echo "$target" | tr '.' '_').csv"
done < targets.txt

Tips and Best Practices

Start new targets with the passive use case to avoid any active interaction with the target’s infrastructure; escalate to footprint/investigate only with proper authorization.
Configure API keys in the web UI’s settings page (or ~/.spiderfoot/spiderfoot.conf) for premium modules like Shodan, Censys, and Have I Been Pwned — many of the most valuable modules are API-key-gated.
Use the web UI’s graph visualization for client presentations; it communicates relationships between findings far more effectively than raw CLI tabular output.
Use -x to strictly scope exported results to the target when a broad scan pulls in tangentially related but out-of-scope entities.
Schedule recurring scans (via cron calling sf.py) for ongoing brand/asset monitoring rather than running one-off manual scans.

Troubleshooting

Problem	Cause	Fix
Web UI won’t load in browser	Server bound to wrong interface/port, or firewall blocking local port	Confirm with `-l 127.0.0.1:5001` and check `netstat -tlnp
Scan takes an extremely long time	`all`/`investigate` use case enables every module including slow ones	Use `-m` to restrict to specific fast modules, or use the `footprint`/`passive` use cases
Many modules show “not enabled” or produce no results	Missing API keys for those modules	Add relevant API keys via web UI Settings → Module Settings
`ModuleNotFoundError` when running from source	Missing Python dependencies	Run `pip3 install -r requirements.txt --break-system-packages` again, ensure correct Python version
Duplicate/noisy results from tangential data	Scan not scoped tightly enough	Use `-x` to restrict output strictly to elements within target scope

References

Official GitHub repository: https://github.com/smicallef/spiderfoot
Official project site and documentation: https://www.spiderfoot.net/
Kali Linux tool page: https://www.kali.org/tools/spiderfoot/