cewl: Generates wordlists from web content

CeWL (Custom Word List generator) is a Ruby-based tool created by Robin Wood (digininja) that spiders a target website and builds a custom wordlist from the words it finds on the pages it crawls. The core idea is simple but powerful: organizations and individuals tend to use words that are meaningful to them — company names, product names, employee names, slogans, technical jargon — as the basis for their passwords. By crawling a target’s own website (or intranet, or public documents), CeWL builds a highly targeted wordlist that is often far more effective than a generic list like rockyou.txt for that specific target.

CeWL can also extract email addresses it finds during the crawl (-e), and it ships with a companion tool, FAB (Files Already Bagged), which extracts author/creator metadata from Office and PDF documents found on the site — useful for discovering employee usernames.

Key Features

• Configurable crawl depth and offsite-following behavior
• Minimum/maximum word length filtering
• Word count output (-c) to prioritize the most frequently used words
• Email address harvesting (-e)
• Metadata harvesting from documents via the bundled FAB tool
• Groups/permutation output for generating word pairs

Installation

CeWL is preinstalled on Kali Linux. To install/update manually:

sudo apt update
sudo apt install cewl -y

Verify:

cewl --help

Installing via RubyGems (if not using apt):

gem install cewl

From source:

git clone https://github.com/digininja/CeWL.git
cd CeWL
bundle install
ruby cewl.rb --help

Syntax

cewl [options] URL

Command-Line Options

Basic Usage

cewl http://example-corp.com -w wordlist.txt

Expected output:

CeWL 6.1 (Max Length) Robin Wood (robin@digi.ninja) (https://digi.ninja/)

(The wordlist itself is written silently into wordlist.txt; no words print to stdout unless -w is omitted.)

cat wordlist.txt

ExampleCorp
Solutions
Innovation
Careers
Contact
Support
Products
Enterprise
Cloud
Security

Practical Examples

Example 1 — Basic crawl with default depth

cewl http://example-corp.com -w corp_wordlist.txt

[*] 214 unique words written to corp_wordlist.txt

Example 2 — Deeper crawl with minimum word length filter

cewl -d 4 -m 5 http://example-corp.com -w corp_deep.txt

[*] 189 unique words (length >=5) written to corp_deep.txt

Example 3 — Word list with frequency counts

cewl -c http://example-corp.com

Word, Count
Security, 42
Cloud, 37
Enterprise, 29
Solutions, 24
Careers, 18

Example 4 — Extracting email addresses alongside the wordlist

cewl -e --email_file emails.txt http://example-corp.com -w corp_wordlist.txt

[*] 214 unique words written to corp_wordlist.txt
[*] 12 email addresses written to emails.txt

cat emails.txt

info@example-corp.com
support@example-corp.com
jsmith@example-corp.com

Example 5 — Extracting document metadata (author names) with FAB

cewl -a --meta_file authors.txt http://example-corp.com/downloads -w corp_wordlist.txt

[*] 3 metadata files processed
[*] Authors written to authors.txt: jsmith, mwilliams, agupta

Example 6 — Following offsite links matching a regex

cewl -o --allowed '(.*\.example-corp\.com.*)' http://example-corp.com -w corp_full.txt

[*] Following offsite links matching pattern
[*] 402 unique words written to corp_full.txt

Example 7 — Authenticated crawl behind basic auth

cewl --auth_type basic --auth_user employee --auth_pass Passw0rd http://intranet.example-corp.com -w intranet_words.txt

[*] Authenticated successfully, crawling intranet...
[*] 356 unique words written to intranet_words.txt

Example 8 — Generating multi-word groups (useful for passphrase guessing)

cewl -g 2 http://example-corp.com -w corp_pairs.txt

CloudSecurity
SecuritySolutions
EnterpriseCloud

Common Use Cases

• Building a company-specific wordlist for use with Hydra, Medusa, John the Ripper, or Hashcat
• Harvesting employee email addresses for phishing simulations or username lists
• Extracting document authorship metadata to discover valid internal usernames
• Feeding a targeted mangled wordlist into a mask/rule-based hash-cracking session
• Generating passphrase-style multi-word candidates (-g) for modern longer password policies

Automation with Bash

#!/bin/bash
# cewl-target-profile.sh — build a full wordlist + email + metadata profile of a target
TARGET="http://example-corp.com"
OUTDIR="./cewl_output"
mkdir -p "$OUTDIR"

cewl -d 3 -m 4 "$TARGET" -w "$OUTDIR/wordlist.txt"
cewl -e --email_file "$OUTDIR/emails.txt" -n "$TARGET"
cewl -a --meta_file "$OUTDIR/authors.txt" -n "$TARGET"

echo "[*] Profile complete. Files in $OUTDIR:"
ls -la "$OUTDIR"

#!/bin/bash
# cewl-to-hashcat.sh — build a CeWL wordlist and immediately feed it into Hashcat
cewl -d 3 -m 5 http://example-corp.com -w custom.txt
hashcat -m 1000 -a 0 ntlm_hashes.txt custom.txt -r /usr/share/hashcat/rules/best64.rule

Tips and Best Practices

• Combine CeWL output with a mutation tool (Hashcat rules, or john --wordlist=... --rules) — raw crawled words rarely match passwords exactly, but mutated (“Enterprise2024!”) often do.
• Use -c to see word frequency and prioritize the most commonly used terms for a shorter, higher-yield candidate list.
• Crawl depth (-d) beyond 2–3 rarely adds meaningfully better words but significantly increases crawl time — tune it to the size of the site.
• Always combine CeWL wordlists with the target organization’s name, product names, and known abbreviations added manually — CeWL misses branding that appears only in images/logos.
• Use -e early in an engagement — harvested email addresses double as a username list for Hydra/Medusa/Ncrack.

Troubleshooting

References

• Official repository: https://github.com/digininja/CeWL
• Author’s site: https://digi.ninja/projects/cewl.php
• Kali Linux tool page: https://www.kali.org/tools/cewl/