Offensive Security Researcher & Quantum Cryptography Analyst

PasteHunter: A Practical Guide to Automated Paste Site Monitoring for Leaked Data

Abdul Wahab Junaid
PasteHunter: A Practical Guide to Automated Paste Site Monitoring for Leaked Data

I got interested in PasteHunter after seeing yet another breach postmortem where the first public sign of a leak was a chunk of source code and an AWS key sitting on a paste site for hours before anyone noticed. That’s exactly the gap PasteHunter is built to close — it continuously scans paste sites (Pastebin, GitHub Gists, and others) using YARA rules designed to catch credentials, API keys, and other sensitive data the moment they’re posted. In this guide, I’ll walk through the tool end to end, and every command, config snippet, and output shown below was actually run and verified against the real project source, not written from assumption.

What PasteHunter Is and Why It Exists

PasteHunter is an open-source Python application, originally created by security researcher Kevin Breen (@KevTheHermit) with contributions from @Plazmaz, that automates the tedious job of manually refreshing paste sites looking for leaked credentials, database dumps, or proprietary source code. Instead of a human watching Pastebin’s “recent pastes” feed, PasteHunter continuously polls multiple paste-style sources, runs every new paste through a bank of YARA detection rules, and forwards anything matching a rule to one or more output destinations — Elasticsearch for a searchable dashboard, Slack for real-time alerts, a SIEM via syslog, and several others.

It’s fundamentally a detection and alerting daemon, not an interactive scanning tool you run once and walk away from. That distinction matters a lot for how you configure and operate it, and I’ll come back to it throughout this guide.

I cloned the actual repository to verify everything in this article:

git clone https://github.com/kevthehermit/PasteHunter.git

The version I tested against is 1.4.2 (confirmed directly from setup.py), with the daemon script itself internally reporting version string 1.4.0 — a small inconsistency in the project’s own versioning between the packaging metadata and the runtime banner, worth knowing so you’re not confused when the two don’t match in your own testing.

Architecture: How PasteHunter Actually Works

PasteHunter’s design is refreshingly modular, and understanding the four-layer architecture makes both configuration and troubleshooting much easier.

1. Input modules — Each supported paste site has its own input module implementing a common interface. I verified the real, current input module list directly from the source tree:

pastehunter/inputs/
├── base_input.py       (shared interface all inputs implement)
├── dumpz.py            (dumpz.org — noted in config as deprecated/API removed)
├── gists.py            (GitHub Gists)
├── github.py           (GitHub code search)
├── ixio.py             (ix.io)
├── pastebin.py         (Pastebin.com)
├── slexy.py            (Slexy.org)
└── stackexchange.py    (Stack Exchange posts)

Each module implements a recent_pastes() function that fetches newly published items since the last poll, using a persisted history file (paste_history.tmp) to avoid reprocessing the same paste twice.

2. YARA scanning core — Every fetched paste’s raw text is run against a compiled set of YARA rules using the yara-python binding. YARA is a pattern-matching engine originally built for malware research, but it’s equally effective here for matching structured secrets like AWS access key formats, private key headers, or database connection strings.

3. Post-processing modules — Optional enrichment steps that run after a YARA match, before the result is sent to an output. I verified the real post-processing modules shipped with the project:

pastehunter/postprocess/
├── post_b64.py         (decodes detected base64 blobs)
├── post_compress.py    (handles compressed paste content)
├── post_email.py       (extracts and normalizes email addresses)
└── post_entropy.py     (calculates Shannon entropy of the raw paste)

4. Output modules — Where a match gets sent once detected. I confirmed the real list from the source tree:

pastehunter/outputs/
├── csv_output.py
├── elastic_output.py
├── http_output.py
├── json_output.py
├── slack_output.py
├── smtp_output.py
├── splunk_output.py
├── syslog_output.py
└── twilio_output.py

That’s nine independent output integrations, meaning you can fan a single detection out to Elasticsearch for long-term search, Slack for immediate team visibility, and SMTP for a formal alert email, all from one config file.

5. The main loop — I read the actual orchestration logic in pastehunter-cli to confirm exactly how execution flows: it compiles all enabled YARA rule files into a single ruleset, then loops indefinitely — populating a multiprocessing pool with fetch jobs for each enabled input, scanning every fetched paste against the compiled rules, dispatching matches to enabled outputs, persisting the paste history, then sleeping for a configurable interval before repeating.

Installation

PasteHunter’s dependencies are declared in requirements.txt, which I read directly from the repo:

requests>=2.20.0
elasticsearch>=5.0.0,<6.0.0
splunk-sdk
yara-python

Note that elasticsearch and splunk-sdk are only actually needed if you plan to use those specific output modules — I confirmed this by testing without them installed (see the troubleshooting section below, where I reproduce the exact failure this causes).

Minimal install for core functionality (YARA scanning plus lightweight local outputs like JSON/CSV):

git clone https://github.com/kevthehermit/PasteHunter.git
cd PasteHunter
pip3 install yara-python requests --break-system-packages

I ran this exact command and it completed cleanly:

Downloading yara_python-4.5.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)
Installing collected packages: yara-python
Successfully installed yara-python-4.5.4

Full install (all outputs including Elasticsearch and Splunk):

pip3 install -r requirements.txt --break-system-packages

Docker is also officially supported — the repo ships both a Dockerfile and a docker-compose.yml:

docker-compose up -d

This is genuinely the easier path if you want Elasticsearch and Kibana running alongside PasteHunter for a full searchable dashboard, since docker-compose.yml wires up the whole stack in one command.

Configuration: The Part Everyone Gets Wrong the First Time

This is the single most important thing I discovered while testing PasteHunter, and it will save you a genuinely confusing debugging session: the repository ships a settings.json.sample file in its root directory, but the running application does not read a local settings.json at all.

I confirmed this directly by reading the real config-loading function in pastehunter/common.py:

def parse_config():
    conf = None
    settings_file = os.path.join(home, ".config", "pastehunter.json")

    if os.path.exists(settings_file):
        conf_file = settings_file
    else:
        conf_file = None

    if conf_file:
        try:
            with open(conf_file, 'r') as read_conf:
                conf = json.load(read_conf)
        except Exception as e:
            logger.error("Unable to parse config file: {0}".format(e))
    else:
        logger.error("Unable to read config file '~/.config/pastehunter.json'")

    return conf

It hardcodes the path to ~/.config/pastehunter.json — your home directory’s config folder — and nothing else. I reproduced the actual failure that happens if you skip this step:

cp settings.json.sample settings.json
python3 pastehunter-cli

Actual captured output:

INFO:pastehunter-cli:Starting PasteHunter Version: 1.4.0
INFO:pastehunter-cli:Reading Configs
ERROR:common.py:Unable to read config file '~/.config/pastehunter.json'

The fix, which I verified works correctly:

mkdir -p ~/.config
cp settings.json.sample ~/.config/pastehunter.json

Understanding the real settings.json structure

I parsed the actual sample config programmatically to give you the true structure rather than a guessed-at one:

import json
with open('settings.json') as f:
    cfg = json.load(f)
print('Input modules:', list(cfg['inputs'].keys()))

Verified output:

Input modules: ['pastebin', 'ixio', 'dumpz', 'gists', 'github', 'slexy', 'stackexchange']

Each input block follows a consistent pattern. Here’s the real Pastebin section, read directly from the sample config:

"pastebin": {
    "enabled": true,
    "module": "pastehunter.inputs.pastebin",
    "api_scrape": "https://scrape.pastebin.com/api_scraping.php",
    "api_raw": "https://scrape.pastebin.com/api_scrape_item.php?i=",
    "paste_limit": 100,
    "store_all": false
}

One thing worth knowing about the Pastebin module specifically: I read its make_request() implementation and confirmed it explicitly checks for an IP-whitelisting rejection message from Pastebin’s scraping API:

def make_request(self, url, timeout=10, headers=None):
    paste_list_request = super().make_request(url, timeout, headers)
    if 'DOES NOT HAVE ACCESS' in paste_list_request.text:
        logger.error("Your IP is not whitelisted visits 'https://pastebin.com/doc_scraping_api'")
        return None
    return paste_list_request

This means the Pastebin scraping API requires your server’s IP to be explicitly whitelisted by Pastebin before it will return results — a common source of “it’s just not fetching anything” confusion for first-time users, and one you’ll only discover by reading the source, as I did here, since the error is logged but easy to miss in a busy log stream.

Output modules follow the same enable/module pattern. Here’s the real json_output block:

"json_output": {
    "enabled": false,
    "module": "pastehunter.outputs.json_output",
    "classname": "JsonOutput",
    "output_path": "logs/json/",
    "store_raw": true,
    "encode_raw": true
}

And the general execution settings:

"general": {
    "run_frequency": 300,
    "process_timeout": 5
}

run_frequency controls the sleep interval (in seconds) between polling cycles — 300 seconds (5 minutes) by default.

Running PasteHunter: A Fully Verified Execution

I put together a safe, network-free test to verify the real startup and main-loop mechanics without hitting any live paste sites — a good way to sanity-check your own install before enabling real inputs. I disabled every input module and enabled only the local json_output:

mkdir -p ~/.config logs/json
python3 -c "
import json
with open('settings.json') as f:
    cfg = json.load(f)
for k in cfg['inputs']:
    cfg['inputs'][k]['enabled'] = False
cfg['elastic_output']['enabled'] = False
cfg['json_output']['enabled'] = True
cfg['general']['run_frequency'] = 2
with open('settings.json', 'w') as f:
    json.dump(cfg, f, indent=2)
"
cp settings.json ~/.config/pastehunter.json
timeout 6 python3 pastehunter-cli

Actual, complete, verified output:

INFO:pastehunter-cli:Starting PasteHunter Version: 1.4.0
INFO:pastehunter-cli:Reading Configs
INFO:pastehunter-cli:Logging to file disabled.
INFO:pastehunter-cli:Setting Log Level to 20
INFO:pastehunter-cli:Configure Inputs
INFO:pastehunter-cli:Configure Outputs
INFO:pastehunter-cli:Enabled Output: json_output
INFO:pastehunter-cli:Compile Yara Rules
INFO:pastehunter-cli:Adding rules from aws.yar
INFO:pastehunter-cli:Adding rules from email_filter.yar
INFO:pastehunter-cli:Adding rules from core_keywords.yar
INFO:pastehunter-cli:Adding rules from password_leak.yar
INFO:pastehunter-cli:Adding rules from powershell.yar
INFO:pastehunter-cli:Enable Blacklist Rules
INFO:pastehunter-cli:Adding rules from blacklist.yar
INFO:pastehunter-cli:Adding rules from database.yar
INFO:pastehunter-cli:Adding rules from CryptoExchangeApi.yar
INFO:pastehunter-cli:Adding rules from api_keys.yar
INFO:pastehunter-cli:Adding rules from general.yar
INFO:pastehunter-cli:Adding rules from hak5.yar
INFO:pastehunter-cli:Adding rules from certificates.yar
INFO:pastehunter-cli:Adding rules from base64.yar
INFO:pastehunter-cli:Populating Queue
INFO:pastehunter-cli:Added 0 Items to the queue
INFO:pastehunter-cli:Sleeping for 2 Seconds
INFO:pastehunter-cli:Populating Queue
INFO:pastehunter-cli:Added 0 Items to the queue
INFO:pastehunter-cli:Sleeping for 2 Seconds

This confirms the real startup sequence: config load, output registration, YARA rule compilation across 13 real rule files, then the infinite fetch/scan/sleep loop. “Added 0 Items to the queue” is expected and correct here since every input was intentionally disabled — with real inputs enabled and network access to the paste sites, this number reflects actual newly discovered pastes each cycle.

YARA Rules: The Detection Engine, Verified Live

This is where PasteHunter’s real value lives. I want to show you an actual, working detection rather than just describing the concept. Here’s the real, unmodified aws.yar rule shipped with the project:

rule aws_cli
{
    meta:
        author = "@KevTheHermit"
        info = "Part of PasteHunter"
        reference = "https://github.com/kevthehermit/PasteHunter"

    strings:
        $a1 = "aws s3 " ascii
        $a2 = "aws ec2 " ascii
        $a3 = "aws ecr " ascii
        $a4 = "aws cognito-identity" ascii
        $a5 = "aws iam "ascii
        $a6 = "aws waf " ascii

    condition:
        any of them
}

rule sw_bucket
{
    meta:
        author = "@KevTheHermit"
        info = "Part of PasteHunter"
        reference = "https://github.com/kevthehermit/PasteHunter"

    strings:
        $a1 = "s3.amazonaws.com" ascii

    condition:
        any of them
}

I compiled this exact rule file with the real yara-python engine and ran it against a synthetic sample paste to prove the detection logic genuinely works:

import yara

rules = yara.compile(filepath='pastehunter/YaraRules/aws.yar')

sample_paste = '''
Deploying my new project, quick backup commands:
aws s3 sync ./build s3://my-demo-bucket/releases/
aws ec2 describe-instances --region us-east-1
Also check https://my-demo-bucket.s3.amazonaws.com/config.json
'''

matches = rules.match(data=sample_paste)
for m in matches:
    print('Rule matched:', m.rule)
    for string_match in m.strings:
        print('  ', string_match)

Actual verified output:

Rule matched: aws_cli
   $a1
   $a2
Rule matched: sw_bucket
   $a1

This is a genuine, reproducible detection — real engine, real shipped rule, real match. It’s a good template for testing any custom rule you write before deploying it live.

Here’s another real rule I verified, generic_api from api_keys.yar, which is a good example of combined-condition logic (requires both an API-key-like keyword and a hash-shaped string, while excluding known false-positive patterns):

rule generic_api
{
    meta:
        author = "@KevTheHermit"
        info = "Part of PasteHunter"

    strings:
        $a1 = "apikey" nocase
        $a2 = "api_key" nocase
        $hash32 = /\b[a-fA-F\d]{32}\b/
        $hash64 = /\b[a-fA-F\d]{64}\b/
        $n1 = "#EXTINF"
        $n2 = "m3u8"
        $n3 = "Chocolatey is running"

    condition:
        (any of ($a*)) and (any of ($hash*)) and (not any of ($n*))
}

The not any of ($n*) clause is a deliberate false-positive filter — the rule’s author noticed that IPTV playlist files (#EXTINF, .m3u8) and Chocolatey package manager logs both commonly contain 32/64-character hex strings that aren’t actually API keys, so they’re explicitly excluded from matching.

Entropy-Based Detection

Beyond pattern matching, PasteHunter includes a Shannon entropy calculator as a post-processing enrichment step, useful for flagging high-randomness strings (a strong indicator of secrets, keys, or tokens) even when they don’t match a specific known key format. I ran the actual function from post_entropy.py directly:

import sys
sys.path.insert(0, '.')
from pastehunter.postprocess.post_entropy import shannon_entropy

sample_normal = 'Hello this is a normal english sentence about deploying code.'
sample_key = 'AKIAIOSFODNN7EXAMPLEwJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'

print('Normal text entropy:', round(shannon_entropy(sample_normal), 3))
print('Key-like string entropy:', round(shannon_entropy(sample_key), 3))

Actual verified output:

Normal text entropy: 4.021
Key-like string entropy: 4.683

The higher entropy value for the key-like string reflects its greater character-level randomness compared to natural language — this is exactly the signal analysts use to build automated “does this look like a secret” heuristics on top of literal pattern matching, and it’s a genuinely effective complement to YARA’s exact-match approach for catching keys in formats not yet covered by a specific rule.

Real-World Digital Forensics and Threat Intelligence Use Cases

Breach early-warning. Security teams run PasteHunter continuously against their own organization’s known keywords (company name, internal hostnames, product names) as custom YARA rules, so that if an employee or attacker posts internal credentials to a paste site, the security team gets a Slack alert within one polling cycle instead of finding out from a journalist weeks later.

Bug bounty and red team OSINT. During authorized engagements, some researchers run PasteHunter with rules tuned to a specific target’s domain names and internal-sounding identifiers, since accidental credential leaks via paste sites remain a genuinely common finding class in bug bounty programs.

Digital forensics investigations. When investigating a suspected data exfiltration incident, matching entropy-flagged pastes or specific YARA hits against the incident timeframe can corroborate whether stolen data was actually posted publicly, and the JSON/CSV output modules make it straightforward to hand structured evidence to a forensic case management system.

Threat intelligence enrichment. The Elasticsearch output, paired with Kibana, turns PasteHunter into a searchable historical archive of every flagged paste — genuinely useful for retroactively checking “was our data ever posted here” during a new incident, since the archive persists long after the original paste itself may have been deleted from the source site.

Integration with Other Security Tools

Performance Optimization

Troubleshooting

“Unable to read config file ‘~/.config/pastehunter.json'” — As demonstrated above, this is the single most common first-run error. Fix: mkdir -p ~/.config && cp settings.json.sample ~/.config/pastehunter.json.

ModuleNotFoundError: No module named 'elasticsearch' — I reproduced this exact error directly:

File "/home/claude/PasteHunter/pastehunter/outputs/elastic_output.py", line 1, in <module>
    from elasticsearch import Elasticsearch
ModuleNotFoundError: No module named 'elasticsearch'

This happens whenever elastic_output.enabled is true in your config but you only installed the minimal dependency set. Fix: either pip3 install elasticsearch (matching the version constraint in requirements.txt, >=5.0.0,<6.0.0) or set elastic_output.enabled to false if you don’t need it.

“Your IP is not whitelisted” — Specific to the Pastebin input module, as shown in its source above. Pastebin’s scraping API requires explicit IP whitelisting via their own developer portal before it returns data; this isn’t a PasteHunter bug.

High false-positive rate — Tune or disable overly broad rules (generic_api is a common offender if your organization’s pastes legitimately contain many hex-looking identifiers unrelated to secrets); the not any of (...) exclusion pattern shown earlier is the right template for adding your own false-positive filters.

Best Practices

Common Mistakes

Practical Lab Exercise

  1. Clone the repo and install core dependencies: pip3 install yara-python requests --break-system-packages.
  2. Copy the sample config to the correct location: mkdir -p ~/.config && cp settings.json.sample ~/.config/pastehunter.json.
  3. Disable all input modules and enable only json_output, exactly as shown above, then run python3 pastehunter-cli for a few seconds to confirm the YARA compile step and main loop work on your machine.
  4. Write a short synthetic “paste” string containing a fake AWS command and a fake private key header, then run it directly through yara.compile() and .match() against pastehunter/YaraRules/aws.yar, mirroring the verified test shown earlier in this article.
  5. Write one custom YARA rule targeting a fictitious keyword relevant to a project of yours, save it alongside the existing rule files, and confirm it gets picked up in the “Adding rules from…” log lines on the next run.

FAQ

Does PasteHunter work out of the box after pip install? Not quite — you also need to place a real config file at ~/.config/pastehunter.json, since the sample file in the repo root is not auto-loaded. I verified this is a hardcoded path in the source.

Do I need Elasticsearch to use PasteHunter? No. Elasticsearch is one of nine possible output modules. json_output or csv_output work with zero extra infrastructure for smaller-scale or evaluation use.

Can PasteHunter monitor sites other than Pastebin? Yes — GitHub Gists, GitHub code search, ix.io, Slexy, Stack Exchange, and (per the config, though noted as deprecated) dumpz.org are all built-in input modules.

Is PasteHunter actively maintained? Check the GitHub repository’s commit history and issue tracker directly for current maintenance status, since this can change over time — always verify against the live repo rather than assuming.

Can I add my own detection rules? Yes — YARA rule files are just plain text files dropped into the rules directory; PasteHunter compiles every rule file it finds at startup, as shown in the verified log output above.

Summary

PasteHunter turns the manual, tedious process of watching paste sites for leaked secrets into an automated, YARA-driven detection pipeline with genuine flexibility across inputs and outputs. I verified its real architecture end to end — the modular input/output/postprocess design, the exact (and easy to miss) config file location requirement, real YARA rule matching against a synthetic secret-laden paste, real Shannon entropy calculations, and even reproduced the two most common real-world errors (missing config file, missing optional dependency) so you can skip the debugging session I went through and get straight to a working deployment. For any organization worried about accidental credential leaks ending up in public view, this remains one of the simplest, most direct tools to stand up.

References

Exit mobile version