When I got asked to lead my first full-scope security assessment years ago, I made a rookie mistake: I opened a scanner, pointed it at the network, and called it a “penetration test.” The report looked impressive — pages of CVEs, colorful severity charts — but it told the client almost nothing about whether an attacker could actually reach their crown-jewel data. That failure taught me the single most important lesson in this field: a scanner is a tool, not a methodology. A real security test needs structure, repeatability, and a clear chain from “what could go wrong” to “here’s proof it can happen, and here’s how to stop it.”
This article walks through how I build and run a comprehensive security test methodology today — from planning through reporting — and why skipping steps almost always produces a report nobody trusts.
Why Methodology Matters More Than Tools
Anyone can run Nmap and Nessus. What separates a professional assessment from a compliance checkbox exercise is methodology: a documented, repeatable process that ensures coverage, consistency, and defensibility. Without one, two testers given the same scope can produce wildly different results, and neither client nor tester can prove due diligence.
A good methodology answers four questions before a single packet is sent:
- What are we testing, and what are we explicitly not testing?
- What does “done” look like?
- How do we prove findings are real, not false positives?
- How do we communicate risk in business terms, not just CVSS scores?
The Core Testing Frameworks
I don’t invent my own methodology from scratch — I anchor to recognized industry frameworks and then tailor them to the engagement. The main ones I lean on:
- OWASP Testing Guide (OTG) / OWASP Web Security Testing Guide (WSTG) — the standard for web application testing.
- PTES (Penetration Testing Execution Standard) — a broad, phase-based methodology covering pre-engagement through reporting.
- NIST SP 800-115 — “Technical Guide to Information Security Testing and Assessment,” useful for aligning testing with US federal and enterprise risk frameworks.
- OSSTMM (Open Source Security Testing Methodology Manual) — strong on measurable, metrics-driven testing.
- MITRE ATT&CK — not a testing methodology per se, but essential for structuring adversary emulation and mapping findings to real-world tactics and techniques.
I typically blend PTES for engagement structure, OWASP WSTG/MSTG for application-layer depth, and MITRE ATT&CK to frame post-exploitation and detection testing.
The Methodology, Phase by Phase
1. Pre-Engagement and Scoping
This is where most badly-run tests actually fail — before testing even starts. I insist on written scope, rules of engagement (RoE), authorized IP ranges/domains, testing windows, emergency contacts, and a signed authorization letter (“get out of jail free” card). I also clarify:
- Testing type: black box, grey box, or white box
- In-scope vs out-of-scope systems (third-party SaaS, production databases, safety-critical OT systems)
- Data handling rules (can I exfiltrate a sample record to prove impact, or just prove access?)
- Escalation path if I find an active breach mid-test
2. Reconnaissance and Information Gathering
Passive first, active second. Passive recon (OSINT, DNS records, certificate transparency logs, job postings, public code repos, breach-data lookups) builds an attack surface map without touching the target. Active recon (port scanning, service enumeration, banner grabbing) confirms what’s actually live.
# Passive subdomain enumeration
subfinder -d example.com -silent | httpx -silent -title -status-code
# Active service discovery
nmap -sV -sC -p- --min-rate 1000 -oA scan_results 203.0.113.10
3. Threat Modeling and Attack Surface Mapping
I map assets to threats using a simple structure: what’s the asset, who would want it, how would they get to it, and what controls sit in the way today. STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) works well for application threat modeling; for infrastructure, I map against ATT&CK tactics.
4. Vulnerability Analysis
Automated scanning (Nessus, OpenVAS, Nuclei) gives breadth. Manual analysis gives depth and eliminates false positives — a scanner will flag a “vulnerable” service version even if the vendor backported the patch, so every finding gets manually verified before it goes in a report.
5. Exploitation
This is where I attempt to prove exploitability safely, within RoE. The goal isn’t to cause damage — it’s to demonstrate real business impact: can I read a customer record, pivot to an internal segment, or escalate to domain admin? I always prefer the least-invasive proof-of-concept that still convincingly demonstrates risk.
6. Post-Exploitation and Lateral Movement
If initial access is achieved, I test how far an attacker could realistically go: privilege escalation, credential harvesting, lateral movement, and — critically — whether the organization’s detection and response controls actually notice.
7. Reporting and Remediation Guidance
A report nobody acts on is a wasted engagement. I structure findings by business risk, not just technical severity, and always pair every finding with a concrete remediation step, not just “patch the system.”
Methodology Flow
flowchart TD
A[Pre-Engagement & Scoping] --> B[Reconnaissance]
B --> C[Threat Modeling]
C --> D[Vulnerability Analysis]
D --> E[Exploitation - Proof of Concept]
E --> F[Post-Exploitation & Lateral Movement]
F --> G[Detection & Response Validation]
G --> H[Reporting & Risk Prioritization]
H --> I[Remediation Verification / Retest]
I -->|Findings Closed| J[Sign-off]
I -->|Findings Open| D
Comparing Testing Approaches
| Approach | Depth | Speed | Best For | Limitation |
|---|---|---|---|---|
| Automated vulnerability scan | Low-Medium | Fast | Continuous coverage, compliance baselines | High false-positive rate, no business context |
| Manual penetration test | High | Slow | Proving real-world exploitability | Point-in-time snapshot |
| Red team engagement | Very High | Very Slow | Testing detection & response, not just vulnerabilities | Expensive, needs mature blue team to be worthwhile |
| Bug bounty | Variable | Continuous | Broad, crowd-sourced coverage of external assets | Inconsistent quality, scope disputes |
| Purple team exercise | High | Medium | Collaborative improvement of detections | Requires blue team buy-in and time |
Real-World Example: A Web App Test That Found More Than a CVE List
On one engagement, an automated scan against a client’s customer portal returned nothing above “medium.” Following methodology instead of stopping there, I manually walked the authentication and authorization flows per OWASP WSTG and found an insecure direct object reference (IDOR): changing a numeric account_id parameter in a URL let any authenticated user view another customer’s invoices — full names, addresses, and partial payment details. No scanner flagged it because the endpoint returned a valid 200 OK regardless of ownership. This is exactly why manual, methodology-driven testing catches what tools miss, and it’s a textbook example of OWASP’s A01:2021 – Broken Access Control.
Common Mistakes I See
- Treating a vulnerability scan as a penetration test. They answer different questions.
- No retest phase. Findings get “fixed” on paper but never verified.
- Ignoring business context. A “critical” CVSS score on an isolated dev server matters less than a “medium” finding on a system touching payment data.
- Skipping detection validation. If nobody in the SOC noticed the simulated attack, that’s itself a critical finding.
- Over-reliance on a single framework. PTES alone won’t give you OWASP-level web app depth; blending frameworks matters.
Risk Mitigation and Best Practices
- Align every finding to a recognized standard (CWE ID, OWASP Top 10 category, MITRE ATT&CK technique ID) so remediation teams have authoritative context.
- Use a risk matrix (likelihood × impact) rather than raw CVSS alone when prioritizing.
- Build a retest cycle into the contract from day one.
- Maintain an evidence chain (screenshots, request/response logs, timestamps) for every proven finding — this is what makes a report defensible in an audit.
FAQs
How often should an organization run a full security test? At minimum annually, plus after major architecture changes, and continuously via automated scanning between full-scope engagements.
What’s the difference between a vulnerability assessment and a penetration test? A vulnerability assessment identifies and lists weaknesses; a penetration test actively attempts to exploit them to prove real-world impact.
Do I need a red team if I already do penetration testing? Not always. Red teaming makes the most sense once an organization has a functioning detection and response capability worth stress-testing.
Is CVSS score enough to prioritize fixes? No — CVSS measures technical severity, not business context. A low-CVSS finding on a system holding sensitive data can be a higher real-world priority.
Summary and Recommendations
A comprehensive security test methodology isn’t about running more tools — it’s about structured, repeatable rigor from scoping through retest. Anchor to recognized standards, blend frameworks to match your target’s nature, verify every finding manually, and always translate technical risk into business impact. That’s what turns a stack of scanner output into an assessment leadership actually acts on.
References and further reading:
- OWASP Web Security Testing Guide: https://owasp.org/www-project-web-security-testing-guide/
- PTES: http://www.pentest-standard.org/
- NIST SP 800-115: https://csrc.nist.gov/pubs/sp/800/115/final
- MITRE ATT&CK: https://attack.mitre.org/
- OSSTMM: https://www.isecom.org/OSSTMM.3.pdf
