The first step to attacking any target is conducting reconnaissance, or simply put, gathering information about the target. Reconnaissance is critically important because it’s how you figure out an application’s attack surface (all of the different points at which an attacker can attempt to exploit the application). To look for bugs most efficiently, you need to discover all the possible ways of attacking a target before deciding on the most effective approach.
By understanding how a target works, anyone can set up a solid foundation for finding vulnerabilities. Recon skills are what separate a good hacker from an ineffective one. For example, if an application doesn’t use PHP, there’s no reason to test it for PHP vulnerabilities, and if the organization doesn’t use Amazon Web Services (AWS), time shouldn’t be wasted trying to crack its buckets [data storage locations].
This article introduces the most useful recon techniques for a bug bounty hunter. It also walks through the basics of writing bash scripts to automate recon tasks and make them more efficient. Bash is a shell interpreter [a program that executes commands] available on macOS and Linux systems. Though this discussion assumes the use of a Linux system, many of these tools can be installed on other operating systems as well. Users will need to install some of the tools discussed before using them.
Important Note: Before proceeding, please verify that you are allowed to perform intrusive recon on your target before attempting any techniques that actively engage with it. In particular, activities like port scanning, spidering, and directory brute-forcing can generate a lot of unwanted traffic on a site and may not be welcomed by the organization, potentially violating their bug bounty policy.
Manually Walking Through the Target Application
Before diving into anything else, it will help to first manually walk through the application to learn more about it. Try to uncover every feature in the application that users can access by browsing through every page and clicking every link. Access the functionalities that are not usually used.
For example, if the target is a large social media platform, try to create an event, play a game, and use the payment functionality if it has never been used before. Sign up for an account at every privilege level to reveal all of the application’s features. For example, on a collaboration platform like Slack, you can create owners, admins, and members of a workspace. Also, create users who are members of different channels under the same workspace. This way, it is possible to see what the application looks like to different users and identify unique features available at each level.
This initial manual walk-through should give a rough idea of what the attack surface looks like, where the data entry points are, and how different users interact with each other. This is a crucial starting point before beginning a more in-depth recon process: finding out the technology and structure of an application.
The Power of Google Dorking
When hunting for bugs, it is often necessary to research the details of a vulnerability. For instance, if exploiting a potential cross-site scripting (XSS) vulnerability, one might want to find a particular payload [a piece of code used to exploit a security vulnerability] seen on GitHub. Advanced search-engine skills will help find the necessary resources quickly and accurately.
In fact, advanced Google searches are a powerful technique that hackers often use to perform recon. Hackers call this Google dorking. For the average user, Google is just a text search tool for finding images, videos, and web pages. But for the hacker, Google can be a means of discovering valuable information such as hidden admin portals, unlocked password files, and leaked authentication keys.
Google’s search engine has its own built-in query language that helps filter searches. Here are some of the most useful operators that can be used with any Google search:
| Operator | Description | Example Search |
| site: | Tells Google to show you results from a certain site only. This helps quickly find the most reputable source. | print site:python.org (Limits results to official Python documentation) |
| inurl: | Searches for pages with a URL that matches the search string. A powerful way to search for potentially vulnerable pages. | inurl:"/course/jumpto.php" site:example.com (Checks for a specific vulnerable-looking path) |
| intitle: | Finds specific strings in a page’s title. Useful for finding pages that contain a particular type of content. | intitle:"index of" site:example.com (Searches for directory listing pages) |
| link: | Searches for web pages that contain links to a specified URL. Useful for finding documentation or discussions about vulnerabilities. | link:"https://en.wikipedia.org/wiki/ReDoS" |
| filetype: | Searches for pages with a specific file extension. Incredible tool for locating potentially sensitive files like logs or password files. | filetype:log site:example.com |
| Wildcard (*) | Means any character or series of characters. | "how to hack * using Google" (Matches strings like “how to hack websites using Google”) |
| Quotes (” “) | Forces an exact match for the phrase within the quotes. | "how to hack" (Searches for the exact phrase) |
| Or () | Searches for one search term or the other, or both at the same time. The pipe character must be surrounded by spaces. | |
| Minus (-) | Excludes certain search results. | "how to hack websites" -php (Searches for hacking websites, but excludes results mentioning PHP) |
Advanced search engine options can be used in many more ways to make reconnaissance work more efficient. Users can even search for the term Google search operators to discover more. These operators can be more useful than one might expect.
For example, look for all of a company’s subdomains by searching as follows:
site:*.example.com
One can also look for special endpoints that can lead to vulnerabilities. Kibana is a data visualization tool that displays server operation data such as server logs, debug messages, and server status. A compromised Kibana instance can allow attackers to collect extensive information about a site’s operation. Many Kibana dashboards run under the path app/kibana, so this query will reveal whether the target has a Kibana dashboard. It is then possible to try to access the dashboard to see if it’s unprotected:
Google can find company resources hosted by a third party online, such as Amazon S3 buckets [cloud storage containers]:
site:example.com inurl:app/kibana
site:s3.amazonaws.com COMPANY_NAME
Look for special extensions that could indicate a sensitive file. In addition to .log, which often indicates log files, search for .php, .cfm, .asp, .jsp, and .pl, the extensions often used for script files:
site:example.com ext:php
site:example.com ext:log
Finally, it is possible to combine search terms for a more accurate search. For example, this query searches the site example.com for text files that contain the word password:
site:example.com ext:txt password
In addition to constructing custom queries, it is highly recommended to check out the Google Hacking Database (available at the Exploit Database website), a resource that hackers and security practitioners use to share Google search queries for finding security-related information. It contains many search queries that could be helpful during the recon process. For example, one can find queries that look for files containing passwords, common URLs of admin portals, or pages built using vulnerable software.
While performing recon using Google search, keep in mind that if a high volume of search queries is being sent, Google will start requiring CAPTCHA challenges for visitors from the network before they can perform more searches. This could be annoying to others on the network, so it is not recommended to use Google dorking extensively on a corporate or shared network.
Scope Discovery: Knowing Your Boundaries
The next step is to dive into the recon process itself. First, always verify the target’s scope. A program’s scope on its policy page specifies which subdomains, products, and applications the hunter is allowed to attack. Carefully verify which of the company’s assets are in scope to avoid overstepping boundaries during the recon and hacking process. For example, if example.com’s policy specifies that dev.example.com and test.example.com are out of scope, no recon or attacks should be performed on those subdomains.
Once this has been verified, the next task is to discover what is actually in scope. Which domains, subdomains, and IP addresses are available to attack? What company assets is the organization hosting on these machines?
WHOIS and Reverse WHOIS
When companies or individuals register a domain name, they need to supply identifying information, such as their mailing address, phone number, and email address, to a domain registrar. Anyone can then query this information by using the whois command, which searches for the registrant and owner information of each known domain. One might be able to find the associated contact information, such as an email, name, address, or phone number:
$ whois facebook.com
This information is not always available, as some organizations and individuals use a service called domain privacy, in which a third-party service provider replaces the user’s information with that of a forwarding service.
One could then conduct a reverse WHOIS search, searching a database by using an organization name, a phone number, or an email address to find domains registered with it. This way, all the domains that belong to the same owner can be found. Reverse WHOIS is extremely useful for finding obscure or internal domains not otherwise disclosed to the public. Use a public reverse WHOIS tool like ViewDNS.info to conduct this search. WHOIS and reverse WHOIS will give a good set of top-level domains to work with.
IP Addresses
Another way of discovering the target’s top-level domains is to locate IP addresses. Find the IP address of a known domain by running the nslookup command. In this example, one can see that facebook.com is located at 157.240.2.35:
$ nslookup facebook.com
Server: 192.168.0.1
Address: 192.168.0.1#53
Non-authoritative answer:
Name: facebook.com
Address: 157.240.2.35
Once the IP address of the known domain has been found, perform a reverse IP lookup. Reverse IP searches look for domains hosted on the same server, given an IP or domain. The same tools like ViewDNS.info can be used for this.
Also, run the whois command on an IP address, and then see if the target has a dedicated IP range by checking the NetRange field. An IP range is a block of IP addresses that all belong to the same organization. If the organization has a dedicated IP range, any IP found in that range belongs to that organization:
$ whois 157.240.2.35
NetRange: 157.240.0.0 - 157.240.255.255
CIDR: 157.240.0.0/16
NetName: THEFA-3
...
Organization: Facebook, Inc. (THEFA-3)
...
Another way of finding IP addresses in scope is by looking at autonomous systems (ASNs), which are routable networks within the public internet. ASNs identify the owners of these networks. By checking if two IP addresses share an ASN, one can determine whether the IPs belong to the same owner.
To figure out if a company owns a dedicated IP range, run several IP-to-ASN translations to see if the IP addresses map to a single ASN. If many addresses within a range belong to the same ASN, the organization might have a dedicated IP range. From the following output, one can deduce that any IP within the 157.240.2.21 to 157.240.2.34 range probably belongs to Facebook:
$ whois -h whois.cymru.com 157.240.2.20
AS | IP | AS Name
32934 | 157.240.2.20 | FACEBOOK, US
$ whois -h whois.cymru.com 157.240.2.27
AS | IP | AS Name
32934 | 157.240.2.27 | FACEBOOK, US
$ whois -h whois.cymru.com 157.240.2.35
AS | IP | AS Name
32934 | 157.240.2.35 | FACEBOOK, US
The -h flag in the whois command sets the WHOIS server to retrieve information from, and whois.cymru.com is a database that translates IPs to ASNs. If the company has a dedicated IP range and doesn’t mark those addresses as out of scope, one could plan to attack every IP in that range.
Certificate Parsing
Another way of finding hosts is to take advantage of the Secure Sockets Layer (SSL) certificates used to encrypt web traffic. An SSL certificate’s Subject Alternative Name field lets certificate owners specify additional hostnames that use the same certificate, so these hostnames can be found by parsing this field. Use online databases like crt.sh, Censys, and Cert Spotter to find certificates for a domain.
For example, by running a certificate search using crt.sh for facebook.com, Facebook’s SSL certificate can be found. It will be seen that many other domain names belonging to Facebook are listed:
X509v3 Subject Alternative Name:
DNS:*.facebook.com
DNS:*.facebook.net
DNS:*.fbcdn.net
DNS:*.fbsbx.com
DNS:*.messenger.com
DNS:facebook.com
DNS:messenger.com
DNS:*.m.facebook.com
DNS:*.xx.fbcdn.net
DNS:*.xy.fbcdn.net
DNS:*.xz.fbcdn.net
The crt.sh website also has a useful utility that lets users retrieve the information in JSON format, rather than HTML, for easier parsing. Just add the URL parameter output=json to the request URL: https://crt.sh/?q=facebook.com&output=json.
Subdomain Enumeration
After finding as many domains on the target as possible, the next goal is to locate as many subdomains on those domains as possible. Each subdomain represents a new angle for attacking the network. The best way to enumerate subdomains is to use automation.
Tools like Sublist3r, SubBrute, Amass, and Gobuster can enumerate subdomains automatically with a variety of wordlists and strategies. For example, Sublist3r works by querying search engines and online subdomain databases, while SubBrute is a brute-forcing tool that guesses possible subdomains until it finds real ones. Amass uses a combination of DNS zone transfers, certificate parsing, search engines, and subdomain databases to find subdomains. A custom tool can be built that combines the results of multiple tools to achieve the best results.
To use many subdomain enumeration tools, it is necessary to feed the program a wordlist of terms likely to appear in subdomains. Some good wordlists made by other hackers can be found online. Daniel Miessler’s SecLists is a pretty extensive one. One can also use a wordlist generation tool like Commonspeak2 to generate wordlists based on the most current internet data. Finally, several wordlists found online or generated can be combined for the most comprehensive results. Here’s a simple command to remove duplicate items from a set of two wordlists:
sort -u wordlist1.txt wordlist2.txt
The sort command line tool sorts the lines of text files. When given multiple files, it sorts all files and writes the output to the terminal. The -u option tells sort to return only unique items in the sorted list.
Gobuster is a tool for brute-forcing to discover subdomains, directories, and files on target web servers. Its DNS mode is used for subdomain brute-forcing. In this mode, one can use the flag -d to specify the domain to brute-force and -w to specify the wordlist to use:
gobuster dns -d target_domain -w wordlist
Once a good number of subdomains have been found, more can be discovered by identifying patterns. For example, if two subdomains of example.com named 1.example.com and 3.example.com are found, one can guess that 2.example.com is probably also a valid subdomain. A good tool for automating this process is Altdns, which discovers subdomains with names that are permutations of other subdomain names.
In addition, more subdomains can be found based on the knowledge about the company’s technology stack. For example, if it has already been learned that example.com uses Jenkins [an open-source automation server], it can be checked if jenkins.example.com is a valid subdomain.
It is also important to look for subdomains of subdomains. After finding, for example, dev.example.com, one might find subdomains like 1.dev.example.com. Subdomains of subdomains can be found by running enumeration tools recursively: add the results of the first run to the Known Domains list and run the tool again.
Automating Reconnaissance with Bash Scripts
Further in this discussion, the basics of writing bash scripts to automate recon tasks and make them more efficient will be walked through. Bash is a shell interpreter available on macOS and Linux systems. Though this assumes the use of a Linux system, many of these tools should be installable on other operating systems as well.
This automation is key to making the reconnaissance process scalable and reliable, allowing a bug bounty hunter to sift through vast amounts of data quickly to find that critical piece of information.
Service Enumeration and Web Hacking Reconnaissance
After identifying the primary domains, subdomains, and IP ranges of a target, the crucial next phase of reconnaissance is service enumeration. This involves discovering exactly what services (like web servers, databases, or mail servers) are running on the machines that have been found. Since services often run on default ports, a good way to find them is by port-scanning the machine using either active or passive scanning methods.
Active vs. Passive Service Scanning
Active Scanning
In active scanning, a hunter directly engages with the server. Active scanning tools send requests to connect to the target machine’s ports to look for open ones. This provides real-time information but is also detectable by the target’s security systems. You can use tools like Nmap (Network Mapper) or Masscan for active scanning.
For example, this simple Nmap command reveals the open ports on scanme.nmap.org:
$ nmap scanme.nmap.org
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up (0.086s latency).
Other addresses for scanme.nmap.org (not scanned): 2600:3c01::f03c:91ff:fe18:bb2f
Not shown: 993 closed ports
PORT STATE SERVICE
22/tcp open ssh
25/tcp filtered smtp
80/tcp open http
135/tcp filtered msrpc
445/tcp filtered microsoft-ds
9929/tcp open nping-echo
31337/tcp open Elite
Nmap done: 1 IP address (1 host up) scanned in 230.83 seconds
Nmap is an indispensable tool that identifies open ports, the services running on those ports (like ssh for secure shell access or http for a web server), and often the version of the software.
Passive Scanning (The Stealth Approach)
In contrast, passive scanning involves using third-party resources to learn about a machine’s ports and services without directly interacting with the server. Passive scanning is stealthier and helps attackers avoid detection.
To find services on a machine without actively scanning it, you can use Shodan, a search engine that lets users find machines connected to the internet based on banners [metadata] collected by the Shodan platform.
With Shodan, you can discover the presence of webcams, web servers, or even industrial control systems based on criteria such as hostnames or IP addresses. For instance, running a Shodan search on an IP address like 45.33.32.156 yields data beyond a simple port scan, providing additional information about the server’s software, versions, and geographical location.
Alternatives to Shodan include Censys and Project Sonar. It is best practice to combine the information gathered from different databases for the best results. With these databases, one might also find the target’s IP addresses, SSL/TLS certificates, and software versions, which are all vital pieces of reconnaissance data.
Directory Brute-Forcing: Uncovering Hidden Paths
The next thing that can be done to discover more of the site’s attack surface is to brute-force the directories of the web servers that have been found. Finding directories on servers is valuable because, through them, one might discover:
- Hidden admin panels
- Configuration files
- Password files
- Outdated functionalities
- Database copies
- Source code files
Directory brute-forcing can sometimes allow a user to directly take over a server! Even if no immediate exploits are found, directory information often tells a great deal about the structure and technology of an application. For example, a pathname that includes phpmyadmin usually means that the application is built with PHP.
You can use Dirsearch or Gobuster for directory brute-forcing. These tools use carefully crafted wordlists to construct potential URLs (e.g., adding /admin, /login, or /backup to the base domain), and then request these URLs from a web server.
- If the server responds with a status code in the 200 range (e.g., 200 OK), the directory or file exists and is accessible. This means one can browse to the page and see what the application is hosting there.
- A status code of 404 (Not Found) means that the directory or file doesn’t exist.
- A status code of 403 (Forbidden) means it exists but is protected. Examine 403 pages carefully to see if the protection can be bypassed to access the content.
Here’s an example of running a Dirsearch command. The -u flag specifies the hostname, and the -e flag specifies the file extension to use when constructing URLs:
$ ./dirsearch.py -u scanme.nmap.org -e php
Extensions: php | HTTP method: get | Threads: 10 | Wordlist size: 6023
Error Log: /tools/dirsearch/logs/errors.log
Target: scanme.nmap.org
[12:31:11] Starting:
[12:31:13] 403 - 290B - /.htusers
[12:31:15] 301 - 316B - /.svn -> http://scanme.nmap.org/.svn/
...
[12:31:40] 301 - 318B - /images -> http://scanme.nmap.org/images/
[12:31:40] 200 - 7KB - /index
[12:31:40] 200 - 7KB - /index.html
...
Task Completed
The Gobuster tool’s Dir mode is also used to find additional content on a specific domain or subdomain, including hidden directories and files. In this mode, the -u flag specifies the target URL and the -w flag specifies the wordlist:
gobuster dir -u target_url -w wordlist
Manually visiting all the pages found through brute-forcing can be time-consuming. Instead, use a screenshot tool like EyeWitness or Snapper to automatically verify what page is hosted at each location. EyeWitness accepts a list of URLs and takes screenshots of each page. In a photo gallery application, one can quickly skim these screenshots to find the interesting-looking ones. Keep an eye out for hidden services, such as developer or admin panels, directory listing pages, analytics pages, and pages that look outdated and ill-maintained. These are all common places for vulnerabilities to manifest.
Spidering the Site: Comprehensive URL Discovery
Another way of discovering directories and paths is through web spidering, or web crawling, a process used to identify all pages on a site. A web spider tool starts with a known page to visit. It then identifies all the URLs embedded on that page and visits them. By recursively visiting all URLs found on all pages of a site, the web spider can uncover many hidden endpoints in an application that might not be found through simple brute-forcing.
OWASP Zed Attack Proxy (ZAP) has a built-in web spider that can be used. This open source security tool includes a scanner, a proxy, and many other features. Burp Suite has an equivalent tool called the crawler, but ZAP’s spider is a strong alternative.
One can access its spider tool by opening ZAP and choosing Tools > Spider.
This opens a window for specifying the starting URL.
Clicking Start Scan causes URLs to pop up in the bottom window.
One should also see a site tree appear on the left side of the ZAP window. This shows the files and directories found on the target server in an organized format, which is very helpful for manually exploring the found paths.
Third-Party Hosting: Uncovering Cloud Assets
It is crucial to take a look at the company’s third-party hosting footprint. For example, look for the organization’s Amazon S3 buckets. S3, which stands for Simple Storage Service, is Amazon’s online storage product. Organizations can pay to store resources in buckets to serve in their web applications, or they can use S3 buckets as a backup or storage location. If an organization uses Amazon S3, its buckets can contain hidden endpoints, logs, credentials, user information, source code, and other highly useful information.
How do you find an organization’s buckets?
- Google Dorking: As mentioned earlier, most buckets use the URL format
BUCKET.s3.amazonaws.comors3.amazonaws.com/BUCKET, so the following search terms are likely to find results:site:s3.amazonaws.com COMPANY_NAMEsite:amazonaws.com COMPANY_NAME
awsands3in their custom bucket URLs, so try these searches:amazonaws s3 COMPANY_NAMEamazonaws bucket COMPANY_NAMEamazonaws COMPANY_NAMEs3 COMPANY_NAME
- GitHub Recon: Another way of finding buckets is to search a company’s public GitHub repositories for S3 URLs. Try searching these repositories for the term
s3. (We will discuss using GitHub for recon in a subsequent section). - Specialized Search Engines: GrayhatWarfare is an online search engine that can be used to find publicly exposed S3 buckets. It allows searching for a bucket by using a keyword. Supply keywords related to your target, such as the application, project, or organization name, to find relevant buckets.
- Brute-Forcing and Permutation Tools: Finally, one can try to brute-force buckets by using keywords. Lazys3 is a tool that helps with this, relying on a wordlist to guess buckets that are permutations of common bucket names. Another good tool is Bucket Stream, which parses certificates belonging to an organization and finds S3 buckets based on permutations of the domain names found on the certificates. Bucket Stream also automatically checks whether the bucket is accessible, saving significant time.
Accessing and Proving S3 Bucket Vulnerability
Once a few buckets belonging to the target organization have been found, use the AWS command line tool (awscli) to see if access is possible. The tool can be installed using:
pip install awscli
Then configure it to work with AWS by following Amazon’s official documentation.
The aws s3 command can now be used to access buckets directly from the terminal. Try listing the contents of the bucket that was found:
aws s3 ls s3://BUCKET_NAME/
If this command works, see if the contents of any interesting files can be read by copying files to the local machine:
aws s3 cp s3://BUCKET_NAME/FILE_NAME /path/to/local/directory
Gather any useful information leaked via the bucket and use it for future exploitation! If the organization reveals information such as active API keys or personal information, this should be reported right away as exposed S3 buckets alone are often considered a vulnerability.
One can also try to upload new files to the bucket or delete files from it to prove write access. For example, this command will copy a local file named TEST_FILE into the target’s S3 bucket:
aws s3 cp TEST_FILE s3://BUCKET_NAME/
And this command will remove the test file that was just uploaded:
aws s3 rm s3://BUCKET_NAME/TEST_FILE
These commands are a harmless way to prove that you have write access to a bucket without actually tampering with the target company’s production files. Always upload and remove your own test files. Do not risk deleting important company resources during testing unless you are prepared to deal with the potential serious consequences.
GitHub Reconnaissance:
One of the most fruitful phases of reconnaissance is GitHub Recon, which involves searching an organization’s GitHub repositories for sensitive data that has been accidentally committed, or information that could lead to the discovery of a vulnerability. This is a common and often fatal mistake made by development teams.
Finding Target Repositories
Start by finding the GitHub usernames relevant to the target organization. You should be able to locate these by searching the organization’s name or product names via GitHub’s search bar, or by checking the GitHub accounts of known employees.
When usernames to audit have been found, visit their pages. Find repositories related to the projects being tested and record them, along with the usernames of the organization’s top contributors, which can help find more relevant repositories.
Analyzing Code, Issues, and Commits
Then, it is time to dive into the code. For each repository, pay special attention to the Issues and Commits sections. These sections are full of potential info leaks:
- They could point attackers to unresolved bugs and problematic code.
- They often reveal the most recent code fixes and security patches. Recent code changes that haven’t stood the test of time are more likely to contain bugs.
Look at any protection mechanisms implemented to see if there are ways to bypass them. One can also search the Code section for potentially vulnerable code snippets. Once a file of interest has been found, check the Blame and History sections at the top-right corner of the file’s page to see how it was developed.
While a deeper dive into reviewing source code is a later step, during the recon phase, the focus should be on looking for hardcoded secrets such as API keys, encryption keys, and database passwords. Search the organization’s repositories for terms like key, secret, and password to locate hardcoded user credentials that can be used to access internal systems. After leaked credentials have been found, KeyHacks (a tool available on GitHub) can be used to check if the credentials are valid and learn how to use them to access the target’s services.
Key Areas of Code to Inspect
You should also search for sensitive functionalities in the project. See if any of the source code deals with important functions such as:
- Authentication
- Password reset
- State-changing actions
- Private info reads
Pay attention to code that deals with user input, such as HTTP request parameters, HTTP headers, HTTP request paths, database entries, file reads, and file uploads, because they provide potential entry points for attackers to exploit the application’s vulnerabilities. Look for any configuration files, as they allow one to gather more information about the infrastructure. Also, search for old endpoints and S3 bucket URLs that can be attacked. Record these files for further review in the future.
Outdated dependencies and the unchecked use of dangerous functions are also a huge source of bugs. Pay attention to dependencies and imports being used and go through the versions list to see if they’re outdated. Record any outdated dependencies. This information can be used later to look for publicly disclosed vulnerabilities (CVEs) that would work on the target.
Automating GitHub Recon
Tools like Gitrob and TruffleHog can automate the GitHub recon process:
- Gitrob locates potentially sensitive files pushed to public repositories on GitHub.
- TruffleHog specializes in finding secrets in repositories by conducting regex [regular expression] searches and scanning for high-entropy strings (random-looking strings often indicative of keys or secrets).
Other Sneaky OSINT Techniques (Open Source Intelligence)
Many of the strategies discussed so far are all examples of open source intelligence (OSINT), or the practice of gathering intelligence from public sources of information. This section details other OSINT sources that can be used to extract valuable information.
Job Postings and Employee Profiles
First, check the company’s job posts for engineering positions. Engineering job listings often reveal the technologies the company uses. For example, an ad like this one can be highly revealing:
Full Stack Engineer
Minimum Qualifications:
- Proficiency in Python and C/C++
- Linux experience
- Experience with Flask, Django, and Node.js
- Experience with Amazon Web Services, especially EC2, ECS, S3, and RDS
From reading this, one immediately knows the company uses Flask, Django, and Node.js to build its web applications. The engineers also probably use Python, C, and C++ on the backend with a Linux machine. Finally, they use AWS (specifically EC2, ECS, S3, and RDS) to outsource their operations and file storage.
If relevant job posts are hard to find, search for employees’ profiles on LinkedIn, and read employees’ personal blogs or their engineering questions on forums like Stack Overflow and Quora. The expertise of a company’s top employees often reflects the technology used in development.
Public Artifacts and Leaks
Another source of information is the employees’ Google calendars. People’s work calendars often contain meeting notes, slides, and sometimes even login credentials. If an employee shares their calendars with the public by accident, access to these artifacts can be gained. The organization or its employees’ social media pages might also leak valuable information. For example, hackers have actually discovered sets of valid credentials on Post-it Notes visible in the background of office selfies!
If the company has an engineering mailing list, sign up for it to gain insight into the company’s technology and development process. Also check the company’s SlideShare or Pastebin accounts. Sometimes, when organizations present at conferences or have internal meetings, they upload slides to SlideShare for reference. Information about the technology stack and security challenges faced by the company might be found there.
Pastebin is a website for pasting and storing text online for a short time. People use it to share text across machines or with others. Engineers sometimes use it to share source code or server logs with their colleagues for viewing or collaboration, so it could be a great source of information. One might also find uploaded credentials and development comments. Go to Pastebin, search for the target’s organization name, and see what happens! Automated tools like PasteHunter can also be used to scan for publicly pasted data.
Archive Websites: The Wayback Machine
Lastly, consult archive websites like the Wayback Machine (available at the Internet Archive), a digital record of internet content. It records a site’s content at various points in time. Using the Wayback Machine, one can find:
- Old endpoints
- Directory listings
- Forgotten subdomains
- URLs, and files that are outdated but still in use.
Tomnomnom’s tool Waybackurls can automatically extract endpoints and URLs from the Wayback Machine, making this historical data easy to consume.
Tech Stack Fingerprinting: Identifying Software Versions
Fingerprinting techniques can help understand the target application even better. Fingerprinting is the process of identifying the software brands and versions that a machine or an application uses. This information allows for targeted attacks on the application because one can search for any known misconfigurations and publicly disclosed vulnerabilities related to a particular version. For example, if it is known that the server is using an old version of Apache that could be impacted by a disclosed vulnerability, one can immediately attempt to attack the server using it.
The security community classifies known vulnerabilities as Common Vulnerabilities and Exposures (CVEs) and gives each CVE a number for reference. It is recommended to search for them on the CVE database.
Direct Engagement Fingerprinting
The simplest way of fingerprinting an application is to engage with the application directly.
- Nmap Version Detection: First, run Nmap on a machine with the
-sVflag enabled to enable version detection on the port scan. Here, one can see that Nmap attempted to fingerprint some software running on the target host:$ nmap scanme.nmap.org -sV Starting Nmap 7.60 ( https://nmap.org ) ... PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 6.6.1p1 Ubuntu 2ubuntu2.13 (Ubuntu Linux; protocol 2.0) ... 80/tcp open http Apache httpd 2.4.7 ((Ubuntu)) ... Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel ... - HTTP Headers Analysis: Next, in Burp Suite, send an HTTP request to the server to check the HTTP headers used to gain insight into the tech stack. A server might leak many pieces of information useful for fingerprinting its technology:
Server: Apache/2.0.6 (Ubuntu) X-Powered-By: PHP/5.0.1 X-Generator: Drupal 8 X-Drupal-Dynamic-Cache: UNCACHEABLE Set-Cookie: PHPSESSID=abcde;- Headers like
ServerandX-Powered-Byare good indicators of technologies. TheServerheader often reveals the software versions running on the server.X-Powered-Byreveals the server or scripting language used. - Also, certain headers are used only by specific technologies. For example, only Drupal uses
X-GeneratorandX-Drupal-Dynamic-Cache. - Technology-specific cookies such as
PHPSESSIDare also clues; if a server sends back a cookie namedPHPSESSID, it’s probably developed using PHP.
- Headers like
- Source Code Inspection: The HTML source code of web pages can also provide clues. Many web frameworks or other technologies will embed a signature in the source code. Right-click a page, select View Source Code, and press CTRL-F to search for phrases like
powered by,built with, andrunning. For instance, one might findPowered by: WordPress 3.3.2written in the source. - File/Directory Signatures: Check technology-specific file extensions, filenames, folders, and directories. For example, a file named
phpmyadminat the root directory, likehttps://example.com/phpmyadmin, means the application runs PHP. A directory namedjinja2that contains templates means the site probably uses Django and Jinja2. One can find more information about a specific technology’s filesystem signatures by visiting its individual documentation.
Automated Fingerprinting Tools
Several applications can automate this process:
- Wappalyzer: A browser extension that identifies content management systems, frameworks, and programming languages used on a site.
- BuiltWith: A website that shows which web technologies a site is built with.
- StackShare: An online platform that allows developers to share the tech they use. It can be used to find out if the organization’s developers have posted their tech stack.
- Retire.js: A tool that detects outdated JavaScript libraries and Node.js packages. It can be used to check for outdated front-end technologies on a site.
Writing Your Own Recon Scripts (Bash Scripting)
It has likely become evident that good recon is an extensive process. But it doesn’t have to be time-consuming or hard to manage. Several tools have already been discussed that use the power of automation to make the process easier.
Sometimes, it may be handy to write your own scripts. A script is a list of commands designed to be executed by a program. They’re used to automate tasks such as data analysis, web-page generation, and system administration. For bug bounty hunters, scripting is a way of quickly and efficiently performing recon, testing, and exploitation. For example, one could write a script to scan a target for new subdomains, or enumerate potentially sensitive files and directories on a server. Once the ability to script has been mastered, the possibilities are endless.
This section covers bash scripts in particular—what they are and why they should be used. Bash scripts, or any type of shell script, are useful for managing complexities and automating recurrent tasks. If commands involve multiple input parameters, or if the input of one command depends on the output of another, entering it all manually could get complicated quickly and increase the chance of error. On the other hand, for a list of commands that should be executed many times, scripts save the trouble of typing the same commands over and over again.
Note on Scripting Knowledge: The following examples assume a basic knowledge of how programming languages work, including variables, conditionals, loops, and functions. If not familiar with these concepts, it is recommended to review introductory programming resources.
Understanding Bash Scripting Basics
Let’s write a first script. The first line of every shell script should be the shebang line (#!), which declares the interpreter to use for the script. We’ll use it to indicate that we’re using bash: #!/bin/bash.
Suppose the goal is to write a script that executes two commands: running Nmap and then Dirsearch on a target.
#!/bin/bash
nmap scanme.nmap.org
/PATH/TO/dirsearch.py -u scanme.nmap.org -e php
This script is not very useful as it can scan only one site. We should let users provide input arguments to the bash script so they can choose the site to scan. In bash syntax, $1 represents the first argument passed in, $2 is the second argument, and so on. $@ represents all arguments passed in, while $# represents the total number of arguments. Let’s allow users to specify their targets with the first input argument, assigned to the variable $1:
#!/bin/bash
nmap $1
/PATH/TO/dirsearch.py -u $1 -e php
Now the commands will execute for whatever domain the user passes in as the first argument.
Path Note: The path /PATH/TO/dirsearch.py must be replaced with the absolute path of the directory where the Dirsearch script is stored. Alternatively, the tool can be made accessible from anywhere by adding its directory to the PATH variable (an environmental variable in Unix systems that specifies where executable binaries are found):
export PATH="PATH_TO_DIRSEARCH:$PATH"
After executing this command, the script can simply call dirsearch.py directly. For permanent changes across terminal restarts, the export command should be added to the ~/.bash_profile file.
The script is ready. Save it with the filename recon.sh. Execute the script in the terminal with the command:
$ ./recon.sh target_domain.com
If a permission denied message appears, the user doesn’t have permission to execute the script. This can be corrected by adding executing rights using the chmod command:
$ chmod +x recon.sh
Or, to grant executing rights for the owner only:
$ chmod 700 recon.sh
Now, when running the script and passing scanme.nmap.org as the first argument, the output of the Nmap and Dirsearch scans will be printed out:
$ ./recon.sh scanme.nmap.org
Starting Nmap 7.60 ( https://nmap.org )
... [Nmap output here]
Extensions: php | HTTP method: get | Threads: 10 | Wordlist size: 6023
... [Dirsearch output here]
Task Completed
Saving Tool Output to a File (Redirection)
To analyze the recon results later, one may want to save the script’s output in a separate file. This is where input and output redirection come into play.
| Operator | Description |
PROGRAM > FILENAME | Writes the program’s output into the file, clearing any existing content. (Creates the file if it does not exist.) |
PROGRAM >> FILENAME | Appends the output of the program to the end of the file, without clearing the file’s original content. |
PROGRAM < FILENAME | Reads from the file and uses its content as the program input. |
PROGRAM1 | PROGRAM2 | Uses the output of PROGRAM1 as the input to PROGRAM2 (piping). |
The results of the Nmap and Dirsearch scans can be written into different files using this structure:
#!/bin/bash
echo "Creating directory $1_recon."
mkdir $1_recon
nmap $1 > $1_recon/nmap
echo "The results of nmap scan are stored in $1_recon/nmap."
/PATH/TO/dirsearch.py -u $1 -e php --simple-report=$1_recon/dirsearch
echo "The results of dirsearch scan are stored in $1_recon/dirsearch."
- The
echocommand prints a message to the terminal. mkdircreates a directory named after the domain with_reconappended.- The
>operator stores thenmapoutput into the file$1_recon/nmap. - Dirsearch’s
--simple-reportflag generates a report directly to the designated file.
Using Variables for Clean Code
The script can be made more manageable by introducing variables to reference files, names, and values. Variables in bash are assigned using the syntax VARIABLE_NAME=VARIABLE_VALUE (no spaces around the equal sign). They are referenced using $VARIABLE_NAME.
#!/bin/bash
PATH_TO_DIRSEARCH="/Users/vickieli/tools/dirsearch"
DOMAIN=$1
DIRECTORY=${DOMAIN}_recon
echo "Creating directory $DIRECTORY."
mkdir $DIRECTORY
nmap $DOMAIN > $DIRECTORY/nmap
echo "The results of nmap scan are stored in $DIRECTORY/nmap."
$PATH_TO_DIRSEARCH/dirsearch.py -u $DOMAIN -e php --simple-report=$DIRECTORY/dirsearch
echo "The results of dirsearch scan are stored in $DIRECTORY/dirsearch."
The use of ${DOMAIN}_recon instead of $DOMAIN_recon is intentional; the curly brackets tell bash that DOMAIN is the variable name, and _recon is the plaintext to be appended, preventing bash from interpreting the entire string as one variable name.
Advanced Scripting: Conditionals and Functions
To add more functionalities, such as adding the date of the scan or selectively running tools, advanced shell scripting concepts are needed.
Command Substitution
Command substitution is operating on the output of a command. Using $(command) tells Unix to execute the command surrounded by the parentheses and assign its output to the value of a variable.
#!/bin/bash
...
TODAY=$(date)
echo "This scan was created on $TODAY"
...
Here, the output of the date command (which displays the current date and time) is assigned to the variable TODAY, which is then printed.
Adding Options with Conditionals (if-elif-else)
To selectively run only certain tools, conditionals are used. The syntax of an if statement ends with the fi keyword:
if [ condition 1 ]
then
# Do if condition 1 is satisfied
elif [ condition 2 ]
then
# Do if condition 2 is satisfied, and condition 1 is not satisfied
else
# Do something else if neither condition is satisfied
fi
This functionality allows the user to specify a scan MODE as a second argument ($2):
...
if [ $2 == "nmap-only" ]
then
nmap $DOMAIN > $DIRECTORY/nmap
echo "The results of nmap scan are stored in $DIRECTORY/nmap."
elif [ $2 == "dirsearch-only" ]
then
$PATH_TO_DIRSEARCH/dirsearch.py -u $DOMAIN -e php --simple-report=$DIRECTORY/dirsearch
echo "The results of dirsearch scan are stored in $DIRECTORY/dirsearch."
else
nmap $DOMAIN > $DIRECTORY/nmap
echo "The results of nmap scan are stored in $DIRECTORY/nmap."
$PATH_TO_DIRSEARCH/dirsearch.py -u $DOMAIN -e php --simple-report=$DIRECTORY/dirsearch
echo "The results of dirsearch scan are stored in $DIRECTORY/dirsearch."
fi
If the user specifies nmap-only, only Nmap is run. If dirsearch-only is specified, only Dirsearch is run. Otherwise (the else block), both scans are executed.
Running Additional Tools with case Statements and Functions
For multiple options, case statements are cleaner than long if-elif-else chains. The syntax ends with esac:
case $VARIABLE_NAME in
case1)
Do something
;;
case2)
Do something
;;
*) # Default case
Do something
;;
esac
This is then combined with functions to reduce repetition in the code. A function is declared using:
FUNCTION_NAME()
{
DO_SOMETHING
}
The refined script using functions and case statements, now including a crt-only mode using the curl command to retrieve certificate data in JSON format:
#!/bin/bash
PATH_TO_DIRSEARCH="/Users/vickieli/tools/dirsearch"
TODAY=$(date)
echo "This scan was created on $TODAY"
DOMAIN=$1
DIRECTORY=${DOMAIN}_recon
echo "Creating directory $DIRECTORY."
mkdir $DIRECTORY
nmap_scan()
{
nmap $DOMAIN > $DIRECTORY/nmap
echo "The results of nmap scan are stored in $DIRECTORY/nmap."
}
dirsearch_scan()
{
$PATH_TO_DIRSEARCH/dirsearch.py -u $DOMAIN -e php --simple-report=$DIRECTORY/dirsearch
echo "The results of dirsearch scan are stored in $DIRECTORY/dirsearch."
}
crt_scan()
{
curl "https://crt.sh/?q=$DOMAIN&output=json" -o $DIRECTORY/crt
echo "The results of cert parsing is stored in $DIRECTORY/crt."
}
case $2 in
nmap-only)
nmap_scan
;;
dirsearch-only)
dirsearch_scan
;;
crt-only)
crt_scan
;;
*)
nmap_scan
dirsearch_scan
crt_scan
;;
esac
The curl command with the -o option is used to download the JSON content from crt.sh and save it to a file. Functions simplify the code by allowing the scan and echo commands to be called repeatedly without rewriting them. Remember that all bash variables are global, but input parameters like $1, $2, and $3 within a function refer only to the arguments the function was called with, not the script’s original arguments.
Parsing the Results with grep
After the scans, the next step is to manually read and make sense of complex output files. This process can be sped up by using Global Regular Expression Print (grep). This command line utility is used to perform searches in text, files, and command outputs.
A simple grep command looks like this:
grep password file.txt
This tells grep to search for the string password in file.txt, and then print the matching lines to standard output.
$ grep 80 TARGET_DIRECTORY/nmap
80/tcp open http
grep can also be used with regular expressions (regex) to perform more flexible searches and trim irrelevant messages from tool output, keeping only the essential part of the report.
grep -E "^\S+\s+\S+\s+\S+$" DIRECTORY/nmap > DIRECTORY/nmap_cleaned
- The
-Eflag tellsgrepthat a regex is being used. - The expression
^\S+\s+\S+\s+\S+$filters lines to include those that start (^) with one or more non-whitespace characters (\S+), followed by a whitespace (\s), then another non-whitespace block, another whitespace, and a final non-whitespace block, ending the line ($). This effectively isolates thePORT STATE SERVICElines from the Nmap output.
Here is a quick overview of some essential regex operators:
| Operator | Description |
\d | Matches any digit. |
\w | Matches any word character (alphanumeric, plus underscore). |
\s | Matches any whitespace character. |
\S | Matches any non-whitespace character. |
. | Matches any single character. |
^ | Matches the start of the string or line. |
$ | Matches the end of the string or line. |
* | Matches the preceding character zero or more times. |
+ | Matches the preceding character one or more times. |
The final section of your original text seems to be cut off mid-sentence.
You’ve reached a crucial part of the reconnaissance process: consolidating all the gathered information into an actionable Master Report and learning how to automate the entire workflow for maximum efficiency. This is where scripting skills shine!
Building a Master Report with jq
To produce a master report from all three output files (Nmap, Dirsearch, and the crt.sh JSON data), you need a way to combine and format the diverse data types, especially the JSON output.
Parsing JSON with jq
You need to parse the JSON file from crt.sh. You can do this with jq, a powerful command-line utility that processes JSON.
If you examine the JSON output file from crt.sh, you’ll see you need to extract the name_value field of each certificate item to extract the domain names. The following command does just that:
$ jq -r ".[] | .name_value" $DOMAIN/crt
- The
-rflag tellsjqto write the output directly to standard output (raw output) rather than format it as JSON strings. - The
.[]iterates through the array of items within the JSON file. - The
.name_valueextracts the value of thename_valuefield for each item. $DOMAIN/crtis the input file to thejqcommand.
Final Report Generation Script
To combine all output files into a master report, the script is updated to execute all scans and then use a series of echo, grep, cat, and jq commands with redirection (>>) to append the formatted data into a single report file:
# ... (Functions nmap_scan, dirsearch_scan, crt_scan, and case statement remain the same) ...
echo "Generating recon report from output files..."
TODAY=$(date)
# 1. Start a new report file with the date (using > to overwrite/create)
echo "This scan was created on $TODAY" > $DIRECTORY/report
# 2. Append Nmap results (cleaned with grep)
echo "Results for Nmap:" >> $DIRECTORY/report
grep -E "^\s*\S+\s+\S+\s+\S+\s*$" $DIRECTORY/nmap >> $DIRECTORY/report
# 3. Append Dirsearch results (using cat to print file content)
echo "Results for Dirsearch:" >> $DIRECTORY/report
cat $DIRECTORY/dirsearch >> $DIRECTORY/report
# 4. Append crt.sh results (parsed with jq)
echo "Results for crt.sh:" >> $DIRECTORY/report
jq -r ".[] | .name_value" $DIRECTORY/crt >> $DIRECTORY/report
Key Redirection Notes:
>(Line 1): Overwrites the file to create a fresh report.>>(Lines 2-4): Appends the subsequent results to the report file.- The
catcommand (Line 3) prints the contents of a file to standard output, but when redirected with>>, it effectively copies the content of one file into another.
Scanning Multiple Domains (Loops and Options)
When reconning a large target, you might start with several of the organization’s domain names (e.g., facebook.com and fbcdn.net). The script needs to be able to scan multiple domains with a single command, like ./recon.sh -m nmap-only facebook.com fbcdn.net.
Command Line Options with getopts
To distinguish which arguments specify the scan MODE and which specify target domains, you use command line options or flags.
The getopts tool parses options from the command line using single-character flags. An option requiring an argument (like -m for mode) must be followed by a colon (:):
getopts "m:" OPTION # recognizes -m followed by an argument
MODE=$OPTARG # stores the argument value (e.g., "nmap-only") into MODE
Important: getopts stops parsing arguments when it encounters an argument that doesn’t start with a - character. You’ll need to place the scan mode before the domain arguments when running the script: ./recon.sh -m nmap-only facebook.com fbcdn.net.
Looping Through Targets (for loop)
To read every domain argument and perform scans on them, a for loop is the best choice since the number of values is known (the list of domain arguments).
for i in "${@:$OPTIND:$#}"
do
# Do the scans for $i
done
for i in ...: Iterates through the provided list of values, assigning the current value to the variablei.${@:$OPTIND:$#}: This array slicing trick is the key.$@represents the array containing all input arguments.$OPTINDis a variable that stores the index of the first argument after the options parsed bygetopts.- The slice essentially says: “Take all elements in the arguments array (
$@), starting from the index where option parsing stopped ($OPTIND), up to the end ($#is the total number of arguments).” This ensures only the domain arguments are looped through.
Final Multi-Domain Script Structure
The multi-domain script wraps the entire scanning and reporting logic inside the for loop:
# ... (Functions and getopts parsing here) ...
for i in "${@:$OPTIND:$#}"
do
DOMAIN=$i
DIRECTORY=${DOMAIN}_recon
echo "Creating directory $DIRECTORY."
mkdir $DIRECTORY
# Run the case statement to execute scans based on $MODE
case $MODE in
# ... (nmap-only, dirsearch-only, crt-only, *) cases call respective functions) ...
esac
echo "Generating recon report for $DOMAIN..."
TODAY=$(date)
echo "This scan was created on $TODAY" > $DIRECTORY/report
# Use conditionals to check if output files exist before reporting
if [ -f $DIRECTORY/nmap ];then
echo "Results for Nmap:" >> $DIRECTORY/report
grep -E "^\s*\S+\s+\S+\s+\S+\s*$" $DIRECTORY/nmap >> $DIRECTORY/report
fi
# ... (Other if statements for dirsearch and crt) ...
done
Useful Conditional Test Flags
The condition [ -f $DIRECTORY/nmap ] checks if the Nmap output file exists. The brackets are shorthand for the test command. Here are some useful test conditions:
| Flag | Purpose | Example |
-f | Tests whether a file exists. | [ -f /bin/bash ] |
-d | Tests whether a directory exists. | [ -d /bin ] |
-r, -w, -x | Tests if a file is readable, writable, or executable. | [ -r /etc/passwd ] |
-z | Tests whether a string is empty. | [ -z "$INPUT" ] |
-n | Tests whether a string is NOT empty. | [ -n "$INPUT" ] |
-eq, -ne | Tests for integer equality / inequality. | [ $COUNT -eq 10 ] |
-gt, -lt | Tests for integer greater than / less than. | [ $COUNT -gt 0 ] |
You can combine test expressions using && (AND) and || (OR):
if [ $3 -gt 1 ] && [ $3 -lt 3 ]: True only if $3 is 2.if [ $3 -gt 1 ] || [ $3 -lt 0 ]: True if $3 is greater than 1, or less than 0.
Writing a Function Library and Interactive Programs
Function Library
As your codebase grows, you should move commonly used functions into a separate file, like scan.lib, to promote code reuse.
- Create
scan.lib: Store all thenmap_scan(),dirsearch_scan(), andcrt_scan()functions inside this file. - Source the Library: In your main script, use the
sourcecommand to load the functions:Bash#!/bin/bash source ./scan.lib # ... now all functions inside scan.lib are available to the script.This is useful when building multiple tools that require the same functionalities.
Building Interactive Programs
To build an interactive program that takes user input during execution (e.g., if the user provides an -i flag for interactive mode), you need the read command and a while loop.
- Parse the Interactive Flag (
-i): Update thegetoptsloop to recognize and handle the-iflag, setting a variable likeINTERACTIVEtotrue. - Use
readto Get Input: Thereadcommand prompts the user and stores the input into a variable:Bashecho "Please enter a domain!" read INPUT - The Interactive
whileLoop: Use awhileloop to repeatedly prompt the user for domains until a specific keyword (likequit) is entered:Bashif [ $INTERACTIVE ];then INPUT="BLANK" while [ $INPUT != "quit" ];do echo "Please enter a domain!" read INPUT if [ $INPUT != "quit" ];then scan_domain $INPUT report_domain $INPUT fi done # ... (else block handles non-interactive multiple domain scan)
Scheduling Automatic Scans (cron Jobs)
To take automation to the next level, you can use cron jobs to schedule your scripts to run periodically, creating an alert system for changes.
Crontab Syntax
You edit your user’s crontab by running crontab -e. All crontabs follow this same syntax:
| Position | Variable | Range |
| A | Minute | 0 – 59 |
| B | Hour | 0 – 23 |
| C | Day of Month | 1 – 31 |
| D | Month | 1 – 12 |
| E | Weekday | 0 – 7 (Sunday is 0 or 7) |
| Command | command_to_be_executed | Full path to the script or command |
Example: To run your recon script (./scan.sh) every day at 9:30 PM:
30 21 * * * /path/to/your/script/scan.sh
Tracking Changes with git diff
You can use the git diff command to compare scan results at different times. This quickly lets you see if the target has changed since you last scanned it, helping to identify any new domains, subdomains, endpoints, and other new assets.
You can combine this with cron and a Git repository to automatically scan, commit results, and let GitHub’s notification feature alert you via email when changes occur.
You are now equipped with the knowledge to perform comprehensive reconnaissance, automate the process with robust Bash scripts, and maintain continuous monitoring of your targets.
That’s an excellent summary of the tools essential for web hacking reconnaissance! You’ve categorized them perfectly based on their function in the overall recon process.
To make this list even more useful, I’ve compiled your mentioned tools into a clear, categorized table format with brief descriptions. This serves as a definitive Recon Toolkit Reference from the chapter.
Reconnaissance Toolkit Reference
| Category | Tool Name | Primary Purpose |
|---|---|---|
| Scope Discovery | WHOIS | Find owner of a domain or IP. |
| ViewDNS.info Reverse WHOIS | Find domains by owner/keyword. | |
nslookup (Built-in OS tool) | Query DNS for IP/host information. | |
| ViewDNS.info Reverse IP | Find domains hosted on the same server. | |
| crt.sh, Censys, Cert Spotter | Find domains via SSL certificates. | |
| Sublist3r, SubBrute, Amass, Gobuster | Enumerate subdomains. | |
| SecLists, Commonspeak2 | Wordlists for brute-forcing. | |
| Altdns | Brute-force subdomains using permutations. | |
| Nmap, Masscan | Scan for open ports. | |
| Shodan, Censys, Project Sonar | Find services without active scanning. | |
| Dirsearch, Gobuster | Brute-force hidden filepaths/directories. | |
| EyeWitness, Snapper | Grab screenshots of URLs for quick review. | |
| OWASP ZAP | Security tool suite; spider for content discovery. | |
| GrayhatWarfare | Search for public Amazon S3 buckets. | |
| Lazys3, Bucket Stream | Brute-force S3 bucket names. | |
| OSINT | Google Hacking Database (GHDB) | Find sensitive info/vulns via Google searches. |
| KeyHacks | Test validity of API keys & credentials. | |
| Gitrob | Find sensitive files in public GitHub repos. | |
| TruffleHog | Find secrets/keys in GitHub repos. | |
| PasteHunter | Scan paste sites for sensitive information. | |
| Wayback Machine | Archive of old website versions. | |
| Waybackurls | Fetch URLs from the Wayback Machine. | |
| Tech Stack Fingerprinting | CVE Database | Search for publicly disclosed vulnerabilities. |
| Wappalyzer | Identify CMS, frameworks, and languages. | |
| BuiltWith | Identify web technologies in use. | |
| StackShare | Research a company’s tech stack. | |
| Retire.js | Detect outdated JavaScript libraries. | |
| Automation | Git (git diff) | Version control; track file changes. |
| Nuclei, Intrigue Core | Automated recon/vulnerability scanning platforms. |
