3. Reconnaissance

Welcome to Module 3 of your ethical cybersecurity training. 

This module focuses on the critical phase of information gathering and reconnaissance, laying the groundwork for effective penetration testing and ethical hacking engagements. You'll delve into DNS enumeration techniques, subdomain discovery, and files and directories discovery using powerful tools.

3.1 Subdomain Discovery

Importance of Subdomain Discovery

Subdomain discovery is a crucial aspect of the reconnaissance phase in penetration testing and ethical hacking or CTF challenges. 

What this phase will reveal:

• Uncovering Hidden Infrastructure
  Subdomains can reveal additional servers, services, and applications that are part of an organization's infrastructure but are not immediately visible on the main domain. These hidden assets might have different security postures and could be potential entry points for attackers.

• Identifying Development and Staging Environments
  Development, testing, and staging environments often reside on subdomains and might not be as secure as production environments. These environments can provide valuable information about the application's structure and potential vulnerabilities that can be exploited.

• Finding Forgotten or Legacy Services
  Organizations sometimes forget to decommission old services and applications, leaving them accessible through subdomains. These legacy systems might have unpatched vulnerabilities that can be exploited.

• Expanding the Attack Surface
  Each discovered subdomain represents an expansion of the attack surface. Attackers can focus on these subdomains to find vulnerabilities that might not be present on the main domain.

• Locating Administrative Interfaces
  Administrative interfaces, dashboards, and management panels are often located on subdomains. Discovering these interfaces can provide attackers with opportunities to attempt login attacks or exploit administrative functionalities.

• Gathering Valuable Information
  Subdomains can contain information about the organization’s internal structure, such as department names, employee names, and project names. This information can be used for social engineering attacks or further reconnaissance.

• Bypassing Security Measures
  Security measures like web application firewalls (WAFs) and intrusion detection systems (IDS) might be configured for the main domain but not for subdomains. Attackers can exploit this oversight to bypass security defenses.

• Improving Security Posture
  From a defensive perspective, subdomain discovery helps organizations identify and secure all parts of their digital footprint. Regular subdomain scanning can help ensure that all services are accounted for and properly secured.

Tools and Techniques

• Sublist3r

  • Sublist3r is a powerful and widely used tool for subdomain enumeration. It helps security professionals, penetration testers, and bug bounty hunters in gathering and discovering subdomains for a given domain. Subdomain enumeration is a crucial step in the reconnaissance phase of a security assessment, as it reveals additional attack surfaces and potential entry points that may otherwise be overlooked.

  • Key Features
    1. Multiple Data Sources: Sublist3r leverages multiple search engines and data sources to gather subdomains. These include Google, Yahoo, Bing, Baidu, Ask, Netcraft, Virustotal, ThreatCrowd, DNSdumpster, and more. This multi-source approach increases the comprehensiveness of the results.
    2. Simple and Efficient: Sublist3r is designed to be simple to use and efficient. It provides quick results with minimal configuration, making it accessible for users with varying levels of technical expertise.
    3. Integration with Other Tools: Sublist3r can be integrated with other tools in the security professional’s toolkit. For example, it can be combined with tools like Nmap for further enumeration and vulnerability scanning of discovered subdomains.
    4. Active and Passive Enumeration: Sublist3r supports both active and passive methods of subdomain enumeration, allowing users to choose the most appropriate approach based on the assessment context.
       
       
  • Installation
    Sublist3r can be easily installed on various operating systems, including Linux, macOS, and Windows.
    Below are the steps to install Sublist3r on a Linux-based system:
    
    1. Python is required before installing Sublist3r
    2. Clone the Sublist3r Repository: git clone https://github.com/aboul3la/Sublist3r.git
    3. Navigate to the Sublist3r Directory:cd Sublist3r
    4. Install the Required Dependencies:
       • If you are using Python2: pip install -r requirements.txt
       • If you are using Python3: pip3 install -r requirements.txt
         
         
  • Usage
    Sublist3r is straightforward to use.
    The basic syntax for running Sublist3r is as follows: python sublist3r.py -d -o

    -d : Specify the target domain for subdomain enumeration.
    -o : (Optional) Save the results to the specified output file.

  • Example
    To enumerate subdomains for the domain example.com and save the results to a file named subdomains.txt, you would use the following command:
    python sublist3r.py -d example.com -o subdomains.txt

[-] Enumerating subdomains now for example.com
[-] Searching in Google...
[-] Searching in Yahoo...
[-] Searching in Bing...
[-] Searching in Baidu...
[-] Searching in Ask...
[-] Searching in Netcraft...
[-] Searching in Virustotal...
[-] Searching in ThreatCrowd...
[-] Searching in DNSdumpster...
[-] Searching in SSL Certificates...
[-] Searching in PassiveDNS...

[-] Total Unique Subdomains Found: 10
www.example.com
mail.example.com
ftp.example.com
blog.example.com
dev.example.com
shop.example.com
api.example.com
admin.example.com
beta.example.com
test.example.com
 

• Amass

  • Amass is a powerful and comprehensive tool designed for in-depth subdomain enumeration and open-source intelligence (OSINT) gathering. Developed by the OWASP Amass Project, it leverages both passive and active reconnaissance techniques to discover a wide range of subdomains, providing valuable insights into the target's digital footprint. Amass is widely used by security professionals, penetration testers, and bug bounty hunters to map out target environments and identify potential attack vectors.
    
    
  • Key Features 
    1. Comprehensive Subdomain Enumeration: Amass combines multiple techniques and data sources to perform extensive subdomain enumeration, resulting in a more thorough discovery process compared to other tools.
    2. Passive and Active Reconnaissance: Amass supports both passive reconnaissance (using public data sources without interacting directly with the target) and active reconnaissance (interacting with the target to gather additional information), providing flexibility based on the user's needs and the rules of engagement.
    3. Data Source Integration: Amass integrates with a variety of data sources, including DNS databases, web archives, certificate transparency logs, and API services. This multi-faceted approach enhances the accuracy and completeness of the enumeration process.
    4. Graphical Visualization: Amass offers features to visualize the collected data, helping users to better understand the relationships between discovered subdomains and the overall structure of the target’s network.
    5. Configurable and Extensible: Amass is highly configurable, allowing users to customize its behavior through configuration files and command-line options. It also supports extensions and custom scripts to enhance its functionality.Key Features
       
       
  • Installation
    Amass can be installed on various operating systems, including Linux, macOS, and Windows.
    Below are the steps to install Amass:
    • With Homebrew (MacOS):
      brew tap owasp-amass/amass
brew install amass
 
    • With Apt-get (Debian / Ubuntu / Kali / Parrot):
      apt update or apt-get update
apt-get install amass or apt install amass
 
    • Verify the Installation
      amass -version
 
  • Usage 
    Amass provides a wide range of options and commands. Here are some basic usage examples for both passive and active reconnaissance
    
    
  • Passive Reconnaissance: Passive reconnaissance involves gathering information without directly interacting with the target. This method is less likely to trigger security alarms and is suitable for stealthy assessments.
    • Example
      amass enum -passive -d example.com

[INF] Enumerating subdomains for: example.com
[INF] example.com: Found 12 subdomains
www.example.com
mail.example.com
ftp.example.com
blog.example.com
dev.example.com
shop.example.com
api.example.com
admin.example.com
beta.example.com
test.example.com
secure.example.com
vpn.example.com
 
  • Active Reconnaissance: Active reconnaissance involves interacting with the target to gather additional information. This method can provide more detailed results but may trigger security defenses.
    • Example
      amass enum -active -d example.com

[INF] Enumerating subdomains for: example.com
[INF] example.com: Found 15 subdomains
www.example.com
mail.example.com
ftp.example.com
blog.example.com
dev.example.com
shop.example.com
api.example.com
admin.example.com
beta.example.com
test.example.com
secure.example.com
vpn.example.com
intranet.example.com
portal.example.com
 
  • Advanced Features
    1. Using a Configuration File: You can create a configuration file to customize Amass's behavior and specify API keys for various data sources.
       • Example: amass enum -config config.ini -d example.com
         
         
    2. Visualizing Results: Amass can generate visualizations of the discovered subdomains to help you understand the target’s network structure.
       • Example: amass viz -d example.com -o amass_graph
         
         

3.2. Files and Directories Discovery

Understanding Files and Directories Discovery

Files and directories discovery is a vital aspect of the reconnaissance phase in penetration testing and ethical hacking. This process involves searching for hidden files and directories on web servers, which can uncover sensitive information and potential entry points for attacks. By identifying these hidden assets, security professionals can gain insights into the target's infrastructure and identify vulnerabilities that might otherwise go unnoticed.

What this phase will reveal:

• Revealing Configuration Files:

  • Configuration files often contain sensitive information such as database credentials, API keys, and server settings. Accessing these files can provide attackers with critical information to compromise the target system.
  • Example: A file named config.php might contain database connection details. If this file is accessible, an attacker could potentially gain access to the database.
    
    

• Identifying Backup Files:

  • Backup files are frequently left on servers and can contain valuable data, including source code, database dumps, and configuration details. These files can be an easy target for attackers.
  • Example: A file named backup.zip might contain an entire website backup, including sensitive data that can be exploited.
    
    

• Exposing Administrative Interfaces:

  • Administrative interfaces are often hidden or not linked from the main website. Discovering these interfaces can allow attackers to attempt unauthorized access and gain control over the server.
  • Example: A directory named /admin could lead to an administrative login page, which might be vulnerable to brute-force attacks.
    
    

• Finding Development and Debugging Files:

  • Development and debugging files are sometimes left on servers unintentionally. These files can provide insights into the application's structure and potential weaknesses.
  • Example: Files like dev.log or debug.php might contain error messages and stack traces that reveal underlying code vulnerabilities.
    
    

• Locating Sensitive Directories:

  • Sensitive directories might contain important information such as user data, financial records, and proprietary information. Gaining access to these directories can have severe security implications.
  • Example: A directory named /users might contain personal information about the users of the web application.
    
    

• Bypassing Security Controls:

  • Discovering hidden files and directories can help attackers bypass security controls like authentication and access controls. This can lead to unauthorized access and data breaches.
  • Example: A directory named /backup might be accessible without authentication, allowing attackers to download sensitive files.
    
    

Tools and Techniques:

Several tools and techniques can be used for files and directories discovery. Here are some popular tools and examples of how to use them:

1. Gobuster:

   • Gobuster is a fast command-line tool for brute-forcing URLs and DNS subdomains.
   • Installation
     • With Homebrew (MacOS):
       brew install gobuster
 
     • With Apt-get (Debian / Ubuntu / Kali / Parrot):
       apt update or apt-get update
apt-get install gobuster or apt install gobuster
     • Verify the Installation
       gobuster version 
       
       
   • Example
     gobuster dir -u http://example.com -w /path/to/wordlist

/admin (Status: 301)
/backup (Status: 403)
/config.php (Status: 200)
/uploads (Status: 200)
 
   • Common Error
     Some websites won't answer as expected in the case of non existant files or directories. The conventional response code from a website should be "404" (not found), but some websites could redirects you (301 or 302), or act differently based on their configurations.
     To test the response code, Gobuster always starts by looking for a file it know it doesn't exist. If the website answers with the right answer, 404, the discovery session will start. But if it answers something else, you'll get response looking like this:
     • Error: the server returns a status code that matches the provided options for non existing urls. https://example.com/feda7955-a131-4201-bd0c-b50391b2cdb9 => 500 (Length: 433). To continue please exclude the status code or the length
     • In this case you'll want to either manually select the postivives status codes (that will flag the files as existing), or the blacklist status codes (files or directory do not exist). To fix the error above we could choose to add the error "500" to our blacklist, which contained only "404" by default, like this:
       gobuster dir -u https://example.com -w /path/to/wordlist -b 404,500 
       
       
   • More commands
     To get all the options available in this discovery function of Gobuster, type:
     gobuster dir -h
     
     

2. dirsearch:

   • Dirsearch is a powerful, command-line tool designed for brute-forcing directories and files on web servers. Written in Python, Dirsearch is favored by security professionals and penetration testers for its speed, flexibility, and ease of use. It can quickly and efficiently enumerate directories and files on a target web server, revealing hidden assets that might be overlooked through conventional web browsing. This makes Dirsearch an essential tool for the reconnaissance phase of penetration testing and ethical hacking.

   • Key Features

     1. Fast and Efficient: Dirsearch is optimized for speed, enabling quick enumeration of directories and files. It achieves this through the use of multiple threads for parallel processing, significantly reducing the time required for comprehensive scans.

     2. Customizable Wordlists: Users can specify their own wordlists, making Dirsearch highly customizable for different targets and contexts. This flexibility allows for tailored scanning based on the specific characteristics of the target web application.

     3. Support for Various HTTP Methods: Dirsearch supports multiple HTTP methods, including GET, HEAD, POST, PUT, DELETE, OPTIONS, and TRACE. This versatility enables it to interact with the target server in various ways to discover hidden directories and files.

     4. Extensible and Configurable: Dirsearch offers a range of configuration options, allowing users to customize scan parameters such as the number of threads, request timeout, user-agent strings, and more. Additionally, its modular design allows for easy extension and integration with other tools.

     5. Proxy and Authentication Support: Dirsearch supports the use of proxies and authentication, making it suitable for environments where direct access is restricted or where authentication is required to access certain resources.

   • Installation

     Dirsearch can be easily installed on various operating systems, including Linux, macOS, and Windows. Below are the steps to install Dirsearch:

     1. python 3.7 or higher is requiered

     2. Clone the Dirsearch Repository: git clone https://github.com/maurosoria/dirsearch.git  

     3. Navigate to the Dirsearch Directory: cd dirsearch

     4. Install the Required Dependencies: pip install -r requirements.txt

   • Usage

     Dirsearch is straightforward to use, with a variety of command-line options to customize scans. Below are some common usage examples:

     • Basic Scan

       To perform a basic directory and file enumeration on a target domain, use the following command:

       python3 dirsearch.py -u http://targetdomain.com -e html,php,js

       In this example:
       - -u specifies the target URL.
       - -e specifies the file extensions to search for (html, php, js).

     • Using a Custom Wordlist

       You can use a custom wordlist for more tailored scanning:

       python3 dirsearch.py -u http://targetdomain.com -w /path/to/wordlist.txt
       
       In this example:
       - -w specifies the path to the custom wordlist.

     • Multithreaded Scanning

       To speed up the scan, you can increase the number of threads:

       python3 dirsearch.py -u http://targetdomain.com -t 50

       In this example:
       - -t specifies the number of threads (50 in this case).

     • Using Proxies and Authentication

       Dirsearch supports the use of proxies and HTTP basic authentication:

       python3 dirsearch.py -u http://targetdomain.com -p http://proxyserver:8080 -a username:password
       
       In this example:
       - -p specifies the proxy server.
       - -a specifies the authentication credentials.

     • Example Output

       As Dirsearch runs, it will display the discovered directories and files in real-time. Here is an example of the output you might see:

       [01:00:00] Starting: http://targetdomain.com
[01:00:01] 200 -  123B  - /admin
[01:00:02] 403 -   23B  - /backup
[01:00:03] 200 -  456B  - /config.php
[01:00:04] 200 -  789B  - /uploads
[01:00:05] 200 -  101B  - /debug.log

3.3 DNS Enumeration

DNS Enumeration is a critical process in the reconnaissance phase of penetration testing and ethical hacking. It involves gathering information about the domain name system (DNS) of a target organization. This process helps in identifying DNS records, subdomains, mail servers, name servers, and other related information. By understanding the DNS infrastructure, security professionals can uncover potential attack vectors and misconfigurations that could be exploited.

Importance of DNS Enumeration

1. Mapping the Target's Infrastructure: DNS enumeration helps in mapping out the target’s network infrastructure. This includes identifying the IP addresses associated with various services, which can reveal the network's topology and the distribution of services.

2. Identifying Misconfigurations: DNS misconfigurations, such as improperly configured zone transfers or outdated DNS records, can expose sensitive information. Enumerating DNS records helps in spotting these issues.

3. Discovering Subdomains: Subdomains can host additional services and applications, which may have different security postures compared to the main domain. Identifying these subdomains can expand the attack surface.

4. Gathering Information for Social Engineering: Information gathered from DNS records, such as email server details and administrative contacts, can be used for social engineering attacks.

5. Uncovering Hidden Services: DNS records might reveal services that are not publicly advertised, such as development servers, internal tools, or backup systems.

Key DNS Record Types

1. A Record: Maps a domain name to an IPv4 address.

2. AAAA Record: Maps a domain name to an IPv6 address.

3. CNAME Record: Alias record that maps one domain name to another.

4. MX Record: Mail exchange record that specifies the mail server responsible for receiving email.

5. NS Record: Name server record that specifies the authoritative DNS servers for a domain.

6. TXT Record: Text record that can contain various types of information, such as SPF (Sender Policy Framework) or domain verification details.

7. PTR Record: Pointer record used for reverse DNS lookups, mapping an IP address to a domain name.

Tools and Techniques

Several tools and techniques can be used for DNS enumeration. Here are some popular tools and examples of how to use them:

1. dig (Domain Information Groper)
   • dig is a powerful command-line tool for querying DNS servers.
     
     
   • Example Command to Query an A Record:
     dig example.com A

; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> example.com A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12345
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.                   IN      A

;; ANSWER SECTION:
example.com.            3600    IN      A       93.184.216.34

;; Query time: 22 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed May 19 12:00:00 UTC 2024
;; MSG SIZE  rcvd: 56
     
     
2. host:
   • host is a simple utility for performing DNS lookups.
     
     
   • Example Command to Query an MX Record:
     host -t mx example.com

example.com mail is handled by 10 mail.example.com.
 
 
3. dnsrecon:
   • dnsrecon is a DNS enumeration tool that can perform various types of queries and checks, including standard record enumeration, zone transfers, and DNSSEC validation.
     
     
   • Install
     
     • git clone https://github.com/darkoperator/dnsrecon.git
     • cd dnsrecon
     • pip install -r requirements.txt (or pip3)
       
       
   • Example Usage for Standard Enumeration:
     python dnsrecon.py -d example.com  (or python3)
     
[*] Performing General Enumeration of Domain: example.com
[*] DNSSEC information found for: example.com
[*] PWNED! Found nameservers: ns1.example.com, ns2.example.com
[*] Found MX record: mail.example.com
[*] Found A record: www.example.com - 93.184.216.34
[*] Found AAAA record: www.example.com - 2606:2800:220:1:248:1893:25c8:1946
     
     
4. fierce:
   • fierce is a DNS reconnaissance tool designed to locate non-contiguous IP space and hostnames against specified domains.
     
     
   • Install
     • git clone https://github.com/mschwager/fierce.git
     • cd fierce
     • pip install -r requirements.txt (or pip3)
     • python fierce/fierce.p (or python3)
        
   • Example Command
     python3 fierce/fierce.py --domain example.com

DNS Servers for example.com:
ns1.example.com
ns2.example.com

Trying zone transfer first...
Unsuccessful in zone transfer (it was worth a shot).

Checking for wildcard DNS...
No wildcard DNS detected.

Found subdomain: www.example.com
Found subdomain: mail.example.com

DNS Enumeration is a fundamental technique in the reconnaissance phase of penetration testing and ethical hacking. By systematically querying and analyzing DNS records, security professionals can uncover valuable information about the target's infrastructure, identify misconfigurations, and expand the attack surface. Utilizing tools like dig, host, dnsrecon, and fierce, testers can perform thorough DNS enumeration to gain insights that are critical for identifying and mitigating potential security risks. Mastering DNS enumeration techniques enhances the overall effectiveness of security assessments and helps ensure a comprehensive understanding of the target environment.

What is a popular tool used for brute-forcing directories and files on web servers to discover hidden paths?