Upcoming Hackers

What is Content Discovery?

Content discovery in cybersecurity refers to the process of identifying and accessing hidden or less obvious content on a web application or server. This includes directories, files, endpoints, and parameters that are not directly linked from the main pages of the website. The primary goal of content discovery is to uncover potential vulnerabilities, misconfigurations, or sensitive information that could be exploited by attackers.

What is Active Content Discovery and Passive Content Discovery?

Active Content Discovery:- Active content discovery involves directly interacting with the target website to uncover hidden resources. This usually means making a large number of HTTP requests to the server, which can be more intrusive and detectable by the server’s logging and monitoring systems.

Passive Content Discovery:- Passive content discovery involves collecting information about the target website without directly interacting with it in a way that would generate significant traffic or alert the server. This method is less intrusive and can help gather useful information that guides further testing.

What are the tool can we use in Content Discovery?

For content discovery, there are several tools available that can effectively uncover hidden directories, files, and other resources on a web server. Here are some of the most popular and widely used tools:-

1. Gobuster
2. Dirsearch
3. DirBuster
4. FFUF (Fuzz Faster U Fool)
5. WFuzz

Let’s briefly understand each of the tools –

Gobuster:- Gobuster is a fast and efficient tool written in Go for brute-forcing URIs (directories and files) in web applications.

• Features:
  • High performance due to being written in Go.
  • Supports directory and file brute-forcing, as well as DNS subdomain discovery.
  • Recursive brute-forcing and proxy support.

So, to use Gobuster, we first have to install it on our Linux terminal. For that, we need to use this command –

sudo apt-get install gobuster

After installing Gobuster on our terminal, it’s time to do some practical exercises. For that, we will use test.php, which is essentially a demo vulnerable website. We can use any website, so first, we copy the URL of test.php and then run it with Gobuster.

gobuster dir -u http://testphp.vulnweb.com/ -w /usr/share/wordlists/dirbuster/directory-list-2.3-medium.txt

In this command, we also need to add the wordlist. Basically, we use a wordlist in Gobuster for directory and file brute-forcing. A wordlist contains a list of common directory and file names that Gobuster uses to systematically try different paths on a web server.

• The -u option stands for “URL“. It specifies the target URL that Gobuster will test for directories and files.
• The -w option stands for “wordlist“. It specifies the path to the wordlist file that Gobuster will use for brute-forcing directories and files on the specified URL.

Now we can see the hidden directories of test.php which are not publicly available. We can use one of the directories from the above list.

From the above list, we have specifically targeted the /pictures directory, and now we can view the hidden content contained within it.

2. Dirsearch:- Dirsearch is a popular command-line tool used for brute-forcing directories and files on web servers. It is made using the python. It is designed to help penetration testers and security researchers discover hidden paths and files that may exist on a target web server.

• Features:
• This feature is essential for discovering potentially sensitive directories and files that are not intended for public access, thus aiding in vulnerability assessment and penetration testing.
• Users can customize the scanning process by providing their own wordlists or using built-in wordlists included with Dirsearch.

To see hidden directories using Dirsearch, you’ll need to install it from GitHub.

So, if we want to use dirsearch in our Linux command line, we should have Python installed. First, we need to check if Python is installed or not. For that, we can use this command-

Python –version

If python is not installed then type this command –

sudo apt install python3

Dirsearch is hosted on GitHub. You can clone the repository to your local machine using git. If git is not installed, install it first with-

sudo apt install git

Then, clone the Dirsearch repository –

git clone https://github.com/maurosoria/dirsearch.git

After cloning the repository, navigate into the dirsearch directory –

cd dirsearch

Using the ls command, we can see the contents of a directory. To check the dependencies, we use the requirements.txt file.

Pip3 install –r requirement.txt

‘requirements.txt’ is a file that typically contains a list of all Python packages and their versions required for a particular project. This ensures that all the necessary libraries and their correct versions are installed.

python3 dirsearch.py -u http://testphp.vulnweb.com/

The command python3 dirsearch.py -u http://testphp.vulnweb.com/ is used to run Dirsearch, a directory brute-forcing tool, against the specified target URL (http://testphp.vulnweb.com/).

In the image, we can clearly see the hidden directory which is not publicly available.

3. DirBuster:- It is a Java-based directory and file brute-forcing tool used to discover hidden directories and files on web servers.

• Features:
• Allows the use of custom wordlists for discovering hidden files and directories.
• Enables simultaneous multiple requests for faster and more efficient brute-forcing.

So, it is made using Java. Therefore, in order to run DirBuster, we first have to install Java on our terminal.

sudo apt install default-jre

after this command we need to download the DirBuster –

wget https://example.com/path/to/dirbuster/DirBuster-1.0-RC1.zip

Extract the downloaded zip file using the unzip

unzip DirBuster-1.0-RC1.zip

Change to the DirBuster directory

cd DirBuster-1.0-RC1

Launch DirBuster by running the JAR file. Use the following command

java -jar DirBuster-1.0-RC1.jar

now it’s time to find the hidden directory using this command –

dirb http://testphp.vulnweb.com/

here we using the dummy website for finding the hidden directory’s

Now you can see the HTTP code 200. This is the hidden directory which is not available in the public.

4. FFUF – ffuf is a fast web fuzzer tool that can be used to discover hidden web directories and files. It is a command-line tool that allows users to perform web application security testing and discovery.

Features –

• FFUF is designed to be fast and efficient, making it a popular choice for security professionals and penetration testers.
• FFUF supports recursive fuzzing, which means it can follow links and directories discovered during the fuzzing.

So run the ffuf in the command terminal first we should have to install the ffuf

Sudo apt install ffuf –y

In the command -y flag is used to automatically answer “yes” to all prompts that may come up during the installation process.

Ffuf –u /usr/share/dirbuster/wordlists/directory-list-2.3-small.txt:FUZZ –u http://testphp.vulnweb.com/FUZZ

In this command, we used the small.txt wordlist and specified our target URL. After running this command, it will find the hidden directories on the specified target URL.

Now we can see all the hidden directories of our targeted URL using the FFUF tool.

5.WFuzz:- It is a web application brute-forcing tool that is used for finding hidden directories, files, and parameters on a web server through brute-force techniques. It is designed to help in the discovery of vulnerabilities in web applications by fuzzing or testing.

Features –

• It does this by systematically testing a large number of potential directory and file names against a target web server.
• It can also be used to brute-force parameters in URLs, forms, and cookies. This involves testing different values or combinations to uncover potential vulnerabilities such as SQL injection, XSS, and other security flaws.

To find a hidden directory using WFuzz, first, we need to install WFuzz in our Linux terminal.

Sudo apt install ffuf –y

The -y flag in the command is used to automatically respond “yes” to all prompts that might appear during the installation process.

wfuzz -c -z file,/usr/share/dirbuster/wordlists/directory-list-2.3-small.txt –hc 404 http://testphp.vulnweb.com/FUZZ

This command runs WFuzz to brute-force directories (FUZZ) on the http://testphp.vulnweb.com website using the directory-list-2.3-small.txt wordlist. It displays results with colorized output (-c) and excludes directories returning a 404 error (--hc 404), helping identify accessible directories and potential points of vulnerability.

Here, we can observe the hidden directory of our specified target URL in the image.

Conclusion:- These tools are commonly used in cybersecurity for discovering hidden files and directories on web servers. Gobuster, Dirsearch, and DirBuster are effective for directory brute-forcing, each offering unique features like speed and customization. FFUF and WFuzz excel in fuzzing parameters and content discovery, supporting extensive customization and automation. Together, they provide comprehensive capabilities for penetration testers and security analysts to assess web application security by identifying vulnerabilities and potential attack surfaces efficiently.