Quick answer: Directory and content discovery is the task of surfacing hidden directories, files and endpoints that exist on the server but are not linked (for example /admin, /backup.zip, /.git, /api/v1). With tools such as gobuster and ffuf, fast requests are sent to the server using a wordlist and existing paths are found. It is a core step of the web reconnaissance phase of a penetration test, because a forgotten backup file or a hidden admin panel is the attacker's easiest entry.

What is directory discovery for?

The surface of a website visible in its menu and links is only part of the surface that actually exists on the server. Behind it there may be many unlinked files and directories: old backups, admin panels, configuration files, test endpoints, version control directories (.git). These do not appear in the menu, but the server responds if they are requested.

Directory discovery surfaces this hidden surface by systematically trying with a wordlist. A backup file found can leak source code, an open .git directory can give the whole project, a forgotten admin panel can provide direct access.

Tools and concepts

Concept Description
Wordlist List of directory and file names to try
Gobuster Fast directory, DNS and virtual host discovery tool
ffuf Flexible web fuzzing tool (parameters, directories, hosts)
Status code filtering Filtering by responses such as 200, 301, 403
Recursion Rescanning under a found directory
Extension testing Trying by adding extensions such as .php, .bak, .zip

Commonly found critical paths

  • An open .git or .svn directory, leaking all source code.
  • Backup files (backup.zip, database.sql, site.bak).
  • Unlinked admin panels (/admin, /manage).
  • Configuration and environment files (.env, config.php.bak).
  • Old and test API endpoints.

Defense: what to do against directory discovery?

  1. Do not keep sensitive files and directories in the web root, and do not leave backups in an accessible place.
  2. Block access to directories and files such as .git, .env at the web server level.
  3. Disable automatic directory listing.
  4. Protect admin panels with access control and, where possible, network restriction.
  5. Notice heavy scanning traffic with rate limiting and anomaly detection.

The KAOS and DSET approach

In web penetration tests DSET surfaces the hidden surface of the site with directory and content discovery and reports exposed backup files, version control directories and admin panels. Our KAOS security engine performs this discovery and validates whether the critical paths found are genuinely reachable and risky. In our field experience the most common critical finding is an exposed .git directory or a reachable database backup. Our goal is to see the site's invisible surface before an attacker does.

Related reading: subdomain discovery and enumeration, web application penetration testing OWASP Top 10, web penetration testing with Burp Suite, leaked API keys and secrets management, security misconfiguration OWASP, path traversal and LFI.

Frequently asked questions

What is the difference between directory discovery and subdomain discovery? Subdomain discovery finds an organization's different subdomains (such as panel.site.com). Directory discovery finds hidden paths and files within a single site (such as /admin, /backup.zip). Both are part of the reconnaissance phase but look at different layers: one at the domain level, the other within a single site.

Why is an open .git directory dangerous? An open .git directory can allow downloading the site's entire source code and history. This can expose the application's internal logic, secret keys and database information. When an attacker obtains the source code, they find flaws much more easily. That is why .git and similar directories must be strictly closed to web access.

How do I prevent directory discovery? Do not keep sensitive files and directories in the web root, do not place backups in an accessible location, block access to paths such as .git and .env at the web server level, and disable automatic directory listing. You can also notice heavy scanning traffic with rate limiting and anomaly detection. The most effective defense is not creating the hidden surface in the first place.

Sources

To discover the hidden surface of your website, you can talk to our team at Ankara Hacettepe Technopark. Contact DSET.