Subdomain enumeration holds a critical role in our reconnaissance and Attack Surface Management (ASM) workflows. In this blog, we’ll dive deep into the complexities of subdomain enumeration, exploring the diverse array of tools and techniques available. I cannot stress enough the significance of achieving comprehensive coverage in our recon and ASM assessments. Furthermore, addressing the challenge of mitigating false negatives, i.e. missing out on subdomains, is paramount for ensuring robust security evaluations. Today, I’ll talk about the landscape of this issue, and a potential solution that involves scalable infrastructure and innovative message queuing techniques to deliver consistent and precise results, even when conducting large-scale assessments.
The Basics of Subdomain Enumeration (we all know it though):
Let’s start by understanding the fundamentals of subdomain enumeration.
Subdomain enumeration involves systematically discovering and listing subdomains associated with a specific domain name. In the context of the Domain Name System (DNS), a subdomain is a domain that is part of a larger domain but is a separate entity with its own unique set of records. For example, “mail.example.com” is a subdomain of “example.com.”
The purpose of subdomain enumeration is to identify potential entry points into a target’s infrastructure and uncover potential attack vectors. By listing all the subdomains associated with a domain, security practitioners can gain valuable insights into the organization’s attack surface, helping them identify potential vulnerabilities and weaknesses.
The common techniques for subdomain enumeration include subdomain brute forcing, LDNS walking, Certificate Transparency reports, reverse DNS lookups, commercial subdomain APIs etc.
Why is this important?
Well, this is a foundational step as during the recon process, the list of subdomains will act as one of the most important seeds. If we miss out on a bunch of important subdomains, no matter how deep and comprehensive the recon/ASM process is, any profiling/risks/etc. on those subdomains will be missed out.
Cool, what is the problem then?
It’s a pivotal process in our cybersecurity arsenal, allowing us to systematically uncover all (as many as possible) subdomains associated with a particular domain name. To achieve this, we leverage a wide array of tools and techniques, each offering unique advantages. From well-established open-source tools like Sublist3r, Amass, and Subfinder to our own custom Python scripts, we all explore various avenues to conduct exhaustive subdomain searches.
While these tools are great and have an amazing output, there are certain limitations that exist on the user’s machine/servers. Wait what, really? Yeah, let’s go deeper into that.
Acknowledging the Role of False Positives and False Negatives:
As we delve into the depths of subdomain enumeration, we encounter both false positives and false negatives. False positive, as in, DNS records which do not even exist but come from the historic records, etc. These are not a problem, as these can be removed by using a simple DNS check. Pretty petty.
The true challenge lies in tackling false negatives. These elusive genuine subdomains, overlooked during enumeration, can lead to incomplete security assessments and conceal critical vulnerabilities. We must take decisive action to minimize the occurrence of false negatives. Why does this happen? Because the resources of the attackers’ machine/server, including, memory, network bandwidth, CPU cores, etc. do restrict the same. While the tools will run and complete the operations as they should, the ultimate output generally has a lot of inconsistency.
Overcoming Inconsistency at Scale:
As we scale up our subdomain enumeration efforts, maintaining consistent and reliable results becomes a daunting task. The vast selection of open-source tools at our disposal often generates inconsistent outcomes, putting the overall reliability and comprehensiveness of our reconnaissance at risk. Again, not blaming the tools, but the resources on the machines. To address this, we need a strategic approach focused on building a scalable infrastructure that can efficiently handle the increased workload.
The Solution: Breaking It Down into Multiple Points
- Scalable Infrastructure: Scaling our infrastructure becomes crucial to conquer the challenges posed by large-scale subdomain enumeration. Cloud computing platforms like Amazon Web Services (AWS) and Google Cloud (GCP) are our go-to options, offering flexibility and elasticity to meet dynamic demand. By dynamically allocating resources and intelligently scaling our infrastructure, we ensure optimal performance, regardless of the scale of our assessments. With infrastructure at our fingertips, it is super easy to auto-scale the ec2 instances and ECS clusters.
- Message Queues to Handle Backlog of Events: Introducing message queuing systems into our workflow is a game-changer. Platforms like Apache Kafka allow us to organize and distribute subdomain enumeration tasks efficiently. The intelligent task management achieved through message queuing ensures balanced workloads and fault tolerance, enhancing the stability and overall effectiveness of our enumeration process.
- Finding the Right Balance of Speed and Accuracy: This is, in my opinion, the most important and overlooked area of improvement. Striking the perfect balance between speed and accuracy should be the key objective. Determining the ideal scanning/querying rate is critical to avoid overlooking critical subdomains or needlessly slowing down our assessments.
Too much multiprocessing can choke the resources, on the local machine, as well as give inconsistent results. Don’t believe me, run a Zmap full port scan with rates of 1000 and 100000 on a bunch of servers, and the results will be miles apart. Shocking?
On the other hand, reducing the rate to a very low number can result in a very slow scan. Think of performing queries for 10,000,000 subdomains, and this might take hours and days depending on the number of targets.
Hence, spend some time, get some data and identify the right balance of speed and accuracy.
- Testing the Accuracy of Data: Continuous validation and verification of enumerated data are cornerstones of our approach. No matter how cool the tech is, we are never going to get the right number on the first go. Regular checks and automated validation tools will solve this issue. This iterative feedback loop empowers us to refine and improve our enumeration process, boosting result accuracy.
- Perfect Workflow with Scalable Infrastructure: Combining intelligent workflow with our scalable infrastructure is where our approach shines. Custom scripts/binaries, equipped with specific logic for filtering and validating subdomains, sharpen our focus and maximize efficiency. This combination guarantees an accurate and targeted subdomain enumeration process, perfectly suited for large-scale assessments.
In conclusion, subdomain enumeration remains an indispensable aspect of our robust reconnaissance and ASM workflows. By capitalizing on scalable infrastructure, leveraging message queuing systems, and optimizing enumeration parameters, we can achieve consistent and precise results, even when dealing with extensive assessments. As the founder and technical leader of our cybersecurity company, I firmly believe that our innovative and comprehensive approach will fortify our digital defenses against ever-evolving threats. With a repertoire of open-source tools, custom scripts, and pioneering techniques, we continue to raise the bar in subdomain enumeration, empowering proactive security assessments and safeguarding the digital landscape.