What is the Difference Between a Penetration Test and a Vulnerability Assessment?

Many clients come to us and ask for a penetration test. But when we talk about their goals, the definition of success, and the level of acceptable risk – we find that what they really want is a vulnerability assessment. So, is one better than the other? No, not necessarily. But there are important differences between the two and an understanding of the distinction can help to ensure that an engagement is effective, efficient, and results in as few surprises as possible for clients and testers alike.

A vulnerability assessment takes a broad view of an environment, leveraging both automated scanning and manual techniques to identify known vulnerabilities, misconfigurations, and other conditions which present weaknesses in the effort to secure an organization’s assets. The findings are analyzed and the potential ramifications are theorized, but no actual exploitation takes place (no hacking here).

A penetration test takes the work of a vulnerability assessment at least one step further and proves risk to the business by actually exploiting vulnerabilities discovered in the environment. This is a great way to discover not only what conditions exist which may aid malicious actors in causing harm to an organization, but also in determining the extent to which a given threat can produce this harm, especially as it relates to exploitation techniques which combine seemingly unrelated vulnerabilities.

Shedding the Opinion – What’s Really In your Environment?

A vulnerability assessment is an excellent way to get an initial “lay of the land”, especially for an organization which is still in the foundational stage of a formal security program. An engagement at this phase can result in a mapping of hosts in the environment (which can oftentimes be more accurate than point-in-time manual inventories, even when performed regularly), an accounting of operating system and exposed services, assets which have not maintained adequate patching levels, and – perhaps most importantly – a root-cause analysis of the policies, procedures, and overall technical/security culture which have produced the conditions ripe for the engagement’s findings, whatever they may be. In other words, a vulnerability assessment can be a great tool for eliminating hopefulness around what might be happening in an environment and instead replacing it with an accurate representation more in-line with reality.

Finding Present Risk

It is common for a vulnerability assessment’s findings to be met with resistance or attempts to explain them away. In these cases, the additional exploitation measures of a penetration test can prove extremely valuable.

We once performed an initial vulnerability assessment of a client’s public-facing assets and discovered that a custom web application was allowed for the enumeration of authorized usernames. We included this condition as a finding in the assessment’s final report and recommended both that the flaw be mitigated by smoothing out the delay in response time when a valid user is provided and by implementing some best-practice controls such as multi-factor authentication.

After we presented our report, there was an internal dispute in the client’s organization as to whether or not the finding should have been included, citing mitigating explanations such as “the web application firewall will take care of it” and “this is standard industry practice”.

As a follow-up, the client asked us to perform a targeted penetration test of the application with a focus on leveraging the user-enumeration finding to compromise the underlying server or associated database. In this case, the flaw allowed us to generate a list of a few dozen authorized users which were used as a pivot point in both credential stuffing (identifying passwords for the account which may have been breached via other services) and password guessing (attempting to login using a select few high-value passwords - such as “Winter2020!” – across many user accounts) attacks.

This resulted in administrative access to the web application, which provided enough functionality to achieve a reverse shell back from the underlying server and eventual compromise of the user database. Additionally, both password reuse and file-system artifacts left behind by system administrators contributed to full ownership of the target organization’s Active Directory domain in a later phase, none of which would have been discovered had the engagement persisted as a pure vulnerability assessment and the associated findings remained in contention.

The Risk Perspective

There are certainly potential risks posed by vulnerability assessments and penetration tests alike. However, the range of acceptable risk can vary greatly between the two. For example, a physical vulnerability assessment might find that a glass-paned exterior door presents an easy bypass to a keycard-based access mechanism. A penetration test of the same location might (at least in theory) result in breaking the glass in order to achieve access to the building, potentially exposing the tester and bystanders to risk presented by the broken glass, as well as the damage to the door itself.

Here’s a more technical and less contrived example:

A vulnerability assessment of an organization’s network includes identifying hosts and their associated operating system versions. Discovery of Linux kernel version 3.8, for example, might indicate the presence of the Dirty COW privilege escalation vulnerability, allowing standard users to elevate to root privileges. Discovering this system and estimating its kernel version did not necessarily introduce a great deal of risk, but even this type of automated scanning can be hazardous to legacy systems, under-resourced hosts, or those with either custom or deprecated networking stacks unequipped to handle the amount of traffic generated by automated scanning.

Attempted exploitation of kernel vulnerabilities such as this can be a different story, though. Exploiting Dirty COW specifically is a notoriously unstable process (especially when associated exploits were first released) and crashing target systems can be common. Additionally, many exploit methods are probabilistic in nature and might require a handful of attempts before achieving the desired result, each of which introduces the potential for unintended consequences.

Full Disclosure: Qualified security professionals understand the risks associated with vulnerability assessments, penetration testing, and other activities. Appropriate steps are always taken to outline risks, mitigate where possible, and approximate production systems where appropriate. However, these risks can never be eliminated entirely.

Which Do You Need? 

The answer here largely depends on how mature the security program is within your organization. For example, my least favorite engagements are when we perform penetration tests for clients who have not yet implemented even the most basic security controls.

Is it fun to pwn an easy target? Maybe the first dozen times. But eventually you start to think of how to provide the most value and wonder whether patch management would be a better use of the funds than showing you can exploit MS08-067 in 2019.

In these cases, a thorough vulnerability assessment can be helpful to establish a starting baseline of what exists in the environment and the associated work to be done (#KnowThyNetwork). However, it is not a perfect world and targeted exploitation of findings might be required in order to build the support, attention, and funding required to implement a comprehensive approach to security. We will gladly oblige and hack all the things in these cases. Later we will cover more advanced exploitation services – such as Red Teaming – which offer a fantastic way of stress-testing mature security programs by simulating real-world attack techniques in a prolonged live-exercise.