Penetration Testing vs Vulnerability Scanning
What is the main difference between vulnerability scanning and penetration testing?
People frequently confuse penetration testing and vulnerability scanning, and it is understandable why. Both services are responsible for revealing weaknesses in your IT infrastructure by examining your systems in a manner similar to how an actual hacker would. However, there is a very important distinction between the two. So what exactly is it?
Penetration testing is a manual security assessment whereby a cyber security professional attempts to find a way to break into your systems. It is an in-depth test which evaluates security controls across a variety of systems, including web application, network and cloud environments. This kind of testing could take several weeks to complete, and due to its complexity and cost, is commonly carried out only on an annual basis.
Vulnerability scanning, on the other hand, is automated and performed by tools which can be either installed directly on your network or accessed online. Vulnerability scanners run thousands of security checks against your systems, producing a list of vulnerabilities with corresponding remediation advice. That being the case, it is possible to run continuous security checks even without having a full-time cyber security expert on the team.
What’s better, one-off pen-testing, or regular vulnerability scanning?
Penetration tests have long been an essential part of many organisation’s strategy to protect themselves from cyber attack, and an excellent way to find flaws at a certain point in time. But the use of penetration testing alone can often leave such organisations defenceless for long periods of time.
Performing annual penetration tests as a primary defence against attackers gained popularity in years gone by, for good reasons, and is still common in the cyber security industry today. And while this strategy is certainly better than doing nothing, it does have a fairly critical drawback —what happens between tests?
For example, what happens when a critical new vulnerability is discovered in the Apache web server operating a sensitive customer portal during that long year between their annual pen tests. Or a security misconfiguration gets introduced by a junior developer. What if a network engineer temporarily opens up a port on a firewall exposing a database to the internet, and forgets to close it? Whose job is it to notice these issues which, if left unchecked, could result in a data breach or compromise?
Without continuous monitoring of issues such as these, would they be identified and fixed before attackers get a chance to take advantage?
Premises with a need for robust physical security often boast 24/7 automated solutions to deter attackers every day of the year. So why do some companies treat cyber security any differently? Especially when on average 20 new vulnerabilities get discovered every single day. We don’t think they should!
So hopefully you can start to see why sparsely scheduled pen testing alone is not enough. It really is the cyber equivalent of checking the locks of your high-security building’s premises once a year, but leaving it unmanned without bothering to check if it’s still secure until your next yearly once over. Sounds a bit crazy, right?
Scanning for security issues on a regular basis helps to complement manual testing, as it provides organisations a good level of ongoing security coverage between manual tests.
A lot of companies today are still using annual penetration testing as their single line of defence, but as understanding of how frequently weaknesses arise continues to mature, our view is that automated vulnerability scanning solutions will become the first port of call for all companies, with manual penetration testing a powerful backup plan.
Thankfully, awareness is increasing of the need for a strategy which provides protection all year round, but we’ve still some way to go.
Perhaps it’s time to wake up and smell the continuous coverage!
Thanks to Chris Wallis
Intruder offers both penetration testing and vulnerability scanning services. Intruder’s continuous vulnerability scanning service helps you keep on top of the latest vulnerabilities and alerts you to emerging threats which affect your most-exposed systems. Get started with a free trial today.
This article was originally published in January 2018, and updated in January 2021.
- Raw CVE Coverage
- Risk Rating Coverage
- Remote Check Types
- Check Publication Lead Time
- Local/Authenticated vs Remote Check Prioritisation
- Software Vendor & Package Coverage
- Headline Vulnerabilities of 2021 Coverage
- Analysis Decisions
Red teamers, security researchers, detection engineers, threat actors have to actively research type of vulnerability, location in vulnerable software and build an associated exploit.
Tenable release checks for 47.43% of the CVEs they cover in this window, and Greenbone release 32.96%.
Red teamers, security researchers, detection engineers and threat actors now have access to some of the information they were previously having to hunt themselves, speeding up potential exploit creation.
Tenable release checks for 17.12% of the CVEs they cover in this window, and Greenbone release 17.69%.
The likelihood that exploitation in the wild is going to be happening is steadily increasing.
Tenable release checks for 10.9% of the CVEs they cover in this window, and Greenbone release 20.69%.
We’re starting to lose some of the benefit of rapid, automated vulnerability detection.
Tenable release checks for 9.58% of the CVEs they cover in this window, and Greenbone release 12.43%.
Any detection released a month after the details are publicly available is decreasing in value for me.
Tenable release checks for 14.97% of the CVEs they cover over a month after the CVE details have been published, and Greenbone release 16.23%.
With this information in mind, I wanted to check what is the delay for both Tenable and Greenbone to release a detection for their scanners. The following section will focus on vulnerabilities which:
- Have CVSSv2 rating of 10
- Are exploitable over the network
- Require no user interaction
These are the ones where an attacker can point their exploit code at your vulnerable system and gain unauthorised access.
We’ve seen previously that Tenable have remote checks for 643 critical vulnerabilities, and OpenVAS have remote checks for 450 critical vulnerabilities. Tenable release remote checks for critical vulnerabilities within 1 month of the details being made public 58.4% of the time, but Greenbone release their checks within 1 month 76.8% of the time. So, even though OpenVAS has fewer checks for those critical vulnerabilities, you are more likely to get them within 1 month of the details being made public. Let’s break that down further.
In Figure 10 we can see the absolute number of remote checks released on a given day after a CVE for a critical vulnerability has been published. What you can immediately see is that both Tenable and OpenVAS release the majority of their checks on or before the CVE details are made public; Tenable have released checks for 247 CVEs, and OpenVAS have released checks for 144 CVEs. Then since 2010 Tenable have remote released checks for 147 critical CVEs and OpenVAS 79 critical CVEs on the same day as the vulnerability details were published. The number of vulnerabilities then drops off across the first week and drops further after 1 week, as we would hope for in an efficient time-to-release scenario.
While raw numbers are good, Tenable have a larger number of checks available so it could be unfair to go on raw numbers alone. It’s potentially more important to understand the likelihood that OpenVAS or Tenable will release a check of a vulnerability on any given day after a CVE for a critical vulnerability is released. In Figure 11 we can see that Tenable release 61% their checks on or before the date that a CVE is published, and OpenVAS release a shade under 50% of their checks on or before the day that a CVE is published.
So, since 2010 Tenable has more frequently released their checks before or on the same day as the CVE details have been published for critical vulnerabilities. While Tenable is leading at this point, Greenbone’s community feed still gets a considerable percentage of their checks out on or before day 0.
I thought I’d go another step further and try and see if I could identify any trend in each organisations release delay, are they getting better year-on-year or are their releases getting later? In Figure 12 I’ve taken the mean delay for critical vulnerabilities per year and plotted them. The mean as a metric is particularly influenced by outliers in a data set, so I expected some wackiness and limited the mean to only checks released 180 days prior to a CVE being published and 31 days after a CVE being published. These seem to me like reasonable limits, as anything greater than 6 months prior to CVE details being released is potentially a quirk of the check details and anything after a 1-month delay is less important for us.
What can we take away from Figure 12?
- We can see that between 2011 and 2014 Greenbone’s release delay was better than that of Tenable, by between 5 and 10 days.
- In 2015 things reverse and for 3 years Tenable is considerably ahead of Greenbone by a matter of weeks.
- But, then in 2019 things get much closer and Greenbone seem to be releasing on average about a day earlier than Tenable.
- For both the trendline over an 11-year period is very close, with Tenable marginally beating Greenbone.
- We have yet to have any data for 2021 for OpenVAS checks for critical show-stopper CVEs.
With the larger number of checks, and still being able to release a greater percentage of their remote checks for critical vulnerabilities Tenable could win this category. However, the delay time from 2019 and 2020 going to OpenVAS, and the trend lines being so close, I am going to declare this one a tie. It’s a tie.
The takeaway from this is that both vendors are getting their checks out the majority of the time either before the CVE details are published or on the day the details are published. This is overwhelmingly positive for both scanning solutions. Over time both also appear to be releasing remote checks for critical vulnerabilities more quickly.