Vulnerability management metrics: how to measure success
How’s your vulnerability management program doing? Is it effective? A success? Let’s be honest, without the right vulnerability management metrics any program is pretty pointless. If you’re not measuring, how do you know it’s working?
And even if you are measuring, faulty reporting or focusing on the wrong metrics can create blind spots and make it harder to communicate any risks to the rest of the business.
“What gets measured gets managed.” (Peter Drucker)
So how do you know what to focus on? Cyber hygiene, scan coverage, average time to fix, vulnerability severity, remediation rates, vulnerability exposure? The list is potentially endless. Every tool and solution on the market offers different metrics, so it can be hard to know what’s really important.
In this article, we will help you identify and define the key metrics that you need to track the state of your vulnerability management program, progress you’ve made, and create audit-ready reports that:
- Prove your security posture
- Meet vulnerability remediation SLAs and benchmarks
- Help pass audits and compliance
- Demonstrate ROI on security tools
- Simplify risk analysis
- Prioritize resource allocation
Why vulnerability management needs metrics
Metrics play a critical role in gauging the effectiveness of your vulnerability and attack surface management. Measuring how quickly you find, prioritize and fix flaws allows you to continuously monitor and therefore optimize your security.
With the right vulnerability management metrics, you can determine which issues are most critical, prioritize what to fix first, and measure the overall performance of your efforts. Ultimately, the right metrics allow you to make properly informed decisions, so you’re allocating the resources in the right places.
The number of vulnerabilities found is always a good starting point, but it doesn’t tell you much in isolation – without prioritization and advisories, where do you start? Finding, prioritizing and fixing your most critical vulnerabilities is far more important to your business operations and data security than simply finding every vulnerability.
Intelligent prioritization and filtering out the noise are so important because overlooking genuine security threats is all too easy when you’re being overwhelmed by non-essential information. Intelligent results make your job easier by prioritizing issues that have real impact on your security, without burdening you with irrelevant findings.
For example, your internet-facing systems are the easiest targets for hackers. Prioritizing issues that leave this exposed makes it easier to minimize your attack surface. Intruder makes vulnerability management simple even for non-experts, by explaining the real risks and providing remediation advice in easy-to-understand language.
Measuring your time to fix
Next up in terms of importance is the average time to fix – because when you do find an issue, you want to be able to fix it as quickly as possible. Especially as the average time between an attacker discovering and exploiting a vulnerability is just 12 days - and can be as little as five hours according to a recent SANS Institute survey of ethical hackers.
Intruder interprets the output from industry-leading vulnerability scanning tools and prioritizes results according to context, saving you time to focus on what really matters.
Discovering issues is what scanners do best, but how long it takes to fix them is down to you. For many of our customers, seven days to fix a critical issue fits their internal SLAs, while for others, fixing a medium severity issue in 24 hours is the benchmark when holding particularly sensitive data. At Intruder, we call this your Cyber Hygiene Score.
What’s the Cyber Hygiene Score?
This score helps you keep track of how long it takes for your team to fix issues and how you stack up against the industry benchmark.
Your Cyber Hygiene Score’s aim is to give you a current snapshot of your ‘cyber hygiene’ - in other words, the scan coverage, the time taken to fix issues over a period of six months and the average time to fix issues overall. Let’s look at these in more detail.
3 top metrics for every vulnerability management program
1. Scan coverage
What are you tracking and scanning? Scan coverage includes all the assets you’re covering and analytics of all business-critical assets and applications, and the type of authentication offered (e.g., username- and password-based, or unauthenticated).
As your attack surface evolves, changes and grows over time, it’s important to monitor any changes to what’s covered and your IT environment, such as recently opened ports and services. A modern scanner will detect deployments you may not have been aware of and prevent your sensitive data from becoming inadvertently exposed. It should also monitor your cloud systems for changes, discover new assets, and automatically synchronize your IPs or hostnames with cloud integrations.
2. Average time to fix
The time it takes your team to fix your critical vulnerabilities reveals how responsive your team is when reacting to the results of any reported vulnerabilities. This should be consistently low since the security team is accountable for resolving issues and delivering the message and action plans for remediation to management. It should also be based on your pre-defined SLA. The severity of the vulnerability should have a corresponding relative or an absolute period of time for planning and remediation.
3. Risk score
The severity of each issue is automatically calculated by your scanner, usually Critical, High or Medium. If you decide not to patch a specific or group of vulnerabilities within a specified time period, this is an acceptance of risk. With Intruder you can snooze an issue if you’re willing to accept the risk and there are mitigating factors.
For example, when you’re preparing for a SOC2 or ISO audit and you can see a critical risk, you may be willing to accept it because the resource required to fix it isn't justified by the actual level of risk or potential impact on the business. Of course, when it comes to reporting, your CTO may want to know how many issues are being snoozed and why!
What vulnerability management metrics do you need to show management?
What metrics you want to report depends on who you’re reporting to. If it's the CTO or senior management, they will just want to know the business is protected and they’re getting ROI. For example, have there been any new critical issues, how quickly were they fixed, and how many are still open (and why).
We’ve produced useful guides on how to create a vulnerability assessment report, and how to report to the board. Of course, you need to go into the weeds a bit more with your technical or security team who are actually fixing the issues!
Make sure everything is covered
We’ve seen why measuring is so important for success, and identified the vulnerability management metrics that should be included in your audits and reporting. But are you capturing everything from every asset in your IT environment?
Modern scanners like Intruder provide automated, audit-ready reports, but it’s important to know where all your digital assets are to avoid blind spots, unpatched systems and inaccurate reporting – which is why asset discovery is integral to successful vulnerability management. By making sure all your digital estate is covered, you can validate what to prioritize in your remediation plans of your most critical systems.
Where are vulnerability management metrics heading?
By Andy Hornegold, VP Product
The goal of Intruder is to reduce the number of vulnerabilities that exist across your attack surface. If vulnerabilities do exist, you need to find them quickly, make sure you get the right recommendations to the right people as quickly as possible, and make sure they’re fixed as quickly as possible before an attacker can compromise your network.
1. Average (or mean) time to detect
This is the point from a vulnerability going public, to us having scanned all targets and detecting the vulnerability. Essentially, how quickly are we detecting vulnerabilities across your attack surface to reduce the window of opportunity for an attacker.
If it's taking four months to detect a vulnerability because you’re not able to scan frequently enough, that's a huge window where you’re potentially exposed. If you want to understand the effectiveness of your vulnerability management processes, mean time to detect is an important metric to understand.
So, what does this mean in practice? If your attack surface is increasing, you may find that it takes you longer to scan everything comprehensively causing your mean time to detect to increase as well. Conversely, even though the number of vulnerabilities that come out every day is increasing, if your attack surface stays flat and your mean time to detect stays flat or goes down, you're using your resources you have effectively. But if you start to see the opposite, you start asking questions, why is it taking us longer to detect things? And if the answer is the attack surface has ballooned, maybe you need to invest more in your tooling and security team.
2. Attack surface visibility
Very few people are lucky enough to manage and see 100% of their attack surface. So that's where attack surface discovery comes in.
You'll have a total number of assets that you know are exposed or that you’ve found, but how many of those are covered by the vulnerability management program? What you want to see is the percentage of assets that are protected by your vulnerability management program across your attack surface, discovered or undiscovered.
But teams increasingly spin up apps too quickly for most vulnerability management programs. Our automatic vulnerability scanner and CloudBot check when a new network service is exposed to help solve that problem.
3. Mean time to inform
What happens if you’re getting results quickly, but you’re not logging into your vulnerability management tool, actioning a ticket or assigning it to somebody to fix? Prioritization – or what we call intelligent results – is increasingly important to measure and help you decide what to fix first, because of their impact on the business.
If the number of issues is exorbitantly high because it's full of noise, there is a tangible impact on your risk profile as an organization because you’re spending so much time going through fluff that you're not fixing the things that are going to prevent an attacker from getting into your environment. And the more time you spend on it, the more likely you’re going to get compromised by one of the other vulnerabilities in that list.
Intruder trims out a whole bunch of noise and helps reduce the false positive rate, which is a key metric to track. And once you reduce the amount of noise, then you can circle back and focus on the most important metric, which is the time to fix.
4. Looking to the future: Time to fix 0
We want the right people – the people who will actually be fixing issues – to get the information they need as quickly as possible to close the window of opportunity on hackers.
This means we’ll be adding features like role-based access control (RBAC) which can reduce the time to fix from hours or days down to a matter of minutes, and integrate Intruder into the full lifecycle and remediation process by automatically sending notifications and recommendations to the individuals who are actually fixing the vulnerabilities. A few steps closer to reducing time to fix down to 0 - watch this space!
Vulnerability management metrics done right with Intruder
Clearly defined and agreed metrics are a critical element of vulnerability management. Without them, it’s impossible to understand the risk to your organization.
Intruder measures what matters most. It provides audit-ready reports for stakeholders and compliance auditors with vulnerabilities prioritized and integrations with your issue tracking tools. See what’s vulnerable and get the exact priorities, remedies, insights, and automation you need to manage your cyber risk.
- Raw CVE Coverage
- Risk Rating Coverage
- Remote Check Types
- Check Publication Lead Time
- Local/Authenticated vs Remote Check Prioritisation
- Software Vendor & Package Coverage
- Headline Vulnerabilities of 2021 Coverage
- Analysis Decisions
Red teamers, security researchers, detection engineers, threat actors have to actively research type of vulnerability, location in vulnerable software and build an associated exploit.
Tenable release checks for 47.43% of the CVEs they cover in this window, and Greenbone release 32.96%.
Red teamers, security researchers, detection engineers and threat actors now have access to some of the information they were previously having to hunt themselves, speeding up potential exploit creation.
Tenable release checks for 17.12% of the CVEs they cover in this window, and Greenbone release 17.69%.
The likelihood that exploitation in the wild is going to be happening is steadily increasing.
Tenable release checks for 10.9% of the CVEs they cover in this window, and Greenbone release 20.69%.
We’re starting to lose some of the benefit of rapid, automated vulnerability detection.
Tenable release checks for 9.58% of the CVEs they cover in this window, and Greenbone release 12.43%.
Any detection released a month after the details are publicly available is decreasing in value for me.
Tenable release checks for 14.97% of the CVEs they cover over a month after the CVE details have been published, and Greenbone release 16.23%.
With this information in mind, I wanted to check what is the delay for both Tenable and Greenbone to release a detection for their scanners. The following section will focus on vulnerabilities which:
- Have CVSSv2 rating of 10
- Are exploitable over the network
- Require no user interaction
These are the ones where an attacker can point their exploit code at your vulnerable system and gain unauthorised access.
We’ve seen previously that Tenable have remote checks for 643 critical vulnerabilities, and OpenVAS have remote checks for 450 critical vulnerabilities. Tenable release remote checks for critical vulnerabilities within 1 month of the details being made public 58.4% of the time, but Greenbone release their checks within 1 month 76.8% of the time. So, even though OpenVAS has fewer checks for those critical vulnerabilities, you are more likely to get them within 1 month of the details being made public. Let’s break that down further.
In Figure 10 we can see the absolute number of remote checks released on a given day after a CVE for a critical vulnerability has been published. What you can immediately see is that both Tenable and OpenVAS release the majority of their checks on or before the CVE details are made public; Tenable have released checks for 247 CVEs, and OpenVAS have released checks for 144 CVEs. Then since 2010 Tenable have remote released checks for 147 critical CVEs and OpenVAS 79 critical CVEs on the same day as the vulnerability details were published. The number of vulnerabilities then drops off across the first week and drops further after 1 week, as we would hope for in an efficient time-to-release scenario.
While raw numbers are good, Tenable have a larger number of checks available so it could be unfair to go on raw numbers alone. It’s potentially more important to understand the likelihood that OpenVAS or Tenable will release a check of a vulnerability on any given day after a CVE for a critical vulnerability is released. In Figure 11 we can see that Tenable release 61% their checks on or before the date that a CVE is published, and OpenVAS release a shade under 50% of their checks on or before the day that a CVE is published.
So, since 2010 Tenable has more frequently released their checks before or on the same day as the CVE details have been published for critical vulnerabilities. While Tenable is leading at this point, Greenbone’s community feed still gets a considerable percentage of their checks out on or before day 0.
I thought I’d go another step further and try and see if I could identify any trend in each organisations release delay, are they getting better year-on-year or are their releases getting later? In Figure 12 I’ve taken the mean delay for critical vulnerabilities per year and plotted them. The mean as a metric is particularly influenced by outliers in a data set, so I expected some wackiness and limited the mean to only checks released 180 days prior to a CVE being published and 31 days after a CVE being published. These seem to me like reasonable limits, as anything greater than 6 months prior to CVE details being released is potentially a quirk of the check details and anything after a 1-month delay is less important for us.
What can we take away from Figure 12?
- We can see that between 2011 and 2014 Greenbone’s release delay was better than that of Tenable, by between 5 and 10 days.
- In 2015 things reverse and for 3 years Tenable is considerably ahead of Greenbone by a matter of weeks.
- But, then in 2019 things get much closer and Greenbone seem to be releasing on average about a day earlier than Tenable.
- For both the trendline over an 11-year period is very close, with Tenable marginally beating Greenbone.
- We have yet to have any data for 2021 for OpenVAS checks for critical show-stopper CVEs.
With the larger number of checks, and still being able to release a greater percentage of their remote checks for critical vulnerabilities Tenable could win this category. However, the delay time from 2019 and 2020 going to OpenVAS, and the trend lines being so close, I am going to declare this one a tie. It’s a tie.
The takeaway from this is that both vendors are getting their checks out the majority of the time either before the CVE details are published or on the day the details are published. This is overwhelmingly positive for both scanning solutions. Over time both also appear to be releasing remote checks for critical vulnerabilities more quickly.