How to Choose A Pentesting Company
We live in the age of automated hacking systems, frequent data breaches and consumer protection regulations such as GDPR and PCI DSS, penetration testing has become an essential security requirement for businesses of all sizes, rather than just the banks and governments of yesteryear. What that means is lots of companies out there find themselves needing to choose a pentest supplier for the first time, and it’s not easy.
Faced with the task of getting a penetration test done, the sheer number of providers can be daunting, and finding one which can deliver a high quality test at a reasonable price is not easy. How do you know if they’re any good? Can you tell what level of security expertise was delivered by reading the report? Was your application secure, or did the tester simply not find the serious weaknesses?
There’s no easy answers to these questions, but the good news is that you can help yourself out by asking the right questions up front. The most important considerations fall into three categories: certifications, experience, and as always - price.
Certifications are the first thing a new buyer should look for, as they can provide a convenient shortcut for building trust with a vendor.
Worldwide there’s no shortage of professional certifications available, but in the UK the most well-recognised certification body is CREST (Council of Registered Ethical Security Testers). CREST was set up by the UK’s leading pen testing consultancies precisely to solve this buyer’s problem, and it is now an internationally recognised hallmark of quality for a variety of cyber security disciplines.
You still need to know what to look for though, as CREST have both a company-level certification, as well as individual certifications where each tester must pass an exam to prove their skills. Having one does not mean you have the other.
The company-wide accreditation (‘CREST member company’) is given to companies that can prove their policies, processes and procedures are up to scratch. This allows penetration testing companies to show that they follow good practices on paper, and use appropriate security testing methodologies. However, asking a ‘CREST member company’ to carry out a pen-test does not guarantee that the consultant performing your test is certified themselves to an appropriate standard - merely that the company is morally obliged to provide you with a suitable tester:
When checking the credentials of a penetration testing company, make sure to ask about the actual tester that will carry out the work — do they have appropriate certifications and experience for the job at hand?
This is a key point to take away, the credentials and experience of the person who will carry out the work are equally important to those of the organisation they work for!
For that reason, CREST also have a range of levels even for the individual testers, from entry-level certificates to complex practical examinations in different specialist areas. It’s important to look at both the level of certifications, and whether they’re specific to the type of penetration testing you are looking for. We’ve outlined the available CREST certifications for penetration testing below:
While certifications are useful, they can’t cover everything. There are many types of technology out there, and you can’t have an exam to cover every single one. As you can see from the diagram above, there is no CREST exam for AWS, or for embedded devices, or mobile applications. Being a penetration tester is sometimes like being a GP, you have a very good set of knowledge and skills, but there isn’t always a textbook for the patient you’re dealing with. That’s when experience can come into play.
Besides a penetration tester’s certifications, another big factor in a pentest’s quality is the breadth of experience your pen tester has under their belt. The more exposure that a tester has had, the more likely they are to be proficient at discovering a wide range of security threats.
It’s also important to note that not all experience is equal, since some types of testing can involve specific skills in particular technologies, like AWS Cognito, or the Real Time Messaging Protocol. As far as possible, make sure your potential provider has relevant experience in the types of technology you’re working with.
Remember though, there may not always be a tester with experience in every technology out there, so you may need to be flexible. A good penetration tester will be able to learn about the technology you need testing, based on skills and principles from other disciplines, but it might take them slightly longer to become familiar with the technology at hand. This could have a knock-on effect on the price…
People often ask what a normal cost for a penetration test is. Unfortunately, due to the variety in size and complexity of IT systems, this is like asking how long is a piece of string. It depends what you are working with, and how much depth you need to go to. If you imagine it like painting a bridge, it depends how big your bridge is, and how many coats of paint you want, just a thin covering might leave you exposed to the elements.
Because of this, pen tests are usually quoted on a ‘day-rate’ basis, and very broadly, you can expect to pay anything in the range of £800-£1250.
Day rates vary from vendor to vendor based on things like reputation, certifications, and special requirements for the tester’s experience, although discounts can be negotiated if you’re buying lots of days (anything more than fifteen days would be considered a large test).
To understand how long your job will take, the vendor will often need to get a demo of your product, or gather information about your environment. As a rule of thumb, the less questions they ask at this stage, the less likely you are to get an accurately quoted piece of work. There’s also no standard when it comes to scoping a piece of work, so you might find estimates differ. One organisation may scope a job as 3 days work, and another as 5, depending on their viewpoint. These are their best estimates, it’s hard to tell for sure until you’re doing the work exactly how long it will take.
You can even buy “fixed-fee” penetration tests, but going back to the bridge analogy, you should probably be worried about coverage if they’re offering it for a fixed fee without asking how big the bridge is.
As with anything in life, the price you are quoted should reflect the quality that your penetration test will be delivered at - but in an industry where the quality of a test is hard to judge, there are bound to be some rogue traders out there. Take care to ask the right questions and don’t skip the due diligence process before deciding on a provider.
Hopefully that explains a few of the most important factors to consider when choosing a penetration testing company to use. We hope the information we’ve provided in this article has been helpful, and please do get in touch if you have any questions around the topics in the article or otherwise.
- Raw CVE Coverage
- Risk Rating Coverage
- Remote Check Types
- Check Publication Lead Time
- Local/Authenticated vs Remote Check Prioritisation
- Software Vendor & Package Coverage
- Headline Vulnerabilities of 2021 Coverage
- Analysis Decisions
Red teamers, security researchers, detection engineers, threat actors have to actively research type of vulnerability, location in vulnerable software and build an associated exploit.
Tenable release checks for 47.43% of the CVEs they cover in this window, and Greenbone release 32.96%.
Red teamers, security researchers, detection engineers and threat actors now have access to some of the information they were previously having to hunt themselves, speeding up potential exploit creation.
Tenable release checks for 17.12% of the CVEs they cover in this window, and Greenbone release 17.69%.
The likelihood that exploitation in the wild is going to be happening is steadily increasing.
Tenable release checks for 10.9% of the CVEs they cover in this window, and Greenbone release 20.69%.
We’re starting to lose some of the benefit of rapid, automated vulnerability detection.
Tenable release checks for 9.58% of the CVEs they cover in this window, and Greenbone release 12.43%.
Any detection released a month after the details are publicly available is decreasing in value for me.
Tenable release checks for 14.97% of the CVEs they cover over a month after the CVE details have been published, and Greenbone release 16.23%.
With this information in mind, I wanted to check what is the delay for both Tenable and Greenbone to release a detection for their scanners. The following section will focus on vulnerabilities which:
- Have CVSSv2 rating of 10
- Are exploitable over the network
- Require no user interaction
These are the ones where an attacker can point their exploit code at your vulnerable system and gain unauthorised access.
We’ve seen previously that Tenable have remote checks for 643 critical vulnerabilities, and OpenVAS have remote checks for 450 critical vulnerabilities. Tenable release remote checks for critical vulnerabilities within 1 month of the details being made public 58.4% of the time, but Greenbone release their checks within 1 month 76.8% of the time. So, even though OpenVAS has fewer checks for those critical vulnerabilities, you are more likely to get them within 1 month of the details being made public. Let’s break that down further.
In Figure 10 we can see the absolute number of remote checks released on a given day after a CVE for a critical vulnerability has been published. What you can immediately see is that both Tenable and OpenVAS release the majority of their checks on or before the CVE details are made public; Tenable have released checks for 247 CVEs, and OpenVAS have released checks for 144 CVEs. Then since 2010 Tenable have remote released checks for 147 critical CVEs and OpenVAS 79 critical CVEs on the same day as the vulnerability details were published. The number of vulnerabilities then drops off across the first week and drops further after 1 week, as we would hope for in an efficient time-to-release scenario.
While raw numbers are good, Tenable have a larger number of checks available so it could be unfair to go on raw numbers alone. It’s potentially more important to understand the likelihood that OpenVAS or Tenable will release a check of a vulnerability on any given day after a CVE for a critical vulnerability is released. In Figure 11 we can see that Tenable release 61% their checks on or before the date that a CVE is published, and OpenVAS release a shade under 50% of their checks on or before the day that a CVE is published.
So, since 2010 Tenable has more frequently released their checks before or on the same day as the CVE details have been published for critical vulnerabilities. While Tenable is leading at this point, Greenbone’s community feed still gets a considerable percentage of their checks out on or before day 0.
I thought I’d go another step further and try and see if I could identify any trend in each organisations release delay, are they getting better year-on-year or are their releases getting later? In Figure 12 I’ve taken the mean delay for critical vulnerabilities per year and plotted them. The mean as a metric is particularly influenced by outliers in a data set, so I expected some wackiness and limited the mean to only checks released 180 days prior to a CVE being published and 31 days after a CVE being published. These seem to me like reasonable limits, as anything greater than 6 months prior to CVE details being released is potentially a quirk of the check details and anything after a 1-month delay is less important for us.
What can we take away from Figure 12?
- We can see that between 2011 and 2014 Greenbone’s release delay was better than that of Tenable, by between 5 and 10 days.
- In 2015 things reverse and for 3 years Tenable is considerably ahead of Greenbone by a matter of weeks.
- But, then in 2019 things get much closer and Greenbone seem to be releasing on average about a day earlier than Tenable.
- For both the trendline over an 11-year period is very close, with Tenable marginally beating Greenbone.
- We have yet to have any data for 2021 for OpenVAS checks for critical show-stopper CVEs.
With the larger number of checks, and still being able to release a greater percentage of their remote checks for critical vulnerabilities Tenable could win this category. However, the delay time from 2019 and 2020 going to OpenVAS, and the trend lines being so close, I am going to declare this one a tie. It’s a tie.
The takeaway from this is that both vendors are getting their checks out the majority of the time either before the CVE details are published or on the day the details are published. This is overwhelmingly positive for both scanning solutions. Over time both also appear to be releasing remote checks for critical vulnerabilities more quickly.