Penetration Testing your AWS environment - a CTO’s guide
So, you've been thinking about getting a Penetration Test done on your Amazon Web Services (AWS) environment. Great! What should that involve exactly?
There are many options available, and knowing what you need will help you make your often limited security budget go as far as possible. Broadly, the key focus areas for most penetration tests involving AWS:
- Your externally accessible cloud infrastructure
- Any application(s) you're building or hosting
- Your internal cloud infrastructure
- Your AWS configuration itself
- Secrets management
We'll look at each one, starting with the most important:
The good news here is that, by default, AWS does its best to help you stay secure. For example, the default security groups don't let your EC2 instances receive communication from the outside world unless you actively specify it by adding additional rules.
That said, AWS still allows you plenty of rope to hang yourself with if you're not careful. Classic mistakes like engineering teams changing security groups to allow all inbound access are still a problem, and the nature of DevOps means services can be coming up and down regularly, not always with the knowledge of team managers.
Though, there is no easier way for a hacker to compromise you than finding a simple security weakness missed in your internet-facing infrastructure, whether that's an exposed database or software with known vulnerabilities. Attackers have the maximum payoff for the minimum effort, so the likelihood of this happening is the highest — therefore should be your first port of call to fix.
It can be challenging to stay on top of cloud vulnerability management due to the dynamic nature of these systems and continuous changes to your environment, with new vulnerabilities being released daily. However, modern vulnerability scanning solutions, such as Intruder, are customised to your cloud environment. You should consider using one of these tools before running a penetration test, as they help continuously manage vulnerabilities in your infrastructure with automatic scans.
As it's your most exposed attack surface, you probably wouldn't want to remove your external infrastructure from the scope of any pen-test. And, still, you shouldn't assign a large proportion of your budget to it if possible, and don't expect to see many results beyond what you've come to expect from your vulnerability scanning tools.
Many companies use AWS to host web application(s) for customers, employees, or partners. Unfortunately, web applications, designed to be exposed by their nature, present attackers with the second easiest way into your systems - if they're not developed securely. This makes them the second most crucial attack surface after your external infrastructure.
Examples of such attacks include the Kaseya incident in 2021, where attackers successfully compromised Kaseya and distributed ransomware to its customers in a supply-chain attack. The right-wing social media site Gab was also compromised early in 2021 and had 70GB of sensitive user data leaked because of a SQL injection vulnerability. Going further back, the famous TalkTalk hack, a 17-year-old customer managed to find his way into their customer database and extract millions of records.
Always consider the impact and likelihood of an attack at this layer. Whether your application is fully accessible to the public or a limited set of customers only should factor into your decision making. For example, applications with "free trials" would allow an attacker to sign up and start having a go. B2B services for paying customers/partners may have a lower threat profile, although still not negligible, and employees' apps are still lower. On the other hand, some applications contain such sensitive information that the impact may seriously outweigh the likelihood.
So, depending on the risk profile of your application, you may find that if you can only afford penetration testers to do a few days work, this is highly likely where you should be looking to spend their time. While automated tools exist for this type of testing and can be helpful to cover the gap between penetration tests, nothing on the market today can replace the quality of a human tester who will understand the business logic of your application and look for ways to impact it.
The next layer of attack is the infrastructure where your application is built. Having covered off the external infrastructure, the internal side is only accessible if an attacker already has breached your defences somehow. So, the threat profile here is secondary to the previous two.
Old-school penetration tests of data centres or corporate networks often revolve around gaining a foothold, then "pivoting" from one system to another, eventually leading to full-blown compromise of administrator accounts or critical systems. Here is where AWS environments can differ from traditional penetration tests, though, as AWS networks' software-defined nature often means tighter controls are maintained between networks, and lateral movement is a challenge. For example, once again, the default "launch-wizard-#" security groups don't let your EC2 instances talk to each other unless you actively specify it by adding them to a VPC or by adding additional rules. However, all but the simplest of AWS accounts get away with such simple configurations. In addition, as shown in the Capital One breach in 2019, attackers can compromise IAM role credentials and use those to access resources.
Additionally, the baked-in access and security controls in AWS mean that you're far less likely to have created compromised environment-wide "administrator" accounts via any of your EC2 instances. Instead, it's more likely that you're using privileged AWS accounts to do this, and so an AWS Config Review can add much more value than an "internal" infrastructure test.
Similarly, while unpatched software and insecure services on internal systems can be an issue, it depends to what extent you've created private networks in your AWS environment and what systems can access others. It's also worth understanding if you have a point-to-point VPN between your on-premises network and your cloud environments. If you do, an internal penetration test may be appropriate to find out whether an attacker can bridge the gap between these two networks.
The more complexity you have, the more an internal penetration test may add value. For example, suppose you're running a handful of EC2's each with their security group, or you're using some of AWS's shared/managed services like lambda functions - you may want to skip a traditional "internal" penetration test and consider a config review instead.
As mentioned, out of the box AWS does a lot for you in terms of security, but an AWS config review can tell you if you've set things up in a robust way.
Classic examples of poor AWS config are the exposed S3 buckets you often hear of or a lack of multi-factor authentication to access the AWS console. But, it can also include things like admin accounts with too many users being able to access them or more complex IAM rules like how a read-only access policy may allow an attacker to gain additional privileges in your environment.
Once again, this can often descend into paying someone to tell you what you already know (or could easily have found out). Before you commission a penetration test, try out some free tools (a quick google throws up a variety of options). The methodology is likely the same, and you may have the answers to your questions already.
If you're not confident in the security stakes or need a third-party audit for compliance reasons, it is valuable to connect with a cyber-security specialist, like Intruder, to uncover how they can help.
Secrets management is how secrets, like access tokens, are stored and used by your people and applications. It is at the bottom of our list, but it affects all the previous areas and deserves some consideration. The AWS configuration review should include, and inform you of, how your users and services access and interact with your AWS environment, including permissions assigned to those users and services. However, this configuration review will likely only be able to assess the configuration in your AWS account, meaning in the process secrets management may be overlooked.
Do your teams use continuous integration or continuous deployment (CI/CD)? If they do, then it's likely that the pipeline used during the CI/CD process will have a level of integration into your AWS environments. For example, they may have to start new EC2 instances or deploy new Lambdas. How are your internal applications or services which integrate with your environment storing secrets? How are your administrators holding secrets?
If an attacker can get access to these secrets, they will be able to access your AWS environment and be able to escalate privileges or maintain access to the cloud environment once they've been cleared off your internal network.
So, when you're considering a penetration test of your AWS environment, you may be interested in including the configuration of other integration systems in the scope of the test. Alternatively, you can split the process across multiple tools/assessments to focus on individual risk areas. An AWS configuration review will give you an understanding of how many things are connecting to your AWS environment using access keys and the AWS API.
Penetration testing in AWS should be treated carefully, as it would be easy to spend time and money in the wrong places. AWS is a vast ecosystem, and it's hard to cover all the ever-expanding number of services within a single point-in-time assessment, especially if you have a significant AWS presence. Sensible use of automation should always come before expensive consultancy hours, and when those are needed, they should always be used most cost-effectively. You may find that the most cost-effective way is a hybrid approach; you provide access to your AWS configuration, which can inform and guide a manual review of your complete AWS estate.
We hope this article serves you well, and if there’s anything we’re missing or you didn’t find here — do get in touch at email@example.com. We’d love to hear your thoughts!
Intruder is a cloud-based vulnerability scanning platform that can be used to check for known vulnerabilities in your AWS environment to reduce your attack surface. For a free trial for up to 30 days, click here.
This article was originally published in April 2020, and updated in October 2021.
- Raw CVE Coverage
- Risk Rating Coverage
- Remote Check Types
- Check Publication Lead Time
- Local/Authenticated vs Remote Check Prioritisation
- Software Vendor & Package Coverage
- Headline Vulnerabilities of 2021 Coverage
- Analysis Decisions
Red teamers, security researchers, detection engineers, threat actors have to actively research type of vulnerability, location in vulnerable software and build an associated exploit.
Tenable release checks for 47.43% of the CVEs they cover in this window, and Greenbone release 32.96%.
Red teamers, security researchers, detection engineers and threat actors now have access to some of the information they were previously having to hunt themselves, speeding up potential exploit creation.
Tenable release checks for 17.12% of the CVEs they cover in this window, and Greenbone release 17.69%.
The likelihood that exploitation in the wild is going to be happening is steadily increasing.
Tenable release checks for 10.9% of the CVEs they cover in this window, and Greenbone release 20.69%.
We’re starting to lose some of the benefit of rapid, automated vulnerability detection.
Tenable release checks for 9.58% of the CVEs they cover in this window, and Greenbone release 12.43%.
Any detection released a month after the details are publicly available is decreasing in value for me.
Tenable release checks for 14.97% of the CVEs they cover over a month after the CVE details have been published, and Greenbone release 16.23%.
With this information in mind, I wanted to check what is the delay for both Tenable and Greenbone to release a detection for their scanners. The following section will focus on vulnerabilities which:
- Have CVSSv2 rating of 10
- Are exploitable over the network
- Require no user interaction
These are the ones where an attacker can point their exploit code at your vulnerable system and gain unauthorised access.
We’ve seen previously that Tenable have remote checks for 643 critical vulnerabilities, and OpenVAS have remote checks for 450 critical vulnerabilities. Tenable release remote checks for critical vulnerabilities within 1 month of the details being made public 58.4% of the time, but Greenbone release their checks within 1 month 76.8% of the time. So, even though OpenVAS has fewer checks for those critical vulnerabilities, you are more likely to get them within 1 month of the details being made public. Let’s break that down further.
In Figure 10 we can see the absolute number of remote checks released on a given day after a CVE for a critical vulnerability has been published. What you can immediately see is that both Tenable and OpenVAS release the majority of their checks on or before the CVE details are made public; Tenable have released checks for 247 CVEs, and OpenVAS have released checks for 144 CVEs. Then since 2010 Tenable have remote released checks for 147 critical CVEs and OpenVAS 79 critical CVEs on the same day as the vulnerability details were published. The number of vulnerabilities then drops off across the first week and drops further after 1 week, as we would hope for in an efficient time-to-release scenario.
While raw numbers are good, Tenable have a larger number of checks available so it could be unfair to go on raw numbers alone. It’s potentially more important to understand the likelihood that OpenVAS or Tenable will release a check of a vulnerability on any given day after a CVE for a critical vulnerability is released. In Figure 11 we can see that Tenable release 61% their checks on or before the date that a CVE is published, and OpenVAS release a shade under 50% of their checks on or before the day that a CVE is published.
So, since 2010 Tenable has more frequently released their checks before or on the same day as the CVE details have been published for critical vulnerabilities. While Tenable is leading at this point, Greenbone’s community feed still gets a considerable percentage of their checks out on or before day 0.
I thought I’d go another step further and try and see if I could identify any trend in each organisations release delay, are they getting better year-on-year or are their releases getting later? In Figure 12 I’ve taken the mean delay for critical vulnerabilities per year and plotted them. The mean as a metric is particularly influenced by outliers in a data set, so I expected some wackiness and limited the mean to only checks released 180 days prior to a CVE being published and 31 days after a CVE being published. These seem to me like reasonable limits, as anything greater than 6 months prior to CVE details being released is potentially a quirk of the check details and anything after a 1-month delay is less important for us.
What can we take away from Figure 12?
- We can see that between 2011 and 2014 Greenbone’s release delay was better than that of Tenable, by between 5 and 10 days.
- In 2015 things reverse and for 3 years Tenable is considerably ahead of Greenbone by a matter of weeks.
- But, then in 2019 things get much closer and Greenbone seem to be releasing on average about a day earlier than Tenable.
- For both the trendline over an 11-year period is very close, with Tenable marginally beating Greenbone.
- We have yet to have any data for 2021 for OpenVAS checks for critical show-stopper CVEs.
With the larger number of checks, and still being able to release a greater percentage of their remote checks for critical vulnerabilities Tenable could win this category. However, the delay time from 2019 and 2020 going to OpenVAS, and the trend lines being so close, I am going to declare this one a tie. It’s a tie.
The takeaway from this is that both vendors are getting their checks out the majority of the time either before the CVE details are published or on the day the details are published. This is overwhelmingly positive for both scanning solutions. Over time both also appear to be releasing remote checks for critical vulnerabilities more quickly.