Cybersecurity Blog

The Limitations of Traditional Penetration Testing

Posted by Chuck Nguyen on Aug 8, 2019, 11:54:38 AM
Chuck Nguyen
Find me on:

INNO4 defines a “traditional” penetration test as a service that is primarily carried out by a human resource or resources using open source, commercial, and proprietary testing tools.shutterstock_1173031021

Vulnerability assessments and penetration testing have traditionally been used as one method to assess the security posture of an organization, business unit, application, among other areas. When performed properly, pen tests can provide a valuable data point in assessing your security. A good penetration test will identify, document, and prioritize areas that an organization should remediate to reduce risk and improve security.

While INNO4 does provide these services, we see three key limitations with traditional pen testing:

  1. Frequency: Businesses most commonly undergo a pen test annually, or even less frequently. The annual interval has been primarily driven by the (high) cost of a traditional pen test and organizations embracing the PCI DSS requirement for pen testing.

    Unfortunately, annual testing in many cases is not sufficient. Pen tests are snapshots in time and only assess a company at this specific point. Companies are dynamic and are constantly changing with business transformations, acquisitions, and new products, applications, employees, technology etc.

    Criminal hackers do not probe and attack organizations on an annual basis. They are constantly probing and testing your environment, looking for new vulnerabilities as your organization evolves.

  2. Time Allocation: Most organizations outsource pen testing, to companies like INNO4, completely or to augment their internal resources. Using an outside party provides a more objective evaluation of a company’s security program.

    Pen testing engagements are most commonly performed as a fixed fee service. In these cases, pen testing companies will allocate a set number of engineering hours for a pen test predicated on the size and objectives of an organization and the complexity of the testing. The pen testing firm will engage in testing until the objectives have been met or until all allocated hours are exhausted. A survey by Enterprise Security Group (ESG) found that: "the majority (75%) of penetration and/or red team exercises last only one to two weeks".

    One issue is that customers often mistakenly equate the end of testing to be a positive result, when in fact it is possible that the test was not long enough. For example, a company may only have funding for 80 hours for an external pen test. If the testing firm cannot gain a foothold into the internal network in that time, the assumption is that the customer’s external “network” is relatively secure. This assumption may be accurate or the pen testing firm could not run a sufficient number of tests in the allocated time.

    Bad actors are only limited by their motivation and have access to an increasing pool of resources including funding and access to support on the tools they employ. Criminal hackers can use broad attacks to identify “low hanging fruit” in addition to more targeted campaigns for key employees and assets.

  3. Testing Methodologies and Techniques: There are two issues related to testing methodologies and techniques. Pen testing can be disruptive and potentially service affecting. Testing firms work closely with customers to approve the methods that they will use during testing to minimize potential disruptions to the production environment.

    The second issue is that pen testers have a tendency to develop a style, which leads to using the same methods and techniques and slower integration of new techniques used by criminal hackers. 

    Attackers are not limited by a scope of work. They will use "low and slow" attacks to more disruptive ones such as ransomware and DDoS attacks. 

This is not to say that an annual pen test is not appropriate for all uses cases. There are scenarios where an annual test will be sufficient. For example, a security team may only perform an annual penetration test on a web application if only minor updates are issued throughout the year and the development team is using other controls such as an application security testing tool.

INNO4 has been testing different automated pen testing and breach and attack simulation tools, that aim to address these shortcomings, with some very positive results. Vendors in this space include: AttackIQ, Cronus-Cyber Technologies, CyCognito, Cymulate, Picus Security, Pycsys, Randori, SafeBreach, SCYTHE, Threatcare, Verodin, WhiteHaX, and XM Cyber.

Topics: Pen testing, Offensive security testing, Cybersecurity, Automated pen testing, Breach and attack simulation