Continuous Automated Pen Testing for Timely Cybersecurity Posture Measurement

Key Points:

  • Useful cybersecurity maturity/posture metrics are very difficult to generate.
  • Most of the metrics that controls generate do not provide reliable indicators of efficacy.
  • Traditional pen testing requires expensive human expertise which limits time and frequency.
  • Automated pen testing enables more frequent, broader, and unbiased controls validation.
  • Automated pen testing enables timely risk evaluation of new threats and new/updated applications.
  • Gartner's "Breach and Attack Simulation" category includes automated penetration testing.
  • INNO4 helps customize the blend of traditional and automated pen testing that's right for you.

Evaluating your deployed asset-level* cybersecurity controls is a critical task that’s part of the cybersecurity risk management and budgeting process. Are these controls performing at the expected levels of efficacy? Is the needed administrative effort what you expected? Have they adversely affected application or infrastructure performance? Have they injected more friction into business processes than you planned for?

*Asset-level security controls are the controls that prevent, detect, and respond to cyber incidents. Examples include network and application firewalls, endpoint threat prevention, detection, and response agents, email security gateways, two-factor authentication, and application authorization, to name a few.

The value of asset-level controls is their ability to reduce the likelihood and severity of cyber incidents. The trade-offs include the dollar costs of the controls, the administrative resources they require, and the friction they inject into user productivity and business processes.

After the controls are deployed, the administrative effort and friction are relatively easy to determine. However measuring efficacy is much more difficult. The metrics asset-level controls generate too often are misleading. For example, if you deploy a new end-point agent and the number of detected attacks increases, does that mean the control is doing a better job of detecting attacks, or has the number of attempted attacks gone up as well? More importantly, how many and what kinds of threats is the control not preventing or detecting?


Variance controls risk management v1

Using “Variance controls” is a more reliable method for measuring asset-level control efficacy. Variance controls by definition, are completely independent of asset-level controls. Penetration testing has been the long-standing variance control of choice.

However traditional pen testing has several key shortcomings. First is expense because they are performed mostly manually by well-trained, specialized security analysts. Second, due high cost, they are only performed once or twice year. Third, due to limited time/budget, each test tends to be narrow in focus. Finally, human pen testers tend to fall into patterns that they get comfortable with which means they may not be using methods that reveal weaknesses in your environment.

Another well-known variance control is Red Team / Blue Team exercises. While valuable, these are more oriented to testing the processes of your Security Operations Center and skill of your analysts, than the efficacy of security controls.

Pen tests ought to be performed regularly in response to the following events that can affect the risk of a cyber incident:

  • New vulnerabilities or threats are published*
  • New applications are deployed or existing applications are updated
  • Infrastructure changes
  • Initial adoption or increased adoption of public cloud or SaaS applications
  • Personnel reorganizations
  • New business initiatives
  • Mergers or acquisitions

*You might ask, why do I need to perform pen tests more regularly when I am already performing vulnerability scans on a regular basis? The answer is that vulnerability scans are focused on finding vulnerabilities in applications and operating systems. Penetration tests are focused on your organization’s overall ability to resist cyber attacks, and include evaluating the efficacy of the asset-level controls you have deployed.

Due to the importance of cost-effectively testing the efficacy of asset-level controls on a regular basis, a new category of variance control has emerged. One might simply call it "Automated Pen Testing." However, Gartner coined the term, “Breach and Attack Simulation (BAS).” Maybe Gartner felt that the initials of “automated pen testing,” APT, would be confused with Advanced Persistent Threats, an unfortunate coincidence. The key point is reducing the human element in order to enable unbiased, repeated, cost-effective, execution. 

We have found a wide range of functionality among the twelve and counting BAS offerings. Some are more oriented to individual security control validation. Others are oriented to a more realistic attack simulation. And among these, there is a wide variation in attack methods used and how far they go.

Actually, there is no accepted definition of the term “simulation.” In fact, at least one vendor does not like the BAS term at all, and prefers “Automated Penetration Testing” despite the "APT" coincidence I mentioned above.

There are other issues to consider as well. Do you need to deploy agents? If so, how many, and where? How many attack surfaces can be tested? Are the results located in the cloud or do they stay on-premise?

Whether you call it Automated Pen Testing or Breach and Attack Simulation, it should have a place in every organization’s variance control strategy because evaluating the efficacy of your asset-level controls once or twice a year is not sufficient.

There are several ways INNO4 can help you leverage pen testing and Breach and Attack Simulation tools:

  • Engagement – INNO4 runs one or more BAS tools in a specific engagement with or without complementary traditional pen testing, and provides analysis and recommendations for improvements.
  • Continuous Product – Deploy one or more BAS tools in your environment and run them on a regular basis. You do your own analysis. INNO4 helps in the tool evaluation and selection.
  • Annual Service – Deploy one or more BAS tools in your environment, and INNO4 performs and analyzes the results, and makes recommendations. Can be complemented with periodic traditional pen testing.

If you would like to learn more about INNO4's pen testing and Breach and Attack Simulation capabilities, please enter your Name, company, email, and phone number.

If you would just like for us to periodically update you on pen testing and Breach and Attack Simulation news, just provide your email address.


Get Free Widget