Save time with unbiased, independent feedback on vendor solutions.
Watch weekly bite-sized webinars hosted by IANS Faculty.
CISO metrics are composed of a measure, trend and goal. These metrics support one of the three C’s: coverage, configuration and capability effectiveness. This piece provides guidance for using these principles to create meaningful metrics to populate a CISO dashboard.
For many metrics, the problem is with their definition. Once what a metric is and how those metrics are built are defined, at least half the problems with creating those metrics vanish. The first challenge to overcome is to know what a metric isn’t. A metric is not a count at a single point in time. For example, “We have 20 critical vulnerabilities, 100 vulnerabilities of high importance, etc.” While counts of things play a part of metrics, there is much more to the story.
A Measure, a Trend and a Risk-Based Goal
A measure is more than a single number or count. Nine times out of 10 it is a ratio or rate. There are five types of measures for security:
Consider starting with the first two of the five measures above.
Burndown rates measure the rate at which risk is removed. This is a cumulative rate. For example, in December, you discover 100 critical (remotely exploitable) vulnerabilities. In that same month you fix 50 of those vulnerabilities. That is a 50% burndown rate. In February, you find an additional 100 vulnerabilities. Of the total remaining, you fix 30. That is 80 out of 100, or a 40% burndown rate. Over time, as you accumulate data, the rate baselines. Baselining means the rate is stable and more accurately reflects the underlying process it measures.
A baselined rate allows you to measure changes in trends over time. For example, the burndown over the last quarter may have baselined at 70%. Perhaps that is a 5% change from last month (75%). If your goal is 95%, your metrics would be at risk. Figure 1 offers an example of what this might look like.
Most metrics evolve to address two of the three C’s: solutions coverage and configuration of that solution. This is exactly where the Center for Internet Security (CIS) metrics focuses. CIS offers the sigma levels displayed in Figure 2 (for those unfamiliar with sigma levels, they are a way of measuring the quality of a process or product).
Imagine these levels applied to the question: What percentage of the organization's networks have not recently been scanned by an active asset discovery tool?
The second C, configuration, is a simple yes or no answer to a question. Figure 3 provides an example of how the CIS V7 metrics address configuration-based questions:
The most mature and most important metrics are capability-based. The burndown metric is a perfect example of this. It measures how efficient you are in meeting risk-based goals over time.
Nearly all CISO metrics should be capability-based. But which capabilities matter most? This is determined by looking at the most plausible threats and losses your business faces. Start with key losses, which can generally be broken down into a minimum of classes:
You may have other loss types critical to your business depending on your individual organization’s needs and experiences. Next, map losses to the threats you focus on. A list of top-of-mind threats includes but is not limited to:
Based on your analysis, you will have threat and loss combinations that represent the largest areas of risk. At this stage, you can “t-shirt size” (project manage and capacity plan for) the combinations that matter most. Figure 4 demonstrates an example of t-shirt sized risk relations:
Looking at this visualization, but without going into quantitative analysis, it’s clear that breach and wire fraud are the outsized losses for this example business. In this case, the business will want to have strategies to address those losses.
Ideally, you will want to invest in controls that offer value across multiple threat use cases. For example, you may decide that stronger IAM controls are needed to limit access to data (to prevent breaches) and purchasing systems (to combat wire fraud). While you may implement a longer term zero trust strategy, two controls related to zero trust are worth immediate consideration: MFA and PAM.
Further, these are examples of capability-based metrics that could be applied to the PAM side:
It’s tempting to say, “Why can’t someone just give me a list with a ready-made set of metrics?” However, metrics in this case must be derived from your risks.
DOWNLOAD: Create Incident Response Metrics Worth Reporting
It is hard work, but it can be rewarding. Following this process, you will find metrics that truly matter to the material risks your business faces.
Although reasonable efforts will be made to ensure the completeness and accuracy of the information contained in our blog posts, no liability can be accepted by IANS or our Faculty members for the results of any actions taken by individuals or firms in connection with such information, opinions, or advice.
February 29, 2024
By IANS Research
Access key data sets from the 2023 -2024 IANS and Artico Search’s Cybersecurity Staff Compensation Benchmark Report. Gain valuable insights on cybersecurity staff roles to hire and retain top security talent.
Access key data from IANS and Artico Search’s Compensation, Budget and Satisfaction for CISOs in Financial Services, 2023-2024 report. Find valuable insights around the Financial Services CISO role to help better understand your situation, improve job satisfaction and drive organizational change.
February 21, 2024
Learn why cloud IR is critical to security and not just another box to check. Find guidance to get started building a strong cloud IR program.