I recently had the privilege of being a guest on the Securabits podcast and, during the session, was asked about other frameworks. I mentioned CVSS (Common Vulnerability Scoring System) in my answer and said I thought it had some serious problems as an analysis and measurement tool (however I also said there were good things about it). Given time constraints, I didn’t go into detail in the podcast about what I thought was good or less-good about CVSS. That’s what this post is about — to clarify and share my thoughts regarding CVSS (version 2.0).
In the interest of keeping this post to a manageable length I’ll constrain my observations to what I believe are the most important strengths and weaknesses of CVSS.
First, I have to acknowledge that what NIST and CMU have tried to accomplish with CVSS is both admirable and difficult. I can only imagine the debates that must have taken place during its development regarding tradeoffs that needed to be made in order to come up with a practical result. I also believe there’s value in CVSS, even as it is today. That said, like any other model or framework there’s always room for improvement. More importantly, like any other tool, its limitations should be well understood so that decisions based on it are made with both eyes open.
What CVSS aims to be
The CVSS guide mentions three key benefits the framework is intended to provide:
- Standardized vulnerability scoring — essentially, a common means of measuring “vulnerabilities”. I think the framework accomplishes this objective for technical vulnerabilities because it does, in fact, provide a standard against which technical vulnerabilities can be scored. Enough said.
- An open framework — i.e., a framework where scoring includes rationale so that the results don’t have to be accepted on blind faith. As described further on, I think the framework hits this target in some respects, and misses completely in others.
- Risk prioritization — i.e., a means of understanding the significance of vulnerabilities so that they can be compared and, thus, prioritized. Here again, in some limited respect CVSS accomplishes this objective. Overall though, as a CISO or other decision-maker, CVSS would not provide me with the information I need to make well-informed risk decisions.
An open framework
Great idea — a framework where justification is provided for the scores/measurements being used. And for the variables a user makes choices about within CVSS (e.g., Exploitability) there is some basic descriptive rationale in the selection matrix. Unfortunately, CVSS equations are also chock-full of weighted values, none of which appear to have clearly documented basis.
For example, the Base Equation multiplies Impact by 0.6 and Exploitability by 0.4. In other words, someone decided that Impact was always 20% more important than Exploitability. What’s the rationale for that? In fact, by my count there are five weighted constants in the base equation alone. Six more weighted values (eleven total) if you include the fact that each Base metric will be given a value that appears to be arbitrarily assigned (e.g., For Confidentiality Impact the score will be 0.0, 0.275, or 0.660 depending on whether the vulnerability is assigned “None”, “Partial”, or “Complete” for that metric). The other CVSS equations use weighted values in a similar fashion. Perhaps there are well-documented and thought-through rationale for each of these, but I haven’t found them.
In my experience weighted values are rarely well-justified. Furthermore, they tend to be very sensitive to specific conditions/assumptions. For example, someone might argue that strong authentication is a more important control than logging. After all, “an ounce of prevention…” Consequently, it might be tempting to “weight” authentication’s value higher than logging. Unfortunately, the logic breaks down if the scenario is focused on privileged insiders as the threat community — i.e., people who are supposed to have access. In that scenario strong authentication isn’t a relevant control at all and logging is much more important.
Unless there’s good rationale for weighted values, they introduce ambiguity, limit the scope of where the analysis can be applied, and can in some cases completely invalidate results. At the very least, if weighted values are going to be used, some well-reasoned rationale should be provided so that users can make an informed choice about whether they agree with the weighted values.
Effective risk prioritization
As a decision-maker, two of the fundamental inputs to any decision are “What’s the likelihood/frequency of bad things happening?” and “How bad are they likely to be if they do happen?”. These are the two values that, taken together, provide me with the loss exposure information I need in order to prioritize effectively. So, in order for CVSS to be an effective aid in risk-informed prioritization it has to provide useful information on both of those parameters.
CVSS tries to hit both targets, but falls short. With regard to frequency/probability of loss, CVSS focuses on the likelihood of attacker success from a couple of different angles, but never addresses the frequency/likelihood of an attack occurring in the first place. Without that metric, the likelihood of attacker success simply does not provide enough information for me to understand the frequency/likelihood of loss. CVSS may be trying to address the likelihood of attack through its Access Vector metric which, it could be argued, implies that the farther away an attacker is from the target, the less likely an attack might be. No argument with the logic (if that is in fact what the metric is supposed to represent), but there are a lot of assumptions built into that, including an assumption that the attacker isn’t an insider.
From a loss magnitude perspective, the Base Metrics include Confidentiality, Integrity, and Availability references but these are actually measuring something pretty different. In a longer post at a later date I might describe a way in which these CVSS metrics could be used in a very interesting way, but that would make this post WAY too long.
CVSS’s Environment Metrics try to include additional loss magnitude considerations. Besides being very qualitative, there appear to be some significant logic flaws in the approach. For example, the Target Distribution metric is essentially a measure of “surface area” (i.e., how many systems could be affected). One problem with this is that there are many scenarios where a single critical or highly sensitive system/asset is exposed (i.e., a small Target Distribution) but gross exposure exists. The way CVSS math works, this exposure would be unaccounted for. Something else to keep in mind is that Target Distribution is also a key consideration in loss event frequency (it may be even more important there in many respects), which isn’t accounted for at all in CVSS.
Setting aside the points above, prioritization of CVSS ratings against anything outside of CVSS isn’t practical because CVSS uses an ordinal scale. You can’t usefully compare something that was measured on a 1-to-10 ordinal scale against something that was measured in monetary values or, for that matter, in a different 1-to-10 scale.
I’ve blogged before about the problems associated with using math on ordinal scales, so I won’t belabor the point here. Suffice it to say that it just doesn’t stand up to scrutiny. That said, if the user recognizes that the results are pretty much meaningless for anything but comparing one CVSS value against another, then I guess no harm, no foul.
For all I know, the people who put CVSS together already thought through all of this (and the other problems within CVSS that I haven’t talked about here) and decided that what they came up with was the only practical result given the constraints they faced and their objectives. Nothing wrong with that. Trade-offs are inevitable. It is important though, for users of the tool to have a realistic and accurate understanding of its capabilities and limitations.
CVSS seems like a decent way to measure and compare technical deficiencies (“vulnerabilities”) against one another from a “(Very roughly) how much weakness does each vulnerability introduce relative to all of the other vulnerabilities measured using CVSS?” perspective, which can be useful information. What it doesn’t provide is meaningful information about how these vulnerabilities stack up in the bigger picture — i.e., “How important are these vulnerabilities relative to the other concerns I have to consider spending resources against?” In other words — “How much do/should I care about the findings?” In order to be useful in answering these questions, CVSS would have to evolve considerably.
Speaking of evolution… RMI has on the drawing board a potential alternative to CVSS that we believe will be both practical and more effective in characterizing the risk associated with vulnerabilities. Stay tuned!