In the LinkedIn discussion mentioned earlier, some very legitimate concerns were raised regarding the inconsistency (variance) that can exist between risk analyses performed by different individuals.  Because not everyone is watching that discussion (and probably many who were got tired of it and moved on) I thought I’d post my thoughts on the consistency problem here.

For the sake of clarity, “consistency” as discussed within this post equates to “the likelihood that two independent analyses of a specific risk scenario will result in similar outcomes”. In other words, analyst A’s results will look very much like analyst B’s.

In order to frame the problem, we should ask ourselves where inconsistency tends to come from. In my experience, there are four key sources of inconsistency within risk analyses:

1) The scenario is scoped differently — i.e., analyst A is operating from a different set of assumptions than analyst B (e.g., is including different threats, different assets, etc.).  This, BTW, is a huge contributor to variance — in many cases it’s the single most significant contributor.

2) The analysts are operating from different analytic models — i.e., one analyst is using a model consisting of variables X, Y, and Z, while the other is using a model consisting of variables X, Y, and W. The models also may have different underlying formulas. The opportunity for inconsistency is especially problematic when analysts are using their own “mental models” for analysis, versus a structured model that can be explicitly referenced.

3) The analysts may have different experience levels and data sources — thus analyst A may estimate variable C to be between “5 and 10″, and analyst B’s estimate for the same variable may be between “40 and 100″.

4) Some people are lousy at estimating.

The first and second sources of inconsistency can be dramatically improved by ensuring that the analysts are singing from the same sheet of music — i.e., using the same model/method for analysis.

The third source of inconsistency can be significantly reduced (but not eliminated) by getting the right subject matter experts involved in the analysis. For example, as a security/risk geek I shouldn’t be estimating reputation damage. That’s the domain of business personnel. It also helps to have more than one person involved in the analysis to increase the experience and perspective the estimates are based on.

The fourth source of inconsistency can be significantly reduced (but not eliminated) through calibration training similar to what Douglas Hubbard presents in his book “How to Measure Anything”. You’d be surprised at how much improvement can be realized.

Bottom line — inconsistency in analyses is manageable to where the degree of variance is not significant relative to the decisions being made and the inherent uncertainty in the data.

Posted on June 1