Risk Rating Litmus Test


  • One of the significant challenges the risk profession faces is the ability to prioritize.  What I see a lot of in the industry are tools and methods that spit out dozens or even hundreds of “High Risk” or even “Critical” findings from a single evaluation.  As a result, typically one of the following happens:

    • Paranoid organizations cripple their operations and/or burn out their people by trying to aggressively remediate those findings, or
    • Non-paranoid organizations schedule remediation efforts for months or even years out.

    In the first case, it’s common to see “committed” closure dates being missed and/or repeatedly pushed out.  This drives auditors nuts (as it should), and sets the organization up for a big fall if a significant loss event occurs.  Unfortunately, in both cases, there may be a handful of issues within the findings that truly are high risk or critical in nature, but because the organization hasn’t differentiated those, they get pushed out with the rest.

    Setting aside for a moment the debate over quantitative vs. qualitative assessment, I have a simple “litmus test” I apply to audit or security findings that helps me perform crude prioritization.  This test is based on a recognition that remediation efforts can/should be characterized in very practical terms and applied consistently.  Consider the following descriptions.  For:

    • Critical Risk findings:  All hands on deck.  Efforts extend into evenings and weekends.  High value business objectives may be postponed, extra resources brought in, and “costs be-damned”.
    • High Risk findings:  Remediation efforts begin immediately, bumping existing priorities and stealing existing resources.
    • Medium Risk:  Remediation efforts scheduled and prioritized amongst other future work to be done.
    • Low Risk:  Either no remediation or “opportunistic” remediation as a part of other activities.

    As a risk professional, if I’m going to label a risk issue “Critical” or “High Risk” and cause the organization to react accordingly, I’d better have a REAL good reason — a reason based on loss exposure (the combination of loss likelihood and impact) vs. “exploitability” or “vulnerability”.  Significant loss is either occurring right now, or it’s imminent.  And forget about formal analysis for a moment — if my intuition is telling me that remediation for issue X doesn’t need to be started immediately, then I am implicitly characterizing it as Medium.

    Some time ago I had a conversation with a friend who was faced with hundreds of “critical” and “high risk” findings from a single security tool.  We spent about 30 minutes categorizing the findings by common traits (e.g., exploitability, frequency of attack, and impact), and then another 30 minutes of evaluating which type of response seemed most appropriate.  At the end of the conversation there were zero Critical and just a couple of  High Risk findings.  Consider what this means to an organization from a resource utilization and remediation focus perspective.  Also consider what it means in terms of the improved accuracy with which the organization’s risk posture is communicated to management and stakeholders.  Finally, consider what it means regarding the accuracy of industry tools and common methods…

    Keep in mind that even though this approach may not require detailed quantitative analysis, it does still require an ability to think numerically in terms of frequency and impact, as well as how to apply critical thinking skills and recognize the difference between what’s possible vs. what’s probable.

    This post reflects my own opinions and positions, and does not necessarily reflect the opinions or position of my employer.

    Posted on

  • 10 comments

    1. MikeO Jun 6

      Whole-heartedly agree.

      Another ‘result’ that comes from communicating large numbers of Criticals and Highs, and I’d probably add this as a third bullet to the two at the top of the article, is the boy who cried wolf situation. At some point those that have the resources to fix the security issues just stop listening….

      (I always seem to remember this as the ‘Peter Wolf syndrome’ – probably from listening to too much J. Geils back in the 70′s in Boston! )

    2. JonesJ Jun 6

      Absolutely right. The credibility issue can often be a crippling problem.

      And yeah, I listened to a lot of Peter Wolf back then too.

      Cheers,
      Jack

    3. jeff g Jun 15

      “what’s possible vs. what’s probable” —> communicating the difference between these are critical!

    4. Aly Nov 3

      Hi JonesJ – Great advice and “litmus test”. It is amazing how much time and energy people can save by simply taking a few moments to go through their risk analysis and prioritize. I work for a risk software company here in Houston, we do schedule risk analysis so its a bit different but I think the prioritizing and planning apply to both.

      Anyways, thanks for sharing! – Aly

    5. Jack Jones Nov 4

      Hi Aly,

      You’re absolutely right. I can’t imagine an organization where prioritizing isn’t important, and the risk component of any significant decision should always be part of the equation.

      Cheers,
      Jack

    6. KY Feb 13

      “My colleague Dylan Evans has developed an online risk intelligence test that readers of your blog may find interesting. Dylan defines risk intelligence as the ability to estimate probabilities accurately, and his research has been featured in a number of blogs such as the Cassandra blog at the Economist (see http://www.economist.com/blogs/theworldin2011/2011/01/predictions_and_risk_intelligence) and Pharyngeal (see http://scienceblogs.com/pharyngula/2010/02/measure_your_rq.php). Dylan has discovered that people with high risk intelligence tend to make better forecasts than those with low RQ. Your readers can take the test for free by going to http://www.projectionpoint.com. Comments, criticisms and suggestions welcome.”

    7. farhang Apr 6

      Hi
      We made a new methodology and new mathematical approach to calculate priority of risks based on the real data, intellectually. For the very first time I’m publishing my ideas in http://risk-assessment.tumblr.com/ would like to use your experiences and may have a chat on that.
      To be honest I’m mostly into software rather than risk , so we may can have some two way communications to see what would be the outcome ;)

    8. Wilmes, LLC RIsk Control Services May 14

      Prioritization is tough for many reasons. RM and CFO’s rarely speak the same language which creates large gaps in the implementation and abatement of those hazards.

    9. Stacey Aug 16

      If you clearly look at it qualitatively you can see how important things are. The litmus test is a brilliant idea to help prioritise. Thanks for sharing.

    1. Pneumonia risk factors higher in modern urban apartment | Stop Smoking Now!

    Leave a reply