If you survey a set of people who deal with risk or security professionally (inside or outside of infosec) and ask them to list key “risks” within their scope of responsibilities, you tend to get an interesting set of answers.  For example, the list you get from an infosec professional might look something like:

• Insiders
• Lack of user awareness
• Data leakage
• Non-compliance
• Reputation
• Web applications

Why it matters

Clearly, all of these can be issues worthy of concern for an organization, so what’s the problem?  Well, maybe nothing.  If all you’re looking for is a list of issues that contribute to the amount of risk an organization has, then a list like this is probably fine.  A problem arises though, when you try to measure, compare, and/or prioritize these, for (at least) two reasons:

• They aren’t the same kind of thing — e.g., Insiders are a threat community, lack of user awareness is a control deficiency, data leakage is a type of loss event, non-compliance is a condition, reputation (damage) is an outcome, and web applications are a type of asset.  A very apples vs. oranges problem.
• They aren’t distinct or solitary in their contribution to risk.  In other words, two or more of them can be combined in different ways to describe different risk scenarios with different probabilities and consequences.  As a result, any individual measurement of significance in terms of risk is invalid.

The definition for risk mentioned above implies a measurement of some sort — i.e., a pair of values (some version of likelihood and consequence) —  yet we use the terms “a risk” and “risks” in a way that implies reference to one or more objects or “things” rather than a value.

Unfortunately, I see a lot of instances where people have tried to characterize “risks” in terms of likelihood and consequence, and it’s never pretty.  The results are very difficult to defend logically, which I suspect contributes to people’s notion that dealing with risk is hard.  My experience has been that once you get clarity around risk terminology the kind of confusion that comes from “risks” goes away and the problem becomes a lot easier to wrap your head around.

By and large, company executives are not stupid. You don’t fall into a job to run a company or a line of business. Being an executive means being good at making decisions with incomplete data. Should we launch this new product or service in Q2? Do we acquire this competitor or supplier? Do we buy this new whiz-bang IT thing?

We in the security profession have allowed this horrible myth to perpetuate our thinking. If you summed up the collective security professionals’ reactions to security failures, it would be something like “Yup, I don’t know what they’re thinking up there” or “Well, that’s stupid!” And yet, there was undoubtedly either an explicit or implicit decision made that allowed the security failure to occur. I ask, “Why was the decision made the way that it was?”

This whole line of thinking was set off by a discussion I had today with a security manager. We were talking about risk and security processes. We put a problem on the table and were collectively working on how to resolve it. The scenario could be summed up as system X may have insiders making fraudulent transaction that would cause us immeasurable reputation damage. However, organizationally, we cannot add any hardware or software controls to the system. We also cannot add or modify any networking devices between us and these systems.

“Is this really even a problem?” I asked.

What I went on to say is that somewhere, somebody decided that the probability of error was far greater than the probability of fraud. What’s important to all involved is whether this decision was made with the right data. For instance, was there a formal assessment of the risk that pointed to specific probabilities for error during implementation versus acceptance of the fraud? Our discussion continued and we came to a very important point: “Well, doing nothing is not an option,” the Security Manager said.

Indeed. Now the Security Manager didn’t say this, but I believe that we security professionals know that we are smarter than executives because our good decisions usually involve doing something. Yes, we must buy DLP. Yes, we must implement segregation. Sometimes though, our decisions are no: No, you cannot offer this new product or service because it’s insecure.

Doing the right thing does not always mean doing something. To hear our collective security conscience, one may think that security is easy: buy more stuff and you’re a good security manager. I think doing the right thing is a little more complicated than that. The right thing means reducing risk to an appropriate level. Sometimes this means not doing anything. Sometimes it means doing specific things (as opposed to anything) that will reduce our loss exposure. It usually always means prioritizing risk issues, working on the big ones, and monitoring the rest. We don’t have an infinite budget, so we can’t be a “good” security manager and just buy everything.

This is a problem bigger than just practice: it’s a sales process as well. I recently watched a presentation by a big consulting firm that showed “security maturity” graphs. Conveniently placed along the maturity continuum was this firm’s product sets.

“Wow!” I said. “The more stuff you buy, the more mature you are.” There was an awkward silence from the firm. The group I was with turned to me and smiled. They didn’t want to buy anything more and were happy that I said the obvious.

So my point is that executives are not stupid because they sometimes choose not to invest in security. I would hope (and it is our professional responsibility to ensure) that these decisions get made with the right data (data and not FUD). As security professionals, I would say that we are likely not responsible for company profit and loss, and at the level our executives work at, they likely know more about the business than we do or even can. Our job is to bring them the right data such that they can make an informed decision. This will sometimes mean that doing nothing is okay.

In the LinkedIn discussion regarding risk analysis, the question came up of whether risk analysis is even useful and, if so, how.  Before diving in — it has been pointed out that context is critical in any discussion, and particularly so when you’re talking about something like “usefulness”.  ”Useful for what?”, is a question that’s begged.  ”Useful to who?” is another.  So, for the sake of clarity in this post, the “who” is business management and the “what” is better informed decisions.

My experience has been that organizations rarely have the resources necessary to address all of the business opportunities, operational costs, and risk issues (of many sorts) that they face.  Consequently, they’re forced to choose what to do now, what to do later, and what not to do at all.  These choices invariably require some form of comparison between the issues, and comparisons are invariably based on some form of measurement (whether formal or informal/intuitive).

Between the three categories of issues (opportunities, costs, and risk), operational costs are probably the easiest to evaluate and forecast, although there’s plenty of opportunity for operational factors to change in ways that weren’t anticipated.  Forecasts regarding opportunities and risk generally are much more speculative, and it’s rare to see realistic business opportunity analyses that aren’t expressed as ranges and/or distributions, with some form of confidence statement.  After all, market demand, the competition, regulations, politics, and the organization itself (amongst other things) may change or behave in ways that weren’t foreseen.  In other words, despite best efforts, what was projected may turn out to be inaccurate.

Nonetheless, despite this inherent uncertainty, I suspect most business decision-makers would agree that they’d prefer to base their big opportunity decisions on some form of structured analysis that can help them understand what’s known, what’s less well known, and what’s highly speculative.  They would probably look pretty skeptically on conclusions and recommendations regarding a large, complex business opportunity that didn’t have a structured analysis behind it, if for no other reason than without the analysis they wouldn’t be able to challenge the assumptions and data the conclusions and recommendations were based on.

In my experience, business decision-makers likewise appreciate the information a well-structured risk analysis can provide.  It allows them to understand how we came up with our conclusions and recommendations, and allows them to challenge our results if our assumptions (particularly regarding loss magnitude) seem off-base.  If the analysis presents the information in more meaningful business-like terms (e.g., annualized loss exposure, or whatever), then so much the better.

Example

A project where I worked had been identified as having a significant amount of risk (a new application required users, a lot of them, to have admin access on their workstations/laptops — sound familiar?).  Unfortunately,  the project was the “pet” of one of our senior executives; a man who wasn’t a fan of infosec to begin with.  Worse, the recommendations we were making were going to delay the project and increase its costs.

As I anticipated, the executive was ummm… not happy with our conclusions and recommendations.  I believe his words were, “You guys are always crying about high risk, why should I pay any attention now.”  At that point I explained that we had recently begun taking a more structured approach to analyzing risk, and I asked if I could describe how we came to our conclusions on his project.  After about a five minute explanation and whiteboard demonstration of the analysis his response was, “Well, it’s hard to argue with that.  Let’s look at your recommendations again.”  In the end, he agreed to all of our recommendations.  More importantly, he became an advocate for us because he had an entirely different perspective on how we approached our work.

Useful?  Yeah, I’d say risk analysis can be useful.

I recently had the opportunity to help an organization decide upon their risk assessment methodology. Being affiliated with RMI, I’m sure you know about which approach I feel the most strongly, but I was looking at this meeting from a different perspective. To me, this was going to be a battle over qualitative versus quantitative. In my view, it’s hardly rational to choose the method without deciding first that you want more objectivity and rigor (ahem…quantitative) versus shooting from the hip (qualitative). Indeed, once you decide you want a truly Quant approach (not just Likert scales) then we’ve already moved the discussion miles away from where the industry is currently.

I can tell you now that the meeting did not go well.

Oh, I won the argument, though. There were discussions about it being impossible to measure this or that, to which I parried with Carl Sagan’s invisible dragon (if some distinction mattered at all, then it would be observable in some way). People attacked Carl Sagan instead of focusing on the message–in all fairness, he can be a loon sometimes :-)  But not about this. I talked about biases, and mapping qualitative scales to quantitative ones unconsciously. The day before I even tabulated the results of a High-Medium-Low project “risk” assessment that was done and showed where 70% of the assessments were 3’s and 4’s (out of 5). We had a discussion about whether this supported good decision making.

We then came to an example. There was to be a presentation later that day to executives about the need for DLP software/hardware to address the “risk” of people taking information out of the organization on USB drives. Apparently, this was a big problem. I asked how often this happened.

Every day. Yes, that’s right: every day there was critical information taken out of the building via USB drives. Every day.

Surely, then, I reasoned, there must be losses associated with this. I mean, daily loss of critical information must have resulted in monetary loss of some type somewhere (anywhere?). Because, of course, if it mattered at all, then it must be observable. Seriously, what kind of people were they hiring? And why wasn’t this data more restricted? So many questions.

I went down a familiar path. I said that at the end of the meeting, you’ll ask for money, and then they’ll ask you how much less risk there will be (after all, this was a “High” risk).

“A whole lot less,” was the answer.

I expressed that I was skeptical of the potential success of this approach (in so many words).

“I don’t want to talk about this anymore,” was the answer.

And that is how I lost this discussion.

I don’t think I actually thought I would win this one. Sure, it would have been nice to drag this organization who so badly needed objective management into just such a place. But I knew there were a few things working against me:

1) Organizational Inertia – We usually use the first part of the definition when we say inertia. I’m meaning it here in the second sense: objects in motion tend to stay in motion. DLP was a crusade where many battles had been fought for over many years. This was an opportunity for them to ensure victory.

2) Confirmation Bias – This wasn’t a meeting to select a risk assessment methodology. It was a meeting to confirm that what was being done was fine. In large measure, they substituted vulnerability and threat assessments for risk assessments. It was, therefore, important for them to validate this approach by uniformly declaring that risk is incalculable.

In the end, this organization will choose a “risk methodology” (we left the meeting with an action item to review all the available methods). I don’t think it will matter what they choose. They all effectively say to multiply impact by likelihood (and don’t ask what those mean, just apply a High, Medium, or Low label to them thank-you-very-much). I shouldn’t complain though: these approaches make my job easier. I’ll just label three fourths of the risks as medium and high and be done. I should be done in an hour. After all, this organization must have an unlimited budget to apply to these problems. </cheekiness>

I share this story with you because I don’t think it is unique. I think everybody in risk and security fights this battle every day. I’ll end with this: if our profession is going to advance, we all need to adopt Carl Sagan’s argument: if some distinction matters, it is observable in some way. Or I should say that we need adopt it in some way. If such a thing could even be measured…

For the sake of clarity, “consistency” as discussed within this post equates to “the likelihood that two independent analyses of a specific risk scenario will result in similar outcomes”. In other words, analyst A’s results will look very much like analyst B’s.

In order to frame the problem, we should ask ourselves where inconsistency tends to come from. In my experience, there are four key sources of inconsistency within risk analyses:

1) The scenario is scoped differently — i.e., analyst A is operating from a different set of assumptions than analyst B (e.g., is including different threats, different assets, etc.).  This, BTW, is a huge contributor to variance — in many cases it’s the single most significant contributor.

2) The analysts are operating from different analytic models — i.e., one analyst is using a model consisting of variables X, Y, and Z, while the other is using a model consisting of variables X, Y, and W. The models also may have different underlying formulas. The opportunity for inconsistency is especially problematic when analysts are using their own “mental models” for analysis, versus a structured model that can be explicitly referenced.

3) The analysts may have different experience levels and data sources — thus analyst A may estimate variable C to be between “5 and 10″, and analyst B’s estimate for the same variable may be between “40 and 100″.

4) Some people are lousy at estimating.

The first and second sources of inconsistency can be dramatically improved by ensuring that the analysts are singing from the same sheet of music — i.e., using the same model/method for analysis.

The third source of inconsistency can be significantly reduced (but not eliminated) by getting the right subject matter experts involved in the analysis. For example, as a security/risk geek I shouldn’t be estimating reputation damage. That’s the domain of business personnel. It also helps to have more than one person involved in the analysis to increase the experience and perspective the estimates are based on.

The fourth source of inconsistency can be significantly reduced (but not eliminated) through calibration training similar to what Douglas Hubbard presents in his book “How to Measure Anything”. You’d be surprised at how much improvement can be realized.

Bottom line — inconsistency in analyses is manageable to where the degree of variance is not significant relative to the decisions being made and the inherent uncertainty in the data. ]]> http://riskmanagementinsight.com/riskanalysis/?feed=rss2&p=726 http://riskmanagementinsight.com/riskanalysis/?p=721 http://riskmanagementinsight.com/riskanalysis/?p=721#comments Mon, 24 May 2010 14:48:30 +0000 JonesJ http://riskmanagementinsight.com/riskanalysis/?p=721 Most of us recognize that the challenges and questions we face in this profession are open-ended in nature.  In other words, there are no universally accepted “right” answers.  We all (or most of us) use the best information and resources available to us at the time, recognizing that they’re imperfect, to form our conclusions and recommendations.  We also seek to constantly learn and improve.

Dealing effectively with open-ended, complex problems requires critical thinking.  An excellent resource I’ve found that helps to characterize different levels of critical thinking maturity can be found here.

I’d be interested in your views regarding critical thinking as it relates to risk analysis vs. the “diligence” approach advocated by some.

There is more than one way to skin a cat.  True.  There are many tools that could be fashioned to varying degrees of efficiency and effectiveness to achieve the skinning of a cat.  What we must do if we are to take on such an endeavor, whether as a hobby or profession, is make sure that we know what a cat is.  We should know what it looks like, the components of a cat should be defined so as to alleviate any confusion.  No one wants to accidentally skin a dog.  That would be silly.

As skinners we may also take the opportunity to learn the anatomy of the cat.  The claws for example.  A cat’s claws may be dangerous and knowing whether they are present may be important.  We may find that we only want to skin those with claws.  As cat-skinning professionals, we should seek to identify cats and compare them so that we may skin only those worth skinning.  After all, there are rarely enough hours in a day.

Tools have and should be fashioned for the endeavor and certainly blacksmiths embraced in the capitalist spirit will embark on the fashioning of such tools.  They will be crude at first, there will be tools better suited for skinning bears and hamsters, there will be failures and successes.  The tools will evolve over time.  Rest assured, they will get better.

But beware: the would-be cat skinners must first define the cat, lest we make tools and in the interest of profit and industry we use them regardless, on all manner of creatures.  Aimless in our herding and poking and slicing, we may find our walls covered with the skins of dogs and hamsters and bears, asking, “Oh my… what is all of that meowing?”

In that instant when we see the signal change color we take in and process a remarkable amount of data and analyze the scenario using whatever mental model we’d developed through experience and education.  We then instantly apply the results of that analysis against our own tolerance for the different forms of risk that are in play (health/safety vs. legal, etc.).  If our data and model are reasonably complete and accurate, then we probably survive these events with an acceptable frequency and magnitude of loss.

That “mental model” is a construct that represents our understanding of how the different elements in the decision play together.  If our mental model is missing key elements — e.g., the effect of icy road conditions on our ability to stop — then our decision is much more likely to have an undesirable outcome.  The same is true if our model contains erroneous or inaccurate structural elements or relationships — e.g., a belief that icy roads will improve our ability to stop.

When we’re faced with a decision regarding information security we will also apply a model.  The model might be an informal mental model or something more formal like FAIR, TARA, CVSS or a host of other candidates.  So the question we want to ask ourselves is; “How accurately does the model we’re using represent the problem we’re trying to understand?”

Where models fit

It’s implied above, but the strategic role risk models play in an organization’s ability to be successful is outlined below:

• Effective management decisions are predicated on…
• Effective comparisons between the issues/options that are in play, which are predicated on…
• The ability to measure the issues/options in a meaningful way, which is predicated on…
• An accurate model (understanding) of the problem and its elements

The above is true regardless of whether you’re using an informal mental model or something formally defined.  Consequently, you can’t expect to consistently and effectively manage a complex problem space if the underlying model for measurement and comparison is badly broken.

An example

My last two posts already described some of the obvious and important ways in which risk models commonly used in infosec are broken.  But how badly broken are they?  For the sake of brevity, this blog post will not include a blow-by-blow analysis — I’ll save that for another time.  I will cite an example, from when I was a CISO, of how a flawed model almost had a significant impact on my employer…

We’d brought in a big-4 consulting firm to perform an attack and penetration exercise against us.  At one point in the exercise they came to the table claiming that they’d identified a number of “high risk” issues that needed to be addressed immediately.  I took one look at those issues, applied a quick mental sniff-test, and told them they were wrong.  I didn’t believe any of those issues represented a level of risk that warranted a high impact (to the business) response.  They agreed to sit down and review the issues with me so that they could show me the error of my ways.  However, after we broke down each of the issues in detail using FAIR, they conceded that none of the issues warranted an immediate, high-impact response.

Their original analysis (measurement) of the issues was shown to be based on an inaccurate model of the problem (risk).  This flawed measurement led to an inaccurate comparison of these issues versus the other issues and priorities the business faced, which would have had a significant negative effect on the business if we hadn’t recognized the flawed analysis.   (The flaws in their model involved how they treated threat event frequency and loss magnitude.)

Now, please don’t interpret this as an indictment of big-4 firms.  They have a lot of very bright people who do marvelous work.  And besides, I’m pretty confident the scenario would have played out similarly with almost any firm because the big-4 firm was using a very common assessment method.  The point is, if your model is broken badly enough, the results can significantly affect your organization.

There are models and then there are “models”

As I see it, there are three types of “models” being used in our industry.

• Checklist “models” (e.g., ISO, PCI, etc.)
• Maturity models  (e.g., SEI’s CMM for software development)
• Analytic models (e.g., FAIR, TARA, CVSS, etc.)

It’s a matter of opinion (/religious debate) whether checklists qualify as models.  It doesn’t matter to me what they’re called as long as we understand what they are.   Checklists are simply a set of security and/or risk management elements somebody believes is relevant and important.  Presumably, if you follow the checklist you’re better off than if you don’t follow the checklist, and for most of the checklists in our industry, I’d agree (up to a point).  So if you’re looking for a quick and dirty “are we generally doing the kinds of things we should be doing” litmus test, then checklists are fine.  They’re also fine for comparing one organization against another (which has some potential pitfalls), and showing progress against, for example, last year’s checklist results.  The downside to checklists is that they tend to be one-size-fits-all, they don’t help us prioritize or compare our options, and they don’t help us understand why the different elements are important or how important they are.

Maturity models tend to focus on measuring process effectiveness and process improvement on a relative basis — two very worthwhile objectives.  What they don’t do is explain the practical effect of process improvements — the “why” or “how much”.  Consequently, similar to checklists, maturity models don’t help us prioritize or compare options.

Analytic models (some might call them scientific models) attempt to describe how things work.  If they’re designed well and used well, they enable the practical and useful measurement of complex systems (systems in the scientific sense vs. the IT sense), explanation of cause and effect, and sophisticated what-if analyses.  In other words — if you want answers to questions like:

• “How much risk do we have?”
• “How much less/more risk will we have if…?”
• “Which of these issues is most significant and by how much?”
• “Which of these mitigation options is likely to be most cost-effective?”

… then analytic models are the way to go.  If, however, they’re designed badly or used poorly, then they can very easily lead to inaccurate conclusions and poor decisions.

Bottom line — all three approaches have their benefits and limitations.  For most organizations, some combination of them will be the best bet for effective risk management.  Speaking of which…

Does organizational maturity play a role?

Steve Dotson raised a very good question in his comment to Lipstick - Part 2.  Paraphrasing — he asked whether more mature organizations are better able to answer/analyze risk-related questions.  The short answer is “probably”.

More mature organizations generally have a more complete picture of their risk landscape — i.e., they likely have better visibility into what their assets are, where their assets are, the control conditions surrounding their assets, the threat landscape, and the loss implications from events.  This information should enable them to provide more precise data for analyses with better confidence than less mature organizations.  That said, less mature organizations can still get accurate and useful results from analyses — just often with less precision.  And, having gone through an analysis, the less mature organization can acquire a very clear idea of where their information gaps are, the significance of those gaps, and what can be done to fill those gaps.

Organizational maturity also can play a role in how much emphasis an organization will likely place on checklists vs. maturity models vs. analytic models.  An organization that’s very immature may place primary emphasis on checklists, just to get things moving in the right general direction.  They may only use maturity models to gauge where a few key processes are today and set preliminary goals for improvement.  They also may initially limit the use of analytic models to key, high-impact decisions.

More mature organizations often are looking to become more cost-effective and/or need to make business cases for continued improvement.  Good risk analyses can make that possible.  Speaking from personal experience, once you get your security/risk organization to a point where management no longer views it as the brightest/hottest fire burning in their landscape, it can become very difficult to get their attention (unless that attention comes in the form of cutbacks…).  Of course, even some “mature” organizations don’t have their security/risk ducks in a row and struggle to get management to care.  For these organizations, being able to explain “why” and “how much” through good risk analyses can be very important.  Conversely, taking lame risk analyses results to management can erode credibility and make future dialog even tougher.

At Jared’s suggestion, I logged into the RC demo and dug into the information in their help file that serves as guidance for its users.  However, rather than post specific observations about RC, I thought it would be more helpful if I simply provided a brief “Thinking Person’s Guide to Risk Assessment Tool Selection”.  Okay, maybe “Things to watch out for” is a better description.  Regardless, I hope you find it useful.

“First, do no harm” (Auguste Francois Chomel)

The phrase above was borrowed from Douglas Hubbard’s book “The Failure of Risk Management”.  (Buy it.  Read it.)  One chapter in the book is entitled “Worse Than Useless”, and in there he describes “structured” scoring methods that can, in fact, lead to worse decisions than if no scoring method was used at all.  To limit the length of this post, I’ll refer you to Douglas’ book rather than repeat it here.  Suffice it to say, amongst other things he describes the same concerns I’ve already posted about ordinal scales and scoring.

There’s likelihood and then there’s “likelihood”

Many information security risk assessment tools view “Likelihood” as a measure of how likely it is that an attack will be successful.  This is VERY different than a measure of how likely it is that an attack will occur and be successful.  Without including the likelihood of occurrence we could rate the “Likelihood” of my being attacked by a polar bear on the streets of Dayton Ohio as “high” because I have no effective defenses from such an event.  Bottom line — understanding likelihood of success is not very useful if I don’t also understand the likelihood of occurrence.

Of course, the first argument that someone’s likely to raise is, “But we don’t know how often some of these events occur!”  I’ll talk more about this in a future post, but the short answer is:

• Baloney.  I sit down regularly with clients who need to evaluate the risk associated with “rare” events or events where no direct evidence exists to draw from, and we’re able to arrive at frequency ranges that make sense and can be defended.  The key here is the term “ranges”.  We may not have the information we need to state exactly how frequently events might occur, but we absolutely have the means to generate frequency as a range.  Again — read Douglas Hubbard’s work.

Ambiguity and overlap

Besides the problems inherent in ordinal scales and scoring, another very significant problem is the lack of clarity and specificity in the elements being measured.  Unfortunately, many of the models I see in use are very poorly defined, with lots of ambiguity and overlap/redundancy between variables.  The result is that things are accounted for and measured multiple times.  Combine this with the ordinal scale problems, and the results are not defensible under any sort of scrutiny.

CMM limitations

Some models uses a CMM scale to rate the effectiveness of controls.  And although CMM is useful for rating process maturity, it’s not intended for nor effective at rating technical controls.

Compensate not, lest ye go awry

Many models have only one place to rate controls, and those control ratings tend to be applied solely to the Likelihood component of risk.  (This was a problem in the first version of FAIR).  Typically, what happens then is that users throw compensating controls in that bucket too, even though some compensating controls (e.g., recovery capabilities) affect Impact rather than Likelihood.  As a result, the effect of these controls are accounted for in the wrong part of the equation.

Chicken Little

The Impact ratings in most assessment models focus on what “can” result — i.e., what’s “possible”.  And, being the paranoid lot that most of us are, we turn this into an estimate of what a worst-case outcome might look like.  I don’t know what your experience has been, but out of all of the incidents I’ve been witness to and victim of over the years, not one has approached a worst-case result despite the fact that some of them had significant potential for really nasty outcomes.  In fact, as I’ve discussed this with colleagues in the past it’s become clear that worst-case outcomes are extremely unusual.  By characterizing risk events purely in terms of worst-case outcomes we provide an exaggerated view of risk, which management recognizes intuitively and writes off as “Chicken Little syndrome”.

The simple fact is that outcomes from incidents can range from inconsequential to catastrophic.  And although we can’t predict precisely which will occur from any future event, there are factors that we can use to help us understand and communicate the range of possible outcomes from worst-case to best-case and even what’s most likely.  If we want to communicate useful and believable risk information to management, we need to be able to deal with loss magnitudes other than just the worst-case outcome.

To summarize…

There are other issues I could raise, but here’s the short list:

• Be very skeptical of methods that use addition, subtraction, multiplication, or division with ordinal scales.  If you do choose to use them, recognize that at the end of the day you’re not going to be able to defend the results as truly quantitative, and you may have a very difficult time defending their legitimacy.
• Make certain that Likelihood includes a frequency component or, better yet, that Frequency is used instead of Likelihood.  Regardless, without some reference to the frequency/probability of occurrence the information’s usefulness is significantly reduced.
• Elements being measured, particularly if math is involved, must be as clearly defined as possible so that redundancies and overlaps can be avoided.  This also helps to prevent having the wrong element in the wrong part of the equation.
• “Quality” scales like CMM should only be used to evaluate the things they’re intended for
• If the tool only allows the user to describe one level of Impact (e.g., “High”), there’s a significant likelihood that users will choose a worst-case outcome.  This almost invariably inflates the risk rating well beyond the actual level of risk, which increases the probability that management won’t take the results seriously.

Bottom line — if we want our risk analyses to be taken seriously, it’s critical that we challenge the assumptions and models (including FAIR) underlying our tools.  Unfortunately, much of what I encounter in our industry’s risk assessment tool kit are examples of faux sophistication and poor definition.  Is it any wonder then, that many within our profession struggle to accept risk analysis as a viable approach?

So how long DOES it take the Sun to orbit the Earth?

I do need to reiterate my concern about a “model-less” analytic tool.  As Jared clearly states, RC is not constrained to any one model for measuring risk.  It’s intended to be an efficiency tool that allows the use of any risk assessment model.  From a marketing perspective that may be pure genius, I don’t know.  I suppose it could translate into a larger potential market because they wouldn’t be locking out those clients who are strict adherents of one model or another.  And certainly, if a user is leveraging a reasonably accurate model, then the tool’s effect should be very positive.  Unfortunately, as I described in the first part of this post, much of what our profession uses to model risk is junk.  In that case, a model-neutral tool is a bit like saying to an astronomer, “Hey, if you want to analyze the solar system by modeling the planets and Sun orbiting the Earth — go for it.  And while you’re at it, if you’d rather measure gravitational pull in bushels rather than units of acceleration, that’s cool too.  We’ll still allow you — in fact we’ll help you — to present the results as valid astronomy.”

The fact is, models matter.  A lot.  In my next post I’m going to talk about the role models play and I’ll also draw a distinction between the different types of models I see our profession using.