Lipstick on Pigs
There are a lot of really slick looking “risk” assessment solutions on the market today, with lots of eye-candy and great marketing verbiage to go with them. Look under the makeup though and, well, you’re going to see pork….
Now, before I start to point out the problems with those products, let me first say that some of them do provide value. Not the nearly the value they’re claiming, but value nonetheless. At the very least, they can provide an interface and structure for their users to identify “stuff” they view as problematic, a way to apply some sort of significance rating for that stuff, and generate reports about the stuff. So in that sense perhaps they make us more efficient at what we already try to do. They also help us look good in front of the auditors and other stakeholders who aren’t too interested, really, in digging in to see whether there’s any meat on those bones.
What these products don’t do, at least none of them that I’ve seen, is actually measure risk in any sort of defensible way. In fact, many of them don’t even really seem know what risk is. For example, let’s take a screenshot from one such product (which shall remain nameless).
Which of these things are not like the others…?
On this screen are shown a number of “risks” and where they (supposedly) stand from a significance perspective. My first observation is that they’re measuring apples, oranges, peas, and carrots here. Some of the “risks” are really control deficiencies (e.g., Unencrypted PII in email). Others are threat events (e.g., DDos), and yet others appear to be security program elements (e.g., Incident Response), policies (e.g., Log Retention), or problematic characteristics of the asset landscape (e.g., Proliferation of PII). Clearly, each of these issues contribute, to some degree, to how much risk an organization has, but they’re fundamentally not the same thing and therefore can’t be measured and compared to one another — at least not the way they’re trying to measure them in this tool. Furthermore, they aren’t independent of one another. For example, the “Proliferation of PII” (assets) is potentially a bigger problem within an organization that has “Rogue Devices” (control deficiency) and that does a poor job of dealing with “Employee Terminations” (threat landscape management).
What are they measuring, anyway?
Their measurement scale for risk is, appropriately, Impact and Likelihood. That should be good news. Unfortunately, their application of it is pretty much useless. For example, looking at the Incident Response issue, what is “Likelihood” supposed to represent? The likelihood of a breach due to poor incident response? The likelihood that incident response is not going to be effective if an incident does occur?
What about Impact? How do they come up with and “8+” Impact score for Incident Response? What kind of incident are they referring to? What if the incident is minor — i.e., someone changed the cafeteria menu?
What’s it mean?
While I’m on the subject of measurement, what does an Impact score represent here? Is it a most likely outcome, average outcome, or a worst case outcome? And what does a “9” represent? How much worse is it than a “7”, or a “2”? Unless these values map to something meaningful to the business, it’s just noise.
Similarly, does the Likelihood scale represent the likelihood of something occurring this year, this week, in our lifetime? What timeframe is this being measured against? And where on the scale do we account for things that might happen many times within whatever timeframe we’re referencing?
Without a lot more clarity regarding what these scales represent, the results are ambiguous to say the least.
Where the problem really lies is…
…with how our profession approaches “risk”. A user of this tool who doesn’t try to measure apples vs. oranges vs. peas, etc., but who instead only enters risk issues that can be measured in terms of frequency and magnitude will be FAR better off. They’ll still be stuck with poorly defined and marginally useful ordinal scales, but at least they’d be measuring stuff that makes sense.
At the end of the day…
All these tools are, are a way for an infosec professional to identify things that bother him/her about their current risk landscape and apply a rating that represents how much sleep they lose at night worrying about them. There’s nothing wrong with that. Unfortunately, the tools tend to be designed poorly, we use them poorly, and then we try to pass the results off as something they’re not. Oink.
Brian Barnier Apr 10
Jack, good analysis and nicely put. Lots of questions that people must answer to be doing real risk analysis, rather than just fill out charts. Keep it coming.
Brian
Jared Pfost Apr 10
I welcome the post but a title of “lipstick on *your own* pig” might be more appropriate. We made the tool above, Risk Communicator, to streamline the process of communicating risk drivers and justifying investments (maybe a good target of a second review?). A couple points to clarify, the value of the heatmap is a visual relative ranking of risks with an interface that’s easy to input and view the evidence for each. Clearly we need to do a better job educating users. In the help text, our website, the videos, we emphasize that tools cannot think for you, but simply communicate your evidence to predict loss events and outcomes.
Looks like we also need to help folks locate the guidance. We include our definitions of the scores. You’re also free to define them yourself. However, no one cares about relative numbers. They care about outcomes. Thus, we make it easy to input your conclusions and focus on the priority drivers, no tool can write it for you.
We have customers who use octave or FAIR but need a tool to visually communicate risk drivers, organize the evidence for each (click on a risk or view in the report), and more importantly, build a case to prioritize mitigation investments. Well documented risks are meaningless without a cost/benefit analysis for solutions.
For folks who aren’t satisfied with existing risk assessment frameworks, Risk Communicator offers a more detailed view breaking down the risk statement (click on Show Risk Model). We don’t fill in the details for you.
I encourage folks to adopt octave, FAIR, whatever they’re comfortable with. The value of Risk Communicator is ready-made “lipstick” backed with the evidence you document. In my past I spent way too much time trying to communicate risks drivers to stakeholders. This is an efficiency tool, not a Gallagher risk-o-matic.
On the apples-oranges comments, I agree. It’s up to the assessor to make sure the level of detail is consistent. However tools can offer consistency and efficient reporting capabilities.
If you think the canvas to document risks is too loose, wait till you see how Risk Communicator enables prioritizing investments. We encourage folks to incorporate subjective qualifiers like IT capability and business drivers. Investments are rarely justified by risk alone.
Sitting with the executives, the ability to quickly communicate what’s important, why, and what’s the best mitigation solution for the business is key. The obj of Risk Communicator is to make this process efficient.
aside: we just released Metrics Manager beta. I’d appreciate your feedback on our goal of making it easier to manage and visually communicate metrics.
- Jared, CEO, Third Defense
Jack Apr 11
Thanks Jared, for your comment and the additional perspective on Risk Communicator (RC). It’s clear that we agree on the efficiency aspect, the importance of outcomes, good documentation of rationale, and the need for the USER to have his/her act together in terms of what they’re measuring. I also agree with you about the importance of a cost/benefit analysis for solutions.
A few concerns remain that perhaps you can provide your perspective on:
* You mention the fact that nobody cares about relative numbers — just outcomes. Yet the quantification method used in RC uses relative (ordinal) numbers. So I’m not sure how the difference is bridged. How does one get from the relative numbers to outcomes? Maybe you can cite an example to clarify.
* It looks as though the risk score in RC is arrived at by multiplying ordinal Likelihood and Impact ratings. And although that’s a common approach, it’s problematic because the result of multiplying two or more ordinal numbers isn’t fundamentally sound. In other words, you can take any ordinal scale and convert it to colors or other qualitative terms like “high”, “low”, “very low”, etc., but you can’t multiply or do other meaningful math on qualitative terms. I don’t think anyone would try to multiply “red” times “yellow”, yet that’s essentially what we’re doing when we multiply ordinal numbers.
* I appreciate your proposed refinement of the title and characterization of RC as lipstick that people can apply to a runway model or a pig. The concern I have is that the value proposition and reputation of any tool will be affected by how it’s used and perceived (e.g., monkeys with hammers). The open “model-less” nature of RC and the fact that the users are free to define things as fundamental as scoring, what’s being measured, etc., would seem to make it highly susceptible to poor use. The double-edged sword of being flexible…
* Given the concerns above, it seems to me the “benefit” component of a cost/benefit analysis might become suspect.
Please recognize that RC wasn’t specifically targeted for this post. If it had been, I would have named it in the post. That screen image just seemed to provide a good example of how many (most?) of the tools in our industry suffer from some fundamental problems and, more importantly, that those fundamental problems are primarily the result of how our industry approaches risk. I’m more than happy to be proven wrong if I’ve mischaracterized RC, and will gladly post any corrections.
I’d be very interested in learning more about your metrics efforts, and would be happy to provide feedback if you believe it would be helpful.
Best regards,
Jack
Jared Pfost Apr 11
I think we’re in violent agreement. The challenge I faced and now hear from customers isn’t the quality of any framework, it’s obtaining quality evidence to justify predictions. For years my advice has been to only prioritize risks where real evidence exists e.g. incidents, assessment findings, peer evidence, etc. I think any modern framework is sufficient when you have the right facts.
The other area we need to invest in is the process surrounding risk prioritization and investments. We plan to publish example process flows, RACI, and metrics to help folks advance their practice. For now we have a few blog entries to help fill the gap.
RC will also do a better incorporating its help guidance into the application as it matures (currently v1.1). Feel free to log into the app from our website and read the guidance under “Help.” It provides a consistent approach to assign likelihood and impact, allows you to map dollar values to relative categories (just like octave, FAIR), helps construct your likelihood assertion, etc.
The value of relative ranking is communication efficiency. If users are misled by the numbers they could be replaced with any ordinal scale, say the alphabet. The 1-10 scale is simply a mapping to qualitative descriptions. It doesn’t matter if you add, multiple, or manually map qualitative descriptions on a x,y axis. I think what matters is the strength and organization of your evidence. We know we can do better and have many enhancements planned e.g. in the Detailed Risk Model, replace the 1-5 numbers with the text in the guidance (and allow users to change the definitions).
Frameworks just help ask the questions and capture evidence to facilitate the best decision for the business. Folks who criticize works like octave, FAIR, RC’s miss this point.
When I conduct an app assessment on say mobile banking I need a way to compare this evidence to the evidence gained from a directory authorization audit. Which is more important to the business, why, and what’s the best course of action. If I were king for a day I’d rename all risk frameworks to “evidence organizing tools.” Apologies for preaching to the choir.
One last note: RC is optimized to prioritize at the portfolio level. Many features are omitted that you see in tactical assessment frameworks e.g. breaking out threat actor and capability. RC is meant to quickly communicate evidence across your assessments, GRC data, industry, reg environment, etc. and conduct a cost/benefit exercise for investments. My old spreadsheet currently does a better job than RC in constructing a risk statement, however it doesn’t scale, facilitate debate, control versions, or have reports that executives can internalize i.e. lipstick :-). We’re also actively listening to users. If our approach is too loose, we’ll tighten it up.
Hopefully we can meet in person someday. If you’re ever in Seattle, please look us up. My direct email is jared@thirddefense.com
Jack Apr 12
I like your “king for a day” idea
although if I had that opportunity I’d probably use it to get people to use the term “risk” to mean just one thing rather than a dozen. My pet peeve.
Regarding data quality, that’s a refrain I run into constantly too. My response has been to suggest that people read Douglas Hubbard’s book “How to Measure Anything”. The fact is, and as Douglas explains in his book, we have more data than we think we do, and we need less data than we think we do.
It sounds like you and your team are putting a lot of thought and effort into RC, and it does appear to have some very cool usability and illustrative features. At your suggestion, I’ve logged into it and have taken a look at the guidance in Help. There’s a lot of information there, which I’m going to spend a couple of days digesting. I’ll then write another post with my observations. If you’d like, we can discuss my observations/conclusions offline before I post anything so that you can set me straight or at least know what’s coming.
Cheers,
Jack
Jared Pfost Apr 12
Jack,
An offline session sounds good. We have many enhancements planned and they might make a good backdrop for comments. Shoot me a mail anytime.
Chris Hayes Apr 14
Jack who??? Welcome back Jack – good to see you. It is amazing how much drama can result from a “probability / impact chart”; aka heat maps. Some thoughts on this subject:
1. Likelihood should be replaced with frequency. The challenge with likelihood is that you have to define the time parameter. Are we talking one month, one year, every five years, or every ten years, etc. By using frequency scales – we are able to capture both the likelihood and loss frequency scale. Same goes with impact; at a minimum ranges of dollars.
2. An effective heat map needs to have additional dimensions. For example, the size or shape of a bubble could be a third or fourth dimension. In FAIR terms, the size of the bubble could be reflective of vulnerability and the shape could be reflective of modifiers (unstable / fragile).
3. Indicating trending – either moving from or moving towards - (for a defined period) can also be useful.
4. There needs to be a fundamental understanding of the relationship between a risk or risk issue being plotted on a heat map and where it falls in an aggregate loss distribution. Ideally, the two should sync up.
At the end of the day – we have to be known for enabling our leaders to perform effective analysis and make effective decisions. If our “artwork” can facilitate that, great- otherwise we should just hang up our artwork up at the local elementary school, sign Johnny’s name to it, and sneak away.
Jack Apr 14
Hi Chris! Thanks for the welcome back, and for your insightful feedback (as always).
Phil Agcaoili Apr 27
Great analysis, Jack, and great responses, Jared.
Good minds think a like and very interesting that a natural intersection on thinking occurred here. Who’s [mostly] right? Right?
Over the past decade, many Big 4 consulting firms have used this exact heat map to help demonstrate, as a Risk Register, organizational priorities.
Jared’s conclusions that have manifest into his tool historically have been hand-built risk charts that I’ve watched these teams build. In fact, the only chart missing is the overall security capability maturity model spider chart: http://www.isaca.org/Template.cfm?Section=Home&CONTENTID=50624&TEMPLATE=/ContentManagement/ContentDisplay.cfm (see figure 4). I’ve seen quite a few firms average this score to represent a company’s relative security posture, set bars to demonstrate comparative industry maturity, and then set another bar to show that company’s targets for the various security domains.
All help CISO’s paint a picture of maturity and priority using risk terms to demonstrate priority and necessity.
In the past, this was done by hand and I welcome the automation.
In the end, if the community thinks this approach is ill-fated, I’d love to hear the opinion because I’ve now witnessed the use of this approach by the Big 4 at 3 companies over the past decade and the representation of the current and targeted security posture with the security organization’s priorities road mapped based on these charts.
Can’t wait to read the next post.