What is a Wise Risk Decision Worth? or ISO 27001 KPIs Follow Up
So yesterday I asked readers to comment on thoughts I had that came from a question asked on the ISO 27001 Google Group:
“How I can communicate the value of an ISO implementation to non-security management?”
This question came to me after one of the posters on the ISO Google Group asked about KPIs for ISO implementation. Got great responses in email, blog comments, and on Twitter from current/former CISO folks and consultants and analysts. Some really great thought and effort, by the way - thank you. It’s really great to be able to have these sorts of conversations online.
First, I have to point out some resources Brian Honan linked to from Gary Hinson, just because they’re so cool. Gary has invested gobs of time and effort to become one of the defacto resources on the ISO (you might also want to read or re-read Gary’s web post on the 7 myths of metrics). Brian links to an implementation guidance document(pdf) and a metrics example(pdf) document.
As full of awesomeness as they are, though, these are simply metrics “mapped” to the ISO (i.e. the ISO isn’t a pre-requisite for generating this information). They are not KPI’s that express the value of ISO implementation. Problem is the metrics created here still require some level of “translation” in order to create some value statement that data owners can understand. As Myrcurial twittered me “27001 is orthoganal to process” meaning (I hope) that metrics have their foundation in events that are generated by processes. 27001 by itself was never meant to create metrics (see above), and so we’re asking a question the ISO can’t answer. But the desire, the need to measure still exists. To that extent we can google “ISO compliance” (whatever that means) and if something can be certifiable or deemed “compliant” we can and are “measuring”. But does that have value? Rybolov (my favorite Guerilla CISO) wrote:
“Whatever you do, don’t start measuring percentage of compliance. Eventually, that’s what all metrics efforts around a framework devolve into.”
I have to agree. Being ISO “compliant/certified” has little expressive business value prima facia. I find that one KPI that absolutely asserts value when expressed properly is risk - and similarly Shrdlu wrote:
“I really have no idea. I personally wouldn’t try to justify an ISO implementation by itself. If I could show traceability on how it affected our overall security risk, then that’s what I’d do.”
And that’s a delightful answer. That “traceability” (geeze-louise Shrdlu - what a word!) is absolutely what I’m after here. How do I get that?
If you’re going to do something with corporate budget (time, money - and goodness knows an ISO implementation is time & money) you better be able to communicate the value. And while the zealotry for ISO implementation differs from person to person, I have yet to come across someone who says that ISO adoption is totally without value. It’s just not apparent what that value of adoption is and how we can measure (metrics) and express it (KPIs).
Jenean Paschalidis wrote what he thought that value was in a very nice email in which she (Edit- whoops!) puts a qualitative name on the value of adoption:
“Transparency and accountability-this is what all executive/senior management (the company) is on the hook for. ISO provides that. If you want to understand and have confidence in your operations as supported by security (because you will know the who, what, where, when, why and how of a system (human, technical etc.) and you want to be able to trace back why a decision (risk-vetted) had been made - then adoption of this best international practice will assist in providing these answers.”
So working with our above thoughts a little here - if we agree with Shrdlu that the only value of an ISO implementation can only be expressed if we can say how said implementation affected our overall security risk - and we agree with Jenean that the primary benefit is an ability to have confidence in operations as supported by security, then….
The value of the ISO should be expressed as a KPI or set of KPIs that cleary explain how the confidence it generates helps us understand (and then reduce) our risk.
If risk is a probability issue, ISO adoption helps generate confidence in our predictive analytics. The dollar value the ISO generates (the ultimate KPI) is part of the cost of being able to make wise risk decisions.
So what is that (making wise risk decisions) worth to you?
SOME CONCLUDING THOUGHTS
First, it occurs to me that this is a real shame. In a sense, an inability to generate a quantitative value statement for ISO use is simply more witch-doctory (“use it because we, the wise men of the tribe say you should”). In some future version, the ISO should include some mechanism for measuring and expressing the worth of adoption to the organization (a better reason to use the ISO than “because we said so”).
Second, It should be noted that of Jack Jones’ 3 true value statements from which all metrics/KPIs should point to - we’re only talking about one of those value statements - the ability to reduce risk. Using the ISO in an organization most certainly could create operational efficiencies (help us do more with less) - but the ISO isn’t a standard that creates operational efficiencies as a primary goal, nor does it give implicit direction on how to create operational efficincies. The ISO folks do, however, play fast and loose with the idea of “risk” and “risk management” so it’s within this context that I interpreted our conversation.
Finally if you’re going to hire someone to help you with ISO adoption in your organization, the deliverables you ask for in your RFP/SOW/what-have-you should include quantitative (probability) statments about risk reduction and the creation of operational efficiencies. If the firms answering can’t tell you what value their work will be to your company, then drop me a note and I’ll gladly point you to some friends of RMI’s that know FAIR & all our Risk Management frameworks and also do great ISO work.
Alex Dec 3
Post Script:
Traceability is, like, a totally and completely valid word that I found after the fact in Dictionary.app on my mac after Shrdlu twittered me about her use of it being a hold over from software dev usage in her past. Sure enough Dictionary.app says:
“Traceability refers to the completeness of the information about every step in a process chain.
The formal definition: Traceability is ability to chronologically interrelate the uniquely identifiable entities in a way that is verifiable.
Traceability is the ability to verify the history, location, or application of an item by means of documented recorded identification.[1]“
Gary Hinson Dec 3
Thanks for the tip o’ the hat, Alex! You’re right, the metrics papers I have written don’t specifically draw out KPIs and in fact my writing is generally rather light on specific metrics. The truth is that, like you, I’m still thinking and searching hard for “a few good security metrics”. I have some favourites but nowhere near a full set.
Progress on developing the set is laborious since:
a) We can identify bad/useless metrics much more easily than we can identify good ‘uns, but this means paring down the *huge* list of “security things we could measure”. We (or at least I) don’t yet have sufficient understanding to generate good security metrics de novo.
and
b) Effective security is generally anticipated to reduce the probability and/or impact of incidents, meaning that we are aiming to reduce bad stuff happening - but there are significant inherent unpredictabilities in our risk models and projections that make it hard to stand by many of the numbers we generate. It’s generally hard to state confidently what probably would have happened if we didn’t have such excellent security controls.
G.
Alex Dec 4
I’m thinking that the structure the ISO provides (and I would argue the process based operational metrics we can create using ISO as a framework) can serve to reduce the inherent uncertainty in measurement of model factors, the models, and posterior projections (”unpredictability”, used here, feels like a kind of loaded word that I’ll shy away from).
This is different from stating what would have happened - but it is creating accuracy in stating the probability of what might happen.
Gary Hinson Dec 4
Maybe I should have said inherent uncertainties rather than unpredictabilities but anyway I think we’re both agreed that reducing variation and increasing accuracy/certainty is a worthwhile goal. I slipped in ’state confidently’ implying not just that we trust our own numbers, but that we can support them confidently when facing a dubious management. I personally think that’s good enough for most real world situations, since in budget discussions other proposers are playing the same games with their investment models. They all have inherent uncertainties.
Vicente Aceituno Dec 8
Alex, if your question was “How I can communicate the value of an ISMS to non-security management?” instead of “How I can communicate the value of ISO implementation to non-security management?”, my answer would be “Explain them how the ISMS contributes to make more likely that business objectives will be achieved despiste of errors, attacks and accidents”. This is what ISM3 based ISMS are about, protecting Business objectives, like: Achieving a vision and mission, continuing to exist, maintaining and growing revenue, attracting, maintaining and fostering talent, maintaining and growing brand and reputation, complying with internal ethics and social responsibility goals and complying with regulations and contracts.
Alex Dec 8
@Vicente
And that’s why I like ISM3. You’re putting important things in place where they need to be. I really ought to write a post advocating it again, soon.