A few days ago, when I was much busier being moral support to the Mrs., the Cyberphobia blog wrote an interesting article on ISO 27004.  27004 is all about measuring the effectiveness of our controls.

Our anonymous friend at Cyberphobia does a great breakdown “word for word” on the language surrounding 004, I won’t replicate it here but forward you to the site.  Go ahead and click over, I’ll still be here when you get back, and we can talk about why there might be issues with the 27004 standard.

Done?  Good, you’ll probably want to go back, bookmark the site and read some of his/her/their other stuff.  Good reads.

Ok, here’s the rub.  Those who have had training on FAIR’s sister framework, our Controls Analysis Framework - you’ll recognize immediately that the inputs into the Controls Analysis Tool are, in fact, relative to other inputs.  For my FAIR-loving readers, this “measurement of control effectiveness” should smell a lot like a “Vulnerability” calculation.

For those who aren’t FAIR-a-riffic - any specific “control” has different levels of effectiveness against different threat communities.  Conceptually speaking, I think we can agree that someone in the top 75% of attackers (in terms of skills and resources) will have an easier time attacking a system than someone in the 50-74%.  So in order to really “measure” effectiveness, there are (multiple) probable threat communities that we would need to measure against.

Second, any controls measurement of “effectiveness” is useless without some assurance discussion.  We all know about the company (not ours, of course) that has invested in some control that they never monitor, haven’t been trained on, and hardly use - but bought as a result of some “perceived” problem.  Some audit, some penetration test, some *something* said we needed XYZ, so we bought XYZ.  To be fair (pun) I’ve not read 27004, so I hope capability and motivation are part of whatever guideline language they develop.

So because the control is relative to factors we don’t (and won’t) have empirical data for (threat information and assurance information) if we’re going to “measure” anything, it will have to be a probability statement.  And, as you, my gentlereaders know, creating a probability statement using the traditional methods we learned in our undergraduate statistics classes just isn’t going to work given the nature of the evidence we’ll have for our estimates (and those trained and using RMI’s Controls Analysis Framework know where we’re coming from).  So color me skeptical, but I have a feeling that control effectiveness measurement is going to be an interesting exercise for the first few years after 27004 is published

I really like the idea of measuring control effectiveness.  I’m a big believer in RMI’s Controls Analysis Framework, and it’s ability to identify the best control solution for a company.  I just hope that the guidance developed by the ISO will be either broad enough to allow creative approaches like ours, or is extremely well thought out.

Posted on February 8