Why Metrics Will Fail
Today’s Dilbert inspired me to put some thoughts down concerning Metrics. You have to ask yourself - how does Adams still keep content fresh? It’s genius.
On To Metrics
{updated content} Now when I say “fail”, what I should really be saying is probably more along the lines of “frustrate immensely in the short term” or “given the same industry patience as Risk Management…” {end updated content}
There’s quite a move towards “metrics” these days. Of course, back in my day, we didn’t have such newfangled things like “measurement” and “statistics” it was all me and Jed out back by the server room with nmap and grep, beef jerky and Mountain Dew. No sir, these days people want dashboards and countin’ stats. They say things like “what gets measured gets done.”
Now I’ve already covered the 3 possible reasons to capture metrics (twice). The question is, “how do we get there?”
And the answer might just be “we don’t get there.” Three reasons:
- Difficulty in coming up with the right risk based metrics. Look, I’ve spent some time with this “risk” thing. ‘taint easy. I think it’s possible, but it’s not as easy as building some clever dashboard that gathers some XML data from some network devices. Heck, we can’t even come up with a logical framework for what risk consists of (oh, wait, maybe we have).
- Folks are interested in precision, not accuracy. I’ve already covered the engineer vs. scientist thing and all the problems with not using stochastic risk methods. All I can say is that if we don’t drop the engineer and accountant world views and start working on problems - good metrics aren’t happening. We’re going to be stuck with quality assurance metrics wrapped around a fillet of ISMS. That’s not science, unless Deming is your Feynman, and W. Edwards just looks waaaay too serious to play a song on the bongos about Orange Juice (mp3).
- Our products fail us. Part their fault, part ours. We’re not telling people what we need to see (because we don’t know) and so they’re just throwing information that they can gather at us. Tripwire might be a good exception.
I’m all for the move to metrics, but unless we know why we’re counting what we’re counting, we can’t even have a conversation about what to do with it once we have counted it!
Take The Poll, Take It, Take It!!!
rybolov May 16
It all starts out when you try to justify something.
For example, I know where most of my incidents occur, I track that metric. When I go in to justify that we need a full-time asset manager, I pull out the incidents relating to asset management.
I think it’s at least the bare minimum that you need to be doing. As we mature, I’ll bring in some more metrics for the managers to look at.