If you think Risk Management is a term that’s been beaten and abused, the state of “metrics” is even worse.

Dear readers, I’ve been doing some research in the name of our little blog journey together (Ok, not really just for you, I have had other motives). For the past year, I’ve been trying to find out all I can about what people are doing right here, right now, about metrics.

The answer? Not a lot. And what most of us are currently reporting is such a major disservice to our industry that it’s almost better that we don’t even try. Seriously.

What I’ve done is read Jaquith’s book, I belong to the mailing list, I’ve talked to several organizations. What I have found is a similarity between the current state of Infosec metrics, and something else I’m passionate about, baseball. If you’ve read Moneyball, you know where I’m going. If you haven’t, that’s OK too.

The influence of big money into baseball has lead to a “metric” revolution in measuring the productivity of players. In baseball, the key is to measure how the individual performance of players on offense and defense, contribute to the won/loss record. Fortunately for baseball statheads - wins and losses are directly correlated to two simple things - the production or prevention of runs.  By throwing out the old stats and focusing on what does correlate to runs, a whole different perspective on the game has been obtained.  That perspective lead to a competitive advantage.  The Oakland A’s have been consistently competitive for the past 8 years - against teams with 2x to 5x their payroll.  So now, gone are archaic metrics like Batting Average and RBI - enter the land of VORP and WARP(3).  In this new baseball, Moneyball, a player is measured by how much they contribute to either producing and/or preventing runs. Why? Runs produced or prevented mean wins, and wins mean fans, and fans mean money.

From what I’ve seen, we’re still all caught up in measuring our equivalent of “old stats”, like baseball’s RBI. As I’ve asked around, most metrics we’re collecting and reporting are vulnerability management and A/V statistics. They are, after all, easy to record and report on. Heck, I’ve even seen InfoSec groups report “patch management” stats as the sole indicator of business value into a Fortune 500 ERM framework.

Here’s the problem. Assume the person across the table, your boss, knows nothing about Nessus, Patch Tuesday, or integrating security into the SDLC. If you report to him that you are “patching 98% of critical systems within 35 days as per policy” what do you think your boss just heard?

Now hopefully your boss is a little sharper than the PHB above. Even if they aren’t, metrics like “patch management” aren’t naturally in their business vocabulary. There isn’t a direct correlation there to what does matter to them, money. So they when we speak, they tend to hear the business equivelant of the teacher from Charlie Brown TV specials (”mwaah, mwah, mwaah, mwah mwaaaah”).

THE MARRIAGE OF RISK AND METRICS

You remember Robert Fort - CIO at Virgin US, and his quote from yesterday? His assertion is that PCI doesn’t help him do his job, sell records CD’s. You and I together cut through the article-speak to understand what he’s really saying here - Information Security doesn’t help him sell MP3’s  CD’s.

Which, of course, is counter-intuitive to you and I.  I hear you say, “Of course we help you sell music, because if we didn’t -  then Robert would pull all his firewalls out and sell them on eBay.”  And you’re absolutely right.  The problem is, nobody’s told Robert in his own language.

Ladies and Gentlemen, Boys and Girls, there are only two categories of IRM metrics that DO speak to non-technical data owners in the language they love.  Loss reduction and Risk reduction.  That’s it.  CIO’s and technical management (like Robert) or the direct boss of the CISO might also be interested in operational efficiencies - but those are more meaningful in light of situational variables (recession, for example), or meaningful only to the CISO.  In addition, most of the time the probable impact of operational metrics on profitability is much less than what we can usually express in risk or loss reduction efforts.

But those are the metrics we’re currently reporting, like number of viruses blocked.  They DO have some correlation to either loss reduction or risk reduction, to be sure.  But in isolation, and expressed independently of their impact on loss or risk reduction, they are practically worthless to data owners.

SO WHAT USE IS PCI?

As we’ve discussed before, PCI is an expression of someone else’s risk tolerance, a risk tolerance that they are asking you to accept and implement.  This may or may not match the risk tolerance of your organization, and unfortunately for you, the Card Companies have left the dirty work - selling PCI, up to you.  The success of your sales effort is going to be dependent upon whether you can successfully correlate the additional investment in controls and processes to risk or loss reduction.  Chances are that much of the PCI standard can mean risk or loss reduction - primarily because the penalties for non-compliance create a feedback loop (a wonderful term used last night by Mike Schiebel) wherein compliance mitigates the risk from it’s own loss factors.  Independent of that feedback loop, I’m not so sure PCI is that effective for companies that already take Information Risk Management seriously.Thing is that desired state where PCI doesn’t matter is irrelevant for now if you aren’t one of the lucky who can simply scan and self-evaluate.

Posted on June 19