By and large, company executives are not stupid. You don’t fall into a job to run a company or a line of business. Being an executive means being good at making decisions with incomplete data. Should we launch this new product or service in Q2? Do we acquire this competitor or supplier? Do we buy this new whiz-bang IT thing?

We in the security profession have allowed this horrible myth to perpetuate our thinking. If you summed up the collective security professionals’ reactions to security failures, it would be something like “Yup, I don’t know what they’re thinking up there” or “Well, that’s stupid!” And yet, there was undoubtedly either an explicit or implicit decision made that allowed the security failure to occur. I ask, “Why was the decision made the way that it was?”

This whole line of thinking was set off by a discussion I had today with a security manager. We were talking about risk and security processes. We put a problem on the table and were collectively working on how to resolve it. The scenario could be summed up as system X may have insiders making fraudulent transaction that would cause us immeasurable reputation damage. However, organizationally, we cannot add any hardware or software controls to the system. We also cannot add or modify any networking devices between us and these systems.

“Is this really even a problem?” I asked.

What I went on to say is that somewhere, somebody decided that the probability of error was far greater than the probability of fraud. What’s important to all involved is whether this decision was made with the right data. For instance, was there a formal assessment of the risk that pointed to specific probabilities for error during implementation versus acceptance of the fraud? Our discussion continued and we came to a very important point: “Well, doing nothing is not an option,” the Security Manager said.

Indeed. Now the Security Manager didn’t say this, but I believe that we security professionals know that we are smarter than executives because our good decisions usually involve doing something. Yes, we must buy DLP. Yes, we must implement segregation. Sometimes though, our decisions are no: No, you cannot offer this new product or service because it’s insecure.

Doing the right thing does not always mean doing something. To hear our collective security conscience, one may think that security is easy: buy more stuff and you’re a good security manager. I think doing the right thing is a little more complicated than that. The right thing means reducing risk to an appropriate level. Sometimes this means not doing anything. Sometimes it means doing specific things (as opposed to anything) that will reduce our loss exposure. It usually always means prioritizing risk issues, working on the big ones, and monitoring the rest. We don’t have an infinite budget, so we can’t be a “good” security manager and just buy everything.

This is a problem bigger than just practice: it’s a sales process as well. I recently watched a presentation by a big consulting firm that showed “security maturity” graphs. Conveniently placed along the maturity continuum was this firm’s product sets.

“Wow!” I said. “The more stuff you buy, the more mature you are.” There was an awkward silence from the firm. The group I was with turned to me and smiled. They didn’t want to buy anything more and were happy that I said the obvious.

So my point is that executives are not stupid because they sometimes choose not to invest in security. I would hope (and it is our professional responsibility to ensure) that these decisions get made with the right data (data and not FUD). As security professionals, I would say that we are likely not responsible for company profit and loss, and at the level our executives work at, they likely know more about the business than we do or even can. Our job is to bring them the right data such that they can make an informed decision. This will sometimes mean that doing nothing is okay.

I recently had the opportunity to help an organization decide upon their risk assessment methodology. Being affiliated with RMI, I’m sure you know about which approach I feel the most strongly, but I was looking at this meeting from a different perspective. To me, this was going to be a battle over qualitative versus quantitative. In my view, it’s hardly rational to choose the method without deciding first that you want more objectivity and rigor (ahem…quantitative) versus shooting from the hip (qualitative). Indeed, once you decide you want a truly Quant approach (not just Likert scales) then we’ve already moved the discussion miles away from where the industry is currently.

I can tell you now that the meeting did not go well.

Oh, I won the argument, though. There were discussions about it being impossible to measure this or that, to which I parried with Carl Sagan’s invisible dragon (if some distinction mattered at all, then it would be observable in some way). People attacked Carl Sagan instead of focusing on the message–in all fairness, he can be a loon sometimes :-)  But not about this. I talked about biases, and mapping qualitative scales to quantitative ones unconsciously. The day before I even tabulated the results of a High-Medium-Low project “risk” assessment that was done and showed where 70% of the assessments were 3’s and 4’s (out of 5). We had a discussion about whether this supported good decision making.

We then came to an example. There was to be a presentation later that day to executives about the need for DLP software/hardware to address the “risk” of people taking information out of the organization on USB drives. Apparently, this was a big problem. I asked how often this happened.

Every day. Yes, that’s right: every day there was critical information taken out of the building via USB drives. Every day.

Surely, then, I reasoned, there must be losses associated with this. I mean, daily loss of critical information must have resulted in monetary loss of some type somewhere (anywhere?). Because, of course, if it mattered at all, then it must be observable. Seriously, what kind of people were they hiring? And why wasn’t this data more restricted? So many questions.

I went down a familiar path. I said that at the end of the meeting, you’ll ask for money, and then they’ll ask you how much less risk there will be (after all, this was a “High” risk).

“A whole lot less,” was the answer.

I expressed that I was skeptical of the potential success of this approach (in so many words).

“I don’t want to talk about this anymore,” was the answer.

And that is how I lost this discussion.

I don’t think I actually thought I would win this one. Sure, it would have been nice to drag this organization who so badly needed objective management into just such a place. But I knew there were a few things working against me:

1) Organizational Inertia – We usually use the first part of the definition when we say inertia. I’m meaning it here in the second sense: objects in motion tend to stay in motion. DLP was a crusade where many battles had been fought for over many years. This was an opportunity for them to ensure victory.

2) Confirmation Bias – This wasn’t a meeting to select a risk assessment methodology. It was a meeting to confirm that what was being done was fine. In large measure, they substituted vulnerability and threat assessments for risk assessments. It was, therefore, important for them to validate this approach by uniformly declaring that risk is incalculable.

In the end, this organization will choose a “risk methodology” (we left the meeting with an action item to review all the available methods). I don’t think it will matter what they choose. They all effectively say to multiply impact by likelihood (and don’t ask what those mean, just apply a High, Medium, or Low label to them thank-you-very-much). I shouldn’t complain though: these approaches make my job easier. I’ll just label three fourths of the risks as medium and high and be done. I should be done in an hour. After all, this organization must have an unlimited budget to apply to these problems. </cheekiness>

I share this story with you because I don’t think it is unique. I think everybody in risk and security fights this battle every day. I’ll end with this: if our profession is going to advance, we all need to adopt Carl Sagan’s argument: if some distinction matters, it is observable in some way. Or I should say that we need adopt it in some way. If such a thing could even be measured…

In the LinkedIn discussion mentioned earlier, some very legitimate concerns were raised regarding the inconsistency (variance) that can exist between risk analyses performed by different individuals.  Because not everyone is watching that discussion (and probably many who were got tired of it and moved on) I thought I’d post my thoughts on the consistency problem here.

For the sake of clarity, “consistency” as discussed within this post equates to “the likelihood that two independent analyses of a specific risk scenario will result in similar outcomes”. In other words, analyst A’s results will look very much like analyst B’s.

In order to frame the problem, we should ask ourselves where inconsistency tends to come from. In my experience, there are four key sources of inconsistency within risk analyses:

1) The scenario is scoped differently — i.e., analyst A is operating from a different set of assumptions than analyst B (e.g., is including different threats, different assets, etc.).  This, BTW, is a huge contributor to variance — in many cases it’s the single most significant contributor.

2) The analysts are operating from different analytic models — i.e., one analyst is using a model consisting of variables X, Y, and Z, while the other is using a model consisting of variables X, Y, and W. The models also may have different underlying formulas. The opportunity for inconsistency is especially problematic when analysts are using their own “mental models” for analysis, versus a structured model that can be explicitly referenced.

3) The analysts may have different experience levels and data sources — thus analyst A may estimate variable C to be between “5 and 10″, and analyst B’s estimate for the same variable may be between “40 and 100″.

4) Some people are lousy at estimating.

The first and second sources of inconsistency can be dramatically improved by ensuring that the analysts are singing from the same sheet of music — i.e., using the same model/method for analysis.

The third source of inconsistency can be significantly reduced (but not eliminated) by getting the right subject matter experts involved in the analysis. For example, as a security/risk geek I shouldn’t be estimating reputation damage. That’s the domain of business personnel. It also helps to have more than one person involved in the analysis to increase the experience and perspective the estimates are based on.

The fourth source of inconsistency can be significantly reduced (but not eliminated) through calibration training similar to what Douglas Hubbard presents in his book “How to Measure Anything”. You’d be surprised at how much improvement can be realized.

Bottom line — inconsistency in analyses is manageable to where the degree of variance is not significant relative to the decisions being made and the inherent uncertainty in the data.