The Cathedral vs. The University, or, Risk Management as a Science
Eric Raymond famously developed a paper that described the different approaches to software development called The Cathedral vs. The Bazaar. I’d like to borrow his idea there to briefly discuss approaches to security program management this morning. I’d like to call it The Cathedral vs. The University.
The other day somebody asked me "What do you have against compliance and best practices, Alex? You seem to be pretty down on them." Now these things have their place, even in the context of true risk management. However, I think we need to put their roles into perspective.
Right now, our profession is groping for answers in the dark. As a result, we’ve built standard upon standard and follow our compliance against these standards to tell us how secure we are. The answers we look for are written on the walls and in the scriptures. "If you abstain from a particular activity, you’ll never suffer negative consequences. If you always do this activity, you’ll be just fine." I compared this approach to the biblical Pharisee in a past post.
Recently, on wellingtongrey.net, I found "science vs. faith" diagrams that jumped out at me. I could see that the diagrams expressed my thoughts a little better. So I smacked them around in OmniGraffle this AM, and here’s what I came up with:
Faith-Based Security Program Management
Risk-Based Security Program Management
You see, good risk management follows an approach not unlike scientific method. Good risk management gives us reasons why we should be doing what we do based on how our world works. Otherwise we end up doing what we do based on how someone says he did it a few years ago — just running off and, as Jack says, "Sacrificing chickens because it’s what our forefathers did, and it’s what the tribe up the river does."
Andy Mar 16
So are you saying there is no place for “best practices” or that there isn’t a substitute for individual risk analysis for every threat and scenario that comes up?
David Hume was a skeptic. He tried to use analysis to prove that we can’t actually be sure of anything. He wasn’t 100% sure that gravity existed since he couldn’t prove it. He didn’t jump out windows though on the expectation that he wouldn’t fall to the ground.
There is a lot of common knowledge out there, we like to categorize things as humans and simplify decisions. We make these sorts of risk analysis decisions on a daily basis. I don’t know how fast I have to be driving to get hurt in an accident if I’m not wearing my seatbelt - but I put it on *every* time just in case. It is cheap and it came with the car.
Maybe a new chart would look like this:
Discover Risk -> see if looks like a previous risk -> decide how much time and energy you have to spend doing analysis vs. a best guess at a remediation -> implement and move on.
The rate at which issues comes up outpaces the number of decisions we have to make. We can’t do analysis for everything.
Perhaps I’m misreading your box “discover risk issue.” Perhaps I should be interpreting it as “discover novel risk/threat we’ve never thought about?”
Alex Hutton Mar 17
Hi Andy:
“there isn’t a substitute for individual risk analysis for every threat and scenario that comes up.”
Yes, that one.
Understand, this chart was just a lark, a bit of fun. It’s designed to contrast management approaches, not be something you put on a corporate white board as a sanctioned process (if you want to, great, but as you point out in your third paragraph - it’s not complete).
Without getting too epistemic about “risk”, I agree - we make “gut” analysis every day based on experience and our understanding of how the world works.
That’s one cool thing about having a good framework for risk. You can be as thorough and detailed as you want (or need to be) or use it to help you refine that “blink” (if you’ve read the book) - bring a level of objectivity into your subjective gut feel. Because, as you rightly say, we’re not going to be able to do *full* analysis for every risk issue that comes up. But at some level, we will be doing an analysis - the only question is how well we’ll be doing that analysis.
Unless, of course, you want to let a “standard” blindly make the decision for you. It’s a good choice for the lazy, but like any bureaucracy - there’s plenty of waste.
Andy Mar 17
Fair enough on the process piece. Though unfortunately in a regulated world we have to set the bar as maintaining compliance. So, we create a minimum set of policies and procedures so ensure we are compliant to a regulation. It is a necessary but not sufficient component of a risk management process.
Failing my PCI audit and/or having the FTC, SEC, FDA, etc. come in and shut me down is a much bigger risk in many situations than many kinds of external threats. Minus hurricanes, etc.
Worth considering when determining what your risk management process looks like and how baselines formulaic you want to be about certain things.
Alex Mar 17
“PCI audit and/or having the FTC, SEC, FDA, etc.come in and shut me down is a much bigger risk”
Is it? Are any of those guys going to really “shut you down”?
Compliance (fines/judgments) is only one of six forms of loss from an incident. It may be (unnecessarily?) the most aggravated loss from in a given scenario, but it’s only one of six to consider w/re: to risk.
“risk management process looks like”
Yes - but risk is bigger. Your risk management process should = your entire security process portfolio. Risk should be why you’re doing what you do, not compliance. Thus the contrast in the two graphs.
Every process - input and output, every “vital record” (sorry in the middle of BIA work), every metric - should all tell you about some risk issue.
Once you have a risk-world view, compliance isn’t so scary and in fact, can be seen as more of a hindrance to addressing high risk issues.
A fundamental question: Why does Compliance Exist?
Adam Mar 17
Two comments:
1) Compliance exists because Congress passes law. Congress passes law in response to percieved failures.
2) Failures happen to a great extent because we let everyone declare their ideas “best practices,” without strong evidence. See also http://www.emergentchaos.com/archives/2006/08/so_this_ummm_friend_of_mi.html
Alex Hutton Mar 18
Hi Adam!
Always the pragmatist. Let me offer that at a more fundamental level, compliance exists because of perceived risk and risk tolerance. GLBA was created because the FED had a different risk tolerance for loss than the F.I.’s. PCI because VISA/Mastercard/Discover had a different risk tolerance than their retailers, etc… Compliance “guidances” and such are simply a codified version of someone’s risk tolerance. As you rightly point out - our problems with them are because everyone’s got a risk tolerance. The auditor, the CISO, her analyst, the CEO, the outside “compliance” pressures, vendors, and on and on and on.
My point for Andy is that compliance should be subservient to (real) risk management.
Andy,
RE: Hume. You’re a Philosophy major - what do you think of Probabilistic Rationalists? Subjective / Objective Bayesians?
I’d be interested to hear (feel free to make it a blog post on your site) your opinions. We share some similar interests, which is pretty cool.
Adam Mar 19
Hi Alex!
I actually don’t agree. Your risk tolerance argument assumes that politics are efficient and wise. I don’t think that the background to either SOX (scandal) or HIPAA (political horse trading) support your argument.
Adam
Alex Mar 20
Let me think this through real quick. I’ve really no clue about the history of HIPAA so I’ll have to defer to you on that one.
But re: SOX - wasn’t/isn’t SOX just a reflection of the politicians risk tolerance? They may have a low tolerance for their own personal reputation damage (one of six forms of loss) and so made up a rather silly law?
Could GLBA be said to be a reflection of the risk tolerance of the Fed (replacement costs), and then the politicos (reputation damage, competitive advantage if you call ability to be re-elected a C.A.)?
Michael R. Farnum Mar 26
Many people think HIPAA is just about securing health information, hence the common misspelling - HIPPA - which people thinks stands for Health Information Protection and Privacy Act or something like that. If that was all HIPAA was about, then I would agree it is essentially useless. But in reality HIPAA stands for “Health Insurance Portability and Accountability Act” (nothing about “information” in there), and it was an attempt to standardize health records to reduce cost and fraud for Medicare and to protect American workers from the “pre-existing condition” issue in health insurance. The infosec piece came to be because of the realization that all this health information floating around could be misused (and that is NOT only information in electronic format - the privacy side of HIPAA deals with paper and the like).
It was actually fairly visionary for the government to think about the dangers of what they were doing. That’s not saying the regulation is worth a crap. But it proves that HIPAA was truly born from risk analysis / tolerance.
A good source of HIPAA history: http://www.hipaadvisory.com/regs/hipaahistorybyzon.htm
Alex Hutton Mar 27
Michael,
Thanks for the post and the mention. I have that much down, it was Adam’s “political horse trading” that kind of led me to believe there was something dark and sinister at work that I wasn’t aware of.
You know, the Illuminati, Skull & Bones, all that jazz