Load of Tosh?
Long time no post… My sincere apologies, and I hope someone out there is still interested. I guess I needed a little prodding, which Stuart King so kindly provided. I’ve provided a response on his site, which I won’t repeat in its entirety here. Suffice it to say that Stuart seems, on occasion, to rant about things he’s not fully informed on. My friend Chris offers a couple of very good posts in response to Stuart as well.
On to something a bit more useful and, hopefully, interesting. An acquaintance of mine recently showed me the controls analysis model he and his colleagues had developed. It was very complex, full of interesting and sophisticated formula, and… weighted variables. Argh. I’ll save my comments regarding sophisticated formula for another post. As for weighting, for those of you who haven’t heard me rant on this before, I’ll summarize: weighted variables seem to me simply another way of saying “We believe X is more important than Y, but we haven’t taken the time to figure out why or by how much“. In the absence of a clear underlying explanation for why/how much, I think it would be very difficult to rationally defend any degree of weighting that’s been applied.
Another problem is that weights are, in many cases, dependent on context. For example, in his model, he had weighted authentication as more important than logging and monitoring. And intuitively we might think, “Yeah, that seems right”. But what about the scenario where the threat agent is the privileged insider - the person that legitimately has credentials and access? In that case, wouldn’t logging and monitoring be more important?
Bottom line - the concept of weighting is attractive but problematic. If you’re going to use it, recognize some of the challenges that come with it. Oh, and let me know if you’ve developed a set of rationale that you use to support the specific weights you’ve applied.
Patrick Florer Mar 23
Hello, Alex -
Nice to read something from you!
It seems to me that the most difficult part of any modeling effort is coming up with reasonable and defensible assumptions that have a clearly documented rationale.
The other piece of modeling that people seem to miss is the need to do a range of scenarios, from best case to worst case - all documented - so that you begin to develop a feel for things.
The last step of all is what some call the “sniff” test - sometimes, no matter how diligent you are, the output just doesn’t make sense and it’s time to go back to the drawing board.
The nice thing about having definitions, taxonomy, and method is that it makes it possible to figure out where the problems are and to take a different approach.
Hope you are well,
Patrick
shrdlu Mar 23
Glad to see you back, Jack!
I agree completely about the weighted variables. I suppose if you had a static, consistent rationale for holding one thing as being more important than other things, you might get away with it, but as you say, you still have to explain why. So much depends on the context, the environment, the controls, the threat landscape, and the business objectives that I don’t see how you could always portray one control as being more important or one threat as being more prevalent.
Chris Hayes Mar 23
Hi Jack. I have recently seen where weighting was used for determining the reliability of data spanning time. For example, let’s say we have six years of data of break-ins for a company’s corporate locations, instead of summing all of them and using the mean value in a distribution, we would take percentages of each years value and then sum (year 1 – 2%, Year 2 – 3%, Year 3 – 10%, Year 4 – 30%, Year 5 – 50%). The justification for such a use of weighting could be current data is more reliable – but does not completely negate the usefulness of older data.
I do agree that weighting with respect to gauging effectiveness of security resistance against threat populations is problematic.
I have some thoughts regarding assumptions as well but will save those for a future blog post.
Chris Hayes Mar 23
BTW - My percentage allocations do not add up to 100% - oversight on my part.
Patrick Florer Mar 23
Ooops -
Sorry, Jack -
Thought I had read Alex’s name at the top of the post.
With regard to weighting, unless you are doing a straightforward calcuation of a weighted mean (200 / 1000 = 20%, 10/100 = 10%, 210/1100 = 19.1%), then you are probably pulling numbers out of the air according to some set of assumptions. Someone else could agree or disagree with what you have done.
The problem with using historical data, whether weighted or not, is that it’s usually hard to know when extrapolation is the best predictor of the future and when something else, like cause and effect, is more appropriate.
I guess if I knew the answer, I wouldn’t be spending any more time reading infosec blogs!
Patrick
Jack Mar 23
Thanks Shrdlu. It’s good to be back.
Chris — I agree that more recent data is likely to be more relevant in many cases, and the usefulness of representing that through weights is clear. The problem I have is defending the weights that are applied.
Patrick — No worries! It’s a complement to be confused with Alex in the blogosphere.
bachroxx Apr 14
Good stuff. I often use (and struggle with) weighting of risks. It is definitely important to explain to management, regulators, etc HOW I came up with the numbers and the limitations of the model. Brian