Bill Brenner has an article in CSO magazine in which “Fortify Co-Founder and Chief Scientist Brian Chess says:

“2009 will mark the end of pen tests as we know them.”

It’s very vogue, this “X technology is dead” mantra I’m hearing from analysts these days.  To be fair, Brian does say:

“…Death doesn’t mean it goes away, it means it transforms. Pen testing will be reborn in the area of production monitoring and measurement…”

Now he doesn’t tell me what he means by production monitoring and measurement, but I’ll give you my thoughts on the subject.

HEY, HEY - MY, MY!  METASPLOIT WILL NEVER DIE!

Me, I’m very bullish on PenTesting.  PenTests, as a tool, have two purposes:

• A discovery tool (find problems)
• A scientific tool (test hypotheses)

The value proposition of the first purpose is fairly self-evident, we’ve been using PenTests as a tool in this manner for years, and I don’t think I need to take any space here to talk about it.

But the value of “hypothesis testing” may not be so self-evident, so lets talk about that.

YEAH, I’M BIG ON SCIENTIFIC METHOD AS A MEANS TO MANAGE QUALITY OF IT SERVICES, WHAT ARE YOU GONNA DO ABOUT IT?

You and I, the risk people who must translate the value of security into business nomenclature, our job is to make educated hypothesis around the probable frequency and probable impact of a threat event.  To do that in the context of tactical risk identification and reporting, we need at least 4 key pieces of information:

1. The probable frequency with which we can expect threat communities to act against us (Threat Event Frequency)
2. The probable impact ($) the business stands to lose in a successful threat event (Probable Loss Magnitude)
3. The probable force a threat can apply against us (Threat Capability)
4. Our probable ability to resist that force (Resistance Strength).

Until you have useful information for those 4 factors, you can forget understanding your exposure to risk for an asset or group of assets (we could collectively call a ‘business process’).

Penetration Testing involves understanding the balance between 3 & 4 (it is, for the FAIR trained, the understanding of “vulnerability”.   I suppose it could also have some relevance to numbers 1 & 2, but relying on a PenTest to be the significant useful prior for those is folly).  Essentially, the FAIR risk analyst makes calibrated estimates for those factors in 3 & 4, placing those estimates within the context of a population distribution.   PenTesting can be the critical way of gathering evidence that reinforces or refutes those hypotheses.

I can hear some of you saying “OK, Alex, but the value of that isn’t self evident to me.  Why would I, the risk executive, spend the money on PenTesting in order to test some hypothesis?”  Well, maybe you will and maybe you won’t.  But I will offer the following.

After hanging out with Jack and Chris this weekend, looking over some new software features and their implications to getting knowledge & wisdom, what I was seeing was so cool that I could only express it in the following Twitter:

“Risk management, tactically speaking, ultimately becomes variability management”

To which Chris Hoff responded:

“This ultimately becomes VISIBILITY management”

Which, is extremely insightful and absolutely correct.  It begged my response:

“which ultimately becomes about measurement, whose usefulness depends on your model.”

PenTesting validates measurements, which validates Hoffs visibility, which validates variability management, which causes strong risk management.  PenTesting will remain an important tool, but it has to get “lean”.

What might “Lean” PenTesting be?

If I were your enterprise CSO, I’d want my PenTesting functions to move from one big “hack the enterprise” multi-week engagements to several small tests designed to rapidly give me useful information about specific strategic points.   So instead of spending (picking a number out of my elbow here) $50k for one enterprise PenTest, I’d rather have 7 smaller $7k tests that test significant hypothesis (i.e. resistance points that occur the most frequently in our analyses), or areas of significant uncertainty (points where the confidence in our calibrated estimates are very low).  When it comes to reporting, spare me the executive summaries, fault chains designed in OmniGraffle and  the Nessus-devined dashboarding in your clever report formats - just give me the following four paragraphs:

1. The hypothesis we’re testing and why (information provided by my analysts)
2. How you chose the skills and resources to emulate the TCap I wanted
3. How you did against our controls (Resistance Strength)
4. Whether you think we should accept that hypothesis and any recommendations.

Seriously, it shouldn’t be more than a page or two of report and depending on the number of assets involved, it shouldn’t take too long, either.  When I was working with @lbhuston we used to do these sort of small tests all the time, just not in as structured of a manner.

PenTesting isn’t going anywhere anytime soon.  If anything, I think it will actually get “cooler”.  But it, like most tools, will evolve over time to provide us with useful information as our view of the world matures.

======================

PS:  And if I were your consultant and you told me you weren’t mature enough as an organization to utilize this approach - I’d tell you to put your money back in your pocket and reallocate it towards someone would could help you get to that level of maturity.  PenTesting just to re-inforce the fact that you’re screwed-up is silly.

Posted on December 8