If you’re really interested in why (good) quantitative analysis works:

Beautiful Probability from the overcoming bias blog starts to explain it.  It makes me want to replicate the whole thing here.  But do allow me to point out some subsets of the discussion:

ON THE DIFFERENCE IN APPROACHES BETWEEN WHAT ANALYSTS WITH BASIC STATISTICAL BACKGROUNDS THINK, AND WHAT WE’RE SUGGESTING 

From Beautiful Probability:

And yet… should rationality be math?  It is by no means a foregone conclusion that probability should be pretty.  The real world is messy - so shouldn’t you need messy reasoning to handle it?  Maybe the non-Bayesian statisticians, with their vast collection of ad-hoc methods and ad-hoc justifications, are strictly more competent because they have a strictly larger toolbox.  It’s nice when problems are clean, but they usually aren’t, and you have to live with that.

After all, it’s a well-known fact that you can’t use Bayesian methods on many problems because the Bayesian calculation is computationally intractable.  So why not let many flowers bloom?  Why not have more than one tool in your toolbox?

That’s the fundamental difference in mindset.  Old School statisticians thought in terms of tools, tricks to throw at particular problems.  Bayesians - at least this Bayesian, though I don’t think I’m speaking only for myself - we think in terms of laws.

Looking for laws isn’t the same as looking for especially neat and pretty tools.  The second law of thermodynamics isn’t an especially neat and pretty refrigerator.

SO WHAT?

As Mike R. (presumably borrowed from Guy Kawasaki’s little man) asks:  “So What”?

You’re going to find those who cling to multiple tools and tool boxes to derive risk.  The tools, as Marcus Ranum says here, are used “to manipulate the perception of those managers by tweaking the inputs to give the desired outputs that will “help management get it.”

Once you focus on law and not tools - then the “bull—-ery” as mjr so eloquently puts it, must stop.

ON ‘SUBJECTIVITY’

No, you can’t always do the exact Bayesian calculation for a problem.  Sometimes you must seek an approximation; often, indeed.  This doesn’t mean that probability theory has ceased to apply, any more than your inability to calculate the aerodynamics of a 747 on an atom-by-atom basis implies that the 747 is not made out of atoms.  Whatever approximation you use, it works to the extent that it approximates the ideal Bayesian calculation - and fails to the extent that it departs.

There is uncertainty in every measurement.  There is subjectivity in all observational data.  Deal with it.  Account for it.  Love it.

Sometimes you can’t use Bayesian methods literally; often, indeed.  But when you can use the exact Bayesian calculation that uses every scrap of available knowledge, you are done.  You will never find a statistical method that yields a better answer.  You may find a cheap approximation that works excellently nearly all the time, and it will be cheaper, but it will not be more accurate.  Not unless the other method uses knowledge, perhaps in the form of disguised prior information, that you are not allowing into the Bayesian calculation; and then when you feed the prior information into the Bayesian calculation, the Bayesian calculation will again be equal or superior.

Love it, because it is useful to you now.  Re-read the above. Note that this is what we should be doing - not working with faulty laws/models like risk=controlsXvulnerability/impact, or threatening to turn some project “yellow” because we are the subject matter expert (which is essentially saying that we are above the law set out in a Bayesian network for risk).

Think laws, not tools.  Needing to calculate approximations to a law doesn’t change the law.  Planes are still atoms, they aren’t governed by special exceptions in Nature for aerodynamic calculations.  The approximation exists in the map, not in the territory.  You can know the second law of thermodynamics, and yet apply yourself as an engineer to build an imperfect car engine.  The second law does not cease to be applicable; your knowledge of that law, and of Carnot cycles, helps you get as close to the ideal efficiency as you can.

We aren’t enchanted by Bayesian methods merely because they’re beautiful.  The beauty is a side effect.  Bayesian theorems are elegant, coherent, optimal, and provably unique because they are laws.

Well, I’m enchanted with FAIR because it is beautiful. and rational. and applicable.

Posted on January 16