Bad Studies, Bad!
What? Folks exaggerating statistics for FUD purposes? NO! I’m kind of surprised Chris Walsh missed that one.
Speaking of Chris, I owe him an email about a study I don’t particularly like, one called The Emperor’s new Security Clothes. There is a nice .pdf of it here. In lieu of an email, Chris, here’s a blog post.
Now before we talk about TENSC let me state that I think the world of the Usable Security movement. Consumers need better indicators about what they are doing online, much better indicators than IE, Mozilla, Apple or HackerSafe are currently giving them. I applaud folks like Dan Houser who are working very hard to create some semblance of language that all our non-technical stakeholders can use to interpret their current network activity within context of their risk tolerance.
But this hack of a study is not the way to go about advancing the movement. I have no qualms with the usability aspects of TENSC study. But that was only one of two purposes in the study.
Researchers set up an already compromised LAN environment where they could perform man in the middle attacks. The kind that would rarely occur between a home, office, or even public WiFi connection and Bank of America online site. That’s fine, but within context of this:
"First, we wanted to evaluate how effectively website-authentication indicators protect users from fraudulent sites."
it makes my mind boggle.
To put it nicely, this report focused solely on effectiveness of site indicators as a detective control for end users, not a preventative control to reduce the number and effectiveness of Phishing attacks against consumers. I would argue that the benefit of these controls is really the latter. And the effectiveness of these controls in that regard aren’t even close to being studied here.
Meaning that the stated goal of the work is not adequately addressed. But that certianly wasn’t what the media picked up. From Slashdot to the New York Times, this report was trumpeted as proving that the controls in place by Bank of America are worthless. They are far from worthless. Especially within context of Defense in Depth – if BoA combines the above controls with, say the Cyota stuff that RSA bought (honestly, I have no idea what EMC is calling it now, sorry), then effectiveness of the control, is increased significantly (and of course, our synthesis from FAIR relationships can mathematically model approx. how much increased effectiveness is afforded the end user). If the researchers really wanted to measure the effectiveness of the control in protecting end users, then they should have studied the control within context.
Another thing : if you’re going to use a real bank, it would be better decorum IMHO to sanitize that bank’s name out of the study. Especially if you’re going to turn your little study into a big media event. What they did there really isn’t fair to our friends and comrades at BoA.
This study was either sensationalist, reckless, or just plain not thought out very well.
Vin McLellan Feb 28
Thank you, Alex!
It was a relief to come across someone who actually read this silly little study, as opposed to simply echoing the Chicken Little “Sky is Falling” analysis that these Ivy Tower researchers have ladled out to the media.
I agree that security usability research is critical, but there seems to be a severe shortage of thoughtful professionals who can skeptically review at this sort of multi-modal research, and are then willing to publicly comment on its credibility.
Vendors often seem hesitant to challenge the credibility of “research” which claims the patina of academic objectivity. Some, like RSA — for which I have been a consultant for many years — may have close ties with the institutions the researchers are affiliated with. Others are fearful of being seen as a Goliath stepping on (or worse, censoring) outspoken little critics. Still others just run from media side-shows.
As a case study, I think this “Harvard/MIT research” has been a salutary example of how outrageous and irresponsible claims, based on severely flawed research, can feed into the media’s appetite for sensationalism.
(The lesson to be learned by industry: cheap shots need a firm and vigorous response. Modest self-restraint can just feed into a media frenzy.)
This social-science research project originally collected 60-odd college students for a study of how gullible on-line consumers are. By design, the researchers willfully subverted the first layer of bank’s multi-layer security architecture — customer authentication — when they used their academic credibility to convince the study participants to freely give up their account data and personal banking passwords. Social engineering at its best. Feedback from two-thirds of the original group was then made irrelevant when these kids were instructed to play a role in a video game scenario: each was given a phony bank account, a new password, and told that they were to act like they were a rich doctor shuffling money between his accounts on a Sunday afternoon.
With the role-playing group compartmentalized, this “research” ultimately boils down to a review of the behavior of just 20 college kids, those who didn’t walk out (or purposely mess up the study protocol) after they were instructed to access their own bank accounts on rigged machines, with their behavior (and private banking data) to be recorded by total strangers.
The researchers sat each member of this gullible group of innocents down in a Harvard classroom and asked them to run through faux bank web pages.
Is this a realistic or representative model for adults using their online banking accounts?
I think not. In fact, I believe the design of the “research project” was so other-worldly, so unrealistic, that it was difficult for the institutional players to take it seriously until the researchers’ overwrought PR campaign was well underway.
The financial institutions which are implementing Site-to-User images for host validation — almost always as one layer in a multi-layer security system that includes stronger-than-password authentication, as well as real-time risk-based analysis of the users’ transactions — have generally done their own research. Typically, as you surmised, they conclude that their customers, in the vast majority, use (or at least claim to use) and value their Personal Security Image. In the midst of a well-documented crisis of confidence among online banking customers, it seems to be a source of particular irritation for these researchers that online consumers usually find their bank’s implementation of PSIs “reassuring” — even when these consumers don’t know about, or understand, the layered security architecture the bank is actually relying upon.
Thank you again for your informed commentary.
Regards,
_Vin