Overloaded Security Metrics
May 20, 2009
I find it interesting when two seemingly disparate disciplines arrive at similar conclusions independently. I got this feeling while reading the following two unrelated articles that appeared on the same page in The Wall Street Journal: Derivatives and the Wisdom of Crowds, and The State of Surveillance. Especially since they both touched on a topic near and dear to my heart: measuring effectiveness by quantifying outcomes.
The first article highlights the banking industry, pointing out that a great deal of granular information was available prior to the recent credit market correction that regulators and others were apparently unable to assimilate: “Indeed, the credit bubble is a reminder that above all regulators need to know what to do with information once it’s disclosed. Bank regulators had access to the information to know who was at risk for how much, which should have alerted them to systemic risk among brokers, AIG and others. Yet regulators weren’t able to predict the implosion any better than the banks.” [my emphasis]
The second article treats a quite different topic, that of government surveillance. And yet, it highlights some of the same conclusions: “…’electronic surveillance is not always augmenting traditional policing; it is more often than not replacing it, with poor results.’ Likewise, huge collections of information gleaned from private sources such as phone companies, banks and credit bureaus (along the lines of America’s renamed but not abandoned Total Information Awareness Program) are apt to be unmanageable and rife with errors. Mr. Clark notes: ‘There is a fundamental rule about databases: the bigger they are, the more useless they become.’” [my emphasis] That last statement seems particularly devastating to the notion that we can glean immaculate knowledge simply through accumulating large volumes of data (and maybe even more devastating to some big database vendors’ business strategies, but I digress).
The second article continues: “Again and again, Mr. Clark finds, high-tech systems that seem at first to be outrageous invasions of privacy turn out to be outrageous boondoggles that don’t succeed at their official goals and actually get in the way of catching the bad guys and protecting the public. ‘The excessive collection of data tends to act as a fog through which authorities struggle to find what they are looking for,’ Mr. Clark writes. ‘The more Big Brother watches, the less he seems to see.’” [my emphasis]
Do these disparate but cogent critiques of data for data’s sake refute the notion that quantitative analysis provides the best justification for security initiatives? Restated more bluntly: is quantification even practical? Can security risk be boiled down to useful numbers?
I’m not ready to throw the baby out with the bathwater yet. The above articles helpfully point out that the push towards quantitative analysis is just as vulnerable to information overload as any other discipline. And therein perhaps lies a lesson. Often we come across organizations building security metrics initiatives involving dozens of measurements undergoing complex manipulations. Would such efforts be more successful if they started from a more simple set of practical measurements?
A recent experience perhaps illuminates. We were moderating a discussion of application security metrics with a group of corporate security leaders from across the Eastern US. The discussion was wide-ranging and covered a number of candidate measurements that could be helpful to drive higher security quality into code. The audience seemed overwhelmed at points, unable to see clearly which metrics might be most applicable to their operations, when a sudden insight was made. Two issues continued to be raised throughout the discussion: age of code, and code re-use. These metrics seemed to be predictors of potential security risk in many scenarios (older code has legacy bugs, newer code has unidentified bugs, heavily re-used routines present greater attack surface, and so on). Furthermore, they were not exotic measurements requiring advanced tools or skillsets to produce consistently. Could one build an effective application security program around tracking solely these two metrics?
Some recent research has opened up our thinking on this topic even further (thanks to Gene Kim of Tripwire for introducing us to this data) . The IT Process Institute (ITPI) has produced a number of research reports and benchmarking studies that illustrate strikingly the “bang for the buck” delivered by excelling at just a handful of IT security practices and controls versus being moderately good (or worse) at many. For example, ITPI’s executive snapshot “Process maturity matters: The key to unlocking the power of IT controls” (July 2007) shows how just 12 of 53 IT controls predict 60% of the performance variation in the 330 companies studied. If you could cut your controls investment by ~75% and still retain 60% of the value, you’d probably think about it, right?
How many other “generally accepted security practices” would benefit from scrutiny like this? Is it time for us to reconsider investing “broad and thin” versus “deep and effective”? Are you picking your battles (and your data) wisely, or are you deluged by information security overload?
Comments
Got something to say?