Home > Intel Communities > Open Port IT Community > IT@Intel > Blog > 2008 > August > 25
Up to Blog Posts in IT@Intel
Previous Next

IT@Intel Blog

Currently Being Moderated
Matthew Rosenquist
8

Are Security ROI Figures Meaningless?

Posted by Matthew Rosenquist on Aug 25, 2008 9:56:19 AM

Recently, security expert Bruce Schneier expounded security ROI figures were meaningless! Is it true? Well, yes and no.

 

The brutal truth.

Well respected information security expert Bruce Schneier recently provided a stark opinion regarding the value of ROI's.

Follow this link to see the story:

http://www.zdnetasia.com/news/security/0,39044215,62037905,00.htm

 

 

 

In brief, Bruce stated security because numbers can be manipulated to justify anything.

He explained that the amount spent on a product can change significantly by simply playing with the equation.
"If the chance of you being attacked is one in a million and I change it to one in two million... I have halved the amount of money you should spend.
"I can make an ROI model say whatever I want. I could justify or not justify anything based on these very, very rare and very, very damaging events," he said.

 

Tell me it is not true!

I believe Bruce is both right and is delivering a message which is a little incomplete. His general message is accurate and shocking enough to garner the right level of attention. Most of the information security ROI's I have read were speculative, could not be validated, were impossible to reproduce, and had great latitude to provide results which benefit the desires of the author. Nowadays audiences are being provided ‘information' under the auspices of ‘fact', when in reality they are more of an opinion. Such valuation assessments are based on qualitative data versus quantitative metrics.

 

I blogged about the The Problem of Measuring Information Security back in August 2007

 

Awareness must be raised. I applaud Bruce in helping to make this happen. His message, as brutal as it sounds, is bringing to light a shadowy area in our industry. I think the follow-up message for audiences is to scrutinize and apply common sense to any ROI they come across. Understand the methodology and if it makes sense in their context. Lifting the curtain can quickly reveal a puppet master pulling the strings to artificially show value.

 

Like Bruce, I too have a jaded perspective. I have seen some WILD ROI's. Much of what I have read from security vendors is pure folly. However, just because most are fiction, it does not mean all methodologies are without merit.

 

Intel published a Whitepaper - Measuring the Return on IT Security Investments which is applicable to some situations. This method, far from being a silver-bullet, is a good start and has proven its truthfulness.

 

For any method, the accuracy should be scrutinized. Can it be validated, repeated? Was the method exclusively developed solely for self serving purposes from someone trying to sell something shiny? Does it make sense? These are the questions I ask myself.

 

On the bright side, many bright sharp people are working very hard to make the industry better and develop more rigid processes to insure both accuracy and confidence.

 

In the end, there is much work to be done in the information security valuation space. In the meanwhile, savvy consumers should be aware of the challenges and dive deeper into prospective ROI's and determine if they are ‘meaningless'.

Tags: model, risk, matthew_rosenquist, rosenquist, security, roi, value, rosi, information_security, optimal_security


Add a comment Leave a comment on this blog post.
May 8, 2008 2:57 PM Guest Göran Sandahl  says:

Hi,

 

I read Measuring the Return on IT Security Investments a few months ago, and at the same time Security Metrics by Jaquith. I had some spontaneous issues and comments with both the book and the methodology described in your paper, especially with the use of incident occurences as a metric for progress and success. Essentially, I think your methodology lack a way of measuring the effectiveness of your detection instruments which may cloud the number of incidents you detect. Any comments on that?

 

For more information on where I'm coming from, see: http://gsandahl.net/2008/03/10/identifying-security-progress-and-success-are-incidents-occurences-really-a-good-metric/

 

/Göran
Reply
May 8, 2008 2:57 PM Guest Lord Volton  says:

On a random aside, is Intel doing anything to help bring down the botnets that send billions of SPAM emails every day? It seems like corporate social responsibility teams at Google, Intel, and Microsoft could work together to figure out how to bring an end to botnets.

 

I doubt the government will be able to do it on its own.
Reply
May 8, 2008 7:17 PM Matthew Rosenquist Matthew Rosenquist    says in response to Göran Sandahl:

Göran,

 

You bring up an interesting area of discussion. Measuring occurrences has both benefits and drawbacks. On the upside, actual events are being captured. This constitutes real data which cannot be argued. This avoids the validity discussions which can swirl around the qualitative to quantitative risk models where ‘experts’ estimate the chances of attacks. You mentioned one problem, the detection system may not capture all events, therefore not reflective of the real situation. Additionally, historical events may not be indicative of future occurrences, especially new types of threats. In this manner, the key is accuracy over time. Using the method outlined in the Intel case study, we were able to derive future predictions for incidences. We then kept track of how accurate those forecasts were. Truth is in the results. After a year, our predictions maintained an 87% accuracy.

 

It is important to understand the objective of the methodology. It is not intended to deliver absolute precision, rather it is designed to provide sufficient accuracy to make good business decisions. Given this, the real question which must be answered is, does the accuracy meet the need? In our case it did. Additionally, I am not aware of any other information security value methodology which calculates its own accuracy. This is significant, as it allows users to determine for themselves if they should believe in the model and if it is applicable to their environment. Blind faith is not required.

 

Lastly, you had a concern about the dependence on the quality of the detection capability to maintain visibility to occurrences. This could be a limiting factor. I too had initial concerns when developing the model, and then I came to an important realization. Reality is brutal. Significant incidents which detection tools may miss, become evident due to their impact. Painful events are typically realized by more than just detection systems. End users, operators, or customers will raise a fury. Those small incidents which are non-consequential and are missed, really don’t matter in the greater scheme of things. As long as the tools are consistent over time, then the numerical ratios derived by the ROSI method will remain valid within accepted tolerances. So in the end, brutal reality is a friend of those trying to derive value.

 

Side note: I tried to view the link you posted and was unable to connect (Connection timed out). Please drop me a note when it is available. Thanks for the comments and great questions!
Reply
May 12, 2008 9:35 AM Guest Göran Sandahl  says in response to Matthew Rosenquist:

Matthew,

 

Thanks for your answers. Some comments below.

 

"Reality is brutal. Significant incidents which detection tools may miss, become evident due to their impact. Painful events are typically realized by more than just detection systems. End users, operators, or customers will raise a fury. "

 

Not sure if information disclosure or corporate espionage would naturally be detectable due to their impact. To the oposit, I think it's likely that such a threat will do everything in its power to evade detection so that the entire incident go unnoticed. While corporate espionage might perhaps be a drastic example (allthough very valid for a company like Intel, I guess?) our everyday trojan, bot and rootkit will have that as one of their primary goals too.

 

"Those small incidents which are non-consequential and are missed, really don’t matter in the greater scheme of things. As long as the tools are consistent over time, then the numerical ratios derived by the ROSI method will remain valid within accepted tolerances."

 

I agree that if the tools you use are consistent with the last time you looked, then you'll at least have a valid comparison. The problem, as I see it, is how do you know that they are? For any security metric to be valid, I think we must put some effort into also defining what constitutes a good detection instrument, and possibly have that as a metric (-very- simplified: with a detection ability of X, we have detected Y incidents). Otherwise we will never be able to compare numbers, since the one with the best security will have zero incidents since he has good defences, and the one with the worst level of security will have zero incidents since he is unable to spot them. Equally happy, but not equally secure.

 

Not trying to bash the methodlogy. I just think we need to put some more effort into defining the inputs before we can begin measuring output. I'm trying to come up with something regarding a detection metric, but it's coming slow

 

P.s. The link in my last post should be live now. Had some system stability problems.

 

Cheers,

Göran Sandahl
Reply
May 12, 2008 12:57 PM Guest Malcolm Harkins  says in response to Lord Volton:

Lord Volton,

Intel works broadley with various industry groups as well as governments on items related information security as well as critical infrastructure protection. We are also continuing to improve the technology to improve the overall security of the platforms we deliver.

 

I dont know though that there will ever be an end to botnets. In their current form they may not exist at some point down the road but as long as it is possible to compromise a system then it will be possible for bots to exist. As you know many systems get compromised and become bots because a user gets socially engineered and clicks on a link to something that compromises their systems.

 

If we can make users more cautious of their behavior on the web and we continue to improve the technology to prevent/mitigate compromises i think we will be able to reduce the amount of botnets and their impact. But i dont think they will be eliminated
Reply
May 14, 2008 1:22 PM Matthew Rosenquist Matthew Rosenquist    says in response to Malcolm Harkins:

Mike Rothman had an interesting perspective on this topic in his blog: http://securityincite.com/TDI-2008-05-13#TBP1

 

Mike contends credibility is key to the discussion. "If you aren't credible, then it doesn't matter what numbers you generate - no one will believe you."

 

Excellent point Mike. Check out his blog for more.

 

I completely agree. Due to a lack of credible and applicable methodologies, much of it comes down to the reputation of the author. Practitioners of security value assessments must prove their expertise, show how they calculate values, and maintain such vigilance to protect their credibility currency.
Reply
Aug 26, 2008 9:15 AM Guest busby seo challenge  says in response to Matthew Rosenquist:

I think it's likely that such a threat will do everything in its power to evade detection so that the entire incident go unnoticed. While corporate espionage might perhaps be a drastic example (allthough very valid for a company like Intel, I guess?) our everyday trojan, bot and rootkit will have that as one of their primary goals too. http://pinayspeak.com
Reply
Oct 14, 2008 12:34 PM Guest captivating capiz  says in response to busby seo challenge:

I dont know though that there will ever be an end to botnets. In their current form they may not exist at some point down the road but as long as it is possible to compromise a system then it will be possible for bots to exist. As you know many systems get compromised and become bots because a user gets socially engineered and clicks on a link to something that compromises their systems.
Reply

Actions

Popular Blog Posts