Dec 13, 2007 12:18 PM Reply Matthew Rosenquist

Pete Lindstrom commented on the whitepaper in his blog “ROSI: It's a bird! It's a plane! It's a real life ROSI calculation!” http://spiresecurity.typepad.com/spire_security_viewpoint/2007/12/rosi-its-a-bird.html (which is always a riot to read) and asked some poignant follow up questions. In addition, members of the ‘securitymetrics’ mail-list have directed a few questions my way. Below is a consolidation of questions and my responses.

Question 1: It would be great to know what the specific categories of consequences were used

• The value assessment took into account the environment and specifically the most important factors contributing to loss. In this case, the company is heavy into manufacturing and R&D, translating into a heavy concern of Availability and Confidentiality. The specific organization in the whitepaper was limited to manufacturing, therefore the most emphasis was places on Availability, as compared to Confidentiality or Integrity. This environment is highly adverse to any type of disruption to manufacturing operations. Even a slight burp can cause an entire factory to do a shutdown, scrap product in process, and begin a controlled startup. Given the fact we produce high value silicon product and not stuffed squeak toys, the costs of downtime can be astronomical. This analysis focused on security incidents which impacted critical manufacturing operations systems and overall production stability.

Question 2. More information on the programs that contributed to the success

• Unfortunately I cannot elaborate on the security programs, products, or processes. These represent a competitive advantage to our organization. I can state these mitigations worked together as part of our Defense-in-Depth strategy and were focused on reducing the Occurrence of incidents, rather than the Effects of incidents.

Question 3. I wasn’t clear on whether you factored in the cost of the programs or not

• Applying the cost of the programs was secondary in this exercise. The biggest rock was to develop a method to determine the value of the security programs as it directly relates to a reduction of impact. The costs were used in follow-up ROI analysis for individual sub-organizations and with the finance folks. In this case, the costs were far outweighed by the conservative estimates of the benefits. At the point we brought the impact-reduction figures and the costs together, the value was obvious.

Question 4. Can you provide further details on the types of incidents you measure, and your detection approach? I am particularly interested in knowing your approach for detecting targeted information espionage attacks, aimed at your corporate trade secrets

• As we focused on incidents which caused, or put at risk, factory operations, it was straightforward where the data could be obtained. The operations folks, responsible for uptime, possessed the data in various places.
• As for detecting targeted information espionage attacks, you are seeking the holy grail of security metrics. We did not explore any innovative methods to identify such stealthy incidents, as it was beyond the scope of impact we were focused on. (..although it is a tantalizing prospect.)

Question 5. I would really be interested in hearing more about the actual types of incidents that you were able to track.

• Not every incident causes loss. We applied some inductive reasoning and noticed a relationship between incidents which occur but cause no loss, those which occur and consume operational time but do not impact factory output, and those which do cause interruption to factory output. So we tracked all types of security incidents and incorporated their relationship as part of the predictions of reduced Occurrence

Question 6. I see where programs can reduce the rate of occurrence for incidents but do you also allow for the possibility that these programs might reduce SLE?

• In this case, we focused solely on the security programs ability to reduce the Occurrence of incidents, and did not evaluate programs which primary focus is to reduce Effects of incidents which do occur. The methodology is applicable only for Occurrence-reduction based programs. In this evaluation, we did not take into account any additional/secondary benefits of SLE reduction for the programs under review.
• The SLE (Single Loss Expectancy) was more a factor of the environment which was effected. Some environments, based upon the data, experienced more loss than others, with the same incident. This is a factor of value for each area as well as the technology/architecture which either facilitates or inhibits the security controls working to contain and recover from the incident. In essence, identical security controls in different areas does not translate to equal losses. The environment itself is a huge factor.
• From a calculation perspective we established an SLE for different areas, based upon historical data. We then applied this figure to the derived ‘avoided incidents’ attributed to each security program, to determine the loss-prevented figures

Dec 31, 2007 1:26 PM Reply Matthew Rosenquist in response to: Matthew Rosenquist

Russell Thomas of http://www.meritology.com has thrown some great questions my way regarding the ROSI whitepaper. I have paraphrased some of his questions.

Question 1. Were the decisions you’d make with this ROSI metric are significantly better or different than you’d make if you had an even simpler/cruder metric such as a security checklist or scorecard

• From a decision perspective, the ROSI had a positive impact in two areas. First, it allowed for a confident conclusion to spend a significant sum of money as a security investment. The programs in question required sizable capital expense and considerable amount of man-hours of the operations folks. Without the ROSI and the vigor of the programs champions, it would have been doubtful such an investment would have taken place. In the words of one manager, “I am not going to blindly invest so heavily…”. With the ROSI, and the validation of their Finance representative giving credibility to the logic and conservative nature of the methodology, a solid decision to invest was reached. Secondly, the all too common after-thrash was avoided. As the audience was not a single decision maker, a normal pattern of doubt and re-validation occurs then re-occurs periodically over time. These professionals run very tight ships and regularly seek opportunities to cut costs and optimize spending. Weeks or months later it is easy to forget why decisions were made and the momentum fades. Without firm justification, security programs appear bloated and become prime targets for cuts. This was avoided. I cannot impress how much pain, effort, and delays were sidestepped.

Question 2. Do the calculations include any discounting for the time value of money and uncertainty associated with cash flow estimates?

• No, in this specific case it was not needed. However, due to the very conservative nature of the analysis, this ROSI can be fed into a NPV template to derive such figures as well as tax implications over time.

Question 3. I don’t see any confidence levels or uncertainty intervals associated with your incident frequency or loss expectancy. Management needs to know how confident to be in the ROSI estimate, and especially when the ROSI method has so much uncertainty that no inferences can be drawn.

• Excellent question! (I was hoping someone was going to ask it. Scooby Snack for you!) Any ROSI analysis is not worth a bag of beans unless it is somewhat accurate and consistently so. It is vital to be able to measure the accuracy! In fact it is a major shortcoming of most security ROI/value methods out there (IMHO). Here is where this model is different. Because we can do predictive analysis we have the ability to measure the models effectiveness, regardless of which path is taken.

Here is how it works over time. Say you are contemplating instituting a security program “Acme Catapult” (very effective against W.Coyote cyber attacks – purely hypothetical for those of a younger generation). You run a predictive analysis against instituting the Acme Catapult as well running the same analysis as if you did not install the Acme Catapult. Now, regardless of which path you choose you have a measure of how many attacks and what losses should occur. Over time this figure can be validated and therefore an accuracy rating derived.

In our case study, the predictions were ~95% accurate for the first 5 months and settled to ~87% accuracy over a year. Frankly, this was much higher than I expected. Almost a curse really, as I fear IT people might expect the normal four-nines (99.99%) mentality to be applied. I am always upfront to say this model is not designed for hyper-accuracy, rather it targets the necessary accuracy to make a good decision. So for the most part I downplay the accuracy.

Additionally, as part of the analysis I established a confidence metric which associated the number of data points over time with the derived averages. The derived incident averages are key to the process and can differ greatly based upon the landing of multiple security solutions. Basically it is an interval scale to compare the confidence between averages calculated with different numbers of instances. I did not need to go math-crazy to realize the more data points, within time periods of non-disruptive technology being introduced, the more accurate the average trend. This helped to both tweak estimations and set expectations with the audience where only small numbers of data points could lead to a wider variance in expected output.

Question 4. After reading your white paper several times, it appears that the scope of your ROSI metric is very narrow and short-term.

• The case study looked at a proposal of landing three complementary security solutions across ~20 worldwide facilities, to protect somewhere in the neighborhood of eighteen thousand systems. We pulled data going back more than 2 years then followed up a year after the dust settled to calculate the accuracy. Value was calculated for the programs individually as well as in combination. The analysis was narrow in that we only looked at the value of the three security initiatives, landing in succession at the different manufacturing sites.

Question 5. I surmise that your ROSI metric is being used to guide program decisions to block malware and unauthorized intrusions, etc. True?

• The ROSI methodology was developed to assign value to security programs which target incident Occurrences rather than incident Effects.

Question 6. Do your “programs” include things such as patch management processes, application lifecycle management (retirement, replacement, new application roll-out, etc.), and architecture decisions, or are they limited to security tools (anti-virus, firewall, etc.)?

• We have successfully embraced a defense-in-depth strategy. It applies itself to the technical as well as behavioral aspects of cyber security. I have a blog on defense-in-depth here - http://communities.intel.com/openport/blogs/it/2007/10/29/defense-in-depth-information-security-strategy or you can listen to Malcolm Harkins, General Manager Intel Risk Security talk about it here - http://blogs.intel.com/it/2007/09/intels_layered_approach_to_inf.php.

Question 7. Is the “Single Loss Expectancy” primarily the expected cost of outage of a particular server or network node. (True? I suggest you consider a loss distribution rather than a single point estimate for losses, so you can include “low probability costs” such as cascading faults (e.g. network outage), decline in manufacturing yields, or even outages for the entire fab, etc. This will make it easier to elicit value/loss estimates from business managers, since they aren’t forced to conflate all this into a single number.)

• Not every event causes loss. Not every outage/impact causes the same amount of loss. We used the historical data to map a simple relationship model:

Events relate to Outages/Impacts, which determines the Loss. SLE is the Losses divided by the Events. Losses in this case were reflecting financial impact due to factory downtime (ie. not providing the service for which it is designed). We looked at loss from the larger ‘corporate’ viewpoint, instead of discrete stand-alone systems. A top-down versus a bottom-up approach. This stratagem worked well given the scope of our value analysis.

Question 8. Your approach does not include low frequency – high impact events that could be material to business unit performance, or even quarterly earnings and/or credit ratings.

• True to a certain extent. As the impact is calculated based upon historical events it is limited. It does not take into account the doomsday fears or integrate in the theoretical potential for rare catastrophic loss. It is justifiable. It does include low frequency events which caused big problems. But those events really happened and nobody is going to argue if they are plausible. This conservative method has strengths and weaknesses, to be sure. It is best for the security practitioner to be aware of them and use as they see fit to align to the environment for the optimal level of accuracy.

Jan 8, 2008 5:00 PM Reply Matthew Rosenquist in response to: Matthew Rosenquist

Lance Spitzner, founder of the Honeynet Project (http://www.honeytech.biz), has some great perspectives and questions of the ROSI whitepaper. See all his comments at his blog: http://lspitzner.blogspot.com/2007/12/rosi.html

Question #1. What happens with this method when your new security program mitigates incidents you never detected in the first place? For example, lets say you counted 400 incidents in your organization last year, but there were really 500. When you implemented your new security measures your incidents drop to zero. Your delta is off by 100 incidents. I'm nit picking here, but your security program actually has far greater ROI. The reason I'm concerned about this is because a good security program mitigates threats/vulnerabilities you did not know about.

• The method is not intended to be hyper-precise. In fact it was designed and wielded to achieve accuracy necessary for better business decisions. In your example, you indicate a number of incidents which were never detected (100 quantity). If we take a pragmatic view, it would probably be safe to assume the collective impact of the un-noticed incidents were not material. If they were material, they should have been noticed at some level. So, given the fact we are trying to evaluate the value of something attempting to affect change, the 100 incidents really have little weight in the big picture.
• Additionally, this method is purposely conservative and defensible. You cannot measure what you don’t know. A ‘good’ program is relative to the impact it has based upon the cost to deliver. How can it be ‘good’ if a program has no measurable affect on the environment? The effect of a program on hidden/immaterial incidents will not be taken into account by the audience as they are unaware of a value impact anyway.

Question #2. However, what I'm even more concerned about is good security includes good detection. Now, what happens if you start with 400 incidents, then implement security controls which includes good detection. Now all of the sudden you are detecting many more incidents you never would have detected before. Even though the total incidents could have gone down, because of your improved detection capabilities management perceives they have gone up.

• I too believe Detection is a critical piece of any defense-in-depth strategy. We have successfully embraced such a posture as it applies itself to the technical as well as behavioral aspects of cyber security. I have a blog on defense-in-depth here - http://communities.intel.com/openport/blogs/it/2007/10/29/defense-in-depth-information-security-strategy or you can listen to Malcolm Harkins, General Manager Intel Risk Security talk about it here - http://blogs.intel.com/it/2007/09/intels_layered_approach_to_inf.php.
• One limitation of this method is, it is only applicable to programs which reduce occurrences of incidents versus programs which reduce effects of those incidents. Detection halfway falls into this category, as detection is worthless if the downstream capabilities cannot use the information to reduce the losses caused by the incidents. Investment in better detection is only worthwhile if you can act on the information provided. So if your investment makes that happen, then the number of successful incidents will reduce or the average losses associated with an incident will be recalculated based upon the better data to show an overall benefit, therefore value will become evident in the ROSI methodology.

Feb 5, 2008 5:20 PM Reply Matthew Rosenquist in response to: Matthew Rosenquist

Kenneth Belva posted a great summary for the whitepaper at www.bloginfosec.com.

"The premise for the paper is simple: the implementation of a security measure (control) should result in a decrease in the number of security incidents for a given environment. Therefore, by quantifying these incidents over time — before and continually after the security control is implemented — we will produce a metric that will demonstrate the effectiveness and return on an information security investment."

On the lighter side, Kenneth listed the reasons he enjoyed the paper, including "...It does not contain marketing fluff" I about fell out of my chair. Yes, there is no marketing, advertising, or sales fluff. Glad to meet that criteria!

Check out his blog for all his comments. http://www.bloginfosec.com/2008/01/24/intel-rosi-paper-sets-practical-guidelines-and-proper-expectations/