Choosing the right method to measure security value is important but not necessarily intuitive.

Some years ago, at the prodding of our department training expert, I developed a class teaching how to think critically while calculating information security value.  The benefits of the course are twofold.  The class helps security practitioners in creating more justifiable value assessments for their programs.  Additionally, it assists audiences of such assessments to question the validity and identify weak justifications.

I offered to teach the class once a year, internally to Intel, and figured the audience would dry up after the first class.  For some odd reason people continued to sign up year after year.  I honestly figured not many people would willingly choose to spend their time on such a dry subject.  In the first year, mostly information security professionals attended.  In subsequent years, to my surprise, a slew of people from finance, manufacturing, marketing, and product development have taken the course.  Sitting in my Inbox is my annual notification for instructing the class, with a list of students from multiple countries already signed up.  Curse you Bruce (training expert)!

With such a diverse audience, I figured I would share some of the materials with the broader community.  This is just a snippet, but one of the key chapters.  Feel free to comment (all comments will be forwarded to Bruce)

This section of the class touches on recommended methods to show value.  This is not an all encompassing list, but probably the most common to information security programs.  These are archetypes of measurement techniques, not specific questions or audits.  Most techniques in use today can be classified into one of these archetypes.  Each has a set of common characteristics with strengths, weaknesses, and applicability considerations.  Knowing these characteristics is to understand how best to validate or challenge the metric.

Information Security Metrics Archetypes

#1 Metric Type: Standards-Based Gap Analysis

Method: Compare the current state against a provided list
Measurement Scale: Nominal
Pro’s: Shows gaps against defined standards.  Can be very fast to accomplish, compared to other methods
Con’s: Does not show actual value, only alignment to a defined state
Applicability: Compliance to regulations, alignment to best-known-methods
Output: Scorecard to expected compliance, gap list of non-compliant areas
Notes: The value of compliance to a predefined standard resides in the applicability and comprehensiveness of the standard itself.  Typically, it is also specific to a particular area of risk.  Interpretation also can skew measures, if the standard is vague.

#2 Metric Type: Raw Gap Analysis

Method: Brainstorm from knowledgeable persons on what they think needs fixing
Measurement Scale: Nominal
Pro’s: Identifies the most apparent issues to correct.  May be as simple or complex as the organizer desires.
Con’s: Reliant on expertise of teams doing the analysis.  Not tied to any quantifiable savings.
Applicability: Response to incidents which already occurred, to prevent recurrence
Output: List of issues to correct
Notes: The value resides in the knowledge of the people conducting the analysis.  A mix of technologists as well as security is best, otherwise the output may lack real benefits

#3 Metric Type: Project Progress Tracking

Method: Metrics which track the start-to-finish progress of a security project
Measurement Scale: Interval
Pro’s: Shows advancement and progress of a project
Con’s: Does not tie the project to any savings or benefits
Applicability: Project management effectiveness
Output: Performance against schedule/budget metrics
Notes: This class of metric is often misused.  Progress of project completion is largely independent of what value it provides once instituted.  This can be used when a security project is a critical path item to another initiative where value is defined.

#4 Metric Type: Qualitative Risk Assessment

Method: Organized collection of concerns from knowledgeable persons on what they believe needs fixing and an explanation statement of the severity of the problems
Measurement Scale: Ordinal
Pro’s: Generates a list of areas to address with prioritized descriptions
Con’s: Reliant on the expertise of teams doing the analysis. Not tied to any quantifiable savings.  Can be time consuming.  May not be comprehensive.  May be skewed to only areas evaluated.  Personalities of the team may significantly alter the priority descriptions of items.
Applicability: Basic state of security gap analysis, scalable to an entire organization.
Output: Description of prioritized line-item gaps
Notes: This is one step above the Raw Gap Analysis method.  Best use is to identify and describe the priority of the most severe issues.  Rarely is this method comprehensive.

#5 Metric Type: Qualitative to Quantitative Risk Assessment

Method: Formal severity ranking, typically on a scale, of problems gathered from a Qualitative Risk exercise
Measurement Scale: Ordinal to Interval
Pro’s: Generates a prioritized list of areas to address, with relative values for comparison.  Can track over time to show incremental changes.
Con’s: Reliant on expertise of teams doing the analysis.  Relative values are not tied to any quantifiable savings.  Time consuming, requires tools for scalability.  Expect +/- 40% accuracy
Applicability: Advanced state of security gap analysis, scalable to an entire organization.
Output: Ranked descriptions of line-item gaps
Notes: This is one step more advanced from the Qualitative Risk assessment, giving numerical values to priority aspects (example: threat, vulnerability, consequences, etc.)

#6 Metric Type: Vulnerability Analysis

Method: Thorough inspection which documents all vulnerabilities
Measurement Scale: Interval
Pro’s: Identifies a list of vulnerabilities which exist
Con’s: Existence of vulnerabilities is not tied to losses.  Output can be overwhelming and underscores only a snap-shot in time of a rapidly changing environment.  Can be very time consuming, requires tools and interpretation.
Applicability: Applied to specific hardening initiatives or fed into a risk assessment
Output: Descriptions of potential vulnerabilities, may be ranked on severity or overall exposure
Notes: Vulnerability analysis poorly correlates to losses.  Just because a vulnerability exists, does not mean it will be exploited.  If exploited, it does not necessarily equate to a meaningful loss.  Question any vulnerability analysis, which claims specific dollar savings!

#7 Metric Type: Against Previous Performance/Operational Efficiency

Method: Statistical comparison against historical data, known costs, and trends (example: actuary tables)
Measurement Scale: Interval to Ratio
Pro’s: Uses actual data to derive the measurement.  Can show the value of a program.  Can be used to both predict value as well as derive sustaining value after project landing.
Con’s: Accuracy may suffer as historical patterns change.  Significant work to accomplish this metric.  Accuracy may be outdated quickly as the environment changes quickly.
Applicability: Before and after comparison of effects for value measurements.
Output: Historical performance and trend graphs showing relative positions.  Net Present Value (NPV) for operational spending.  Forecasts of high-level changes to risk.  Can provide a ‘value’ in terms of dollars.
Notes: Depending upon the historical data, it may not tie to actual security value.  Data trends in the security field tend to be incomplete, limited, and can be manipulated.  Operations costs may not reflect the benefit of security.  Best when used to compare data prior and after landing a security program.

#8 Metric Type: Value Calculation for a Return on Security Investment

Method: Financial model quantifying the dollar benefits of a security program
Measurement Scale: Interval to Ratio
Pro’s: Uses actual data to derive the measurement, based upon trends and control groups.  Potential to generate dollar values derived for both losses and loss prevented.  May comprehend defense-in-depth solutions, showing the individual as well as cumulative value.  Statistical predictions quantify accuracy
Con’s: Extremely difficult to produce.  Must have significant amounts of accurate data and understanding of the security environment.  Must use complex calculations and factor in unknowns.  Very difficult to scale.  Tools and processes are not well defined or mature in the industry.
Applicability: When sufficient historical data is available, an intuitive understanding of the security environment is present, and business values can be measured.  For use when justifiable estimates of dollar value of a security program is needed.
Output: Incident reduction metrics, estimated losses, and loss prevented metrics.  Single Loss Expectancy (SLE), incident and loss predictions.  Derived dollar value of individual security projects as well the value for multiple overlapping/complementary security systems.
Notes: Not for the faint of heart.  These types of analysis are ugly monsters to produce and validate.  All assumptions, calculations, and data sources must be documented.  Complete raw data sets must be provided.   May include limited aspects of other measurement archetypes to fill in gaps, thereby affecting accuracy.

Lastly, there is another choice which can be made: the decision to not measure the value of a security program.  I think this option is pursued more often than not and done for the entirely wrong reasons.  Measuring value is not easy.  It consumes time, resources, requires expertise, and once it is published the author may be under the spotlight to answer and justify the analysis for years to come.  But for all the sweat, tears, and pain, having a good understanding of the value, has merit for security programs of significant investment.

On the other hand, the simple reality is that in many cases a full blown analysis does not make sense.  For example, when a program is required to meet regulatory requirements or when the security investment is very small.  I would not do a comprehensive value assessment for justification to purchase a $10 cable lock.  Let common sense prevail.  If the value must be understood to compare to other options, articulate security posture, or justify spending, then do an assessment.  Otherwise, ask yourself if it is really needed.

After spending the last 6 months researching emerging technologies around the IT Client platform, I have identified two must have technologies when considering your client refresh.  The first is Solid State Hard disks.  While the cost is a concern at initial glance, the benefit you receive from this technology is incredible.  We have seen benefits such as no more hard drive failures do to failures from moving parts.  Increased performance from faster machine startup and resume times.  Increased application responsiveness from quicker access on a SSD versus traditional platform.  Fragmented hard drives become an issue of the past and you can now save costs on 3rd party defrag tools and/or custom solutions you develop in house.  These are just some of the many benefits we have seen, for more in depth review check out our recently released whitepaper - http://communities.intel.com/docs/DOC-2524. But beyond all of these benefits are the ones you may need in the future.  As IT moves to more and more of a Virtualized Client environment, technologies like these help make adoption much easier.  When testing the Solid State Disks, we noticed that our Virtualized IT environment running in a traditional Type-2 Client Hypervisor actually ran 27% faster than the same virtual environment on a traditional platter based drive.  This brings me to the next technology, VT-d.  This is the next evolution in client support of virtualization.  While todays more common systems have VT-x, VT-d is now available on many newer systems today.  VT-s offers what we refer to as "direct pass through" interface for virtual machines to communicate with the system hardware.  What this means for you is that you can have a virtualized OS that can talk directly to certain parts of your systems hardware without having to go through a virtualization layer in a Host OS.  This will also enable better use of Type-1 Hypervisors or "Native Client" hypervisors that will allow side by side, on at the same time OS operation on a single platform.  Imagine being able to support a corporate and personal build on the same machine but keeping them isolated from each other.  This opens the door to a host of possibilities for future IT shops.  Not all of these technologies are ready to run full speed today, but with most shops carrying a 2-3 year refresh cycle, it is important to buy the right technologies at the right time so when you want to deploy these, you have systems that support them.  So make sure you check these two technologies out and get them into your client roadmap as soon as possible.

Most security programs show value by either reducing the number of incident occurrences or reducing the impact of those incidents.Understanding where a program draws its value and how it fits in a defense-in-depth strategy, gives insights to where it can be maximized for optimal benefit and raise flags when expectations are mismatched with function.

Occurrence and Impact

Simply put, security incidents happen and they cause discomfort. Effective programs will either affect the number of times they occur and/or will lessen the negative impact. These aspects of Occurrence and Impact are important when we look at the complexities of measuring security value in the real world. It is a basic first step, but this type of framing establishes boundaries and clarifies expectations.

Once understood, it may be possible to measure the effectiveness to a level which determines general value and applicability. It can paint an important piece of the picture showing how the collection of security programs provides coverage to the landscape of attacks. Additionally, the big picture can identify inefficient duplications of security services.

Do all security programs manifest value in this way? No. Some efforts are tailored to meet regulatory, ethical, or emotional needs. For those types of initiatives, this general framework has limited applicability to measure value.

Intersection of Defense-in-Depth

The diagram below is an overlay of the Occurrence/Impact domains with the Defense-in-Depth categories as they intersect a typical attack lifecycle.Mapping security capabilities, tools, and services will show coverage and gaps for different types of attacks.

Defense in Depth Information Security Strategy

Information Security Defense In Depth Whitepaper is Now Available

Everyone wants information security to be easy. Wouldn't it be nice if it were simple enough to fit snugly inside a fortune cookie? Well, although I don't try to promote such foolish nonsense, I do on occasion pass on readily digestible nuggets to reinforce security principles and get people thinking how security applies to their environment.

Common Sense

I think the key to fortune cookie advice is ‘common sense' in the context of security. It must be simple, succinct, and make sense to everyone, while conveying important security aspects.

Here is my Fortune Cookie advice for September:

In information security, like in sports, knowing your adversary is far more important than knowing the condition of the field.

Information security is an adversarial pursuit. It all begins with threat agents, those people who will negatively affect your organization. Some are malicious, others are not. The key is they are living, breathing opponents whose motivations drive actions which cause loss. They learn, adapt, and change as they seek their objectives.

Know your threats. This is an important first step. Knowing all your vulnerabilities is fine, but secondary in importance.

For those who are malicious, understand what they target and the likely methods they will employ. Only then can the vulnerabilities be narrowed to show the most probable exposures. This prediction gives the security professional a focus on what to protect, how best to monitor, and preparations necessary to respond when needed.

So am I contributing to the problem of over simplifying security? Or am I reaching out to those who might not take an inordinate amount of time necessary to understand the complexities and nuances of our industry? You decide and feel free to share your knowledge-nuggets.

Fortune Cookie Security Advice - August 2008

Fortune Cookie Security Advice - June 2008

Fortune Cookie Security Advice - May 2008

Deconstructing Cyber Security Attacks - Threat Model

Defense in Depth Information Security Strategy

Recently, security expert Bruce Schneier expounded security ROI figures were meaningless! Is it true? Well, yes and no.

The brutal truth.

Well respected information security expert Bruce Schneier recently provided a stark opinion regarding the value of ROI's.

Follow this link to see the story:

http://www.zdnetasia.com/news/security/0,39044215,62037905,00.htm

In brief, Bruce stated security because numbers can be manipulated to justify anything.

He explained that the amount spent on a product can change significantly by simply playing with the equation.
"If the chance of you being attacked is one in a million and I change it to one in two million... I have halved the amount of money you should spend.
"I can make an ROI model say whatever I want. I could justify or not justify anything based on these very, very rare and very, very damaging events," he said.

Tell me it is not true!

I believe Bruce is both right and is delivering a message which is a little incomplete. His general message is accurate and shocking enough to garner the right level of attention. Most of the information security ROI's I have read were speculative, could not be validated, were impossible to reproduce, and had great latitude to provide results which benefit the desires of the author. Nowadays audiences are being provided ‘information' under the auspices of ‘fact', when in reality they are more of an opinion. Such valuation assessments are based on qualitative data versus quantitative metrics.

I blogged about the The Problem of Measuring Information Security back in August 2007

Awareness must be raised. I applaud Bruce in helping to make this happen. His message, as brutal as it sounds, is bringing to light a shadowy area in our industry. I think the follow-up message for audiences is to scrutinize and apply common sense to any ROI they come across. Understand the methodology and if it makes sense in their context. Lifting the curtain can quickly reveal a puppet master pulling the strings to artificially show value.

Like Bruce, I too have a jaded perspective. I have seen some WILD ROI's. Much of what I have read from security vendors is pure folly. However, just because most are fiction, it does not mean all methodologies are without merit.

Intel published a Whitepaper - Measuring the Return on IT Security Investments which is applicable to some situations. This method, far from being a silver-bullet, is a good start and has proven its truthfulness.

For any method, the accuracy should be scrutinized. Can it be validated, repeated? Was the method exclusively developed solely for self serving purposes from someone trying to sell something shiny? Does it make sense? These are the questions I ask myself.

On the bright side, many bright sharp people are working very hard to make the industry better and develop more rigid processes to insure both accuracy and confidence.

In the end, there is much work to be done in the information security valuation space. In the meanwhile, savvy consumers should be aware of the challenges and dive deeper into prospective ROI's and determine if they are ‘meaningless'.

Measuring the value of information security programs is difficult and a problem for the entire industry. In the second of the three part series, Intel discusses a practical approach to determine value of information security initiatives. Intel security professionals Tim Casey, Enrique Herrera, and Matthew Rosenquist discussed the success of Intel’s security value methodology outlined in the Whitepaper - Measuring the Return on IT Security Investments

Listen to how Intel utilizes this strategy as one means to measure the value of security programs. The whitepaper is available for download.

The 30 minute discussion can be replayed here:

The last of the three part series, Future State of Security Measurement, will occur on Wednsday June 4th. Everyone is welcome to participate or just listen in. Details can be found here:

http://communities.intel.com/openport/blogs/it/2008/05/12/how-do-you-measure-something-that-doesnt-happen

Good security conversations benefit all involved. The more we share, discuss, and challenge each other, the more we advance our industry. Thankfully, I have the benefit of working closely with a brigade of information security professionals and we banter at every opportunity, for the sheer pleasure and insights. In that same spirit, we hosted our first Blog-Talk radio session. This was a general discussion of the problems of measuring security.

The 30 minute discussion can be replayed here

Two other internet chats are planned. Everyone is welcome to participate or just listen in. Details can be found here.