Oct 5, 2007 3:07 PM Reply Matthew Rosenquist in response to: Mark Greisiger

Mark
What you propose is truly ambitions! If I understand the insurance industry correctly, what you're really attempting to determine the impact of incidents to calculate the average losses and occurrence rates over a period of time, as well as the primary contributing factors for both. This is huge. I've been mentally chewing on this particular problem for a while and there are several areas which may cause you some problems. Here is a short list to get the conversation started…

Survey bias: It sounds like you are already taking this into consideration, but the respondents to your survey may be a very narrow sample within the field. Your information may be limited only to those who have insurance or those were looking to obtain it. If your survey is open to the general populace then you're gathering data from those sources that have the time, the inclination, or the desire to respond to a survey. This may be far from representative and ultimately reflect in your overall model accuracy.

Orders of magnitude: Given a good cross-sample I would expect wildly varying loss figures and occurrence rates, even for similar incidents. As an example, loss of customer Personally Identifiable Information (PII) could result in little-to-no loss or a $256 million rock, as in the TJX case discussed in this blog: http://communities.intel.com/openport/blogs/it/2007/09/26/landmark-information-data-breach-suit-ramifications. Additionally, occurrence rates as they relate to loss may be disjointed. A company may experience thousands of infections without any noticeable loss while a near identical company may have only one infection which wreaks financial and regulatory havoc.
Computing environments nowadays have to worry about single points of failure. Many organizations, dare I say all organizations, have a number of these critical pressure points which can bring it to its knees. In some ways, it is a game of Russian roulette, never knowing if the next incident will hit a benign or important area. An identical type of threat or attack could have very little impact on one organization and crater the next. The late, and some would say great, Dr. Bill Hancock used to talk of an airline for which he conducted a risk assessment. The single point of failure was found in an area nobody had imagined. He explained the ‘weights and measures’ computer for the airline, which simply calculates weight distribution within a plane prior to take off, was not considered critical and therefore not protected with a high degree of security. Dr. Hancock concluded if this system was compromised it could have a catastrophic effect on planes attempting to take off. As Bill so deftly put it, “after four or five big fiery lawn darts at the end of a runway, people would no longer fly on that airline”, in effect the company would cease to exist as a viable business. The airline quickly corrected the problem but that single point of failure was very vulnerable for a long time without anyone having a sense of the potential danger. Not all single points of failure are this catastrophic, but they do represent pitfalls which can very quickly cause damage and impact. In some cases the size of the loss is so great the organization cannot recover. The loss spectrum for security incidents varies so greatly it will be supremely challenging to temper the figures into a workable model.

Accuracy of losses: Measuring the tangible and intangible aspects of loss is very problematic. As of right now there are no good methods for measuring those intangibles such as an increase in legal liability, customer goodwill, brand recognition, etc. Even measuring the aspects considered tangible such as downtime and operational impact, can be calculated in different ways with greatly varying results. Most of these methods rarely take into account other cascading issues which crop up due to the incident.
It is almost guaranteed if two different groups assess the damage within an organization, they will come up with two greatly different figures. Sticking to rigid accounting practices, a finance department tends to undershoot the impact, while the marketing-sales and operations people have a tendency to overestimate initially.
Lastly, from a timing perspective it can take a long while to even begin to understand the financial impact. Given time, the losses may increase, which is difficult to incorporate in a snapshot-of-the-moment survey.

Completeness and consistency of occurrences: Respondents may not report all the issues and they may not include all costs and losses. This could be on purpose or could simply be the fact that they just don't realize when they've been attacked and understand what losses have actually occurred. In fact some losses may take months or many years to surface, if they are to be understood at all.
At a practical level, take for example virus infections. It is doubtful that any of the respondents would report every time a system within their environment is infected with a virus, however the one infection that does cause significant downtime or business impact would be reported. I think understanding both is important in calculating the overall incident occurrence rates.
Some of the loss types are not at the forefront of people’s minds, yet. This will change as the industry matures and I think we will see organizations begin to estimate losses of customer goodwill, brand reputation, etc. The losses occur now, but nobody knows how to measure them. So your data will inherently change as this begins to take root.

Human unpredictability: At its core, security incidents are spawned from people, the attackers. People are unpredictable especially when it comes to the wild west of Internet and computer security. Your model must take into account the human element as it will drive trends, new types of attacks, and the value of security controls.
On the other side of the battle line is the people who defend the organizations and the potential victims. Changes in the target environment and with the people who operate within, is pivotal to understanding loss trends. Half of security is technical, the other half is behavioral.

My initial recommendation is to look at a few different aspects:
1. Narrow the focus of what you are trying to measure and predict, based upon attacks, loss types, security posture, organization type, timelines, etc.
2. Incorporate Threats into the model, ie. Attackers-Methods-Objectives
3. Comprehend security posture and controls as it relates to occurrence and loss. (vulnerabilities, exposures, and acceptable loss)
4. Help map the consequences of different types attacks
5. Understand the type of organization being protected
6. Frame the windows of time in which trending data remains valid (between disruptive technologies)

Narrow the focus of what you are trying to measure and predict, based upon attacks, loss types, security posture, organization type, timelines, etc.
o I would start small in your development cycle. Pick a specific type loss for a single target class and attempt to put all the pieces together. Figure out the key elements and a way to express them mathematically

Incorporate Threats into the model, ie. Attackers-Methods-Objectives
o Comprehend the complex nature of the evolving threats. Unlike any other area I know, the threat landscape for information security changes most rapidly over the months and years. All attackers have objectives. They will seek to take advantage of methods to achieve those objectives. In most cases, they will follow the path of least resistance. Other paths and vulnerabilities are not as important. However, when the most luring path is blocked, they may quickly shift to the next most attractive method. With all the variations of new technology, vulnerabilities, security capabilities, and end-user behaviors, the threat map changes very rapidly. To some level this must be understood.
o Intel been working on creating a threat agent library as a foundational structure to evaluate threats. A whitepaper was just released. http://communities.intel.com/openport/docs/DOC-1151 It's a good place to start. The next step would be to keep an eye on the emerging threats to see how the attacks are changing and what the threat agents are going after. Overlay this with your customer base to get a better understanding of the differing aspects of the data

Comprehend security posture and controls as it relates to occurrence and loss. (vulnerabilities, exposures, and acceptable loss)
o The other half of the equation is how organizations are protecting themselves. The security mitigation controls in place will either reduce the occurrences or reduce the effects of incidents which do happen. This is of great importance to the primary aspects of what you're trying to derive. The data you are gathering is dependant on how the organization is defending itself coupled with the attacker’s ability and desire to cause damage
o Nobody has perfect security and everybody experiences loss. It would be nice to know at what level management is willing to accept this residual loss within an organization
o In order for the attackers to achieve their objectives they need a method. Vulnerabilities are the opportunities the attackers require. The more you understand the vulnerabilities within an environment the better you are in a position to be able to predict future occurrences. Understanding what vulnerabilities exist for an individual organization will highlight the exposure that organization currently has, which in turn will help you understand both their occurrence and impact data. For environments which do not change much, a generic level of understanding is sufficient as the resulting loss trends will tell the story. For environments with a high degree of change, a more detailed understanding is necessary.
o Whenever you capture data be sure to get a clear picture of the technical and business environment in which the losses occurred. Otherwise the data is meaningless

Help map the consequences of different types attacks
o Not all security incidents are equal. What type of losses are being declared? Do other losses exist which will have a cascading effect? Looking only at the declared losses of a security incident is the equivalent of a disaster occurring and the news reporter simply stating something “bad” happened. The value is going to be the next level of granularity. I believe you'll need to understand many of the factors for every type of loss situation which occurs, but don’t get too far in the details. These factors become the dials in your overall algorithm, having a fundamental impact on the results you derive
o Many organizations don't understand the different losses associated with different types of attacks. I would recommend you create some standard templates to help respondents understand the scope of losses and more consistently document them

Understand the type of organization being protected
o Not all organizations are the same. Different types of attacks have varying impacts on dissimilar organizations. In some ways the differences in losses can reflect how the organization chooses to configure its defenses
o If we take into consideration traditional Confidentiality, Integrity, and Availability concepts as they pertain to different organizations, we can gain insights to how they might set up their defenses. As a general rule of thumb, a manufacturing organization is most concerned with Availability, a human resources department is most worried about Confidentiality, and finance departments are most fearful of Integrity issues. So the manufacturing organization would likely align its security defenses to protect its operational stability and to recover quickly when a security incident does occur. Such a robust defense would be unlikely in the other two areas due to cost considerations. Finance and human resource organizations would present a different strategy and configuration, aligned to what they hold most dear. The type of organization as well as a type of defenses which were in place is good to take into account for any given security incident which is reported. It will help makes sense of the numbers.

Frame the windows of time in which trending data remains valid (between disruptive technologies)
o Predictive trending for information security incidents can be very valuable. But there is a downside. Due to the disruptive nature of technology introduction and adoption, the window of prediction is very small and has a short effective life. As an example, the introduction and adoption of the Internet vastly increased the risk for the average company. Any trend data gathered before that event is worthless and would need to be ignored from a future trending perspective. In general, the more data the better a predictive model can be refined, up until the point a new disruptive technology is unleashed. Then it's time to flush the data and start anew.
o The good news is I have a white paper in the works showing this very example. Back in 2005 I developed a methodology to predict infection rates based on different types of security controls for our worldwide factory environment. This allowed us to both measure the value of discrete and overlapping security programs and to estimate the number of future incidents if a security program was to be integrated or not. Very powerful. Additionally, we tied in value calculations to derive actual dollar figures for several product technology environments. In 2005, this methodology was rightfully tagged as ‘competitive advantage’ and could not be released. That veil was lifted this year and the white paper should be published to this site by the end of the quarter.