In a previous blog on IT Security Metrics as part of a Good IT Security Program, I described the need for a security metrics program that can provide was to show value of the security program to the organization. A simple way to begin with metrics is to define the reason or goal, question, and metric for what needs to be measured. After that, an effort can begin on data collection, analysis, and reporting.
The process by which the data is collected should be well understood and documented so that as changes occur in the security processes and controls, these documents can be reviewed for what changes may be necessary in data collection update methods. The data collection methods are vital as there could be the need for measurement units to be transposed in order to provide correlating analysis to other units of measurement. Additionally, if there are qualitative measurements being analyzed, the approach to this transposition of units for measurement must be well documented so that future review can take into considerations changes that could affect the analysis.
The analysis process for a given metric should be documented so that if necessary, other individuals can take ownership of the metrics analysis to interpret the results and provide a conclusion. Quantitative metric data provides numerical values for the units of measurement and once collected over time, determination of a mean and standard deviation can be calculated to describe a range for the metric. For qualitative data to be used as a metric, the unit of measurements should also be clearly document measurement techniques with security analysts. To avoid bias during analysis of qualitative metrics, a purposeful effort should be made to get agreement from all stakeholders and members of the analysis team on how qualitative data will be analyzed.
When presenting results, it is important to understand the audience to which information will be reported to or presented. Some audiences may not be interested in technical details of the data but more information in the cost to benefit ratio or return on Investment. Indicators can be very valuable to the business group in presenting the current state of security within their area of responsibility. Key Performance Indicators can be presented on a scorecard dashboard by which the team can collectively review the status of security and provide input for more creative analysis from other perspectives.
One more important aspect of a security metrics program is that it doesn’t have to be a set it and forget about it approach even if much of the data collection methods are automated. Future efforts can be ongoing to review previously defined metrics along with the attempt to find other relevant metrics for continuous improvement.