The Limitations of Security Data
We are constantly being bombarded by cybersecurity data, reports, and marketing collateral—and not all of this information should be treated equally. Security data inherently has limitations and biases, which result in varying value and relevance in how it should be applied. It is important to understand which is significant and how best to allow it to influence your decisions.
There is a tsunami of security metrics, reports, analyses, blogs, papers, and articles vying for attention. Sources range from reporters, researchers, professional security teams, consultants, dedicated marketing groups, and even security-operations people who are adding data, figures, and opinions to the cauldron. We are flooded with data and all those who have opinions on it.
It was not always this way. Over a decade ago, it was an information desert, where even speculations were rare. Making decisions driven by data has always been a good practice. Years ago, many advocates were working hard to convince the industry to share information. Even a drop is better than none. Most groups that were capturing metrics were too frightened or embarrassed to share. Data was kept secret by everyone while decision makers were clamoring for security insights based upon industry numbers, which simply were not available.
What Was the Result?
Fear, uncertainty, and doubt ruled. People began to dread the worst and unscrupulous security marketing advocates took advantage, fanning the flames to sell products and snake oil. They were dark times, promulgated with outlandish claims like “we solve security,” “total protection,” and “complete security solution.” Why customers chose to believe such nonsense (when the problem and the effectiveness of potential solutions could not be quantified) is beyond me, but many did and trust in the security solutions industry was lost for a period of time.
Slowly, a trickle of informative sources began to produce reports and publish data. Such initiatives gained momentum with others joining in to share in limited amounts. It was a turning point. Armed with data and critical thinking, clarity and common sense began to take root. It was not perfect or quick, but the introduction of data from credible sources empowered security organizations to better understand the challenge and effective ways to maneuver against threats.
As the size of the market and competition grew, additional viewpoints joined the fray. Today, we are bombarded by all manner of cybersecurity information. Some are credible while others are not. There are several types of data being presented, ranging from speculations to hard research. Being well-informed is extremely valuable to decision makers. Now, the problem is figuring out how to filter and organize the data so one is not mislead.
As part of my role as a cybersecurity strategist, I both publish information to the community and consume vast amounts of industry data. To manage the burden and avoid the risks of believing less-than-trustworthy information, I have a quick guide to help structure the process. It is burned into my mind as a set of filters and rules, but I am committing it to paper in order to share.
I categorize data into four buckets. These are: Speculation, Survey, Actuarial, and Research. Each has its pros and cons. The key to managing security data overload is to understand the limitations of each class, its respective value and its recommended usage.
For example, Survey data is the most unreliable, but does have value in understanding the fears and perceptions of the respondent community. Research data is normally very accurate but notoriously narrow in scope and may be late to the game. One of my favorites is Actuarial data. I am a pragmatic guy. I want to know what is actually happening so I can make my own conclusions. But there are limitations to Actuarial data as well. It tends to be very limited in size and scope, so you can’t look too far into it and it is a reflection of the past, which may not align to the future.
I hear lots of different complaints and criticisms when it comes to the validity, scope, intent, and usage of data. I personally have my favorites and those which I refuse to even read. Security data is notoriously difficult. There are so many limitations and biases, it is far easier to point out issues than to see the diamond in the rough. But data can be valuable if filtered, corrected for bias, and the limitations are known. Don’t go in blind. Common sense must be applied. Have a consistent method and structure to avoid pitfalls and maximize the data available to help you manage and maintain an optimal level of security.
Below are a few examples, in my opinion, of credible cybersecurity data across the spectrum of different categories. Again keep in mind the limitations of each group and don’t make the mistake of using the information improperly! Look to Speculation for the best opinions, Survey for the pulse of industry perceptions, Actuarial for real events, and Research for deep analysis:
- 2016 Cybersecurity Threat PredictionsMcAfee Labs
- $243 billion - $1 trillion. Potential cost of a single attack against the US Power Grid, per Lloyds Insurance
- ~3 Trillion aggregate economic impact of cybersecurity on technology trends, through 2020. World Economic Forum 2014 report Risk and Responsibility in a Hyperconnected World
- $90 trillionAtlantic Council’s report estimate.
- 55% CAGR. Growth of global IoT Security market 2016-2020, per researchandmarkets.com
- My 2016 Cybersecurity Predictions, in fact most of my blogs
- Threat Intelligence Sharing surveyMcAfee Labs Threats Report
- 20% jump in cybercrimePwC
- 25% Americans believe they have experienced a data breachTravelers survey
- 43%ESG IT spending intentions research
- 61% of CEO’s believe cyber threats pose a danger to corporate growth per PwC survey
- 3 out of 5 Californians2016 California Data Breach Report
- ~35% of the US population. Top 10 Healthcare breaches of 2015, affected almost 35% of the US population Source: Office of Civil Rights
- Data Breach Investigations Reportreport by Verizon
- 2016 Annual Security ReportCisco
- 42 millionMcAfee Labs Threat Report
- Security Intelligence ReportMicrosoft
- $325M losses attributed to Cryptowall v3 ransomware, analysis from the Cyber Threat Alliance
- $13.1 billion. U.S. Government spends on cybersecurity in 2015. Source: FISMA report from OMB
By the way, yes, this very blog would be considered Speculation. Treat it as such.