Despite all the forward progress, how some organizations internally report on the state of their cybersecurity still contains too much illusion. Without universal standards and metrics, security professionals self-reporting to their leadership are left on their own devices. We hear that everything is mostly alright, and “you need to trust us”, or “we have defense-in-depth and mitigating controls!”. Then there is always an ask for a bigger budget for more staff or cutting-edge tools. To confuse the situation even more, we hear the obligatory, “Look how hard our understaffed team is working, and the thousands of attacks prevented and vulnerabilities patched!” And let us not forget the regular update on all the current security “initiatives” (e.g. – rolling out two-factor authentication, or SOAR for the SOC) underway which promise to reduce risk.
In our more than 19 years of industry experience, working with numerous organizations across all markets, there is a systemic problem in genuine cybersecurity reporting which is keeping our industry from advancing.
Let’s examine the current state of cybersecurity reporting and some other examples of “vanity” metrics we often hear and see reported, which we’ve highlighted below.
- Budget / Team Size: “My budget is $10M, we have 120 staff.”
- Tools Stack: “We use [insert Gartner Magic Quadrant leader here].”
- Blocked Attacks: “Our latest gen firewall blocks 70k attacks a minute!”
- Patching: “Our team patches 90k vulnerabilities a month!”
- Logging: “Our SIEM team is optimizing the search head cluster!”
- Incident Response: “Our IR team investigated 32 incidents this month, 30 were false positives!”
- Initiatives: “Our bake-off of 3rd party risk tools is going well, we’ve narrowed it down to two vendors.”
- Outside Support: “We use [insert top tier cybersecurity consulting firm here] for APT hunting.”
- Frameworks / Compliance: “We use [insert a framework here] compliance framework and are using the MITRE ATT&CK framework.
Does this sound familiar? At first glance, one might think, “Wow, the team and tools we’re paying large sums of money for are doing a lot and working hard – this is very commendable, this must be the limit of what’s possible, and all they can do!” Right and wrong – security is a thankless job that is difficult and complex. It is not however universally managed in a way to maximize the value of the organization’s security investment.
Upon closer examination – lots of activity and minimal achievement – with none of the above examples offering much substance around the actual effectiveness of a security program.
The Infamous Equifax Breach
Leading up to their 2017 breach, Equifax likely had similar ‘awesomeness’ with their reported $85M annual cybersecurity budget. They had a large cyber team with vulnerability management and network monitoring tools, processes, and governance. Yet, there was no enterprise visibility into cybersecurity effectiveness. The Board and C-level leadership did not understand their true picture or what implications it had on overall risk to the business.
It took Equifax:
- 145 days to patch
- 76 days to identify their breach
- 15 & 22 days to inform the CEO and Board of a likely data breach
Not For the Faint of Heart
Security is hard. Ops generally isn’t a fan and it requires a particular set of skills along with an infusion of security into the corporate culture. Couple this with protecting against sophisticated Nation State actors, budget constraints, and an ISC2 projected shortage of 4 million worldwide trained cybersecurity positions, and you have a thankless uphill battle where not everyone will make it out alive. Given the landscape, let’s all pause and give our security leaders the sincere credit and thanks they deserve as they hold arguably the hardest of all C-level positions. Still not convinced? Here‘s a quick unexhaustive list of daily challenges they face:
| Competing priorities | Patching | Ransomware |
| Collaboration across business units | Maturating security posture | Data classification |
| Internal threat actors | Asset inventory | Communicating value to leadership |
| Maintaining a defensible security posture | Access & Identity Mgt | Justifying budget |
| User awareness / phishing | Endpoint security / encryption | Staffing shortfalls / retention |
| Quantifying risk | Compliance / Audits | Understanding true security posture |
| Incident response | Insider threats | Choosing between 3k+ products |
| Tool integration | 3rd party risk | Incident recovery |
Every CISO we’ve met is intelligent and has nothing but the best intentions of doing good and protecting the enterprise while enabling the business. CISO’s and other security leaders need to report on the state of their efforts to gain support and funding, and to ensure everyone is aware of all the good work being done. For better or worse, reporting requirements are not universal, meaning what is reported at one institution may not be reported at another. There is no standard. Security professionals tend to default to those “metrics” that are relatively easy to derive and tell a story of great effort and perpetuate their employment. After all, most teams are understaffed and take the burdensome approach of wrestling countless unwieldy spreadsheets to build their own manual reports derived from multiple tools and across teams.
Most security metrics used today paint a picture of confidence and competence that the uninitiated dare not question (or most likely don’t know how to question). These vanity metrics buy the security professional more time to complete an ongoing initiative (that rarely lives up to its potential), or hire the SOC analyst who certainly will take the team to the next level, etc., etc. The reality is these well-intentioned hopes and dreams rarely materialize or pay the dividends envisioned.
There is an element of security theater, a bit of a kabuki dance between security professionals, vendors, and business executives, managing the tension of egos, marketing hype, and a language barrier. Besides, we’re all busy and should be able to trust what our experts are telling us, especially when they, “have things under control” or need yet another of today’s 3,000+ cyber tools to shore up a recent audit finding. When you’re told you have policies, procedures, perhaps a SOC and the latest SIEM with UAB to include proprietary, black box algorithms driving cutting edge AI and ML risk scoring technologies – all is right in the world. Throw in real live threat hunters and wow, the CEO and board can’t stop talking about how sophisticated their security program is – congratulations!
However, your gut is right, here lies residual systemic risk: your organization is not as secure, mature, or effective as you’ve been told.
It’s Time to Achieve More in Cyber
All lightheartedness aside, if we want to truly make meaningful strides in cybersecurity, we need to change the paradigm from activity to achievement. Consider new research out of Stanford University by Nicolas Bloom asserting that management practices account for more than 20% of productivity variations. The research goes on to claim this is a similar, or greater, percentage as that accounted for by R&D, Information and Communication Technologies, or human capital. Simply put: well-managed teams using Key Performance Indicators (KPIs) can perform at a higher rate as those teams that have additional tech or staff. To determine this, the researchers collected volumes of US census data to better understand management practices. They focused on the degree in which KPIs were established, visible, reviewed, and embedded as part of employee performance conversations and incentives. Something intuitive, now backed by research.
In our next blog, we’ll talk about how using KPIs can make cybersecurity teams more effective.
