Measuring Security Metrics & Continuous Improvement in DevSecOps [Video]

  • Share on Twitter
  • Share on LinkedIn
  • Share on Instagram

Many security departments and management teams want to improve their processes. DevSecOps introduces the ability for much more granular measurements than traditional manual security testing. Even simple measures can highlight gaps and areas for improvement where the budget can be spent.

We’re taking a closer look at the challenges of KPI metrics for software testing.


Problems of measuring security

They say you can’t improve what you can’t measure. However, if the act of measuring something affects its performance, it’s not an ideal method. In traditional security environments, there are a few problems to measuring security:

  1. The frequency of the measure. If you’re pen-testing every six months, that’s not enough granularity to know what happened in between tests.
  2. The scale of the measure. When there are more projects to test than security can handle, then projects are going to be skipped. In some companies, this could be the majority of projects. If they’re not tested, they’re not measured.
  3. The time taken to measure. It’s not just a matter of counting issues - it’s normalising issues, so you're comparing apples with apples.

DevSecOps processes and tooling gives the opportunity to automate the measurements and metrics that help drive an effective policy that works for everyone.  

If development drives down lots of security bugs, you want that to be recognised. If security teams add automation that either highlights or helps drive down risk, that should be seen and replicated where possible.

Since DevSecOps introduces the ability to measure security more often and introduces a wider scope of testing, you can get much more data to play with.

Through using day-to-day metrics from your DevSecOps security guardrails, you can avoid wasting efforts on security programs that don’t effectively reduce your risk. Instead, you can see the bigger picture where security time and resources can be most impactful. With effective metrics, you can determine what toolings will help the most.

One of the best parts is that since the metrics are recorded automatically, you can record as much as you desire and interpret insights on your security that work at all levels, from individual projects up to teams and departments across the whole company.

How DevSecOps is used to improve security metrics

Here are a few ways we’ve seen DevSecOps used to improve security metrics and the overall security processes:

Mean Time-to-Fix 

Always a popular metric, this really comes down to measurement in days. If you’re automating the discovery and the subsequent fix of that issue (i.e. CI/CD pipeline found the security issue at time X and then the same CI/CD pipeline showed the issue had been fixed at time Y), then you can accurately measure the time to fix without human interaction (Y minus X).  Automatically recording this at scale across thousands of issues gives a rich data source for performance insights.

Are any actions or processes having an effect on the speed of fixes being applied? Since DevSecOps is doing this automatically, security teams are not having to spend their time finding or calculating this information, it’s automatically available to everyone in real time.

Comparison Across Projects or Teams 

Is the Belfast team coming across more security issues than the New York team? Are their projects of greater risk? Is that because one team’s working on risky financial products and the other team is changing colours on backend systems? 

Inserting security automation in CI/CD pipelines for projects/teams lets you know where to invest in that training course or reward teams who are helping the business through their amazing cyber skills. These metrics can be used to gamify security across teams, to see who can compete on security.

What Have We Spent Our Budget On? 

Cyber teams have an interesting job. They’re given a budget to reduce risk to the business, however, it can be hard to prove a negative (we stopped a successful breach) and things change so quickly.

Let’s say you had 10,000 security issues at the start of a year, you spend $10 million on security during the year and then have 9,000 issues at the end of the year, what does that tell you? Did you simply fix 1,000 issues? Were 5,000 new issues introduced and you then fixed 6,000 issues? Through continual automated measurement of issues, risk from those issues and daily changes, you can present more data on what really happened throughout the year.

ROI of Security Tools 

Some security tools are free, some are expensive, some require little usage, some need week-long training courses to use and then time to onboard and operate. When a DevSecOps tooling process is able to show the real, side-by-side comparisons of varying tools in terms of coverage and false positives, then you’re able to make informed decisions on the true costs of security tooling, what value they’re bringing and what changes make sense.

Categories of Security Issues

We’ve seen organisations categorise the issues found in ways that make sense, maybe according to OWASP ASVS categories or NIST/CSA. By doing this, it quickly becomes clear what categories of security issues are being found and which are not. 

Perhaps nothing related to authentication or authorisation is being covered or none of the Python projects are having any injections discovered. Mapping this to the tooling you’re running informs you if there are gaps in your tool coverage that need to be filled or if dev teams have been successful at coding security in those areas.

How the Uleska Platform helps

The Uleska Platform is able to record data during CI/CD pipeline scans, facilitating these metrics and more. It’s been used to highlight categories of security issues that are causing the most risk to a company. It has built-in charts that show the changes in risk and numbers of issues, from test to test and from week to week. 

The Uleska Platform can be used to prove the amount of risk being removed from projects, teams, departments and the overall organisation, all without increasing the time needed, due to its DevSecOps automation.

Since the Uleska Platform is integrated into CI/CD Pipelines, but is run and configured outside of them, this delivers the flexibility to extend the metrics you want to record over time, without affecting the CI/CD pipeline code or technology. Even better, since the Uleska Platform records scan performance and data, you can query historical information to answer questions you didn’t know you had when you started measuring.

Given the ability of the Uleska Platform to map projects to varying levels of criticality, these security metrics can then be aligned to the sensitivity of the projects. Easy identification and sampling of this data can improve where security or development resources are focused.

With the ability of the Uleska Platform to apply fuller risk analysis to vulnerabilities found, you have a better dimension when it comes to metrics that align with the risk focus of newer security standards and regulations.

Yet, adding a risk dimension allows you to show that while the number of issues in the backlog may be growing, the overall risk to the company is decreasing due to the highest risk issues being fixed and burned down. Effectively, you’re reducing the number of issues with the highest risk first, which makes sense to the business.

To discover more about the challenges of automating DevSecOps and how to overcome them, check out our playbook. 

Learn more about the challenges of DevSecOps

The problem with DevSecOps is incorporating many layers of security tasks into the fast-paced software development cycle. Thankfully, there are a variety of things you can do to overcome the challenges faced. 

In our playbook, we cover the top 10 challenges of automating DevSecOps, while also delivering actionable advice on how to overcome them. Click the link below to download it today.

overcome the challenge of devsecops

Subscribe to the Uleska blog

You may unsubscribe at any time using the unsubscribe link in the newsletter.

Popular Articles
Visit the Blog

Open Source Security Testing Tools

Security tools are an essential part of software development today, especially with the ever-increasing number of attacks we see every year....


Security Orchestration Automation and Response (SOAR)

Security teams frequently struggle with the volume of alerts and issues they are tasked with daily. On average, most enterprises receive between...


Secure Software Development Life Cycle

Software development has evolved into an incredibly complex machine, with several moving parts to keep track of. Teams get more extensive, and...


Application Security Orchestration & Correlation

Application Security is a constantly evolving industry, with new threats and methods to combat them appearing regularly. One of the more recent...


Top 5 AppSec Productivity Hacks 2022

The application security (AppSec) industry moves fast. Development, security and operations (DevSecOps) practitioners are having to find creative...


How to improve security tool selection and customisation with Uleska Toolkits

We know starting your application security (AppSec) journey can be a little overwhelming. After all, choosing your tools from scratch and setting...

Application Security

What is Application Security? A Beginner’s Guide

What is Application Security? Application Security is defined by developing, adding, and testing security features in an application or website....


Vulnerability Assessments in Application Security

Did you know that over 79% of developers surveyed in 2020 stated their applications had 20 or more vulnerabilities on average? As the digital world...


Defining and breaking down Vulnerability Management

No system is perfectly secure, as proven by software analysis firm CAST, which reviewed 278 million lines of code and discovered more than 1.3...

Company News, Featured

Toolkits: Taking the guesswork out of security tool selection and customisation

There are thousands of amazing AppSec tools out there, but this can be both a blessing and a curse. While the headway and innovation we are seeing...


How to eliminate risk when scaling application security

Building robust application security is a lot like building a house—you want it done thoroughly, without any missing parts. However, there is a...


What is the OWASP Top 10 and how to use it?

Cybersecurity has been a rising concern in the last decade. In 2021, researchers have seen 50% more attacks per week on corporate networks compared...


What is Shift Left? Ultimate Guide to Shift Left Security

With today’s fast development speeds, it’s hard to keep up with security practices for some organisations. This is especially true in the last few...


What is Software Composition Analysis?

Open-source software has become a vital part of development in the last decade. However, utilising these components often comes with several caveats,...


DevSecOps tool examples that will alleviate your workload

The saying goes: “Many hands make light work.” Nowhere is this more apparent than in DevSecOps where developers and releases outnumber security...