Measuring Security Metrics & Continuous Improvement in DevSecOps

  • Share on Twitter
  • Share on LinkedIn
  • Share on Instagram

Many security departments and management teams want to improve their processes. DevSecOps introduces the ability for much more granular measurements than traditional manual security testing. Even simple measures can highlight gaps and areas for improvement where the budget can be spent.

We’re taking a closer look at the challenges of KPI metrics for software testing.


Problems of measuring security

They say you can’t improve what you can’t measure. However, if the act of measuring something affects its performance, it’s not an ideal method. In traditional security environments, there are a few problems to measuring security:

  1. The frequency of the measure. If you’re pen-testing every six months, that’s not enough granularity to know what happened in between tests.
  2. The scale of the measure. When there are more projects to test than security can handle, then projects are going to be skipped. In some companies, this could be the majority of projects. If they’re not tested, they’re not measured.
  3. The time taken to measure. It’s not just a matter of counting issues - it’s normalising issues, so you're comparing apples with apples.

DevSecOps processes and tooling gives the opportunity to automate the measurements and metrics that help drive an effective policy that works for everyone.  

If development drives down lots of security bugs, you want that to be recognised. If security teams add automation that either highlights or helps drive down risk, that should be seen and replicated where possible.

Since DevSecOps introduces the ability to measure security more often and introduces a wider scope of testing, you can get much more data to play with.

Through using day-to-day metrics from your DevSecOps security guardrails, you can avoid wasting efforts on security programs that don’t effectively reduce your risk. Instead, you can see the bigger picture where security time and resources can be most impactful. With effective metrics, you can determine what toolings will help the most.

One of the best parts is that since the metrics are recorded automatically, you can record as much as you desire and interpret insights on your security that work at all levels, from individual projects up to teams and departments across the whole company.

How DevSecOps is used to improve security metrics

Here are a few ways we’ve seen DevSecOps used to improve security metrics and the overall security processes:

Mean Time-to-Fix 

Always a popular metric, this really comes down to measurement in days. If you’re automating the discovery and the subsequent fix of that issue (i.e. CI/CD pipeline found the security issue at time X and then the same CI/CD pipeline showed the issue had been fixed at time Y), then you can accurately measure the time to fix without human interaction (Y minus X).  Automatically recording this at scale across thousands of issues gives a rich data source for performance insights.

Are any actions or processes having an effect on the speed of fixes being applied? Since DevSecOps is doing this automatically, security teams are not having to spend their time finding or calculating this information, it’s automatically available to everyone in real time.

Comparison Across Projects or Teams 

Is the Belfast team coming across more security issues than the New York team? Are their projects of greater risk? Is that because one team’s working on risky financial products and the other team is changing colours on backend systems? 

Inserting security automation in CI/CD pipelines for projects/teams lets you know where to invest in that training course or reward teams who are helping the business through their amazing cyber skills. These metrics can be used to gamify security across teams, to see who can compete on security.

What Have We Spent Our Budget On? 

Cyber teams have an interesting job. They’re given a budget to reduce risk to the business, however, it can be hard to prove a negative (we stopped a successful breach) and things change so quickly.

Let’s say you had 10,000 security issues at the start of a year, you spend $10 million on security during the year and then have 9,000 issues at the end of the year, what does that tell you? Did you simply fix 1,000 issues? Were 5,000 new issues introduced and you then fixed 6,000 issues? Through continual automated measurement of issues, risk from those issues and daily changes, you can present more data on what really happened throughout the year.

ROI of Security Tools 

Some security tools are free, some are expensive, some require little usage, some need week-long training courses to use and then time to onboard and operate. When a DevSecOps tooling process is able to show the real, side-by-side comparisons of varying tools in terms of coverage and false positives, then you’re able to make informed decisions on the true costs of security tooling, what value they’re bringing and what changes make sense.

Categories of Security Issues

We’ve seen organisations categorise the issues found in ways that make sense, maybe according to OWASP ASVS categories or NIST/CSA. By doing this, it quickly becomes clear what categories of security issues are being found and which are not. 

Perhaps nothing related to authentication or authorisation is being covered or none of the Python projects are having any injections discovered. Mapping this to the tooling you’re running informs you if there are gaps in your tool coverage that need to be filled or if dev teams have been successful at coding security in those areas.

How the Uleska Platform helps

The Uleska Platform is able to record data during CI/CD pipeline scans, facilitating these metrics and more. It’s been used to highlight categories of security issues that are causing the most risk to a company. It has built-in charts that show the changes in risk and numbers of issues, from test to test and from week to week. 

The Uleska Platform can be used to prove the amount of risk being removed from projects, teams, departments and the overall organisation, all without increasing the time needed, due to its DevSecOps automation.

Since the Uleska Platform is integrated into CI/CD Pipelines, but is run and configured outside of them, this delivers the flexibility to extend the metrics you want to record over time, without affecting the CI/CD pipeline code or technology. Even better, since the Uleska Platform records scan performance and data, you can query historical information to answer questions you didn’t know you had when you started measuring.

Given the ability of the Uleska Platform to map projects to varying levels of criticality, these security metrics can then be aligned to the sensitivity of the projects. Easy identification and sampling of this data can improve where security or development resources are focused.

With the ability of the Uleska Platform to apply fuller risk analysis to vulnerabilities found, you have a better dimension when it comes to metrics that align with the risk focus of newer security standards and regulations.

Yet, adding a risk dimension allows you to show that while the number of issues in the backlog may be growing, the overall risk to the company is decreasing due to the highest risk issues being fixed and burned down. Effectively, you’re reducing the number of issues with the highest risk first, which makes sense to the business.

To discover more about the challenges of automating DevSecOps and how to overcome them, check out our playbook. 

Learn more about the challenges of DevSecOps

The problem with DevSecOps is incorporating many layers of security tasks into the fast-paced software development cycle. Thankfully, there are a variety of things you can do to overcome the challenges faced. 

In our playbook, we cover the top 10 challenges of automating DevSecOps, while also delivering actionable advice on how to overcome them. Click the link below to download it today.

overcome the challenge of devsecops

Subscribe to the Uleska blog

You may unsubscribe at any time using the unsubscribe link in the newsletter.

Popular Articles
Visit the Blog

DevSecOps tool examples that will alleviate your workload

Nowhere is this more apparent than in DevSecOps where developers and releases outnumber security experts by thousands to one. It’s simply not...


What is CI/CD? A Complete Guide to CI/CD

Software development cycles have changed immensely in the last ten years. New practices and design philosophies are being tried every day. One of...


How to source the right tools to scale an AppSec programme

Everyone’s development is different, it stands to reason everyone’s perfect security toolkit will also be different. But finding the right tools to...


What is Penetration Testing (Pen Testing)?

Penetration testing (also known as pen testing) is the process of checking if your infrastructure and applications are robust enough to protect...


What is Dynamic Application Security Testing (DAST)?

DAST, meaning Dynamic Application Security Testing, is a form of black-box security testing. It simulates external attacks on a live application,...


Ultimate Guide to DevSecOps

What is DevSecOps?  DevSecOps meaning (Development, Security, and Operations) primarily aims to automate security in each part of the software...

Company News, Security

Uleska and Log4Shell

Summary TL;DR: If you are a cloud customer of Uleska, you are not vulnerable to log4shell. If you are an on-premise customer of Uleska and are using...


The Top Application Security Tools in 2021

In modern businesses, applications have assumed a pivotal role. And while applications help with operational processes, the majority of cyber-attacks...


The Ultimate Guide to Application Security Tools

With the emergence of new software security threats, businesses need robust, flexible and affordable methods to ensure their applications are...


Introducing the DevSecOps Toolkit: A guide to scaling an AppSec programme

Imagine you’ve been asked to build a house from scratch. You don’t have any tools. You don’t have any experience. In fact, all you have is an empty...


What is Static Application Security Testing (SAST) and how does it work?

What is SAST? Static Application Security Testing (SAST), or static analysis, is a method of testing and analysing source code. This method allows...

Tools, Featured

Choosing the Best AppSec Tools: Advice from Experienced Engineers

In our latest webinar Gary Robinson and Martin Hewitt from Uleska gave us a fascinating and comprehensive look into how experienced security teams...

Managing Risk

How to Use Risk Based Security Testing [With Video]

Last week we discussed how using risk-based decisions can help speed up pipelines. You can watch the webinar on demand and read a summary of the...


Can DevSecOps Tools Open Security Testing To Everyone?

At Uleska, we focus on moving security testing away from experts running manual tests and move it to automating security checks into existing...

Company News

Start your DevSecOps journey with the Uleska free plan

Companies are developing and shipping software faster than ever before. The very nature of DevOps means that developers can work in an always-on...