Product
Pricing
Resources
Docs
Product
Pricing
Resources
Docs

Top 10 DevSecOps Challenges #9: Security Metrics, Insights and Continuous Improvement

  • Share on Twitter
  • Share on LinkedIn
  • Share on Instagram

Many security departments and management teams want to improve their processes. DevSecOps introduces the ability for much more granular measurements than traditional manual security testing. Even simple measures can highlight gaps and areas for improvement where the budget can be spent.

We’re taking a closer look at the challenges of KPI metrics for software testing.

Problems of measuring security

They say you can’t improve what you can’t measure. However, if the act of measuring something affects its performance, it’s not an ideal method. In traditional security environments, there are a few problems to measuring security:

  1. The frequency of the measure. If you’re pen-testing every six months, that’s not enough granularity to know what happened in between tests.
  2. The scale of the measure. When there are more projects to test than security can handle, then projects are going to be skipped. In some companies, this could be the majority of projects. If they’re not tested, they’re not measured.
  3. The time taken to measure. It’s not just a matter of counting issues - it’s normalising issues, so you're comparing apples with apples.

DevSecOps processes and tooling gives the opportunity to automate the measurements and metrics that help drive an effective policy that works for everyone.  

If development drives down lots of security bugs, you want that to be recognised. If security teams add automation that either highlights or helps drive down risk, that should be seen and replicated where possible.

Since DevSecOps introduces the ability to measure security more often and introduces a wider scope of testing, you can get much more data to play with.

Through using day-to-day metrics from your DevSecOps security guardrails, you can avoid wasting efforts on security programs that don’t effectively reduce your risk. Instead, you can see the bigger picture where security time and resources can be most impactful. With effective metrics, you can determine what toolings will help the most.

One of the best parts is that since the metrics are recorded automatically, you can record as much as you desire and interpret insights on your security that work at all levels, from individual projects up to teams and departments across the whole company.

How DevSecOps is used to improve security metrics

Here are a few ways we’ve seen DevSecOps used to improve security metrics and the overall security processes:

Mean Time-to-Fix 

Always a popular metric, this really comes down to measurement in days. If you’re automating the discovery and the subsequent fix of that issue (i.e. CI/CD pipeline found the security issue at time X and then the same CI/CD pipeline showed the issue had been fixed at time Y), then you can accurately measure the time to fix without human interaction (Y minus X).  Automatically recording this at scale across thousands of issues gives a rich data source for performance insights.

Are any actions or processes having an effect on the speed of fixes being applied? Since DevSecOps is doing this automatically, security teams are not having to spend their time finding or calculating this information, it’s automatically available to everyone in real time.

Comparison Across Projects or Teams 

Is the Belfast team coming across more security issues than the New York team? Are their projects of greater risk? Is that because one team’s working on risky financial products and the other team is changing colours on backend systems? 

Inserting security automation in CI/CD pipelines for projects/teams lets you know where to invest in that training course or reward teams who are helping the business through their amazing cyber skills. These metrics can be used to gamify security across teams, to see who can compete on security.

What Have We Spent Our Budget On? 

Cyber teams have an interesting job. They’re given a budget to reduce risk to the business, however, it can be hard to prove a negative (we stopped a successful breach) and things change so quickly.

Let’s say you had 10,000 security issues at the start of a year, you spend $10 million on security during the year and then have 9,000 issues at the end of the year, what does that tell you? Did you simply fix 1,000 issues? Were 5,000 new issues introduced and you then fixed 6,000 issues? Through continual automated measurement of issues, risk from those issues and daily changes, you can present more data on what really happened throughout the year.

ROI of Security Tools 

Some security tools are free, some are expensive, some require little usage, some need week-long training courses to use and then time to onboard and operate. When a DevSecOps tooling process is able to show the real, side-by-side comparisons of varying tools in terms of coverage and false positives, then you’re able to make informed decisions on the true costs of security tooling, what value they’re bringing and what changes make sense.

Categories of Security Issues

We’ve seen organisations categorise the issues found in ways that make sense, maybe according to OWASP ASVS categories or NIST/CSA. By doing this, it quickly becomes clear what categories of security issues are being found and which are not. 

Perhaps nothing related to authentication or authorisation is being covered or none of the Python projects are having any injections discovered. Mapping this to the tooling you’re running informs you if there are gaps in your tool coverage that need to be filled or if dev teams have been successful at coding security in those areas.

How the Uleska Platform helps

The Uleska Platform is able to record data during CI/CD pipeline scans, facilitating these metrics and more. It’s been used to highlight categories of security issues that are causing the most risk to a company. It has built-in charts that show the changes in risk and numbers of issues, from test to test and from week to week. 

The Uleska Platform can be used to prove the amount of risk being removed from projects, teams, departments and the overall organisation, all without increasing the time needed, due to its DevSecOps automation.

Since the Uleska Platform is integrated into CI/CD Pipelines, but is run and configured outside of them, this delivers the flexibility to extend the metrics you want to record over time, without affecting the CI/CD pipeline code or technology. Even better, since the Uleska Platform records scan performance and data, you can query historical information to answer questions you didn’t know you had when you started measuring.

Given the ability of the Uleska Platform to map projects to varying levels of criticality, these security metrics can then be aligned to the sensitivity of the projects. Easy identification and sampling of this data can improve where security or development resources are focused.

With the ability of the Uleska Platform to apply fuller risk analysis to vulnerabilities found, you have a better dimension when it comes to metrics that align with the risk focus of newer security standards and regulations.

Yet, adding a risk dimension allows you to show that while the number of issues in the backlog may be growing, the overall risk to the company is decreasing due to the highest risk issues being fixed and burned down. Effectively, you’re reducing the number of issues with the highest risk first, which makes sense to the business.

To discover more about the challenges of automating DevSecOps and how to overcome them, check out our playbook. 

Learn more about the challenges of DevSecOps

The problem with DevSecOps is incorporating many layers of security tasks into the fast-paced software development cycle. Thankfully, there are a variety of things you can do to overcome the challenges faced. 

In our playbook, we cover the top 10 challenges of automating DevSecOps, while also delivering actionable advice on how to overcome them. Click the link below to download it today.

overcome the challenge of devsecops

Subscribe to the Uleska blog

You may unsubscribe at any time using the unsubscribe link in the newsletter.

Popular Articles
Visit the Blog
Managing Risk

Speed up Pipelines Using Automated Risk-Based Decisions

Last week we discussed how using risk-based decisions can help speed up pipelines. You can watch the webinar on demand and read a summary of the...

DevSecOps

Can DevSecOps Tools Open Security Testing To Everyone?

At Uleska, we focus on moving security testing away from experts running manual tests and move it to automating security checks into existing...

Company News

Start your DevSecOps journey with the Uleska free plan

Companies are developing and shipping software faster than ever before. The very nature of DevOps means that developers can work in an always-on...

DevSecOps

Top 10 DevSecOps Challenges #10: Communication

Adding automation to one part of a process can then flood another part of a process. With DevSecOps, we’re allowing more security tools to find more...

Collaboration

AppSec ❤️ DevOps: Bridging the DevSecOps Disconnect

It’s a tale as old as time: developers want to ship an app but are lambasted with security requests, and security teams want to secure an app but are...

Managing Risk

Top 10 DevSecOps Challenges #8: Adding Risk Prioritisation to Your Pipeline Security

DevSecOps increases the number of issues found and the speed at which they’re to be dealt with. In reality, only a small number of issues will pose a...

DevSecOps

Top 10 DevSecOps Challenges #7: Mapping security automation to how development works

All teams present in the app development process have pressures on them to get work done fast and efficiently.  With DevOps processes and CI/CD...

DevSecOps

Top 10 DevSecOps Challenges #6: The all-important triaging of security issues

Security tools can be noisy. In 20 years, we haven’t seen a single security tool return a set of issues that are 100% what needs to be worked on....

DevSecOps

Top 10 DevSecOps Challenges #5: Running too many security tools in CI/CD

DevSecOps involves setting up many different automated security tools to cover all bases. It’s not uncommon for organisations to run tons of security...

DevSecOps

Top 10 DevSecOps Challenges #4: Using DevSecOps to reduce and focus issues raised

One of the biggest challenges when rolling out a DevSecOps process is the volume of issues it can bring to light. 

DevSecOps

Top 10 DevSecOps Challenges #3: Doing DevSecOps without constant CI/CD changes

Better collaboration between teams, faster time to market, improved overall productivity and enhanced customer satisfaction are just some of the...

DevSecOps

Top 10 DevSecOps Challenges #2: Fitting DevSecOps into CI/CD Pipelines

Put simply, the goal of CI/CD pipelines is automation and a key goal of DevSecOps is to alert someone to a problem as early in the automated-delivery...

DevSecOps

Top 10 DevSecOps Challenges #1: How to approach DevSecOps security automation

DevSecOps encourages security tasks to be wrapped and enabled with software development and operations tasks. The aim is to make them as seamless as...

Company News

Introducing Uleska: The Future of DevSecOps Automation Tools

Few industries have seen such scrutiny and shifts in recent years as cybersecurity. In an increasingly connected world, speed and agility throughout...