Can DevSecOps Tools Open Security Testing To Everyone?

  • Share on Twitter
  • Share on LinkedIn
  • Share on Instagram

At Uleska, we focus on moving security testing away from experts running manual tests and move it to automating security checks into existing processes. We believe in continually testing ourselves in this mission, which is why we tasked a group of people without software or security skills to test 1000 projects and feed the results back to the project teams.

Here’s why we did it, how we did it and what we learned.

Why did we conduct this experiment?

Firstly, many of our customers have security teams or developers who are interested in security that work with our product. We want to push that boundary so anyone can undertake a security test. There’s also a trend in many teams for security testing to be more transparent to developers, yet still produce meaningful results.

We want to give something back to the open-source community by alerting them to any issues they may need to look at. Plus, as many security tools have UI/UX aimed at security professionals, we want to test our UI for the simplest ease-of-use we can find.

The experiment: Javascript source code security testing tools

To begin our experiment, we chose 1,000 Javascript projects on GitHub that were recently updated and had between 50 and 200 stars in rating. Next, we went to and paid non-professionals £5/hour to use the Uleska Platform to automatically run three Javascript source code security testing tools. We consolidated the results generated by the platform to the project teams and made sure they didn’t have security or programming skills.

We didn’t set up any of our cyber value-at-risk enumerations, as they would apply to how an open-source library is used within the end-user application (i.e. types of data processed, quantity of data, environment etc.).

We also made the decision not to triage the results, as the Uleska Platform has functionality to automatically remove false positives and duplicates, a feature heavily used by our customers.

Since this experiment was for open source projects, instead of enterprise software teams, we wanted to see how difficult (or easy) reports containing a mix of real and false-positive issues were for open source teams.

unnamed (1)

Some of the vulnerabilities found, split by the users and applications (projects).

DevSecOps tools outcomes

The experiment was a great success, providing a variety of valuable insights, including some improvements we can make to our DevSecOps orchestration usage. Some of our main learnings were:

  1. With simple instructions, non-skilled people were able to onboard hundreds of projects. They were also able to run a number of security tools easily, mainly due to the abstraction the Uleska Platform provided. Instead of starting command lines or setting up profiles, this was done easily by the click of a button. Plus, the onboarding/execution only took a few days.
  2. As operations run so frequently, we discovered ways to speed up our own UI/UX and our API and make it even simpler, aiming for 2-3 clicks to set up a project/application test. The feedback we received from the group showed the team using the Uleska Platform UI to kick off the testing instead of any triggers from GitHub or DevOps tools or continuous integration. However, the effect would be the same.
  3. There were over 35,000 issues registered by the tooling, some of which were false positives, with others acknowledged by the project teams as issues to be fixed. Around 10% of the projects tested didn’t return any issues at all. 

In the next stage of this experiment, we will look to automatically remove the false positives. 

DevSecOps challenges with this experiment

During this experiment, we ran into some logistical challenges. Firstly, we created new GitHub accounts to find/extract the GitHub URL for the project to pass that into the Uleska Platform so the codeline can be tested. 

These new GitHub accounts were also used to update the projects with the report of security issues. This meant these GitHub accounts were now creating GitHub projects or code which were later flagged by GitHub. This meant they were no longer able to submit issue reports to projects. For this reason, we stopped short of the full 1,000 projects, ending at around the 730 project mark.

We also had a few projects react negatively to being passed security reports out of the blue. Sometimes this was because the false positives weren't removed, other times our reports were perceived as spam. Now we understand where we need to improve, in our next iteration of this experiment, we’re looking to remove the likely false positives using the Uleska technology and look forward to helping more open source projects stay secure.

To discover more about the challenges of automating DevSecOps and Uleska can help overcome them, check out our playbook. 

Discover how to overcome the challenges of DevSecOps

The problem with DevSecOps is incorporating many layers of security tasks into the fast-paced software development cycle. Thankfully, there are a variety of things you can do to overcome the challenges faced. In our playbook, we cover the top 10 challenges of automating DevSecOps, while also delivering actionable advice on how to overcome them.overcome the challenge of devsecops

Subscribe to the Uleska blog

You may unsubscribe at any time using the unsubscribe link in the newsletter.

Popular Articles
Visit the Blog
Managing Risk

Speed up Pipelines Using Automated Risk-Based Decisions

Last week we discussed how using risk-based decisions can help speed up pipelines. You can watch the webinar on demand and read a summary of the...

Company News

Start your DevSecOps journey with the Uleska free plan

Companies are developing and shipping software faster than ever before. The very nature of DevOps means that developers can work in an always-on...


Top 10 DevSecOps Challenges #10: Communication

Adding automation to one part of a process can then flood another part of a process. With DevSecOps, we’re allowing more security tools to find more...


Top 10 DevSecOps Challenges #9: Security Metrics, Insights and Continuous Improvement

Many security departments and management teams want to improve their processes. DevSecOps introduces the ability for much more granular measurements...


AppSec ❤️ DevOps: Bridging the DevSecOps Disconnect

It’s a tale as old as time: developers want to ship an app but are lambasted with security requests, and security teams want to secure an app but are...

Managing Risk

Top 10 DevSecOps Challenges #8: Adding Risk Prioritisation to Your Pipeline Security

DevSecOps increases the number of issues found and the speed at which they’re to be dealt with. In reality, only a small number of issues will pose a...


Top 10 DevSecOps Challenges #7: Mapping security automation to how development works

All teams present in the app development process have pressures on them to get work done fast and efficiently.  With DevOps processes and CI/CD...


Top 10 DevSecOps Challenges #6: The all-important triaging of security issues

Security tools can be noisy. In 20 years, we haven’t seen a single security tool return a set of issues that are 100% what needs to be worked on....


Top 10 DevSecOps Challenges #5: Running too many security tools in CI/CD

DevSecOps involves setting up many different automated security tools to cover all bases. It’s not uncommon for organisations to run tons of security...


Top 10 DevSecOps Challenges #4: Using DevSecOps to reduce and focus issues raised

One of the biggest challenges when rolling out a DevSecOps process is the volume of issues it can bring to light. 


Top 10 DevSecOps Challenges #3: Doing DevSecOps without constant CI/CD changes

Better collaboration between teams, faster time to market, improved overall productivity and enhanced customer satisfaction are just some of the...


Top 10 DevSecOps Challenges #2: Fitting DevSecOps into CI/CD Pipelines

Put simply, the goal of CI/CD pipelines is automation and a key goal of DevSecOps is to alert someone to a problem as early in the automated-delivery...


Top 10 DevSecOps Challenges #1: How to approach DevSecOps security automation

DevSecOps encourages security tasks to be wrapped and enabled with software development and operations tasks. The aim is to make them as seamless as...

Company News

Introducing Uleska: The Future of DevSecOps Automation Tools

Few industries have seen such scrutiny and shifts in recent years as cybersecurity. In an increasingly connected world, speed and agility throughout...