At Uleska, we focus on moving security testing away from experts running manual tests and move it to automating security checks into existing processes. We believe in continually testing ourselves in this mission, which is why we tasked a group of people without software or security skills to test 1000 projects and feed the results back to the project teams.
Here’s why we did it, how we did it and what we learned.
Firstly, many of our customers have security teams or developers who are interested in security that work with our product. We want to push that boundary so anyone can undertake a security test. There’s also a trend in many teams for security testing to be more transparent to developers, yet still produce meaningful results.
We want to give something back to the open-source community by alerting them to any issues they may need to look at. Plus, as many security tools have UI/UX aimed at security professionals, we want to test our UI for the simplest ease-of-use we can find.
To begin our experiment, we chose 1,000 Javascript projects on GitHub that were recently updated and had between 50 and 200 stars in rating. Next, we went to UpWork.com and paid non-professionals £5/hour to use the Uleska Platform to automatically run three Javascript source code security testing tools. We consolidated the results generated by the platform to the project teams and made sure they didn’t have security or programming skills.
We didn’t set up any of our cyber value-at-risk enumerations, as they would apply to how an open-source library is used within the end-user application (i.e. types of data processed, quantity of data, environment etc.).
We also made the decision not to triage the results, as the Uleska Platform has functionality to automatically remove false positives and duplicates, a feature heavily used by our customers.
Since this experiment was for open source projects, instead of enterprise software teams, we wanted to see how difficult (or easy) reports containing a mix of real and false-positive issues were for open source teams.
Some of the vulnerabilities found, split by the users and applications (projects).
The experiment was a great success, providing a variety of valuable insights, including some improvements we can make to our DevSecOps orchestration usage. Some of our main learnings were:
In the next stage of this experiment, we will look to automatically remove the false positives.
During this experiment, we ran into some logistical challenges. Firstly, we created new GitHub accounts to find/extract the GitHub URL for the project to pass that into the Uleska Platform so the codeline can be tested.
These new GitHub accounts were also used to update the projects with the report of security issues. This meant these GitHub accounts were now creating GitHub projects or code which were later flagged by GitHub. This meant they were no longer able to submit issue reports to projects. For this reason, we stopped short of the full 1,000 projects, ending at around the 730 project mark.
We also had a few projects react negatively to being passed security reports out of the blue. Sometimes this was because the false positives weren't removed, other times our reports were perceived as spam. Now we understand where we need to improve, in our next iteration of this experiment, we’re looking to remove the likely false positives using the Uleska technology and look forward to helping more open source projects stay secure.
To discover more about the challenges of automating DevSecOps and Uleska can help overcome them, check out our playbook.
The problem with DevSecOps is incorporating many layers of security tasks into the fast-paced software development cycle. Thankfully, there are a variety of things you can do to overcome the challenges faced. In our playbook, we cover the top 10 challenges of automating DevSecOps, while also delivering actionable advice on how to overcome them.
You may unsubscribe at any time using the unsubscribe link in the newsletter.
Security tools are an essential part of software development today, especially with the ever-increasing number of attacks we see every year....
Security teams frequently struggle with the volume of alerts and issues they are tasked with daily. On average, most enterprises receive between...
Software development has evolved into an incredibly complex machine, with several moving parts to keep track of. Teams get more extensive, and...
Application Security is a constantly evolving industry, with new threats and methods to combat them appearing regularly. One of the more recent...
The application security (AppSec) industry moves fast. Development, security and operations (DevSecOps) practitioners are having to find creative...
We know starting your application security (AppSec) journey can be a little overwhelming. After all, choosing your tools from scratch and setting...
What is Application Security? Application Security is defined by developing, adding, and testing security features in an application or website....
Did you know that over 79% of developers surveyed in 2020 stated their applications had 20 or more vulnerabilities on average? As the digital world...
No system is perfectly secure, as proven by software analysis firm CAST, which reviewed 278 million lines of code and discovered more than 1.3...
There are thousands of amazing AppSec tools out there, but this can be both a blessing and a curse. While the headway and innovation we are seeing...
Building robust application security is a lot like building a house—you want it done thoroughly, without any missing parts. However, there is a...
Cybersecurity has been a rising concern in the last decade. In 2021, researchers have seen 50% more attacks per week on corporate networks compared...
With today’s fast development speeds, it’s hard to keep up with security practices for some organisations. This is especially true in the last few...
Open-source software has become a vital part of development in the last decade. However, utilising these components often comes with several caveats,...
The saying goes: “Many hands make light work.” Nowhere is this more apparent than in DevSecOps where developers and releases outnumber security...