By Chris Aniszczyk (CNCF), Adam Korczynski (Ada Logics), David Korczynski (Ada Logics)
CNCF maintains a high level of security for its projects by way of a series of initiatives such as security auditing, supply-chain assessments and security automation work. In this blogpost we will go over CNCF’s fuzzing initiative and its impact in 2024
Fuzzing a technique for finding security and reliability bugs by way of executing vast amounts of arbitrary inputs against a given API or codebase. We have previously written about the impact of fuzzing the CNCF project landscape and a detailed update of CNCF Fuzzing’s impact in 2023. Furthermore, CNCF Fuzzing has published an in-depth fuzzing handbook that goes into detail with what fuzzing is and how to apply it.
Managing fuzzing for a project can be a tedious and comprehensive task. The more fuzz harnesses a project has, the more time or hardware resources it needs to run those fuzz tests. The fuzzers will build up corpus that the project needs to manage over time in a dynamic manner, as this is likely to change every time a fuzz test runs. Finally, the fuzzer might find crashes in its target source which the project team needs to assess and potentially address.
We build the CNCFs fuzzing efforts around OSS-Fuzz which is a continuous fuzzing service run by way of a Github project here, run by Google. Critical open source projects can integrate into OSS-Fuzz and instruct OSS-Fuzz to build and run their fuzz tests. In doing so, they get two high level benefits: 1) OSS-Fuzz runs their fuzz tests at scale with thousands of CPU cores which allows the fuzz tests to explore the code they test faster than if a developer runs them one at a time on their local development machine. 2) OSS-Fuzz manages the fuzzing workflow in the background in a fully automated manner from running the fuzz tests to optimizing the accumulated corpus (through a process called “pruning”) to notifying maintainers about findings. The workflow of running fuzz tests can be complicated and grows in complexity with more fuzz tests.
A lot of CNCF projects run fuzzing suites by way of OSS-Fuzz, and each year more and more CNCF projects integrate into OSS-Fuzz. To give an estimate of this, the below timeline shows the CNCF projects currently integrated into OSS-Fuzz and when they did the CNCF integration.
CNCF Fuzzing Audits
CNCF projects undergo dedicated fuzzing audits in different stages of their graduation process ranging from the sandbox stage to post-graduation. During a CNCF fuzzing audit, the auditors assess a given project’s existing fuzzing efforts and identify areas that need improvements. The auditing team then implements the improvements.
In 2024, we completed three fuzzing audits with a fourth being underway. The first audit was for Lima, the CNCF command line tool for launching virtual machines. During this audit, Lima integrated into OSS-Fuzz and saw 8 fuzz tests added to its fuzzing suite. The fuzzers found a number of bugs in third-party parsing libraries and continued to find crashes after the audit finished. The report for the Lima fuzzing audit can be found here. The second audit was for Keycloak, a project widely used in production for managing identity and access for applications and services. The audit resulted in Keycloak integrating into OSS-Fuzz with 24 fuzzers that target widely different types of APIs in Keycloak from parsing routines to targets requiring structured randomization and extensive mocking. During this audit, the fuzzers found a single issue, however, after the audit had completed and the fuzzers had run for a longer time, one of them found a high-severity security vulnerability in a regular expression that could lead to denial of services by exhausting system resources. The report for Keycloaks fuzzing audit is available here. The third audit was of the Opentelemetry Collector which the auditors integrated into OSS-Fuzz and has 49 fuzz tests added for its core repository and its collector-contrib repository. The audit for Opentelemetrys fuzzing audit is available here.
Findings
In this section we take a look at the issues found in CNCF projects from their fuzz tests running on OSS-Fuzz. We have extracted the data from OSS-Fuzz’s bug tracker at https://issues.oss-fuzz.com/ where we have searched by project name and extracted all issues reported by OSS-Fuzz. We removed issues related to failing builds which refer to cases where OSS-Fuzz cannot successfully build all the fuzzers in a project’s build script. The issues reported by OSS-Fuzz can include false positives from flakes or other untrue bugs in the context of the threat model of a given project. We estimate the subset of false positives in the sense of not being security or reliability bugs to be around 20% of the dataset.
Below we show 2 sets of data that each consists of three graphs. In the first set, the first graph contains bugs found in all CNCF throughout 2024. The second graph in the first set contains bugs found in the two memory-unsafe CNCF projects running on OSS-Fuzz throughout 2024. The third graph in the first set contains bugs found in all the memory-safe CNCF projects throughout 2024.
The second set consists of three similar graphs that cover 2019 to December 2024.
In the below graphs, the “New” graph shows the number of currently open issues in OSS-Fuzz and the “Verified” graphs show the number of issues that have been verified to no longer reproduce their given bug, i.e. they show the number of fixed issues across the data set.
Issues found throughout 2024
Bugs found in all CNCF projects throughout 2024
Throughout 2024, OSS-Fuzz found bugs in CNCF projects at a steady rate of 10 bugs per month.
Memory unsafe
Like in previous years, memory-unsafe languages continue to take the top of the charts of projects with most issues reported by OSS-Fuzz. The following graph shows OSS-Fuzz issues in the CNCF memory unsafe projects integrated into OSS-Fuzz, showing that around 50 issues were fixed throughout 2024 with some open issues left to be fixed. These projects claim half of all issues found in CNCF projects, which is proportionately higher from a bug density perspective since we have fewer memory unsafe projects in comparison to memory safe projects.
Memory-safe projects
The number of issues found in memory-safe languages was not trivial. In total across CNCF’s projects, OSS-Fuzz reported just over 60 issues throughout 2024 with the majority being found in the second half.
Issues of all time
In the above we showed issues from 2024. In this section we will go through issues throughout last five years
All CNCF projects
Over the last five years, OSS-Fuzz has reported over 2000 verified issues in CNCF projects. The finding of new issues has slowed down from early 2023 to the end of 2024 whereas 2022 was a very productive year for discovering new issues in CNCF projects.
Memory unsafe
Memory unsafe projects have been the top projects for finding issues by way of fuzzing throughout the last 5 years. The finding of new issues has been slower since early 2022. Nonetheless, these two projects benefit from fuzzing many years after they first integrated into OSS-Fuzz.
Memory-safe projects
The following graph shows the issues OSS-Fuzz has reported in memory-safe CNCF projects.. Fuzzing continues to find issues in memory-safe CNCF projects with almost 800 total issues found. We can see from the graph from when the majority of memory-safe projects were integrated at the end of 2021. From that time until early 2023, fuzzing found around 400 issues, and since then the development has slowed down. This makes a lot of sense since we expect the number of issues to be found by fuzzing to decrease per project, as each project is being fuzzed.
Selected security vulnerabilities found from CNCF’s fuzzers in 2024
We saw on multiple occasions in 2024 that different CNCF projects’ fuzz harnesses found security vulnerabilities in their respective projects after we completed their fuzzing security audits. This demonstrates the value of building the CNCF’s ecosystems fuzzing efforts around OSS-Fuzz in that the fuzzers can run continuously, including after an active engagement. Below we highlight two cases where the fuzzers found security vulnerabilities after their fuzzing audit completions. The first is Cert-Manager that had a set of fuzz tests set up during its security audit. The second is Keycloak where one of its fuzzers from its fuzzing audit found a security vulnerability right after we completed the audit.
Cert-Manager: Potential slowdown / DoS when parsing specially crafted PEM inputs
A fuzzer found a vulnerability in Cert-Manager that allowed an attacker to slow down Cert-Manager when processing PEM data. An attacker would need to cause Cert-Manager to process their maliciously crafted PEM data which in turn would cause Cert-Manager to significantly slow down. This could prevent other users in the cluster from using Cert-Manager, and as a result, an attacker could cause denial for other users if they would continuously send a limited number of requests that would slow down Cert-Manager.
Fuzzing is particularly well suited to find such vulnerabilities given that it had its root cause in a complex parsing routine. Interestingly, OSS-Fuzz did not report this as a single point of failure that pointed to a specific line of code. The reason for this is that the vulnerability was not found by a panic on a specific line of code. Instead, OSS-Fuzz detected that a specific testcase was able to reliably take longer than 60 seconds to process in the OSS-Fuzz runtime environment. As such, by monitoring the execution time, OSS-Fuzz was able to report this as a true positive finding.
GHSA: https://github.com/cert-manager/cert-manager/security/advisories/GHSA-r4pg-vg54-wxx4
Keycloak: REDoS from malicious search query
Right after we finished Keycloaks fuzzing audit, one of Keycloak’s fuzzers found a REDoS in the SearchQueryUtils class. The vulnerability could allow a malicious threat actor who could invoke the vulnerable method with a maliciously crafted string to exhaust system resources and deny other users from using any of the system resources.
The vulnerability is of high severity with a CVSS score of 7.1 and can be seen in detail here: https://github.com/advisories/GHSA-wq8x-cg39-8mrr.
Conclusions
Fuzzing continues to positively impact the CNCF landscape’s security posture by finding new code-level issues of both reliability and security relevance. As of the end of 2024, OSS-Fuzz has run CNCF fuzzing harnesses for tens of thousands of CPU hours and reported more than than diverse issues 2,000 crashes. The results are enabled by the state-of-the-art fuzzing engines and techniques from the fuzzing community that has built and improved the fuzzing techniques for years, and a lot of hand on work writing fuzzing harnesses for the relevant projects. OSS-Fuzz is at the center of open source fuzzing, and it has had a high impact on the CNCF landscape until now by finding thousands of crashes. At the same time, new initiatives are being developed with the goal of improving open-source fuzzing results. For example, OSS-Fuzz is working on LLM-powered capabilities to automate fuzzing harness construction at https://github.com/google/oss-fuzz-gen with the goal of making these available to integrated projects. CNCF projects will benefit from new ways of using fuzzing simply by being integrated into OSS-Fuzz; Improvements on OSS-Fuzz are often rolled out with little to no required changes in projects’ integration files. We are excited to see how fuzzing continues to impact CNCF projects through 2025.