Posted on April 18, 2023

CNCF projects highlighted in this post

Cilium logo containerd logo Crossplane logo Dapr logo Envoy logo etcd logo Flux logo Helm logo Istio logo Knative logo Linkerd logo

By Chris Aniszczyk, Adam Korczynski, David Korczynski

Introduction

In this blog post we will present an overview of the state of fuzzing CNCF projects.

We published a blog post on this in June 2022 titled Improving Security by Fuzzing the CNCF landscape and in this blog post we will highlight the efforts we’ve made since then and present our plans for the future. Our efforts in the recent year has been closely related to the goals we set out then. As such, this blog takes a retrospective look on the goals we set out and the efforts made to achieve them.

Fuzzing is one of several steps that CNCF is undertaking to secure the CNCF landscape. The fuzzing efforts are orthogonal to our efforts with respect to security auditing engagements. Fuzzing is one of the techniques software developers should use to ensure security and reliability of their software. Fuzzing is a proven technique for finding softwar bugs and the benefits of fuzzing is that the technique is closely-related to the developer’s workflow as it’s similar to testing. Fuzzing also has the benefit from a large community continuously improving and extending the infrastructure around fuzzing. This is also why CNCF has focused on integrating projects into the OSS-Fuzz fuzzing infrastructure, which is a free service for continuously analysing projects by way of fuzzing.

Improving the state of CNCF fuzzing in 2022 and 2023

In the CNCF blog post Improving Security by Fuzzing the CNCF landscape from June 2022 we outlined two major goals for the future:

  1. Extend the existing set ups to include more fuzzers and also Integrate more projects into OSS-Fuzz
  2. Improve the sustainability of the fuzzing efforts by increasing maintainer involvement.

In the following, we will highlight more explicitly the efforts made to achieve these goals:


Goal 1: Extend the existing set ups to include more fuzzers and also Integrate more projects into OSS-Fuzz

In respect to the first goal, since June 2022 we’ve added roughly 100 new fuzzers to the fuzzing suites of various CNCF projects and we’ve integrated fuzzing into Cilium, Cluster-API, Crossplane, Notary, Knative and Dapr. Meanwhile, we’ve also extended the fuzzing suites of many of the existing CNCF projects. This means, as of this writing, the following CNCF projects, listed in alphabetical order, are all integrated into OSS-Fuzz:

Goal 2: Improve the sustainability of the fuzzing efforts by increasing maintainer involvement and education

In respect to the second goal, we’ve taken several efforts. This goal is more qualitative in comparison to the first goal since each project has a unique set of preferences as to how to manage their software development. The efforts we’ve taken includes steps such as:

Go native fuzzing: Writing fuzzers for the projects using go-native fuzzing (https://go.dev/security/fuzz/) and ensuring they can run on OSS-Fuzz. Go-native fuzzing makes it much easier for developers to create fuzz drivers as opposed to the state before go-native fuzzing. OSS-Fuzz supports go-native fuzzing, which means that the developers can write the fuzzers in an intuitive development environment and still expect them to run continuously in the cloud. This has had a positive impact in terms of developers intuitively managing the fuzzers. In fact, the CNCF projects currently have around 110 go native fuzzers.

Engaging the community: A central part in making fuzzing a first-class citizen in the development of CNCF projects is to promote knowledge-sharing and sharing experiences in the public domain. SInce June 2022 there has been several public talks and alike promoting the value of fuzzing for CNCF projects, including:

Working closely with maintainers: It’s a key part of our goal to involve maintainers into the fuzzing process. One of the actions we’ve taken is to migrate fuzzers from our dedicated fuzzing repository to upstream repositories following the desired set up of maintainers. We’ve also kept close communication over the maintainers’ preferred communication channels to assist as much as we could in fuzzing-related tasks.

Assist in related, but non-fuzzing, tasks: Assist in related, albeit non-fuzzing, tasks. For example, when the fuzzers found a security vulnerability  in Crossplane it was important to the maintainers that a procedure was established for reporting security vulnerabilities. This led the Crossplane maintainers to implement a security policy that will be used for future cases of security reports https://github.com/crossplane/crossplane/security/policy. The Crossplane security team had a chance to test the new security policy on the issues found by the fuzzers.

Fuzzing booth at KubeCon

CNCF had arranged for a booth in the project area of KubeCon North America 2022 to be specifically focused on the CNCF fuzzing efforts. The goal behind this booth was to speak to maintainers about fuzzing, answer questions and in general to promote the value of fuzzing. Throughout the whole conference there were two fuzzing experts at the booth and would answer questions from the visitors, show demos and engage with maintainers.

Results from fuzzing CNCF projects

In this section we will present insights and details about the results, in terms of issues, from the fuzzing efforts.

High-level perspective from issues

From a high-level perspective, we can get an understanding of the results from fuzzing by querying the OSS-Fuzz bug tracker. This bug tracker will report all issues found by OSS-Fuzz, which includes issues that may not be security-relevant, e.g. false positives due to erroneous fuzzers, reliability bugs over security bugs and also bugs that are due to over-approximations in the fuzzers.

The following figure shows the results in terms of bugs reported and handled by OSS-Fuzz for the CNCF projects integrated into OSS-Fuzz. The Verified tag means issues that were reported by OSS-Fuzz and OSS-Fuzz later notified the issue has been fixed. This data was found by searching Monorail for all the CNCF projects, which can be replicated using this query.

A total number of 200 issues have been reported and verified up until March 2023, and the count went from approximately 1200 to 2000 between February 2022 and March 2023.

Line graph showing issues over time from Mar 2022 to Mar 2023 filtered on accepted, new, won't fix and verified

Some of the larger projects and memory unsafe projects account for a major proportion of the issues above. Removing the projects Envoy, Fluent Bit and Istio from the query the results we get are as follows: 

Line graph showing issues over time from Mar 2022 to Mar 2023 filtered on fixed, new, won't fix, and verified

The number of issues reported and then verified went from around 150 to 500 between February 2022 and March 2023. The above data can be found using this query.


The macro-level overview of the bugs gives an understanding about the volume. It’s important to acknowledge that for almost all of these issues there will be some level of human-effort involved for triaging the case. This would be impossible to do without the support from the maintainers of the many projects.

Detailed inspection of the issues

The macro-level overview given above misses a bit of insights with respect to what the bugs actually are. To give a better understanding about the issues on a detailed level and how they manifest themselves in code, we have picked a set of issues and provided a brief technical description from a code-level perspective.

Helm

Helms fuzzing audit (report) found several security and reliability issues, including 4 issues that have been assigned CVEs. 

Helm, CVE-2022-36055, Denial of service through string value parsing (1)

This is the first of two vulnerabilities found in Helm’s strvals parser. The parser takes a single string as its parameter and returns a map[string]interface{}. One use case of the strvals parser is when a user sets the –set or –set-string flags with the Helm client. A fuzzer was able to create a string that would crash Helm with a non-recoverable out-of-memory panic when the string was passed to Helms strvals Parser. 

Github advisory: https://github.com/helm/helm/security/advisories/GHSA-7hfp-qfw3-5jxh

Helm, CVE-2022-23524, Denial of service through string value parsing (2)

This is the second vulnerability that Helm’s strvals parser fuzzer found. Similar to the first vulnerability, the fuzzer was able to create a string that could crash Helm with an unrecoverable out-of-memory panic. 

Github Advisory: https://github.com/helm/helm/security/advisories/GHSA-6rx9-889q-vv2r 

Helm, CVE-2022-23526, Denial of service through schema file

Helms chartutil package implements functionality that loads a json schema validation file. When Helm then renders a chart, it will validate the fields of the cart against the loaded schema file. A fuzzer was able to create a schema file that would crash Helm when loaded with Helms chartutil schema file loader. The crash is un-recoverable and since the API is part of the Helm SDK, which must accept untrusted input, the issue was deemed a security vulnerability.

Github advisory: https://github.com/helm/helm/security/advisories/GHSA-67fx-wx78-jx33

Helm, CVE-2022-23525, Denial of service through through repository index file

A helm chart repository is an HTTP server that contains package charts and an index file, index.yaml which is an index of all charts that exists in the repository. The Helm SDK has an API that parses a repository index file for easier managing the values in it. One of Helm’s fuzzers was able to create a malicious index.yaml file that would crash Helm with an unrecoverable panic.

Github advisory: https://github.com/helm/helm/security/advisories/GHSA-53c4-hhmh-vw5q 

Notary

Notary’s fuzzing audit found a vulnerability in https://github.com/notaryproject/notation-go which is a collection of libraries based on the Notary v2 standard to support signing, verifying, pushing and pulling of OCI artifacts. 

The vulnerability was found in notation-go’s signature verification API, https://github.com/notaryproject/notation-go.Verify(). The fuzzer was able to craft a malicious trustpolicy that could cause a dependency of notation-go to allocate excessive memory resulting in Denial of Service from an unrecoverable out-of-memory panic. This issue was assigned CVE-2023-25656.

Github advisory: https://github.com/notaryproject/notation-go/security/advisories/GHSA-87×9-7grx-m28v 

containerd

containerd implements an API for importing OCI images and fuzzing found a vulnerability in containerd’s OCI image importer, where maliciously-crafted files in the image would exhaust memory on the machine causing Denial-of-Service. This issue was assigned CVE-2023-25153.

Github advisory: https://github.com/containerd/containerd/security/advisories/GHSA-259w-8hf6-59c2 

Crossplane

Fuzzing found a vulnerability in Crossplane-runtime that allowed a user to directly manage the memory Crossplane-runtime would allocate. The Crossplane-runtime project is a collection of libraries to write Crossplane controllers. The vulnerability would happen when a user could control the value of SetValue() of a Paved object. The vulnerable code was used by the Crossplane project, and as a result both projects were vulnerable from the same root cause, and 2 CVE’s were assigned CVE-2023-28494 and CVE-2023-27483:

Github advisories:

  1. https://github.com/crossplane/crossplane/security/advisories/GHSA-v829-x6hh-cqfq
  2. https://github.com/crossplane/crossplane-runtime/security/advisories/GHSA-vfvj-3m3g-m532

FluxCD

Fuzzing found a high-severity vulnerability in FluxCD’s flux2 project, https://github.com/fluxcd/flux2. The root cause of the vulnerability was one of Helms vulnerabilities that were also found by fuzzing, and the Helm and Flux teams worked together on triaging and mitigating the issue. This issue was found by a fuzzer for Flux’s helm controller which had been running continuously for approximately 6 months before finding the crash which speaks to the importance of continuity in fuzzing. 

The vulnerability was assigned CVE-2022-36049.

Github advisory: https://github.com/fluxcd/flux2/security/advisories/GHSA-p2g7-xwvr-rrw3.

Future work and outlook

We look forward to continuing to support CNCF projects by way of fuzzing. In general, we  maintain the two goals from the previous year:

  1. Extend the existing set ups to include more fuzzers and also Integrate more projects into OSS-Fuzz.
  2. Improve the sustainability of the fuzzing efforts by increasing maintainer involvement.


There is still more work to be done in each of these goals and we expect the process to have many similarities as the last 12 months.

In addition to this, we have added a new set of goals:

      3. Improve the use of custom sanitizers. We have a lot of data in terms of code coverage and we have a lot of inputs to trigger code paths. An option is to check for additional security invariants that we can check for.

      4. Add introspection capabilities by way of Fuzz Introspector as a way of measuring how well the CNCF projects are fuzzed to identify potential missing gaps and opportunities.


We will have a Fuzzing booth at Kubecon North America again this year, and look forward to welcoming you!

If you want to get involved, please see the https://github.com/cncf/cncf-fuzzing/ repo.