Posted on March 2, 2023 by Adam Korczynski + Phil Estes

CNCF projects highlighted in this post

containerd logo Kubernetes logo

Community post by Adam Korczynski and Phil Estes

The containerd project is happy to announce the completion of a comprehensive fuzzing audit which added 28 fuzzers covering a wide range of container runtime functionality. During this audit a vulnerability was uncovered in the OCI image importer. The audit is a part of a larger initiative by the CNCF to improve the security posture of the cloud-native landscape by way of fuzzing. The audit was carried out by Ada Logics over the course of 2021 and 2022. The Ada Logics team is thankful for the opportunity to help improve containerd’s security posture and were impressed with the low count of issues found given the large number of fuzzers created. This is a testament to the well-written and well-maintained codebase of the containerd project. 

The full results of the containerd audit are available here: https://containerd.io/img/ADA-fuzzing-audit-21-22.pdf

containerd

At a high level, a container runtime is a lifecycle manager that can create, run, and manage the state of a containerized process on an operating system which provides container isolation primitives. Container runtimes are generally split into two categories:

  1. Low-level container runtimes
  2. High-level container runtimes

Generally speaking, low level container runtimes are responsible for abstracting the Linux container primitives, whereas high-level container runtimes are responsible for container image management and managing state and lifecycle operations in concert with a low-level runtime.

containerd is a high-level container runtime with an emphasis on simplicity, robustness and portability. It was started at Docker in 2014 to manage the low-level runtime for the Docker engine. For most use cases, the low-level engine is the OCI runc implementation, but the containerd-shim offers an easy way to replace runc with other low-level isolation technologies, like micro VMs. containerd was donated to the CNCF in 2017 and was well received by the community; Over the following two years, the number of companies that contributed to the project grew from 53 to 136 – a 157% increase. containerd graduated from the CNCF in 2019.

containerd implements a long-lived daemon that listens for requests to start and stop containers and report their status. It also manages image transfer operations—such as pulling and pushing images to and from an OCI-compliant container registry—and handles metadata operations for resources like tasks, labels, annotations, among others. containerd also implements the Kubernetes Container Runtime Interface (CRI) and is widely used as the underlying runtime in Kubernetes deployments.

Fuzzing

Fuzzing is a technique for testing software for bugs. To test your software by way of fuzzing, you need two things:

  1. A fuzzing engine
  2. A test harness

The fuzzing engine is responsible for generating a testcase that it makes available in the test harness. It is the test harness’s job to pass the testcase onto the API that the harness is meant to test. containerd is implemented in Go, and its initial fuzzers utilised the go-fuzz engine, which at the time was the most mature engine. During the audit, Go released its native fuzzing engine and the community started writing new fuzzers using this engine as well as migrate go-fuzz harnesses to the native Go fuzzing engine. As a result, containerd’s fuzzers are implemented both as go-fuzz harnesses and native Go harnesses. 

Fuzzing is a valuable technique in finding software bugs. In Go, this includes bugs such as nil-dereference, out-of-bounds panics, out-of-range panics, out-of-memory panics as well as logic bugs.

Fuzzing containerd

The audit progressed along two stages: In the first stage, the auditors wrote fuzzers to increase test coverage of the containerd code base. At this stage they added the fuzzers to the CNCF’s dedicated fuzzing repository: https://github.com/cncf/cncf-fuzzing. The auditors wrote 28 fuzzers covering a wide range of containerd functionality including:

In the second stage, the containerd maintainers migrated the fuzzers from cncf-fuzzing upstream to https://github.com/containerd/containerd, where they are located now. During the entire audit, OSS-Fuzz ran the fuzzers continuously against the latest master branch of containerd. OSS-Fuzz is a service offered by Google for critical open source projects that runs their fuzzers with excessive hardware resources allowing the fuzzers to run on thousands of cores. At the end of the audit, OSS-Fuzz had run fuzzers for almost 40,000 hours collectively. 

In addition, the fuzzers run in the CI on pull requests with the corpus from OSS-Fuzz which allows them to test new PRs with corpus accumulated by hundreds or thousands of hours of fuzzing. Fuzzing during CI is an important step to catch bugs before they are merged into the code base. 

An important utility in containerd’s fuzzing suite is go-fuzz-headers which is used to transform the testcase provided by the fuzzing engine into structured data. This is done in containerd’s CRI fuzzers, where go-fuzz-headers is used to create the requests to containerd’s CRI endpoints. For example, consider the following program iteration, where testcase represents the testcase provided by the fuzzer. In this program, the fuzzer randomizes the CreateContainerRequest.

package main

import (
	"fmt"

	fuzz "github.com/AdaLogics/go-fuzz-headers"
	runtime "k8s.io/cri-api/pkg/apis/runtime/v1"
)

func main() {
	testcase := []byte{0, 0, 0, 13, 112, 111, 100, 115, 97, 110, 100, 98, 111, 120, 49, 50, 51}
	ff := fuzz.NewConsumer(testcase)
	r := &runtime.CreateContainerRequest{}
	ff.GenerateStruct(r)
	fmt.Printf("%+v\n", r)
}

On the last line we print the request, which now has the PodSandboxId value: podsandbox123:

&CreateContainerRequest{PodSandboxId:podsandbox123,Config:&ContainerConfig{Metadata:&ContainerMetadata{Name:,Attempt:0,},Image:nil,Command:[],Args:[],WorkingDir:,Envs:[]*KeyValue{},Mounts:[]*Mount{},Devices:[]*Device{},Labels:map[string]string{},Annotations:map[string]string{},LogPath:,Stdin:false,StdinOnce:false,Tty:false,Linux:nil,Windows:nil,},SandboxConfig:nil,}

Over time, the fuzzer will generate test cases that also randomise the Config and the SandboxConfig fields along with the values in these structs. In containerd’s cri fuzzers, each request is randomized in a similar way, for example:

func createContainerFuzz(c server.CRIService, f *fuzz.ConsumeFuzzer) error {
	r := &runtime.CreateContainerRequest{}
	err := f.GenerateStruct(r)
	if err != nil {
		return err
	}
	_, _ = c.CreateContainer(context.Background(), r)
	...
}

Found issues

The fuzzers found four unique issues during the audit. Three of those were in containerd itself, and one was in a 3rd-party dependency. At the completion of the audit, all issues have been fixed in the upstream project. Most notably is the issue found in OCI image import handling which could result in Denial-of-Service of the node if a maliciously-crafted image was imported by the victim. The issue was assigned CVE-2023-25153, and containerd has fixed the issue in containerd 1.5.18 and 1.6.18. The issue is detailed in https://github.com/containerd/containerd/security/advisories/GHSA-259w-8hf6-59c2.

In summary, finding only four issues from 28 fuzzers is impressively low, and is a testament to a well-written and well-maintained codebase.

Thanks

Thanks to the containerd maintainers and community for their help over the last few years in implementing and improving the fuzzing implementation in the containerd project.

Specifically we would like to thank:

As well as the rest of the containerd maintainer team for their help with reviews, merging PRs, and helping with refactoring of the fuzzing implementation as different pieces were moved between projects.

If you want to get involved in the containerd project please check out the #containerd-dev channel on the CNCF Slack, or join the twice monthly project call listed on the CNCF calendar.