Project post by Liz Rice, Isovalent, for the Cilium project
One of the benefits for CNCF projects is the funding of third-party security audits and testing. These help projects identify potential vulnerabilities in their code and process improvements to improve their security posture. Today, the Cilium project is publishing two reports: the Security Audit and the Fuzzing Audit.
From the executive summary of the Security Audit:
“The overall conclusion is that Cilium is a well-secured project. The audit found no critical vulnerabilities and found a lot of positives about the security of Cilium. This included both the code displaying positive security awareness as well as the maintainers having thorough understanding about the security posture of Cilium. “
Both these reports were produced by Ada Logics, and commissioned by the CNCF and the Open Source Technology Improvement Fund. Ada Logics have previously provided several other projects assessments and audits including Istio, CRI-O, and Argo.
This is an important milestone on Cilium’s path to CNCF Graduation.
Overview
Across the two reports the Ada Logics team provided the Cilium project with:
- Threat model formalization
- Manual code audit
- OSS-Fuzz tests
- SLSA compliance review
The audit and fuzz testing found a total of 30 issues, two of which were medium severity with the remainder low severity or informational. The reports also provided several recommendations for process improvements.
Security Audit
The security audit considered the main Cilium codebase, which includes code for the networking components of the Cilium project as well as the Hubble observability pieces. The Tetragon sub-project was not included as it’s still considered beta-level and changing rapidly.
Ada Logics created a threat model, which pointed out the importance of careful security review particularly in the policy enforcement code. Users rely on Cilium to correctly enforce security policies, so any bugs here would be particularly problematic! Fortunately, the review did not uncover any issues in this area.
This is helped by the fact that policy enforcement (as well as connectivity and observability) is performed by Cilium eBPF programs. The eBPF verifier analyzes these programs as they are loaded into the kernel, ensuring “a high standard of security by default with respect to memory corruption issues”. The verifier’s checks are an added layer of defense against the possibility of memory-related security vulnerabilities.
Manual review and additional fuzzing
The Ada Logics team carried out a manual review of the Cilium codebase, and also added ten new fuzzers. These activities identified 22 issues in total. The first medium severity issue was the absence of easy-to-find documentation on how to run Cilium securely, which we’re working on under this pull request. The second was a mutex that might not be unlocked in the event of a configuration error, which has been fixed here. Several of the low severity and informational issues have also been fixed, and we’re tracking the status in this issue.
SLSA review and Supply Chain Scorecard
As part of the Security Audit, Ada Logics reviewed the Cilium project against SLSA (Supply chain Levels for Software Artifacts). SLSA is “a security framework, a check-list of standards and controls to prevent tampering, improve integrity, and secure packages and infrastructure in your projects, businesses or enterprises. It’s how you get from safe enough to being as resilient as possible, at any link in the chain.” The SLSA specification is still at “alpha” level and is likely to change, but even this stage gives useful guidance for projects on improving their supply chain security posture.
The SLSA specification covers four different categories: Source, Build, Provenance, and Common, and has four levels for each category, with level 4 being the most secure. As you’ll see from the report, Cilium achieves level 4 for most criteria. The Provenance category isn’t yet addressed as we don’t yet generate a provenance report for releases (although we will have a Software Bill of Materials generated starting with release 1.13). We’re tracking SLSA improvements under a GitHub issue.
The audit also recommends that we add the OpenSSF’s Security Scorecard assessment, alongside the existing OpenSSF best practices and CLOMonitor badges that you’ll see on the Ciium README today. We’re tracking this here.
Fuzzing Audit
Ada Logics added 14 new fuzzers using the OSS-Fuzz framework to stress various Cilium components. This uncovered 8 issues in total, of which two have been fixed and two low-severity items are being tracked in GitHub issues. Another concerned the parser for MetalLB configuration that we’re planning to deprecate, and analysis by the Cilium security team determined that three are essentially impossible to hit in real-world usage of Cilium. Let’s take a moment to consider why that’s the case.
Fuzzing generates random inputs to functions to check whether they handle them correctly. This can be a powerful way to find bugs since the fuzzer generates inputs that human developers (and testers!) might not anticipate. However, in the real world inputs might be constrained by the component that creates those inputs.
For the three issues that we don’t believe can occur in the real world, the input comes from the Cilium agent and is parsed by the cilium monitor command, which can be run by a privileged user within the Cilium pod. Since the code for that command is fully under the control of the Cilium project, and is well tested, the likelihood is very low that it could generate the same malformed input as the fuzzer. Even if it did somehow happen, the consequence is that the cilium monitor command would crash, so the user wouldn’t get the information they wanted, but there would be no further impact on other components of Cilium.
This leads to a broader point noted in the Security Audit that the assumptions made about expected inputs aren’t always clear when reading the Cilium code. It’s a complex codebase, and we could do a better job within the code itself of documenting these assumptions so that future changes don’t break them.
Next steps
Securing a project is an ongoing process, and if you’d like to get involved in Cilium security we invite you to reach out on the Cilium Slack, or join our weekly developer meeting.
We’d like to thank the Ada Logics team for their work on the reports, and their helpful feedback that will go on to make Cilium even more safe to use. Cilium maintainer Joe Stringer also deserves gratitude for technical input and recommendations he made to support the audit process. Finally, our thanks to CNCF and OSTIF for commissioning these security projects.