Last December, as we were all counting down the days until 2020 was no more, FireEye posted an announcement that caused various beveragewares across the world to fall and shatter on the floor: some sort of state-sponsored attacker had stolen their red-teaming tools. Why would a state-sponsored attacker— especially one who had already clearly demonstrated they could break through a security firm like FireEye’s defenses (reminding us all that even security professionals are fallible)— need the tools FireEye themselves used to break through other’s defenses? Within a week, it became clear that this was just the tip of the iceberg.
These attackers, whose identities are yet to be confirmed but whose initials probably rhyme with SVR, had compromised SolarWinds Orion to deliver trojanized updates to clients all across the globe. As became clear in the weeks that followed, this attack, along with similar exploits of Microsoft and VMWare products, had exposed and compromised thousands of organizations, including NATO, the UK Government, the EU Parliament, the US Departments of Commerce, Defense, Homeland Security, Justice, State, Treasury (and more), Microsoft itself, and numerous other state and local governments and private sector entities. The true expanse of this attack is yet to be determined. The compromises went back at least 9 months, possibly longer, and represented a stunning breach of many high-profile targets through a supply chain vector — a documented growing area of concern by organizations including the Linux Foundation and the CNCF.
Preventing supply chain attacks is still a nascent, rapidly developing field. While the CNCF Security Technical Advisory Group (TAG) has documented instances of supply chain compromises going back at least to 2003, with the cadence of such attacks picking up substantially since about 2017, there is still little general consensus on how to prevent such attacks. This is why the Security TAG has published a new paper, Software Supply Chain Security Best Practices, designed to provide the cloud native and open source communities with a holistic approach to architecting a secure supply chain regardless of whether they are a software producer or consumer.
The approach outlined by the CNCF Security TAG group has four key elements:
First, establishing and verifying “trust” at every step in the process through a combination of code-signing, metadata, and cryptographic validation. Essentially, the goal is to establish a “we trust because we’ve verified” architecture for each point within the entire supply chain.
Second, “automation”: everything that can be automated should be automated and documented. This helps to prevent accidental errors and makes it easier to spot when things have gone awry.
The third element is “clarity.” This takes two forms. Every step in the software build and supply chain process should be clearly defined with a precise, limited scope. Next, deriving from this design, every actor within the supply chain (whether they be human or machine) must have a clearly defined role. This, in turn, allows us to limit the permissions of these actors to exactly those needed.
Finally, every entity in our system must engage in “mutual authentication.” This means that no human, software process, or machine should be trusted to be who they say they are, they must demonstrate through a hardened technique (such as MFA) that they are in fact who they purport to be. The mutuality of this (developer demonstrates their identity to server, server simultaneously demonstrates their identity to developer) helps to prevent exploits such as Man-in-the-Middle or replay attacks.
These principles can be operationalized across first party source code repositories, third party dependencies, build pipelines, artefact repositories, and deployments. Each of these stages involves different considerations and requirements, and comes with its own set of best practices. As a companion, we are also providing an “adoption framework” in this blog post that can be used to assess your own supply chain architecture and help identify the areas where you may need to place the most emphasis.
This paper fits within the broader work of the CNCF Security TAG group. Last year, the group published a paper on Cloud Native Security, emphasizing the importance of automation, shifting security left, and putting security issues on the same footing as other software “bugs” or development priorities. The CNCF Security TAG is also working on:
- a landscape map of cloud native security products and projects to make it easier for teams to find and adopt the tools they need,
- a Lexicon of security terms so teams can better understand the importance of security issues, and a catalog of “gaps” in the security ecosystem so the industry as a whole can improve its coverage of security needs,
- and additional papers and efforts to detail security considerations for specific areas within the ecosystem.
Within that body of work, this paper continues the emphasis on automation and treating security as a first order concern in the software development process. We outline many of the tools available for supply chain security. Indeed, one of the threads running through this paper is that while developing a secure “software factory” is a significant undertaking, most of the pieces to do so are readily available for teams willing to take up the challenge. Our hope is that by elevating the priority of security within software producing organizations, we can establish a clearer baseline that enhances the security of us all. This is especially true of supply chain security concerns, which, as we have seen, can have a wide ranging impact not just on the breached organization itself but on their downstream and indirect customers, too.
A Framework for Supply Chain Evaluation
So how does your supply chain stack up? Here are some questions to ask yourself:
Verify Source Code
- Do you require signed commits?
- Do you use git hooks or automated scans to prevent committing secrets to source control?
- Have you defined an unacceptable risk level for vulnerabilities? For example: no code may be committed that includes Critical or High vulns
- Do you use automated scanning to detect and prevent security issues, vulnerable dependencies, etc. from being committed to the repo that are not in compliance with your defined risk threshold?
- Have you defined clear contributors roles? Are they documented and discoverable?
- Do you enforce review and approval of contributions prior to merging?
- Are branch protection rules in place?
- Do you enforce MFA and SSH keys for human-entities? Do you have a plan in place for rotating SSH keys at regular intervals or following a key leak?
- Do you limit the access of automation agents (like CI/CD pipelines) following the principles of least privilege and just-in-time?
Verify Materials
- Do you verify that dependencies meet your minimum thresholds for quality and reliability?
- Do you automatically scan dependencies for security issues and license compliance?
- Do you automatically perform Software Composition Analysis on dependencies when they are downloaded/installed?
- Do you monitor dependencies for updates and security issues?
- Do you build dependencies yourself instead of relying on public package managers?
- Do you create an SBOM of your own artefacts?
Protecting Build Pipelines
- Do you use hardened, minimal containers as the foundation for your build workers?
- Do you maintain your build and test pipelines as Infrastructure-as-Code?
- Do you automate every step in your build pipeline outside of code reviews and final sign-offs?
- Do you sign the output of every step in your build pipeline to provide a verifiable guarantee?
- Do you validate the signatures and checksums of all dependencies before ingesting them in a build stage?
- Do you use separate build workers/containers for each step in your build pipeline?
- Do your pipeline orchestrator pass in the environments, tools, and commands allowed on each build worker?
- Do you network isolate your build workers and pipeline as much as possible?
- Do you produce verifiable, reproducible builds?
Protecting Artefacts and Deployments
- Is every artefact your produce (including metadata and intermediate artefacts) signed?
- Do you distribute metadata in a way that can be verified by downstream consumers of your products?
- Can your downstream consumers verify/validate any artefact they ingest from you before they use/deploy it?