Kubernetes Annual Report 2021

Published: June 1, 2022

This is a summary of the Kubernetes project’s contributor community and activities.

This report documents both quantitative measures of community health (project milestones and snapshot) as well as qualitative measures of the community as reported by community leaders and contributors to the project.

Authors: The Kubernetes Steering Committee
Editor: Tim Bannister, @sftim, SIG Docs Tech Lead

KCS Celebration EU 2022

Terminology

This report uses the following terminology:

  • Special Interest Group (SIG): a body of contributors, responsible on an ongoing basis for an area of work in the Kubernetes project. They own code, docs, and/or policy.
  • Working Group (WG): a body of contributors, responsible for an area of work in the project. Unlike SIGs, WGs dissolve once the scoped work is complete. Working groups are cross-functional efforts sponsored by a SIG.
  • Community Groups: all of our official groups of the upstream project. Special Interest Groups + Working Groups + Committees = community groups. For a full list, visit the Kubernetes Contributor Site at: https://k8s.dev/groups
  • Chair and/or Tech Lead: a contributor who organizes and leads a community group.
  • Contributor: an individual who creates an event in GitHub like a Pull Request, Issue, Review, or Comment
  • KEP: a Kubernetes Enhancement Proposal
  • OWNER: a GitHub user who reviews, approves, and/or merges commits and is listed in an OWNERS file. Maintainer is a good industry synonym.
  • Contributor Ladder: member, reviewer, approver, subproject owner.

For the community group mailing list, meeting times, and other contact info visit: https://k8s.dev/groups

For community groups governance:


Data collection

The Kubernetes Steering Committee sent out a survey to all community group leads to collect data for this report. Each individual group report may be found in their respective directory inside the Kubernetes Community repo.

For more, see: Program Documentation


Contributor snapshot

62000
contributors all time

10000
new contributors this year

1
new sig SIG K8s Infra, converted from WG

1
new working group

5
new chairs and tech leads

10 or less
unique reviewers in 8 groups

8.29
average active meeting participants in each group

~70000
slack members in SIG/WG rooms

Accolades

On behalf of the project, we’d like to say thanks to the following contributors, community groups, and ecosystem for the following highlights. As always, give praise to an effort in #shoutouts on Kubernetes slack.

Feature Maturity and Stability

Thanks to our groups for continuing the efforts from 2020, many SIGs continue to drive long standing beta features to graduate to stable.

Several features that graduated to stable or made notable progress include:

Other project processes are maturing, too, and not just the code. A new way to cast votes in elections (like Steering Committee and more) runs via Elekto. The Kubernetes Monthly Community meeting was rebooted to include discussions and not just presentations.

Showing up and sticking around

Climbing the contributor ladder is a trust-building exercise as much as it is a skills one. Sticking around, chopping wood, and carrying water is the main formula for growing OWNERs and leaders on the project.

An example of an intentional contributor ladder growth effort happened in SIG Docs by growing its contributor and reviewer base in 2021. They introduced a shadow program for PR Wrangling and dedicated more time to being active in the #sig-docs Slack channel, helping grow the community. SIG Docs also worked on a leadership transition strategy to bring community members into leadership roles via a specialized six-month group mentorship program. They were able to cultivate leaders for the SIG and some of its subgroups, adding new co-chairs and tech leads.

SIG CLI deserves another great shoutout for having long-standing Chairs and Tech Leads take the emeritus route while growing new leaders into the roles. Thanks for your service and great job, team!

Amping up Kubernetes security

Every group in Kubernetes has a responsibility to make sure we are putting our best foot forward with supply chain security. Accolades to all of SIG Release, SIG Auth, and SIG Security for their sustained efforts in this area that include:

Alongside those improvements specifically to supply-chain security, we’ve seen:

Things that no longer spark joy

There are plenty of processes, tools, and policy that are put together in a project lifecycle that eventually need to be phased out for whatever reason. A contributor painpoint that we’ve had with a codebase this large is bazel. The crews in SIG Testing and SIG Release put in a lot of time and attention on removing bazel from kubernetes/kubernetes. There are some pieces left in kubernetes/test-infra but needless to say, we are on the road to moving on in our build processes.

Growing Windows support

Thanks to the SIG Windows team and surrounding groups for their efforts in growing the support in this space! A true testament to the power of the ecosystem. They have more upcoming work to do and we are looking forward to seeing their growth in 2022 and beyond.

Details:

Themes / Trends

Prioritizing Quality

The project saw an increase in regression-related backports in the two most recent releases (1.22 and 1.23). Many of these regressions were related to a couple types of changes:

What have we done?

Adjustments are being made in several areas throughout the release cycle to reverse this trend:

Independent contributors play a critical role on the project

A misconception is that this project is just cloud providers maintaining it; however, one of our biggest contributor bases are “independent” that is, not affliated with an organization.

There is space for everyone here.

What have we done?

Connect folks to jobs! While not all indie contributors are looking for employment, many are. This year we worked with CNCF to add a feature to the [cncf.jobs.io site], which allows employers to indicate a percentage of time that they would support upstream activities. The Kubernetes project needs more contributors with employer-backed time, and this was a great step toward that goal. Aligning contributors with the right incentives is the sweet spot for lasting contributions.

Areas to research?

As part of upcoming surveys, we will poll the indpedenet contributors on various topics and how we can support them more. As always, we welcome feedback via SIG Contributor Experience or for high level governance matters, the Steering Committee.

Niche contributor documentation /help-wanted

With one of the largest decentralized distributed open-source projects out there, expect our contribution guides to be in-depth and extensive. k8s.dev/guide is our primary guide; no matter where you contribute to the project, you start there. But because the project is so large, some groups have other style guides, code review processes, and more that define how they do business and operationalize. This is an important part of our [values]. Same thing at big employers: everyone gets the standard onboarding docs, but your department might have an additional “here’s how to get work done” document floating around.

Many of our groups reported in that they have a hard time keeping this information up to date, if they even have this kind of documentation at all. This is a great way to get involved if you are new to a group! Want to become an OWNER? Set someone up for success behind you by creating documentation for your area.

What have we done?

In late 2020, SIG Leads were tasked with auditing their area specific documentation, with many removing out-dated information and creating follow-up items calling out things things that should be documented. These audits made it easy for companies to bring on Tech Writers to help shore up this needed documentation.

Additional processes have been put in place, such as a documentation review as part of the annual report process should ensure that project contributing docs remain (relatively) up-to-date.

Areas to research

Updating documentation is usually a good onboarding path for interns and new contributors but this can get murky with some of the complexities of the code and doc set. It can take up to 3 months to onboard on to the project before suggestion and submitting changes. Is there a program that SIGs could create as an onboarding path towards OWNERship here?

Burnout

The topics of burnout and workload management are frequent in our Leads and group meetings, Steering Committee, and even the growing voices at ecosystem level during talks and events at KubeCon/CloudNativeCons. This is an industry wide problem that we need to solve together. With a mix of reasons why contributors are burning out, there is no one “solve all” solution here. Aligning incentives to grow OWNERs seems to be one of the main challenges in this space.

What have we done

Growth Areas

This section represents an area of the project that we’ve identified as having a growth opportunity or need.

What’s project health anyway?

Some of the more mature groups like SIG Instrumentation or those with industry open-source veterans can quickly identify areas of their components that need help and tell stories about what’s flourishing. Yet, it can be challenging to establish universal indicators of “project health” in a project as large and diverse as Kubernetes. We need to develop these indicators to provide signal to the leads so that they may detect, pre-empt, or bubble up this information to keep their area healthy.

Every group needs more reviewers

If you’ve been watching open source news over the last year, supply chain security has made headlines. According to OpenSSF and other security groups, code reviews are an important piece to putting prioritization on security. Kubernetes strives for two reviews (a reviewer and approver) to match our values of quality and high-trust. Reviewers are a key part of our success in quality code and documentation changes upstream. Reviewer is the next step on our contributor ladder post Org Member; however, you don’t need to be in an OWNERs file as a Reviewer to review. Anyone can leave a comment or an “/lgtm” (looks good to me!). As we try and grow top level approvers, sticking around and getting into an OWNERs file as a Reviewer is crucial for our sustainability as a project to keep pace.

As a data point, in 1.19, the average daily PR reviewers were down -15% across Kubernetes Orgs and -24% in the Kubernetes/Kubernetes repo while still receiving the same amount of issues and PRs. If you are trying to get a change into Kubernetes, this will effect you.

The 9 to 5 contributor is almost over and we have to adjust

Only a handful of our OWNERs, some of our most active contributors, will tell you that they work 80-100% upstream. These folks know the codebase and docs extensively and are some of our most experienced reviewer eyes. But anecdotally, the number of experienced and very active core folks able to contribute has decreased in recent years. Ensuring continuity and growing more people into senior roles is becoming critical for the project to continue to deliver a robust and reliable releases.

In 2022 we have started discussion the CNCF Governing Board to see how we can tackle long term strategies together.

This reporting process and its summary

This process takes us 6 months. This is both not sustainable and not helpful. Between our groups being heads down shipping reliable and stable enhancements, societal challenges and atrocities that affect us such the war in Ukraine, not to mention a global pandemic, we have a lot of leniency for groups getting this together. Our contributors live all over the world, have day jobs, and might have their own challenges that they are living through.

With the theme of burnout, how can we support groups without bogging them down with paperwork? How can we communicate our needs at a level that hears and takes action on them? We need to build more tooling in this area and will be putting out a call for interns soon. Have other advice for us?


Help Wanted

SIG API Machinery

SIG Apps

SIG Apps is looking to grow their pool of [reviewers and appprovers]. Contributors looking at growing into these roles can join the SIG Apps / SIG CLI Review club.

SIG Auth

SIG Auth keeps a running list of KEPs that need help and tracks their progress on their SIG Auth project board. They are also looking for help in enhancing their own onboarding guide and PR review guidance.

Specifically SIG Auth is looking for help in these initatives:

SIG CLI

SIG CLI has three areas where they’re looking for more help:

SIG Cloud Provider

SIG Cloud Provider needs more support from cloud providers to extract the provider specific code from the main Kubernetes repo. Spinning them out will create a smaller and more secure core, while enabling the Cloud Providers to release and update their components on their own cadence.

SIG Contributor Experience

The SIG is looking for a full time community manager. Also, there are three subprojects where SIG Contributor Experience could use assistance.

SIG Docs

There are two initatives where SIG Docs could use assistance.

The blog subproject is particularly short on resources and attention. At the moment a very small pool of active editors are the constraint / most critical resource for article publication. One editor is involved in the majority of published articles; other editors are perhaps even more stretched with other Kubernetes contributions and involvement with other SIGs.

The Ukrainian localization team is primarily worked on by people based in Ukraine, where the ongoing and intensifying conflict creates challenges that take priority over open source contribution.

SIG Instrumentation

The Prometheus Adapter subproject is in need of additional contributors that can grow and commit to becoming reviewer/approvers. It currently only has one active approver and is used a number of endusers.

SIG K8s Infra

SIG K8s Infra is looking for engineers to help build tools to automate more of the project’s infrastructure and to help migrate more tests to community owned resources. Please show up to #sig-k8s-infra on Slack to help with this important group. (You can get an invitation to Slack from https://slack.k8s.io/)

SIG Release

SIG Release is looking for more contributors in a number of subprojects

SIG Scalability

SIG Scalability is looking to grow their contributors base across all their subprojects. Good entry points for new scalability contributors are the Scalability Test Framework and Performance Tests & Validaiton subproject.

SIG Scheduling

The Scheduler Simulator, a project that allows for simulating and testing of scheduling profiles/plugins needs more reviewers and approvers.

SIG Security

The SIG Security docs subproject is always looking for security-minded contributors of all experience levels to share their learning and knowledge with the community. This subproject has consistently been a place where people merge their first Kubernetes PRs. There’s always room for continuous improvement in our documentation, and contributing to this provides an opportunity to learn more about Kubernetes security while helping everyone run their clusters more safely. We’re really proud of the way Docs encourages and welcomes new contributors, and we’d love to encourage you to become a part of it!

SIG Storage

SIG Storage is broadly looking for more help fixing bugs and growing reviewers across the board.

Full time contributors in the following areas:

SIG Testing

SIG Testing is broadly looking for more contributors that can become reviewers / approvers.

Looking for help in the following projects:

SIG Windows

SIG Windows has several areas it is looking for support, the largest being related to Windows Storage support/CSI Proxy.

Looking for full time contributors to help with:

WG API Expression

No Report

WG Data Protection

WG IoT/Edge

Spinning down inside of Kubernetes and heading to CNCF level

WG Multitenancy

No specific help needed! Contributions are still welcome.

WG Structured Logging


Initiatives

SIG API Machinery

API Machinery is evaluating the potential for generics in go1.19. There are a number of other initiatives.

SIG Apps

Kubernetes Enhancements:

SIG Auth

SIG CLI

SIG CLI made progress on a number of initiatives in 2021:

Kubernetes Enhancements

SIG Cloud Provider

No Report

SIG Contributor Experience

During 2021, SIG Contributor Experience continued to provide a number of services to the project and it’s 75,000 contributors. Some achievements include the migration of the large public kubernetes-dev mailing list to to managed a project owned Google workspace, developing Elekto, a replacement for the CIVS voting system, and the seamless migration of the CLA system to EasyCLA.

SIG Contributor Experience also ran the North America Contributor Summit, the end of year Contributor Celebration, ran three successful mentoring cohorts, and the [Contributor Comms team] automated and started using the @k8scontributors twitter account to reach 5700 follows with a number of them being contributors.

Contributor Experience (“ContribEx”) is a service and program orientated SIG. Most of its initiatives cover long term services for the Kubernetes project.

SubprojectInitiative / Program
CommunityCommunity Repo Stewardship
Community ManagementCalendar Admin
Community ManagementLeadership Operations
Community Managementdiscuss.k8s.io End User Forum Admin
Community ManagementMailing List Admin
Community ManagementSlack Admin
Community ManagementZoom / YouTube Admin
Contributor DocumentationContributor Guide Stewardship
Contributor DocumentationContributor Site
Contributor DocumentationDeveloper Guide Audit
Contributor DocumentationDeveloper Guide Stewardship
Contributor CommsContributor / SIG Profiling
Contributor CommsSIG Outreach and Support
Contributor CommsContributor Events Outreach
Contributor CommsStewardship of k8scontributors twitter
DevstatsDevstats Dashboard Update
EventsMonthly Community Meeting
EventsOffice Hours
EventsElections
EventsContributor Summits
GitHub ManagementGitHub Admin / Moderation
GitHub ManagementGitHub Master -> Main rename
GitHub ManagementGitHub New Membership Coordinator
MentoringGroup Mentoring
MentoringLFX Mentor Program
Slack Infraslack-infra

SIG Docs

Kubernetes Enhancements:

SIG Instrumentation

SIG Instrumentation had several large accomplishments in 2021.

Kubernetes Enhancements:

SIG Release

After finalizing the rewrite of the release process from bash into golang, the release engineering team focused its efforts on two main areas:

Kubernetes Enhancements

SIG Scalability

SIG Scalability spent significant effort on validating the scalability and reliability impact of many Kubernetes features across 2021; growing the scalability tests of large services to cover 1000+ pods. Additional work was put into adding support for modules in tests, measuring the availability of the api-server and adding support for measuring cilium propagation delay & dns latency.

Kubernetes Enhancements:

SIG Scheduling

During 2021, SIG Scheduling focused on improving the overall performance of the scheduler, some highlights include:

Kubernetes Enhancements:

SIG Security

Most of SIG Security’s initiatives are out of scope for KEPs, and instead are largelty service and process oriented.

In 2021 they had several notable achievements:

Kubernetes Enhancements:

SIG Storage

In addition to a number of KEPs, SIG Storage has been working on CBT (Change Blocking Tracking)] in conjunction with the Data Protection WG

Kubernetes Enhancements:

SIG Testing

SIG Testing is largely service-oriented and their initatives are not often tracked as KEPs, yet they have had a number of achievements in the past year improving testing infrastructure and features.

Highlights of some of these initiatives include:

Kubernetes Enhancements

SIG Windows

SIG Windows has made progress on a number of lower level features. They implemented hostProcess container support (now in beta) which has now been adopted by a number of other OSS Projects. Other achievements include better node-level logging, improving the Windows Kubernetes developer experience with sig-windows-dev-tools, defining a set of operational readiness standards, and removed Dockershim from Windows nodes.

Kubernetes Enhancements

WG API Expression

WG Data Protection

The Data Protection WG identified the missing building blocks for supporting data protection in Kubernetes and published in their whitepaper. Features such as Volume Backups, Change Block Tracking, Volume Populator, Volume Group Group Snapshot, and Backup Repositories are owned by SIG Storage. Features such as Quiesce and Unquiesce Hooks are owned by SIG Node, with SIG Storage and SIG Apps participating. Features such as Application Snapshots and Backups are owned by SIG Apps, with SIG Storage participating. We will continue to work on them until all the missing pieces are available in Kubernetes.

The following items have been under development and have not yet been captured in a KEP:

WG IoT/Edge

The IoT/Edge Working Group is moving to the CNCF ecosystem.

WG Structured Logging

In 2021 The structured logging WG migrated kubelet, kube-scheduler, kube-proxy to the new standard format.

Kubernetes Enhancements Beta: