Posted on November 4, 2024 by Dotan Horovits + Adriel Perkins

CNCF projects highlighted in this post

OpenTelemetry logo

SIG post by Dotan Horovits and Adriel Perkins, Project Leads, SIG CI/CD Observability, OpenTelemetry

We’ve been talking about the need for a common “language” for reporting and observing CI/CD pipelines for years, and finally, we see the first “words” of this language entering the “dictionary” of observability – the OpenTelemetry open specification. With the recent release of OpenTelemetry’s Semantic Conventions, v1.27.0, you can find designated attributes for reporting CI/CD pipelines. 

This is the result of the hard work of the CI/CD Observability Special Interest Group (SIG) within OpenTelemetry. As we accomplish the core milestone for the first phase, we thought it’d be a good time to share it with the world.

Engineers need observability into their CI/CD pipelines 

CI/CD observability is essential for ensuring that software is released to production efficiently and reliably. Well-functioning CI/CD pipelines directly impact business outcomes by shortening Lead Time for Changes DORA metric, and enabling fast identification and resolution of broken or flaky processes. By integrating observability into CI/CD workflows, teams can monitor the health and performance of their pipelines in real-time, gaining insights into bottlenecks and areas that require improvement. 

Leveraging the same well-established tools used for monitoring production environments, organizations can extend their observability capabilities to include the release cycle, fostering a holistic approach to software delivery. Whether open source or proprietary tools, there’s no need to reinvent the wheel when choosing the observability toolchain for CI/CD pipelines.

The need for standardization

However, the diverse landscape of CI/CD tools creates challenges in achieving consistent end-to-end observability. With each tool having its own means, format and semantic conventions for reporting the pipeline execution status, fragmentation within the toolchain can hinder seamless monitoring. Migrating between tools becomes painful, as it requires reimplementing existing dashboards, reports and alerts.

Things become even more challenging, when needing to monitor multiple tools involved in the release pipeline in a uniform manner. This is where open standards and specifications become critical. They create a common uniform language, one which is tool- and vendor-agnostic, enabling cohesive observability across different tools and allowing teams to maintain a clear and comprehensive view of their CI/CD pipeline performance.

The need for standardization is relevant for creating the semantic conventions mentioned above, the language for reporting what goes on in the pipeline. Standardization is also needed for the means in which this reporting is propagated through the system, such as upon spawning processes during the pipeline execution. This led us to promote standardization for using environment variables for context and baggage propagation between processes, another important milestone that was recently approved and merged.

OpenTelemetry: the natural home for CI/CD observability specification

This realization drove us to look for the right way to approach creating a specification. OpenTelemetry emerges as the standard for telemetry generation and collection. The OpenTelemetry specification is tasked with exactly this problem: creating a common uniform and vendor-agnostic specification for telemetry. And housed under the Cloud Native Computing Foundation (CNCF) can ensure it remains open and vendor-neutral. As long standing advocates of OpenTelemetry, it only made sense to extend OpenTelemetry to cover this important DevOps use case. 

We started with an OpenTelemetry extension proposal (OTEP #223) a couple of years ago, proposing our idea to extend OpenTelemetry to cover the CI/CD observability use case. In parallel, we’ve started a slack channel on the CNCF slack to gather fellow enthusiasts behind the idea and start brainstorming what that should look like. The slack channel grew and we quickly discovered that the problem is common across many organizations.

With the feedback from the Technical Oversight Committee and others within the CNCF, we’ve taken the path of asking the mandate to start a dedicated Working Group for the topic under OpenTelemetry’s Semantic Conventions SIG (SIG SemConv in short). With their blessing, we launched the formal CI/CD Observability SIG to formalize our previous slack group discussions and goals. 

OpenTelemetry’s CI/CD Observability SIG

Since November of 2023, the SIG has been actively working to develop the standard for semantics around CI/CD observability in collaboration with experts from multiple companies and Open-Source projects. At its inception, we decided to focus on a few key areas for 2024:

At first, our SIG met during the larger Semantic Conventions Working Group meetings every Monday. This provided a good opportunity for us to get our bearings as we researched and discussed how we would accomplish the goals on our roadmap. This also enabled us to get to know many members of the larger OpenTelemetry community, solicit feedback on our designs, and get direction on how to proceed. The OpenTelemetry Semantic Convention Working Group has been extraordinarily supportive of the CI/CD initiative.

Upon completion and release of its initial milestone (see below), our SIG was granted its own dedicated meeting slot on the OpenTelemetry calendar, every Thursday at 0600 PT. The group gets together here to discuss current and future work prior to bringing to the larger Semantic Conventions meetings on Monday. We greatly look forward to the continued support and participation of the community as we continue to drive forward this critical area of standardization.

CI/CD Is part of the latest OpenTelemetry Semantic Conventions

Over the course of months of iteration and feedback, the first set of Semantic Conventions was merged in for the v1.27.0 release. This change brought forth the first set of foundational semantics for CI/CD under the CICD, artifacts, VCS, test, and deployment namespaces. This was a significant milestone for the CI/CD Observability SIG and industry as a whole. This creates the foundation for which all of our group’s other goals can begin to take form, and reach implementation.

But what does that actually mean? What value does it provide? Let’s consider real world examples for two of the namespaces

Tracking release revisions from Version Control Systems (VCS)

Version Control System (VCS) attributes cover multiple areas common in a VCS like refs and changes (pull/merge requests). The vcs.repository.ref.revision attribute is a key piece of metadata. As Version Control Systems like GitHub and GitLab emit events, they can now have this semantically compliant attribute. That means when integrating code, releasing it, and deploying it to environments, systems can include this attribute and trace the code revision across bounds more easily. In the event a deployment fails, you can quickly look at the revision of code and track it back to the buggy release. This attribute is actually a key piece of metadata for DORA metrics too as you calculate Change lead time and Failed deployment recovery time.

Artifacts for supply chain security, aligned with the SLSA specification

The artifact attribute namespace had multiple attributes for its first implementation. One key set of attributes within this namespace cover attestations that closely align with the SLSA model. This is really the first time a direct connection is being made between Observability and Software Supply Chain Security. Consider the following supply chain threat model defined by SLSA:

image

These new attributes for artifacts and attestations help observe the sequence of events modeled in the above diagram in real time. Really, the conventions that exist today and those that will be added in the future enable interoperability between core software delivery capabilities like security and platform engineering via observability semantics.

What’s Next for CI/CD Observability Working Group

The first major milestone we shared above, was the merge of the OTEP for extending the semantic conventions with the new attributes, which is now part of the OpenTelemetry Semantic Conventions latest release.

The other important milestone was OTEP #258 for Environment Variable Context Propagation that was just approved and merged. This OTEP sets the ground for writing the specification.

Since we’ve made progress on our initial milestones, we’ve updated the CI/CD Observability SIG milestones for the remainder of 2024. Our goal is to finish out as many of the defined milestones as possible by the end of the year. Notably, we’re focused on: 

All that has been mentioned thus far is just the beginning! We have lots of work defined on our CICD Project Board, and we have work in progress! We’ll continue to iterate on the above milestones that we’ve set out for the remainder of 2024. Here’s a couple things to look out for.

And much more!

It takes a village to extend OpenTelemetry

Woah, that’s a lot to do! Most certainly this SIG will continue beyond 2024 and through 2025. Standards are hard, but essential. And, we have some amazing folks that are part of the SIG and contributing to these standards! Who you may ask? 

Firstly we’d like to acknowledge key members of OpenTelemetry leadership committees who have heavily enabled the work we’ve done thus far, and will continue to do.

From the OpenTelemetry Technical Committee we have two core sponsors, Carlos Alberto from Lightstep and Josh Suereth from Google. Both Carlos and Josh have been so supportive of the CICD work, really guiding us through the process and details we need to be successful.

From the OpenTelemetry Governance Committee we’ve had Trask Stalnaker from Microsoft act as an exceptional ally, and Daniel Blanco from Skyscanner who now acts as our current Liaison. Both Trask and Daniel have been instrumental in supporting the SIG and enabling us to have our own meeting in the OpenTelemetry community.

In addition to those folks, we’ve had significant feedback, support, and contributions from the following key folks:

That was a lot of names to name! We greatly appreciate everyone who has supported this initiative and helped bring it to fruition! It takes significant thinking ability and time to build industry wide standards. Hard problems are hard, but these folks have risen to the challenge to make the world of observability and CICD systems a better, more interoperable place!

Join the Working Group discourse and make an impact

Want to learn more? Want to get involved in shaping CI/CD Observability? 

We invite developers and practitioners to participate in the discussions, contribute ideas, and help shape the future of CI/CD observability and the OpenTelemetry semantic conventions. Discussion takes place in the CNCF slack workspace under the #cicd-o11y channel, and you can chime in on GitHub and join the CICD SIG weekly calls every Thursday at 0600 PT.