Verify the integrity of the Helm Charts stored in OCI-compliant registries as OCI artifacts

Posted on November 24, 2022 by Flux maintainers

CNCF projects highlighted in this post

Guest post originally published on Flux’ blog

Cosign integration was one of the most important features we shipped in the Flux v0.35 release. After that, we wrote a blog post which explains how to use the feature with OCIRepository resources which enables fetching OCI artifacts from container registries. If you haven’t read it yet, we highly encourage you to go and check it out first.

Flux v0.36.0 allows you to prove the authenticity of HelmChart resources with the help of the cosign integration. Here we will demonstrate how to use the cosign integration to verify the integrity of the Helm charts stored in OCI-compliant registries as OCI artifacts.

Not at #kubecon so I had time to prepare Flux and Flagger releases. Flagger v1.24 comes with signed releases & OCI Helm charts. Flux v0.36 adds support for verifying Helm charts with Cosign. https://t.co/6MYwzfMA3W
— Stefan Prodan (@stefanprodan) October 27, 2022

Starting with Helm v3.8.0, Helm supports the OCI registry as a one of the storage option for Helm charts as an alternative to Helm repositories. The Helm CLI can push and pull Helm charts to and from OCI-compliant registries.

Note: Prior to Helm v3.8.0, OCI support was experimental. To use it there, you need to enable the feature by setting the HELM_EXPERIMENTAL_OCI environment variable to 1.

As we store Helm charts in OCI-compliant registries as OCI artifacts, we can now use the cosign integration to sign and verify them. Also, thanks to Flux, you can reconcile resources such as plain-text Kubernetes YAML manifests, Terraform modules, etc. from OCI-compliant registries with the help of OCIRepository resources. You can achieve the same thing for Helm charts with HelmRepository resources. This means that you can store Helm charts in OCI-compliant registries as OCI artifacts and use Flux to reconcile them like the following:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: podinfo
  namespace: default
spec:
  type: "oci"
  interval: 5m0s
  url: oci://ghcr.io/stefanprodan/charts

Note:

Here you can review the complete registries lists that support the OCI artifact specification: OCI-Conformant Products.

You will notice that when you open the list, DockerHub is not included into the list but it will be added soon because they recently announced OCI Artifacts support, and you can read more about it from here.

Let’s jump right into the details of how we can actually use it.

We will deploy Prometheus by using its community Helm charts stored as OCI artifacts in OCI registry. Recently, Prometheus’ community started to publish their Helm charts to OCI registries and sign them with cosign using the keyless approach, you can learn more the process here. Then we are going to verify it with cosign and configure Flux to verify the Helm chart’s signatures before they are downloaded and reconciled. As the Prometheus community signed their Helm Charts without providing a key pair, we do not need to specify any key in the HelmChart resource’ provider.cosign spec to enable keyless verification for Flux.

For the sake of simplicity, we’ve deployed Prometheus alone but if you want to learn more about installing the Prometheus stack including Grafana, Alertmanager, etc., please refer to the official Flux page that can help you to do that.

You need three things to complete this demo;

cosign CLI
- https://docs.sigstore.dev/cosign/installation/
A Kubernetes cluster
- https://kind.sigs.k8s.io/#installation-and-usage
Flux CLI
- https://fluxcd.io/flux/cmd/

Let’s start by creating a simple Kubernetes cluster:

kind create cluster

Use the Flux CLI to do pre-flight checks:

$ flux check --pre
► checking prerequisites
✔ Kubernetes 1.25.3 >=1.20.6-0
✔ prerequisites checks passed

If the checks are successful, you can install Flux on the cluster.

Let’s install Flux on it – if you need to use other options, check out the installation page.

export GITHUB_USER=developer-guy
export GITHUB_TOKEN=$GITHUB_TOKEN

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=flux-cosign-helm-oci-demo \
  --branch=main \
  --path=./clusters/my-cluster \
  --personal

Note: Don’t forget to change the values with your own details!

As we stick to GitOps practices, we only create files that contain the HelmRepository and HelmRelease resources. After committing and pushing those changes into the upstream repository, Flux will watch for changes and use them as source-of-truth for the configuration:

git clone git@github.com:developer-guy/flux-cosign-helm-oci-demo.git
cd flux-cosign-helm-oci-demo

Let’s create the HelmRepository resource first:

flux create source helm prometheus-community \
    --url=oci://ghcr.io/prometheus-community/charts \
    --interval=10m \
    --export > ./clusters/my-cluster/prometheus-community-helmrepository.yaml

Now, let’s move on with creating the HelmRelease resource:

flux create helmrelease prometheus \
    --source=HelmRepository/prometheus-community \
    --chart=prometheus \
    --interval=10m \
    --release-name prometheus \
    --target-namespace=monitoring \
    --create-target-namespace \
    --export > ./clusters/my-cluster/prometheus-helmrelease.yaml

and run the following command to add the verify section to the HelmRelease resource’ .spec.chart.spec section to enable keylesss verification:

yq e '.spec.chart.spec|=({"verify": { "provider": "cosign" } } +.)' ./clusters/my-cluster/prometheus-helmrelease.yaml

This command above will add the following part to the HelmRelease resource’s .spec.chart.spec section:

verify:
  provider: cosign

then, commit and push the changes:

git commit -m "Add prometheus HelmRelease and HelmRepository resources"
git push

After a couple of seconds, Flux will have applied these changes. Now let’s check the status of them:

Or, you can trigger the reconciliation immediately by running the simple command: flux reconcile source git flux-system

For the HelmRepository resource:

$ flux get sources helm
NAME                    REVISION        SUSPENDED       READY   MESSAGE
prometheus-community                    False           True    Helm repository is ready

For the HelmRelease resource:

$ flux get helmreleases
NAME            REVISION        SUSPENDED       READY   MESSAGE
prometheus      15.18.0         False           True    Release reconciliation succeeded

If everything is fine, you can check the pods in the monitoring namespace:

$ kubectl get pods -n monitoring
NAME                                             READY   STATUS    RESTARTS   AGE
prometheus-alertmanager-54b7d7cf45-2b7zf         2/2     Running   0          115s
prometheus-kube-state-metrics-67f68d64bb-vlmvd   1/1     Running   0          115s
prometheus-node-exporter-46gm6                   1/1     Running   0          115s
prometheus-pushgateway-596cd99697-t79zt          1/1     Running   0          115s
prometheus-server-c458cf6f9-nvstw                2/2     Running   0          115s

Great! Now, you have installed Prometheus with Flux by using the Helm chart stored in the OCI registry and verified it with cosign.

We can assume that the Helm chart’s signature is verified as we let it be deployed in a cluster but let’s do have double check and check the status of the HelmRelease to see whether the verification is successful or not:

$ kubectl get helmcharts -n flux-system flux-system-prometheus -ojsonpath='{.status.conditions[?(@.type=="SourceVerified")]}'
{"lastTransitionTime":"2022-11-09T13:27:38Z","message":"verified signature of version 15.18.0","observedGeneration":1,"reason":"Succeeded","status":"True","type":"SourceVerified"}

That’s super cool! Because Flux is going to add a condition to the HelmChart resource’s status section to show the verification status. If the verification is successful, it will add a condition like the one above.

DIY (Do it yourself) Approach

The Prometheus community Helm charts only serve as an example. Here is how you can do the same thing with your own Helm charts.

Create a Helm chart
Package the Helm chart as .tar.gz file
Login to the OCI-compliant registry that you want to use to store your Helm chart
Push the Helm chart as OCI artifact

Let’s create a sample directory that will contain our Helm chart:

mkdir -p helm-oci-demo
cd helm-oci demo

Now package the Helm chart as .tar.gz file:

 helm create nginx

Let’s package the Helm chart as .tar.gz file:

helm package nginx

Now, let’s login to the OCI-compliant registry that you want to use to store your Helm chart. In this example, we’ll be using the GitHub Container Registry (ghcr.io):

echo $GHCR_PAT | helm registry login ghcr.io -u $USER --password-stdin

Note: Don’t forget to change the values with your own details!

At this point, we are ready to push the Helm chart as OCI artifact:

$ helm push nginx-0.1.0.tgz oci://ghcr.io/$USER
Pushed: ghcr.io/developer-guy/nginx:0.1.0
Digest: sha256:21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7

So, the Helm chart is pushed to the OCI registry. It’s time to sign it with cosign. As cosign recommends we should always sign images based on their digests (@sha256:) rather than a tag. So, we should grab the digest from the command output above which is 21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7 in this case, and use that digest while signing the image:

As we saw the keyless approach before, let’s try the key-based approach this time. To do that, we should create public/private key pairs first:

cosign generate-key-pair

This command will generate two files, a cosign.pub which is a publickey and cosign.key which is a private key pair and store them in the current directory directory.

Now, let’s sign the image with the private key:

cosign sign --key cosign.key ghcr.io/$USER/nginx@21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7

Cool! Now we have signed the image with the private key. Let’s check the signature:

cosign verify --key cosign.key ghcr.io/$USER/nginx@21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7

Yay! It’s verified. But in order to make the public key accessible by Flux, we need to create a Kubernetes secret to store the public key:

kubectl -n flux-system create secret generic cosign-pub \
  --from-file=cosign.pub=cosign.pub

Now, we can use it in our Flux configuration. The rest of the steps are the same as the previous section. For the sake of simplicity, we won’t repeat them here other than the HelmRepository and HelmRelease resources’ creation steps.

Let’s create the HelmRepository resource first:

flux create source helm $USER-charts\
    --url=oci://ghcr.io/$USER \
    --interval=10m \
    --export > ./clusters/my-cluster/nginx-helmrepository.yaml

Let’s move on with creating the HelmRelease resource:

flux create helmrelease nginx \
    --source=HelmRepository/$USER-charts \
    --chart=nginx \
    --interval=10m \
    --release-name nginx \
    --target-namespace=default \
    --export > ./clusters/my-cluster/nginx-helmrelease.yaml

Don’t forget to run the following command to add the verify section to the HelmRelease resource’ .spec.chart.spec section to enable verification:

yq e '.spec.chart.spec|=({"verify": { "provider": "cosign", "secretRef": { "name": "cosign-pub" } } } +.)' ./clusters/my-cluster/nginx-helmrelease.yaml

This command above will add the following part to the HelmRelease resource’s .spec.chart.spec section:

verify:
  provider: cosign
  secretRef:
    name: cosign-pub

That’s all you need to do folks!

Congratulations! You have successfully signed your Helm chart with cosign with key-based approach and used it with Flux.

London, United Kingdom

DIY (Do it yourself) Approach