Elevating system resilience: leveraging LitmusChaos and Backstage integration

Posted on April 1, 2024 by Namkyu Park

CNCF projects highlighted in this post

Project post by Namkyu Park, Maintainer of LitmusChaos (LinkedIn | GitHub)

This blog post provides step-by-step instructions for injecting chaos using LitmusChaos and managing it with Backstage.

Table of Contents

• Chaos Engineering, LitmusChaos, and Backstage
• Pre-Requisites
• What does our demo consist of?
• Demo environment
• Injecting chaos using LitmusChaos
• Manage LitmusChaos stuff in Backstage
• Summary

Chaos Engineering, LitmusChaos, and Backstage

As cloud-native technologies, including Kubernetes, become more sophisticated, more components make up an application. Does your system have resilience? For example, if one of your containers goes down due to an OOM, is it automatically recreated and ready for it? Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. And LitmusChaos (CNCF incubating project) is a Cloud-Native Chaos Engineering Framework with cross-cloud support. You can inject chaos into different infrastructure layers using LitmusChaos.

Backstage is an open platform for building developer portals and is one of the most popular CNCF projects. It allows developers to manage cloud-native applications’ numerous services and codes from a single point.

LitmusChaos can be integrated with Backstage via the backstage-plugin. In this tutorial, we will build a local Kubernetes cluster via minikube and install the demo application and LitmusChaos. Afterward, we will run a chaos experiment with LitmusChaos and easily view the stuff in LitmusChaos with Backstage.

Pre-Requisites

• Docker, minikube (If you use your k8s cluster, skip this)
• LitmusChaos pre-requisites
• Backstage pre-requisites

What does our deployment consist of?

Luckily, We don’t have to make k8s-based services. microservice-demo is a great microservice we can use. The picture below shows the architecture of the demo we’ll be building.

Demo environment

Let’s install a local k8s cluster and the demo app, then install LitmusChaos.

minikube (local k8s cluster)

In this tutorial, we are using the local k8s cluster, minikube. Follow this guide if you have not installed minikube yet. Once minikube start executes, you can access the local k8s cluster using kubectl command.

Online Boutique (microservices-demo)

Installing microservices-demo is more simple. All you have to do is execute the below code. Here’s a more detailed guide.

// clone the repo
git clone https://github.com/GoogleCloudPlatform/microservices-demo
cd microservices-demo/
// move dir
cd microservices-demo/
// install using a manifest file
kubectl apply -f ./release/kubernetes-manifests.yaml

We need to access the demo with the URL generated by the following command.

minikube service frontend-external --url

Now we can access the service with http://<<generated_url>>.GIF

LitmusChaos

Let’s open a new shell and install LitmusChaos.

// Add LitmusChaos Helm repo
helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
// Create the namespace
kubectl create ns litmus
// Install the chart
helm install chaos litmuschaos/litmus --namespace=litmus --set portal.frontend.service.type=NodePort

As before, we need to create a URL to access the frontend of the litmuschaos(chaos center).

minikube service chaos-litmus-frontend-service -n litmus --url

login as admin / litmus

The first time you log in to LitmusChaos, you’ll see a popup asking you to enable chaos infrastructure like this

Once you’ve gone through all the steps to create your environment, you’ll see your chaos infrastructure connected as shown below.

If you’re curious about the architecture of LitmusChaos, check out this document.

Injecting chaos using LitmusChaos

We’ve done all the preparation to perform Chaos Engineering. Let’s discuss a scenario. Suppose you are the manager of an online boutique service, and you want to run an experiment to see if the cartservice component recovers when you remove its pod. To check if a pod exists, use the following command. kubectl get pods -n default | grep cartservice | grep Running | wc -l.

LitmusChaos supports various types of probes. Learn more about probe here and here’s a great tutorial video.

Setup Probe

You can create a new probe in the Resilience Probes tab. In this tutorial, we will create a Command Probe. Enter a value like below.

// Command
kubectl get pods -n default | grep cartservice | grep Running | wc -l
// Type
Int
// Comparison Criteria
>
// Value
0

One more thing, because cartservice deployment doesn’t have labels, we added labels like below

kubectl label deployment cartservice app=cartservice

Let’s inject chaos

All we have to do is create a chaos experiment. As we discussed before, we delete the cartservice pod and check whether this pod regenerates. Here’s a detailed tutorial and gif below. I just want to point out some important features.

First, we select the pod-delete chaos fault in this experiment.

Second, we provide information about the target applications like below.

Last, we use a probe we made before.

When you set up and run a chaos experiment, chaos is injected into the target application and the probe checks to see if the application is resilient. If yes, You’ve now mastered chaos engineering. Congratulations 🎉

Manage LitmusChaos stuff in Backstage

You can start building your Backstage app by following this Getting Started Guide and running it with yarn dev.

LitmusChaos plugin for Backstage

Let’s add a LitmusChaos plugin to Backstage

• In the ./packages/app project add the plugin.

yarn add backstage-plugin-litmus

• Get LitmusChaos API Token through the LitmusChaos UI
  1. Click “SETTINGS” in the sidebar.
  2. Click “+ New Token” Button in the ‘API Tokens’ section.
  3. Enter your information and click “Confirm”.
  4. Get the token in the “VALUE” column.
• In the app-config.yaml file root directory, add litmus proxy and info like below

   proxy:
     '/litmus':
       target: 'your-own-litmus-ui-url'
       changeOrigin: true
       headers:
         Authorization: 'Bearer ${LITMUS_AUTH_TOKEN}'
   litmus:
     baseUrl: 'your-own-litmus-ui-url'
     apiToken: ${LITMUS_AUTH_TOKEN}

• Add your auth key to the environmental variables

   export LITMUS_AUTH_TOKEN="your-own-token"

• To get the project ID, copy it directly from the URL from the portal after ‘/project/’
• Append a new component to ./examples/entities.yaml

# ...
---
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: backstage-litmus-demo
  description: An example of a Backstage application.
  ## append here
  annotations:
    litmuschaos.io/project-id: your-own-project-id
  ## 
spec:
  type: service
  owner: john@example.com
  lifecycle: experimental

• To Enable frontend, we need to edit ./packages/app/src/components/catalog/EntityPage.tsx

   // packages/app/src/components/catalog/EntityPage.tsx
   import { isLitmusAvailable, EntityLitmusCard, EntityLitmusContent } from 'backstage-plugin-litmus'
   // ...
   const overviewContent = (
     <Grid container spacing={6} alignItems="stretch">
       // ...
       <EntitySwitch>
         <EntitySwitch.Case if={isLitmusAvailable}>
           <Grid item md={4} xs={12}>
             <EntityLitmusCard />
           </Grid>
         </EntitySwitch.Case>
       </EntitySwitch>
       // ...
     </Grid>
   )
   // ...
   const serviceEntityPage = (
     <EntityLayout>
       // ...
       <EntityLayout.Route path="/litmus" title="Litmus">
         <EntityLitmusContent />
       </EntityLayout.Route>
       // ...
     </EntityLayout>
   )

yarn dev again, Now you can manage all about LitmusChaos in one place, Backstage. The Overview tab provides an at-a-glance view of some of the important metrics in LitmusChaos.

The LitmusChaos tab is where you can see all the information about LitmusChaos on one page, including chaos experiments, chaos infrastructure, and more.

What I want to highlight is that we can re-run an existing chaos experiment. All we have to do is just click ‘Run Experiment’ button 🚀

Summary

In this tutorial, we’ve explored how to utilize LitmusChaos and Backstage integration to inject chaos into a Kubernetes environment and manage it effectively.

The integration of LitmusChaos and Backstage offers a unified platform to manage all aspects of Chaos Engineering right from injecting chaos to monitoring the impact and analyzing the results.

If you are interested in LitmusChaos, Join the community! You can join the LitmusChaos community on GitHub and Slack.

Thank you for reading 🙏

Namkyu Park
Maintainer of LitmusChaos
LinkedIn | GitHub