Guest post by Da Yin, engineer at Alibaba Cloud and maintainer of KubeVela
Background
Dating back to year 2019, Kubernetes is gradually being widely adopted as the de facto standard for deploying and managing infrastructures. More and more platform engineers start to build platforms on top of Kubernetes and provide services to their end-users. The containerized deployment and declarative configuration have greatly reduced difficulites of complex system operating.
However, when thousands of workloads and related resources are held in the cluster, it is hard for operators to identify the logical topology and make proper accurate management based on their internal relationships.
At the same time, tools like Helm or Kustomize have explored solutions for packaging and delivering Kubernetes resources. These have been proven to be highly effective and Helm Chart is now obviously the most popular way for publishing and distributing artifacts along with the built image in the Kubernetes world.
On the other hand, various projects also started the journey of defining applications. Instead of simply packaging resources, these attempts seek for solutions from a more top down perspect. Compared to grouping discrete objects for more convenient usage, applications are designed to bridge the cognition gap between app users and underlying infrasturctures. The birth of Open Application Model (OAM) and it’s implementation KubeVela starts here.
The Birth of the Application Model
The propose of Open Application Model (OAM), backed by Microsoft & Alibaba Cloud, intends to give a theoretical model for what a cloud native application should be look like. It is designed initially not to be bond with the Kubernetes implementation, and tends to provide unified interfaces for operating applications.
OAM raises the concept of Component and Trait to make abstractions to the architecture of applications. This is a big change for native Kubernetes users. But it comes for reasons.
In Kubernetes, resources like Deployments or Services focus on the implementation. Each type of resources provide specific functionalities. Some low-level functionalities like running containers goes to basic units like Pods. Some higher-level ones that built upon lower ones like orchestrating containers, handling rollback like Deployment or StatefulSet. There’s no doubt that “the platform for platform builders” has received great success. But for app developers, the community is forcing them to become Kubernetes expert to get things work. Besides, A superficial understanding of low-level resources may actually result in a significant amount of risk due to incorrect operations.
Component
OAM chooses a different starting point. What really composes an application? What do app developers need?
Something that runs the application. This is absolutely the central part for an application. Inside Kubernetes, it could be Deployment, Pods or other things. Outside Kubernetes, it could be Docker container, Virtual machines or cloud execution engine. 99.9% app developers create programs and run it. OAM defines the whole stuff that runs the app as Component. It is NOT merely a group of resources. Without Component, you will not be able to know what the application is for.
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
name: webserver-demo
spec:
components:
- name: hello-world
type: webserver # claim to deploy webserver component definition
properties: # setting parameter values
image: crccheck/hello-world
port: 8000 # this port will be automatically exposed to public
env:
- name: "foo"
value: "bar"
cpu: "100m"
^^ Example of a component inside an OAM application
As microservice becomes more and more prevelant, lots of app developers start to break giant monolitithic program into small pieces. Each runnable piece in OAM’s definition, could be modeled as a Component and as a whole, they together compose a complete OAM Application. Their internal logical relationships are also sketched inside the Component.
Trait
To make an application run properly, there are also other things that needed. For example, in Kubernetes, we have Service, ConfigMap, PersistentVolumeClaim, and many other resources that provide specific capabilities for Deployment to use. But these resources are created to satisfy some technical demands, and are not purpose-oriented. The ConfigMap provides a way for Kubernetes users to store configuration data but to let the container use the configuration, users need to add volume sections in Deployment and set container args or environment variables.
If we change the view again and think about what app developers really want to do when they use these resources, we could possibly reach a conclusion that many of the resources serve for extra capabilities for the application Component. The Service or Ingress objects are created for providing access for the application. The PersistentVolumeClaim and volume sections in the Deployment are for providing storage. Third-party objects like ServiceMonitor in Prometheus are for providing observation rules. These auxiliary resources and modifications that serve for providing capabilities are defined as Trait in OAM.
Trait does not work independently. These capabilities are attached to Component and work as decorations. Without it, the application’s main purpose will not change, but become an incomplete one.
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
name: my-example-app
spec:
components:
- name: publicweb
type: web-ui
properties: # properties targeting component parameters.
image: example/web-ui:v1.0.2@sha256:verytrustworthyhash
param_1: "enabled" # param_1 is defined on the web-ui component
traits:
- type: ingress # ingress trait providing a public endpoint for the publicweb component of the application.
properties: # properties are defined by the trait CRD spec. This example assumes path and port.
path: /
port: 8080
- name: backend
type: company/test-backend # test-backend is referenced from other namespace
properties:
debug: "true" # debug is a parameter defined in the test-backend component.
traits:
- type: scaler # scaler trait to specify the number of replicas for the backend component
properties:
replicas: 4
^^Example of an OAM application containing two components with traits attached
And more
Aside by the Component and Trait, which mostly describe the application from the functionality aspect, there are also other things that works for the application in different levels.
Each application could contain multiple atomic functional units, aka Component, and each Component could be decorated by several Traits. If we want to define some application-level behaviours or strategies, which do not have a clear one-to-one relationship with the running Component, we need more for the applictaion definition.
By far, OAM does not have a decisive solution for this part of modeling. Concepts including Scope, Policy, Workflow have been proposed. Scope is added to define the usage scope of the application. Policy is added to describe the common strategies and behaviours. Workflow focuses on the delivery aspect of the application. Some of them are adopted in the KubeVela, which is one of the Kubernetes-based implemetation for OAM, that can help us drive the design with real world practices.
From Theory to Practice
The specification of Open Application Model conceptually, provides guides to developers who want to model application and build platforms from the app developers perspective. Adopters including KubeVela, Crossplane, Verrazzano and Intuit have made various attempts in the past years. Partly due to the technical limitation, although OAM spec is raised to be infrastructure agnostic, the majority of the implemetations are all based on Kubernetes by far.
As summarized after looking back to the history, application modeling is especially critical on Kubernetes. When landing OAM spec on Kubernetes, KubeVela did a lot of job to tackle various detail technique problems. They can be categorized into three different tracks: abstraction, delivery and management.
Abstraction
The Component in OAM defines the running instance of the application. Different types of Component goes for different type of running instances. On Kubernetes, it corresponds to different workload implementation, like Deployment or StatefulSet. KubeVela declare these TYPES as ComponentDefinition and introduce CUE for an extensible solution. CUE is used for templating Kubernetes resources and platform builders could therefore make arbitrary customizations.
In terms of Trait, the method of abstraction is similar, aka use TraitDefinition to declare different types. One of the key differences to Component is that some types of Trait needs to make modifications to the resources abstracted by ComponentDefinition. For example, to expose ports, we need to add Service object and expose ports in the Deployment spec at the same time. CUE itself does not provide ways to do so but KubeVela somehow managed to achieve that by making extensions to CUE.
In addition to the static rendering, KubeVela’s abstraction layer is also runtime-aware. It means the rendering logic could be altered according to the runtime environment, like the Kubernetes version it currently sticks to. This is particularly useful when KubeVela is used as the unified control plane across clusters.
In summary, similar to Go Template used by Helm Chart or making spec overlays via Kustomization, the implementation of the abstraction layer in KubeVela does not show huge technical differences. On the other hand, KubeVela grounded these abstractions into ComponentDefinitions and TraitDefinitions for better reuse. This gives finer-grained choices for platform builders compared to directly templating the whole application using Helm Chart. This in turn draws a clearer boundary between the responsibilities of platform engineers and app developers: one works on X-Definitions and the other works on Application.
Delivery
The abstraction layer mainly solves the problem of building, packaging and distributing applications. When it comes to deliverying the application and make them as a unified one, things get more complex. The application as a whole, is treated as the basic unit for delivery in KubeVela. The internal relationships between different Components inside one application could decide how these Components should be delivered, which are indicated in the dependsOnfield.
Except for dispatching resources, the delivery of application sometimes involves more, like manual reviewing for risky delivery to production environment, notification on the completion of the automatic delivery, differentiated delivery across multi-clusters, etc. To empower users to customize the delivery process, KubeVela adds Workflow to the Application model, where WorkflowStepDefinition leverages CUE to define the atomic execution unit.
Some users also might encounter cases when multiple correlated applications are intended to be delivered and KubeVela even gives higher level solution for that, known as Pipeline.
It is also a frequently debated question that whether the application model should be jointly designed with the delivery. Independent delivery tools like Tekton, Argo Workflow have been invented for years. Other CI tools like Jenkins or GitHub Actions also evolves themselves to support various delivery scenarios. KubeVela has made a pioneering step in combining application model and the delivery process together because it considers this method to be a practical solution for landing application model into production usage. But the evolvement of OAM spec holds back, as the answer to the theoretical modeling still wait for broader discussions.
Management
The application model is not only for delivery. The declared state is always wanted. So in practice, the application model is being used for KubeVela’s consistent reconciliation as well. KubeVela ensures that there is no drift from the desired state once the delivery process finished successfully, which is accord with the final-state-oriented philosophy of Kubernetes.
Addtionally, KubeVela also embeds many other management actions, including version management, privileges control, resource sharing and garbage collection. These are not included in the OAM spec for now but are considered to be general capabilies that could be parts of the OAM spec in the future.
In general, to satisfy all kinds of needs for the application usage on Kubernetes, KubeVela has made continuous attempts beyond OAM and explore more possible ways to operate applications.
If you’re interested with more technical details, please refer to this blog.
Future
Many ones have questioned why OAM seems to stop evolving recently. For people who mainly cares for the production usage, the answer is that KubeVela, as an OAM implementation, has always been evolving. The user number is growing and it has just become a CNCF incubation project.
As for the theoretical model, KubeVela is still waiting for more feedbacks and evidences from the industrial usage before proposing its extra added concepts like Workflow or Policy to the OAM spec. Hoping more and more people will join the community with the help of CNCF, to make deploying and operating applications across today’s hybrid, multi-cloud environments easier, faster and more reliable.