Guest post from Huawei by David Eads (@deads2k), Kevin Wang (@kevin-wangzefeng)
Current users will know that KubeFed was an early answer to the issue of centralized management of multiple Kubernetes clusters– but as community engagement has slowed with the project since 2020, it’s now reached its EOL, leaving many searching for alternatives. Today we have two CNCF projects providing modern answers to the issue KubeFed accounted for. With active and vibrant communities to support them, Karmada and Open Cluster Management (OCM) are here to take up the challenge of managing fleets of clusters across the hybrid and multi-cloud landscape.
Limitations with KubeFed
The Kubernetes SIG KubeFed project allowed users to coordinate the configuration of multiple Kubernetes clusters from a single set of APIs in a hosting cluster. Federation v1 could distribute services to multiple Kubernetes clusters but could not handle other kinds of objects. The project has been archived by Kubernetes-SIG and never became a core feature. KubeFed v2 quickly replaced v1 and is used by operations staff around the world. It allows a single Kubernetes cluster to deploy multiple kinds of objects to multiple other Kubernetes clusters. KubeFed v2 also allows the “control plane” main cluster to manage the other clusters, including their resources and policy.
Impediments to wider KubeFed adoption:
- Incompatibility of Kubernetes APIs – the federated-resources API results in extra learning and adoption efforts when users migrate from single cluster to multi-cluster architecture.
- Lack of extensibility – users were unable to satisfy varying use cases with the rigid nature of KubeFed. This led to an explosion of highly diverged forks and made it hard to build a community around a standardized implementation.
Given the difficulty of proliferating CRDs and extensibility limitations, there is a lack of roadmap or meaningful development in the KubeFed project to address these issues. Let’s take a look at two alternatives that are quickly adopted by users around the world.
Karmada: Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration
Karmada (Kubernetes Armada) is a Kubernetes management system that enables you to run your cloud-native applications across multiple Kubernetes clusters and clouds, with no changes to your applications. By using Kubernetes-native APIs and providing advanced scheduling capabilities, Karmada enables truly open, multi-cloud Kubernetes.
Karmada aims to provide turnkey automation for multi-cluster application management in multi-cloud and hybrid cloud scenarios, with key features such as centralized multi-cloud management, high availability, failure recovery, and traffic scheduling.
As the natural successor of Federation (v1) and KubeFed (v2), Karmada inherited a lot of concepts from these two versions. For example:
- Resource template: Karmada uses Kubernetes Native API definition for its federated resource template, to make it easy to integrate with existing tools that already adopt Kubernetes.
- Propagation Policy: Karmada offers a standalone Propagation(placement) Policy API to define multi-cluster scheduling and spreading requirements.
- Override Policy: Karmada provides a standalone Override Policy API for specializing cluster relevant configuration automation.
For more details please refer to Karmada Concepts.
Key features of Karmada
- Kubernetes Native API Compatible
- Karmada don’t require users have a “federated CRD”
- Seamless integration of existing Kubernetes tool chain, like
- Fruitful Multi-Cluster Scheduling Policies
- Propagate workloads into member clusters as per powerful schedule policies.
- Taint based failover across multi-clusters.
- Supports both Push and Pull cluster management modes
- Push mode: like KubeFed, Karmada control-plane talks to member clusters directly.
- Pull mode: delegate karmada-agent to apply workloads to member clusters.
Most of the features in KubeFed have been reformed in Karmada. In addition, Karmada also provides extra features like, more powerful scheduling policies, Aggregated Kubernetes API, Customizing Resource Interpreter, Multi-Cluster Service Discovery, Multi-Cluster Ingress, etc.
Open Cluster Management: Simplify fleet management across the open hybrid cloud at scale
Open Cluster Management (OCM) provides a framework to enable any capability within the Kubernetes ecosystem to orchestrate itself across multiple clusters and cloud providers. The built-in primitives allow an understanding of cluster inventory, cluster workload placement, and cluster workload distribution. The hub-agent model of OCM allows for new kinds of agent behavior to be injected into managed clusters easily. New behaviors can further configure clusters or collect information about clusters (such as observability data or search indexing). The project also provides two multicluster capabilities out of the box: application delivery, orchestration, and policy-based governance.
Key features of OCM
- Highly scalable, easy-to-use Kubernetes-native API control plane
- No federated CRD required, easy adoption from single cluster to multicluster.
- Distributed “Hub-spoke” architecture.
- Streamlined cluster registration process, compared to KubeFed’s more manual intensive process.
- Powerful workload distribution with cluster placement
- Placement API – Labels, Taints, Tolerations, Spread Policies, and Placements all allow the consumer to extensively schedule across multiple clouds, data centers, and the edge.
- ManifestWork API for dispatching resources from the hub cluster to the spoke clusters.
- Implements Kubernetes-SIG Work API project.
- Flexible extensibility framework
- Modular API design provides an integration point for making other Kubernetes projects multi-cluster aware.
- Addon-framework to help developers to develop extensions for the purpose of working with multiple clusters in custom cases.
- Collaborated with popular projects like Submariner.io, ArgoCD, Istio, etc. to ease their management over multiple clusters.
Differences between Karmada and OCM
Both projects are ready to take up the challenge of managing fleets of clusters across the hybrid and multi-cloud landscape, but they have different philosophies when it comes to solving it. Karmada provides a more complete full stack end to end solution. OCM provides a robust modular framework and APIs that enable other Kubernetes ecosystem projects to integrate with it, to unlock multicluster capabilities.
In the future, there will be many use cases where both Karmada and OCM can be complementary to each other. There is already an ongoing collaboration between both project maintainers in the Kubernetes SIG-Multicluster community to standardize the Work API, which is a project that distributes Kubernetes objects between clusters.
Final thoughts
The Kubernetes community is desperately looking for alternatives to KubeFed. Today we explored two exciting projects Karmada and Open Cluster Management, both of which enhance the scale and the capabilities of multicluster management well beyond what KubeFed provided. We call upon the Kubernetes community to give both projects a try and let’s continue to head towards the multicluster future.
Authors
- David Eads (@deads2k) is a senior principal software engineer at Red Hat and a Kubernetes contributor and co-lead for Kubernetes SIG-apimachinery and an emeritus lead for SIG-auth.
- Kevin Wang (@kevin-wangzefeng) is the lead of the Cloud Native Open Source Team at HUAWEI CLOUD and a Kubernetes contributor. Kevin’s team made many significant contributions to the Kubernetes Federation v1 and v2(KubeFed).
References
- KubeFed
- Karmada
- Open Cluster Management
- Work API
- Kubernetes SIG-multicluster