Guest Post from Weidong Cai and Ye Yin of Tencent

Recently, the Tencent Games container team named Tenc has open sourced the Kubernetes (K8s) multi-cluster scheduling project tensile-kube. This blog will briefly introduce the tensile-kube.

Birth of tensile-kube

There are dozens of online K8s clusters on the Tenc computing platform. These clusters have some fragmented resources that cannot be effectively used. A common scenario is that a job requires N resources, but the remaining resources of the existing clusters A, B, C, etc. do not satisfy N, and the total resources of clusters A, B, and C can also satisfy N. In addition, we have another scenario that users have services that need to be published to multiple clusters at the same time.

In fact, the industry has had similar needs for a long time and solutions such as federation have been born. However, these will be a large transformation cost for the existing K8s while adding a lot of complicated CRDs. Therefore, we independently developed a more lightweight solution named tensile-kube based on virtual-kubelet.

Introduction to virtual-kubelet (VK)

VK is a basic library that is open sourced by Microsoft. Its implementation is usually called a provider. The early VK included various providers. At present, the VK community has separated it into three parts: virtual-kubelet, node-cli and provider. The virtual-kubelet provides an abstract capability. By implementing and register the provider in a K8s cluster, users’ pods can be scheduled to this virtual node. The core capability of virtual-kubelet is the life cycle control of Node and Pod.

Architecture of tensile-kube

Tensile-kube architecture

There are 4 core components of tensile-kube:

Notice:

Features of tensile-kube

All of these make tensile-kube achieve the scheduling and management of multiple clusters easily. In addition, it is suitable for most scenarios.

Main abilities of tensile-kube

Other possible applications

Feature work

It has been 6 years since Tenc brought K8s into production environment in 2014. Tensile-kube is born from our years of container-related experience. At present, based on tensile-kube, we have expanded the original cluster from two thousand nodes to more ten thousand nodes. Previously, we have submitted many PRs to the VK community to expand VK functions and optimize its performance. In the future, we will continue to improve VK and tensile-kube in the following aspects:

  1. Multi-cluster scheduler optimization;
  2. Real-time optimization of provider status synchronization;
  3. Resource name conflicts in sub clusters.

Project address: https://github.com/virtual-kubelet/tensile-kube