Kubernetes turns 10: triumphs, trials, and the ongoing battle for security

Member post by Nigel Douglas, Senior Developer Advocate at Sysdig

Kubernetes will celebrate its 10th anniversary at the “KuberTENes Birthday Bash” this week.  This milestone is a perfect time to commemorate the platform’s decade-long journey and the significant advancements it has experienced. It also poses an opportunity to reflect on the security challenges Kubernetes has overcome in the last decade. 

Overview of Kubernetes and Security

Kubernetes, an open-source container orchestration platform, originated from Google’s internal Borg project and was publicly launched in 2014.  The main emphasis was on container orchestration, scalability, and automation, offering a revolutionary method for deploying and managing web applications. Initially, security was not a primary focus, with the emphasis on speed and management for complex applications.

Competing projects like Docker Swarm were also vying for dominance in the open-source orchestration space. However, as Kubernetes emerged as the de facto standard for container orchestration, the critical need for robust security measures became apparent. As more developers – and their companies – wanted to shift to cloud-native computing, adding more security support was essential to support those organizations transitioning from small-scale test environments to large-scale production deployments. As Kubernetes adoption grew rapidly, the community recognized the importance of implementing strong security practices to protect against emerging threats.

Kubernetes has played a pivotal role in the evolution of cloud-native computing. It provided a standardized way to manage containers at scale, fostering the growth of microservices architecture. However, as Kubernetes evolved, security challenges became more complex. The security landscape had to catch up with this rapid development to ensure that applications running on Kubernetes were secure from vulnerabilities and attacks.

Kubernetes and Security Issues

Over the past decade, Kubernetes has faced several significant security challenges. When Kubernetes was first released in 2014, it lacked several critical security features that are now considered essential. Notably, Kubernetes did not initially include Role-Based Access Control (RBAC) and network policies. These features were crucial for managing permissions and securing network traffic within a Kubernetes cluster. Kubernetes introduced Network Policies in version 1.3, released in July 2016 through a merged PR from Casey Davenport of Project Calico. RBAC was introduced in Kubernetes v.1.6, which was released in March 2017.

Kubernetes has seen its share of high profile bugs as well. Notable vulnerabilities included the CVE-2018-1002105, a critical flaw that allowed unauthorised users to escalate their privileges via the Kubernetes API server. Another significant bug was CVE-2020-8554, which allowed attackers to intercept traffic bound for other services. In the latter example, recently-graduated CNCF project Falco was used to detect this Man-in-the-Middle attack against LoadBalancer and Cluster IPs. However, for such a high profile project that underpins so many cloud deployments, the number of critical issues in Kubernetes that could be used by attackers has remained low.

After Pod Security Policies (PSPs) were deprecated in v.1.21 of Kubernetes in favour of more flexible and manageable security solutions. This indicated a shift towards more sophisticated security practices, as the Kubernetes community had identified serious usability problems in PSPs. For example, these included vulnerabilities where users could accidentally grant broader permissions than intended, with users finding it difficult to inspect which PSPs were applied in each situation.

Best Practices for Developers

So how can you improve your Kubernetes security practices across your developer and DevOps teams? Firstly, with the deprecation of PSPs, utilise Pod Security Admission or third-party solutions like OPA/Gatekeeper to enforce security policies. You can define different isolation levels for your pods, and these standards let you define how you want to restrict the behaviour of pods in a clear, consistent fashion.

It should be in place as standard but adopt RBAC across your developers. Implementing role-based access control ensures that users and applications have the minimal permissions necessary to function. Similar to the shortcomings of PSPs, it’s important to avoid using wildcard permissions where possible, avoid using the cluster-admin role, and avoid adding users to the system:masters group to reduce the risk of excessive access leading to security incidents. With access credentials being the first target that attackers go for once they can get access, reducing the number of accounts that can get access to additional resources can prevent lateral movement.

Similarly, use network policies to control traffic between pods, preventing unauthorized communication. Since the beginning of the Kubernetes project, it has defined what is known as a flat network design whereby all workloads can freely communicate with each other. It’s up to DevOps teams to decide which traffic is actually required between pods, and the rest should really be dropped in an implicit network policy design.

Lastly, scanning your images for vulnerabilities should be in place. In addition to managing pod security and access controls, it is crucial to scan container images for vulnerabilities or older, insecure software components. Using tools like the CNCF project Trivy, you can identify and mitigate vulnerabilities in your container images before they are deployed. Trivy performs comprehensive scans and provides actionable insights, helping you maintain secure and compliant container environments.

The Future of Kubernetes Security

It’s hard to say where the Kubernetes project will evolve from here, but it’s fair to say that from the rapid development of the Kubernetes ecosystem we can expect significant changes to come over the next 10 years. For example, the CNCF has graduated several projects like Cilium and Falco, over the last year, reflecting the increasing demand for advanced security solutions within the Kubernetes ecosystem. Cilium offers advanced network security features, while Falco provides runtime threat detection. As these projects graduate from the CNCF, they have satisfied the Due Diligence requirements of the Technical Oversight Committee (TOC) that require stringent security checks on these open-source projects to improve the overall supply-chain security of the Kubernetes ecosystem.

Network security has also evolved over the past decade, from basic Kubernetes-native network policies that block basic port and IP activity to advanced use cases with Cilium, including DNS policies and multi-cluster networking. This is made perfectly clear from the number of training labs provided by the Isovalent team on topics such as Advanced Gateway API Use Cases, BGP Networking, Host Firewalling, Mutual Authentication, and more.

The shift from sidecar containers to a more reliable DaemonSet approach has improved service mesh implementations, enhancing security and efficiency. What started off as just an Istio Service Mesh offering, saw the rise of projects such as Linkerd and Cilium Service Mesh which ultimately expedited the speed at which service mesh technologies evolved to address the demand for security and reliability.

Similarly, Falco has expanded from simply monitoring raw system call activity taking place in the Linux kernel to providing a flexible plugin architecture for sources such as Kubernetes Audit Logs and AWS Cloudtrail logs. Falco’s improved observability through FalcoSidekick UI makes the project more approachable in production environments, whereas the recent introduction of Falco Talon adds a dedicated threat mitigation engine for Kubernetes, allowing for automated mitigation of threats based on Kubernetes primitives such as labelling workloads, gracefully terminating pods, enforcing network policies and more.

Need for OWASP Top 10 for Kubernetes

The OWASP Kubernetes Top 10 was published in 2021 by Jimmy Mesta, Co-founder and CTO at KSOC. This list was created to help security practitioners, system administrators, and software developers prioritise risks associated with Kubernetes deployments. The goal was to address the growing security challenges as Kubernetes adoption surged, providing a structured framework for identifying and mitigating security vulnerabilities within Kubernetes environments.

Since 2021, the OWASP Top 10 guide for Kubernetes has become a crucial resource for security practitioners, focusing on areas such as insecure workload configurations, overly permissive RBAC configurations, and inadequate logging and monitoring, most of which are mentioned above.

Timeline of Key Kubernetes Ecosystem Projects

The timeline below highlights the introduction of key projects that have significantly contributed to the overall security of the Kubernetes ecosystem, enhancing its capabilities in monitoring, security, policy enforcement, and networking.

As Kubernetes Evolves, So Must Security

Kubernetes has not yet reached full maturity, but the strides made in improving its security and the broader cloud-native security landscape are commendable. The platform’s evolution, coupled with the continuous efforts of the CNCF and the community, ensures that Kubernetes remains a robust and secure choice for modern applications. As we celebrate a decade of Kubernetes, it’s also clear that the journey toward better security is ongoing and vital for the future of cloud-native computing.