Member post originally published on Nirmata’s blog by Boris Kurktchiev
Introduction
As organizations increasingly turn to Kubernetes to deploy and manage containerized applications, they face unique challenges in efficiently allocating resources. The dynamic nature of Kubernetes often leads to inefficient defaults, underperforming applications, and inflated cloud bills. While Kubernetes and the CNCF ecosystem provide many tools, most organizations require help getting started.
This blog explores these complexities and presents an easy-to-adopt and intuitive solution that leverages Kyverno policies to simplify and automate resource management in Kubernetes environments.
- Conflicting goals: In most organizations, the platform team manages Kubernetes clusters and their worker nodes, while application teams deploy and manage workloads. Application owners will want to allocate the maximum available resources to optimize performance, while platform teams want to minimize wastage and optimize resource usage across workloads. Given these competing requirements, who is responsible for right-sizing resources?
- Dynamic Workloads and Variable Utilization: Cloud-native workloads are dynamic and respond quickly and automatically to fluctuating resource demands. This volatility often results in over or under-provisioning, creating inefficiencies. For instance, applications may require more CPU and memory during peak times, while off-peak hours see these resources underutilized. Additionally, the same application may behave differently in different environments.
- Misconfigured workloads: Kubernetes configuration is declarative and fine-grained, supporting many workloads and use cases. The flexible configuration options and non-optimal defaults make it easy for application teams to misconfigure resources, leading to unexpected behaviors. For example, using an emptyDir volume type or a misconfigured Pod Disruption Budget (PDB) may prevent scaling down pods, leading to wasted resources. Similarly, a missing resource request can impact how Kuberenets interprets the workload’s desired Quality of Service (QoS) requirements and cause eviction under load.
The Need for Policy-Driven Automation
Kubernetes policies are configuration resources that manage other configurations or workloads. Policies allow application developers, platform teams, and other stakeholders, such as security, to collaborate on organizational requirements and goals.
Using policies, platform teams can enable common behaviors and best practices across all workloads.
The ideal solution must be straightforward and robust for a complex topic like resource management, providing immediate value to the platform and application teams without steep learning curves. It will meet the following requirements:
- Automatically provide resource recommendations based on actual usage data
- Allow flexible opt-in and opt-out capabilities
- Provide standardized reporting via native APIs and tools
- Enable self-service capabilities for collaboration across teams
- Ease the knowledge gap for newer development teams
- Balance simplicity and ease of use with powerful cloud native capabilities
Technical Deep Dive
The solution leverages the Kubernetes Vertical Pod Autoscaler (VPA).
The Kubernetes VPA consists of three components: 1) Recommender, 2) Updater, and 3) Admission Plugin.
For this solution, only the Recommender component is used. The VPA updater and webhook are not required.
An initial and effective strategy is shown below. Organizations can use Kyverno policies that automate the management of VPA resources in recommendation-only mode and leverage policy reports to inform application teams of violations and misconfigurations. Application teams can then address policy violations by configuring resources effectively, which then clears the violations.
This solution cleanly segments responsibility and delivers helpful information to busy developers using native tools and APIs, eliminating the need to open and check a solution-specific interface.
More advanced organizations can easily extend the solution to automatically adjust (mutate) resources. Additional flexibility can be achieved using the VPA updater or policies.
The following section describes the policies used for this initial strategy, and future posts will dive into advanced topics.
Kyverno Policies in Action
Kyverno policies are the backbone of this solution, serving as versatile tools that can validate, mutate, generate, clean up resources, and verify image signatures and attestations. They are easy to integrate with Kubernetes controllers like the VPA.
The solution uses two Kyverno policies described below:
Generate VerticalPodAutoscaler Resources
The Kyverno policy automatically generates VPA resources for pod controllers like Deployments, StatefulSets, and DaemonSets. This automated approach ensures consistent policy application across clusters:
This policy automatically creates a VPA resource for each pod controller, allowing the VPA Recommender to observe and provide recommendations without making automatic changes.
The policy is available in this GitHub file.
NOTE: The policy rule does not set a minimum or maximum allowed per container. Instead, a global minimal value is configured for the VPA recommender.
Check Resource Configurations
Another Kyverno policy performs periodic checks that ensure that the configured container resources align with VPA recommendations, thereby maintaining optimal performance and cost efficiency.
In this issue comment, Marco Voelz described the logic used by the policy rule to determine when resource configurations should be adjusted.
The policy rule checks if the resource request exceeds the upper or lower bounds and recommends the target settings, preventing overprovisioning or underprovisioning.
Since the VPA recommender improves its results over time, the policy rule evaluates recommendations after a minimum period. A 20% variation (customizable) is also allowed to dampen reporting noise.
The full policy is available in this GitHub file.
Integration with Nirmata Policy Manager
Nirmata’s unified governance platform builds on Kyverno and offers several capabilities organizations need to collaborate and manage policies effectively. The Nirmata Policy Manager provides global policy enforcement, unified governance, and workflows for DevSecOps collaboration across clusters, pipelines, and cloud services.
With Nirmata, Kubernetes namespaces and workloads can be easily auto-assigned to application teams, and all policy violations can be routed to them using their interface of choice.
Connecting the solution to Nirmata Policy Manager (NPM) is straightforward. After onboarding your cluster to NPM, you can easily categorize violations under a category called ‘Resource Optimization.’
Key Benefits
The Kubernetes landscape is competitive, with solutions ranging from open-source tools to enterprise-grade platforms. However, Nirmata’s policy-driven resource optimization solution stands out due to its unique approach:
Automation and Integration: Kyverno’s seamless integration with VPA provides fully automated recommendations and adjustments, eliminating misconfigurations and wasted resources.
User Experience: Nirmata prioritizes simplicity and ease of use, offering a user-friendly interface that reduces complexity and accelerates team adoption.
Flexibility: Unlike rigid solutions, the solution is highly customizable, catering to diverse workload needs and deployment scenarios.
Comparison to Alternatives
While several Kubernetes cost management solutions exist, they do not prevent resource misconfigurations or help optimize resource settings.
The Kubernetes VPA Updater and webhook components are powerful, but they may only sometimes be suitable for use in production, are not easily customizable per workload, and lack reporting.
Other solutions integrating with the VPA provide visibility but require access to GUIs to check recommendations.
Extensions and Future Work
The solution presented in this blog post provides an excellent way for organizations looking to start right-sizing resource allocations based on usage patterns.
Automatically Updating Resources
The Kubernetes VPA Updater already allows automated resource updates. Combining the VPA with Kyverno policies can also update resources and provide more flexibility or controls when needed.
In-place Pod Resource Resize
The In-Place Pod Resource Resize feature allows resizing CPU and memory without restarts. This feature enables new behaviors which can be combined with policy logic.
Cluster Proportional Autoscaling
The Cluster Proportional Autoscaler allows horizontal and vertical auto-scaling of a workload based on the cluster size. This is useful for running the same workload across different clusters of varying sizes and can be integrated with policy logic.
Horizontal Pod Autoscaler (HPA) Interactions
The best practice guidance is using a different horizontal and vertical auto-scaling metric.
However, you can safely use the VPA recommender with an HPA for the same workload to gain insights into resource usage patterns and make recommendations based on long-term observations. In other cases, using a combination of horizontal and vertical scaling may be desirable. For example, you should apply vertical scaling and create multiple replicas when a limit is used. More research is required on these interactions to establish best practices.
Conclusion and Strategic Call to Action
In today’s competitive landscape, cloud costs can rapidly escalate beyond intended bounds, and effective resource management is an operational necessity and can be a strategic advantage.
Nirmata’s policy-driven resource optimization is designed to address this challenge head-on, offering a sophisticated yet user-friendly approach to Kubernetes resource optimization that aligns seamlessly with technical requirements and business objectives. The easy-to-adopt nature of the solution ensures that it doesn’t introduce extra steps and hurdles for end users, making them feel at ease with the solution. This frees them to do what they want, knowing their workloads will always be right-sized.
In an era where cloud costs and application performance are critical business concerns, effective Kubernetes resource optimization is a strategic imperative. By automating resource management with Kyverno policies, Nirmata empowers organizations to achieve their cloud-native goals while staying competitive in today’s fast-paced market.
Visit the Nirmata website to learn more about Nirmata Policy Manager, Nirmata Enterprise for Kyverno, and Kyverno OSS.
Feel free to contact us at hello@nirmata.com for a discussion and demo of the proposed solution!