Member post by AccuKnox
Introduction:
In the realm of cybersecurity, ensuring that virtualized or cloud-based infrastructures security is paramount. One crucial aspect is safeguarding applications where most of our crown-jewel sits and are susceptible to dynamic changes. In modern infrastructure Kubernetes deployments are gaining popularity by over 55% annual rate (ref). And hence, securing Kubernetes deployment (applications) and going one-step further to protecting the underlying host system becomes critical. This technical blog explores the enhanced security the user can get by combined use of AWS Bottlerocket and KubeArmor (a runtime security engine). Both are powerful tools that work in harmony to provide robust runtime protection for Kubernetes environments end-to-end. AWS Bottlerocket is originally developed and currently maintained by Amazon Web Services (AWS) and is well-optimized for use with Amazon Elastic Kubernetes Service (EKS), although it can also be used with other container orchestration platforms like Kubernetes.
What is AWS Bottlerocket?
AWS Bottlerocket: A Security-Focused Linux OS Distribution which is tailored for securely running containerized workloads in the cloud environment. It was developed with a view to have a security-focused, open-source Linux-based operating system which will be specifically designed for running any type of containerized workloads in all kinds of cloud environments. AWS Bottlerocket does not allow installing any package management tools or maintenance binaries on the host system. Users can only utilize these tools through a “control pod” or an admin port, ensuring strict control over their underlying os infrastructure.
What is the significance of AWS Bottlerocket?
Enhanced Security: One of the primary reasons for the significance of Bottlerocket is its strong emphasis on security. It is purpose-built to provide a secure and minimalistic environment for running containers. By reducing the attack surface and removing unnecessary components, Bottlerocket helps mitigate potential vulnerabilities and enhances the overall security posture of containerized workloads.
- Has a read-only filesystem and gets checked via dm-verity at bootup
- No SSH Server and has Control Container enabled by default. Could enable AWS SSM agent to run shell sessions
- Admin Container is disabled by default
- Configured with Security-Enhanced Linux (SELinux) policies for isolation
Bottlerocket can be used with –
- Amazon EKS
- Amazon ECS
- VMware or
- Baremetal
Secure Updates with The Update Framework (TUF)
Another essential aspect of AWS Bottlerocket’s security is the implementation of secure updates for the operating system image and root file system. This is achieved through The Update Framework (TUF), a CNCF graduated project designed for secure system updates. Leveraging TUF ensures that updates are tamper-proof and come from trusted sources.
Locking Down with Kernel Integrity Features
To prevent unauthorized access and tampering, AWS Bottlerocket uses a kernel feature called “integrity,” which prohibits processes from writing to the kernel memory or loading/unloading kernel modules. Additionally, the distribution utilizes “dm-verity” (device mapper verity), a kernel module feature that offers integrity verification of the root file system through cryptographic digests. Any attempt to write to sensitive file paths in the root file system is denied and reported.
What is KubeArmor?
KubeArmor is a runtime security engine which is currently a CNCF sandbox project. The aim of KubeArmor is to implement Zero Trust Security by auto discovering application behavior and auto generating granular policies based on the observed app behavior. These policies are usually enforced as guardrails using Linux Security Modules (LSMs) to prevent any unknown binary execution, or application behavior
Leverage the unfair KubeArmor Advantage
KubeArmor at its core support observability of the application behavior via eBPF and enforces granular policies at the kernel-level for in-line mitigation of those zero day attacks in a proactive way.
As Zero Trust security calls for ‘never trust always verify” and ‘assume breach’ requirements its imperative to have multi-layer defense for the modern workload. Here is how KubeArmor helps its customer to reach their Zero Trust journey
- Application Level Security
- Least permissive access to crown jewels and
- Granular controls for Application Hardening
- Application Isolation and containing the blast radius
- Transport Level Security –
- Use of secure endpoints
- Ensure secure TLS and certification configuration
- Network Security
- Microsegmentation with Ingress/egress control
- Process based Network Access Whitelisting
- System Security
- Process Whitelisting
- Files integrity or Volume mont access whitelisting
- Kernel security sensitive access primitives whitelisting
Problem Statement:
A Remote Code Execution and other attack vectors were detected within an already ‘secured’ cluster and code repositories. Customer X helps its users to detect the delta difference the PR (pull request) will be impacting the main repository and highlight the security implications of the GitHub action.
Upon any trigger such as PR, Customer X Atlas module (depicted below) invokes a pod that has 2 containers –
- Init-dependencies container
The functionality of Init-dependencies container is to clone the target user’s repo and its associated dependencies locally and then exit. These repo and dependencies are downloaded into a volume mount point.
- Analyzer container
This volume mount point location is then assessed by Analyzer container
Point of concern:
After the dependencies are downloaded, there are events where there is a “postscript” that could be invoked and an attacker could leverage this “postscript” to launch an potential (RCE) attack.
Their requirements to mitigate such attacks were well articulated and thought-upon which are as follows –
- How to stop unknown binaries and reverse shell from getting executed
- How to whitelist specific binaries from Init-dependencies container to access the repo and dependencies folder
- How to apply those granular controls on specific Init-dependencies container and not on Analyzer container within a same pod
Solution:
- How to stop unknown binaries and reverse shell from getting executed
- KubeArmor can block *.binaries from execution
- How to whitelist specific binaries from Init-dependencies container to access the repo and dependencies folder
- KubeArmor can help in whitelisting specific binary to access specific directories
- How to apply those granular controls on specific Init-dependencies container and not on Analyzer container within a same pod
- KubeArmor can also help in applying those granular controls on per Container basis [Refer – Container based Policy].
While AWS Bottlerocket safeguards the host and worker nodes, KubeArmor steps in to protect the Kubernetes pods and their applications. Many applications installed by vendors or users lack sufficient security understanding, making them vulnerable. KubeArmor minimizes the attack surface of such pods by allowing only specific processes to execute, access certain sensitive file paths, and use specific network primitives.
- Securing Kubernetes Secrets and Service Account Tokens
KubeArmor secures Kubernetes service account tokens and secrets by tightly controlling access to these critical resources. The security policies can restrict access to a specific set of processes, ensuring only authorized entities can interact with sensitive data.
- Applying Least Permissive Policies
KubeArmor supports the implementation of least permissive policies, following the tenets of zero-trust security. This approach grants explicit access to a predetermined set of binaries and operations while denying all other activities, thus minimizing potential vulnerabilities.
Leveraging eBPF for Enhanced Visibility and Observability
KubeArmor utilizes eBPF (extended Berkeley Packet Filter) to enhance observability and visibility. eBPF is a powerful technique that provides real-time telemetry and alerting, replacing older mechanisms like kernel audit. This ensures comprehensive monitoring without compromising system performance.
Demonstration: Combining AWS Bottlerocket and KubeArmor
To demonstrate the synergy between AWS Bottlerocket and KubeArmor, we present a hands-on demo within an EKS cluster with Bottlerocket nodes.
Scenario 1: Blocking Access to Kubernetes Service Account Tokens
In this scenario, we use KubeArmor to implement a policy that denies access to the Kubernetes service account token. This token, mounted within a predefined volume bound point, is vital for interacting with the Kubernetes API server. The policy effectively blocks any unauthorized access to this critical token.
Scenario 2: Denying Execution of Certain Processes
KubeArmor is employed to prevent the execution of certain processes like “apt” and “apt-get” within pods. These package management tools are unnecessary and can increase the attack surface, making them unfit for production environments. The policy restricts their execution, minimizing potential risks.
Scenario 3: Enabling Specific Network Primitives
In this final scenario, we showcase how KubeArmor allows specific processes, like Apache and Ping, to access certain network primitives (UDP, TCP, and RAW). This enables fine-grained control over network operations while denying access to other protocols, enhancing security.
Conclusion:
By combining AWS Bottlerocket and KubeArmor, Kubernetes deployments can achieve robust runtime protection. While Bottlerocket ensures a secure foundation for the host and worker nodes, KubeArmor provides granular control over pods and applications, enhancing overall security in the cloud-based ecosystem. The powerful features of both tools complement each other, creating a formidable defense against cybersecurity threats within Kubernetes environments. KubeArmor provides enhanced security by using BPF-LSM to protect Bottlerocket containers from within by limiting system behavior with respect to processes, files, etc. For e.g., a k8s security access token that is mounted within the pod is accessible by default across all the containers. KubeArmor can restrict access to such tokens only for certain processes. Similarly KubeArmor can be used to protect other sensitive information e.g., k8s secrets, x509 certs, within the container.