This guest post was written by Viswajith Venugopal, Member Of Technical Staff at StackRox and was originally posted on Stackrox.
The container orchestrator war is over, and Kubernetes has won. With companies large and small rapidly adopting the platform, security has emerged as an important concern — partly because of the learning curve inherent in understanding any new infrastructure, and partly because of recently announced vulnerabilities.
Kubernetes brings another security dynamic to the table — its defaults are geared towards making it easy for users to get up and running quickly, as well as being backward compatible with earlier releases of Kubernetes that lacked important security features. Consequently, many important Kubernetes configurations are not secure by default.
One important configuration that demands attention from a security perspective is the network policies feature. Network policies specify how groups of pods are allowed to communicate with each other and other network endpoints. You can think of them as the Kubernetes equivalent of a firewall.
On a side note, if you haven’t already done so, upgrade to the latest Kubernetes version because some of the most critical Kubernetes security issues have been addressed by recent Kubernetes updates.
How to set up network policies
We lay out here a step-by-step guide on how to set up network policies. The network policy spec is intricate, and it can be difficult to understand and use correctly. In this guide, we provide recommendations that significantly improve security. Users can easily apply these recommendations without needing to know the spec in detail.
A quick note: this guide focuses just on ingress network policies. When starting out, the biggest security gains come from applying ingress policies, so we recommend focusing on them first, and then adding egress policies. We will discuss egress policies in detail and provide recommendations in a subsequent post in this series.
Use a network plugin that supports network policies
First things first — use a network plugin that actually enforces network policies. Although Kubernetes always supports operations on the NetworkPolicy resource, simply creating the resource without a plugin that implements it will have no effect. Example plugins include Calico, Cilium, Kube-router, Romana and Weave Net.
“Isolate” your pods
Each network policy has a podSelector field, which selects a group of (zero or more) pods. When a pod is selected by a network policy, the network policy is said to apply to it.
Each network policy also specifies a list of allowed (ingress and egress) connections. When the network policy is created, all the pods that it applies to are allowed to make or accept the connections listed in it. In other words, a network policy is essentially a whitelist of allowed connections — a connection to or from a pod is allowed if it is permitted by at least one of the network policies that apply to the pod.
This tale, however, has an important twist: based on everything described so far, one would think that, if no network policies applied to a pod, then no connections to or from it would be permitted. The opposite, in fact, is true: if no network policies apply to a pod, then all network connections to and from it are permitted.
This behavior relates to the notion of “isolation”: pods are “isolated” if at least one network policy applies to them; if no policies apply, they are “non-isolated”. Network policies are not enforced on non-isolated pods. Although somewhat counter-intuitive, this behavior exists to make it easier to get a cluster up and running — a user who does not understand network policies can run their applications without having to create one.
Therefore, we recommend you start by applying a “default-deny-all” network policy. The effect of the following policy specification is to isolate all pods, which means that only connections explicitly whitelisted by other network policies will be allowed.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
spec:
podSelector: {}
policyTypes:
- Ingress
Without such a policy, it is very easy to run into a scenario where you delete a network policy, hoping to forbid the connections listed in it, but find that the result is that all connections to some pods suddenly become permitted — including ones that weren’t allowed before. Such a scenario occurs when the network policy you deleted was the only one that applied to a particular pod, which means that the deletion of the network policy caused the pod to become “non-isolated”.
Important Note: Since network policies are namespaced resources, you will need to create this policy for each namespace. You can do so by running kubectl -n create -f for each namespace.
Explicitly allow internet access for pods that need it
With just the default-deny-all policy in place in every namespace, none of your pods will be able to talk to each other or receive traffic from the Internet. For most applications to work, you will need to allow some pods to receive traffic from outside sources. One convenient way to permit this setup would be to designate labels that are applied to those pods to which you want to allow access from the Internet and to create network policies that target those labels. For example, the following network policy allows traffic from all (including external) sources for pods having the `networking/allow-internet-access=true` label:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: internet-access
spec:
podSelector:
matchLabels:
networking/allow-internet-access: "true"
policyTypes:
- Ingress
ingress:
- {}
For a more locked-down set of policies, you would ideally want to specify more fine-grained CIDR blocks as well as explicitly list out allowed ports and protocols. However, this policy provides a good starting point, with much greater security than the default.
Explicitly allow necessary pod-to-pod communication
After taking the above steps, you will also need to add network policies to allow pods to talk to each other. You have a few options for how to enable pod-to-pod communications, depending on your situation:
If you don’t know which pods need to talk to each other
In this case, a good starting point is to allow all pods in the same namespace to talk to each other and explicitly whitelist communication across namespaces, since that is usually more rare. You can use the following network policy to allow all pod-to-pod communication within a namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-same-namespace
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- podSelector: {}
If You Know The Sources and Sinks for Communication
Often, communication between pods in an application follows a hub-and-spoke paradigm, with some central pods that many other pods need to talk to. In this case, you could consider creating a label which designates pods that are allowed to talk to the “hub.” For example, if your hub is a database pod and has an app=db label, you could allow access to the database only from pods that have a networking/allow-db-access=true label by applying the following policy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-db-access
spec:
podSelector:
matchLabels:
app: "db"
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
networking/allow-db-access: "true"
You could do something similar if you have a server that initiates connections to many other pods. If you want to explicitly whitelist the pods that the server is allowed to talk to, you can set the networking/allow-server-to-access=true label on them, and apply the following network policy (assuming your server has the label app=server) on them:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-server-to-access
spec:
podSelector:
matchLabels:
networking/allow-server-to-access: "true"
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: "server"
If you know exactly which connections should be allowed
Within the same namespace
Advanced users who know exactly which pod-to-pod connections should be allowed in their application can explicitly allow each such connection. If you want pods in deployment A to be able to talk to pods in deployment B, you can create the following policy to whitelist that connection, after replacing the labels with the labels of the specific deployment:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-server-to-access
spec:
podSelector:
matchLabels:
deployment-b-pod-label-1-key: deployment-b-pod-label-1-value
deployment-b-pod-label-2-key: deployment-b-pod-label-2-value
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
deployment-a-pod-label-1-key: deployment-a-pod-label-1-value
deployment-a-pod-label-2-key: deployment-a-pod-label-2-value
Across namespaces
To allow connections across namespaces, you will need to create a label for the source namespace (unfortunately, Kubernetes does not have any labels on namespaces by default) and add a namespaceSelector query next to the podSelector query. To label a namespace, you can simply run the command: kubectl label namespace networking/namespace=
With this namespace label in place, you can allow deployment A in namespace N1 to talk to deployment B in namespace N2 by applying the following network policy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-n1-a-to-n2-b
namespace: N2
spec:
podSelector:
matchLabels:
deployment-b-pod-label-1-key: deployment-b-pod-label-1-value
deployment-b-pod-label-2-key: deployment-b-pod-label-2-value
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
networking/namespace: N1
podSelector:
matchLabels:
deployment-a-pod-label-1-key: deployment-a-pod-label-1-value
deployment-a-pod-label-2-key: deployment-a-pod-label-2-value
What about new deployments?
Although explicitly whitelisting connections in this manner is great for security, this approach does affect usability. When you create new deployments, they will not be able to talk to anything by default until you apply a network policy. To mitigate this potentially frustrating user experience, you could create the following pair of network policies, which allow pods labelled networking/allow-all-connections=true to talk to all other pods in the same namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-from-new
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
networking/allow-all-connections: "true"
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-to-new
spec:
podSelector:
matchLabels:
networking/allow-all-connections: "true"
policyTypes:
- Ingress
ingress:
- from:
- podSelector: {}
You can then apply the networking/allow-all-connections=true label to all newly created deployments, so that your application works until you create specially crafted network policies for them, at which point you can remove the label.
Summary
While these recommendations provide a good starting point, network policies are a lot more involved. If you’re interested in exploring them in more detail, be sure to check out the Kubernetes tutorial as well as some handy network policy recipes.