Posted on September 29, 2020 by Mohamed Ahmed

CNCF projects highlighted in this post

Kubernetes logo

Guest post originally posted on Magalix blog by Mohamed Ahmed

In this section of our OPA series, we define policies that ensure that no bad Ingress definitions will be deployed to our cluster. If you haven’t already done so, please go through our previous articles in this series to learn more about OPA, and how it can be integrated with Kubernetes to enforce policies. This article assumes that you have a working knowledge of Kubernetes and OPA, and that you already have admin access to a Kubernetes cluster that has OPA deployed. We also assume that an Ingress controller is installed (in the lab, we used the nginx controller).

As you probably know, the Kubernetes Ingress controller allows you to serve different URLs on the same Load Balancer. It receives the HTTP request and routes it to the appropriate backend service depending on the path or the host header. Obviously, this saves you a lot of extra cost required to create a Load Balancer for each public-facing service. However, and like Uncle Ben said:

” With great power, comes great responsibility “

In the rest of this article, we get to know some of the problems that may occur when working with Ingresses in the absence of an administrative policy.

Requirement: No Route Conflicts Exist Among Ingress Resources

So, you have your Ingress controller set, and users start creating Ingress resources for different routes – as more and more routes are defined, there is a chance of collision. For example, having an Ingress with oscorp.com host defined in one namespace, and another Ingress with the same host defined in another namespace. OPA can help here by creating a policy that intercepts the request for a new Ingress and:

  1. Checking the existing hosts in all other Ingresses.
  2. Determining if there’s an already defined host in an existing Ingress that matches the one in the request.
  3. Depending on the result, the request is approved or denied.

The following diagram depicts this workflow:

Diagram showing enforce ingress best practices using OPA

Describing The Policy In Rego

When we need to define a policy for OPA to execute, we need to use the Rego language, which was designed specifically for this purpose. Our enforce-ingress-hostnames.rego file may look as follows:

package kubernetes.admission

import data.kubernetes.ingresses

deny[msg] {
	# We define variables that we use as keys when iterating through the ingresses dictionary
    some other_ns, other_ingress
    # We are only interested in Ingress requests
    input.request.kind.kind == "Ingress"
    # that asks for creation (other requests should be ignored by this policy)
    input.request.operation == "CREATE"
    # Extract the host part of the requests JSON object
    host := input.request.object.spec.rules[_].host
    # Get all the existing ingresses. Make sure you identify the namespace (other_ns) and the ingress name (other_ingress)
    ingress := ingresses[other_ns][other_ingress]
    # We are not interested in Ingress requests in the same namespace
    other_ns != input.request.namespace
    # Do we have an existing ingress that has a host matching the one in the ingress definition?
    ingress.spec.rules[_].host == host
    # If yes, then this policy is vilated. We need to send an informative message to the client detailing which part of the ingress violated the policy
    msg := sprintf("invalid ingress host %q (conflicts with %v/%v)", [host, other_ns, other_ingress])
}

Copy

As usual, we’ve added comments to explain what each code line does, but let’s focus on some important points:

Applying The Policy

Before uploading our Rego file to OPA, we need to instruct the kube-mgmt sidecar container to load the existing ingresses, so that they are available to our code. This can be done by modifying the command arguments of the container so that they look as follows:

- name: kube-mgmt
     image: openpolicyagent/kube-mgmt:0.8
       args:
         - "--replicate=extensions/v1beta1/ingresses"

Copy

If you’ve been following this article series from the start, you can apply the above change either by editing the OPA deployment: kubectl -n opa edit deployment opa or by making the change in the YAML file and applying it.

Now, let’s apply the policy by creating a ConfigMap in the OPA namespace that holds the contents of our Rego file:

kubectl create configmap  enforce-ingress-hostnames --from-file=enforce-ingress-hostnames.rego

Copy

Then, ensure that OPA did not complain about any syntax errors:

kubectl get cm enforce-ingress-hostnames -o json | jq '.metadata.annotations'
{
  "openpolicyagent.org/policy-status": "{\"status\":\"ok\"}"
}

Copy

Exercising The Policy

In order to ensure that our policy is working as expected, we apply the following procedure:

  1. Create an Ingress resource in the default namespace and ensure that the creation request was admitted successfully.
  2. Create a new Ingress in another namespace. Make sure that you use the same host in the definition file.
  3. The request should be denied, and you should see an informative message.

Our Ingress definition looks as follows:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-ok
  namespace: production
spec:
  rules:
  - host: oscorp.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80

Copy

$ kubectl apply -f ingress.yaml
ingress.extensions/ingress-ok created
$ kubectl create ns production
namespace/production created

Copy

Now, modify the ingress.yaml file so that the namespace is production instead of default and apply the definition again:

$ kubectl apply -f ingress.yaml
Error from server (invalid ingress host "oscorp.com" (conflicts with default/ingress-ok)): error when creating "ingress.yaml": admission webhook "validating-webhook.openpolicyagent.org" denied the request: invalid ingress host "oscorp.com" (conflicts with default/ingress-ok)

Copy

TL;DR