Kubernetes Mutating Web Hooks to Configure Ingress

I’m working on a project that dynamically creates Pods/Services/Ingress objects using the Kubernetes API.

This all works pretty well, until we have to support a bunch of different Ingress Controllers. Currently the code supports 2 different options.

  1. Using the nginx ingress controller with it set as the default IngressClass
  2. Running on AWS EKS with a ALB Ingress Controller.

If does this by having an if block and a settings flag that says it’s running on AWS where it then injects a bunch of annotations into the Ingress object

apiVersion: networking.k8s.io/v1
kind: Ingress
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: flowforge
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
  name: graceful-whiskered-tern-1370
  namespace: flowforge

While this works it doesn’t scale well as we add support for more types of Ingress Controller that require different annotations to configure them e.g. to use cert-manager to request LetsEncrypt certificates to HTTPS.

Luckily Kubernetes provides a mechanism for modifying objects as they are being created via something called a MutatingAdmissionWebhook. This is a HTTPS endpoint hosted inside the cluster that is passed the object at specific lifetime events and is allowed to modify that object before it is instantiated by the control plane.

There are a few projects that implement this pattern and allow you to declare rules to be applied to objects. Such as KubeMod and Patch Operator from RedHat. I may end up using one of these for the production solution, but as this didn’t sound too complex so I thought I would have a go at implementing a Webhook first myself just to help understand how they work.

Here are the Kubernetes docs on creating Webhooks.

So the first task was to write a simple web app to host the Webhook. I decided to use express.js as that is what I’m most familiar with to get started.

By default the Webhook is a POST to the /mutate route, the body is a JSON object with the AdmissionReview which has the object being created in the request.object field.

Modifications to the original object need to be sent as a base64 JSONPatch.

  apiVersion: admissionReview.apiVersion,
  kind: admissionReview.kind,
  response: {
    uid: admissionReview.request.uid,
    allowed: true,
    patchType: 'JSONPatch',
    patch: Buffer.from(patchString).toString('base64')

The JSONPatch to add the AWS ALB annotations mentioned earlier look like:

    "value":"[{\"HTTPS\":443}, {\"HTTP\":80}]"

A basic hook that adds the AWS ALB annotations fits in 50 lines of code (and some of that is for HTTPS, which we will get to in a moment).

Webhooks need to be called via HTTPS, this means we need to create a server certificate for the HTTP server. Normally we could use something like LetsEncrypt to generate a certificate, but that will only issue certificates that have host names that are publicly resolvable. And since we will be accessing this as a Kubernetes Service that means it’s hostname will be something like service-name.namespace. Luckily we can create our own Certificate Authority and issue certificates that match any name we need because as partof the configuration we can upload our own CA root certificate.

The following script creates a new CA, then uses it to sign a certificate for a service called ingress-mutartor and adds all the relevant `SAN entries that are needed.

cd ca
rm newcerts/* ca.key ca.crt index index.* key.pem req.pem
touch index

openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -out ca.crt -subj "/C=GB/ST=Gloucestershire/O=Hardill Technologies Ltd./OU=K8s CA/CN=CA" 

openssl req -new -subj "/C=GB/CN=ingress-mutator" \
    -addext "subjectAltName = DNS.1:ingress-mutator, DNS.2:ingress-mutator.default, DNS.3:ingress-mustator.default.svc, DNS.4:ingress-mutator.default.svc.cluster.local"  \
    -addext "basicConstraints = CA:FALSE" \
    -addext "keyUsage = nonRepudiation, digitalSignature, keyEncipherment" \
    -addext "extendedKeyUsage = serverAuth" \
    -newkey rsa:4096 -keyout key.pem -out req.pem \

openssl ca -config ./sign.conf -in req.pem -out ingress.pem -batch

If I was building more than one Webhook I could break out the last 2 lines to generate and sign multiple different certificates.

Now we have the code and the key/certificate pair we can bundle them up in the Docker container that can be pushed to a suitable container registry and we then create the deployment YAML files needed to make all this work.

The Pod and Service definitions are all pretty basic, but we also need a MutatingWebhookConfiguration. As well as identifying which service hosts the Webhook it also includes the filter that decides which new objects should be passed to the Webhook to be modified.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
  name: ingress-annotation.hardill.me.uk
- name: ingress-annotation.hardill.me.uk
    - key: kubernetes.io/metadata.name
      operator: In
        - flowforge
  - apiGroups:   [ "*" ]
    apiVersions: [ "networking.k8s.io/v1" ]
    resources:   [ "ingresses" ]
    operations:  [ "CREATE" ]
    scope:       Namespaced
      namespace: default
      name: ingress-mustator
      path: /mutate
    caBundle: < BASE64 encoded CA bundle>
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5
  reinvocationPolicy: "Never"

The rules secition says to match all new Ingress objects and the namespaceSelector says to only apply it to objects from the flowforge namespace to stop us stamping on anything else that might be creating new objects on the cluster.

The caBundle value is the output of cat ca.crt | base64 -w0

This all worked as expected when deployed. So the next step is to remove the hard coded JSONPatch and make it apply a configurable set of different options based on the target environment.

The code for this is all on github here.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.