Kubernetes Mutating Web Hooks to Configure Ingress

I’m working on a project that dynamically creates Pods/Services/Ingress objects using the Kubernetes API.

This all works pretty well, until we have to support a bunch of different Ingress Controllers. Currently the code supports 2 different options.

  1. Using the nginx ingress controller with it set as the default IngressClass
  2. Running on AWS EKS with a ALB Ingress Controller.

If does this by having an if block and a settings flag that says it’s running on AWS where it then injects a bunch of annotations into the Ingress object

apiVersion: networking.k8s.io/v1
kind: Ingress
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: flowforge
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
  name: graceful-whiskered-tern-1370
  namespace: flowforge

While this works it doesn’t scale well as we add support for more types of Ingress Controller that require different annotations to configure them e.g. to use cert-manager to request LetsEncrypt certificates to HTTPS.

Luckily Kubernetes provides a mechanism for modifying objects as they are being created via something called a MutatingAdmissionWebhook. This is a HTTPS endpoint hosted inside the cluster that is passed the object at specific lifetime events and is allowed to modify that object before it is instantiated by the control plane.

There are a few projects that implement this pattern and allow you to declare rules to be applied to objects. Such as KubeMod and Patch Operator from RedHat. I may end up using one of these for the production solution, but as this didn’t sound too complex so I thought I would have a go at implementing a Webhook first myself just to help understand how they work.

Here are the Kubernetes docs on creating Webhooks.

So the first task was to write a simple web app to host the Webhook. I decided to use express.js as that is what I’m most familiar with to get started.

By default the Webhook is a POST to the /mutate route, the body is a JSON object with the AdmissionReview which has the object being created in the request.object field.

Modifications to the original object need to be sent as a base64 JSONPatch.

  apiVersion: admissionReview.apiVersion,
  kind: admissionReview.kind,
  response: {
    uid: admissionReview.request.uid,
    allowed: true,
    patchType: 'JSONPatch',
    patch: Buffer.from(patchString).toString('base64')

The JSONPatch to add the AWS ALB annotations mentioned earlier look like:

    "value":"[{\"HTTPS\":443}, {\"HTTP\":80}]"

A basic hook that adds the AWS ALB annotations fits in 50 lines of code (and some of that is for HTTPS, which we will get to in a moment).

Webhooks need to be called via HTTPS, this means we need to create a server certificate for the HTTP server. Normally we could use something like LetsEncrypt to generate a certificate, but that will only issue certificates that have host names that are publicly resolvable. And since we will be accessing this as a Kubernetes Service that means it’s hostname will be something like service-name.namespace. Luckily we can create our own Certificate Authority and issue certificates that match any name we need because as partof the configuration we can upload our own CA root certificate.

The following script creates a new CA, then uses it to sign a certificate for a service called ingress-mutartor and adds all the relevant `SAN entries that are needed.

cd ca
rm newcerts/* ca.key ca.crt index index.* key.pem req.pem
touch index

openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -out ca.crt -subj "/C=GB/ST=Gloucestershire/O=Hardill Technologies Ltd./OU=K8s CA/CN=CA" 

openssl req -new -subj "/C=GB/CN=ingress-mutator" \
    -addext "subjectAltName = DNS.1:ingress-mutator, DNS.2:ingress-mutator.default, DNS.3:ingress-mustator.default.svc, DNS.4:ingress-mutator.default.svc.cluster.local"  \
    -addext "basicConstraints = CA:FALSE" \
    -addext "keyUsage = nonRepudiation, digitalSignature, keyEncipherment" \
    -addext "extendedKeyUsage = serverAuth" \
    -newkey rsa:4096 -keyout key.pem -out req.pem \

openssl ca -config ./sign.conf -in req.pem -out ingress.pem -batch

If I was building more than one Webhook I could break out the last 2 lines to generate and sign multiple different certificates.

Now we have the code and the key/certificate pair we can bundle them up in the Docker container that can be pushed to a suitable container registry and we then create the deployment YAML files needed to make all this work.

The Pod and Service definitions are all pretty basic, but we also need a MutatingWebhookConfiguration. As well as identifying which service hosts the Webhook it also includes the filter that decides which new objects should be passed to the Webhook to be modified.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
  name: ingress-annotation.hardill.me.uk
- name: ingress-annotation.hardill.me.uk
    - key: kubernetes.io/metadata.name
      operator: In
        - flowforge
  - apiGroups:   [ "*" ]
    apiVersions: [ "networking.k8s.io/v1" ]
    resources:   [ "ingresses" ]
    operations:  [ "CREATE" ]
    scope:       Namespaced
      namespace: default
      name: ingress-mustator
      path: /mutate
    caBundle: < BASE64 encoded CA bundle>
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5
  reinvocationPolicy: "Never"

The rules secition says to match all new Ingress objects and the namespaceSelector says to only apply it to objects from the flowforge namespace to stop us stamping on anything else that might be creating new objects on the cluster.

The caBundle value is the output of cat ca.crt | base64 -w0

This all worked as expected when deployed. So the next step is to remove the hard coded JSONPatch and make it apply a configurable set of different options based on the target environment.

The code for this is all on github here.

Joining the Herd

With the news about what’s happening on the BirdSite… I thought it might be time to explore what the state of the art alternative is these days.

As an aside, this isn’t the first time I’ve looked at Twitter alternatives, back in the very early days of Twitter I build and ran a project called BlueTwit inside IBM. This was ground up clone that was there to see how a micro blogging platform (how quaint a term…) would work in a large organisation. It had a Firefox plugin client, supported 250+ characters long before Twitter. The whole was written in Java ran on a modest AMD64 box under my desk for many years and was even played with by the IBM Research team before similar functionality ended up in IBM/Lotus Connections. (Someday I should do a proper writup about it…)

Anyway back to the hear and now… I have a propensity for wanting to know how things work under the covers (which is why I run my own web, DNS, mail, SIP, finger and, gopher server). So I thought I’d have a go at running my own Mastodon server for a little while to see how it all fits together.

A little digging shows that a Mastodon instance isn’t just one thing, it needs the following:

  • Mastodon Web interface
  • Mastodon Streaming interface
  • Mastodon Job server (sidekiq)
  • A PostgreSQL instance (for persistence)
  • A Redis Instance
  • A Elastic Search Instance (optional)

Given this list of parts trying to run it in a containerised environment made sense. I have both a Docker Compose setup and a Kubernetes setup at home for testing FlowForge on while I’m working, so that wouldn’t be a major problem (though I understand I’m the outlier here). I decided to go with Kubernetes, because that cluster is a bit bigger and I like a challenge.

Deploying to Kubernetes

A bit of Googling turned up that while there isn’t a published Helm chart, there is one included in the project. So I cloned the project to my laptop.

git clone https://github.com/mastodon/mastodon.git


I then started to create a local-values.yml file to contain my instances specific configuration details. To get a feel for what values I’d need I started by looking in the chart/values.yml file to see what the defaults are and what I could override.

I also started to read the Mastodon server install guide as it had explanations to what each option means.

The first choice was what to call the instance. I went with a suggestion from @andypiper for the server name, but I’ll have the users hosted at my root domain

This means that the server is called bluetoot.hardill.me.uk and the instance is called hardill.me.uk so users will be identified for example as @ben@hardill.me.uk. These are configured as local_domain and web_domain

Next up was setting up the details of my mail server so that thinks like alerts and password resets can be sent. That was all pretty self explanatory.

The first tricky bit was setting up some secrets for the instance. There are secret keys for authentication tokens and a seed for the OTP setup. The documentation says to use rake secret but that implies you have Ruby environment already setup. I don’t work with Ruby so this wasn’t available. A bit more searching suggested that OpenSSL could be used:

openssl rand -hex 64

The next set of secrets are the vapid public and private keys. Here the documentation again has a rake command, but this time it appears to be a Mastodon specific Gem. To get round this I decided to pull the pre-built docker image and see if I could run the command in the container.

docker pull tootsuite/mastodon
docker run -it --rm --entrypoint /bin/bash tootsuite/mastodon:latest
mastodon@ab417e0a893a:~$ RAILS_ENV=production bundle exec rake mastodon:webpush:generate_vapid_key

(These keys are purely for demonstration and have not been used)

I made the decision to skip the elastic search instance to start with just to try and limit just how many resources I need to provide.

There are a couple of other bits I tweaked to make things work with my environment (forcing the bitnami charts to run on the AMD64 nodes rather than the ARM64) but the local-values.yml ended up looking like

    enabled: true
    username: fat_controller
    email: mastodon@example.com
  local_domain: example.com
  web_domain: bluetoot.example.com
      accessMode: ReadWriteMany
          storage: 10Gi
      accessMode: ReadWriteMany
          storage: 10Gi
    auth_method: plain
    delivery_method: smtp
    enable_starttls_auto: true
    port: 587
    server: mail.example.com
    login: user
    password: password
    from_address: mastodon@example.com
    domain: example.com

    secret_key_base: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    otp_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      private_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      public_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  enabled: false

  - host: bluetoot.example.com
    - path: '/'

    password: xxxxxxx
      beta.kubernetes.io/arch: amd64

  password: xxxxxxx
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/arch: amd64


The first part is to run helm

helm upgrade --install  --namespace mastodon --create-namespace mastodon ../mastodon/chart -f ./local-values.yml

The first time I ran this it failed to start most of the Mastodon pods with a bunch of errors around the PostgreSQL user password. I tracked it down to a recent Pull Request so I raised a new issue and a matching PR to fix things. This got merged really quickly which was very nice to see.

Once I’d modified the helm chart templates a little it deployed cleanly.


Next up was setting up my Internet facing web proxy. The Ingress controller on my Kubernetes cluster is not directly exposed to the internet so I needed to add an extra layer of proxying.

First up was to setup an new host entry and renew my LetsEncrypt certificate with the new SAN entry.

location / {
    proxy_pass https://kube-three.local;
    proxy_ssl_verify off;
    proxy_ssl_server_name on;
    proxy_redirect off;
    proxy_read_timeout 900;
    proxy_set_header Host $http_host; # host:$server_port;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-NginX-Proxy true;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_http_version 1.1;

It was important to make sure that the proxy target was over https otherwise the mastodon web app would trigger a redirect loop. the proxy_ssl_verify off; is because I’m using the default https certificate on the kubernetes ingress controller so it will fail the hostname check.

The other bit that needed doing was to add a proxy on the hardill.me.uk server for the .well-known/webfinger path to make sure user discovery works properly.

location /.well-known/webfinger {
    return 301 https://bluetoot.example.com$request_uri;

First Login

Now all the proxying and https setup is complete I can point my browser at https://bluetoot.hardill.me.uk and I get the intial sigin/signup screen.

As part of the configuration I created an admin user (fat_controller) but didn’t have a way to login as I don’t know what the generated password is. I tried follwing the lost password flow but couldn’t get it to work so followed the documentation about using the admin cli to do a password reset. I did this by using kubectl to get a shell in the mastodon web app pod in the cluster.

kubectl -n mastodon exec -it mastodon-web-5dd6764d4-mvwnl /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
mastodon@mastodon-web-5dd6764d4-mvwnl:~$ RAILS_ENV=production bin/tootctl accounts modify fat_controller --reset-password
New password: fc20b54eddb312bf9462abfec77a27d8

I could then log in as the administrator, update things like the contact email address, change the default user creation to require approval for new users.

What next?

I’m not sure if I’m going to keep this instance yet. At the moment it’s a testing environment and I’m not sure yet how much I’m going to actually use Mastodon. A lot will depend on what Mr Musk decides to do with his new play thing and if/when the communities I follow move.

But I will probably play with writing a few bots for some of the home automation things I have on the go, e.g. as I write this I’m waiting for a WiFi CO2 monitor to be delivered. Having a bot that Toots the level in my office sounds like a good first project.

Setting up mDNS CNAME entries for K8S Ingress Hostnames

As I hinted at in the end of my last post, I’ve been looking for a way to take the hostnames setup for Kubernetes Ingress endpoints and turn them into mDNS CNAME entries.

When I’m building things I like to spin up a local copy where possible (e.g. microk8s on a Pi 4 for the Node-RED on Kubernetes and the Docker Compose environment on another Pi 4 for the previous version). These setups run on my local network at home and while I have my own DNS server set up and running I also make extensive use of mDNS to be able to access the different services.

I’ve previously built little utilities to generate mDNS CNAME entries for both Nginx and Traefik reverse proxies using Environment Variables or Labels in a Docker environment, so I was keen to see if I can build the same for Kubernetes’ Ingress proxy.

Watching for new Ingress endpoints

The kubernetes-client node module supports for watching certain endpoints, so can be used to get notifications when an Ingress endpoint is created or destroyed.

const stream = client.apis.extensions.v1beta1.namespaces("default").ingresses.getStream({qs:{ watch: true}})
const jsonStream = new JSONStream()
jsonStream.on('data', async obj => {
  if (obj.type == "ADDED") {
    for (x in obj.object.spec.rules) {
      let hostname = obj.object.spec.rules[x].host
  } else if (obj.type == "DELETED") {
    for (x in obj.object.spec.rules) {
      let hostname = obj.object.spec.rules[x].host

Creating the CNAME

For the previous versions I used a python library called mdns-publish to set up the CNAME entries. It works by sending DBUS messages to the Avahi daemon which actually answers the mDNS requests on the network. For this version I decided to try and send those DBUS messages directly from the app watching for changes in K8s.

The dbus-next node module allows working directly with the DBUS interfaces that Avahi exposes.

const dbus = require('dbus-next');
const bus = dbus.systemBus()
bus.getProxyObject('org.freedesktop.Avahi', '/')
.then( async obj => {
	const server = obj.getInterface('org.freedesktop.Avahi.Server')
	const entryGroupPath = await server.EntryGroupNew()
	const entryGroup = await bus.getProxyObject('org.freedesktop.Avahi',  entryGroupPath)
	const entryGroupInt = entryGroup.getInterface('org.freedesktop.Avahi.EntryGroup')
	var interface = -1
	var protocol = -1
	var flags = 0
	var name = host
	var clazz = 0x01
	var type = 0x05
	var ttl = 60
	var rdata = encodeFQDN(hostname)
	entryGroupInt.AddRecord(interface, protocol, flags, name, clazz, type, ttl, rdata)

Adding a signal handler to clean up when the app gets killed and we are pretty much good to go.

process.on('SIGTERM', cleanup)
process.on('SIGINT', cleanup)
function cleanup() {
	const keys = Object.keys(cnames)
	for (k in keys) {


Once it’s all put together it runs as follows:

$ node index.js /home/ubuntu/.kube/config ubuntu.local

The first argument is the path to the kubectl config fileand the second is the hostname the CNAME should point to.

If the Ingress controller is running on ubuntu.local then Ingress YAML would look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
  name: manager-ingress
  - host: "manager.ubuntu.local"
      - pathType: Prefix
        path: "/"
            name: manager
              number: 3000 

I’ve tested this with my local microk8s install and it is working pretty well (even on my folks really sketchy wifi). The code is all up here.