Quick and Dirty Finger Daemon

I’ve been listening to more Brad & Will Made a Tech Pod and the current episode triggered a bunch of nostalgia about using finger to work out what my fellow CS students at university were up to. I won’t go into to too much detail about what Finger is as the podcast covers it all.

This podcast has triggered things like this in the past, like when I decided to make this blog (and Brad & Will’s podcast) available via Gopher.

On the podcast they had Ben Brown as a guest who had written his own Finger Daemon and linked it up to a site called Happy Net Box where users can update their plan file. Then anybody can access it using the finger command e.g. finger hardillb@happynetbox.com . The finger command is shipped by default on Windows, OSx and Linux so can be accessed from nearly anywhere.

I really liked the idea of resurrecting finger and as well as having a play with Happy Net Box I decided to see if I could run my own.

I started to look at what it would take to run a finger daemon on one of my Raspberry Pis, but while there are 2 packaged they don’t appear to run on current releases as they rely on init.d rather than Systemd.

Next up I thought I’d have a look at the protocol, which is documented in RFC1288. It is incredibly basic, you just listen on port 79 and read the username terminated with a new line & carriage return. This seamed to be simple enough to implement so I thought I’d give it a try in Go (and I needed something to do while all tonight’s TV was taken up with 22 men chasing a fall round a field).

The code is on Github here.

package main

import (
  "io"
  "os"
  "fmt"
  "net"
  "path"
  "time"
  "strings"
)

const (
  CONN_HOST = "0.0.0.0"
  CONN_PORT = "79"
  CONN_TYPE = "tcp"
)

func main () {
  l, err := net.Listen(CONN_TYPE, CONN_HOST+":"+CONN_PORT)
  if err != nil {
    fmt.Println("Error opening port: ", err.Error())
    os.Exit(1)
  }

  defer l.Close()
  for {
    conn, err := l.Accept()
    if err != nil {
      fmt.Println("Error accepting connection: ", err.Error())
      continue
    }
    go handleRequest(conn)
  }
}

func handleRequest(conn net.Conn) {
  defer conn.Close()
  currentTime := time.Now()
  buf := make([]byte, 1024)
  reqLen, err := conn.Read(buf)
  fmt.Println(currentTime.Format(time.RFC3339))
  if err != nil {
    fmt.Println("Error reading from: ", err.Error())
  } else {
    fmt.Println("Connection from: ", conn.RemoteAddr())
  }

  request := strings.TrimSpace(string(buf[:reqLen]))
  fmt.Println(request)

  parts := strings.Split(request, " ")
  wide := false
  user := parts[0]

  if parts[0] == "/W" && len(parts) == 2 {
    wide = true
    user = parts[1]
  } else if parts[0] == "/W" && len(parts) == 1 {
    conn.Write([]byte("\r\n"))
    return
  }

  if strings.Index(user, "@") != -1 {
    fmt.Println("remote")
    conn.Write([]byte("Forwarding not supported\r\n"))
  } else {
    if wide {
      //TODO
    } else {
      pwd, err := os.Getwd()
      filePath := path.Join(pwd, "plans", path.Base(user + ".plan"))
      filePath = path.Clean(filePath)
      fmt.Println(filePath)
      file, err := os.Open(filePath)
      if err != nil {
        //not found
        // io.Write([]byte("Not Found\r\n"))
      } else {
        defer file.Close()
        io.Copy(conn,file)
        conn.Write([]byte("\r\n"))
      }
    }
  }
}

Rather than deal with the nasty security problems with pulling .plan files out of peoples home directories it uses a directory called plans and loads files that match the pattern <username>.plan

I’ve also built it in a Docker container and mounted a local directory to allow me to edit and add new plan files.

You can test it with finger ben@hardill.me.uk

Setting up WireGuard IPv6

I’ve been having a quick play with setting up another VPN solution for getting an IPv6 address on my mobile devices this time using WireGuard.

WireGuard is a relatively new VPN tunnel implementation that has been written to be as stripped back as possible to keep the codebase as small as possible to help make it easier to audit.

Setup

A lot of the instructions for running WireGuard on RaspberryPi OS talk about adding debian testing repos or building the code from scratch, but it looks like recent updates have included the packages needed in the core repositories.

# apt-get install wireguard

I set up UDP port forwarding on my router for port 53145 and got my ISP to route another /64 IPv6 subnet to my line, both of these are forwarded on to the Raspberry Pi that is running that is also running my OpenVPN setup. This is useful as it’s already setup to do NAT for the 10.8.0.0/24 range I’m issuing to OpenVPN clients so having it do itfor the 10.9.0.0/24 range for WireGuard is easy enough.

WireGuard on Linux is implemented as a network device driver so can be configured on the command line with the ip command e.g.

# ip link add dev wg0 type wireguard
# ip address add dev wg0 10.9.0.1/24

Which brings the device up and sets the IP addresses but you still need to add the Private Key and remote address and Public Key which can be done with the wg command

# wg set wg0 listen-port 53145 private-key /path/to/private-key peer ABCDEF... allowed-ips 0.0.0.0/0 endpoint 209.202.254.14:8172

Or more easily it can read from a config file

# wg setconf wg0 myconfig.conf

Or the whole setup and configured with wg-quick

# wg-quick up /path/to/wg.conf

Server Config

[Interface]
Address = 10.9.0.1/24, 2001:8b0:2c1:xxx::1/64
ListenPort = 53145
PrivateKey = oP3TAHBctNVcnPTxxxxxxxxzNRLSF5CwII4s8gVAXg=

#nexus
[Peer]
PublicKey = 4XcNbctkGy0s73Dvxxxxxxxxx++rs5BAzCGjYmq21UM=
AllowedIPs = 10.9.0.2/32, 2001:8b0:2c1:xxx::2/128

The Server config includes:

  • Address is the local address on the VPN tunnel, here has both IPv4 and IPv6.
  • ListenPort is which port to listen for client connections on. WireGuard doesn’t have a assigned port.
  • PrivateKey to identify the host.
  • There can be multiple Peers which represent which clients can connect and the AllowedIPs is the IP addresses for each client.

Client Config

[Interface]
Address = 10.9.0.2/32, 2001:8b0:2c1:4b50::2/128
PrivateKey = UFIJGgtKsor6xxxxxxxxxxxbWeKmw+Bb5ODpyNblEA=
DNS = 8.8.8.8

[Peer]
PublicKey = jMB2oMu+YTKigGxxxxxxxxxxSYcTde/7HT+QlQoZFm0=
AllowedIPs = 0.0.0.0/0, ::0/0
Endpoint = hardill.me.uk:53145

The differences from the Server config are:

  • Interface has a DNS entry for the client to use while the tunnel is running.
  • Peer has an Endpoint which is the public address and port to connect to
  • AllowedIPs are which IPs to route over the tunnel, in this case it’s everything

Key Generation

Both ends of the connection need a PublicKey and a PrivateKey so they can mutually authenticate each other. These are generated with the wg command

# wg keygen > privateKey
# wg pubkey < privateKey > publicKey

Sharing Config

The WireGuard Android app that you can manually add all the details in the config file or it supports reading config files from QR codes. This makes it really easy to setup and removes the chance of getting a typo in the Keys and IP addresses.

You can generate QR codes from the config file as follows:

# qrencode -t png -o nexus.png < nexus.conf
# qrencode -t ansiutf8 < nexus.conf

The first generates a PNG file with the QR code, the second prints the code out as ASCII art.

Conclusion

It all looks to be working smoothly. I can see the advantages over OpenVPN being that you don’t need to worry about certificate maintenance and distribution.

I’ll give it a proper work out and see how it holds up running things like SIP connections along with general access to my home network.

As well as running it on the phone, I’ll set up a client config for my laptop to use when out and about. The only issues is that the Gnome Network Manager integration for WireGuard isn’t available in the standard repos for Ubuntu 20.04 so it needs to be started/stopped from the command line.

Google Assistant Sensors

Having built my 2 different LoRA connected temperature/humidity sensors I was looking for something other than the Graphana instance that shows the trends.

Being able to ask Google Assistant the temperature in a room seemed like a good idea and an excuse to add the relatively new Sensor device type my Google Assistant Bridge for Node-RED.

I’m exposing 2 options for the Sensor to start with, Temperature and Humidity. I might look at adding Air Quality later.

Once the virtual device is setup, you can feed data in the Google Home Graph using a flow similar to the following

The join node is set to combine the 2 incoming MQTT messages into a single object based on their topics. The function node then builds the right payload to pass to the Google Home output node and finally it feeds it through an RBE node just to make sure we only send updates when the data changes.

msg.payload = {
  params: {
    temperatureAmbientCelsius: msg.payload["bedroom/temp"],
    humidityAmbientPercent: Math.round(msg.payload["bedroom/humidity"])
  }
}

Working with multiple EFS file system in EKS

I’ve been building a system recently on AWS EKS and using EFS filesystems as volumes for persistent storage.

I initially only had one container that required any storage, but as I added a second I ran into the issue that there didn’t look to be a way to bind a EFS volume to a specific PersistentVolumeClaim so no way to make sure the same volume was mounted into the same container each time.

A Pod requests a volume by referencing a PersistentVolumeClaim as follows:

apiVersion: v1
kind: Pod
metadata:
  name: efs-app
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    volumeMounts:
    - name: efs-volume
      mountPath: /data
  volumes:
  - name: efs-volume
    persistentVolumeClaim:
      claimName: efs-claim

The PersistentVolumeClaim would look:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi

You can bind the EFS volume to a PersistentVolume as follows

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-persistent-volume
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-6eb2fc16

The volumeHandle points to the EFS volume you want to back it.

If there is only one PersistentVolume then there is not a problem as the PersistentVolumeClaim will grab the only one available. But if there are more than one then you can include the volumeName in the PersistentVolumeClaim description to bind the two together.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi
  volumeName: efs-persistent-volume

After a bit of poking around I found this Stack Overflow question which pointed me in the right direction.

Viewing Node-RED Credentials

A question popped up on the Node-RED Slack yesterday asking how to recover an entry from the credentials file.

Background

The credentials file can normally be found in the Node-RED userDir, which defaults to ~/.node-red on Unix like platforms (and is logged near the start of the output when Node-RED starts). The file has the same name as the flow file with _cred appended before the .json e.g. the flows_localhost.json will have a coresponding flows_localhost_creds.json

The content of the file will look something a little like this:

{"$":"7959e3be21a9806c5778bd8ad216ac8bJHw="}

This isn’t much use on it’s own as the contents are encrypted to make it harder for people to just copy the file and have access to all the stored passwords and access tokens.

The secret that is used to encrypt/decrypt this file can be found in one of 2 locations:

  • In the settings.js file in the credentialsSecret field. The user can set this if they want to use a fixed known value.
  • In the .config.json (or .config.runtime.json in later releases) in the __credentialSecret field. This secret is the one automatically generated if the user has not specifically set one in the settings.js file.

Code

In order to make use of thex

const crypto = require('crypto');

var encryptionAlgorithm = "aes-256-ctr";

function decryptCreds(key, cipher) {
  var flows = cipher["$"];
  var initVector = Buffer.from(flows.substring(0, 32),'hex');
  flows = flows.substring(32);
  var decipher = crypto.createDecipheriv(encryptionAlgorithm, key, initVector);
  var decrypted = decipher.update(flows, 'base64', 'utf8') + decipher.final('utf8');
  return JSON.parse(decrypted);
}

var creds = require("./" + process.argv[1])
var secret = process.argv[2]

var key = crypto.createHash('sha256').update(secret).digest();

console.log(decryptCreds(key, creds))

If you place this is a file called show-creds.js and place it in the Node-RED userDir you can run it as follows:

$ node show-creds flows_localhost_creds.json [secret]

where [secret] is the value stored in credentialsSecret or _credentialsSecret from earlier. This will then print out the decrypted JSON object holding all the passwords/tokens from the file.

Adding a TPM to My Offline Certificate Authority

Back at the start of last year, I built an offline Certificate Authority based around Pi Zero and a RTC module.

The idea was to run the CA on the pi that can only be accesses when it’s plugged in via a USB cable to another machine. This means that the CA cert and private key are normally offline and only potentially accessible by an attacker when plugged in.

For what’s at stake if my toy CA gets compromised this is already overkill, but I was looking to see what else I could do to make it even more secure.

TPM

A TPM or Trusted Platform Module is a dedicated CPU paired with some dedicated NVRAM. The CPU is capable of doing some pretty basic crypto functions, provide a good random number generator and NVRAM is used to store private keys.

TPM & RTC on a Raspberry Pi Zero

TPMs also have a feature called PCRs which can be used to validate the hardware and software stack used to boot the machine. This means you can use this to detect if the system has been tampered with at any point. This does require integration into the bootloader for the system.

You can set access policies for keys protected by the TPM to allow access if the PCRs match a known pattern, some Disk Encryption systems like LUKS on Linux and Bitlocker on Windows1 can use this to automatically unlock the encrypted drive.

You can get a TPM for the Raspberry Pi from a group called LetsTrust (that is available online here).

It mounts on to the SPI bus pins and is enabled by adding a Device Tree Overlay to the /boot/config,txt similar to the RTC.

dtoverlay=i2c-rtc,ds1307
dtoverlay=tpm-slb9670

Since the Raspberry Pi Bootloader is not TPM aware the PCRs are not initialised in this situation, so we can’t use it to automatically unlock an encrypted volume.

Using the TPM with the CA

Even without the PCRs the TPM can be used to protect the CA’s private key so it can only be used on the same machine as the TPM. This makes the private key useless if anybody does manage to remotely log into the device and make a copy.

Of course since it just pushes on to the Pi header if anybody manages to get physical access they can just take the TPM and sdcard, but as with all security mechanisms once an attacker has physical access all bets are usually off.

There is a plugin for OpenSSL that enables it to use keys stored in the TPM. Once compiled it can be added as OpenSSL Engine along with a utility called tpm2tss-genkey that can be used to create new keys or an existing key can be imported.

Generating New Keys

You can generate a new CA certificate with the following commands

$ tpm2tss-genkey -a rsa -s 2048 ca.tss
$ openssl req -new -x509 -engine tpm2tss -key ca.tss  -keyform engine -out ca.crt

This certificate can now be used to sign CSRs

$ openssl ca -config openssl.cnf -engine tpm2tss -key ca.tss -keyform engine -in cert.csr -out cert.pem

Importing Keys

For an existing ca.key private key file.

$ tpm2_createprimary --hierarchy=o --hash-algorithm=sha256 --key-algorithm=rsa --key-context=primiary_owner_key.ctx
$ HANDLE=$(tpm2_evictcontrol --hierarchy=o --object-context=primiary_owner_key.ctx | cut -d ' ' -f 2 | head -n 1)
$ tpm2_import -C primiary_owner_key.ctx -G rsa -i ca.key -u ca-pub.tpm -r ca.tpm
$ tpm2tss-genkey --public ca-pub-tpm --private ca.tpm --parent $HANDLE --password secret ca.tss

And we can then sign new CSRs the same way as with the generated key

$ openssl ca -config openssl.cnf -engine tpm2tss -key ca.tss -keyform engine -in cert.csr -out cert.pem

Once the keys have been imported the it’s important to remember to clean up the original key file (ca.key) so any attacker can’t just use them instead of using the one protected by the TPM. Any attacker now needs both the password for the key and the TPM device that was used to cloak it.

Web Interface

At the moment the node-openssl-cert node that I’m using to drive the web interface to CA doesn’t look to support passing in engine arguments so I’m having to drive it all manually on the command line, but I’ll be looking at a way to add support to the library. I’ll try and generate a pull request when I get something working.


1Because of it’s use with Bitlocker, a TPM is now required for all machines that want to be Windows 10 certified. This means my second Dell XPS13 also has one (it was an optional extra on the first version and not included in the Sputnik edition)

Email Autoconfiguration

I finally got round to setting up a new version of a Virtual Machine I had on my old laptop. It’s purpose is basically to host an email client that accesses a bunch of email addresses I have set up on my domain.

It was all going smoothly until I actually got round to adding the account details to Thunderbird

It sat and span like this for a while then pops up the manual configuration view.

Which is fine as I know the difference between pop3 and imap but it’s the sort of thing that really confuses the most users (I’ve lost count of the number of times I’ve had to talk people through this over the phone).

The problem is I thought I’d already fixed particular probelm. Back last time I set up a bunch of email addresses I remember setting up a bunch of DNS SRV records to point to both the inbound mail server and the IMAP server.

SRV Records

SRV records allow you to say which servers to use for a particular protocol using a given domain. The entries are made up of the protocol followed by the transport type and then the domain e.g.

_submission._tcp.example.com

The mail client would look the SRV record for this hostname to find the mail submission protocol server for the example.com domain and would get a response that looks like this:

_submission._tcp.example.com.	3600 IN	SRV	0 1 587 mail.example.com.

where:

  • 3600 is the Time to Live (how long to cache this result in seconds)
  • IN is the Internet Protocol
  • SRV Record type
  • 0 Weight (If multiple records try the lowest first)
  • 1 Priority (If multiple records with the same Weight pick the highest first)
  • 587 The port number of the service
  • mail.example.com the host where to find the service.

I have SRV records for the following protocols enabled:

  • Mail Submission _submission._tcp
  • IMAPS _imaps._tcp
  • SIP _sip._tcp & _sip._udp
  • SIPS _sips._tcp

Using SRV records for email discovery is covered by RFC6186. SRV records are also used in the VoIP space to point to SIP servers.

So the question is why this doesn’t work. The problem looks to be that Thunderbird hasn’t implemented support for RFC6186 just yet. A bit of digging found this document which covers what the current design for Thunderbird is and which bits are still to be implemented. It looks like the only option that currently works in the XML configuration file

config-v1.1.xml file

The document lists a few locations that a file can be placed relative to the domain that holds details of how to configure the email account. This includes http://example.com/.well-known/autoconfig/mail/config-v1.1.xml where example.com is the domain part of the email address.

The schema for config-v1.1.xml can be found here. A basic minimal entry would look something like this:

<?xml version="1.0"?>
<clientConfig version="1.1">
    <emailProvider id="example.com">
      <domain>example.com</domain>
      <displayName>Example Mail</displayName>
      <displayShortName>Example</displayShortName>
      <incomingServer type="imap">
         <hostname>mail.example.com</hostname>
         <port>995</port>
         <socketType>SSL</socketType>
         <username>%EMAILADDRESS%</username>
         <authentication>password-cleartext</authentication>
      </incomingServer>

      <outgoingServer type="smtp">
         <hostname>mail.example.com</hostname>
         <port>587</port>
         <socketType>STARTTLS</socketType> 
         <username>%EMAILADDRESS%</username> 
         <authentication>password-cleartext</authentication>
         <addThisServer>true</addThisServer>
      </outgoingServer>
    </emailProvider>
    <clientConfigUpdate url="https://www.example.com/config/mozilla.xml" />
</clientConfig>

Apart from the obvious parts that say which servers to connect to the other useful bit is found in the <username> tags. Here I’m using %EMAILADDRESS% which says to use the whole email address as the username. You can also use %EMAILLOCALPART% which is everything before the @ sign and %EMAILDOMAIN% which is everything after the @ sign.

The documentation includes options for setting up remote address books and calendar information, though it doesn’t look like Thunderbird supports all of these options just yet.

With this file now in place on my HTTP server Thunderbird now sets everything up properly.

Setting up mDNS CNAME entries for K8S Ingress Hostnames

As I hinted at in the end of my last post, I’ve been looking for a way to take the hostnames setup for Kubernetes Ingress endpoints and turn them into mDNS CNAME entries.

When I’m building things I like to spin up a local copy where possible (e.g. microk8s on a Pi 4 for the Node-RED on Kubernetes and the Docker Compose environment on another Pi 4 for the previous version). These setups run on my local network at home and while I have my own DNS server set up and running I also make extensive use of mDNS to be able to access the different services.

I’ve previously built little utilities to generate mDNS CNAME entries for both Nginx and Traefik reverse proxies using Environment Variables or Labels in a Docker environment, so I was keen to see if I can build the same for Kubernetes’ Ingress proxy.

Watching for new Ingress endpoints

The kubernetes-client node module supports for watching certain endpoints, so can be used to get notifications when an Ingress endpoint is created or destroyed.

const stream = client.apis.extensions.v1beta1.namespaces("default").ingresses.getStream({qs:{ watch: true}})
const jsonStream = new JSONStream()
stream.pipe(jsonStream)
jsonStream.on('data', async obj => {
  if (obj.type == "ADDED") {
    for (x in obj.object.spec.rules) {
      let hostname = obj.object.spec.rules[x].host
      ...
    }
  } else if (obj.type == "DELETED") {
    for (x in obj.object.spec.rules) {
      let hostname = obj.object.spec.rules[x].host
      ...
    }
  }
}

Creating the CNAME

For the previous versions I used a python library called mdns-publish to set up the CNAME entries. It works by sending DBUS messages to the Avahi daemon which actually answers the mDNS requests on the network. For this version I decided to try and send those DBUS messages directly from the app watching for changes in K8s.

The dbus-next node module allows working directly with the DBUS interfaces that Avahi exposes.

const dbus = require('dbus-next');
const bus = dbus.systemBus()
bus.getProxyObject('org.freedesktop.Avahi', '/')
.then( async obj => {
	const server = obj.getInterface('org.freedesktop.Avahi.Server')
	const entryGroupPath = await server.EntryGroupNew()
	const entryGroup = await bus.getProxyObject('org.freedesktop.Avahi',  entryGroupPath)
	const entryGroupInt = entryGroup.getInterface('org.freedesktop.Avahi.EntryGroup')
	var interface = -1
	var protocol = -1
	var flags = 0
	var name = host
	var clazz = 0x01
	var type = 0x05
	var ttl = 60
	var rdata = encodeFQDN(hostname)
	entryGroupInt.AddRecord(interface, protocol, flags, name, clazz, type, ttl, rdata)
	entryGroupInt.Commit()
})

Adding a signal handler to clean up when the app gets killed and we are pretty much good to go.

process.on('SIGTERM', cleanup)
process.on('SIGINT', cleanup)
function cleanup() {
	const keys = Object.keys(cnames)
	for (k in keys) {
		//console.log(keys[k])
		cnames[keys[k]].Reset()
    	cnames[keys[k]].Free()
	}
	bus.disconnect()
	process.exit(0)
}

Running

Once it’s all put together it runs as follows:

$ node index.js /home/ubuntu/.kube/config ubuntu.local

The first argument is the path to the kubectl config fileand the second is the hostname the CNAME should point to.

If the Ingress controller is running on ubuntu.local then Ingress YAML would look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: manager-ingress
spec:
  rules:
  - host: "manager.ubuntu.local"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: manager
            port:
              number: 3000 

I’ve tested this with my local microk8s install and it is working pretty well (even on my folks really sketchy wifi). The code is all up here.

Multi Tenant Node-RED with Kubernetes

Having built a working example of Multi Tenant Node-RED using Docker I thought I’d have a look at how to do the same with Kubernetes as a Christmas project.

I started with installing the 64bit build of Ubuntu Server on a fresh Pi4 with 8gb RAM and then using snapd to install microk8s. I had initially wanted to use the 64bit version of Raspberry Pi OS, but despite microk8s claiming to work on any OS that support snapd, I found that containerd just kept crashing on Raspberry Pi OS.

Once installed I enabled the dns and ingress plugins, this got me a minimal viable single node Kubernetes setup working.

I also had to stand up a private docker registry to hold the containers I’ll be using. That was just a case of running docker run -d -p 5000:5000 --name registry registry on a local machine e.g private.example.com . This also means adding the URL for this to microk8s as described here.

Since Kubernetes is another container environment I can reuse most of the parts I previously created. The only bit that really needs to change is the Manager application as this has to interact with the environment to stand up and tear down containers.

Architecture

As before the central components are a MongoDB database and a management web app that stands up and tears down instances. The MongoDB instance holds all the flows and authentication details for each instance. I’ve deployed the database and web app as a single pod and exposed them both as services

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-red-multi-tenant
  labels:
    app: nr-mt
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nr-mt
  template:
    metadata:
      labels:
        app: nr-mt
    spec:
      containers:
      - name: node-red-manager
        image: private.example.com/k8s-manager
        ports:
        - containerPort: 3000
        volumeMounts:
        - name: secret
          mountPath: /usr/src/app/config
        env:
        - name: MONGO_URL
          value: mongodb://mongo/nodered
        - name: ROOT_DOMAIN
          value: example.com
      - name: mongodb
        image: mongo
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: mongo-data
          mountPath: /data/db
      - name: registry
        image: verdaccio/verdaccio
        ports:
        - containerPort: 4873
        volumeMounts:
        - name: registry-data
          mountPath: /verdaccio/storage
        - name: registry-conf
          mountPath: /verdaccio/conf
      volumes:
      - name: secret
        secret:
          secretName: kube-config
      - name: mongo-data
        hostPath:
          path: /opt/mongo-data
          type: Directory
      - name: registry-data
        hostPath:
          path: /opt/registry-data
          type: Directory
      - name: registry-conf
        secret:
          secretName: registry-conf

This Deployment descriptor basically does all the heavy lifting. It sets up the mangment app, MongoDB and the private NPM registry.

It also binds 2 sets of secrets, the first holds holds the authentication details to interact with the Kubernetes API (the ~/.kube/config file) and the settings.js for the management app. The second is the config for the Veraccio NPM registry.

I’m using the HostPath volume provider to store the MongoDB and the Veraccio registry on the filesystem of the Pi, but for a production deployment I’d probably use the NFS provider or a Cloud Storage option like AWS S3.

Manager

This is mainly the same as the docker version, but I had to swap out dockerode for kubernetes-client.

This library exposes the full kubernetes API allowing the creation/modification/destructions of all entities.

Standing up a new instance is a little more complicated as it’s now a multi step process.

  1. Create a Pod with the custom-node-red container
  2. Create a Service based on that pod
  3. Expose that service via the Ingress addon

I also removed the Start/Stop buttons since stopping pods is not really a thing in Kubernetes.

All the code for this version of the app is on github here.

Catalogue

In the Docker-Compose version the custom node `catalogue.json` file is hosted by the management application and had to be manually updated each time a new or updated node was push to the repository. For this version I’ve stood up a separate container.

This container runs a small web app that has 2 endpoints.

  • /catalogue.json – which returns the current version of the catalogue
  • /update – which is triggered by the the notify function of the Verdaccio private npm registry

The registry has this snippet added to the end of the config.yml

notify:
  method: POST
  headers: [{'Content-Type': 'application/json'}]
  endpoint: http://catalogue/update
  content: '{"name": "{{name}}", "versions": "{{versions}}", "dist-tags": "{{dist-tags}}"}'

The code for this container can be found here.

Deploying

First clone the project from github

$ github clone https://github.com/hardillb/multi-tenant-node-red-k8s.git

Then run the setup.sh script, passing in the base domain for instances and the host:port combination for the local container registry.

$ ./setup.sh example.com private.example.com:5000

This will update some of the container locations in the deployment and build the secrets needed to access the Kubernetes API (reads the content of ~/.kube/config)

With all the configuration files updated the containers need building and pushing to the local container registry.

$ docker build ./manager -t private.example.com:5000/k8s-manager
$ docker push private.example.com:5000/k8s-manager
$ docker build ./catalogue -t private.example.com:5000/catalogue
$ docker push private.example.com:5000/catalogue
$ docker build ./custom-node-red -t private.example.com:5000/custom-node-red
$ docker push private.example.com:5000/custom-node-red

Finally trigger the actual deployment with kubectl

$ kubectl apply -f ./deployment

Once up and running the management app should be available on http://manager.example.com, the private npm registry on http://registry.example.com and an instance called “r1” would be on http://r1.example.com.

A wildcard DNS entry needs to be setup to point all *.example.com hosts to the Kubernetes clusters Ingress IP addresses.

As usual the whole solution can be found on github here.

What’s Next

I need to work out how to set up Avahi CNAME entries for each deployment as I had working with both nginx and traefik so I can run it all nicely on my LAN without having to mess with /etc/hosts or the local DNS. This should be possible by using a watch call one the Kubernetes Ingress endpoint.

I also need to back port the new catalogue handling to the docker-compose version.

And finally I want to have a look at generating a Helm chart for all this to help get rid of needing the setup.sh script to modify the deployment YAML files.

p.s. If anybody is looking for somebody to do this sort of thing for them drop me a line.

Getting a Little Rusty

After using building a tool to populate a Gopher server as an excuse to learn the Go programming language I’ve recently been wanting to try my hand at Rust.

The best way to learn a new programming language is to use it to actually solve a problem, rather than just copying exercises out of a tutorial. So this time I thought I’d try and build my own Gopher server.

Specification

The initial version of the Gopher specification is laid down in RFC1436. It’s pretty simple. The client basically sends a string that represents the path to the document it wants followed by \r\n. In the case of the root document the client sends just the line ending chars.

The server responds with either the raw content of the document or if the path points to a directory then it sends the content of a file called gophermap if found in that directory.

The gophermap file holds a list of links a bit like a index.html for a HTTP server.

Ben's Place - Gopher

Just a place to make notes about things I've 
been playing with

0CV	cv.txt
1Blog	/blog
1Brad & Will Made a Tech Pod	/podcast

The lines that start with 0 are direct links to a file and have the label and then the file name. Where as 1 are links to another directory. The fields are delimited with tabs.

You can also link to files/directories on other servers by including the server and port after the filename/dir path again separated by tabs.

1Blog	/blog	gopher.hardill.me.uk	70

There is also something called Gopher+ which is an extended version that looks to never have been formally adopted as a standard but both my gopher client and PyGopherd look to support. A copy of the draft is here.

Rust

Similar to things like NodeJS, Rust has a package manager called cargo that can be used to create a new project, running cargo new rust_gopher will create a Cargo.toml file that looks a bit like this:

[package]
name = "rust_gopher"
version = "0.1.0"
authors = ["Ben Hardill <hardillb@gmail.com"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

It also create a src directory with a main.rs and calls git init to start a new repository.

The src/main.rs is pre-populated with “Hello World”

fn main() {
    println!("Hello, world!");
}

I’ve replaced the main() function with one that uses the clap create to parse some command line arguments.

let matches = App::new("rust_gopher")
	.version("0.1.0")
	.author("Ben Hardill")
	.about("Basic Gopher Server")
	.arg(Arg::with_name("hostname")
		.short("h")
		.long("hostname")
		.takes_value(true)
		.help("The hostname of this server"))
	.arg(Arg::with_name("port")
		.short("p")
		.long("port")
		.takes_value(true)
		.help("Port number to listen on"))
	.arg(Arg::with_name("dir")
		.short("d")
		.long("dir")
		.takes_value(true)
		.help("path to gopher content"))
	.get_matches();

let hostname = matches.value_of("hostname").unwrap_or("localhost");
let port :i16 = matches.value_of("port").unwrap_or("70").parse().unwrap();
let dir = matches.value_of("dir").unwrap_or("root");

Which gives really nicely formatted help output:

$ rust_gopher --help
rust_gopher 0.1.0
Ben Hardill
Basic Gopher Server

USAGE:
    rust_gopher [OPTIONS]

FLAGS:
        --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -d, --dir <dir>              path to gopher content
    -h, --hostname <hostname>    The hostname of this server
    -p, --port <port>            Port number to listen on

Reading the gophermap

To add the required bits to a basic gophermap file into what actually gets sent, the files get parsed into the following structure.

struct Gophermap {
	row_type: char,
	label: String,
	path: String,
	server: String,
	port: i16
}

fn read_gophermap(path: &Path, config: &Config) -> Vec<Gophermap>{
	let mut entries: Vec<Gophermap> = Vec::new();

	let file = File::open(path).unwrap();
	let reader = BufReader::new(file);

	for line in reader.lines() {
		let mut l = line.unwrap();
...
		let  entry = Gophermap {
 			row_type: t,
 			label: label,
 			path: p.to_string(),
 			server: s.to_string(),
 			port: port
 		};
		entries.push(entry);
	}
	entries;
}

What’s next?

It still needs a bunch of work, mainly around adding a bunch of error handling. I’ll probably keep poking at it over the holidays.

As usual the code is up on github here.