Email Autoconfiguration

I finally got round to setting up a new version of a Virtual Machine I had on my old laptop. It’s purpose is basically to host an email client that accesses a bunch of email addresses I have set up on my domain.

It was all going smoothly until I actually got round to adding the account details to Thunderbird

It sat and span like this for a while then pops up the manual configuration view.

Which is fine as I know the difference between pop3 and imap but it’s the sort of thing that really confuses the most users (I’ve lost count of the number of times I’ve had to talk people through this over the phone).

The problem is I thought I’d already fixed particular probelm. Back last time I set up a bunch of email addresses I remember setting up a bunch of DNS SRV records to point to both the inbound mail server and the IMAP server.

SRV Records

SRV records allow you to say which servers to use for a particular protocol using a given domain. The entries are made up of the protocol followed by the transport type and then the domain e.g.

_submission._tcp.example.com

The mail client would look the SRV record for this hostname to find the mail submission protocol server for the example.com domain and would get a response that looks like this:

_submission._tcp.example.com.	3600 IN	SRV	0 1 587 mail.example.com.

where:

  • 3600 is the Time to Live (how long to cache this result in seconds)
  • IN is the Internet Protocol
  • SRV Record type
  • 0 Weight (If multiple records try the lowest first)
  • 1 Priority (If multiple records with the same Weight pick the highest first)
  • 587 The port number of the service
  • mail.example.com the host where to find the service.

I have SRV records for the following protocols enabled:

  • Mail Submission _submission._tcp
  • IMAPS _imaps._tcp
  • SIP _sip._tcp & _sip._udp
  • SIPS _sips._tcp

Using SRV records for email discovery is covered by RFC6186. SRV records are also used in the VoIP space to point to SIP servers.

So the question is why this doesn’t work. The problem looks to be that Thunderbird hasn’t implemented support for RFC6186 just yet. A bit of digging found this document which covers what the current design for Thunderbird is and which bits are still to be implemented. It looks like the only option that currently works in the XML configuration file

config-v1.1.xml file

The document lists a few locations that a file can be placed relative to the domain that holds details of how to configure the email account. This includes http://example.com/.well-known/autoconfig/mail/config-v1.1.xml where example.com is the domain part of the email address.

The schema for config-v1.1.xml can be found here. A basic minimal entry would look something like this:

<?xml version="1.0"?>
<clientConfig version="1.1">
    <emailProvider id="example.com">
      <domain>example.com</domain>
      <displayName>Example Mail</displayName>
      <displayShortName>Example</displayShortName>
      <incomingServer type="imap">
         <hostname>mail.example.com</hostname>
         <port>995</port>
         <socketType>SSL</socketType>
         <username>%EMAILADDRESS%</username>
         <authentication>password-cleartext</authentication>
      </incomingServer>

      <outgoingServer type="smtp">
         <hostname>mail.example.com</hostname>
         <port>587</port>
         <socketType>STARTTLS</socketType> 
         <username>%EMAILADDRESS%</username> 
         <authentication>password-cleartext</authentication>
         <addThisServer>true</addThisServer>
      </outgoingServer>
    </emailProvider>
    <clientConfigUpdate url="https://www.example.com/config/mozilla.xml" />
</clientConfig>

Apart from the obvious parts that say which servers to connect to the other useful bit is found in the <username> tags. Here I’m using %EMAILADDRESS% which says to use the whole email address as the username. You can also use %EMAILLOCALPART% which is everything before the @ sign and %EMAILDOMAIN% which is everything after the @ sign.

The documentation includes options for setting up remote address books and calendar information, though it doesn’t look like Thunderbird supports all of these options just yet.

With this file now in place on my HTTP server Thunderbird now sets everything up properly.

Setting up mDNS CNAME entries for K8S Ingress Hostnames

As I hinted at in the end of my last post, I’ve been looking for a way to take the hostnames setup for Kubernetes Ingress endpoints and turn them into mDNS CNAME entries.

When I’m building things I like to spin up a local copy where possible (e.g. microk8s on a Pi 4 for the Node-RED on Kubernetes and the Docker Compose environment on another Pi 4 for the previous version). These setups run on my local network at home and while I have my own DNS server set up and running I also make extensive use of mDNS to be able to access the different services.

I’ve previously built little utilities to generate mDNS CNAME entries for both Nginx and Traefik reverse proxies using Environment Variables or Labels in a Docker environment, so I was keen to see if I can build the same for Kubernetes’ Ingress proxy.

Watching for new Ingress endpoints

The kubernetes-client node module supports for watching certain endpoints, so can be used to get notifications when an Ingress endpoint is created or destroyed.

const stream = client.apis.extensions.v1beta1.namespaces("default").ingresses.getStream({qs:{ watch: true}})
const jsonStream = new JSONStream()
stream.pipe(jsonStream)
jsonStream.on('data', async obj => {
  if (obj.type == "ADDED") {
    for (x in obj.object.spec.rules) {
      let hostname = obj.object.spec.rules[x].host
      ...
    }
  } else if (obj.type == "DELETED") {
    for (x in obj.object.spec.rules) {
      let hostname = obj.object.spec.rules[x].host
      ...
    }
  }
}

Creating the CNAME

For the previous versions I used a python library called mdns-publish to set up the CNAME entries. It works by sending DBUS messages to the Avahi daemon which actually answers the mDNS requests on the network. For this version I decided to try and send those DBUS messages directly from the app watching for changes in K8s.

The dbus-next node module allows working directly with the DBUS interfaces that Avahi exposes.

const dbus = require('dbus-next');
const bus = dbus.systemBus()
bus.getProxyObject('org.freedesktop.Avahi', '/')
.then( async obj => {
	const server = obj.getInterface('org.freedesktop.Avahi.Server')
	const entryGroupPath = await server.EntryGroupNew()
	const entryGroup = await bus.getProxyObject('org.freedesktop.Avahi',  entryGroupPath)
	const entryGroupInt = entryGroup.getInterface('org.freedesktop.Avahi.EntryGroup')
	var interface = -1
	var protocol = -1
	var flags = 0
	var name = host
	var clazz = 0x01
	var type = 0x05
	var ttl = 60
	var rdata = encodeFQDN(hostname)
	entryGroupInt.AddRecord(interface, protocol, flags, name, clazz, type, ttl, rdata)
	entryGroupInt.Commit()
})

Adding a signal handler to clean up when the app gets killed and we are pretty much good to go.

process.on('SIGTERM', cleanup)
process.on('SIGINT', cleanup)
function cleanup() {
	const keys = Object.keys(cnames)
	for (k in keys) {
		//console.log(keys[k])
		cnames[keys[k]].Reset()
    	cnames[keys[k]].Free()
	}
	bus.disconnect()
	process.exit(0)
}

Running

Once it’s all put together it runs as follows:

$ node index.js /home/ubuntu/.kube/config ubuntu.local

The first argument is the path to the kubectl config fileand the second is the hostname the CNAME should point to.

If the Ingress controller is running on ubuntu.local then Ingress YAML would look like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: manager-ingress
spec:
  rules:
  - host: "manager.ubuntu.local"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: manager
            port:
              number: 3000 

I’ve tested this with my local microk8s install and it is working pretty well (even on my folks really sketchy wifi). The code is all up here.

Multi Tenant Node-RED with Kubernetes

Having built a working example of Multi Tenant Node-RED using Docker I thought I’d have a look at how to do the same with Kubernetes as a Christmas project.

I started with installing the 64bit build of Ubuntu Server on a fresh Pi4 with 8gb RAM and then using snapd to install microk8s. I had initially wanted to use the 64bit version of Raspberry Pi OS, but despite microk8s claiming to work on any OS that support snapd, I found that containerd just kept crashing on Raspberry Pi OS.

Once installed I enabled the dns and ingress plugins, this got me a minimal viable single node Kubernetes setup working.

I also had to stand up a private docker registry to hold the containers I’ll be using. That was just a case of running docker run -d -p 5000:5000 --name registry registry on a local machine e.g private.example.com . This also means adding the URL for this to microk8s as described here.

Since Kubernetes is another container environment I can reuse most of the parts I previously created. The only bit that really needs to change is the Manager application as this has to interact with the environment to stand up and tear down containers.

Architecture

As before the central components are a MongoDB database and a management web app that stands up and tears down instances. The MongoDB instance holds all the flows and authentication details for each instance. I’ve deployed the database and web app as a single pod and exposed them both as services

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-red-multi-tenant
  labels:
    app: nr-mt
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nr-mt
  template:
    metadata:
      labels:
        app: nr-mt
    spec:
      containers:
      - name: node-red-manager
        image: private.example.com/k8s-manager
        ports:
        - containerPort: 3000
        volumeMounts:
        - name: secret
          mountPath: /usr/src/app/config
        env:
        - name: MONGO_URL
          value: mongodb://mongo/nodered
        - name: ROOT_DOMAIN
          value: example.com
      - name: mongodb
        image: mongo
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: mongo-data
          mountPath: /data/db
      - name: registry
        image: verdaccio/verdaccio
        ports:
        - containerPort: 4873
        volumeMounts:
        - name: registry-data
          mountPath: /verdaccio/storage
        - name: registry-conf
          mountPath: /verdaccio/conf
      volumes:
      - name: secret
        secret:
          secretName: kube-config
      - name: mongo-data
        hostPath:
          path: /opt/mongo-data
          type: Directory
      - name: registry-data
        hostPath:
          path: /opt/registry-data
          type: Directory
      - name: registry-conf
        secret:
          secretName: registry-conf

This Deployment descriptor basically does all the heavy lifting. It sets up the mangment app, MongoDB and the private NPM registry.

It also binds 2 sets of secrets, the first holds holds the authentication details to interact with the Kubernetes API (the ~/.kube/config file) and the settings.js for the management app. The second is the config for the Veraccio NPM registry.

I’m using the HostPath volume provider to store the MongoDB and the Veraccio registry on the filesystem of the Pi, but for a production deployment I’d probably use the NFS provider or a Cloud Storage option like AWS S3.

Manager

This is mainly the same as the docker version, but I had to swap out dockerode for kubernetes-client.

This library exposes the full kubernetes API allowing the creation/modification/destructions of all entities.

Standing up a new instance is a little more complicated as it’s now a multi step process.

  1. Create a Pod with the custom-node-red container
  2. Create a Service based on that pod
  3. Expose that service via the Ingress addon

I also removed the Start/Stop buttons since stopping pods is not really a thing in Kubernetes.

All the code for this version of the app is on github here.

Catalogue

In the Docker-Compose version the custom node `catalogue.json` file is hosted by the management application and had to be manually updated each time a new or updated node was push to the repository. For this version I’ve stood up a separate container.

This container runs a small web app that has 2 endpoints.

  • /catalogue.json – which returns the current version of the catalogue
  • /update – which is triggered by the the notify function of the Verdaccio private npm registry

The registry has this snippet added to the end of the config.yml

notify:
  method: POST
  headers: [{'Content-Type': 'application/json'}]
  endpoint: http://catalogue/update
  content: '{"name": "{{name}}", "versions": "{{versions}}", "dist-tags": "{{dist-tags}}"}'

The code for this container can be found here.

Deploying

First clone the project from github

$ github clone https://github.com/hardillb/multi-tenant-node-red-k8s.git

Then run the setup.sh script, passing in the base domain for instances and the host:port combination for the local container registry.

$ ./setup.sh example.com private.example.com:5000

This will update some of the container locations in the deployment and build the secrets needed to access the Kubernetes API (reads the content of ~/.kube/config)

With all the configuration files updated the containers need building and pushing to the local container registry.

$ docker build ./manager -t private.example.com:5000/k8s-manager
$ docker push private.example.com:5000/k8s-manager
$ docker build ./catalogue -t private.example.com:5000/catalogue
$ docker push private.example.com:5000/catalogue
$ docker build ./custom-node-red -t private.example.com:5000/custom-node-red
$ docker push private.example.com:5000/custom-node-red

Finally trigger the actual deployment with kubectl

$ kubectl apply -f ./deployment

Once up and running the management app should be available on http://manager.example.com, the private npm registry on http://registry.example.com and an instance called “r1” would be on http://r1.example.com.

A wildcard DNS entry needs to be setup to point all *.example.com hosts to the Kubernetes clusters Ingress IP addresses.

As usual the whole solution can be found on github here.

What’s Next

I need to work out how to set up Avahi CNAME entries for each deployment as I had working with both nginx and traefik so I can run it all nicely on my LAN without having to mess with /etc/hosts or the local DNS. This should be possible by using a watch call one the Kubernetes Ingress endpoint.

I also need to back port the new catalogue handling to the docker-compose version.

And finally I want to have a look at generating a Helm chart for all this to help get rid of needing the setup.sh script to modify the deployment YAML files.

p.s. If anybody is looking for somebody to do this sort of thing for them drop me a line.

Getting a Little Rusty

After using building a tool to populate a Gopher server as an excuse to learn the Go programming language I’ve recently been wanting to try my hand at Rust.

The best way to learn a new programming language is to use it to actually solve a problem, rather than just copying exercises out of a tutorial. So this time I thought I’d try and build my own Gopher server.

Specification

The initial version of the Gopher specification is laid down in RFC1436. It’s pretty simple. The client basically sends a string that represents the path to the document it wants followed by \r\n. In the case of the root document the client sends just the line ending chars.

The server responds with either the raw content of the document or if the path points to a directory then it sends the content of a file called gophermap if found in that directory.

The gophermap file holds a list of links a bit like a index.html for a HTTP server.

Ben's Place - Gopher

Just a place to make notes about things I've 
been playing with

0CV	cv.txt
1Blog	/blog
1Brad &amp; Will Made a Tech Pod	/podcast

The lines that start with 0 are direct links to a file and have the label and then the file name. Where as 1 are links to another directory. The fields are delimited with tabs.

You can also link to files/directories on other servers by including the server and port after the filename/dir path again separated by tabs.

1Blog	/blog	gopher.hardill.me.uk	70

There is also something called Gopher+ which is an extended version that looks to never have been formally adopted as a standard but both my gopher client and PyGopherd look to support. A copy of the draft is here.

Rust

Similar to things like NodeJS, Rust has a package manager called cargo that can be used to create a new project, running cargo new rust_gopher will create a Cargo.toml file that looks a bit like this:

[package]
name = "rust_gopher"
version = "0.1.0"
authors = ["Ben Hardill <hardillb@gmail.com"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

It also create a src directory with a main.rs and calls git init to start a new repository.

The src/main.rs is pre-populated with “Hello World”

fn main() {
    println!("Hello, world!");
}

I’ve replaced the main() function with one that uses the clap create to parse some command line arguments.

let matches = App::new("rust_gopher")
	.version("0.1.0")
	.author("Ben Hardill")
	.about("Basic Gopher Server")
	.arg(Arg::with_name("hostname")
		.short("h")
		.long("hostname")
		.takes_value(true)
		.help("The hostname of this server"))
	.arg(Arg::with_name("port")
		.short("p")
		.long("port")
		.takes_value(true)
		.help("Port number to listen on"))
	.arg(Arg::with_name("dir")
		.short("d")
		.long("dir")
		.takes_value(true)
		.help("path to gopher content"))
	.get_matches();

let hostname = matches.value_of("hostname").unwrap_or("localhost");
let port :i16 = matches.value_of("port").unwrap_or("70").parse().unwrap();
let dir = matches.value_of("dir").unwrap_or("root");

Which gives really nicely formatted help output:

$ rust_gopher --help
rust_gopher 0.1.0
Ben Hardill
Basic Gopher Server

USAGE:
    rust_gopher [OPTIONS]

FLAGS:
        --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -d, --dir <dir>              path to gopher content
    -h, --hostname <hostname>    The hostname of this server
    -p, --port <port>            Port number to listen on

Reading the gophermap

To add the required bits to a basic gophermap file into what actually gets sent, the files get parsed into the following structure.

struct Gophermap {
	row_type: char,
	label: String,
	path: String,
	server: String,
	port: i16
}

fn read_gophermap(path: &amp;Path, config: &amp;Config) -> Vec<Gophermap>{
	let mut entries: Vec<Gophermap> = Vec::new();

	let file = File::open(path).unwrap();
	let reader = BufReader::new(file);

	for line in reader.lines() {
		let mut l = line.unwrap();
...
		let  entry = Gophermap {
 			row_type: t,
 			label: label,
 			path: p.to_string(),
 			server: s.to_string(),
 			port: port
 		};
		entries.push(entry);
	}
	entries;
}

What’s next?

It still needs a bunch of work, mainly around adding a bunch of error handling. I’ll probably keep poking at it over the holidays.

As usual the code is up on github here.

Multi Tenant Node-RED Working Example

I’ve now completed all the parts I outlined in the first post in this series and it’s time to put it all together to end up with a running system.

Since the stack is already running on Docker, using docker-compose to assemble the orchestration seemed like the right thing to do. The whole docker-compose project can be found on GitHub here.

Once you’ve checked out the project run the setup.sh script with the root domain as the first argument. This will do the following:

  • Checkouts the submodules (Management app and Mongoose schema objects).
  • Create and set the correct permissions on the required local directories that are mounted as volumes.
  • Build the custom-node-red docker container that will be used for each of the Node-RED instances.
  • Change the root domain for all the virtual hosts to match the value passed in as an argument or if left blank the current hostname with .local appended.
  • Create the Docker network that all the images will be attached to.

The following docker-compose.yaml file starts the Nginx proxy, the MongoDB instance, the custom npm registry and finally the management application.

version: "3.3"

services:
  nginx:
    image: jwilder/nginx-proxy
    networks:
      - internal
    volumes:
      - "/var/run/docker.sock:/tmp/docker.sock:ro"
    ports:
      - "80:80"

  mongodb:
    image: mongo
    networks:
      - internal
    volumes:
      - "./mongodb:/data/db"
    ports:
      - "27017:27017"

  registry:
    image: verdaccio/verdaccio
    networks:
      - internal
    volumes:
      - "./registry:/verdaccio/conf"
    ports:
      - "4873:4873"

  manager:
    image: manager
    build: "./manager"
    depends_on:
      - mongodb
    networks:
      - internal
    volumes:
      - "/var/run/docker.sock:/tmp/docker.sock"
      - "./catalogue:/data"
    environment:
      - "VIRTUAL_HOST=manager.example.com"
      - "MONGO_URL=mongodb://mongodb/nodered"
      - "ROOT_DOMAIN=docker.local"


networks:
  internal:
    external:
      name: internal

It mounts local directories for the MongoDB storage, the Verdaccio registry configuration/storage so that this data is persisted across restarts.

The ports for direct access to the MongoDB and the registry are currently exposed on the docker host. In a production environment it would probably make sense to not expose the MongoDB instance as it is currently unsecured, but it’s useful while I’ve been developing/debugging the system.

The port for the private NPM registry is also exposed to allow packages to be published from outside the system.

And finally it binds to the network that was created by the setup.sh script, this is used for all the containers to communicate. This also means that the containers can address each other by their names as docker runs a DNS resolver on the custom network.

The management application is also directly exposed as manager.example.com, you should be able to log in with the username admin and the password of password (both can be changed in the manager/settings.js file) and create a new instance.

Conclusions

This is about as bare bones as a Multi Tenant Node-RED deployment can get, there are still a lot of things that would need to be considered for a commercial offering (Things like cgroup based memory and CPU limits))based on something like this, but it should be enough for a class room deployment allowing each student to have their own instance.

Next steps would be to look what more the management app could do e.g. expose the logs to the instance admins and look at would be needed to deploy this to something like OpenShift.

If you build something based on this please let both me and the Node-RED community know.

And if you are looking for somebody to help you build on this please get in touch.

Advanced Multi Tenant Node-RED topics

Across the last 6 posts I’ve walked through deploying a Multi Tenant Node-RED service. In this post I’m going to talk about how you go about customising the install to make it more specific to your needs.

Custom Node Catalogue

As I mentioned in the post about creating the custom docker container you can use the nodeExcludes in settings.js to disable nodes and if needed the package.json can be edited to remove any core nodes that you want.

But you might have a collection of nodes that are specific to your deployment that are not of use outside your environment and you may not want to publish to npmjs.org. In extreme cases you may even want to remove all the core nodes and only allow users to use your own set of nodes.

Node-RED downloads the list of nodes available to install from catalogue.nodered.org and allows you to add either additional URLs to pull in a list of extra nodes or replace the URL with a custom list. The documentation for this can be found here. The list is kept in the settings.js under the editorTheme entry.

...
editorTheme: {
  pallete: {
    catalogues: [
      "http://catalogue.nodered.org/catalogue.json", //default catalogue
      'http://manager.example.com/catalogue.json'
    ]
  }
},
...

The URL should point to a JSON file that has the following format

{
  "name":"Ben's custom catalogue",
  "updated_at": "2016-08-05T18:37:50.673Z",
  "modules": [
    {
      "id": "@ben/ben-red-random",
      "version": "1.3.0",
      "description": "A node-red node that generates random numbers",
      "keywords": [
        "node-red",
        "random"
      ],
      "updated_at": "2016-08-05T18:37:50.673Z",
      "url": "http://flows.example.com/node/ben-red-random"
    },
    ...
  ]
}

There is a small script called build-catalogue.js in the manager app that will generate a catalogue.json file from a given npm repository.

Now there is a custom list of nodes we need to be able to load them with the npm command. There are a few options

  • Publish the node to npmjs.org but without the node-red keyword so they don’t end up being index on flows.nodered.org (and not manually submit them as you now need to)
  • Publish the node to a private npm repository that also acts as a pass through proxy to npmjs.org
  • Publish the node to a private npm repository with a scope and configure npm to use a different repository for a given scope.

The first option doesn’t need anything special settings up, you just add the nodes you want to the catalogue. The second two options need a private npm package repository. For the second it needs to act as a pass through so all the dependencies can also be loaded which is what makes the third option probably best.

I’ve been playing with running verdaccio on the same docker infrastructure as everything else and set npm to map the @private scope http://registry:4873 in the custom docker container.

...
RUN npm config set @private:registry http://registry:4873

Verdaccio can also proxy to npmjs.org if needed. (Like the nginx-proxy container I had to rebuild the verdaccio container to get it to run on my Pi4 since it only ships a AMD64 version)

I’m hosting my catalogue.json from the same Express application as is used to provision new Node-RED instances.

Screen shot of nodes listed on Verdaccio

Or if you want to prevent users being able to install/remove nodes then you can add:

...
editorTheme: {
  pallete: {
    editable: false
  }
},
...

Skinning Node-RED

The last thing on my list is to give the Node-RED instances a custom look and feel.

The basics like the page title, header image, favicon and logon screen graphic can all be set directly from the settings.js with the option to also link to a custom CSS style sheet so the colour scheme and shape/size of elements be changed as well. The design document can be found here.

editorTheme: {
  projects: {
    // To enable the Projects feature, set this value to true
    enabled: false
  },
  page: {
    title: "Ben-RED"
  },
  header: {
    title: "Ben-RED"
  },
  palette: {
    catalogues: [
      'https://catalogue.nodered.org/catalogue.json',
      'http://manager.example.com/catalogue.json'
    ]
  }
},

Managing Multi Tenant Node-RED Instances

Over the last series of posts I’ve outlined how to build all the components that are needed for a Multi Tenant Node-RED service. What’s missing is a way to automate the spinning up of new instances.

One option would be to do this with Node-RED it’s self, you can drive docker using the node-red-contrib-dockerode, but for this I’ve decided to create a dedicated application.

I’ve written a small Express app that uses dockerode directly and also will populate the Users collection in the MongoDB with the admin password for the editor and then spin up a new instance.

docker.createContainer({
  Image: "custom-node-red",
  name: req.body.appname,
  Env: [
    "VIRTUAL_HOST=" + req.body.appname + "." + settings.rootDomain,
    "APP_NAME="+ req.body.appname,
    "MONGO_URL=mongodb://mongodb/nodered"
  ],
  AttachStdin: false,
  AttachStdout: false,
  AttachStderr: false,
  HostConfig: {
    Links: [
      "mongodb:mongodb"
    ]
  }
})
.then(container => {
  console.log("created");
  cont = container;
  return container.start()
})
.then(() => {
  res.status(201).send({started: true, url: "http://" + req.body.appname + "." + settings.rootDomain});
})

It’s pretty basic but it does just enough to get started. It is being exposed using the same nginx-proxy that exposes the Node-RED instances, so the management interface is available on the manger.docker-pi.local domain. If it was being deployed in a production environment it should probably not be internet facing and should have some basic access control to prevent anybody from being able to stand up a new Node-RED instance.

When the app has completed creating a new instance a link to that instance is displayed.

You can also Start/Stop/Remove the instance as well as streaming the logs via a websocket.

Thanks to Dave Conway-Jones (@ceejay) for help with making my very utilitarian UI look a lot better.

The code for the management app is on github here

More MQTT VirtualHost Proxying

A really quick follow up to the earlier post about using TLS SNI to host multiple MQTT brokers on a single IP address.

In the previous post I used nginx to do the routing, but I have also worked out that the required input to Traefik would be.

The static config file looks like this

global:
  checkNewVersion: false
  sendAnonymousUsage: false
entryPoints:
  mqtts:
    address: ":1883"
api:
  dashboard: true
  insecure: true
providers:
  file:
    filename: config.yml
    directory: /config
    watch: true

And the dynamic config like this
tcp:
  services:
    test1:
      loadBalancer:
        servers:
          - address: "192.168.1.1:1883"
    test2:
      loadBalancer:
        servers:
          - address: "192.168.1.2:1883"
  routers:
    test1:
      entryPoints:
        - "mqtts"
      rule: "HostSNI(`test1.example.com`)"
      service: test1
      tls: {}
    test2:
      entryPoints:
        - "mqtts"
      rule: "HostSNI(`test2.example.com`)"
      service: test2
      tls: {}

tls:
  certificates:
    - certFile: /certs/test1-chain.crt
      keyFile: /certs/test1.key
    - certFile: /certs/test2-chain.crt
      keyFile: /certs/test2.key

Of course all the dymaic stuff can be generated via any of the Traefik providers.

Proxying for Mulitple Node-RED instances

So far we have worked out how to set up Node-RED to store flows in a database, use authentication to prevent unauthorised access and how to start multiple containerised instances under Docker.

In this post I will cover how to expose those multiple instances so their users can access them.

The easiest way to do this is to stick something like Nginx or Traefik in front of the docker containers and have it act as a reverse proxy. There are two ways we can set this up

  • Virtual Host based proxying – where each instance has it’s own hostname e.g. http://r1.example.com
  • Path based proxying – where each instance has a root path on the same hostname e.g. http://example.com/r1

In this case I’m going to use the first option of virtual hosts as currently Node-RED uses browser local storage to hold the Admin API security token and this is scoped to the host the editor is loaded from which means that you can only access one instance at a time if you use path based proxying. This is on the Node-RED backlog to get fixed soon.

To do that I’m going to use the nginx-proxy container. This container runs in the same Docker environment as the Node-RED instances and monitors all Docker to watch for containers starting/stopping. When it sees a new container start up it automatically create the right entry in the nginx configuration files and triggers it to reload the config.

To make this work I needed to add an extra environment variable to the command used to start the Node-RED containers

$ docker run -d --rm -e VIRTUAL_HOST=r1.example.com -e MONGO_URL=mongodb://mongodb/nodered -e APP_NAME=r1 --name r1 --link mongodb custom-node-red

I added the VIRTUAL_HOST environment variable which contains the hostname to use for this container. Th is means I can access this specific instance of Node-RED on http://r1.docker.local.

Traefik can be run in a similar way using labels instead of environment variables.

To make is all work smoothly I’ve added a wildcard domain entry to my local DNS that maps anything that matches *.example.com to the docker-pi.local machine that the container are running on.

Security

If I was going to run this exposed to the internet I’d probably want to enable HTTPS. To do this there are 2 options again

  • Use a separate certificate for each Virtual Host
  • Use a wildcard certificate that matches the wildcard DNS entry

I would probably go with the second option here as it is just one certificate you have to manage and even LetsEncrypt will issue a wildcard domain these days if you have access to the DNS.

For the first option there is a companion docker container for nginx-proxy that will use LetsEncrypt to issue certificates for each Virtual Host as they are started. It’s called letsencrypt-nginx-proxy-companion you can read how to use it in the nginx-proxy README.md

Limitations

Exposing Node-RED via a HTTP proxy does have one drawback. This approach means that only HTTP requests can directly reach the instances.

While this helps to offer a little more security, it also means that you will not be able to use the TCP-in or UDP-in nodes in server mode, that would allow arbitrary network connections into the instance. You will still be able to connect out from the instances to hosts as Docker provides NAT routing from containers to the outside world.

Sidebar

I’m testing all this on a Raspberry Pi4 running the beta of 64bit Raspberry Pi OS. I need this to get the official MongoDB container to work as they only formally support 64bit. As a result of this I had to modify and rebuild the nginx-proxy container because it only ships with support for AMD64 architectures. I had to build a ARM64 version of the forego and docker-gen packages and manually copy these into the container.

There is an outstanding pull-request open against the project to use a multistage-build which will build target specific binaries of forego and docker-gen which will fix this.

Hostname Based Proxying with MQTT

An interesting question came up on Stack Overflow recently that I suggested a hypothetical answer for how to do hostname based proxying for MQTT.

In this post I’ll explore how to actually implement that hypothetical solution.

History

HTTP added the ability to do hostname based proxying when it instroduced the Host header in HTTP v1.1. This meant that a single IP address could be used for many sites and the server would decide which content to serve based on the his header. Front end reverse proxies (e.g. nginx) can use the same header to decide which backend server to forward the traffic to.

This works well until we need encrypt the traffic to the HTTP server using SSL/TLS where the headers are encrypted. The solution to this is to use the SNI header in the TLS handshake, this tells the server which hostname the client is trying to connect to. The front end proxy can then either use this information to find the right local copy of the certificate/key for that site if it’s terminating the encryption at the frontend or it can forward the whole connection directly to the correct backend server.

MQTT

Since the SNI header is in the initial TLS handshake and is nothing to do with the underlying protocol it can be used for ay protocol, in this case MQTT. This measn if we set up a frontend proxy that uses SNI to pick the correct backend server to connect to.

Here is a nginx configuration file that proxies for 2 different MQTT brokers based on the hostname the client uses to connect. It is doing the TLS termination at the proxy before forwarding the clear version to the backend.

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

stream {  
  map $ssl_server_name $targetBackend {
    test1.example.com  192.168.1.1:1883;
    test2.example.com  192.168.1.2:1883;
  }

  map $ssl_server_name $targetCert {
    test1.example.com /certs/test1-chain.crt;
    test2.example.com /certs/test2-chain.crt;
  }

  map $ssl_server_name $targetCertKey {
    test1.example.com /certs/test1.key;
    test2.example.com /certs/test2.key;
  }
  
  server {
    listen 1883         ssl; 
    ssl_protocols       TLSv1.2;
    ssl_certificate     $targetCert;
    ssl_certificate_key $targetCertKey;
        
    proxy_connect_timeout 1s;
    proxy_pass $targetBackend;
  } 
}

Assuming the the DNS entries for test1.example.com and test2.example.com both point to the host running nginx then we can test this with the mosquitto_sub command as follows

$ mosquitto_sub -v -h test1.example.com -t test --cafile ./ca-certs.pem

This will be proxied to the broker running on 192.168.1.1, where as

$ mosquitto_sub -v -h test2.example.com -t test --cafile ./ca-certs.pem

will be proxied to the broker on 192.168.1.2.

Cavets

The main drawback with this approach is that it requires that all the clients connect using TLS, but this is not a huge problem as nearly all devices are capable of this now and for any internet facing service it should be the default anyway.

Acknowledgment

How to do this was mainly informed by the following Gist