An interesting question came up on Stack Overflow recently that I suggested a hypothetical answer for how to do hostname based proxying for MQTT.
In this post I’ll explore how to actually implement that hypothetical solution.
History
HTTP added the ability to do hostname based proxying when it instroduced the Host header in HTTP v1.1. This meant that a single IP address could be used for many sites and the server would decide which content to serve based on the his header. Front end reverse proxies (e.g. nginx) can use the same header to decide which backend server to forward the traffic to.
This works well until we need encrypt the traffic to the HTTP server using SSL/TLS where the headers are encrypted. The solution to this is to use the SNI header in the TLS handshake, this tells the server which hostname the client is trying to connect to. The front end proxy can then either use this information to find the right local copy of the certificate/key for that site if it’s terminating the encryption at the frontend or it can forward the whole connection directly to the correct backend server.
MQTT
Since the SNI header is in the initial TLS handshake and is nothing to do with the underlying protocol it can be used for ay protocol, in this case MQTT. This measn if we set up a frontend proxy that uses SNI to pick the correct backend server to connect to.
Here is a nginx configuration file that proxies for 2 different MQTT brokers based on the hostname the client uses to connect. It is doing the TLS termination at the proxy before forwarding the clear version to the backend.
Assuming the the DNS entries for test1.example.com and test2.example.com both point to the host running nginx then we can test this with the mosquitto_sub command as follows
$ mosquitto_sub -v -h test1.example.com -t test --cafile ./ca-certs.pem
This will be proxied to the broker running on 192.168.1.1, where as
$ mosquitto_sub -v -h test2.example.com -t test --cafile ./ca-certs.pem
will be proxied to the broker on 192.168.1.2.
Cavets
The main drawback with this approach is that it requires that all the clients connect using TLS, but this is not a huge problem as nearly all devices are capable of this now and for any internet facing service it should be the default anyway.
Acknowledgment
How to do this was mainly informed by the following Gist
The brackets and the back ticks mean you have to escape both if you try to add the label on the command line:
$ docker run --label traefik.enable=true --label traefik.http.routers.r1.rule=Host\(\`r1.docker.local\`\) nodered/node-red
I was struggling to get the Go template language that docker-gen uses to extract the hostname from the label value so I decided to write my own parser this time. It uses Dockerode to monitor the Docker events (for starting/stopping containers) and then parses out the needed details.
This is actually easier than the nginx version as the start event actually contains the container labels so there is no need to inspect the new container to get the environment variables.
if (eventJSON.status == "start") {
var keys = Object.keys(eventJSON.Actor.Attributes)
keys.forEach(key => {
if (re.test(key)) {
var host = eventJSON.Actor.Attributes[key].match(re2)[1]
cnames.push(host)
fs.writeFile("cnames",cnames.join('\n'),'utf8', err =>{})
}
})
} else if (eventJSON.status == "stop") {
var keys = Object.keys(eventJSON.Actor.Attributes)
keys.forEach(key => {
if (re.test(key)) {
var host = eventJSON.Actor.Attributes[key].match(re2)[1]
var index = cnames.indexOf(host)
if (index != -2) {
cnames.splice(index,1)
}
fs.writeFile("cnames",cnames.join('\n'), 'utf8', err => {})
}
})
}
I’ve also rolled in the nodemon to watch the output file and restart the python app that interfaces with the docker hosts instance of avahi. This removes the need for the forego app that was used for the nginx version.
I was recently approached by a company that wanted to sponsor adding Multi Tenant support to Node-RED. This would be to enable multiple users to run independent flows on a single Node-RED instance.
This is really not a good idea for a number of reasons. But mainly it is because the NodeJS runtime is inherently single threaded and there is no way to get real separation between different users. For example, if one user uses a poorly written node (or function in a function node) it is possible to block the event loop starving out all the other users or in extreme cases an uncaught asynchronous exception will cause the whole application to exit.
The best approach is to run a separate instance of Node-RED per user, which gives you the required separation between users. The usual way to do this is to use a container based system and a HTTP reverse proxy to control access to the instances.
Over the next months worth of posts I’ll outline the components required to build a system like I described and at the end should hopefully have a fully working Proof of Concept that people can use as the base to build their own deployments.
As the future posts are published I will add links here.
Required components
Flow Storage
Because containers file systems are not persistent we are going to need somewhere to reliably store the flows each user creates.
Node-RED supplies an API that lets you control how/where flows (and a bunch of things that would normally end up on disk)
We are going to need a way to only allow right users access to the Node-RED editor, again there is a plugin API that allows this to be wired into nearly any existing authentication source of your choice.
I wrote a simple implementation of this API which uses LDAP as the source of users not long after Node-RED was released . But in this series of post I’ll write a new one that uses the same backend database as the flow storage plugin.
Once we’ve built storage and authentication plugins we will need to build a custom version of the Node-RED Docker container that includes these extras.
Now we have a collection of containers each running a users instance of Node-RED we are going to need a way to expose these to the outside world. Since Node-RED is a web application then a HTTP Reverse Proxy is probably the right way forward.
Once all of the other pieces are in place we need is a way to control the creation/deletion of Node-RED instances. It will also be useful to see the instances’ logs.
Having built the nginx-proxy-avahi-helper container to expose proxied instances as mDNS CNAME entries on my LAN I also wanted a way to allow these containers to also be able to resolve other devices that are exposed via mDNS on my LAN.
By default the Docker DNS service does not resolve mDNS hostnames, it either takes the docker hosts DNS server settings or defaults to using Goolge’s 8.8.8.8 service.
The containers themselves can’t do mDNS resolution unless you set their network mode to host mode which isn’t really what I want.
You can pass a list of DNS servers to a docker container when it’s started with the --dns= command line argument which means that if I can run a bridge that will convert normal DNS requests into mDNS resquests on my LAN I should be able to get the containers to resolve the local devices.
I’ve played with writing DNS proxies before when I was looking at DoH so I had a reasonably good idea where to start.
This worked well but I ran into a small problem with the mdns-resolver library which wouldn’t resolve CNAMEs, but a small pull request soon fixed that.
The next spin of the code added support to send any request not for a .local domain to an upstream DNS server to resolve which means I don’t need to add as may DNS servers to each container.
I’ve been playing with the jwilder/nginx-proxy docker container recently. It automatically generates reverse proxy entries for other containers.
It does this by monitoring when new containers are started and then inspecting the environment variables attached to the new container. If the container has the VIRTUAL_HOST variable is uses it’s value as the virtual host for nginx to proxy access to the container.
Normally for this type of setup you would set up a wildcard DNS that points to the docker host so all DNS lookups for a machine in the root domain will return the same IP address.
If all the virtual hosts are in the example.com domain e.g. foo.example.com and bar.example.com you would setup the *.example.com DNS to point to the docker hosts IP address.
When working on my home LAN I normally use mDNS to access local machines so there is no where to set up the wildcard DNS entry. Instead I have build a container to add CNAME mDNS entries for the docker host for each of the virtual hosts.
In my case Docker is running on a machine called docker-pi.local and I’m using that as the root domain. e.g. manager.docker-pi.local or registry.docker-pi.local.
The container is using the docker-gen application that use templates to generate configuration files based on the running containers. In this case it generates a list of virtual hosts and writes them to a file.
The file is read by a small python application that connects to d-bus to talk to the avahi daemon on the docker host and configures the CNAMEs.
Both Docker and d-bus use unix domain sockets as their transport protocol so you have to mount the sockets from the host into the container.
docker run -d -v /run/dbus/system_bus_socket:/run/dbus/system_bus_socket -v /var/run/docker.sock:/tmp/docker.sock --name avahi-cname hardillb/nginx-proxy-avahi-helper
I’ve put the code on github here and the container is on docker hub here.
A LONG time ago I started to work out how to build a linear clock using a strip of 60 LEDs. It’s where all my playing with using Pi Zeros and Pi 4s as USB devices started from.
I’ve had a version running on the desk with jumper wires hooking everything up, but I’ve always wanted to try and do something a bit neater. So I’ve been looking at building a custom Pi pHAT to link everything together.
The design should be pretty simple
Break out pin 18 on the Pi to drive the ws2812b LEDs.
Supply 5v to the LED strip.
Include an I2C RTC module so the Pi will keep accurate time, especially when using a Pi Zero (not a Pi Zero W) with no network connectivity.
I know that the Pi can supply enough power from the 5v pin in the 40 pin header to drive the number of LEDs that the clock should have lit at any one time.
Also technically the data input for the ws2812b should also be 5v but I know that at least with the strip that I have it will work with the 3.3v supplied from GPIO pin 18.
I had started to work on this with Eagle back before it got taken over by Autodesk, while there is still a free version for hobbyists I thought I’d try something different this time round and found librePCB.
Designing the Board
All the PCB board software looks to work in a similar way, you start by laying out the circuit logically before working out how to lay it out physically.
The RTC is going to need a battery to keep it up to date if the power goes out so I’m using the Keystone 500 holder which is for a 12mm coin cell. These are a lot smaller than the more common 20mm (cr2032) version of coin cells, so should take up lot less space on the board.
I also checked that the M41T81 has a RTC Linux kernel driver and it looks to be included in the rtc-m41t80 module, so should work.
Finally I’m using the terminal block because I don’t seem to be able to find a suitable board mountable connector for the JST SM connector that comes on most of the LED strips I’ve seen. The Wikipedia entry implies that the standard is only used for wire to wire connectors.
Circuit layout
LibrePCB has a number of component libraries that include one for the Raspberry Pi 40 pin header.
But I had to create a local library for some of the other parts, especially the RTC and the battery holders. I will look at how to contribute these parts to the library once I’ve got the whole thing up and running properly.
The Pi has built in pull up resistors on the I2C bus pins so I shouldn’t need to add any.
Board layout
Now I have all the components linked up it was time to layout the actual board.
The board dimensions are 65mm x 30mm which matches the Pi Zero and with the 40 pin header across the top edge.
The arrangement of the pins on the RTC mean that no matter which way round I mount it I always end up with 2 tracks having to cross which means I have one via to the underside of the board. Everything else fits into one layer. I’ve stuck the terminal block on the left hand edge so the strip can run straight out from there and the power connector for the Pi can come in to the bottom edge.
Possible Improvements
An I2C identifier IC – The Raspberry Pi HAT spec includes the option to use the second I2C bus on the Pi to hold the device tree information for the device.
A power supply connector – Since the LED strip can draw a lot of power when all are lit, it can make sense to add either a separate power supply for just the LEDs or a bigger power supply that can power the Pi via the 5v GPIO pins.
A 3.3v to 5v Level shifter – Because not all the LED strips will work with 3.3v GPIO input.
Find a light level sensor so I can adjust the brightness of the LED strip based on the room brightness.
I now need to send off the design to get some boards made, order the components and try and put one together.
Getting boards made is getting cheaper and cheaper, an initial test order of 5 boards to test from JLC PCB came in at £1.53 + £3.90 shipping (this looks to include a 50% first order discount). This has a 12 day shipping time, but I’m not in a rush so this just feels incredibly cheap.
Soldering the surface mount RTC is going to challenge my skills a bit but I’m up for giving it a go. I think I might need to buy some solder paste and a magnifier.
I’ll do another post when the parts all come in and I’ve put one together.
While playing with the log analyser I mentioned in my post about building fail2ban rules, I accidentally overwrote the index.html file for my homepage.
After a bit of digging around and not being able to find a copy of it anywhere (it was probably on my old work laptop, so might be on the backup USB drive in the bottom draw…) I started to write a new one. To make sure I could always find a copy I decided I’d check it into GitHub.
As part of setting up the new repo I thought I’d have a look at the relatively new GitHib Actions that allow you to run jobs on your own hardware.
Actions are GitHubs implementation of a CI pipeline, you can attach an action to any number of events that happen on a repo e.g.
Post a welcome message when somebody new opens an issue
Run tests on any new pull requests
Deploy the project when code is merged into a specific branch
This attaches to the push trigger and asks that it is run on a self-hosted, linux machine. Where it will checkout the /hardillb/homepage repository and then execute the deploy.sh script found in the scripts directory of the repo.
Github supply runners for Windows/Linux/macOS on a range of architectures including ARM which means I can deploy one on the Raspberry Pi that hosts my website. Details of how to add a Action Runner to your specific project and download the package can be found here.
I went with running my own Action Runner on the same machine as the site is hosted because it means I don’t need to worry about setting up a custom ssh key and storing it as a secret so a cloud runner could access the device and upload the changed files. This way the script runs on the same machine and I can make sure the user has access rights to the right directory to copy the files to.
Having deployed my Adafruit Feather version of a Temperature/Humidity LoRa Sensor to make use of my The Things Network gateway it’s now time to build another sesnor, this time with a Raspberry Pi Zero.
I have a Pi Supply LoRa node pHAT that adds the radio side of things, but I needed a sensor. The pHAT has a header on the right hand edge which pulls out the 3.3v, 2, 3, 4 and Gnd, pins from the pi.
These just happen to line up with the I2C bus. They are also broken out in the same order as used by the Pimoroni Breakout Garden series of I2C sensors which is really useful.
Pimoroni’s Breakout Garden offer a huge range of sensors and output devices in either I2C or SPI. In this case I grabbed the BME680 which offers temperature, pressure, humidity and air quality in a single package.
Pimoroni supply a python library to read from the sensor so it was pretty easy to combine this with python to drive the LoRa pHat
I’m only using the temperature and humidity values at the moment to help keep the payload compatible with the feather version so I can use a single “port” in the TTN app.
I’ve put all the code for both this and the Feather version on GitHub here.
As well as the Pi Zero version I’ve upgraded the Feather with a 1200mAh LiPo battery and updated the code to also transmit the battery voltage so I can track battery life and work out how often I need to charge it.
Adding the battery also meant I could stick it in my bag and go for a walk round the village to see what sort of range I’m getting.
I’m currently using a little coil antenna on the gateway that is laid on it’s side, I’ll find something to prop it up with which should help. The feather is also only using a length of wire antenna so this could be seen as a worst case test.
It’s more than good enough for what I need right now, but it would be good to put a proper high gain one in then it might be useful to more people. It would be great if there was a way to see how many unique TTN applications had forwarded data through a gateway to see if anybody else has used it.
After deploying the TTN Gateway in my loft last year, I’ve finally got round to deploying a sensor to make use of it.
As the great lock down of 2020 kicked off I stuck in a quick order with Pimoroni for a Adafruit Feather 32u4 with an onboard LoRa radio (In hindsight I probably should have got the M0 based board).
I already had a DHT22 temperature & humidity and the required resistor so it was just a case of soldering on an antenna of the right length and adding the headers so I could hook up the 3 wires needed to talk to the DHT22.
There is an example app included in with the TinyLoRa library that reads from the sensor and sends a message to TTN every 30 seconds.
(There are full instructions on how to set everything up on the Adafruit site here)
Now the messages are being published to TTN Application I needed a way to consume them. TTN have a set of Node-RED nodes which make this pretty simple. Though installing them did lead to a weekend adventure of having to upgrade my entire home automation system. The nodes wouldn’t build/install on Node-RED v0.16.0, NodeJS v6 and Raspbian Jessie so it was time to upgrade to Node-RED v1.0.4, NodeJS v12 and Raspbian Buster.
This flow receives messages via MQTT direct from TTN, it then separates out the Temperature and Humidity values, runs them through smooth nodes to generate a rolling average of the last 20 values. This is needed because the DHT22 sensor is pretty noisy.
It outputs these smoothed values to 2 charts on a Node-RED Dashboard. which show the trend over the last 12 hours.
It also outputs to 2 MQTT topics that mapped to my public facing broker, which means they can be used to resurrect the temperature chart on my home page, and when I get an outdoor version of the sensor (hopefully solar/battery powered) up and running it will have both lines that my old weather station used to produce.
The last thing the flow does is to update the temperature reading for my virtual thermostat in the Google Assistant HomeGraph. This uses the RBE node to make sure that it only sends updates when the value actually changes.
The next step is to build a couple more and find a suitable battery and enclosure so I can stick one in the back garden.
