Building a Kubernetes Test Environment

Over the last couple of weekends I’ve been noodling around with my home lab set up to build a full local environment to test out FlowForge with both the Kubernetes Container and Docker Drivers.

The other reason to put all this together is to help to work the right way to put together a proper CI pipeline to build, automatically test and deploy to our staging environment.

Components

NPM Registry

This is somewhere to push the various FlowForge NodeJS modules so they can then be installed while building the container images for the FlowForge App and the Project Container Stacks.

This is a private registry so that I can push pre-release builds without them slipping out in to the public domain, but also so I can delete releases and reuse version numbers which is not allowed on the public NPM registry.

I’m using the Verdaccio registry as I’ve used this in the past to host custom Node-RED nodes (which it will probably end up doing again in this set up as things move forwards). This runs as Docker container and I use my Nginx instance to reverse proxy for it.

As well as hosting my private builds it can proxy for the public npmjs.org regisry which speeds up local builds.

Docker Container Registry

This is somewhere to push the Docker containers that represent both the FlowForge app it’s self and the containers that represent the Project Stacks.

Docker ship a container image available that will run a registry.

As well as the registry I’m also running second container with this web UI project to help keep track of what I’ve pushed to the registry and also allows me to delete tags which is useful when testing

Again my internet facing Nginx instance is proxying for both of these (on the same virtual host since their routes do not clash and it makes CORS easier since the UI is all browser side JavaScript)

Helm Chart Repository

This isn’t really needed, as you can generate all the required files with the helm command and host the results on any Web server, but this lets me test the whole stack end to end.

I’m using a package called ChartMuseum which will automatically generate index.yaml manifest file when charts are uploaded via it’s simple UI.

Nginx Proxy

All of the previous components have been stood up as virtual hosts on my public Nginx instance so that they can get HTTPS certificates from LetsEncrypt. This is makes things a lot easier because both Docker and Kubernetes basically require the container registry be secure by default.

While it is possible to add exceptions for specific registries, these days it’s just easier to do it “properly” up front.

MicroK8s Cluster

And finally I need a Kubernetes cluster to run all this on. In this case I have a 3 node cluster made up of

  • 2 Raspberry Pi 4s with 8gb of RAM each
  • 1 Intel Celeron based mini PC with 8gb of RAM

All 3 of these are running 64bit Ubuntu 20.04 and MicroK8s. The Intel machine is needed at the moment because the de facto standard PostrgresSQL Helm Chat only have amd64 based containers at the moment so won’t run on the Raspberry Pi based nodes.

The cluster uses the NFS Persistent Volume provisioner to store volumes on my local NAS so they are available to all the nodes.

Usage

I’ll write some more detailed posts about how I’ve configured each of these components and then how I’m using them.

As well as testing the full Helm chart builds, I can also use this to run the FlowForge app locally and have the Kubernetes Container driver running locally on my development machine and have it create Projects in the Kubernetes cluster.

Pimoroni Keybow Upgrade

I’ve had a little 3 key Pimoroni Keybow sat on my desk for a while. It was running the same basic config I had setup when I bought it, namely mapping the three buttons to volume down, mute and volume up respectively.

Pimoroni Keybow Mini

While this was useful, it felt like there where better uses for it.

With more and more time being spent in video meetings having quick shortcuts to mute the mic or toggle the camera on/off sounded like a good idea. But then I wondered if I could find a way to switch the key mapping on the fly.

The key mapping is done by editing a short Lua script. This is stored on the sdcard that the Pi Zero that holds the Keybow boots from. This means the layout is normally fixed. Except the latest version (0.0.4) of the sdcard image on the Pimoroni Github page added support for starting a USB serial link as well as the HID used to send the keyboard events. This is exposed in the Lua environment so I managed to build the following script.

require "keybow"

-- Keybow MINI volume/zoom controls --

function setup()
    keybow.use_mini()
    keybow.auto_lights(false)
    keybow.clear_lights()
    keybow.set_pixel(0, 0, 255, 255)
    keybow.set_pixel(1, 255, 0, 255)
    keybow.set_pixel(2, 0, 255, 255)
end

-- Key mappings --

state = 'zoom'

function handle_minikey_02(pressed)
    if state == 'zoom' then
	if pressed then
            keybow.set_modifier(keybow.LEFT_ALT, keybow.KEY_DOWN)
            keybow.tap_key("v")
            keybow.set_modifier(keybow.LEFT_ALT, keybow.KEY_UP)
        end
    elseif state == 'media' then
        keybow.set_media_key(keybow.MEDIA_VOL_UP, pressed)
    end
end

function handle_minikey_01(pressed)
    if state == 'zoom' then
	if pressed then
            keybow.set_modifier(keybow.LEFT_ALT, keybow.KEY_DOWN)
            keybow.tap_key("a")
            keybow.set_modifier(keybow.LEFT_ALT, keybow.KEY_UP)
        end
    elseif state == 'media' then
        keybow.set_media_key(keybow.MEDIA_MUTE, pressed)
    end
end

function handle_minikey_00(pressed)
    if state == 'zoom' then
	if pressed then
             keybow.set_modifier(keybow.LEFT_ALT, keybow.KEY_DOWN)
             keybow.tap_key("n")
             keybow.set_modifier(keybow.LEFT_ALT, keybow.KEY_UP)
        end
    elseif state == 'media' then
        keybow.set_media_key(keybow.MEDIA_VOL_DOWN, pressed)
    end
end

local function isempty(s)
  return s == nil or s == ''
end

function tick()
    local line
    line = keybow_serial_read()
    if not isempty(line) then 
        -- keybow_serial_write( line .. "\n" )
        if line == 'zoom' then
            keybow.clear_lights()
            keybow.set_pixel(0, 0, 255, 255)
            keybow.set_pixel(1, 255, 0, 255)
            keybow.set_pixel(2, 0, 255, 255)
            state = 'zoom'
        elseif line == 'media' then
            keybow.clear_lights()
            keybow.set_pixel(0, 255, 0, 255)
            keybow.set_pixel(1, 0, 255, 255)
            keybow.set_pixel(2, 255, 0, 255)
            state = 'media'
        end
    end

end

The serial port gets setup on /dev/ttyACM0 on my laptop so I’m toggling between the 2 modes with echo media > /dev/ttyACM0 and echo zoom > /dev/ttyACM0.

In media mode it works exactly the same as before, but in zoom mode it toggles the camera on/off, toggles mute on/off and cycles through the available cameras.

This worked but keybow_serial_read() call added 1 second of latency to each call to the tick function which really wasn’t great as it was possible to miss key presses.

A bit of digging in the git rep turned up the file that implemented the serial access and this bit of code:

int serial_open(){
    if(port_fd > -1) return 0;

    port_fd = open(KEYBOW_SERIAL, O_RDWR);

    if(port_fd > -1){
        printf("Open success\n");
        tcgetattr(port_fd, &termios);
        termios.c_lflag &= ~ICANON;
        termios.c_cc[VTIME] = 10;
        termios.c_cc[VMIN] = 0;
        tcsetattr(port_fd, TCSANOW, &termios);
    }
    return 0;
}

The termios.c_cc[VTIME] = 10; was what was causing the delay. I rebuilt the library changing the value to 1 and 0. The value is in deciseconds (1/10 seconds)

With 1 the delay was cut to a tenth of a second, which was OK, but meant you had to be very deliberate in pushing the button to make sure it didn’t get missed, which with a mute toggle is a little risky.

With 0 it worked perfectly.

The script also changes the backlight colour for the keys based on mode so I can see which is active. It should be possible to add more modes as needed.

Next up is to see if I can script the toggling the mode based on if Zoom is the currently active window. Looks like it should be possible with tools like xprop or xdotool.

Determining which Linux Distro you are on to install NodeJS

I’ve recently been working on an install script for a project. As part of the install I need to check if there is a suitable version of NodeJS installed and if not install one.

The problem is that there are 2 main ways in which NodeJS can be installed using the default package management systems for different Linux Distributions. So I needed a way to work out which distro the script was running on.

The step was to work out if it is actually Linux or if it’s OSx, since I’m using bash as the interpreter for the script there is the OSTYPE environment variable that I can check.

case "$OSTYPE" in
  darwin*) 
    MYOS=darwin
  ;;
  linux*)
    MYOS=$(cat /etc/os-release | grep "^ID=" | cut -d = -f 2 | tr -d '"')
  ;;
  *) 
    # unknown OS
  ;;
esac

Once we are sure we are on Linux the we can check the /etc/os-release file and cut out the ID= entry. The tr is to cut the quotes off (Amazon Linux I’m looking at you…)

MYOS then contains one of the following:

  • debian
  • ubuntu
  • raspbian
  • fedora
  • rhel
  • centos
  • amzon

And using this we can then decide how to install NodeJS

if [[ "$MYOS" == "debian" ]] || [[ "$MYOS" == "ubuntu" ]] || [[ "$MYOS" == "raspbian" ]]; then
      curl -sSL "https://deb.nodesource.com/setup_$MIN_NODEJS.x" | sudo -E bash -
      sudo apt-get install -y nodejs build-essential
elif [[ "$MYOS" == "fedora" ]]; then
      sudo dnf module reset -y nodejs
      sudo dnf module install -y "nodejs:$MIN_NODEJS/default"
      sudo dnf group install -y "C Development Tools and Libraries"
elif [[ "$MYOS" == "rhel" ]] || [[ "$MYOS" == "centos" || "$MYOS" == "amzn" ]]; then
      curl -fsSL "https://rpm.nodesource.com/setup_$MIN_NODEJS.x" | sudo -E bash -
      sudo yum install -y nodejs
      sudo yum group install -y "Development Tools"
elif [[ "$MYOS" == "darwin" ]]; then
      echo "**************************************************************"
      echo "* On OSx you will need to manually install NodeJS            *"
      echo "* Please install the latest LTS release from:                *"
      echo "* https://nodejs.org/en/download/                            *"
      echo "**************************************************************"
      exit 1
fi

Now that’s out of the way time to look at how to nicely setup a Systemd service…

Debugging Node-RED nodes with Visual Code

A recent Stack Overflow post had me looking at how to run Node-RED using Visual Code to debug custom nodes. Since I’d not tried Visual Code before (I tend to use Sublime Text 4 as my day to day editor) I thought I’d give it a go and see if I could get it working.

We will start with a really basic test node as an example. This just prints the content of msg.payload to the console for any message passing through.

test.js

module.exports = function(RED) {
    function test(n) {
        RED.nodes.createNode(this,n)
        const node = this
        node.on('input', function(msg, send, done){
            send = send || function() { node.send.apply(node,arguments) }
            console.log(msg.payload)
            send(msg)
            done()
        })
    }
    RED.nodes.registerType("test", test)
}

test.html

<script type="text/html" data-template-name="node-type">
</script>

<script type="text/html" data-help-name="node-type">
</script>

<script type="application/javascript">
    RED.nodes.registerType('test',{
        category: 'test',
        defaults: {},
        inputs: 1,
        outputs: 1,
        label: "test"
    })
</script>

package.json

{
  "name": "test",
  "version": "1.0.0",
  "description": "Example node-red node",
  "keywords": [
    "node-red"
  ],
  "node-red": {
    "nodes": {
      "test": "test.js"
    }
  },
  "author": "ben@example.com",
  "license": "Apache-2.0"
}

Setting up

All three files mentioned above are placed in a directory and then the following steps are followed:

  • In the Node-RED userDir (normally ~/.node-red on a Linux machine) run the following command to create a symlink in the node_modules directory. This will allow Node-RED to find and load the node.
    npm install /path/to/test/directory
  • Add the following section to the package.json file
...
  ],
  "scripts": {
    "debug": "node /usr/lib/node_modules/node-red/red.js"
  },
  "node-red": {
...

Where usr/lib/node_modules/node-red/red.js is the output from readlink -f `which node-red`.

You can then add a breakpoint to the code

View of node's javascript code with break point set on line 7

And then start Node-RED by clicking on the Play button just above the scripts block.

view of node's package.json with play symbol and Debug above the scripts block

This will launch Node-RED and attach the debugger and stop when the breakpoint if hit. You can also enable the debugger to stop the application on exceptions, filtering on if they are caught or not.

This even works when using Visual Code’s remote capabilities for editing, running and debugging projects on remote machines. I’ve tested this running over SSH to a Raspberry Pi Zero 2 W (which is similar to the original StackOverflow question as they were trying to debug nodes working with the Pi’s GPIO system). The only change I had to make on the Pi was to increase the default swap file size from 100mb to 256mb as squeezing the Visual Code remote agent and Node-RED into 512mb RAM is a bit of a squeeze.

I might give Visual Code a go as my daily driver in the new year.

Test Certificates for localhost

While answering a couple of Stack Overflow questions recently I needed to create some certificates to use with localhost so I thought I’d record the steps to I would have something to link to next time.

Generate CA cert

$ openssl genrsa -out ca.key 2048
$ openssl req -new -x509 -days 365 -key ca.key \ 
  -subj "/C=GB/ST=Gloucestershire/O=localhost CA/CN=locahost Root CA" \
  -out ca.pem

Generate Server cert

$ openssl req -newkey rsa:2048 -nodes -keyout server.key \
  -subj "/C=GB/ST=Gloucestershire/O=Localhost CA/CN=localhost" \
  -out server.csr
$ openssl x509 -req \
  -extfile <(printf "subjectAltName=DNS:localhost,IP:127.0.0.1,IP:::1") \
  -days 365 -in server.csr -CA ca.pem -CAkey ca.key \
  -CAcreateserial -out server.pem

The outputs are

  • ca.key the private key for the CA
  • ca.pem the CA certificate
  • server.key the private key for the server
  • server.pem the certificate fro the server

Traditionally the certificates Subject’s CN value has contained the hostname of the machine the certificate is representing. But the spec doesn’t actually assign any specific meaning to this field and it was deprecated as part of RFC2818.

v3 of the x509 spec adds an extension for storing hostnames and IP addresses called Subject Alternative Names (known as SAN). The last line in the instructions adds SANs for the hostname localhost and the IP addresses 127.0.0.1 and ::1. This means it should be valid for all possible ways of accessing localhost.

The Linear Clock Ticks Again

I’ve had a background project ticking over slowly in the background for a number of years.

Last year I designed and had built a number of PCBs to be used as HATs for a Raspberry Pi Zero. They included a RTC and a terminal block to attach the LED strip.

I did say that I would write another post when the boards where delivered and I had assembled the first prototype. Unfortunately I had made a small, but critical mistake when designing the boards, I slightly messed up the package package size for the RTC so it wasn’t possible to get assemble the boards correctly. I didn’t get round to re-doing the PCB layout with the correct sized parts so the whole thing just sat for a while.

In the meantime the Raspberry Pi Foundation went and released a new product, the Raspberry Pi Pico, which is based on the RP2040 chip. As well as the Pico they are also making the RP2040 chip available to other folk to include it directly in their own projects.

Pimoroni have created a number of different boards but their latest is the Plasma 2040 which is specifically designed to drive LED strips.

B.O.M.

Assembly

  • Solder the RTC on to the breakout section of the Plasma 2040, the terminals are labelled so just make sure you match up the pins, I used the headers that came with the RTC and arranged it so the breakout was over the top of the Plasma2040
  • Loosen the screw terminals for the connections marked 5V, DA and -. Insert the Red wire of the adapter in the 5V, Green wire in DA and White wire in –
  • Clip the LED strip to the end of the adapter.
Plasma 2040

Code

When you first attach the Plasma2040 to your computer it will show up as a USB flash drive. This is so you can install the runtime. In this case we’ll be using the Pimoroni Micropython build that comes with support for the board. You can grab a version from the release page on GitHub here. Once downloaded copy it into the root of the drive. When the copy has finished the board will reboot and be ready to run Python code.

You can use the Thonny IDE to both write and push code to the device. You will need at least version 3.3.3 to support the Plasma2040.

The fist version of the code was as follows:

import plasma
from plasma import plasma2040
from pimoroni import RGBLED, Button
import time

NUM_LEDS = 60
LOW = 32
MED = 64
HIGH = 128
BRIGHTNESS = [LOW,MED,HIGH]
BRIGHTNESS_LEVEL = 0

button_brightness = Button(plasma2040.BUTTON_A)

led = RGBLED(plasma2040.LED_R, plasma2040.LED_G, plasma2040.LED_B)
led.set_rgb(0, 0, 0)
led_strip = plasma.WS2812(NUM_LEDS, 0, 0, plasma2040.DAT)

led_strip.start()

while True:
    RED = [0]*NUM_LEDS
    GREEN = [0]*NUM_LEDS
    BLUE = [0]*NUM_LEDS
    t = time.localtime()

    hour = (t[3] % 12) * 5
    #Hours
    RED[hour] = BRIGHTNESS[BRIGHTNESS_LEVEL]
    RED[hour + 1] = BRIGHTNESS[BRIGHTNESS_LEVEL]
    RED[hour + 2] = BRIGHTNESS[BRIGHTNESS_LEVEL]
    RED[hour + 3] = BRIGHTNESS[BRIGHTNESS_LEVEL]
    RED[hour + 4] = BRIGHTNESS[BRIGHTNESS_LEVEL]
    #Mins
    GREEN[t[4]] = BRIGHTNESS[BRIGHTNESS_LEVEL]
    #Secs
    BLUE[t[5]] = BRIGHTNESS[BRIGHTNESS_LEVEL]
    
    #set the LEDS
    for i in range (NUM_LEDS):
        led_strip.set_rgb(i, RED[i], GREEN[i], BLUE[i])
    
    #change brightness
    if button_brightness.read():
        BRIGHTNESS_LEVEL += 1
        BRIGHTNESS_LEVEL %= 3
    
    time.sleep(1)
 

This works well when triggered from Thonny as it syncs the laptop’s time to the RP2040 each time it connects. But when the clock is powered by a USB power supply or a battery, the clock starts at 00:00:01 Jan 1st 2021 and has no way to be updated to match now.

This is why we need the RTC module, it keeps track of the time while the clock is powered down.

It also has a way to change the brightness, by pressing the A button it will cycle through 3 different brightness levels.

Setting the RTC Time

With a little bit of playing I worked out how to sync the RTC to the current time in the Thonny console

>>> from pimoroni_i2c import PimoroniI2C
>>> from breakout_rtc import BreakoutRTC
>>> import time
>>> PINS_PLASMA = {"sda": 20, "scl": 21}
>>> i2c = PimoroniI2C(**PINS_PLASMA)
>>> rtc = BreakoutRTC(i2c)
>>> rtc.set_unix(time.time())
>>> rtc.set_time(54,18,17,6,18,9,2021)
True
>>> rtc.update_time()
True
>>> print(rtc.string_time())
17:18:54
>>> rtc.set_backup_switchover_mode(3)

The most important line is the last one, which enables the battery backup for the RTC so it remembers the time you just set.

I was going to use the rtc.set_unix() function and pass in time.time() but it appears that the unix timestamp is maintained independently of the “Real” time on the RTC.

The set_time() function takes values in the order

  • seconds (0-60)
  • minutes (0-60)
  • hours (0-23)
  • day of the week (1-7 -> mon-sun)
  • day of month (1-31)
  • monthe (1-12)
  • year (2000-2099)

With the RTC set correctly a small update to the code to read from the RTC rather than from the time object and we are good to go.

import plasma
from plasma import plasma2040
from pimoroni import RGBLED, Button
from pimoroni_i2c import PimoroniI2C
from breakout_rtc import BreakoutRTC
import time

PINS_PLASMA = {"sda": 20, "scl": 21}

i2c = PimoroniI2C(**PINS_PLASMA)
rtc = BreakoutRTC(i2c)

if rtc.is_12_hour():
    rtc.set_24_hour()

if rtc.update_time():
    print(rtc.string_time())
    print(rtc.string_date())

NUM_LEDS = 60
LOW = 32
MED = 64
HIGH = 128
BRIGHTNESS = [LOW,MED,HIGH]
BRIGHTNESS_LEVEL = 0

button_brightness = Button(plasma2040.BUTTON_A)

led = RGBLED(plasma2040.LED_R, plasma2040.LED_G, plasma2040.LED_B)
led.set_rgb(0, 0, 0)
led_strip = plasma.WS2812(NUM_LEDS, 0, 0, plasma2040.DAT)

led_strip.start()

rtc.enable_periodic_update_interrupt(True)

while True:
    RED = [0]*NUM_LEDS
    GREEN = [0]*NUM_LEDS
    BLUE = [0]*NUM_LEDS
    t = time.localtime()

    if rtc.read_periodic_update_interrupt_flag():
        rtc.clear_periodic_update_interrupt_flag()
         
        rtc.update_time()
        hour = (rtc.get_hours() % 12) * 5
        RED[hour] = BRIGHTNESS[BRIGHTNESS_LEVEL]
        RED[hour + 1] = BRIGHTNESS[BRIGHTNESS_LEVEL]
        RED[hour + 2] = BRIGHTNESS[BRIGHTNESS_LEVEL]
        RED[hour + 3] = BRIGHTNESS[BRIGHTNESS_LEVEL]
        RED[hour + 4] = BRIGHTNESS[BRIGHTNESS_LEVEL]
        GREEN[rtc.get_minutes()] = BRIGHTNESS[BRIGHTNESS_LEVEL]
        BLUE[rtc.get_seconds()] = BRIGHTNESS[BRIGHTNESS_LEVEL]

        for i in range (NUM_LEDS):
            led_strip.set_rgb(i, RED[i], GREEN[i], BLUE[i])
        
        if button_brightness.read():
            BRIGHTNESS_LEVEL += 1
            BRIGHTNESS_LEVEL %= 3
    
    time.sleep(1)
2021 Edition

Next Steps

There are a few things that need doing next. The first is to build a case for the clock, I’m thinking about something made up of layers of thin plywood with a channel for the LED strip and maybe a layer of smoked/mat acrylic to act as a diffuser.

The second part is to work out a way to work with DST, Micropython doesn’t support timezones as the database needed to keep track of all the different timezones takes up a huge amount of space. I could hard code in the dates for my location, but I’ll probably just make use of the B button to toggle an hours difference on/off.

Optionally I might add another 31 LED strip (probably at 30/meter) to be used as a calendar showing the current month with markers for weekends and the current day.

Another option is to use 4 of these to build a 60 LED ring for something a little more conventionally shaped.

And the final extra hack is to daisy chain the Light level sensor (e.g. one of these) on top of the RTC and dynamically adjust the brightness based on ambient light levels.

I’ll also probably keep tinkering with the Raspberry Pi Zero W version as that will allow oAuth to link to things like Google Calendar to show meetings in the clock view and add Holidays to the Calendar view. It will also have access to the full timezone database and NTP for time syncing over the network.

Router swap

With all the working from home over the last 18months and the fact I now work for a 100% remote company I decided it was time to have another look at my home broadband setup.

I currently have a FTTC install supplied by A&A which currently tops out at about 60/15 and while a FTTP setup would be nice I’ll have to wait until OpenReach get their finger out and actually fully enable my exchange (A recent new build development is already full fibre, but the existing properties will have to wait).

The line has been pretty reliable but I decided it was time to add some backup capability if I’m going to be relying on it all the time. I decided to add an LTE/4G link (no 5G out here in the sticks yet either).

I already had a LTE USB stick but the Ubiquiti EdgeRouter X that I was running didn’t have a USB port so I looked at putting the stick in Pi and adding a second low priority default route via the Pi. This worked but meant that I lost IPv6 (finding a UK cell provider that will offer IPv6 on Pay&Go is a problem I’ve looked at before) and others won’t be able to reach my web server or the other services I host at home. I’ll cover the 4G network provision later.

A&A offer a L2TP service which can route the fixed IPv4 and IPv6 ranges over any connection if your main line is down for any reason. This can easily run over a LTE connection, but it does have one slight niggle. If the L2TP tunnel is running at the same time as the FTTC line then it will take priority which means it should only be started when the FTTC line goes down.

The EdgeRouter X only supports L2TP Tunnels when paired with IPSEC so can’t easily be used with this option. I could run something like xl2tp on the Pi with the LTE USB stick but then I would need a way to trigger it on the Pi when the PPPoE link goes down on the EdgeRouter. All of this combined with Ubiquiti’s apparent pulling back from the EdgeRouter line as they focus more on their Dream Machine range I thought I’d see what else was available.

MikroTik

If you poke around the internet in the places where people talk about Ubiquiti kit they also mention MikroTik and RouterOS so I thought I’d have a look and see what was available.

MikroTik hEX s router

The closest match to the EdgeRouter X looked to be a MikroTik hEX S. It has the same 5 Gigabit Ethernet ports, PoE powered and also has a USB port and a SFP port for if I ever want to add fibre support.

I already had a Huawei E3372-200 LTE stick to plug into the side. This supports up to 150Mbps connections and has connectors to add external antenna if needed to get the best signal. I also grabbed a 90° USB adapter, because everybody knows that USB sticks work better when pointed straight up.

Router & Switch

I plugged the hEX S into my desktop ISP setup to work out how configure it and play with some of the settings.

There are 3 ways to configure most RouterBoard/RouterOS devices

  • Winbox – a native application that supports Windows (can be run under Wine on Linux and OSx)
  • WebFig – a web interface
  • Console/SSH – a command line interface

I’ve not tried Winbox, I did most of the setup via the console interface, but I used the WebFig to check. Most of the time WebFig works just fine, but occasionally it would throw javascript errors. I’m hoping that most of this is down to the fact I had to install a 7.1 release candidate build to get LTE stick to work properly. I’ll check back once 7.1 gets a proper release.

Using the console I managed to setup the LAN IP address range, DHCP server and pre-reserved all the static IP addresses to match my old setup.

Getting the port forwarding and hairpin NAT setup was a little bit more challenging than on the EdgeRouter but I have something that looks to behave the same for everything I had setup before.

I set the LTE device to be always on but with a static route to the L2TP endpoint and a script that run when the PPPoE device goes up or down. When the PPPoE goes down it will connect the L2TP client and disconnect it when the PPPoE device comes back up. The easiest way to test is to unplug the ethernet cable between the router and the modem running in bridge mode.

Cellular Contract

The next question is what mobile data plan to use, this is meant to be only used as a fall back, so I don’t really want to be paying for a monthly contract and then not using it, which means I’m looking for a Pay & Go sim card. I also want a plan that has the longest possible lifetime for any credit. Luckily Terrence Eden had recently collated a list of the best deals for this kind of data sim. It looks like the Three 24GB or the matching Vodafone 24GB plan are the best fit.

I opted for the Three as I have reasonable coverage at home, it comes with 24GB pre-loaded and it will last for up to 2 years (unlike a lot of the others that expire every month). It’s list price at time of writing is £44.99, I got mine for $39.96, but it’s been as low as £31.29 on offer recently.

Next

At the moment the router only fails over if the PPPoE connection goes down, it would be nice to try and detect if the PPPoE link stays up, but traffic stops flowing and change over. The challenge here is how to know to switch back since the L2TP tunnel takes priority. I’ll have to think about that one.

IKEA VINDRIKTNING PM2.5 Sensor

Having seen a tweet to a Hackaday article (/ht Andy Piper) about adding a ESP8266 to the new IKEA VINDRIKTNING air quality sensor.

IKEA Air Quality Sensor showing Green Light

The sensor is a little stand alone platform that measures the amount of PM 2.5 particles in the air and it has an array of coloured LEDs on the front to show a spectrum from green when the count is low and red when high.

Sören Beye opened one up and worked out that the micro controller that reads the sensor to control the leds does so over a uart serial connection and that the Tx/Rx lines were exposed via a a set of test pads along with 5v and Ground power. This makes it easy to attach a second micro controller to the Rx line to read the response when the sensor is polled.

Sören has written some code for an ESP8266 to decode that response and publish the result via MQTT.

Making the hardware modification is pretty simple

Wemos D1 Mini attached to sensor
  • Unscrew the case
  • Strip the ends on 3 short pieces of wire
  • Solder the 3 leads to the test pads labelled 5v, G and REST
  • Solder the 5V to 5V, G to G and REST to D2 (assuming using a Wemos D1 Mini)
  • Place the Wemos in the empty space above the sensor
  • Screw the case back together

The software is built using the Ardunio IDE and is easily flashed via the USB port. Once installed when the ESP8266 boots it will set up a WiFi Access Point to allow you to enter details for the local WiFi network and the address, username and password for a MQTT broker.

When connected the sensor publishes a couple of messages to allow auto configuration for people who use Home Assistant but it also publishes messages like this:

{
  "pm25":12,
  "wifi":{
    "ssid":"IoT Network",
    "ip":"192.168.1.58",
    "rssi":-60
  }
}

It includes the pm25 value and information about which network it’s connected to and it’s current IP address. I’m subscribing to this with Node-RED and using it to convert the numerical value, which has units of μg/m3 into a recognised scale (found on page 4).

let pm25 = msg.payload.pm25
if ( pm25 < 12 ) {
  msg.payload.string = "good"
} else if (pm25 >= 12 && pm25 < 36) {
  msg.payload.string = "moderate"
} else if (pm25 >= 36 && pm25 < 56) {
  msg.payload.string = "unhealthy for sensitive groups"
} else if (pm25 >= 56 && pm25 < 151 ) {
  msg.payload.string = "unhealthy"
} else if (pm25 >= 151 && pm25 < 251 ) {
  msg.payload.string = "very unhealthy"
} else if (pm25 >= 251 ) {
  msg.payload.string = "hazardous"
}
return msg;

I’m feeding this into a Google Smart Home Assistant Sensor device that has the SensorState trait, this takes the scale values as input, but you can also include the raw values as well.

msg.payload = {
  "params":{
    "currentSensorStateData":[
      {
        "name":"AirQuality",
        "currentSensorState":msg.payload.string
      },
      {
        "name":"PM2.5",
        "rawValue": msg.payload.pm25
      }
    ]
  }
}
return msg;

I will add the an Air Quality trait to the Node-RED Google Assistant Bridge shortly.

I’m also routing it to gauge in a Node-RED Dashboard setup.

Quick and Dirty Finger Daemon

I’ve been listening to more Brad & Will Made a Tech Pod and the current episode triggered a bunch of nostalgia about using finger to work out what my fellow CS students at university were up to. I won’t go into to too much detail about what Finger is as the podcast covers it all.

This podcast has triggered things like this in the past, like when I decided to make this blog (and Brad & Will’s podcast) available via Gopher.

On the podcast they had Ben Brown as a guest who had written his own Finger Daemon and linked it up to a site called Happy Net Box where users can update their plan file. Then anybody can access it using the finger command e.g. finger hardillb@happynetbox.com . The finger command is shipped by default on Windows, OSx and Linux so can be accessed from nearly anywhere.

I really liked the idea of resurrecting finger and as well as having a play with Happy Net Box I decided to see if I could run my own.

I started to look at what it would take to run a finger daemon on one of my Raspberry Pis, but while there are 2 packaged they don’t appear to run on current releases as they rely on init.d rather than Systemd.

Next up I thought I’d have a look at the protocol, which is documented in RFC1288. It is incredibly basic, you just listen on port 79 and read the username terminated with a new line & carriage return. This seamed to be simple enough to implement so I thought I’d give it a try in Go (and I needed something to do while all tonight’s TV was taken up with 22 men chasing a ball round a field).

The code is on Github here.

package main

import (
  "io"
  "os"
  "fmt"
  "net"
  "path"
  "time"
  "strings"
)

const (
  CONN_HOST = "0.0.0.0"
  CONN_PORT = "79"
  CONN_TYPE = "tcp"
)

func main () {
  l, err := net.Listen(CONN_TYPE, CONN_HOST+":"+CONN_PORT)
  if err != nil {
    fmt.Println("Error opening port: ", err.Error())
    os.Exit(1)
  }

  defer l.Close()
  for {
    conn, err := l.Accept()
    if err != nil {
      fmt.Println("Error accepting connection: ", err.Error())
      continue
    }
    go handleRequest(conn)
  }
}

func handleRequest(conn net.Conn) {
  defer conn.Close()
  currentTime := time.Now()
  buf := make([]byte, 1024)
  reqLen, err := conn.Read(buf)
  fmt.Println(currentTime.Format(time.RFC3339))
  if err != nil {
    fmt.Println("Error reading from: ", err.Error())
  } else {
    fmt.Println("Connection from: ", conn.RemoteAddr())
  }

  request := strings.TrimSpace(string(buf[:reqLen]))
  fmt.Println(request)

  parts := strings.Split(request, " ")
  wide := false
  user := parts[0]

  if parts[0] == "/W" && len(parts) == 2 {
    wide = true
    user = parts[1]
  } else if parts[0] == "/W" && len(parts) == 1 {
    conn.Write([]byte("\r\n"))
    return
  }

  if strings.Index(user, "@") != -1 {
    fmt.Println("remote")
    conn.Write([]byte("Forwarding not supported\r\n"))
  } else {
    if wide {
      //TODO
    } else {
      pwd, err := os.Getwd()
      filePath := path.Join(pwd, "plans", path.Base(user + ".plan"))
      filePath = path.Clean(filePath)
      fmt.Println(filePath)
      file, err := os.Open(filePath)
      if err != nil {
        //not found
        // io.Write([]byte("Not Found\r\n"))
      } else {
        defer file.Close()
        io.Copy(conn,file)
        conn.Write([]byte("\r\n"))
      }
    }
  }
}

Rather than deal with the nasty security problems with pulling .plan files out of peoples home directories it uses a directory called plans and loads files that match the pattern <username>.plan

I’ve also built it in a Docker container and mounted a local directory to allow me to edit and add new plan files.

You can test it with finger ben@hardill.me.uk

Setting up WireGuard IPv6

I’ve been having a quick play with setting up another VPN solution for getting an IPv6 address on my mobile devices this time using WireGuard.

WireGuard is a relatively new VPN tunnel implementation that has been written to be as stripped back as possible to keep the codebase as small as possible to help make it easier to audit.

Setup

A lot of the instructions for running WireGuard on RaspberryPi OS talk about adding debian testing repos or building the code from scratch, but it looks like recent updates have included the packages needed in the core repositories.

# apt-get install wireguard

I set up UDP port forwarding on my router for port 53145 and got my ISP to route another /64 IPv6 subnet to my line, both of these are forwarded on to the Raspberry Pi that is running that is also running my OpenVPN setup. This is useful as it’s already setup to do NAT for the 10.8.0.0/24 range I’m issuing to OpenVPN clients so having it do itfor the 10.9.0.0/24 range for WireGuard is easy enough.

WireGuard on Linux is implemented as a network device driver so can be configured on the command line with the ip command e.g.

# ip link add dev wg0 type wireguard
# ip address add dev wg0 10.9.0.1/24

Which brings the device up and sets the IP addresses but you still need to add the Private Key and remote address and Public Key which can be done with the wg command

# wg set wg0 listen-port 53145 private-key /path/to/private-key peer ABCDEF... allowed-ips 0.0.0.0/0 endpoint 209.202.254.14:8172

Or more easily it can read from a config file

# wg setconf wg0 myconfig.conf

Or the whole setup and configured with wg-quick

# wg-quick up /path/to/wg.conf

Server Config

[Interface]
Address = 10.9.0.1/24, 2001:8b0:2c1:xxx::1/64
ListenPort = 53145
PrivateKey = oP3TAHBctNVcnPTxxxxxxxxzNRLSF5CwII4s8gVAXg=

#nexus
[Peer]
PublicKey = 4XcNbctkGy0s73Dvxxxxxxxxx++rs5BAzCGjYmq21UM=
AllowedIPs = 10.9.0.2/32, 2001:8b0:2c1:xxx::2/128

The Server config includes:

  • Address is the local address on the VPN tunnel, here has both IPv4 and IPv6.
  • ListenPort is which port to listen for client connections on. WireGuard doesn’t have a assigned port.
  • PrivateKey to identify the host.
  • There can be multiple Peers which represent which clients can connect and the AllowedIPs is the IP addresses for each client.

Client Config

[Interface]
Address = 10.9.0.2/32, 2001:8b0:2c1:xxx::2/128
PrivateKey = UFIJGgtKsor6xxxxxxxxxxxbWeKmw+Bb5ODpyNblEA=
DNS = 8.8.8.8

[Peer]
PublicKey = jMB2oMu+YTKigGxxxxxxxxxxSYcTde/7HT+QlQoZFm0=
AllowedIPs = 0.0.0.0/0, ::0/0
Endpoint = hardill.me.uk:53145

The differences from the Server config are:

  • Interface has a DNS entry for the client to use while the tunnel is running.
  • Peer has an Endpoint which is the public address and port to connect to
  • AllowedIPs are which IPs to route over the tunnel, in this case it’s everything

Key Generation

Both ends of the connection need a PublicKey and a PrivateKey so they can mutually authenticate each other. These are generated with the wg command

# wg keygen > privateKey
# wg pubkey < privateKey > publicKey

Sharing Config

The WireGuard Android app that you can manually add all the details in the config file or it supports reading config files from QR codes. This makes it really easy to setup and removes the chance of getting a typo in the Keys and IP addresses.

You can generate QR codes from the config file as follows:

# qrencode -t png -o nexus.png < nexus.conf
# qrencode -t ansiutf8 < nexus.conf

The first generates a PNG file with the QR code, the second prints the code out as ASCII art.

Conclusion

It all looks to be working smoothly. I can see the advantages over OpenVPN being that you don’t need to worry about certificate maintenance and distribution.

I’ll give it a proper work out and see how it holds up running things like SIP connections along with general access to my home network.

As well as running it on the phone, I’ll set up a client config for my laptop to use when out and about. The only issues is that the Gnome Network Manager integration for WireGuard isn’t available in the standard repos for Ubuntu 20.04 so it needs to be started/stopped from the command line.