Home MicroK8s Cluster

I started to write about my home test environment for FlowForge a while ago, having just had to rebuild my K8s cluster due to a node failure I thought I should come back to this and document how I set it up (as much for next time as to share).

Cluster Nodes

I’m using the following nodes

Base OS

I’m working with Ubuntu 20.04 as this is the default OS of choice for MicroK8s and it’s available for both x86_64 and Arm8 for the Raspberry Pi 4.

Installing MicroK8s

$ sudo snap install microk8s --classic --channel=1.24

Once deployed on all 3 nodes, then we need to pick one of the nodes as the manager. In this case I’m using the Intel Celeron machine as the master and will run the following:

$ microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join

Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join --worker

If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join

And then on the other 2 nodes run the following

$ microk8s join --worker

You can verify the nodes are joined to the cluster with:

$ microk8s.kubectl get nodes
kube-two     Ready    <none>   137m   v1.24.0-2+f76e51e86eadea
kube-one     Ready    <none>   138m   v1.24.0-2+f76e51e86eadea
kube-three   Ready    <none>   140m   v1.24.0-2+59bbb3530b6769

Once the nodes are added to the cluster we need to enable a bunch of plugins, on the master node run:

$ microk8s enable dns:192.168.1.xx ingress helm helm3

dns:192.168.1.xx overrides the default of using Google’s DNS server to resolve names outside the cluster. This is important because I want it to point to my local DNS as I have set *.flowforge.loc and *.k8s.loc to point to the cluster IP addresses for Ingress.

Install kubectl and helm

By default Microk8s ships with a bunch of tools baked in, these include kubectl and helm that can be accessed as microk8s.kubectl and microk8s.helm respectively.


Instructions for installing standalone kubectl can be found here. Once installed you can generate the config by running the following on the master node:

$ microk8s config > ~/.kube/config

This can be copied to other machines that you want to be able to administrate the cluster.


Instructions for installing helm can be found standalone here.

This will make use of the same ~/.kube/config credentials file as kubectl.

NFS Persistent Storage

In order to have a consistent persistence storage pool across all 3 nodes I’m using a NFS share from my NAS. This is controlled using the nfs-subdir-external-provisioner. This creates a new directory on the NFS share for each volume created.

All the nodes need to have all the NFS client tools installed, this can be achieved with:

$ sudo apt-get install nfs-common

This is deployed using helm

$ helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
$ helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
    --set nfs.server= \
    --set nfs.path=/volume1/kube

To set this as the default StorageClass run the following:

kubectl patch storageclass standard -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'


That is enough for the basic Kubernetes cluster setup, there are some FlowForge specific bits that are needed (e.g. tagging nodes) but I’ll leave that for the FlowForge on Kubernetes install docs (which I have to finish writing before the next release).

FlowForge v0.1.0

So it’s finally time to talk a bit more about what I’ve been up to for the last few months since joining FlowForge Inc.

The FlowForge platform is a way to manage multiple instances of Node-RED at scale and to control user access to those instances.

The platform comes with 3 different backend drivers

  • LocalFS
  • Docker Compose
  • Kubernetes


This is the driver to use for evaluating the platform or as a home user that doesn’t want to install all the overhead that is required for the other 2 drivers. I starts Projects (Node-RED instances) as separate processes on the same machine and runs each one on a separate port. It keeps state in a local SQLite database.

Docker Compose

This version is a little more complicated, it uses the Docker runtime to start containers for the FlowForge runtime, a PostgreSQL database and Nginx reverse proxy. Each Project lives in it’s own container and is accessed by a unique hostname prepended to a supplied hostname. This can still run on a single machine (or multiple if Docker Swarm mode is used)


This is the whole shebang, similar to Docker Compose the FlowForge platform all runs in containers and the Projects end up in their own containers. But the Kubernetes platform provides more ways to manage the resources behind the containers and to scale to even bigger deployments.


Today we have released version 0.1.0 and made all the GitHub projects public.

The initial release is primarily focused on getting the core FlowForge platform out there for feedback and we’ve tried to make the LocalFS install experience as smooth as possible. There are example installers for the Docker and Kubernetes drivers but the documentation around these will improve very soon.

You can read the official release announcement here which has a link to the installer and also includes a walk through video.

Debugging Node-RED nodes with Visual Code

A recent Stack Overflow post had me looking at how to run Node-RED using Visual Code to debug custom nodes. Since I’d not tried Visual Code before (I tend to use Sublime Text 4 as my day to day editor) I thought I’d give it a go and see if I could get it working.

We will start with a really basic test node as an example. This just prints the content of msg.payload to the console for any message passing through.


module.exports = function(RED) {
    function test(n) {
        const node = this
        node.on('input', function(msg, send, done){
            send = send || function() { node.send.apply(node,arguments) }
    RED.nodes.registerType("test", test)


<script type="text/html" data-template-name="node-type">

<script type="text/html" data-help-name="node-type">

<script type="application/javascript">
        category: 'test',
        defaults: {},
        inputs: 1,
        outputs: 1,
        label: "test"


  "name": "test",
  "version": "1.0.0",
  "description": "Example node-red node",
  "keywords": [
  "node-red": {
    "nodes": {
      "test": "test.js"
  "author": "ben@example.com",
  "license": "Apache-2.0"

Setting up

All three files mentioned above are placed in a directory and then the following steps are followed:

  • In the Node-RED userDir (normally ~/.node-red on a Linux machine) run the following command to create a symlink in the node_modules directory. This will allow Node-RED to find and load the node.
    npm install /path/to/test/directory
  • Add the following section to the package.json file
  "scripts": {
    "debug": "node /usr/lib/node_modules/node-red/red.js"
  "node-red": {

Where usr/lib/node_modules/node-red/red.js is the output from readlink -f `which node-red`.

You can then add a breakpoint to the code

View of node's javascript code with break point set on line 7

And then start Node-RED by clicking on the Play button just above the scripts block.

view of node's package.json with play symbol and Debug above the scripts block

This will launch Node-RED and attach the debugger and stop when the breakpoint if hit. You can also enable the debugger to stop the application on exceptions, filtering on if they are caught or not.

This even works when using Visual Code’s remote capabilities for editing, running and debugging projects on remote machines. I’ve tested this running over SSH to a Raspberry Pi Zero 2 W (which is similar to the original StackOverflow question as they were trying to debug nodes working with the Pi’s GPIO system). The only change I had to make on the Pi was to increase the default swap file size from 100mb to 256mb as squeezing the Visual Code remote agent and Node-RED into 512mb RAM is a bit of a squeeze.

I might give Visual Code a go as my daily driver in the new year.

The Linear Clock Ticks Again

I’ve had a background project ticking over slowly in the background for a number of years.

Last year I designed and had built a number of PCBs to be used as HATs for a Raspberry Pi Zero. They included a RTC and a terminal block to attach the LED strip.

I did say that I would write another post when the boards where delivered and I had assembled the first prototype. Unfortunately I had made a small, but critical mistake when designing the boards, I slightly messed up the package package size for the RTC so it wasn’t possible to get assemble the boards correctly. I didn’t get round to re-doing the PCB layout with the correct sized parts so the whole thing just sat for a while.

In the meantime the Raspberry Pi Foundation went and released a new product, the Raspberry Pi Pico, which is based on the RP2040 chip. As well as the Pico they are also making the RP2040 chip available to other folk to include it directly in their own projects.

Pimoroni have created a number of different boards but their latest is the Plasma 2040 which is specifically designed to drive LED strips.



  • Solder the RTC on to the breakout section of the Plasma 2040, the terminals are labelled so just make sure you match up the pins, I used the headers that came with the RTC and arranged it so the breakout was over the top of the Plasma2040
  • Loosen the screw terminals for the connections marked 5V, DA and -. Insert the Red wire of the adapter in the 5V, Green wire in DA and White wire in –
  • Clip the LED strip to the end of the adapter.
Plasma 2040


When you first attach the Plasma2040 to your computer it will show up as a USB flash drive. This is so you can install the runtime. In this case we’ll be using the Pimoroni Micropython build that comes with support for the board. You can grab a version from the release page on GitHub here. Once downloaded copy it into the root of the drive. When the copy has finished the board will reboot and be ready to run Python code.

You can use the Thonny IDE to both write and push code to the device. You will need at least version 3.3.3 to support the Plasma2040.

The fist version of the code was as follows:

import plasma
from plasma import plasma2040
from pimoroni import RGBLED, Button
import time

LOW = 32
MED = 64
HIGH = 128

button_brightness = Button(plasma2040.BUTTON_A)

led = RGBLED(plasma2040.LED_R, plasma2040.LED_G, plasma2040.LED_B)
led.set_rgb(0, 0, 0)
led_strip = plasma.WS2812(NUM_LEDS, 0, 0, plasma2040.DAT)


while True:
    RED = [0]*NUM_LEDS
    GREEN = [0]*NUM_LEDS
    BLUE = [0]*NUM_LEDS
    t = time.localtime()

    hour = (t[3] % 12) * 5
    #set the LEDS
    for i in range (NUM_LEDS):
        led_strip.set_rgb(i, RED[i], GREEN[i], BLUE[i])
    #change brightness
    if button_brightness.read():

This works well when triggered from Thonny as it syncs the laptop’s time to the RP2040 each time it connects. But when the clock is powered by a USB power supply or a battery, the clock starts at 00:00:01 Jan 1st 2021 and has no way to be updated to match now.

This is why we need the RTC module, it keeps track of the time while the clock is powered down.

It also has a way to change the brightness, by pressing the A button it will cycle through 3 different brightness levels.

Setting the RTC Time

With a little bit of playing I worked out how to sync the RTC to the current time in the Thonny console

>>> from pimoroni_i2c import PimoroniI2C
>>> from breakout_rtc import BreakoutRTC
>>> import time
>>> PINS_PLASMA = {"sda": 20, "scl": 21}
>>> i2c = PimoroniI2C(**PINS_PLASMA)
>>> rtc = BreakoutRTC(i2c)
>>> rtc.set_unix(time.time())
>>> rtc.set_time(54,18,17,6,18,9,2021)
>>> rtc.update_time()
>>> print(rtc.string_time())
>>> rtc.set_backup_switchover_mode(3)

The most important line is the last one, which enables the battery backup for the RTC so it remembers the time you just set.

I was going to use the rtc.set_unix() function and pass in time.time() but it appears that the unix timestamp is maintained independently of the “Real” time on the RTC.

The set_time() function takes values in the order

  • seconds (0-60)
  • minutes (0-60)
  • hours (0-23)
  • day of the week (1-7 -> mon-sun)
  • day of month (1-31)
  • monthe (1-12)
  • year (2000-2099)

With the RTC set correctly a small update to the code to read from the RTC rather than from the time object and we are good to go.

import plasma
from plasma import plasma2040
from pimoroni import RGBLED, Button
from pimoroni_i2c import PimoroniI2C
from breakout_rtc import BreakoutRTC
import time

PINS_PLASMA = {"sda": 20, "scl": 21}

i2c = PimoroniI2C(**PINS_PLASMA)
rtc = BreakoutRTC(i2c)

if rtc.is_12_hour():

if rtc.update_time():

LOW = 32
MED = 64
HIGH = 128

button_brightness = Button(plasma2040.BUTTON_A)

led = RGBLED(plasma2040.LED_R, plasma2040.LED_G, plasma2040.LED_B)
led.set_rgb(0, 0, 0)
led_strip = plasma.WS2812(NUM_LEDS, 0, 0, plasma2040.DAT)



while True:
    RED = [0]*NUM_LEDS
    GREEN = [0]*NUM_LEDS
    BLUE = [0]*NUM_LEDS
    t = time.localtime()

    if rtc.read_periodic_update_interrupt_flag():
        hour = (rtc.get_hours() % 12) * 5
        GREEN[rtc.get_minutes()] = BRIGHTNESS[BRIGHTNESS_LEVEL]
        BLUE[rtc.get_seconds()] = BRIGHTNESS[BRIGHTNESS_LEVEL]

        for i in range (NUM_LEDS):
            led_strip.set_rgb(i, RED[i], GREEN[i], BLUE[i])
        if button_brightness.read():
            BRIGHTNESS_LEVEL += 1
            BRIGHTNESS_LEVEL %= 3
2021 Edition

Next Steps

There are a few things that need doing next. The first is to build a case for the clock, I’m thinking about something made up of layers of thin plywood with a channel for the LED strip and maybe a layer of smoked/mat acrylic to act as a diffuser.

The second part is to work out a way to work with DST, Micropython doesn’t support timezones as the database needed to keep track of all the different timezones takes up a huge amount of space. I could hard code in the dates for my location, but I’ll probably just make use of the B button to toggle an hours difference on/off.

Optionally I might add another 31 LED strip (probably at 30/meter) to be used as a calendar showing the current month with markers for weekends and the current day.

Another option is to use 4 of these to build a 60 LED ring for something a little more conventionally shaped.

And the final extra hack is to daisy chain the Light level sensor (e.g. one of these) on top of the RTC and dynamically adjust the brightness based on ambient light levels.

I’ll also probably keep tinkering with the Raspberry Pi Zero W version as that will allow oAuth to link to things like Google Calendar to show meetings in the clock view and add Holidays to the Calendar view. It will also have access to the full timezone database and NTP for time syncing over the network.


Having seen a tweet to a Hackaday article (/ht Andy Piper) about adding a ESP8266 to the new IKEA VINDRIKTNING air quality sensor.

IKEA Air Quality Sensor showing Green Light

The sensor is a little stand alone platform that measures the amount of PM 2.5 particles in the air and it has an array of coloured LEDs on the front to show a spectrum from green when the count is low and red when high.

Sören Beye opened one up and worked out that the micro controller that reads the sensor to control the leds does so over a uart serial connection and that the Tx/Rx lines were exposed via a a set of test pads along with 5v and Ground power. This makes it easy to attach a second micro controller to the Rx line to read the response when the sensor is polled.

Sören has written some code for an ESP8266 to decode that response and publish the result via MQTT.

Making the hardware modification is pretty simple

Wemos D1 Mini attached to sensor
  • Unscrew the case
  • Strip the ends on 3 short pieces of wire
  • Solder the 3 leads to the test pads labelled 5v, G and REST
  • Solder the 5V to 5V, G to G and REST to D2 (assuming using a Wemos D1 Mini)
  • Place the Wemos in the empty space above the sensor
  • Screw the case back together

The software is built using the Ardunio IDE and is easily flashed via the USB port. Once installed when the ESP8266 boots it will set up a WiFi Access Point to allow you to enter details for the local WiFi network and the address, username and password for a MQTT broker.

When connected the sensor publishes a couple of messages to allow auto configuration for people who use Home Assistant but it also publishes messages like this:

    "ssid":"IoT Network",

It includes the pm25 value and information about which network it’s connected to and it’s current IP address. I’m subscribing to this with Node-RED and using it to convert the numerical value, which has units of μg/m3 into a recognised scale (found on page 4).

let pm25 = msg.payload.pm25
if ( pm25 < 12 ) {
  msg.payload.string = "good"
} else if (pm25 >= 12 && pm25 < 36) {
  msg.payload.string = "moderate"
} else if (pm25 >= 36 && pm25 < 56) {
  msg.payload.string = "unhealthy for sensitive groups"
} else if (pm25 >= 56 && pm25 < 151 ) {
  msg.payload.string = "unhealthy"
} else if (pm25 >= 151 && pm25 < 251 ) {
  msg.payload.string = "very unhealthy"
} else if (pm25 >= 251 ) {
  msg.payload.string = "hazardous"
return msg;

I’m feeding this into a Google Smart Home Assistant Sensor device that has the SensorState trait, this takes the scale values as input, but you can also include the raw values as well.

msg.payload = {
        "rawValue": msg.payload.pm25
return msg;

I will add the an Air Quality trait to the Node-RED Google Assistant Bridge shortly.

I’m also routing it to gauge in a Node-RED Dashboard setup.

Working with multiple AWS EKS instances

I’ve recently been working on a project that uses AWS EKS managed Kubernetes Service.

For various reasons too complicated to go into here we’ve ended up with multiple clusters owned by different AWS Accounts so flipping back and forth between them has been a little trickier than normal.

Here are my notes on how to manage the AWS credentials and the kubectl config to access each cluster.


First task is to authorise the AWS CLI to act as the user in question. We do this by creating a user with the right permissions in the IAM console and then export the Access key ID and Secret access key values usually as a CSV file. We then take these values and add them to the ~/.aws/credentials file.

aws_access_key_id = AKXXXXXXXXXXXXXXXXXX
aws_secret_access_key = xyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxy

aws_access_key_id = AKYYYYYYYYYYYYYYYYYY
aws_secret_access_key = abababababababababababababababababababab

aws_access_key_id = AKZZZZZZZZZZZZZZZZZZ
aws_secret_access_key = nmnmnmnmnmnmnmnmnmnmnmnmnmnmnmnmnmnmnmnm

We can pick which set of credential the AWS CLI uses by adding the --profile option to the command line.

$ aws --profile dev sts get-caller-identity
    "Account": "111111111111",
    "Arn": "arn:aws:iam::111111111111:user/dev"

Instead of using the --profile option you can also set the AWS_PROFILE environment variable. Details of all the ways to switch profiles are in the docs here.

$ export AWS_PROFILE=test
$ aws sts get-caller-identity
    "Account": "222222222222",
    "Arn": "arn:aws:iam::222222222222:user/test"

Now we can flip easily between different AWS accounts we can export the EKS credential with

$ export AWS_PROFILE=prod
$ aws eks update-kubeconfig --name foo-bar --region us-east-1
Updated context arn:aws:eks:us-east-1:333333333333:cluster/foo-bar in /home/user/.kube/config

The user that created the cluster should also follow these instructions to make sure the new account is added to the cluster’s internal ACL.


If we run the previous command with each profile it will add the connection information for all 3 clusters to the ~/.kube/config file. We can list them with the following command

$ kubectl config get-contexts
CURRENT   NAME                                                  CLUSTER                                               AUTHINFO                                              NAMESPACE
*         arn:aws:eks:us-east-1:111111111111:cluster/foo-bar   arn:aws:eks:us-east-1:111111111111:cluster/foo-bar   arn:aws:eks:us-east-1:111111111111:cluster/foo-bar   
          arn:aws:eks:us-east-1:222222222222:cluster/foo-bar   arn:aws:eks:us-east-1:222222222222:cluster/foo-bar   arn:aws:eks:us-east-1:222222222222:cluster/foo-bar   
          arn:aws:eks:us-east-1:333333333333:cluster/foo-bar   arn:aws:eks:us-east-1:333333333333:cluster/foo-bar   arn:aws:eks:us-east-1:333333333333:cluster/foo-bar 

The star is next to the currently active context, we can change the active context with this command

$ kubectl config set-context arn:aws:eks:us-east-1:222222222222:cluster/foo-bar
Switched to context "arn:aws:eks:us-east-1:222222222222:cluster/foo-bar".

Putting it all together

To automate all this I’ve put together a collection of script that look like this

export AWS_PROFILE=prod
aws eks update-kubeconfig --name foo-bar --region us-east-1
kubectl config set-context arn:aws:eks:us-east-1:222222222222:cluster/foo-bar

I then use the shell source ./setup-prod command (or it’s shortcut . ./setup-prod) , this is instead of adding the shebang to the top and running it as a normal script. This is because when environment variables are set in scripts they go out of scope. Leaving the AWS_PROFILE variable in scope means that the AWS CLI will continue to use the correct account settings when it’s used later while working on this cluster.

Joining FlowForge Inc.

FlowForge Logo

Today is my first day working for FlowForge Inc. I’ll be employee number 2 and joining Nick O’Leary working on all things based around Node-RED and continuing to contribute to the core Open Source project.

We should be building on some of the things I’ve been playing with recently.

Hopefully I’ll be able to share some of the things I’ll be working on soon, but in the mean time here is the short post that Nick wrote when he announced FlowForge a few weeks ago and a post welcoming me to the team

To go with this announcement Hardill Technologies Ltd will be going dormant. It’s been an good 3 months and I’ve built something interesting for my client which I hope to see it go live soon.

Google Assistant Sensors

Having built my 2 different LoRA connected temperature/humidity sensors I was looking for something other than the Graphana instance that shows the trends.

Being able to ask Google Assistant the temperature in a room seemed like a good idea and an excuse to add the relatively new Sensor device type my Google Assistant Bridge for Node-RED.

I’m exposing 2 options for the Sensor to start with, Temperature and Humidity. I might look at adding Air Quality later.

Once the virtual device is setup, you can feed data in the Google Home Graph using a flow similar to the following

The join node is set to combine the 2 incoming MQTT messages into a single object based on their topics. The function node then builds the right payload to pass to the Google Home output node and finally it feeds it through an RBE node just to make sure we only send updates when the data changes.

msg.payload = {
  params: {
    temperatureAmbientCelsius: msg.payload["bedroom/temp"],
    humidityAmbientPercent: Math.round(msg.payload["bedroom/humidity"])

Google Assistant Camera Feeds

As mentioned in a previous post I’ve been playing with Streaming Camera feeds to my Chromecast.

The next step is to enabling accessing these feeds via the Google Assistant. To do this I’m extending my Node-RED Google Assistant Service.

You should now be able to add a device with the type Camera and a CameraStream trait. You can then ask the Google Assistant to “OK Google, show me View Camera on the Livingroom TV”

This will create an input message in Node-RED that looks like:

  "topic": "",
  "name": "View Camera",
  "payload": {
    "command": "action.devices.commands.GetCameraStream",
    "params": {
      "StreamToChromecast": true,
      "SupportedStreamProtocols": [
      "online": true

The important part is mainly the SupportedStreamProtocols which shows the types of video stream the display device supports. In this case because the target is a ChromeCast it shows the full list.

Since we need to reply with a URL pointing to the stream the Node-RED input node can not be set to Auto Acknowledge and must be wired to a Response node.

The function node updates the msg.payload.params with the required details. In this case

msg.payload.params = {
    cameraStreamAccessUrl: "",
    cameraStreamProtocol: "hls"
return msg;

It needs to include the cameraStreamAccessUrl which points to the video stream and the cameraStreamProtocol which identifies which of the requested protocols the stream uses.

This works well when the cameras and the Chromecast are on the same network, but if you want to access remote cameras then you will want to make sure that they are secured to prevent them being scanned by a IoT search engine like Shodan and open to the world.

Viewing Node-RED Credentials

A question popped up on the Node-RED Slack yesterday asking how to recover an entry from the credentials file.


The credentials file can normally be found in the Node-RED userDir, which defaults to ~/.node-red on Unix like platforms (and is logged near the start of the output when Node-RED starts). The file has the same name as the flow file with _cred appended before the .json e.g. the flows_localhost.json will have a coresponding flows_localhost_creds.json

The content of the file will look something a little like this:


This isn’t much use on it’s own as the contents are encrypted to make it harder for people to just copy the file and have access to all the stored passwords and access tokens.

The secret that is used to encrypt/decrypt this file can be found in one of 2 locations:

  • In the settings.js file in the credentialsSecret field. The user can set this if they want to use a fixed known value.
  • In the .config.json (or .config.runtime.json in later releases) in the __credentialSecret field. This secret is the one automatically generated if the user has not specifically set one in the settings.js file.


In order to make use of thex

const crypto = require('crypto');

var encryptionAlgorithm = "aes-256-ctr";

function decryptCreds(key, cipher) {
  var flows = cipher["$"];
  var initVector = Buffer.from(flows.substring(0, 32),'hex');
  flows = flows.substring(32);
  var decipher = crypto.createDecipheriv(encryptionAlgorithm, key, initVector);
  var decrypted = decipher.update(flows, 'base64', 'utf8') + decipher.final('utf8');
  return JSON.parse(decrypted);

var creds = require("./" + process.argv[2])
var secret = process.argv[3]

var key = crypto.createHash('sha256').update(secret).digest();

console.log(decryptCreds(key, creds))

If you place this is a file called show-creds.js and place it in the Node-RED userDir you can run it as follows:

$ node show-creds creds.json [secret]

where [secret] is the value stored in credentialsSecret or _credentialsSecret from earlier. This will then print out the decrypted JSON object holding all the passwords/tokens from the file.