Building a Kubernetes Test Environment

Over the last couple of weekends I’ve been noodling around with my home lab set up to build a full local environment to test out FlowForge with both the Kubernetes Container and Docker Drivers.

The other reason to put all this together is to help to work the right way to put together a proper CI pipeline to build, automatically test and deploy to our staging environment.

Components

NPM Registry

This is somewhere to push the various FlowForge NodeJS modules so they can then be installed while building the container images for the FlowForge App and the Project Container Stacks.

This is a private registry so that I can push pre-release builds without them slipping out in to the public domain, but also so I can delete releases and reuse version numbers which is not allowed on the public NPM registry.

I’m using the Verdaccio registry as I’ve used this in the past to host custom Node-RED nodes (which it will probably end up doing again in this set up as things move forwards). This runs as Docker container and I use my Nginx instance to reverse proxy for it.

As well as hosting my private builds it can proxy for the public npmjs.org regisry which speeds up local builds.

Docker Container Registry

This is somewhere to push the Docker containers that represent both the FlowForge app it’s self and the containers that represent the Project Stacks.

Docker ship a container image available that will run a registry.

As well as the registry I’m also running second container with this web UI project to help keep track of what I’ve pushed to the registry and also allows me to delete tags which is useful when testing

Again my internet facing Nginx instance is proxying for both of these (on the same virtual host since their routes do not clash and it makes CORS easier since the UI is all browser side JavaScript)

Helm Chart Repository

This isn’t really needed, as you can generate all the required files with the helm command and host the results on any Web server, but this lets me test the whole stack end to end.

I’m using a package called ChartMuseum which will automatically generate index.yaml manifest file when charts are uploaded via it’s simple UI.

Nginx Proxy

All of the previous components have been stood up as virtual hosts on my public Nginx instance so that they can get HTTPS certificates from LetsEncrypt. This is makes things a lot easier because both Docker and Kubernetes basically require the container registry be secure by default.

While it is possible to add exceptions for specific registries, these days it’s just easier to do it “properly” up front.

MicroK8s Cluster

And finally I need a Kubernetes cluster to run all this on. In this case I have a 3 node cluster made up of

  • 2 Raspberry Pi 4s with 8gb of RAM each
  • 1 Intel Celeron based mini PC with 8gb of RAM

All 3 of these are running 64bit Ubuntu 20.04 and MicroK8s. The Intel machine is needed at the moment because the de facto standard PostrgresSQL Helm Chat only have amd64 based containers at the moment so won’t run on the Raspberry Pi based nodes.

The cluster uses the NFS Persistent Volume provisioner to store volumes on my local NAS so they are available to all the nodes.

Usage

I’ll write some more detailed posts about how I’ve configured each of these components and then how I’m using them.

As well as testing the full Helm chart builds, I can also use this to run the FlowForge app locally and have the Kubernetes Container driver running locally on my development machine and have it create Projects in the Kubernetes cluster.

Advanced Multi Tenant Node-RED topics

Across the last 6 posts I’ve walked through deploying a Multi Tenant Node-RED service. In this post I’m going to talk about how you go about customising the install to make it more specific to your needs.

Custom Node Catalogue

As I mentioned in the post about creating the custom docker container you can use the nodeExcludes in settings.js to disable nodes and if needed the package.json can be edited to remove any core nodes that you want.

But you might have a collection of nodes that are specific to your deployment that are not of use outside your environment and you may not want to publish to npmjs.org. In extreme cases you may even want to remove all the core nodes and only allow users to use your own set of nodes.

Node-RED downloads the list of nodes available to install from catalogue.nodered.org and allows you to add either additional URLs to pull in a list of extra nodes or replace the URL with a custom list. The documentation for this can be found here. The list is kept in the settings.js under the editorTheme entry.

...
editorTheme: {
  palette: {
    catalogues: [
      "http://catalogue.nodered.org/catalogue.json", //default catalogue
      'http://manager.example.com/catalogue.json'
    ]
  }
},
...

The URL should point to a JSON file that has the following format

{
  "name":"Ben's custom catalogue",
  "updated_at": "2016-08-05T18:37:50.673Z",
  "modules": [
    {
      "id": "@ben/ben-red-random",
      "version": "1.3.0",
      "description": "A node-red node that generates random numbers",
      "keywords": [
        "node-red",
        "random"
      ],
      "updated_at": "2016-08-05T18:37:50.673Z",
      "url": "http://flows.example.com/node/ben-red-random"
    },
    ...
  ]
}

There is a small script called build-catalogue.js in the manager app that will generate a catalogue.json file from a given npm repository.

Now there is a custom list of nodes we need to be able to load them with the npm command. There are a few options

  • Publish the node to npmjs.org but without the node-red keyword so they don’t end up being index on flows.nodered.org (and not manually submit them as you now need to)
  • Publish the node to a private npm repository that also acts as a pass through proxy to npmjs.org
  • Publish the node to a private npm repository with a scope and configure npm to use a different repository for a given scope.

The first option doesn’t need anything special settings up, you just add the nodes you want to the catalogue. The second two options need a private npm package repository. For the second it needs to act as a pass through so all the dependencies can also be loaded which is what makes the third option probably best.

I’ve been playing with running verdaccio on the same docker infrastructure as everything else and set npm to map the @private scope http://registry:4873 in the custom docker container.

...
RUN npm config set @private:registry http://registry:4873

Verdaccio can also proxy to npmjs.org if needed. (Like the nginx-proxy container I had to rebuild the verdaccio container to get it to run on my Pi4 since it only ships a AMD64 version)

I’m hosting my catalogue.json from the same Express application as is used to provision new Node-RED instances.

Screen shot of nodes listed on Verdaccio

Or if you want to prevent users being able to install/remove nodes then you can add:

...
editorTheme: {
  palette: {
    editable: false
  }
},
...

Skinning Node-RED

The last thing on my list is to give the Node-RED instances a custom look and feel.

The basics like the page title, header image, favicon and logon screen graphic can all be set directly from the settings.js with the option to also link to a custom CSS style sheet so the colour scheme and shape/size of elements be changed as well. The design document can be found here.

editorTheme: {
  projects: {
    // To enable the Projects feature, set this value to true
    enabled: false
  },
  page: {
    title: "Ben-RED"
  },
  header: {
    title: "Ben-RED"
  },
  palette: {
    catalogues: [
      'https://catalogue.nodered.org/catalogue.json',
      'http://manager.example.com/catalogue.json'
    ]
  }
},