Managing Multi Tenant Node-RED Instances

Over the last series of posts I’ve outlined how to build all the components that are needed for a Multi Tenant Node-RED service. What’s missing is a way to automate the spinning up of new instances.

One option would be to do this with Node-RED it’s self, you can drive docker using the node-red-contrib-dockerode, but for this I’ve decided to create a dedicated application.

I’ve written a small Express app that uses dockerode directly and also will populate the Users collection in the MongoDB with the admin password for the editor and then spin up a new instance.

docker.createContainer({
  Image: "custom-node-red",
  name: req.body.appname,
  Env: [
    "VIRTUAL_HOST=" + req.body.appname + "." + settings.rootDomain,
    "APP_NAME="+ req.body.appname,
    "MONGO_URL=mongodb://mongodb/nodered"
  ],
  AttachStdin: false,
  AttachStdout: false,
  AttachStderr: false,
  HostConfig: {
    Links: [
      "mongodb:mongodb"
    ]
  }
})
.then(container => {
  console.log("created");
  cont = container;
  return container.start()
})
.then(() => {
  res.status(201).send({started: true, url: "http://" + req.body.appname + "." + settings.rootDomain});
})

It’s pretty basic but it does just enough to get started. It is being exposed using the same nginx-proxy that exposes the Node-RED instances, so the management interface is available on the manger.docker-pi.local domain. If it was being deployed in a production environment it should probably not be internet facing and should have some basic access control to prevent anybody from being able to stand up a new Node-RED instance.

When the app has completed creating a new instance a link to that instance is displayed.

You can also Start/Stop/Remove the instance as well as streaming the logs via a websocket.

Thanks to Dave Conway-Jones (@ceejay) for help with making my very utilitarian UI look a lot better.

The code for the management app is on github here

Proxying for Mulitple Node-RED instances

So far we have worked out how to set up Node-RED to store flows in a database, use authentication to prevent unauthorised access and how to start multiple containerised instances under Docker.

In this post I will cover how to expose those multiple instances so their users can access them.

The easiest way to do this is to stick something like Nginx or Traefik in front of the docker containers and have it act as a reverse proxy. There are two ways we can set this up

  • Virtual Host based proxying – where each instance has it’s own hostname e.g. http://r1.example.com
  • Path based proxying – where each instance has a root path on the same hostname e.g. http://example.com/r1

In this case I’m going to use the first option of virtual hosts as currently Node-RED uses browser local storage to hold the Admin API security token and this is scoped to the host the editor is loaded from which means that you can only access one instance at a time if you use path based proxying. This is on the Node-RED backlog to get fixed soon.

To do that I’m going to use the nginx-proxy container. This container runs in the same Docker environment as the Node-RED instances and monitors all Docker to watch for containers starting/stopping. When it sees a new container start up it automatically create the right entry in the nginx configuration files and triggers it to reload the config.

To make this work I needed to add an extra environment variable to the command used to start the Node-RED containers

$ docker run -d --rm -e VIRTUAL_HOST=r1.example.com -e MONGO_URL=mongodb://mongodb/nodered -e APP_NAME=r1 --name r1 --link mongodb custom-node-red

I added the VIRTUAL_HOST environment variable which contains the hostname to use for this container. Th is means I can access this specific instance of Node-RED on http://r1.docker.local.

Traefik can be run in a similar way using labels instead of environment variables.

To make is all work smoothly I’ve added a wildcard domain entry to my local DNS that maps anything that matches *.example.com to the docker-pi.local machine that the container are running on.

Security

If I was going to run this exposed to the internet I’d probably want to enable HTTPS. To do this there are 2 options again

  • Use a separate certificate for each Virtual Host
  • Use a wildcard certificate that matches the wildcard DNS entry

I would probably go with the second option here as it is just one certificate you have to manage and even LetsEncrypt will issue a wildcard domain these days if you have access to the DNS.

For the first option there is a companion docker container for nginx-proxy that will use LetsEncrypt to issue certificates for each Virtual Host as they are started. It’s called letsencrypt-nginx-proxy-companion you can read how to use it in the nginx-proxy README.md

Limitations

Exposing Node-RED via a HTTP proxy does have one drawback. This approach means that only HTTP requests can directly reach the instances.

While this helps to offer a little more security, it also means that you will not be able to use the TCP-in or UDP-in nodes in server mode, that would allow arbitrary network connections into the instance. You will still be able to connect out from the instances to hosts as Docker provides NAT routing from containers to the outside world.

Sidebar

I’m testing all this on a Raspberry Pi4 running the beta of 64bit Raspberry Pi OS. I need this to get the official MongoDB container to work as they only formally support 64bit. As a result of this I had to modify and rebuild the nginx-proxy container because it only ships with support for AMD64 architectures. I had to build a ARM64 version of the forego and docker-gen packages and manually copy these into the container.

There is an outstanding pull-request open against the project to use a multistage-build which will build target specific binaries of forego and docker-gen which will fix this.

Custom Node-RED container for a Multi Tenant Environment

In this post I’ll talk about building a custom Node-RED Docker container that adds the storage and authentication plugins I build earlier, along with disabling a couple of the core nodes that don’t make much sense on a platform where the local disk isn’t really usable.

Settings.js

The same modifications to the settings.js from the last two posts need to be added to enabled the storage and authentication modules. The MongoDB URI and the AppName are populated with environment variables that we can pass in different values when starting the container.

We will also add the nodeExcludes entry which removes the nodes that interact with files on the local file system as we don’t want users saving things into the container which will get lost if we have to restart. It also removes the exec node since we don’t want users running arbitrary commands inside the container.

Setting the autoInstallModules to true means that if the container gets restarted then any extra nodes the user has installed with the Palette Manager will get reinstalled.

...
storageModule: require('node-red-contrib-storage-mongodb'),
mongodbSettings: {
  mongoURI: process.env["MONGO_URL"],
  appname: process.env["APP_NAME"]
},
adminAuth: require('node-red-contrib-auth-mongodb').setup({
  mongoURI: process.env["MONGO_URL"],
  appname: process.env["APP_NAME"]
}),
nodesExcludes:['90-exec.js','28-tail.js','10-file.js','23-watch.js'],
autoInstallModules: true,
...

Dockerfile

This is a pretty easy extension of the default Node-RED container. We add the modified settings.js from above to the /data directory in the container (and set the permissions on it) and then install the plugins. Everything else stays the same.

FROM nodered/node-red

COPY settings.js /data/
USER root
RUN chown -R node-red:root /data
USER node-red
RUN npm install --no-fund --no-update-notifier --save node-red-contrib-auth-mongodb node-red-contrib-storage-mongodb

We can build this with the following command

$ docker build -t custom-node-red .

It is important to node here I’ve not given a version tag in the FROM tag, so every time the container is rebuilt it will pull the very latest shipped version. This might not be what you want for a deployed environment where making sure all users are on the same version is probably a good idea from a support point of view. It may also be useful to use the -minimal tag postfix to use the version of the container based on alpine to reduce the size.

Starting an instance

You can start a new instance with the following command

$ docker run -d -rm -p 1880:1880 -e MONGO_URL=mongodb://mongodb/nodered -e APP_NAME=r1 --name r1 custom-node-red

In this example I’ve mapped the containers port 1880 to the host, but that will only work for a single container or every container will need to be on a different port on the host, for a Multi Tenant solution we need something different and I’ll cover that in the next post.

Node-RED Authentication Plugin

Next in the Multi Tenant Node-RED series is authentication.

If you are going to run a multi user environment one of the key features will be identifying which users are allowed to do what and where. We need to only allow users to access their specific instances of Node-RED.

Node-RED provides a couple of options, the simplest is just to include the username/password/permissions details directly in the settings.js but this doesn’t allow for dynamic updates like adding/removing users or changing passwords.

// Securing Node-RED
// -----------------
// To password protect the Node-RED editor and admin API, the following
// property can be used. See http://nodered.org/docs/security.html for details.
adminAuth: {
    type: "credentials",
    users: [{
        username: "admin",
        password: "$2a$08$zZWtXTja0fB1pzD4sHCMyOCMYz2Z6dNbM6tl8sJogENOMcxWV9DN.",
        permissions: "*"
    }]
},

The documentation also explains how to use PassportJS strategies to authenticate against oAuth providers, meaning you can do things like have users sign in with their Twitter credentials or use an existing Single Sign On solution if you want.

And finally the documentation covers how to implement your own authentication plugin, which is what I’m going to cover in this post.

In the past I have built a version of this type of plugin that uses LDAP but in this case I’ll be using MongoDB. I’ll be using the same database that I used in the last post about building a storage plugin. I’m also going to use Mongoose to wrap the collections and I’ll be using the passport-local-mongoose plugin to handle the password hashing.

const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const passportLocalMongoose = require('passport-local-mongoose');

const Users = new Schema({
  appname: String,
  username: String,
  email: String,
  permissions: { type: String, default: '*' },
});

var options = {
  usernameUnique: true,
  saltlen: 12,
  keylen: 24,
  iterations: 901,
  encoding: 'base64'
};

Users.plugin(passportLocalMongoose,options);
Users.set('toObject', {getters: true});
Users.set('toJSON', {getters: true});

module.exports = mongoose.model('Users',Users);

We need the username and permissions fields for Node-RED, the appname is the same as in the storage plugin to allow us to keep the details for multiple Node-RED instances in the same collection. I added the email field as a way to contact the user if we need do something like a password reset. You can see that there is no password field, this is all handled by the passport-local-mongoose code, it injects salt and hash fields into the schema and adds methods like authenticate(password) to the returned objects that will check a supplied password against the stored hash.

Required API

There are 3 function that need to be implemented for Node-RED

users(username)

This function just checks if a given user exists for this instance

  users: function(username) {
    return new Promise(function (resolve, reject){
      Users.findOne({appname: appname, username: username}, {username: 1, permissions: 1})
      .then(user => {
        resolve({username:user.username, permissions:user.permissions);
      })
      .catch(err => {
        reject(err)
      })
    });
  },

authenticate(username, password)

This does the actual checking of the supplied password against the database. It looks up the user with the username and appname and then has passport-local-mongo check it against the hash in the database.

authenticate: function(username, password) {
      return new Promise(function(resolve, reject){
        Users.findOne({appname: appname, username})
        .then((user) => {
          user.authenticate(password, function(e,u,pe){
            if (u) {
              resolve({username: u.username, permissions: u.permissions)
            } else {
              resolve(null);
            }
          })
        })
        .catch(err => {
          reject(err)
        })
      }) 
  },

default()

In this case we don’t want a default user so we just return a null object.

default: function() {
    return Promise.resolve(null);
  }

Extra functions

setup(settings)

Since authentication plugins do not get the whole setting object passed in like the storage plugins we need include a setup() function to allow details about the MongoDB to be passed in.

type: "credentials",
setup: function(settings) {
  if (settings && settings.mongoURI) {
    appname = settings.appname;
    mongoose.connect(settings.mongoURI, mongoose_options)
    .catch(err => {
      throw err;
    });
  }
  return this;
},

Using the plugin

This is again similar to the storage plugin where an entry is made in the settings.js file. The difference is that this time the settings object isn’t explicitly passed to the plugin so we need to include the call to setup in the entry.

...
adminAuth: require('node-red-contrib-auth-mongodb').setup({
   mongoURI: "mongodb://localhost/nodered",
   appname: "r1"
})

How to add users to the database will be covered in a later post about managing the creation of new instances.

Source code

You can find the code here and it’s on npmjs here

Node-RED Storage Plugin

As part of my series of posts about the components needed to build a Multi Tenant Node-RED system in this post I’ll talk about writing a plugin to store the users flow in the database rather than on disk.

There are a number of existing Storage plugins, the default local filessystem, the CloudantDB plugin that is used with Node-RED in the IBM Cloud.

I’m going to use MongoDB as the backend storage and the Mongoose library to wrap the reading/writing to the database (I’ll be reusing the Mongoose schema definiations later in the Authentication plugin and the app to manager Node-RED instances).

The documentation from the Storage API can be found here. There are a number of methods that a plugin needs to provide:

init()

This sets up the plugin, it reads the settings and then opens the connection to the database.

init: function(nrSettings) {
  settings = nrSettings.mongodbSettings || {};

  if (!settings) {
    var err = Promise.reject("No MongoDB settings for flow storage found");
    err.catch(err => {});
    return err;
  }

  appname = settings.appname;

  return new Promise(function(resolve, reject){
    mongoose.connect(settings.mongoURI, mongoose_options)
    .then(() => {
      resolve();
    })
    .catch(err => {
      reject(err);
    });
  })
},

getFlows()/saveFlows(flows)

Here we retrieve/save the flow to the database. If there isn’t a current flow (such as the first time the instance is run) we need to return an empty array ([])

getFlows: function() {
  return new Promise(function(resolve, reject) {
    Flows.findOne({appname: appname}, function(err, flows){
      if (err) {
        reject(err);
      } else {
        if (flows){
          resolve(flows.flow);
        } else {
          resolve([]);
        }
      }
    })
  })
},
saveFlows: function(flows) {
  return new Promise(function(resolve, reject) {
    Flows.findOneAndUpdate({appname: appname},{flow: flows}, {upsert: true}, function(err,flow){
      if (err) {
        reject(err)
      } else {
        resolve();
      }
    })
  })
},

The upsert: true in the options passed to findOneAndUpdatet() triggers an insert if there isn’t an existing matching document.

getCredentials/saveCredentials(credentials)

Here we had to convert the credentials object to a string because MongoDB doesn’t like root object keys that start with a $ (the encrypted credentials string is held in the $_ entry in the object.

getCredentials: function() {
  return new Promise(function(resolve, reject) {
    Credentials.findOne({appname: appname}, function(err, creds){
      if (err) {
        reject(err);
      } else {
        if (creds){
          resolve(JSON.parse(creds.credentials));
        } else {
          resolve({});  
        }
      }
    })
  })
},
saveCredentials: function(credentials) {
  return new Promise(function(resolve, reject) {
    Credentials.findOneAndUpdate({appname: appname},{credentials: JSON.stringify(credentials)}, {upsert: true}, function(err,credentials){
      if (err) {
        reject(err)
      } else {
        resolve();
      }
    })
  })
},

getSessions()/saveSessions(sessions)/getSettings()/saveSettings(settings)

These are pretty much just carbon copies of the getFlows()/saveFlows(flows) functions since they are just storing/retrieving a single JSON object.

getLibraryEntry(type,name)/saveLibraryEntry(type,name,meta,body)

saveLibraryEntry(type,name,meta,body) is pretty standard with a little bit of name manipulation to make it look more like a file path.

getLibraryEntry(type,name,meta,body) needs a bit more work as we need to build the directory structure as well as being able to return the actual file content.

getLibraryEntry: function(type,name) {
  if (name == "") {
    name = "/"
  } else if (name.substr(0,1) != "/") {
    name = "/" + name
  }

  return new Promise(function(resolve,reject) {
    Library.findOne({appname: appname, type: type, name: name}, function(err, file){
      if (err) {
        reject(err);
      } else if (file) {
        resolve(file.body);
      } else {
        var reg = new RegExp('^' + name , "");
        Library.find({appname: appname, type: type, name: reg }, function(err, fileList){
          if (err) {
            reject(err)
          } else {
            var dirs = [];
            var files = [];
            for (var i=0; i<fileList.length; i++) {
              var n = fileList[i].name;
              n = n.replace(name, "");
              if (n.indexOf('/') == -1) {
                var f = fileList[i].meta;
                f.fn = n;
                files.push(f);
              } else {
                n = n.substr(0,n.indexOf('/'))
                dirs.push(n);
              }
            }
            dirs = dirs.concat(files);
            resolve(dirs);
          }
        })
          
      }
    })
  });
},
saveLibraryEntry: function(type,name,meta,body) {
  return new Promise(function(resolve,reject) {
    var p = name.split("/");    // strip multiple slash
    p = p.filter(Boolean);
    name = p.slice(0, p.length).join("/")
    if (name != "" &amp;&amp; name.substr(0, 1) != "/") {
      name = "/" + name;
    }
    Library.findOneAndUpdate({appname: appname, name: name}, 
      {name:name, meta:meta, body:body, type: type},
      {upsert: true, useFindAndModify: false},
      function(err, library){
        if (err) {
          reject(err);
        } else {
          resolve();
        }
      });
  });
}

Using the plugin

To use the plugin you need to include it in the settings.js file. This is normally found in the userDir (the location of this file is logged when Node-RED starts up).

...
storageModule: require('node-red-contrib-storage-mongodb'),
mongodbSettings: {
    mongoURI: "mongodb://localhost/nodered",
    appname: "r1"
},
...

The mongodbSettings object contains the URI for the database and the appname is a unique identifier for this Node-RED instance allowing the same database to be used for multiple instances of Node-RED.

Source code

You can find the code for this module here and it’s hosted on npmjs here

Multi Tenant Node-RED

I was recently approached by a company that wanted to sponsor adding Multi Tenant support to Node-RED. This would be to enable multiple users to run independent flows on a single Node-RED instance.

This is really not a good idea for a number of reasons. But mainly it is because the NodeJS runtime is inherently single threaded and there is no way to get real separation between different users. For example, if one user uses a poorly written node (or function in a function node) it is possible to block the event loop starving out all the other users or in extreme cases an uncaught asynchronous exception will cause the whole application to exit.

The best approach is to run a separate instance of Node-RED per user, which gives you the required separation between users. The usual way to do this is to use a container based system and a HTTP reverse proxy to control access to the instances.

Over the next months worth of posts I’ll outline the components required to build a system like I described and at the end should hopefully have a fully working Proof of Concept that people can use as the base to build their own deployments.

As the future posts are published I will add links here.

Required components

Flow Storage

Because containers file systems are not persistent we are going to need somewhere to reliably store the flows each user creates.

Node-RED supplies an API that lets you control how/where flows (and a bunch of things that would normally end up on disk)

Authentication

We are going to need a way to only allow right users access to the Node-RED editor, again there is a plugin API that allows this to be wired into nearly any existing authentication source of your choice.

I wrote a simple implementation of this API which uses LDAP as the source of users not long after Node-RED was released . But in this series of post I’ll write a new one that uses the same backend database as the flow storage plugin.

Custom Container

Once we’ve built storage and authentication plugins we will need to build a custom version of the Node-RED Docker container that includes these extras.

HTTP Reverse Proxy

Now we have a collection of containers each running a users instance of Node-RED we are going to need a way to expose these to the outside world. Since Node-RED is a web application then a HTTP Reverse Proxy is probably the right way forward.

Management

Once all of the other pieces are in place we need is a way to control the creation/deletion of Node-RED instances. It will also be useful to see the instances’ logs.

Extra bits

Finally I’ll cover some extra bits that help make a deployment specific to a particular environment, such as

Custom node repository

This allows you to host your own private nodes and still install them using the Manage Palette menu.

Custom Theme

Tweaking page titles and colour schemes can make Node-RED fit in better with your existing look and feel.

Node-RED Google Home Smart Home Action Generally Available

I started this post back in November 2017, it’s been a long slog, but we are finally here. We had a false start back in January 2019 when I got the bulk of the code all working and I thought it would take a few weeks get it certified and released and again at the beginning of this year when the Action got approved but still required you to be a member of a Google Group to be able to sign up . Unfortunately that wasn’t the case. But all that is behind us now.

A Node-RED flow with lots of Google Home Assistant nodes

You can find the full docs for how to install and configure the Action here, but the short version is:

  • Create an account here
  • Create some devices using the wizard
  • Link the NR-GAB Action to your Google Account in the Google Home app, it should have the following icon.
  • Install the node-red-contrib-googlehome node in Node-RED
  • Drag the googlehome-in node on to the canvas and start building flows that can be triggered by the Google Assistant.

As always the doc probably needs a little bit more work, so I’ll keep updating it as folks run into issues with it. If you have any questions the best place to ask them is in the #google-home-assistant channel on the Node-RED Slack team.

Next steps are I have a working version of the Google Assistant Local Control API that I will release as soon as Google open that up for general availability. This sends commands directly to Node-RED from the Smart Speaker device which reduces the latency in triggering actions.