Node-Red – Delay Node (formally Pause Node)

I’ve just been updating my Pause node for Node-Red after a request to support message rate limiting as well as pausing individual messages.

To help make it a little clearer the node has also been renamed to Delay (thanks to deldrid1 for the suggestion)

Delay Mode

In this mode the node allows you to delay any message passing through it by a given number of milliseconds, seconds, hours or days.

Rate Limit Mode

This time the node ensures that no more than the given number of messages are delivered per millisecond, second, hour or day.

All the code is in my fork of the original project for now and soon to be rolled up in to the main stream.

Taking a look at the neighbourhood

A recent article about detecting offline social networks from information about preferred WIFI networks leaked by mobile devices led me to have another look at the work I’d done looking at WIFI location detection to see what other information I could derive.

I had been having problems with finding WIFI adapters with the right chipsets to allow me to use them in monitor mode in order to capture all the available packets from different networks, but recent updates to some of the WIFI drivers in the Linux kernel have enabled some more of the devices I have access to.

Previously I build a small application which works with the Kismet wireless scanner application to publish details of each device spotted to a MQTT topic tree. With a small modification it now also published the data about the WIFI networks that are being searched for.

Then using 2 simple Node-RED flows this data is stored into a MongoDB instance

You can have a look at the flow here.

From the device’s MAC address it is possible to determine the manufacture, so with another little node application to query the MongoDB store I can generate this d3js view of what type of devices are in use in the area round my flat.

The view dynamically updates every 5 seconds to pick up the latest information.

Now I know who owns what type of device, time to see who might know who. By plotting a force directed graph of all the clients detected and linking them based on the networks they have been searching for I can build up a view of which devices may belong to people who know each other.

Force directed network graph

There are a couple of clusters in the data so far, but most of them are from public WIFI networks like BTOpenzone and O2 Wifi. After filtering these services out there was still the 3 devices that look to be using Mike’s Lumina 800 for internet access and 4 devices connected to the same Sky Broadband router. I expect the data to be a lot more interesting when I get to run it somewhere with a few more people.

At the moment this is all running on my laptop, but it should run fine on my raspberry pi or my home server, as soon as I’ve transferred it over I’ll put a link up to live version of the charts.

Node-Red – Digispark RGB Node

Following on from last weekends post about adding a pause node to Node-Red this weekend I’ve been trying to create a node to support my Digispark RGB USB LED.

The RGB shield for the Digispark came with a sketch and a some python code to send colours to the device using the HID protocol. I had a bit of a poke round and I found node-hid and thought things where going to be easy.

Unfortunately just trying to duplicate the python code in nodejs didn’t seam to work no matter what I tried.

After a bit more digging I found this blog post by Dougal Campbell who looked to have been pursuing a similar idea a little earlier in the year. I couldn’t see any follow up post on his blog but a quick comment later and I had pointers to what he’d managed to get working. I was hoping to leave the default sketch on the digiSpark but it needed a little change to the USB library to make things work properly, you can find the changes here.

88-digiRGB.js


// Require main module
var RED = require("../../red/red");
var HID = require('node-hid');
var device;
var node;

// The main node definition - most things happen in here
function DigiRGBNode(n) {
    // Create a RED node
    RED.nodes.createNode(this,n);
    node=this;
    //look up the matching devices
    var devices = HID.devices(0x16c0,0x05df);
    for (var i=0; i< devices.length; i++) {
      if (devices[i].product == 'DigiUSB') {
        path = devices[i].path;
        node.log("found: " + path);
        try {
          device = new HID.HID(devices[i].path);
          //only work with the first one found
          break;
        } catch (e) {
          node.log(e)
        }
      }  
   }
   
   if (device) {   
      this.on("input", function(msg) {
        if (msg != null) {
          var args = msg.payload.split(',');
          if (args.length == 3) {
            device.sendFeatureReport([115,parseInt(args[0]),parseInt(args[1]),parseInt(args[2])]);
          }
        }
      });
   } else {
      node.warn("no digispark RGB found");
   }
   
}

// Register the node by name. This must be called before overriding any of the
// Node functions.
RED.nodes.registerType("digiRGB",DigiRGBNode);


DigiRGBNode.prototype.close = function() {
    // Called when the node is shutdown - eg on redeploy.
    // Allows ports to be closed, connections dropped etc.
    // eg: this.client.disconnect();
    device.close();
}

If you want to have a play unzip 78-digiRGB.zip in the node-red directory to unpack the whole node into the nodes/hardware directory. I have now saved this a Github GIST which you can check out directly into /node-red/nodes/hardware directory as follows:

[user@node node-red]$ git clone https://gist.github.com/6573158.git

You will also need to run npm import node-hid in the node-red directory to add the dependency.

(Remember you will need to modify the DigiUSB.cpp and the usbconfig.h in the DigisparkUSB example as mentioned in the link above).

This is unlikely to make it into core Node-RED as the hardware is a little specialist, but if Nick and Dave decide to host a collection of hardware nodes it may end up in there.

Node-Red – Pause Node

Two guys from my team (Nick O’Leary and Dave Conway-Jones) at IBM released a very cool tool for wiring up IoT last week called Node-Red.

Node-Red lets you wire together a selection of inputs, things like MQTT messages, Twitter feeds, serial port and a whole lot more, perform transforms on them with Javascript functions before outputting them to a similar collection of options.

I’ve been playing with it to replace a couple things that used to be standalone scripts, but one of the things that was missing was the way to delay an action happening until a certain time after a message has arrived. For example if I want to set my Blink(1) clone to green when somebody mentions me in the tweet I may want to reset it to off 20 seconds later

To help with this I have written a new node called pause that delays passing on a given message by a configurable amount of time. Nodes are made up of 2 files and are placed in the node-red/nodes directory. As well as all the built in nodes there is a example node (99-sample) which you can use to

Each node is made up of 2 parts, the first is javascript file which controls how the node should behave and a html file which contains the descriptive text,

99-pause.js

...
// Simple node to introduce a pause into a flow

//Require main module
var RED = require("../red/red");

function PauseNode(n) {
   RED.nodes.createNode(this,n);
   
   this.timeout = n.timeout * 1000;
   
   this.on("input", function(msg) {
       var node= this;
       setTimeout(function(){node.send(msg);}, node.timeout);
   });
}

RED.nodes.registerType("pause",PauseNode);

You can see in the code above I’ve attached a small function to the ‘input’ event for the pause node, all it does setup a callback after the timeout expires to send the passed in message object to the next node in the flow.

99-pause.html

...
<!-- First, the content of the edit dialog is defined.                       -->
<script type="text/x-red" data-template-name="pause">
  <div class="form-row">
     <label for="node-input-topic"><i class="icon-tasks"></i> Pause</label>
    <input type="text" id="node-input-timeout" placeholder="Time">
  </div>
    
  <div class="form-row">
    <label for="node-input-name"><i class="icon-tag"></i> Name</label>
    <input type="text" id="node-input-name" placeholder="Name">
  </div>
</script>

<!-- Next, some simple help text is provided for the node.                   -->
<script type="text/x-red" data-help-name="pause">
    <p>Introduces a pause into a flow</p>
    <p>Default pause is 5 seconds but can be configured</p>
</script>

<!-- Finally, the node type is registered along with all of its properties   -->
<script type="text/javascript">
    RED.nodes.registerType('pause',{
        category: 'function',      // the palette category
        color:"#E6E0F8",
        defaults: {             // defines the editable properties of the node
            name: {value:""},   //  along with default values.
            timeout: {value:"5", required:true, validate:RED.validators.number()}
        },
        inputs:1,                // set the number of inputs - only 0 or 1
        outputs:1,               // set the number of outputs - 0 to n
        icon: "arrow-in.png",    // set the icon (held in public/icons)
        label: function() {      // sets the default label contents
            return this.name||this.topic||"pause";
        },
        labelStyle: function() { // sets the class to apply to the label
            return this.name?"node_label_italic":"";
        }
    });
</script>

The HTML sets up configuration dialog for the node, the help text, the nodes look and behaviour and the default values for any properties the node has.

The configuration dialog for the pause node looks like this:

I’ve checked this new node into my fork of the Node-Red project here, but I’ll be submitting a pull request to get it included in the main project as soon as I get a chance to have a chat with Nick and Dave.

EDIT:
This is now a core node called delay and is also capable of rate limiting messages

d3 MQTT Tree visualiser updated

I’ve been having a bit of a play updating my d3 based MQTT topic tree visualiser this weekend.

  1. I’ve been trying to tidy things up a bit and break the code out into a little standalone library and only leave the MQTT connection code in the index.html.
  2. I’ve been improving the handling of long payload. There is now a nicer popup tooltip with a longer version of the payload, it’s still cropped at 100 chars but I’m still working on wrapping it and showing a few lines to get more in.
  3. I’ve been moving the MQTT connection code over to use binary WebSocket connections* rather than the JSON based proxy I was using before. The version hosted here is still using the proxy for now, but the binary versions work and I’ll move it over once I’ve finished playing with my internal broker setup.

I still need to try and make the whole thing resize or at least not have hard coded dimensions, but that might have to wait for next time.

I’ve stuck the updated version up here, I’ll stick the code up on github once I get sign off from the boss.

https://github.com/hardillb/d3-MQTT-Topic-Tree

* There is a new IBM WebSphere MQ feature pack supporting this, details can be found here

RGB LED Meeting Warning Light (Lotus Notes Edition)

At the end of my last post I mentioned trying to get the same set up working with my work Lotus Notes calendar.

After a little poking around with the Lotus Notes Java API here is the result:

package uk.me.hardill.notes;

/**
 * NextCalEntry
 * 
 * Sets the RGB values for Blink(1)/Digispark+RGB
 * according to the time to the next meeting in 
 * your Lotus Notes Calendar
 * 
 * This should be run with the JRE that ships with
 * Lotus notes as it has the required classes on 
 * classpath and have Lotus notes directory on 
 * library path e.g.
 * LD_LIBRARY_PATH=/opt/ibm/lotus/notes
 * /opt/ibm/lotus/notes/jvm/bin/java NextCalEntry -d
 */

import java.text.SimpleDateFormat;
import java.util.Date;

import lotus.domino.Database;
import lotus.domino.DateTime;
import lotus.domino.DbDirectory;
import lotus.domino.Document;
import lotus.domino.NotesFactory;
import lotus.domino.NotesThread;
import lotus.domino.Session;
import lotus.domino.View;
import lotus.domino.ViewEntry;
import lotus.domino.ViewEntryCollection;

public class NextCalEntry extends NotesThread {

   static SimpleDateFormat dateFormat = new SimpleDateFormat(
         "dd/MM/yy H:m:s z");

   static int rgb[] = { 0, 200, 0 };

   static int pollInterval = 300;
   static boolean mqtt = false;
   static String topic = "";
   static boolean digi = true;

   public static void main(String argv[]) {

      for (int i = 0; i < argv.length; i++) {
         if (argv[i].equals("-b")) {

         } else if (argv[i].equals("-d")) {

         } else if (argv[i].equals("-t")) {
            mqtt = true;
            topic = argv[++i];
         }
      }

      NextCalEntry nextCalEntry = new NextCalEntry();
      nextCalEntry.start();
   }

   public void runNotes() {
      try {
         Session s = NotesFactory.createSession();

         DbDirectory dir = s.getDbDirectory(null);
         Database db = dir.getFirstDatabase(DbDirectory.DATABASE);

         db = dir.openMailDatabase();

         if (db.isOpen() == false)
            db.open();

         db = dir.openMailDatabase();
         View calendarView = db.getView("($Calendar)");

         DateTime sdt = s.createDateTime("today");
         sdt.setNow();
         DateTime edt = s.createDateTime("today");
         edt.setNow();
         edt.adjustDay(+1);

         ViewEntryCollection vec = calendarView.getAllEntries();

         ViewEntry entry = vec.getFirstEntry();

         int offset = 3600;
         boolean poisonPill = false;
         while (entry != null) {
            Document caldoc = entry.getDocument();
            String sub = caldoc.getItemValueString("Subject");
            DateTime startDate = null;
            try {
               startDate = (DateTime) caldoc.getItemValueDateTimeArray(
                     "StartDate").firstElement();

            } catch (Exception e) {

            }

            if (startDate != null) {

               for (int i = 0; i < caldoc.getItemValueDateTimeArray(
                     "StartDateTime").size(); i++) {

                  int st = sdt.timeDifference((DateTime) caldoc
                        .getItemValueDateTimeArray("StartDateTime")
                        .get(i));
                  int en = edt.timeDifference((DateTime) caldoc
                        .getItemValueDateTimeArray("EndDateTime")
                        .get(i));

                  Date start = dateFormat.parse(caldoc
                        .getItemValueDateTimeArray("StartDateTime")
                        .get(i).toString());
                  Date end = dateFormat.parse(caldoc
                        .getItemValueDateTimeArray("EndDateTime")
                        .get(i).toString());
                  Date now = new Date();

                  if ((st <= 0) & (en >= 0)) {
                     if ((-1 * st) < offset) {
                        offset = (-1 * st);
                     }

                  } else if (now.after(start) && now.before(end)) {
                     offset = -1;
                     poisonPill = true;
                     break;
                  }
               }
            }
            if (poisonPill) {
               break;
            }
            entry = vec.getNextEntry();
         }

         if (offset > 0 && offset <= 300) {
            // red
            rgb[0] = 20;
            rgb[1] = 0;
            rgb[2] = 0;
         } else if (offset > 300 && offset <= 600) {
            // redish
            rgb[0] = 15;
            rgb[1] = 5;
            rgb[2] = 0;
         } else if (offset > 600 && offset <= 900) {
            // greenish/redish
            rgb[0] = 10;
            rgb[1] = 10;
            rgb[2] = 0;
         } else if (offset > 900 && offset <= 1200) {
            // greenish
            rgb[0] = 5;
            rgb[1] = 15;
            rgb[2] = 0;
         } else if (offset == -1) {
            // blue
            rgb[0] = 0;
            rgb[1] = 0;
            rgb[2] = 20;
         } else {
            // green
            rgb[0] = 0;
            rgb[1] = 20;
            rgb[2] = 0;
         }

         if (mqtt) {
            // TODO
            // connect to broker and publish
         } else {
            Runtime runtime = Runtime.getRuntime();
            String cmd[];
            if (digi) {
               System.out.println(rgb[0] + " " + rgb[1] + " " + rgb[2]);
               cmd = new String[4];
               cmd[0] = "DigiRGB.py";
               cmd[1] = rgb[0] + "";
               cmd[2] = rgb[1] + "";
               cmd[3] = rgb[2] + "";
            } else {
               System.out.println(rgb[0] + "," + rgb[1] + "," + rgb[2]);
               cmd = new String[3];
               cmd[0] = "blink1-tool";
               cmd[1] = "--rgb";
               cmd[2] = rgb[0] + "," + rgb[1] + "," + rgb[2];
            }
            Process proc = runtime.exec(cmd);
            proc.waitFor();
         }

      } catch (Exception e) {
         e.printStackTrace();
      }
   }
}

It is also available as gist here (not embedded as there seams to be CSS background colour issue)

Run this in a script every 5 mins either with a sleep loop or as a crontab entry similar to the ones described at the end of the previous post

RGB LED Meeting Warning Light

In my previous post I talked about the Digispark board and the RGB LED shield I had just received.

Digispark and RGB Shield

This afternoon in a spare 30mins I had while some tests ran I built my first little hack to use one of them. It is inspired by this really cool hack that got mentioned on twitter by Dave CJ while we were debating Blink(1) vs Digispark/RGB Shields. The idea of a strip of LEDs that represent the working day with different colours for free time and meetings is cool, but I only have 1 LED to work with, so I thought how about just showing how close to the next meeting I am.

The idea is that if the next entry in my calendar is more than 20mins away then the light is green, as the meeting start time gets closer the LED changes closer to red and finally during the meeting if glows blue.

I started out prototyping this against my google calendar as it’s pretty easy to get at the data, if you poke around in the settings page you can get hold of a URL that points to either a XML or ICAL version of the data.

I’m running this on my Raspberry Pi so python seamed like a good choice to run this up in.

#!/usr/bin/python

import sys
import pycurl
import StringIO
from icalendar import Calendar, Event
from datetime import datetime, timedelta

curl = pycurl.Curl()
curl.setopt(pycurl.URL,sys.argv[1])
body = StringIO.StringIO()
curl.setopt(pycurl.WRITEFUNCTION, body.write)
curl.perform()

cal = Calendar.from_ical(body.getvalue())
body.close()

now = datetime.now()

offset = 3600

for component in cal.walk():
  if component.name == "VEVENT":
    dtstart = component["dtstart"].to_ical()
    dtend = component["dtend"].to_ical()
    if "T" in dtstart:
      dt = datetime.strptime(dtstart, "%Y%m%dT%H%M%SZ")
      end = datetime.strptime(dtend, "%Y%m%dT%H%M%SZ")

      if dt > now:
        delta = dt - now
        if delta.days == 0:
          if delta.seconds < offset:
            offset = delta.seconds
      elif end > now and dt < now:
        offset = -1
        break

if offset > 0 and offset < 300:
  print "200 0 0"
elif offset >= 300 and offset < 600:
  print "150 50 0"
elif offset >= 600 and offset < 900:
  print "100 100 0"
elif offset >= 900 and offset < 1200:
  print "50 150 0"
elif offset == -1:
  print "0 0 200"
else:
  print "0 200 0"

This script outputs RGB values that can be fed directly to DigiBlink.py script like this in a cron job

*/5 * * * * DigiBlink.py `calColur.py <calendar URL>`

Or since I’ve set up a little script to control my DigiSpark via MQTT the following publishes the output of the script on the ‘digiblink’ topic on the local broker.


*/5 * * * * calColur.py <calendar URL> | mosquitto_pub -h localhost -t digiblink -l

This script should also work with the command line tool for the Blink(1)

I still need to tweak the colours for each step and next work out how to get the same feed out of the company calendaring system, but it’s a good start.

Budget Blink(1)

Production blink(1) units

About a month ago I spotted Kickstarter project for a device called a Blink(1). This was a little USB device that acts like Ambient Orb and it made by the same folk that make the BlinkM 3 colour LED board that is used in them. I had loads of ideas with what to do with one.

The project had already reached it’s funding target and it had actually delivered. The devices where available to order directly from ThinkM here. I was all set to buy one except for 1 thing, the price. While I don’t really have a problem with ThinkM wanting $30 each for them, the problem is not even with the fact that HMRC will want to charge me VAT (at 20% on anything imported with a price of more than £15) on this when it is imported the real issue is that which ever courier handles the UK end of the delivery will charge me at least £8 in “handling” fee for the privileged.

Digispark and RGB Shield

I put the idea to the back of my mind for a while with the hope that the price would come down or a UK supplier would be found. While waiting I came across another Kickstarter project for a really tiny Arduino called a Digispark from the Digistump team. As well as the digispark boards themselves they where doing a range of shield, one of which was a RGB LED shield. Being a miniature arduino means I should be able to add the odd sensor or two to the board which I can use to influence the colour shown by the LED.

And to top it off I could order 2 Digisparks and 2 RGB LED shields + shipping for $26 which comes in just under the import limit for VAT.

The package was waiting for me when I got back from a weekend away and I managed to grab 5 mins today at the office to use the soldering iron to put one of the shields together and add the headers to the first digispark.

I plugged it into my laptop and the standard blink sketch seamed to be running fine. I started to follow the instructions on the wiki to set up the development environment under Linux and I think I followed it all properly but I could not get the tools to upload a new sketch to the device. I have added a comment to a similar question on the forum. In the mean time I have managed to get it all to work on my Windows machine.

EDIT: Following a comment on the forum suggesting I try plugging the digispark into a powered hub to program it I have managed to get my second digispark programmed from Linux.

Along with the RGB shield there is some sample code that makes the digispark behave as a HID device and some python code that allows you to send it RGB values to set the colour shown much the same way the Blink(1) works. When I get 5 mins I will be doing the obvious MQTT hack to allow RGB values to be posted to a topic to set the colour then look at hooking it up to some of the data feeds available.