Tag Archives: javascript

Unpacking binary data from MQTT in Javascript

While doing trawl of Stackoverflow for questions I might be able to help out with I came across this interesting looking question:

Receive binary with paho mqttws31.js

The question was how to unpack binary MQTT payloads into double precision floating point numbers in javascript when using the Paho MQTT over WebSockets client.

Normally I would just send floating point numbers as strings and parse them on the receiving end, but sending them as raw binary means much smaller messages, so I thought I’d see if I could help to find a solution.

A little bit of Googling turned up this link to the Javascript typed arrays which looked like it probably be in the right direction. At that point I got called away to look at something else so I stuck a quick answer in with a link and the following code snippet.

function onMessageArrived(message) {
  var payload = message.payloadByte()
  var doubleView = new Float64Array(payload);
  var number = doubleView[0];
  console.log(number);
}

Towards the end of the day I managed to have a look back and there was a comment from the original poster saying that the sample didn’t work. At that point I decided to write a simple little testcase.

First up quick little Java app to generate the messages.

import java.nio.ByteBuffer;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;

public class MessageSource {

  public static void main(String[] args) {
    try {
      MqttClient client = new MqttClient("tcp://localhost:1883", "doubleSource");
      client.connect();

      MqttMessage message = new MqttMessage();
      ByteBuffer buffer = ByteBuffer.allocate(8);
      buffer.putDouble(Math.PI);
      System.err.println(buffer.position() + "/" + buffer.limit());
      message.setPayload(buffer.array());
      client.publish("doubles", message);
      try {
        Thread.sleep(1000);
      } catch (InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
      }
      client.disconnect();
    } catch (MqttException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
  }
}

It turns out that using typed arrays is a little more complicated and requires a bit of work to populate the data structures properly. First you need to create an ArrayBuffer of the right size, then wrap it in a Uint8Array in order to populate it, before changing to the Float64Array. After a little bit of playing around I got to this:

function onMessageArrived(message) {
  var payload = message.payloadBytes
  var length = payload.length;
  var buffer = new ArrayBuffer(length);
  uint = new Uint8Array(buffer);
  for (var i=0; i<length; i++) {
	  uint[i] = payload[i];
  }
  var doubleView = new Float64Array(uint.buffer);
  var number = doubleView[0];
  console.log("onMessageArrived:"+number);
};

But this was returning 3.207375630676366e-192 instead of Pi. A little more head scratching and the idea of checking the byte order kicked in:

function onMessageArrived(message) {
  var payload = message.payloadBytes
  var length = payload.length;
  var buffer = new ArrayBuffer(length);
  uint = new Uint8Array(buffer);
  for (var i=0; i<length; i++) {
	  uint[(length-1)-i] = payload[i];
  }
  var doubleView = new Float64Array(uint.buffer);
  var number = doubleView[0];
  console.log("onMessageArrived:"+number);
};

This now gave an answer of 3.141592653589793 which looked a lot better. I still think there may be a cleaner way to do with using a DataView object, but that’s enough for a Friday night.

EDIT:

Got up this morning having slept on it and came up with this:

function onMessageArrived(message) {
  var payload = message.payloadBytes
  var length = payload.length;
  var buffer = new ArrayBuffer(length);
  uint = new Uint8Array(buffer);
  for (var i=0; i<length; i++) {
	  uint[i] = payload[i];
  }
  var dataView = new DataView(uint.buffer);
  for (var i=0; i<length/8; i++) {
      console.log(dataView.getFloat64((i*8), false));
  }
};

This better fits the original question in that it will decode an arbitrary length array of doubles and since we know that Java is big endian, we can set the little endian flag to false to get the right conversion without having to re-order the array as we copy it into the buffer (which I’m pretty sure wouldn’t have worked for more than one value).

d3 MQTT Tree visualiser updated

I’ve been having a bit of a play updating my d3 based MQTT topic tree visualiser this weekend.

  1. I’ve been trying to tidy things up a bit and break the code out into a little standalone library and only leave the MQTT connection code in the index.html.
  2. I’ve been improving the handling of long payload. There is now a nicer popup tooltip with a longer version of the payload, it’s still cropped at 100 chars but I’m still working on wrapping it and showing a few lines to get more in.
  3. I’ve been moving the MQTT connection code over to use binary WebSocket connections* rather than the JSON based proxy I was using before. The version hosted here is still using the proxy for now, but the binary versions work and I’ll move it over once I’ve finished playing with my internal broker setup.

I still need to try and make the whole thing resize or at least not have hard coded dimensions, but that might have to wait for next time.

I’ve stuck the updated version up here, I’ll stick the code up on github once I get sign off from the boss.

https://github.com/hardillb/d3-MQTT-Topic-Tree

* There is a new IBM WebSphere MQ feature pack supporting this, details can be found here

d3 MQTT topic tree visualiser

As part of writing my talks for the developerWorks Days Zurich conference next week I’ve been working on some code for the demo in the MQTT session. I was looking for a nice way to display a MQTT topic structure and the most recent value posted to topic.

In the past I have used a plugin to the freemind mindmap tool that plots the topics as and their values. This worked well but it does require a machine with Java and a very old version of freemind to make it work so it was time to see if there was another approach.

Nick O’Leary had created a neat little python MQTT to Websockets* bridge that we have been using round the office for a while and there are some really great javascript visualisation libraries out there now, so I thought I would see how hard it would be to run up a browser based version.

Starting with the d3 Collapsible Tree Layout example seamed like a good idea and it didn’t take too long to tweak it a little to make it dynamically add new topic branches when a message was received.

function addNode(topic, body) {
	var parts = topic.split("/");
	if (root.children[0]===undefined){
		var newnode = {"name": parts.shift(), "children":[]};
		root.children = [newnode];
		walk(parts,newnode,body);
	} else {
		walk(parts,root,body);
	}
	update(root);
}

function walk(parts, node, body) {
	if (parts.length != 0) {
		var current = parts.shift()
		if (node.children && node.children.length != 0) {
			var z=0;
			for(z=0; z < node.children.length; z++) {
				if (node.children[z].name == current) {
					walk(parts,node.children[z], body);
					break;
				}
			}
			if (z == node.children.length) {
				var newnode = {"name": current, "children":[]};
				node.children.push(newnode);
				walk(parts,node.children[z],body);
			}
		} else {
			var newnode = {"name": current, "children":[]};
			node.children = [newnode];
			walk(parts,node.children[0],body);
		}
	}
}

I also swapped out the where the data was loaded to use a static root rather than loading it from a json file on the server.

var root = {"name": "/","children": []};
root.x0 = h / 2;
root.y0 = 0;
update(root);

This worked pretty well so next I set about looking to see if I could add the payload of the message. To do this I added a data field to the json object for each node when populating the tree.

function walk(parts, node, body) {
	if (parts.length != 0) {
		var current = parts.shift()
		if (node.children && node.children.length != 0) {
			var z=0;
			for(z=0; z < node.children.length; z++) {
				if (node.children[z].name == current) {
					walk(parts,node.children[z], body);
					break;
				}
			}
			if (z == node.children.length) {
				var newnode = {"name": current, "children":[]};
				node.children.push(newnode);
				walk(parts,node.children[z],body);
			}
		} else {
			var newnode = {"name": current, "children":[]};
			node.children = [newnode];
			walk(parts,node.children[0],body);
		}
	} else {
		console.log("body");
		node.data = body;
	}
}

Then updated where the SVG is inserted to add a text element in the right place.

...
  nodeEnter.append("svg:text")
      .attr("x", function(d) { return d.children || d._children ? -10 : 10; })
      .attr("dy", ".35em")
      .attr("text-anchor", function(d) { return d.children || d._children ? "end" : "start"; })
      .text(function(d) { return d.name; })
      .style("fill-opacity", 1e-6);
  
  // Add message payload
  nodeEnter.append("svg:text")
  	.attr("class","data")
	.attr("x", 10)
	.attr("y", 10)
	.attr("dy", ".35em")
	.text(function(d) { return d.data || ""; })
	.style("fill-opacity", 1);
  
  // Transition nodes to their new position.
  var nodeUpdate = node.transition()
      .duration(duration)
      .attr("transform", function(d) { return "translate(" + d.y + "," + d.x + ")"; });
...

Again this worked very well, except that it would not update the payload value after adding the first version. I tried a number of things to get this to work, but didn’t get very far, until James Thomas pointed me at this page about how some of the internals of d3 work. It talks about entry, update and exit methods to work on new, existing and removed elements in the data. There are examples of entry and exit functions in the example so I tried adding an update method but kept getting errors that it wasn’t found. I went back and had another look at the examples in the page James had linked to and noticed that update code doesn’t actually use a functions.

...
// Add message payload
  nodeEnter.append("svg:text")
  	.attr("class","data")
	.attr("x", 10)
	.attr("y", 10)
	.attr("dy", ".35em")
	.text(function(d) { return d.data || ""; })
	.style("fill-opacity", 1);

  // Update payload if changed  
  var nodeChange = node.selectAll("text.data")
      .text(function(d) {return d.data || "";});

  // Transition nodes to their new position.
  var nodeUpdate = node.transition()
      .duration(duration)
      .attr("transform", function(d) { return "translate(" + d.y + "," + d.x + ")"; });
...

An that was basically it. I’ve added a bit more code to update values when the tree branches are collapsed and to trim payloads to no more than 20 chars so the display is not too cluttered. I’m planning on improving the style sheet a little soon and possibly adding an effect so you see which payloads have been updated recently and maybe add hover text with the full payload for the shortened ones.

You can see the whole thing running here

* There is also some work being done to support Websockets with the Mosquitto MQTT broker, you can see the testing page here