Tag Archives: d3

Finally got round to updating site home page

I’ve had the same basic home page for this domain since I set it up. It was never meant to last, it was just there until I got wordpress set up to host the blog.

I was playing with the Google Webmaster tools last week and it pointed out some broken links on the home page so I decided it was time to finally update it.

The old version shows just how badly I understood the HTML box model and lack in graphic design skills. Also widgets are very old.

home page
My original home page

Rather than work out how to do it all properly these days there are any number of frameworks that will do it all for you. I’ve picked Bootstrap which supports responsive layouts which should mean that it should look good no matter what the screen size of the device that views it. The old version really didn’t work on phone/tablet screens.

I wanted to keep the live feed from the power meter and weather centre, but it was time to update them from the dojo and Google charts versions to something a bit newer. The line charts are now Rickshaw/D3JS. I’ve used D3JS before for a things like my MQTT topic tree viewer.

The wind direction indicator is just a background image with a ‘↑’ character at a large font size rotated using CSS transforms.

All the data comes in as a set of subscriptions to the MQTT broker running on my box via Websockets.

New home page
New home page

You can see the results for real here.

Taking a look at the neighbourhood

A recent article about detecting offline social networks from information about preferred WIFI networks leaked by mobile devices led me to have another look at the work I’d done looking at WIFI location detection to see what other information I could derive.

I had been having problems with finding WIFI adapters with the right chipsets to allow me to use them in monitor mode in order to capture all the available packets from different networks, but recent updates to some of the WIFI drivers in the Linux kernel have enabled some more of the devices I have access to.

Previously I build a small application which works with the Kismet wireless scanner application to publish details of each device spotted to a MQTT topic tree. With a small modification it now also published the data about the WIFI networks that are being searched for.

Then using 2 simple Node-RED flows this data is stored into a MongoDB instance

You can have a look at the flow here.

From the device’s MAC address it is possible to determine the manufacture, so with another little node application to query the MongoDB store I can generate this d3js view of what type of devices are in use in the area round my flat.

The view dynamically updates every 5 seconds to pick up the latest information.

Now I know who owns what type of device, time to see who might know who. By plotting a force directed graph of all the clients detected and linking them based on the networks they have been searching for I can build up a view of which devices may belong to people who know each other.

Force directed network graph

There are a couple of clusters in the data so far, but most of them are from public WIFI networks like BTOpenzone and O2 Wifi. After filtering these services out there was still the 3 devices that look to be using Mike’s Lumina 800 for internet access and 4 devices connected to the same Sky Broadband router. I expect the data to be a lot more interesting when I get to run it somewhere with a few more people.

At the moment this is all running on my laptop, but it should run fine on my raspberry pi or my home server, as soon as I’ve transferred it over I’ll put a link up to live version of the charts.

d3 MQTT Tree visualiser updated

I’ve been having a bit of a play updating my d3 based MQTT topic tree visualiser this weekend.

  1. I’ve been trying to tidy things up a bit and break the code out into a little standalone library and only leave the MQTT connection code in the index.html.
  2. I’ve been improving the handling of long payload. There is now a nicer popup tooltip with a longer version of the payload, it’s still cropped at 100 chars but I’m still working on wrapping it and showing a few lines to get more in.
  3. I’ve been moving the MQTT connection code over to use binary WebSocket connections* rather than the JSON based proxy I was using before. The version hosted here is still using the proxy for now, but the binary versions work and I’ll move it over once I’ve finished playing with my internal broker setup.

I still need to try and make the whole thing resize or at least not have hard coded dimensions, but that might have to wait for next time.

I’ve stuck the updated version up here, I’ll stick the code up on github once I get sign off from the boss.

https://github.com/hardillb/d3-MQTT-Topic-Tree

* There is a new IBM WebSphere MQ feature pack supporting this, details can be found here

d3 MQTT topic tree visualiser

As part of writing my talks for the developerWorks Days Zurich conference next week I’ve been working on some code for the demo in the MQTT session. I was looking for a nice way to display a MQTT topic structure and the most recent value posted to topic.

In the past I have used a plugin to the freemind mindmap tool that plots the topics as and their values. This worked well but it does require a machine with Java and a very old version of freemind to make it work so it was time to see if there was another approach.

Nick O’Leary had created a neat little python MQTT to Websockets* bridge that we have been using round the office for a while and there are some really great javascript visualisation libraries out there now, so I thought I would see how hard it would be to run up a browser based version.

Starting with the d3 Collapsible Tree Layout example seamed like a good idea and it didn’t take too long to tweak it a little to make it dynamically add new topic branches when a message was received.

function addNode(topic, body) {
	var parts = topic.split("/");
	if (root.children[0]===undefined){
		var newnode = {"name": parts.shift(), "children":[]};
		root.children = [newnode];
		walk(parts,newnode,body);
	} else {
		walk(parts,root,body);
	}
	update(root);
}

function walk(parts, node, body) {
	if (parts.length != 0) {
		var current = parts.shift()
		if (node.children && node.children.length != 0) {
			var z=0;
			for(z=0; z < node.children.length; z++) {
				if (node.children[z].name == current) {
					walk(parts,node.children[z], body);
					break;
				}
			}
			if (z == node.children.length) {
				var newnode = {"name": current, "children":[]};
				node.children.push(newnode);
				walk(parts,node.children[z],body);
			}
		} else {
			var newnode = {"name": current, "children":[]};
			node.children = [newnode];
			walk(parts,node.children[0],body);
		}
	}
}

I also swapped out the where the data was loaded to use a static root rather than loading it from a json file on the server.

var root = {"name": "/","children": []};
root.x0 = h / 2;
root.y0 = 0;
update(root);

This worked pretty well so next I set about looking to see if I could add the payload of the message. To do this I added a data field to the json object for each node when populating the tree.

function walk(parts, node, body) {
	if (parts.length != 0) {
		var current = parts.shift()
		if (node.children && node.children.length != 0) {
			var z=0;
			for(z=0; z < node.children.length; z++) {
				if (node.children[z].name == current) {
					walk(parts,node.children[z], body);
					break;
				}
			}
			if (z == node.children.length) {
				var newnode = {"name": current, "children":[]};
				node.children.push(newnode);
				walk(parts,node.children[z],body);
			}
		} else {
			var newnode = {"name": current, "children":[]};
			node.children = [newnode];
			walk(parts,node.children[0],body);
		}
	} else {
		console.log("body");
		node.data = body;
	}
}

Then updated where the SVG is inserted to add a text element in the right place.

...
  nodeEnter.append("svg:text")
      .attr("x", function(d) { return d.children || d._children ? -10 : 10; })
      .attr("dy", ".35em")
      .attr("text-anchor", function(d) { return d.children || d._children ? "end" : "start"; })
      .text(function(d) { return d.name; })
      .style("fill-opacity", 1e-6);
  
  // Add message payload
  nodeEnter.append("svg:text")
  	.attr("class","data")
	.attr("x", 10)
	.attr("y", 10)
	.attr("dy", ".35em")
	.text(function(d) { return d.data || ""; })
	.style("fill-opacity", 1);
  
  // Transition nodes to their new position.
  var nodeUpdate = node.transition()
      .duration(duration)
      .attr("transform", function(d) { return "translate(" + d.y + "," + d.x + ")"; });
...

Again this worked very well, except that it would not update the payload value after adding the first version. I tried a number of things to get this to work, but didn’t get very far, until James Thomas pointed me at this page about how some of the internals of d3 work. It talks about entry, update and exit methods to work on new, existing and removed elements in the data. There are examples of entry and exit functions in the example so I tried adding an update method but kept getting errors that it wasn’t found. I went back and had another look at the examples in the page James had linked to and noticed that update code doesn’t actually use a functions.

...
// Add message payload
  nodeEnter.append("svg:text")
  	.attr("class","data")
	.attr("x", 10)
	.attr("y", 10)
	.attr("dy", ".35em")
	.text(function(d) { return d.data || ""; })
	.style("fill-opacity", 1);

  // Update payload if changed  
  var nodeChange = node.selectAll("text.data")
      .text(function(d) {return d.data || "";});

  // Transition nodes to their new position.
  var nodeUpdate = node.transition()
      .duration(duration)
      .attr("transform", function(d) { return "translate(" + d.y + "," + d.x + ")"; });
...

An that was basically it. I’ve added a bit more code to update values when the tree branches are collapsed and to trim payloads to no more than 20 chars so the display is not too cluttered. I’m planning on improving the style sheet a little soon and possibly adding an effect so you see which payloads have been updated recently and maybe add hover text with the full payload for the shortened ones.

You can see the whole thing running here

* There is also some work being done to support Websockets with the Mosquitto MQTT broker, you can see the testing page here