An interesting question came up on Stack Overflow recently that I suggested a hypothetical answer for how to do hostname based proxying for MQTT.
In this post I’ll explore how to actually implement that hypothetical solution.
History
HTTP added the ability to do hostname based proxying when it instroduced the Host header in HTTP v1.1. This meant that a single IP address could be used for many sites and the server would decide which content to serve based on the his header. Front end reverse proxies (e.g. nginx) can use the same header to decide which backend server to forward the traffic to.
This works well until we need encrypt the traffic to the HTTP server using SSL/TLS where the headers are encrypted. The solution to this is to use the SNI header in the TLS handshake, this tells the server which hostname the client is trying to connect to. The front end proxy can then either use this information to find the right local copy of the certificate/key for that site if it’s terminating the encryption at the frontend or it can forward the whole connection directly to the correct backend server.
MQTT
Since the SNI header is in the initial TLS handshake and is nothing to do with the underlying protocol it can be used for ay protocol, in this case MQTT. This measn if we set up a frontend proxy that uses SNI to pick the correct backend server to connect to.
Here is a nginx configuration file that proxies for 2 different MQTT brokers based on the hostname the client uses to connect. It is doing the TLS termination at the proxy before forwarding the clear version to the backend.
Assuming the the DNS entries for test1.example.com and test2.example.com both point to the host running nginx then we can test this with the mosquitto_sub command as follows
$ mosquitto_sub -v -h test1.example.com -t test --cafile ./ca-certs.pem
This will be proxied to the broker running on 192.168.1.1, where as
$ mosquitto_sub -v -h test2.example.com -t test --cafile ./ca-certs.pem
will be proxied to the broker on 192.168.1.2.
Cavets
The main drawback with this approach is that it requires that all the clients connect using TLS, but this is not a huge problem as nearly all devices are capable of this now and for any internet facing service it should be the default anyway.
Acknowledgment
How to do this was mainly informed by the following Gist
I’ve been meaning to get round to this ever since the Pi Supply Kickstarter delivered my LoRa Gateway HAT and the LoRa Node pHAT.
They have been sat in their boxes waiting until I had some spare time (and I’d finally finished moving a few things around to free up a spare Pi).
LoRa is a long range, low bandwidth radio system that uses the unlicensed spectrum. When combined with the higher level LoRaWAN protocol it makes great IoT platform for low power devices that want to send low volumes of data in places where there is no WiFi coverage and can’t justify the cost of a cellular connection.
LoRaWAN allows you to deploy a collection of Gateway devices that can act as receivers for a large number of deployed devices. These gateways then forward on messages to central point for processing.
The Things Network
A group called The Things Network run a LoRaWAN deployment. They are aiming for as large a coverage area as possible. To do this they allow users to deploy their own gateways and join these to the network. By joining the network you get to use everybody elses gateways in exchange for letting other people use yours.
Setting up the Gateway
This was particularly easy. I just had to download an image and flash it to a SD card. Stick that into the pi along with an ethernet cable and some power.
After the pi boots up you point your browser at http://iotloragateway.local and fill in a couple of values generated when I registered the gateway on the TTN site and that was it. The gateway is now up and running and ready to send/receive packets from any devices in range.
Testing
In order to test the gateway I need to set up a Pi Zero with the LoRa Node pHAT. This was a little trickier, but not much.
Fist I had to disable the Linux serial console, this can be done using the raspi-config command. I also had to add dtoverlay=pi3-miniuart-bt /boot/config.txt.
That was all that was needed to get the hardware configured, as for the software there is a rak811 python package that supplies the api and utilities to work with pHAT.
I now needed to declare an application on The Things Network site, this is how messages get routed to be processes. Taking the values for this application I could now write the following helloWorld.py
First up will be to get a better antenna for the gateway and to move the whole things up in the attic, from there it should get a good view north out towards the River Severn. After that I want to get a small battery powered LoRa/GPS board, like a TTGO T-Beam and ride round on my bike to get a feel for what the range/coverage actually is.
I’ll also be keeping an eye on the stats from the gateway to see if anybody else near by is deploying TTN LoRaWAN devices.
I’ve been looking for a project for a bunch of ESP-8266 ESP-01 boards I’ve had kicking around for a while.
The following describes a simple button that when pushed publishes a MQTT message that I can subscribe to with Node-RED to control different tasks.
It’s been done many times before, but I wanted to have a go at building my own IoT button.
Software
The code is pretty simple:
The MQTT PubSubClient (Thank’s Nick)
Some hard coded WiFi and MQTT Broker details
The setup function which connects to the network
The reconnect function that connects to the MQTT broker and publishes the message
The loop function which flashes the LED to show success then go into deep sleep
In order to get the best battery life you want the ESP8266 to be in deep sleep mode for as much as possible, so for this reason the loop function sends the message to signify the button has been pushed then indefinitely enters the deepest sleep state possible. This means the loop function will only run once on start up.
By using a momentary push button wake the ESP-01 I bridged reset pin to ground the chip resets each time it’s pushed, this wakes it from the deep sleep state and runs the all the code, then drops back into the sleep state.
The green line is the Chip enable
The blue line is the the links reset to ground via the push button
Now I had the basic circuit and code working I needed to pick a power supply. The ESP-01 needs a 3.3v supply and most people seem to opt for using a small LiPo cell. A single cell has a nominal fully charged voltage of 3.7v which is probably close enough to use directly with the ESP-01, but the problem is you normally need to add a circuit to cut them out before the voltage gets too low as they discharge to prevent permanently damage them. You would normally add a charging circuit to allow recharging from a USB source.
This wasn’t what I was looking for, I wanted to use off the shelf batteries so I went looking for a solution using AAA batteries. The board will run directly from a fresh set of 2 AAA batteries, but the voltage can quickly drop too low. To help with this I found a part from Pololu that would take an input between 0.5v and 5v and generate a constant 3.3v. This meant that even as the batteries discharged I should be able to continue to run the device.
At first things didn’t look to work, because converter was not supplying enough current for the ESP-01 at start up, to get round this I added a 100uF capacitor across the outputs of the regulator. I don’t really know how properly to size this capacitor so I basically made a guess.
The final step was to use the soldering iron to remove the red power LED from the board as this was consuming way more power than the rest of the system.
Next Steps
Make the MQTT topic based on the unique id of the ESP-01 so it isn’t hard coded
Look at adding Access Point mode to allow configuration of the WiFi details if the device can not connect to the existing configuration
Design a circuit board to hold the voltage converter, capacitor, button and the ESP-01
Create a case to hold it all
Work out just how long a set of batteries will last
I’ve been working on integrating the new IKEA TRÅDFRI Lights into my Home Automation system. I’d really like a native NodeJS system so I can plug it directly into Node-RED, but I’ve not found a working CoAP over DTLS setup just yet.
So in the mean time I’ve been working on a very basic MQTT to CoAP client bridge in Java using the Eclipse Californium library.
It still needs some work, but here is the first pass:
package uk.me.hardill.coap2mqtt;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.HashMap;
import java.util.logging.Level;
import org.eclipse.californium.core.CaliforniumLogger;
import org.eclipse.californium.core.CoapClient;
import org.eclipse.californium.core.CoapHandler;
import org.eclipse.californium.core.CoapResponse;
import org.eclipse.californium.core.Utils;
import org.eclipse.californium.core.coap.MediaTypeRegistry;
import org.eclipse.californium.core.network.CoapEndpoint;
import org.eclipse.californium.core.network.config.NetworkConfig;
import org.eclipse.californium.scandium.DTLSConnector;
import org.eclipse.californium.scandium.ScandiumLogger;
import org.eclipse.californium.scandium.config.DtlsConnectorConfig;
import org.eclipse.californium.scandium.dtls.pskstore.StaticPskStore;
import org.eclipse.paho.client.mqttv3.IMqttDeliveryToken;
import org.eclipse.paho.client.mqttv3.MqttCallback;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence;
import org.json.JSONArray;
import org.json.JSONObject;
/**
* @author hardillb
*
*/
public class Main {
static {
CaliforniumLogger.disableLogging();
ScandiumLogger.disable();
// ScandiumLogger.initialize();
// ScandiumLogger.setLevel(Level.FINE);
}
private DTLSConnector dtlsConnector;
private MqttClient mqttClient;
private CoapEndpoint endPoint;
private String ip;
private HashMap<String, Integer> name2id = new HashMap<>();
Main(String psk, String ip) {
this.ip = ip;
DtlsConnectorConfig.Builder builder = new DtlsConnectorConfig.Builder(new InetSocketAddress(0));
builder.setPskStore(new StaticPskStore("", psk.getBytes()));
dtlsConnector = new DTLSConnector(builder.build());
endPoint = new CoapEndpoint(dtlsConnector, NetworkConfig.getStandard());
MemoryPersistence persistence = new MemoryPersistence();
try {
mqttClient = new MqttClient("tcp://localhost", MqttClient.generateClientId(), persistence);
mqttClient.connect();
mqttClient.setCallback(new MqttCallback() {
@Override
public void messageArrived(String topic, MqttMessage message) throws Exception {
// TODO Auto-generated method stub
System.out.println(topic + " " + message.toString());
String parts[] = topic.split("/");
int id = name2id.get(parts[1]);
System.out.println(id);
String command = parts[3];
System.out.println(command);
try{
JSONObject json = new JSONObject("{\"9001\":\"Living Room Light\",\"9020\":1491515804,\"9002\":1491158817,\"9003\":65537,\"9054\":0,\"5750\":2,\"3\":{\"0\":\"IKEA of Sweden\",\"1\":\"TRADFRI bulb E27 opal 1000lm\",\"2\":\"\",\"3\":\"1.1.1.0-5.7.2.0\",\"6\":1},\"9019\":1,\"3311\":[{\"5850\":1,\"5851\":10,\"9003\":0}]}");
JSONObject settings = json.getJSONArray("3311").getJSONObject(0);
if (command.equals("dim")) {
settings.put("5851", Integer.parseInt(message.toString()));
} else if (command.equals("on")) {
if (message.toString().equals("0")) {
settings.put("5850", 0);
settings.put("5851", 0);
} else {
settings.put("5850", 1);
settings.put("5851", 128);
}
}
String payload = json.toString();
Main.this.set("coaps://" + ip + "//15001/" + id, payload);
} catch (Exception e) {
e.printStackTrace();
}
}
@Override
public void deliveryComplete(IMqttDeliveryToken token) {
// TODO Auto-generated method stub
}
@Override
public void connectionLost(Throwable cause) {
// TODO Auto-generated method stub
}
});
mqttClient.subscribe("TRÅDFRI/+/control/+");
} catch (MqttException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private void discover() {
try {
URI uri = new URI("coaps://" + ip + "//15001");
CoapClient client = new CoapClient(uri);
client.setEndpoint(endPoint);
CoapResponse response = client.get();
JSONArray array = new JSONArray(response.getResponseText());
for (int i=0; i<array.length(); i++) {
String devUri = "coaps://"+ ip + "//15001/" + array.getInt(i);
this.watch(devUri);
}
client.shutdown();
} catch (URISyntaxException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private void set(String uriString, String payload) {
System.out.println("payload\n" + payload);
try {
URI uri = new URI(uriString);
CoapClient client = new CoapClient(uri);
client.setEndpoint(endPoint);
CoapResponse response = client.put(payload, MediaTypeRegistry.TEXT_PLAIN);
if (response.isSuccess()) {
System.out.println("Yay");
} else {
System.out.println("Boo");
}
client.shutdown();
} catch (URISyntaxException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private void watch(String uriString) {
try {
URI uri = new URI(uriString);
CoapClient client = new CoapClient(uri);
client.setEndpoint(endPoint);
CoapHandler handler = new CoapHandler() {
@Override
public void onLoad(CoapResponse response) {
System.out.println(response.getResponseText());
JSONObject json = new JSONObject(response.getResponseText());
if (json.has("3311")){
MqttMessage message = new MqttMessage();
int state = json.getJSONArray("3311").getJSONObject(0).getInt("5850");
message.setPayload(Integer.toString(state).getBytes());
message.setRetained(true);
String topic = "TRÅDFRI/" + json.getString("9001") + "/state/on";
String topic2 = "TRÅDFRI/" + json.getString("9001") + "/state/dim";
name2id.put(json.getString("9001"), json.getInt("9003"));
MqttMessage message2 = new MqttMessage();
int dim = json.getJSONArray("3311").getJSONObject(0).getInt("5851");
message2.setPayload(Integer.toString(dim).getBytes());
message2.setRetained(true);
try {
mqttClient.publish(topic, message);
mqttClient.publish(topic2, message2);
} catch (MqttException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} else {
System.out.println("not bulb");
}
}
@Override
public void onError() {
// TODO Auto-generated method stub
}
};
client.observe(handler);
} catch (URISyntaxException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
/**
* @param args
*/
public static void main(String[] args) throws InterruptedException {
String psk = args[0];
String ip = args[1];
Main m = new Main(psk, ip);
m.discover();
}
}
I’ve tagged the code onto the gist as well for now, but I’ll check the whole thing in as a separate project soon.
I recently got asked to help out building a demo for one of the projects the team has been working on.
The project uses situational awareness to vary the level of authentication needed to carry out different tasks. Rather than make the person using the demo run all over the place to change location or log in and out of various systems we have stubbed out the inputs and needed a way to toggle them locally. The demo runs on a tablet so we wanted something that could sit near by and allow the inputs to be varied.
To do this I thought I’d have a play with a Raspberry Pi and the GPIO pins. The plan was to attach 5 buttons to the toggle the state of the inputs, the buttons have lights that will show the state.
This would have been a perfect use case for the new Raspberry Pi Zero as it only needs 1 USB port and the GPIO pins, but I couldn’t convince any of the guys on the team lucky enough to grab one to give it up.
I’d uses something very similar in the past for the MQTT Powered Video Walls in IBM Southbank Client Centre.
The first single button prototype worked nicely after I’d worked out what was needed with pull-up resistors for the buttons and current limiting resistors for the LEDs.
Node-RED was set up to read the state of the buttons from the GPIO pints and to update toggle the LED output the same way.
Node-RED flow to read buttons
To make the demo totally standalone the rPi acts as a WiFi access point that the tablet connects to. It uses this connection to load the demo and to read the updates from the buttons via MQTT over Websockets.
The USB WiFi adapter available was a Edimax EW-7811UN, The default Linux driver for this doesn’t support HostAPD, luckily there is a page with instructions and a github project with a functional version.
Mosquitto 1.4.2 was installed to act as the the MQTT broker and to support MQTT over websockets so the web app could connect directly to get the button updates.
Having finished off the wiring and configuring the rPi the whole thing was mounted in a display board put together by Andy Feltham.
While playing with my Moo.comsticker engine I’ve added a new option, this one will create a single 90 sticker book with 45 MQTT stickers and 45 Node-RED stickers.
To have this set added to your moo cart click here.
A few years ago I create a web app to allow people to order their own sets of Moo stickers with the MQTT, Node-RED and Owntracks(at the time called MQTTitude) logos.
These worked well with people ordering about 1 pack a month until Moo changed the way they did authentication. I didn’t have time to fix it until today.
I’ve moved the app from being a J2EE app over to one written in NodeJS using express, this along with the change in Moo’s authentication method has made the code a lot shorter and easier to read. I’ve published the new code on github here.
To order a set of stickers click on the appropriate image:
MQTT
Node-RED
Owntracks