Listing AWS Lambda Runtimes

For the last few weeks I’ve been getting emails from AWS about Node 6.10 going end of life and saying I have deployed Lambda using this level.

The emails don’t list which Lambda or which region they think are at fault which makes tracking down the culprit difficult. I only really have 1 live instance deployed across multiple regions (and 1 test instance on a single region).

AWS Lambda region list

Clicking down the list of regions is time consuming and prone to mistakes.

In the email AWS do provide a command to list which Lambda are running with Node 6.10:

aws lambda list-functions --query="Functions[?Runtime=='nodejs6.10']"

But what they fail to mention is that this only checks your current default region. I can’t find a way to get the aws command line tool to list the Lambda regions, the closest I’ve found is the list of ec2 regions which hopefully match up. Pairing this with the command line JSON search tool jq and a bit of Bash scripting I’ve come up with the following:

for r in `aws ec2 describe-regions --output text | cut -f3`; 
do
  echo $region;
  aws --region $r lambda list-functions | jq '.[] | .FunctionName + " - " + .Runtime';
done

This walks over all the regions as prints out all the function names and the runtime they are using.

eu-north-1
ap-south-1
eu-west-3
eu-west-2
eu-west-1
"Node-RED - nodejs8.10"
"oAuth-test - nodejs8.10"
ap-northeast-2
ap-northeast-1
sa-east-1
ca-central-1
ap-southeast-1
ap-southeast-2
eu-central-1
us-east-1
"Node-RED - nodejs8.10"
us-east-2
us-west-1
us-west-2
"Node-RED - nodejs8.10"

In my case it only lists NodeJS 8.10 so I have no idea why AWS keep sending me these emails. Also since I’m only on the basic level I can’t even raise a technical help desk query to find out.

Anyway I hope this might be useful to others with the same problem.

DNS-over-HTTPS update

My post on DNS-over-HTTPS from last year is getting a fair bit more traffic after a few UK news paper articles (mainly crying that the new UK Government  censoring won’t work if Google roll it out in Chrome… what a shame). The followning article has a good overview [nakedsecurity].

Anyway I tweeted a link to the old post and it started a bit of a discussion and the  question about the other side of system came up. Namely how to use a DNS resolver that pushed traffic over DNS-over-HTTPS rather than provide a HTTPS endpoint that supported queries. The idea being that at the moment only Firefox & Chrome can take advantage of the secure lookups.

I did a bit of poking around and found things like stubby which DNS-over-TLS (another approach to secure DNS lookups) and also Cloudflare have cloudflared which can proxy for DNS-over-HTTPS to Cloudflare’s DNS server (it also is used to set up the VPN tunnel to Cloudflare’s Argo service, which is also worth a good look at.)

Anyway, while there are existing solutions out there I thought I’d have a really quick go at writing my own, to go with the part I’d written last year, just to see how hard it could be.

It turned out a really basic first pass could be done in about 40 lines of Javascript:

const dgram = require('dgram')
const request = require('request')
const dnsPacket = require('dns-packet')

const port = process.env["DNS_PORT"] || 53
//https://cloudflare-dns.com/dns-query
const url = process.env["DNS_URL"] 
    || "https://dns.google.com/experimental" 
const allow_selfSigned = 
    (process.env["DNS_INSECURE"] == 1) 

const server = dgram.createSocket('udp6')

server.on('listening', function(){
  console.log("listening")
})

server.on('message', function(msg, remote){
  var packet = dnsPacket.decode(msg)
  var id = packet.id
  var options = {
    url: url,
    method: 'POST',
    body: msg,
    encoding: null,
    rejectUnauthorized: allow_selfSigned ? false : true,
    headers: {
      'Accept': 'application/dns-message',
      'Content-Type': 'application/dns-message'
    }
  }

  request(options, function(err, resp, body){
    if (!err && resp.statusCode == 200) {
      var respPacket = dnsPacket.decode(body)
      respPacket.id = id
      server.send(body,remote.port)
    } else {
      console.log(err)
    }
  })

})

server.bind(port)

It really could do with some caching and some more error handling and I’d like to add support for Google JSON based lookups as well as the binary DNS format, but I’m going to add it to the github project with the other half and people can help extend it if they want.

The hardest part was working out I needed the encoding: null in the request options to stop it trying to turn the binary response into a string but leaving it as a Buffer.

I’m in the process of migrating my DNS setup to a new machine, I’ll be adding a DNS-over-TLS (using stunnel) & a DNS-over-HTTPS listeners for the public facing sides.

Building a Bluetooth speaker

Recently I’ve been playing with how to build a Bluetooth audio device using a Raspberry Pi Zero. The following are some notes on what I found.

First question is why build one when you can buy one for way less than the cost of the parts. There are a couple of reasons:

  • I build IoT prototypes for a living, and the best way to get a feel for the challenges is to actually face them.
  • Hacking on stuff is fun.

The Hardware

I’m starting out with a standard Raspberry Pi Zero W. This gets me a base high level platform that includes a WiFi and Bluetooth.

Raspberry Pi Zero W

The one thing that’s missing is an audio output (apart from the HDMI) but Raspberry Pi’s support audio using the I2S standard. There are several I2S pHATs available and I’m going to be using a pHAT DAC from Pimoroni. I’ve used these before for a project so I’m reasonably happy with how to set it up, but Pimoroni have detailed instructions.

I’m going to add a screen to show things like the current track title & artist along with the volume. I’m also going to need some buttons to send Play/Pause, Next & Previous commands to the connected device. I have a PaPiRus e-ink display that has 5 buttons built in which I was going to use but this clashes with the GPIO pins used for the DAC so instead I’ve opted for the Inky pHAT and the Button Shim.

The Software

I knew the core components of this had to be a problem others had solved and this proved to be the case. After a little bit of searching I found this project on github.

As part of the configuration we need to generate the Bluetooth Class bitmask. This can be done one this site.


Class options

This outputs a hex value of 0x24043C which is added to the /etc/bluetooth/main.conf

With this up and running I had a basic Bluetooth speaker that any phone can connect to without a pin and play music, but nothing else. The next step is to add some code to handle the button pushes and to update the display.

The Bluetooth stack on Linux is controlled and configured using DBus. Dbus is a messaging system supporting IPC and RPC

A bit of Googling round turned up this askubuntu question that got me started with the following command:

dbus-send --system --print-reply --dest=org.bluez /org/bluez/hci0/dev_44_78_3E_85_9D_6F org.bluez.MediaControl1.Play

This sends a Play command to the connected phone with the Bluetooth mac address of 44:78:3E:85:9D:6F. The problem is knowing what the mac address is as the system allows multiple devices to pair with the speaker. Luckily you can use DBus to query the system for the connected device. DBus also has some really good Python bindings. So with a bit more poking around I ended up with this:

#!/usr/bin/env python
import signal
import buttonshim
import dbus
bus = dbus.SystemBus()
manager = dbus.Interface(
bus.get_object("org.bluez","/"), 
"org.freedesktop.DBus.ObjectManager")

@buttonshim.on_press(buttonshim.BUTTON_A)
def playPause(button, pressed):
	objects = manager.GetManagedObjects()
	 
	for path in objects.keys():
	    interfaces = objects[path]
	    for interface in interfaces.keys():
	        if interface in [
	        "orge.freedesktop.DBus.Introspectable",
	        "org.freedesktop.DBus.Properties"]:
	            continue
	 
	        if interface == "org.bluez.Device1":
	            props = interfaces[interface]
	            if props["Connected"] == 1:
	                media = objects[path + "/player0"]["org.bluez.MediaPlayer1"]
	 
	                mediaControlInterface = dbus.Interface(
	                bus.get_object("org.bluez",path + "/player0"),
	                "org.bluez.MediaPlayer1")
	 
	                if media["Status"] == "paused":
	                    mediaControlInterface.Play()
	                else:
	                    mediaControlInterface.Pause()

signal.pause()

When button A is pressed this looks up the connected device, and also checks the current state of the player, is it playing or paused and toggles the state. This means that one button can be Play and Pause. It also uses the org.bluez.MediaPlay1 API rather than the org.bluez.MediaControl1 which is marked as deprecated in the doc.

The button shim also comes with Python bindings so putting it all together was pretty simple.

DBus also lets you register to be notified when a property changes, this paired with the Track property on the org.bluez.MediaPlay1 as this holds the Artist, Track Name, Album Name and Track length information supplied by the source. This can be combined with the Inky pHAT python library to show the information on the screen.

#!/usr/bin/env python

import dbus
from dbus.mainloop.glib import DBusGMainLoop
from gi.repository import GLib

def trackChanged(*args, **kw):
	target = args[0]
	if target == "org.bluez.MediaPlayer1":
		data = args[1].get("Track",0)
		if data != 0:
			artist = data.get('Artist')
			track = data.get('Title')
			print(artist)
			print(track)


DBusGMainLoop(set_as_default=True)
system_bus = dbus.SystemBus()
system_bus.add_signal_receiver(trackChanged, 
	dbus_interface="org.freedesktop.DBus.Properties", 
	signal_name="PropertiesChanged", 
	path='/org/bluez/hci0/dev_80_5A_04_12_03_0E/player0')
loop = GLib.MainLoop()
loop.run()

This code attaches a listener to the MediaPlayer object and when it spots that the Track has changed it prints out the new Artist and Title. The code matches all PropertiesChanged events which is a little messy but I’ve not found a way to use wildcards or partial matches for the DBus interface in python (since we don’t know the mac address of the connected device at the time we start listening for changes).

Converting the Artist/Title information into an image with the Pyton Image Library then getting the Inky pHAT to render that is not too tricky

from PIL import Image, ImageDraw, ImageFont
from font_fredoka_one import FredokaOne
from inky import InkyPHAT

...

disp = InkyPHAT("yellow")
font = ImageFont.truetype(FredokaOne, 22)

img = Image.new("P", (inky_display.WIDTH, inky_display.HEIGHT))
draw = ImageDraw.Draw(img)

draw.text((), "Artist: "+ artist, disp.WHITE, font=font)
draw.text((), "Track: "+ track, disp.WHITE, font=font)

disp.set_image(img)
disp.show()


That’s the basics working, now I need to find/build a case for it and then look at seeing if I can add Chromecast Audio and Airplay support.

Node-RED Google Home Smart Home Action

Google Home

Following on from my Alexa Home Skill for Node-RED it’s time to see about showing some love to the Google Home users (OK, I’ve been slowly chipping away at this for ages, but I’ve finally found a bit of time).

One of the nice things about Google Assistant is that it works all over the place, I can use it via the text interface if I’m somewhere and can’t talk, or even from the car via Android Auto.

Screenshot_20190101-170716

Google offer a pretty similar API for controlling Smart home devices to the one offered by Amazon for the Alexa so the implementation of this was very similar. The biggest difference is the is no requirement to use something like Amazon’s Lambda to interface with the service so it’s just a single web endpoint.

I’ve taken pretty much the same approach as with the Alexa version in that I have a Web Site where you can sign up for an account and then define virtual devices with specific names and characteristics.

Virtual devices

Google support a lot more different types of devices and characteristics than Amazon with Alexa at the moment, but to start with I’m just supporting Sockets/Light/Switches and Thermostats. I intend to add more later as I work out the best way to surface the data.

The other big change is that Google Assistant supports asynchronously updating the device state and the ability for the Assistant backend to query the state of a device. To support this I’m going to allow the response node to be configured with a specific device and to accept input that has not come from an input node.

The node is currently being beta tested, if you are interested post in #google-home-assistant on the Node-RED Slack and I can add you to the ACL for the beta.

Google Assistant Node-RED Node

I’ll do another post when the node has finished testing and has been accepted by Google.

DIY IoT button

I’ve been looking for a project for a bunch of ESP-8266 ESP-01 boards I’ve had kicking around for a while.

The following describes a simple button that when pushed publishes a MQTT message that I can subscribe to with Node-RED to control different tasks.

It’s been done many times before, but I wanted to have a go at building my own IoT button.

Software

The code is pretty simple:

  • The MQTT PubSubClient (Thank’s Nick)
  • Some hard coded WiFi and MQTT Broker details
  • The setup function which connects to the network
  • The reconnect function that connects to the MQTT broker and publishes the message
  • The loop function which flashes the LED to show success then go into deep sleep

In order to get the best battery life you want the ESP8266 to be in deep sleep mode for as much as possible, so for this reason the loop function sends the message to signify the button has been pushed then indefinitely enters the deepest sleep state possible. This means the loop function will only run once on start up.

#include <ESP8266WiFi.h>
#include <PubSubClient.h>

#define ESP8266_LED 1

const char* ssid = "WifiName";
const char* passwd = "GoodPassword";
const char* broker = "192.168.1.114";

WiFiClient espClient;
PubSubClient client(espClient);

void setup() {

  Serial.begin(115200);
  delay(10);
  
  pinMode(ESP8266_LED, OUTPUT);

  WiFi.hostname("Button1");
  WiFi.begin(ssid, passwd);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");  
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
  client.setServer(broker, 1883);
  reconnect();
}

void reconnect() {
  while(!client.connected()) {
    if (client.connect("button1")){
      client.publish("button1", "press");
    } else {
      delay(5000);
    }
  }
}

void loop() {

  if (client.connected()) {
    reconnect();
  }
  client.loop();
  
  // put your main code here, to run repeatedly:
  digitalWrite(ESP8266_LED, HIGH);
  delay(1000);
  digitalWrite(ESP8266_LED, LOW);
  delay(1000);
  ESP.deepSleep(0);
}

Hardware

By using a momentary push button wake the ESP-01 I bridged reset pin to ground the chip resets each time it’s pushed, this wakes it from the deep sleep state and runs the all the code, then drops back into the sleep state.

Button diagram

  • The green line is the Chip enable
  • The blue line is the the links reset to ground via the push button

Now I had the basic circuit and code working I needed to pick a power supply. The ESP-01 needs a 3.3v supply and most people seem to opt for using a small LiPo cell. A single cell has a nominal fully charged voltage of 3.7v which is probably close enough to use directly with the ESP-01, but the problem is you normally need to add a circuit to cut them out before the voltage gets too low as they discharge to prevent permanently damage them. You would normally add a charging circuit to allow recharging from a USB source.

This wasn’t what I was looking for, I wanted to use off the shelf batteries so I went looking for a solution using AAA batteries. The board will run directly from a fresh set of 2 AAA batteries, but the voltage can quickly drop too low. To help with this I found a part from Pololu that would take an input between 0.5v and 5v and generate a constant 3.3v. This meant that even as the batteries discharged I should be able to continue to run the device.

At first things didn’t look to work, because converter was not supplying enough current for the ESP-01 at start up, to get round this I added a 100uF capacitor across the outputs of the regulator. I don’t really know how properly to size this capacitor so I basically made a guess.

The final step was to use the soldering iron to remove the red power LED from the board as this was consuming way more power than the rest of the system.

Prototype

Next Steps

  • Make the MQTT topic based on the unique id of the ESP-01 so it isn’t hard coded
  • Look at adding Access Point mode to allow configuration of the WiFi details if the device can not connect to the existing configuration
  • Design a circuit board to hold the voltage converter, capacitor, button and the ESP-01
  • Create a case to hold it all
  • Work out just how long a set of batteries will last

SIP to SMS bridging

I’ve recently updated my local Asterisk PBX. The main reason for the update was that processing the logs in order to set up the firewall rules to block the folk that hammer on it all day long trying to make long distance calls or run up big bills on premium rate numbers was getting too much for the original Mk i Raspberry Pi B (it now runs on a Pi 3 b+ which more up to the task).

As part of the set up I have 3G USB dongle that also supports voice calls using the chan_dongle plugin. This plugin also supports sending and receiving SMS messages (this is different from the MMS/SMS gateway I also run on a separate pi) and they are handled by the dial plan.

My first pass at the dial plan just publishes the incoming message to a MQTT topic and it is then processed by Node-RED, which emails a copy to me as well as logging it to a file.

[dongle-incoming]
exten => sms,1,Verbose(1,Incoming SMS from ${CALLERID(num)} ${BASE64_DECODE(${SMS_BASE64})})
exten => sms,n,AGI(/opt/asterisk/agi-mqtt/mqtt,/opt/asterisk/agi-mqtt/mqtt.cfg,foo/sms,${BASE64_DECODE(${SMS_BASE64})})
exten => sms,n,Hangup()

This works OK for incoming messages but sending them is a bit harder, as the only way to send them from outside the dialplan (from within the dialpan you can use DongleSendSMS) is to use the asterisk CLI tool which is a bit clunky.

What I was really looking for was a way to send/receive SMS messages like you do with a mobile phone. To make calls I normally use Linphone on my tablet and this includes support for SIP messaging. This lets you send messages between SIP clients and you can get Asterisk to consume these.

You can send SIP messages with the MessageSend asterisk dailplan function

The following is a basic echo bot:

[messaging]
exten => _.,1,Answer()
exten => _.,n,Verbose(1,Text ${MESSAGE(body)})
exten => _.,n,Verbose(1,Text from ${MESSAGE(from)})
exten => _.,n,Verbose(1,Text to ${MESSAGE(to)})
exten => _.,n,Set(FROM=${MESSAGE(from)})
exten => _.,n,Set(TO=${REPLACE(FROM,<'>,)})
exten => _.,n,MessageSend(pj${TO},${CUT(MESSAGE(to),:,2)})
exten => _.,n,Hangup()

Because I’m using the PJSIP module rather than the legacy SIP module I need to prefix the outbound address with pjsip: rather than sip:. This also matches any target extension which will be useful a little later.

To enable a specific context for SIP messages you need to add message_context to the PJSIP endpoint for the SIP user:

[tablet]
type = endpoint
context = internal
message_context = messaging
...

Now if we put the 2 bits together we get a dialplan that looks like this:

[dongle-incomming]
exten => sms,1,Verbose(1,Incoming SMS from ${CALLERID(num)} ${BASE64_DECODE(${SMS_BASE64})})
exten => sms,n,AGI(/opt/asterisk/agi-mqtt/mqtt,/opt/asterisk/agi-mqtt/mqtt.cfg,foo/sms,${BASE64_DECODE(${SMS_BASE64})})
exten => sms,n,Set(MESSAGE(body))=${BASE64_DECODE(${SMS_BASE64})})
exten => sms,n,MessageSend(pjsip:nexus7,${CALLERID(num)})
exten => sms,n,Hangup()

[messaging]
exten => _.,1,Answer()
exten => _.,n,Verbose(1,Text ${MESSAGE(body)})
exten => _.,n,Verbose(1,Text from ${MESSAGE(from)})
exten => _.,n,Verbose(1,Text to ${MESSAGE(to)})
exten => _.,n,Set(FROM=${MESSAGE(from)})
exten => _.,n,Set(TO=${REPLACE(FROM,<'>,)})
exten => _.,n,DongleSendSMS(dongle0,${EXTEN},${MESSAGE(body)},1440,no)
exten => _.,n,Hangup()

The first part handles the incoming SMS messages delivered to the dongle and passed to the sms extension in the dongle-incomming context. This logs the message to the console and via MQTT then fires it off to my tablet as a SIP message. The second is the context for the incoming SIP messages from the tablet, this will accept messages to any extension, logs the message, who it’s to/from then sends it to the number in the extension via the dongle.

Using SIP Client to send SMS

DNS-Over-HTTPS

I saw the recent announcements from Mozilla, Cloudflare and Google about running a trial to try and make DNS name resolution more secure.

The basic problem is that most users get their DNS server set via DHCP which is controlled by who ever runs the network (at home this tends to be their ISP, but when using public wifi this could be anybody). The first approach to help with this was Google’s 8.8.8.8 public DNS service (followed by the IBM’s 9.9.9.9 and Cloudflares 1.1.1.1). This helps if people are technically literate enough know how to change their OS’s DNS settings and fix them to one of these providers. Also DNS is UDP based protocol which makes it particularly easy for a bad actor on the network to spoof responses.

The approach the 3 companies are taking is to run DNS over an existing secure protocol, in this case HTTPS. From Firefox version 60 (currently in beta) it is possible to set it up to do name host name resolution via DNS-Over-HTTPS.

There are currently 2 competing specifications for how to actually implement DNS-Over-HTTPS.

DNS Wireformat

This uses exactly the same data structure as existing DNS. Requests can be made via a HTTP GET or POST. For a POST the body is the binary request and the Content-Type is set to application/dns-udpwireformat.

For GET requests the payload is BASE64 encoded and passed as the dns query parameter.

In both cases the response is the same binary payload as would be made by a normal DNS server.

This approach is currently covered by this draft RFC

JSON

For this approach the request are made as a HTTP GET request with the hostname (or IP address) being passed as the name and the query type being passed as the type query parameters.

A response looks like this:

{
    "Status": 0,
    "RA": true,
    "RD": true,
    "TC": false,
    "AD": false,
    "CD": true,
    "Additional": [],
    "Answer": [
        {
            "TTL": 86400,
            "data": "93.184.216.34",
            "name": "example.com",
            "type": 1
        }
    ],
    "Question": [
        {
            "name": "example.com",
            "type": 1
        }
    ]
}

With a Content-Type of application/dns-json

You can find the spec for this scheme from Google here and Cloudflare here.

Both of these schemes have been implemented by both Google and Cloudflare and either can be used with Firefox 60+.

Privacy Fears

There has already been a bit of a backlash against this idea, mainly around privacy fears. The idea of Google/CloudFlare being able to collect information about all the hosts your browser resolves scared some people. Mozilla has an agreement in place with CloudFlare about data retention for the initial trial.

Given these fears I wondered if people might still want to play with DNS-Over-HTTPS but not want to share data with Google/Cloudflare. With this in mind I thought I’d try and see how easy it would be to implement a DNS-Over-HTTPS server. Also people may want to try this out on closed networks (for things like performance testing or security testing).

It turned out not to be too difficult, I started with a simple ExpressJS based HTTP server and then started to add DNS support. Initially I tried a couple of different DNS NodeJS nodes to get all the require details and in the end settled on dns-packet and actually sending my own UDP packets to the DNS server.

I’ve put my code up on github here if anybody wants a play. The README.md should include details about how to set up Firefox to use an instance.

Adding audio to the CCTV of your burgular

In a full on throw back to the 2nd ever post to this blog, back in February 2010, I’ve recently been updating the system that sends me a video when there is movement in my flat via MMS and email.

I thought I’d try and add audio to the video that gets sent. A quick Google turned up two options, one was to use the sox command and it’s silence option, the second uses the on_event_start triggers in motion as a way to record the audio at the same time as capturing the movement video. I went with the second option and tweaked it a bit to make it pick the right input for my system and to direct encode the audio to MP3 rather than WAV to save space.

on_event_start arecord --process-id-file /var/motion/arecord.pid -D sysdefault:CARD=AF -f S16_LE -r 22050 - | lame  -m m - /var/motion/%Y%m%d_%H%M%S.mp3

The other useful addition was the –process-id-file /var/motion/arecord.pid which writes the process id to a file so I can just use this to stop the recording rather than having to use grep and awk to find the process in the ps output.

on_event_end kill `cat /var/motion/arecord.pid`

Now it’s just a case of combining the video from motion with the audio. I can do this with ffmpeg when I re-encode the video into the 3gp container to make it suitable for sending via a MMS message.

ffmpeg -i movement.avi -i movement.mp3 -map 0:v -map 1:a -c:a aac -strict -2 -s qcif -c:v h263 -y /var/www/html/cam1/intruder.3gp

This seams to work but the output file is a little large. The default audio encoding seams to be at 160k bitrate, I wound it down to 32k and the file size got a lot better.

ffmpeg -i movement.avi -i movement.mp3 -map 0:v -map 1:a -c:a aac -b:a 32k -strict -2 -s qcif -c:v h263 -y /var/www/html/cam1/intruder.3gp

I’d like to try the AMR audio codec but I can’t get ffmpeg to encode with it at the moment, so I’m just going to email the mp3 of the audio along with the high res AVI version of the video and just send the low res 3GP version via MMS.

Multipart HTTP Post requests with Volley on Android

It’s been a little while since I’ve done any really serious Android development, but a couple of projects have brought me back to it.

Early on in one of those projects I had to make some HTTP requests, my first thought was to make use of the Apache HTTP Client classes as I had done many times before on Android. Which is why I was a little surprised when the usual ctrl-space didn’t auto complete any of the expected class names.

It turns out the classes were removed in Android 6.0 and the notice suggests using the HttpURLConnection class. A little bit more digging turned up a wrapper for this called Volley.

Volley is a wrapper round the HttpURLConnection class to provides a neat asynchronous interface that does IO in the background and then delivers results to the Main thread so UI updates can be done with out further faffing around switching threads. There is also a nice set of tutorials on the Android Developers pages.

The first few requests all worked fine, but there was one which was a little bit more tricky. The HTTP endpoint in question accepts a multipart-form payload. A bit of googling/searching on Stackoverflow turned up a number of approaches to this and best seamed to be documented in this gist

This was close to what I wanted but not quite what I needed so I have taken some of the core concepts and built my own MultipathRequest object.

...
MultipartRequest request = new MultipartRequest(url, headers, 
    new Response.Listener<NetworkResponse>() {
        @Override
        public void onResponse(NetworkResponse response) {
        ...
        }
    },
    new Response.ErrorListener() {
        @Override
        public void onErrorResponse(VolleyError error) {
        ...
        }
    });
    
request.addPart(new FormPart(fieldName,value));
request.addPart(new FilePart(fileFieldName, mimeType, fileName, data);

requestQueue.add(request);
...

I’ve stuck the code up on github here. You can include it in your Android Project by adding the following to the build.gradle in the root of the project:

allprojects {
  repositories {
    ...
    maven { url 'https://jitpack.io' }
  }
}

And then this to the dependencies section of the modules build.gradle:

dependencies {
  compile 'com.github.hardillb:MultiPartVolley:0.0.3'
}

Fist pass TRÅDFRI MQTT Bridge

I’ve been working on integrating the new IKEA TRÅDFRI Lights into my Home Automation system. I’d really like a native NodeJS system so I can plug it directly into Node-RED, but I’ve not found a working CoAP over DTLS setup just yet.

So in the mean time I’ve been working on a very basic MQTT to CoAP client bridge in Java using the Eclipse Californium library.

It still needs some work, but here is the first pass:

package uk.me.hardill.coap2mqtt;

import java.net.InetSocketAddress;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.HashMap;
import java.util.logging.Level;

import org.eclipse.californium.core.CaliforniumLogger;
import org.eclipse.californium.core.CoapClient;
import org.eclipse.californium.core.CoapHandler;
import org.eclipse.californium.core.CoapResponse;
import org.eclipse.californium.core.Utils;
import org.eclipse.californium.core.coap.MediaTypeRegistry;
import org.eclipse.californium.core.network.CoapEndpoint;
import org.eclipse.californium.core.network.config.NetworkConfig;
import org.eclipse.californium.scandium.DTLSConnector;
import org.eclipse.californium.scandium.ScandiumLogger;
import org.eclipse.californium.scandium.config.DtlsConnectorConfig;
import org.eclipse.californium.scandium.dtls.pskstore.StaticPskStore;
import org.eclipse.paho.client.mqttv3.IMqttDeliveryToken;
import org.eclipse.paho.client.mqttv3.MqttCallback;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence;
import org.json.JSONArray;
import org.json.JSONObject;

/**
 * @author hardillb
 *
 */
public class Main {
  
  static {
    CaliforniumLogger.disableLogging();
    ScandiumLogger.disable();
//    ScandiumLogger.initialize();
//    ScandiumLogger.setLevel(Level.FINE);
  }
  
  private DTLSConnector dtlsConnector;
  private MqttClient mqttClient;
  private CoapEndpoint endPoint;
  
  private String ip;
  
  private HashMap<String, Integer> name2id = new HashMap<>();
  
  Main(String psk, String ip) {
    this.ip = ip;
    DtlsConnectorConfig.Builder builder = new DtlsConnectorConfig.Builder(new InetSocketAddress(0));
    builder.setPskStore(new StaticPskStore("", psk.getBytes()));
    dtlsConnector = new DTLSConnector(builder.build());
    endPoint = new CoapEndpoint(dtlsConnector, NetworkConfig.getStandard());
    
    MemoryPersistence persistence = new MemoryPersistence();
    try {
      mqttClient = new MqttClient("tcp://localhost", MqttClient.generateClientId(), persistence);
      mqttClient.connect();
      mqttClient.setCallback(new MqttCallback() {
        
        @Override
        public void messageArrived(String topic, MqttMessage message) throws Exception {
          // TODO Auto-generated method stub
          System.out.println(topic + " " + message.toString());
          String parts[] = topic.split("/");
          int id = name2id.get(parts[1]);
          System.out.println(id);
          String command = parts[3];
          System.out.println(command);
          try{
            JSONObject json = new JSONObject("{\"9001\":\"Living Room Light\",\"9020\":1491515804,\"9002\":1491158817,\"9003\":65537,\"9054\":0,\"5750\":2,\"3\":{\"0\":\"IKEA of Sweden\",\"1\":\"TRADFRI bulb E27 opal 1000lm\",\"2\":\"\",\"3\":\"1.1.1.0-5.7.2.0\",\"6\":1},\"9019\":1,\"3311\":[{\"5850\":1,\"5851\":10,\"9003\":0}]}");
            JSONObject settings = json.getJSONArray("3311").getJSONObject(0);
            if (command.equals("dim")) {
              settings.put("5851", Integer.parseInt(message.toString()));
            } else if (command.equals("on")) {
              if (message.toString().equals("0")) {
                settings.put("5850", 0);
                settings.put("5851", 0);
              } else {
                settings.put("5850", 1);
                settings.put("5851", 128);
              }
            }
            String payload = json.toString();
            Main.this.set("coaps://" + ip + "//15001/" + id, payload);
          } catch (Exception e) {
            e.printStackTrace();
          }
        }
        
        @Override
        public void deliveryComplete(IMqttDeliveryToken token) {
          // TODO Auto-generated method stub
        }
        
        @Override
        public void connectionLost(Throwable cause) {
          // TODO Auto-generated method stub
        }
      });
      mqttClient.subscribe("TRÅDFRI/+/control/+");
    } catch (MqttException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
  }
  
  private void discover() {
    try {
      URI uri = new URI("coaps://" + ip + "//15001");
      CoapClient client = new CoapClient(uri);
      client.setEndpoint(endPoint);
      CoapResponse response = client.get();
      JSONArray array = new JSONArray(response.getResponseText());
      for (int i=0; i<array.length(); i++) {
        String devUri = "coaps://"+ ip + "//15001/" + array.getInt(i);
        this.watch(devUri);
      }
      client.shutdown();
    } catch (URISyntaxException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
  }
  
  private void set(String uriString, String payload) {
    System.out.println("payload\n" + payload);
    try {
      URI uri = new URI(uriString);
      CoapClient client = new CoapClient(uri);
      client.setEndpoint(endPoint);
      CoapResponse response = client.put(payload, MediaTypeRegistry.TEXT_PLAIN);
      if (response.isSuccess()) {
        System.out.println("Yay");
      } else {
        System.out.println("Boo");
      }
      
      client.shutdown();
      
    } catch (URISyntaxException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
  }
  
  private void watch(String uriString) {
    
    try {
      URI uri = new URI(uriString);
      
      CoapClient client = new CoapClient(uri);
      client.setEndpoint(endPoint);
      CoapHandler handler = new CoapHandler() {
        
        @Override
        public void onLoad(CoapResponse response) {
          System.out.println(response.getResponseText());
          JSONObject json = new JSONObject(response.getResponseText());
          if (json.has("3311")){
            MqttMessage message = new MqttMessage();
            int state = json.getJSONArray("3311").getJSONObject(0).getInt("5850");
            message.setPayload(Integer.toString(state).getBytes());
            message.setRetained(true);
            String topic = "TRÅDFRI/" + json.getString("9001") + "/state/on";
            String topic2 = "TRÅDFRI/" + json.getString("9001") + "/state/dim";
            name2id.put(json.getString("9001"), json.getInt("9003"));
            MqttMessage message2 = new MqttMessage();
            int dim = json.getJSONArray("3311").getJSONObject(0).getInt("5851");
            message2.setPayload(Integer.toString(dim).getBytes());
            message2.setRetained(true);
            try {
              mqttClient.publish(topic, message);
              mqttClient.publish(topic2, message2);
            } catch (MqttException e) {
              // TODO Auto-generated catch block
              e.printStackTrace();
            }
          } else {
            System.out.println("not bulb");
          }
        }
        
        @Override
        public void onError() {
          // TODO Auto-generated method stub
          
        }
      };
      client.observe(handler);
    } catch (URISyntaxException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }
    
  }

  /**
   * @param args
   */
  public static void main(String[] args) throws InterruptedException {
    String psk = args[0];
    String ip = args[1];
    Main m = new Main(psk, ip);
    
    m.discover();
  }

}

I’ve tagged the code onto the gist as well for now, but I’ll check the whole thing in as a separate project soon.

EDIT: now with it’s own Github repo here