Updated AWS Lambda NodeJS Version checker

I got another of those emails from Amazon this morning that told me that the version of the NodeJS runtime I’m using in the Lambda for my Node-RED Alexa Smart Home Skill is going End Of Life.

I’ve previously talked about wanting a way to automate checking what version of NodeJS was in use across all the different AWS Availability Regions. But when I tried my old script it didn’t work.

for r in `aws ec2 describe-regions --output text | cut -f3`; 
do
  echo $r;
  aws --region $r lambda list-functions | jq '.[] | .FunctionName + " - " + .Runtime';
done

This is most likely because the output from awscli tool has changed slightly.

The first change looks to be in the listing of the available regions

$ aws ec2 describe-regions --output text
REGIONS	ec2.eu-north-1.amazonaws.com	opt-in-not-required	eu-north-1
REGIONS	ec2.ap-south-1.amazonaws.com	opt-in-not-required	ap-south-1
REGIONS	ec2.eu-west-3.amazonaws.com	opt-in-not-required	eu-west-3
REGIONS	ec2.eu-west-2.amazonaws.com	opt-in-not-required	eu-west-2
REGIONS	ec2.eu-west-1.amazonaws.com	opt-in-not-required	eu-west-1
REGIONS	ec2.ap-northeast-2.amazonaws.com	opt-in-not-required	ap-northeast-2
REGIONS	ec2.ap-northeast-1.amazonaws.com	opt-in-not-required	ap-northeast-1
REGIONS	ec2.sa-east-1.amazonaws.com	opt-in-not-required	sa-east-1
REGIONS	ec2.ca-central-1.amazonaws.com	opt-in-not-required	ca-central-1
REGIONS	ec2.ap-southeast-1.amazonaws.com	opt-in-not-required	ap-southeast-1
REGIONS	ec2.ap-southeast-2.amazonaws.com	opt-in-not-required	ap-southeast-2
REGIONS	ec2.eu-central-1.amazonaws.com	opt-in-not-required	eu-central-1
REGIONS	ec2.us-east-1.amazonaws.com	opt-in-not-required	us-east-1
REGIONS	ec2.us-east-2.amazonaws.com	opt-in-not-required	us-east-2
REGIONS	ec2.us-west-1.amazonaws.com	opt-in-not-required	us-west-1
REGIONS	ec2.us-west-2.amazonaws.com	opt-in-not-required	us-west-2

This looks to have added something extra to the start of the each line, so I need to change which filed I select with the cut command by changing -f3 to -f4.

The next problem looks to be with the JSON that is output for the list of functions in each region.

$ aws --region $r lambda list-functions
{
    "Functions": [
        {
            "TracingConfig": {
                "Mode": "PassThrough"
            }, 
            "Version": "$LATEST", 
            "CodeSha256": "wUnNlCihqWLXrcA5/5fZ9uN1DLdz1cyVpJV8xalNySs=", 
            "FunctionName": "Node-RED", 
            "VpcConfig": {
                "SubnetIds": [], 
                "VpcId": "", 
                "SecurityGroupIds": []
            }, 
            "MemorySize": 256, 
            "RevisionId": "4f5bdf6e-0019-4b78-a679-12638412177a", 
            "CodeSize": 1080463, 
            "FunctionArn": "arn:aws:lambda:eu-west-1:434836428939:function:Node-RED", 
            "Handler": "index.handler", 
            "Role": "arn:aws:iam::434836428939:role/service-role/home-skill", 
            "Timeout": 10, 
            "LastModified": "2018-05-11T16:20:01.400+0000", 
            "Runtime": "nodejs8.10", 
            "Description": "Provides the basic framework for a skill adapter for a smart home skill."
        }
    ]
}

This time it looks like there is an extra level of array in the output, this can be fixed with a minor change to the jq filter

$aws lambda list-functions | jq '.[] | .[] | .FunctionName + " - " + .Runtime'
"Node-RED - nodejs8.10"

Putting it all back together to get

for r in `aws ec2 describe-regions --output text | cut -f4`;  do
  echo $r;
  aws --region $r lambda list-functions | jq '.[] | .[] | .FunctionName + " - " + .Runtime'; 
done

eu-north-1
ap-south-1
eu-west-3
eu-west-2
eu-west-1
"Node-RED - nodejs8.10"
ap-northeast-2
ap-northeast-1
sa-east-1
ca-central-1
ap-southeast-1
ap-southeast-2
eu-central-1
us-east-1
"Node-RED - nodejs8.10"
us-east-2
us-west-1
us-west-2
"Node-RED - nodejs8.10"

Quick and Dirty Touchscreen Driver

I spent way too much time last week at work trying to get a Linux kernel touchscreen driver to work.

The screen vendor supplied the source for the driver with no documentation at all, after poking through the code I discovered it took it’s configuration parameters from a Device Tree overlay.

Device Tree

So started the deep dive into i2c devices and Device Tree. At first it all seemed so easy, just a short little overlay to set the device’s address and to set a GPIO pin to act as an interrupt, e.g. something like this:

/dts-v1/;
/plugin/;

/ {
    fragment@0 {
        target = <&i2c1>;
        __overlay__ {
            status = "okay";
            #address-cells = <1>;
            #size-cells = <0>;

            pn547: pn547@28 {
                compatible = "nxp,pn547";
                reg = <0x28>;
                clock-frequency = <400000>;
                interrupt-gpios = <&gpio 17 4>; /* active high */
                enable-gpios = <&gpio 21 0>;
            };
        };
    };
};

All the examples are based around a hard wired i2c device attached to a permanent system i2c bus, this is where my situation differs. Due to “reasons” too complicated to go into here, I have no access to either of the normal i2c buses available on a Raspberry Pi so I’ve ended up using a Adafruit Trinket running the i2c_tiny_usb firmware as a USB i2c adapter and attaching the touchscreen via this bus. The kernel driver for the i2c_tiny_usb devices is already baked into the default Raspbian Linux kernel so meant I didn’t have to build anything special.

The problem is that USB devices are not normally represented in the Device Tree as they can be hot plugged. After being plugged in they are enumerated to discover what modules to load to support the hardware. The trick now was to work out where to attach the touchscreen i2c device, so the interrupt configuration would be passed to the driver when it was loaded.

I tried all kinds of different overlays, but no joy. The Raspberry Pi even already has a Device Tree entry for a USB device, because the onboard Ethernet is actually a permanently wired device and has an entry in the Device Tree. I tried copying this pattern and adding an entry for the tiny_i2c_usb device and then the i2c device but still nothing worked.

I have an open Raspberry Pi Stack Exchange question and an issue on the tiny-i2c-usb github page that hopefully somebody will eventually answer.

Userspace

Having wasted a week and got nowhere this morning I decided to take a different approach (mainly for the sake of my sanity). This is a basic i2c device with a single GPIO pin to act as an interrupt when new data is available. I knew I could write userspace code that would watch the pin and read from the device, so I set about writing a userspace device driver.

Python has good i2c and GPIO bindings on the Pi so I decided to start there.

import smbus
import RPi.GPIO as GPIO
import signal

GPIO.setmode(GPIO.BCM)
GPIO.setup(27, GPIO.IN, pull_up_down.PUD_UP)

bus=smbus.SMBus(3)

def callback(c):
  ev = bus.read_i2c_block_data(0x38,0x12,2)
  x = ev[0]
  y = ev[1]
  print("x=%d y=%d" % (x, y))

GPIO.add_event_detect(27,GPIO.FALLING,callback=callback)
signal.pause()

This is a good start but it would be great to be able to use the standard /dev/input devices like a real mouse/touchscreen. Luckily there is the uinput kernel module that exposes an API especially for userspace input devices and there is the python-uinput module.

import smbus
import RPi.GPIO as GPIO
import uinput
import signal

GPIO.setmode(GPIO.BCM)
GPIO.setup(27, GPIO.IN, pull_up_down.PUD_UP)

bus=smbus.SMBus(3)

device = uinput.device([
  uinput.ABS_X,
  uinput.ABS_Y
])

def callback(c):
  ev = bus.read_i2c_block_data(0x38,0x12,3)
  down = ev[0]
  x = ev[1]
  y = ev[2]
  if down == 0:
    device.emit(uinput.BTN_TOUCH, 1, syn=False)
    device.emit(uinput.ABS_X, x, syn=False)
    device.emit(uinput.ABS_Y, y)
  else:
    device.emit(uinput.BTN_TOUCH, 0)   

GPIO.add_event_detect(27,GPIO.FALLING,callback=callback)
signal.pause()

This injects touchscreen coordinates directly into the /dev/input system, the syn=False in the X axis value tells the uinput code to batch the value up with the Y axis value so it shows up as an atomic update.

This is a bit of a hack, but it should be more than good enough for what I need it for, but I’m tempted to keep chipping away at the Device Tree stuff as I’m sure it will come in handy someday.

Static IP Addresses and Accounting

Over the last few posts I’ve talked about how to set up the basic parts needed to run a small ISP.

In this post I’m going to cover adding a few extra features such as static IP addresses, Bandwidth accounting and Bandwidth limiting/shaping.

Static IP Addresses

We can add a static IP address by adding a field to the users LDAP entry. To do this first we need to add the Freeradius schema to the list of fields that the LDAP server understands. The Freeradius schema files can be found in the /usr/share/doc/freeradius/schemas/ldap/openldap/ and have been gzipped. I unzipped them and copied them to /etc/ldap/schema then imported it with

$ sudo ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/freeradius.ldif

Now we have the schema imported we can now add the radiusprofile objectClass to the user along with a radiusFramedIPAddress entry with the following ldif file.

dn: uid=isp1,ou=users,dc=hardill,dc=me,dc=uk
changetype: modify
add: radiusFramedIPAddress
rediusFramedIPAddress: 192.168.5.2

We then use ldapmodify to update the isp1 users record

$ ldapmodify -f addIPAddress.ldif -D cn=admin,dc=hardill,dc=me,dc=uk -w password

Now we have the static IP address stored against the user, we have to get the RADIUS server to pass that information back to the PPPoE server after it has authenticated the user. To do this we need to edit the /etc/freeradius/3.0/mods-enabled/ldap file. Look for the `update` section and add the following

update {
  ...
  reply:Framed-IP-Address     := 'radiusFramedIPAddress'
}

Running radtest will now show Framed-IP-Address in the response message and when pppoe-server receives the authentication response it will use this as the IP address for the client end of the connection.

Accounting

Out of the box pppoe-server will send accounting messages to the RADIUS server at the start and end of the session.

Sat Aug 24 21:35:17 2019
	Acct-Session-Id = "5D619F853DBB00"
	User-Name = "isp1"
	Acct-Status-Type = Start
	Service-Type = Framed-User
	Framed-Protocol = PPP
	Acct-Authentic = RADIUS
	NAS-Port-Type = Virtual
	Framed-IP-Address = 192.168.5.2
	NAS-IP-Address = 127.0.1.1
	NAS-Port = 0
	Acct-Delay-Time = 0
	Event-Timestamp = "Aug 24 2019 21:35:17 BST"
	Tmp-String-9 = "ai:"
	Acct-Unique-Session-Id = "290b459406a25d454fcfdf3088a2211c"
	Timestamp = 1566678917

Sat Aug 24 23:08:53 2019
	Acct-Session-Id = "5D619F853DBB00"
	User-Name = "isp1"
	Acct-Status-Type = Stop
	Service-Type = Framed-User
	Framed-Protocol = PPP
	Acct-Authentic = RADIUS
	Acct-Session-Time = 5616
	Acct-Output-Octets = 2328
	Acct-Input-Octets = 18228
	Acct-Output-Packets = 32
	Acct-Input-Packets = 297
	NAS-Port-Type = Virtual
	Acct-Terminate-Cause = User-Request
	Framed-IP-Address = 192.168.5.2
	NAS-IP-Address = 127.0.1.1
	NAS-Port = 0
	Acct-Delay-Time = 0
	Event-Timestamp = "Aug 24 2019 23:08:53 BST"
	Tmp-String-9 = "ai:"
	Acct-Unique-Session-Id = "290b459406a25d454fcfdf3088a2211c"
	Timestamp = 1566684533

The Stop message includes the session length (Acct-Session-Time) in seconds and the number of bytes downloaded (Acct-Output-Octets) and uploaded (Acct-Input-Octets).

Historically in the days of dial up that probably would have been sufficient as sessions would probably only last for hours at a time, not weeks/months for a DSL connection. pppoe-server can be told to send updates at regular intervals, this setting is also controlled by a field in the RADIUS authentication response. While we could add this to each user, it can be added to all users with a simple update to the /etc/freeradius/3.0/sites-enabled/default file in the post-auth section.

post-auth {
   update reply {
      Acct-Interim-Interval = 300
   }
   ...
}

This sets the update interval to 5mins and the log now also contains entries like this.

Wed Aug 28 08:38:56 2019
	Acct-Session-Id = "5D62ACB7070100"
	User-Name = "isp1"
	Acct-Status-Type = Interim-Update
	Service-Type = Framed-User
	Framed-Protocol = PPP
	Acct-Authentic = RADIUS
	Acct-Session-Time = 230105
	Acct-Output-Octets = 10915239
	Acct-Input-Octets = 17625977
	Acct-Output-Packets = 25918
	Acct-Input-Packets = 31438
	NAS-Port-Type = Virtual
	Framed-IP-Address = 192.168.5.2
	NAS-IP-Address = 127.0.1.1
	NAS-Port = 0
	Acct-Delay-Time = 0
	Event-Timestamp = "Aug 28 2019 08:38:56 BST"
	Tmp-String-9 = "ai:"
	Acct-Unique-Session-Id = "f36693e4792eafa961a477492ad83f8c"
	Timestamp = 1566977936

Having this data written to a log file is useful, but if you want to trigger events based on it (e.g. create a rolling usage graph or restrict speed once a certain allowance has been passed) then something a little more dynamic is useful. Freeradius has a native plugin interface, but it also has plugins that let you write Perl and Python functions that are triggered at particular points. I’m going to use the Python plugin to publish the data to a MQTT broker.

To enable the Python plugin you need to install the freeradius-python package

$ sudo apt-get install freeradius-python

And then we need to symlink the mods-available/python to mods-enabled and then edit the file. First we need to set the path that the plugin will use to file Python modules and files. And then enable the events we want to pass to the module.

python {
    python_path = "/etc/freeradius/3.0/mods-config/python:/usr/lib/python2.7:/usr/local/lib/python/2.7/dist-packages"
    module = example

    mod_instantiate = ${.module}
    func_instantiate = instantiate

    mod_accounting = ${.module}
    func_accounting = accounting
}

The actual code follows, it publishes the number of bytes used in the session to the topic isp/[username]/usage. Each callback gets pass a tuple containing all the values available.

import radiusd
import paho.mqtt.publish as publish

def instantiate(p):
  print "*** instantiate ***"
  print p
  # return 0 for success or -1 for failure

def accounting(p):
  print "*** accounting ***"
  radiusd.radlog(radiusd.L_INFO, '*** radlog call in accounting (0) ***')
  print
  print p
  d = dict(p)
  if d['Acct-Status-Type'] == 'Interim-Update':
      topic = "isp/" + d['User-Name'] + "/usage"
      usage = d['Acct-Output-Octets']
      print "publishing data to " + topic
      publish.single(topic, usage, hostname="hardill.me.uk", retain=True)
      print "published"
  return radiusd.RLM_MODULE_OK

def detach():
  print "*** goodbye from example.py ***"
  return radiusd.RLM_MODULE_OK

I was going to talk about traffic shaping next, but that turns out to be real deep magic and I need to spend some more time playing before I have something to share.

LDAP & RADIUS

As mentioned in the last post, I’m building a PoC ISP and to do this I need to set both an LDAP and RADIUS servers.

I’m going to run all of this on the latest version of Raspbian Buster.

LDAP

Lets start by installing the LDAP server.

$ sudo apt-get install ldap-server

This will install OpenLDAP. The first thing to do is to set the admin password and configure the base dn. To do this we first create a hashed version of the password with slappasswd

$ slappasswd
New password:
Re-enter new password: 
{SSHA}FRtFAY09RdZN76rZiVfgyqs2F3J9jXPN

We can then create the following ldif file called config.ldif. This sets the admin password and updates the base dn

dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcRootPW
olcRootPW: {SSHA}TXcmvaldskl312012cKsPK1cY2321+aj
-
replace: olcRootDN
olcRootDN: cn=admin,dc=hardill,dc=me,dc=uk
-
replace: olcSuffix
olcSuffix: dc=hardill,dc=me,dc=uk

We the apply these changed with the ldapmodify command

$ ldapmodify -a -Q -Y EXTERNAL -H ldapi:/// -f config.ldif

Now we have the admin user setup we can start to add the normal users. Again we need to use the slappasswd command to create password that we can use in the user.ldif file. I’ve added the inetOrgPerson in to the user entry so I can also include the mail item.

dn: uid=isp1,ou=users,dc=hardill,dc=me,dc=uk
objectClass: top
objectClass: person
objectClass: inetOrgPerson
displayName: Joe Blogs
cn: Joe
sn: Blogs
mail: isp1@hardill.me.uk
uid: isp1
userPassword: {SSHA}rozJD+T37NqRQp36myXf1KJ35+7tf2LN

And since we’ve set the admin password we need to modify the ldapmodify command as well

$ ldapadd -f user.ldif -D cn=admin,dc=hardill,dc=me,dc=uk -w password 

RADIUS

Next we need to install the RADIUS

$ sudo apt-get install freeradius

Once installed we need to enable the LDAP module and configure it to use the server we have just setup. To do this we need to symlink the ldap file from /etc/freeradius/3.0/mods-available to /etc/freeradius/3.0/mods-enabled. Next edit the identity, password and base_dn in the ldap config file to match the settings in config.ldif.

...
	#  additional schemes:
	#  - ldaps:// (LDAP over SSL)
	#  - ldapi:// (LDAP over Unix socket)
	#  - ldapc:// (Connectionless LDAP)
	server = 'localhost'
#	server = 'ldap.rrdns.example.org'
#	server = 'ldap.rrdns.example.org'

	#  Port to connect on, defaults to 389, will be ignored for LDAP URIs.
#	port = 389

	#  Administrator account for searching and possibly modifying.
	#  If using SASL + KRB5 these should be commented out.
	identity = 'cn=admin,dc=hardill,dc=me,dc=uk'
	password = password

	#  Unless overridden in another section, the dn from which all
	#  searches will start from.
	base_dn = 'ou=users,dc=hardill,dc=me,dc=uk'

	#
	#  SASL parameters to use for admin binds
...

Once we’ve restarted freeradius we can test if we can authenticate the isp1 user with the radtest command.

$ radtest isp1 secret 127.0.0.1 testing123
Sent Access-Request Id 159 from 0.0.0.0:42495 to 127.0.0.1:1812 length 78
	User-Name = "isp1"
	User-Password = "secret"
	NAS-IP-Address = 127.0.1.1
	NAS-Port = 0
	Message-Authenticator = 0x00
	Cleartext-Password = "secret"
Received Access-Accept Id 159 from 127.0.0.1:1812 to 127.0.0.1:42495 length 51

testing123 is the default password for a RADIUS client connecting from 127.0.0.1, you can change this and add more clients in the /etc/freeradius/3.0/clients.conf file.

In the next post I’ll talk about setting up PPPoE

Building an ISP

I’ve had this idea in the back of my head for ages, it’s centred round buying a building (something like an old Yorkshire mill, or better yet a private island) and dividing it up into a number of homes/offices/co-working paces.

To go with this fantasy I’ve been working out how to build a small scale boutique ISP (most of this would probably work for a small town community fibre or wireless mesh system) to share the hugely expensive high bandwidth symmetric dedicated fibre .

Over the next few posts I’m going to walk through building the PoC for this (which is likely to be where it stays unless I win the lottery)

To work out what I’d need lets first look roughly how home internet connections works.

At the advent of Home Internet there were two methods of delivering IP packets over a telephone/serial line, SLIP and PPP protocol. PPP became the dominant player and was extended to encapsulate PPP packets carried over both ATM (PPPoA) and Ethernet (PPPoE) frames in order to facilitate the move to DSL Home Broadband connections. PPPoE became the standard for the next evolution, FTTX (Where X can be B for building, P for premisses, or H for Home ). Modern home routers include a modem that converts DSL signal back to Ethernet frames and a PPPoE client to unpack the PPP connection back into IP packets to forward on to the network.

This means we need a PPPoE server for the users router to connect to, Linux has PPPoE support both as a client and as a server. I’ve already used the PPPoE client when the router for my FTTC Broadband service was late arriving.

Now we have the basic connection between the users equipment and the ISPs network we need to be able to authenticate each user so we know who is actually trying to connect. You can hard code credentials and details into the PPPoE configuration files, but this doesn’t scale and means you need to restart everything when ever something changes.

The better solution is something called a RADIUS server. RADIUS is a AAA service that can be used to not only authenticate users, but also supply information to the PPPoE server about that user, e.g. a static IP address allocation. RADIUS can also be used for accounting to record how much bandwidth each user has consumed.

A rasperry Pi and a Acer Revo hooked up to a ethernet switch
Initial testing

RADIUS servers can be backed by a number of different databases but the usual approach is to use LDAP.

In the next post I’ll cover installing the LDAP and RADIUS servers, then configuring them.

Listing AWS Lambda Runtimes

For the last few weeks I’ve been getting emails from AWS about Node 6.10 going end of life and saying I have deployed Lambda using this level.

The emails don’t list which Lambda or which region they think are at fault which makes tracking down the culprit difficult. I only really have 1 live instance deployed across multiple regions (and 1 test instance on a single region).

AWS Lambda region list

Clicking down the list of regions is time consuming and prone to mistakes.

In the email AWS do provide a command to list which Lambda are running with Node 6.10:

aws lambda list-functions --query="Functions[?Runtime=='nodejs6.10']"

But what they fail to mention is that this only checks your current default region. I can’t find a way to get the aws command line tool to list the Lambda regions, the closest I’ve found is the list of ec2 regions which hopefully match up. Pairing this with the command line JSON search tool jq and a bit of Bash scripting I’ve come up with the following:

for r in `aws ec2 describe-regions --output text | cut -f3`; 
do
  echo $r;
  aws --region $r lambda list-functions | jq '.[] | .FunctionName + " - " + .Runtime';
done

This walks over all the regions as prints out all the function names and the runtime they are using.

eu-north-1
ap-south-1
eu-west-3
eu-west-2
eu-west-1
"Node-RED - nodejs8.10"
"oAuth-test - nodejs8.10"
ap-northeast-2
ap-northeast-1
sa-east-1
ca-central-1
ap-southeast-1
ap-southeast-2
eu-central-1
us-east-1
"Node-RED - nodejs8.10"
us-east-2
us-west-1
us-west-2
"Node-RED - nodejs8.10"

In my case it only lists NodeJS 8.10 so I have no idea why AWS keep sending me these emails. Also since I’m only on the basic level I can’t even raise a technical help desk query to find out.

Anyway I hope this might be useful to others with the same problem.

DNS-over-HTTPS update

My post on DNS-over-HTTPS from last year is getting a fair bit more traffic after a few UK news paper articles (mainly crying that the new UK Government  censoring won’t work if Google roll it out in Chrome… what a shame). The followning article has a good overview [nakedsecurity].

Anyway I tweeted a link to the old post and it started a bit of a discussion and the  question about the other side of system came up. Namely how to use a DNS resolver that pushed traffic over DNS-over-HTTPS rather than provide a HTTPS endpoint that supported queries. The idea being that at the moment only Firefox & Chrome can take advantage of the secure lookups.

I did a bit of poking around and found things like stubby which DNS-over-TLS (another approach to secure DNS lookups) and also Cloudflare have cloudflared which can proxy for DNS-over-HTTPS to Cloudflare’s DNS server (it also is used to set up the VPN tunnel to Cloudflare’s Argo service, which is also worth a good look at.)

Anyway, while there are existing solutions out there I thought I’d have a really quick go at writing my own, to go with the part I’d written last year, just to see how hard it could be.

It turned out a really basic first pass could be done in about 40 lines of Javascript:

const dgram = require('dgram')
const request = require('request')
const dnsPacket = require('dns-packet')

const port = process.env["DNS_PORT"] || 53
//https://cloudflare-dns.com/dns-query
const url = process.env["DNS_URL"] 
    || "https://dns.google.com/experimental" 
const allow_selfSigned = 
    (process.env["DNS_INSECURE"] == 1) 

const server = dgram.createSocket('udp6')

server.on('listening', function(){
  console.log("listening")
})

server.on('message', function(msg, remote){
  var packet = dnsPacket.decode(msg)
  var id = packet.id
  var options = {
    url: url,
    method: 'POST',
    body: msg,
    encoding: null,
    rejectUnauthorized: allow_selfSigned ? false : true,
    headers: {
      'Accept': 'application/dns-message',
      'Content-Type': 'application/dns-message'
    }
  }

  request(options, function(err, resp, body){
    if (!err &amp;&amp; resp.statusCode == 200) {
      var respPacket = dnsPacket.decode(body)
      respPacket.id = id
      server.send(body,remote.port)
    } else {
      console.log(err)
    }
  })

})

server.bind(port)

It really could do with some caching and some more error handling and I’d like to add support for Google JSON based lookups as well as the binary DNS format, but I’m going to add it to the github project with the other half and people can help extend it if they want.

The hardest part was working out I needed the encoding: null in the request options to stop it trying to turn the binary response into a string but leaving it as a Buffer.

I’m in the process of migrating my DNS setup to a new machine, I’ll be adding a DNS-over-TLS (using stunnel) & a DNS-over-HTTPS listeners for the public facing sides.

Building a Bluetooth speaker

Recently I’ve been playing with how to build a Bluetooth audio device using a Raspberry Pi Zero. The following are some notes on what I found.

First question is why build one when you can buy one for way less than the cost of the parts. There are a couple of reasons:

  • I build IoT prototypes for a living, and the best way to get a feel for the challenges is to actually face them.
  • Hacking on stuff is fun.

The Hardware

I’m starting out with a standard Raspberry Pi Zero W. This gets me a base high level platform that includes a WiFi and Bluetooth.

Raspberry Pi Zero W

The one thing that’s missing is an audio output (apart from the HDMI) but Raspberry Pi’s support audio using the I2S standard. There are several I2S pHATs available and I’m going to be using a pHAT DAC from Pimoroni. I’ve used these before for a project so I’m reasonably happy with how to set it up, but Pimoroni have detailed instructions.

I’m going to add a screen to show things like the current track title & artist along with the volume. I’m also going to need some buttons to send Play/Pause, Next & Previous commands to the connected device. I have a PaPiRus e-ink display that has 5 buttons built in which I was going to use but this clashes with the GPIO pins used for the DAC so instead I’ve opted for the Inky pHAT and the Button Shim.

The Software

I knew the core components of this had to be a problem others had solved and this proved to be the case. After a little bit of searching I found this project on github.

As part of the configuration we need to generate the Bluetooth Class bitmask. This can be done one this site.


Class options

This outputs a hex value of 0x24043C which is added to the /etc/bluetooth/main.conf

With this up and running I had a basic Bluetooth speaker that any phone can connect to without a pin and play music, but nothing else. The next step is to add some code to handle the button pushes and to update the display.

The Bluetooth stack on Linux is controlled and configured using DBus. Dbus is a messaging system supporting IPC and RPC

A bit of Googling round turned up this askubuntu question that got me started with the following command:

dbus-send --system --print-reply --dest=org.bluez /org/bluez/hci0/dev_44_78_3E_85_9D_6F org.bluez.MediaControl1.Play

This sends a Play command to the connected phone with the Bluetooth mac address of 44:78:3E:85:9D:6F. The problem is knowing what the mac address is as the system allows multiple devices to pair with the speaker. Luckily you can use DBus to query the system for the connected device. DBus also has some really good Python bindings. So with a bit more poking around I ended up with this:

#!/usr/bin/env python
import signal
import buttonshim
import dbus
bus = dbus.SystemBus()
manager = dbus.Interface(
bus.get_object("org.bluez","/"), 
"org.freedesktop.DBus.ObjectManager")

@buttonshim.on_press(buttonshim.BUTTON_A)
def playPause(button, pressed):
	objects = manager.GetManagedObjects()
	 
	for path in objects.keys():
	    interfaces = objects[path]
	    for interface in interfaces.keys():
	        if interface in [
	        "orge.freedesktop.DBus.Introspectable",
	        "org.freedesktop.DBus.Properties"]:
	            continue
	 
	        if interface == "org.bluez.Device1":
	            props = interfaces[interface]
	            if props["Connected"] == 1:
	                media = objects[path + "/player0"]["org.bluez.MediaPlayer1"]
	 
	                mediaControlInterface = dbus.Interface(
	                bus.get_object("org.bluez",path + "/player0"),
	                "org.bluez.MediaPlayer1")
	 
	                if media["Status"] == "paused":
	                    mediaControlInterface.Play()
	                else:
	                    mediaControlInterface.Pause()

signal.pause()

When button A is pressed this looks up the connected device, and also checks the current state of the player, is it playing or paused and toggles the state. This means that one button can be Play and Pause. It also uses the org.bluez.MediaPlay1 API rather than the org.bluez.MediaControl1 which is marked as deprecated in the doc.

The button shim also comes with Python bindings so putting it all together was pretty simple.

DBus also lets you register to be notified when a property changes, this paired with the Track property on the org.bluez.MediaPlay1 as this holds the Artist, Track Name, Album Name and Track length information supplied by the source. This can be combined with the Inky pHAT python library to show the information on the screen.

#!/usr/bin/env python

import dbus
from dbus.mainloop.glib import DBusGMainLoop
from gi.repository import GLib

def trackChanged(*args, **kw):
	target = args[0]
	if target == "org.bluez.MediaPlayer1":
		data = args[1].get("Track",0)
		if data != 0:
			artist = data.get('Artist')
			track = data.get('Title')
			print(artist)
			print(track)


DBusGMainLoop(set_as_default=True)
system_bus = dbus.SystemBus()
system_bus.add_signal_receiver(trackChanged, 
	dbus_interface="org.freedesktop.DBus.Properties", 
	signal_name="PropertiesChanged", 
	path='/org/bluez/hci0/dev_80_5A_04_12_03_0E/player0')
loop = GLib.MainLoop()
loop.run()

This code attaches a listener to the MediaPlayer object and when it spots that the Track has changed it prints out the new Artist and Title. The code matches all PropertiesChanged events which is a little messy but I’ve not found a way to use wildcards or partial matches for the DBus interface in python (since we don’t know the mac address of the connected device at the time we start listening for changes).

Converting the Artist/Title information into an image with the Pyton Image Library then getting the Inky pHAT to render that is not too tricky

from PIL import Image, ImageDraw, ImageFont
from font_fredoka_one import FredokaOne
from inky import InkyPHAT

...

disp = InkyPHAT("yellow")
font = ImageFont.truetype(FredokaOne, 22)

img = Image.new("P", (inky_display.WIDTH, inky_display.HEIGHT))
draw = ImageDraw.Draw(img)

draw.text((), "Artist: "+ artist, disp.WHITE, font=font)
draw.text((), "Track: "+ track, disp.WHITE, font=font)

disp.set_image(img)
disp.show()


That’s the basics working, now I need to find/build a case for it and then look at seeing if I can add Chromecast Audio and Airplay support.

Node-RED Google Home Smart Home Action

Google Home

Following on from my Alexa Home Skill for Node-RED it’s time to see about showing some love to the Google Home users (OK, I’ve been slowly chipping away at this for ages, but I’ve finally found a bit of time).

One of the nice things about Google Assistant is that it works all over the place, I can use it via the text interface if I’m somewhere and can’t talk, or even from the car via Android Auto.

Screenshot_20190101-170716

Google offer a pretty similar API for controlling Smart home devices to the one offered by Amazon for the Alexa so the implementation of this was very similar. The biggest difference is the is no requirement to use something like Amazon’s Lambda to interface with the service so it’s just a single web endpoint.

I’ve taken pretty much the same approach as with the Alexa version in that I have a Web Site where you can sign up for an account and then define virtual devices with specific names and characteristics.

Virtual devices

Google support a lot more different types of devices and characteristics than Amazon with Alexa at the moment, but to start with I’m just supporting Sockets/Light/Switches and Thermostats. I intend to add more later as I work out the best way to surface the data.

The other big change is that Google Assistant supports asynchronously updating the device state and the ability for the Assistant backend to query the state of a device. To support this I’m going to allow the response node to be configured with a specific device and to accept input that has not come from an input node.

The node is currently being beta tested, if you are interested post in #google-home-assistant on the Node-RED Slack and I can add you to the ACL for the beta.

Google Assistant Node-RED Node

I’ll do another post when the node has finished testing and has been accepted by Google.

DIY IoT button

I’ve been looking for a project for a bunch of ESP-8266 ESP-01 boards I’ve had kicking around for a while.

The following describes a simple button that when pushed publishes a MQTT message that I can subscribe to with Node-RED to control different tasks.

It’s been done many times before, but I wanted to have a go at building my own IoT button.

Software

The code is pretty simple:

  • The MQTT PubSubClient (Thank’s Nick)
  • Some hard coded WiFi and MQTT Broker details
  • The setup function which connects to the network
  • The reconnect function that connects to the MQTT broker and publishes the message
  • The loop function which flashes the LED to show success then go into deep sleep

In order to get the best battery life you want the ESP8266 to be in deep sleep mode for as much as possible, so for this reason the loop function sends the message to signify the button has been pushed then indefinitely enters the deepest sleep state possible. This means the loop function will only run once on start up.

#include <ESP8266WiFi.h>
#include <PubSubClient.h>

#define ESP8266_LED 1

const char* ssid = "WifiName";
const char* passwd = "GoodPassword";
const char* broker = "192.168.1.114";

WiFiClient espClient;
PubSubClient client(espClient);

void setup() {

  Serial.begin(115200);
  delay(10);
  
  pinMode(ESP8266_LED, OUTPUT);

  WiFi.hostname("Button1");
  WiFi.begin(ssid, passwd);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");  
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
  client.setServer(broker, 1883);
  reconnect();
}

void reconnect() {
  while(!client.connected()) {
    if (client.connect("button1")){
      client.publish("button1", "press");
    } else {
      delay(5000);
    }
  }
}

void loop() {

  if (client.connected()) {
    reconnect();
  }
  client.loop();
  
  // put your main code here, to run repeatedly:
  digitalWrite(ESP8266_LED, HIGH);
  delay(1000);
  digitalWrite(ESP8266_LED, LOW);
  delay(1000);
  ESP.deepSleep(0);
}

Hardware

By using a momentary push button wake the ESP-01 I bridged reset pin to ground the chip resets each time it’s pushed, this wakes it from the deep sleep state and runs the all the code, then drops back into the sleep state.

Button diagram

  • The green line is the Chip enable
  • The blue line is the the links reset to ground via the push button

Now I had the basic circuit and code working I needed to pick a power supply. The ESP-01 needs a 3.3v supply and most people seem to opt for using a small LiPo cell. A single cell has a nominal fully charged voltage of 3.7v which is probably close enough to use directly with the ESP-01, but the problem is you normally need to add a circuit to cut them out before the voltage gets too low as they discharge to prevent permanently damage them. You would normally add a charging circuit to allow recharging from a USB source.

This wasn’t what I was looking for, I wanted to use off the shelf batteries so I went looking for a solution using AAA batteries. The board will run directly from a fresh set of 2 AAA batteries, but the voltage can quickly drop too low. To help with this I found a part from Pololu that would take an input between 0.5v and 5v and generate a constant 3.3v. This meant that even as the batteries discharged I should be able to continue to run the device.

At first things didn’t look to work, because converter was not supplying enough current for the ESP-01 at start up, to get round this I added a 100uF capacitor across the outputs of the regulator. I don’t really know how properly to size this capacitor so I basically made a guess.

The final step was to use the soldering iron to remove the red power LED from the board as this was consuming way more power than the rest of the system.

Prototype

Next Steps

  • Make the MQTT topic based on the unique id of the ESP-01 so it isn’t hard coded
  • Look at adding Access Point mode to allow configuration of the WiFi details if the device can not connect to the existing configuration
  • Design a circuit board to hold the voltage converter, capacitor, button and the ESP-01
  • Create a case to hold it all
  • Work out just how long a set of batteries will last