Another LoRa Temperature/Humidity Sensor

Having deployed my Adafruit Feather version of a Temperature/Humidity LoRa Sensor to make use of my The Things Network gateway it’s now time to build another sesnor, this time with a Raspberry Pi Zero.

Pi Supply LoRa node pHat with sensor

I have a Pi Supply LoRa node pHAT that adds the radio side of things, but I needed a sensor. The pHAT has a header on the right hand edge which pulls out the 3.3v, 2, 3, 4 and Gnd, pins from the pi.

These just happen to line up with the I2C bus. They are also broken out in the same order as used by the Pimoroni Breakout Garden series of I2C sensors which is really useful.

Pimoroni’s Breakout Garden offer a huge range of sensors and output devices in either I2C or SPI. In this case I grabbed the BME680 which offers temperature, pressure, humidity and air quality in a single package.

Pimoroni supply a python library to read from the sensor so it was pretty easy to combine this with python to drive the LoRa pHat

#!/usr/bin/env python3
from rak811 import Mode, Rak811
import bme680
import time

lora = Rak811()
lora.hard_reset()
lora.mode = Mode.LoRaWan
lora.band = 'EU868'
lora.set_config(app_eui='xxxxxxxxxxxxx',
                app_key='xxxxxxxxxxxxxxxxxxxxxx')
lora.join_otaa()
lora.dr = 5

sensor = bme680.BME680(bme680.I2C_ADDR_PRIMARY)
sensor.set_humidity_oversample(bme680.OS_2X)
sensor.set_temperature_oversample(bme680.OS_8X)
sensor.set_filter(bme680.FILTER_SIZE_3)

while True:
    if sensor.get_sensor_data():
        temp = int(sensor.data.temperature * 100)
        humidity = int(sensor.data.humidity * 100)
        foo = "{0:04x}{1:04x}".format(temp,humidity)
        lora.send(bytes.fromhex(foo))
        time.sleep(20)

lora.close()

I’m only using the temperature and humidity values at the moment to help keep the payload compatible with the feather version so I can use a single “port” in the TTN app.

I’ve put all the code for both this and the Feather version on GitHub here.

As well as the Pi Zero version I’ve upgraded the Feather with a 1200mAh LiPo battery and updated the code to also transmit the battery voltage so I can track battery life and work out how often I need to charge it.

Adding the battery also meant I could stick it in my bag and go for a walk round the village to see what sort of range I’m getting.

I’m currently using a little coil antenna on the gateway that is laid on it’s side, I’ll find something to prop it up with which should help. The feather is also only using a length of wire antenna so this could be seen as a worst case test.

It’s more than good enough for what I need right now, but it would be good to put a proper high gain one in then it might be useful to more people. It would be great if there was a way to see how many unique TTN applications had forwarded data through a gateway to see if anybody else has used it.

LoRa Temperature/Humidity Sensor

After deploying the TTN Gateway in my loft last year, I’ve finally got round to deploying a sensor to make use of it.

As the great lock down of 2020 kicked off I stuck in a quick order with Pimoroni for a Adafruit Feather 32u4 with an onboard LoRa radio (In hindsight I probably should have got the M0 based board).

Adafruit Feather LoRa Module

I already had a DHT22 temperature & humidity and the required resistor so it was just a case of soldering on an antenna of the right length and adding the headers so I could hook up the 3 wires needed to talk to the DHT22.

There is an example app included in with the TinyLoRa library that reads from the sensor and sends a message to TTN every 30 seconds.

(There are full instructions on how to set everything up on the Adafruit site here)

Now the messages are being published to TTN Application I needed a way to consume them. TTN have a set of Node-RED nodes which make this pretty simple. Though installing them did lead to a weekend adventure of having to upgrade my entire home automation system. The nodes wouldn’t build/install on Node-RED v0.16.0, NodeJS v6 and Raspbian Jessie so it was time to upgrade to Node-RED v1.0.4, NodeJS v12 and Raspbian Buster.

Node-RED flow consuming messages from Temp/Humidity sensor from TTN

This flow receives messages via MQTT direct from TTN, it then separates out the Temperature and Humidity values, runs them through smooth nodes to generate a rolling average of the last 20 values. This is needed because the DHT22 sensor is pretty noisy.

Plots of Temperature and Humdity

It outputs these smoothed values to 2 charts on a Node-RED Dashboard. which show the trend over the last 12 hours.

It also outputs to 2 MQTT topics that mapped to my public facing broker, which means they can be used to resurrect the temperature chart on my home page, and when I get an outdoor version of the sensor (hopefully solar/battery powered) up and running it will have both lines that my old weather station used to produce.

The last thing the flow does is to update the temperature reading for my virtual thermostat in the Google Assistant HomeGraph. This uses the RBE node to make sure that it only sends updates when the value actually changes.

The next step is to build a couple more and find a suitable battery and enclosure so I can stick one in the back garden.

Gophering around

I’ve been listening to a podcast called “Brad & Will Made a Tech Pod.” . It’s a fun geeky hour that comes out every Sunday.

In their last episode (44-alt-barney-dinosaur-die-die-die) they went back to their first contact with the Internet and this lead to a discussion of “The Gopher Space”, which is basically what the Internet was before “The Web” took over.

In my case my first trip online was a dial up educational BBS on a BBC Micro while at junior school, I dread to think how big the school phone bill must have been and how often I cut off the school Secretary off mid call by flicking the switch that hooked the line up to the acoustic coupler modem.

By the time I got to secondary school we had a 486 PC at home and I remember convincing my parents to let me buy a modem and posting 12 forward dated £12.00 cheques to Deamon Internet for a years worth of dial up access. At this point the Mosaic graphical Web browser had just turned up so I missed out on digging too deep into the Gopher Space.

But all the talk got me wondering how hard it would be to set up a gopher server to have a play with. I already run my own webserver to host this blog so why not see if I can host my posts on gopher as well.

A little bit of digging and I found pygopherd which comes ready packaged on Rasbian/RaspberryOS and just needs the servername setting in /etc/pygopherd/pygopherd.conf and a entry in the port forwarding rules on the router to get up and running.

I can now use the command gopher gopher.hardill.me.uk to start exploring. This is a terminal based tool which gives an authentic experience, but there are plugins for Firefox and Chrome if you want to play on the desktop.

A gopher site is sort of like a very stripped back webpage, it’s text only with no real formatting but it can contain links to other pages and to binary files that can be downloaded but not viewed inline. For example here is my “homepage” is in a file called gophermap (a bit like index.html) in /var/gopher

Ben's Place - Gopher

Just a place to make notes about things I've 
been playing with

1Blog	/blog
1Brad & Will Made a Tech Pod	/podcast
A screen shot of the gophermap file listed above rendered by the gophner command line applicatio

The lines starting with a 1 point to another gophermap with titles Blog and Brad & Will Made a Tech Pod in /var/gopher/blog and /var/gopher/podcast respectively. If the line had started with 0 it would have pointed to a text file. A full list of the prefixes can be found here.

The plan is to generate text versions of all the posts on the blog. I’ll probably write a script that takes the ATOM feed and do the conversion, but in the mean time, there was a small challenge thrown down in podcast to host all the podcast over gopher.

I was also looking for an excuse to start to doing some playing in Go which just seemed apt.

A quick search found a library to do the RSS download and parsing so the whole thing only took about 50 lines. You can find the code on github here.

You run the code as follows:

./gopherPodcast http://some.podcast.com/rss > gophermap

I’ll be running this every Sunday morning with a crontab entry to grab the latest episode and add it to the list.

Creating Fail2Ban rules

I recently installed some log charting software to get a view NGINX logs for the instance that servers up this site. While looking at some of the output I found some strange results so went to have a closer look at the actual logs

Chart showing hits/visitor to site

There were a whole bunch of entries with with a similar pattern, something similar to the following:

94.130.58.174 - - [15/Jul/2020:11:43:39 +0100] "CONNECT rate-market.ru:443 HTTP/1.1" 400 173 "-" "-"

CONNECT is a HTTP verb that is used by HTTP Proxy clients to tell the proxy server to set up a tunnel to the requested machine. This is needed because is the client is trying to connect to a HTTP server that uses SSL/TLS then it needs to be able to do the SSL/TLS handshake directly with that server to ensure it has the correct certificates.

Since my instance of NGINX is not configured to be a HTTP Proxy server it is rightly rejecting these requests as can been seen by the 400 (Bad Request) after the request string. Even though the clients making these request are all getting the 400 error they are continuing to send the same request over and over again.

I wanted to filter these bogus request out for 2 reasons

  • They clog the logs up, making it harder to see real problems
  • They take up resources to process

The solution is to use a tool called fail2ban, which watches log files for signs of abuse and then sets up the correct firewall rules to prevent the remote clients from being able to connect.

I already run fail2ban monitoring the SSH logs for all my machines that face the internet as there are plenty of scripts out there running trying standard username/password combinations attempting to log into any machine listening on port 22.

Fail2ban comes with a bunch of log filters to work will a applications, but the ones bundled on raspbian (RaspberryOS) at this time didn’t include one that would match this particular pattern. Writing filters is not that hard but it does require writing regular expressions which is a bit of dark art. The trick is to get a regex that only matches what we want.

The one I came up with is this:

^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400

Let’s break that down into it’s specific parts

  • ^ this matches the start of the log line
  • <HOST> fail2ban has some special tags that match an IP address or Host name
  • - - this matches the 2 dashes in the log
  • \[.*\] this matches the date/timestamp in the square brackets, I could have matched the date more precisely, but we don’t need to in this case. We have to escape the [ as it has meaning in regex as we will see later.
  • \"CONNECT this matches the start of the request string and limits it to just CONNECT requests
  • .* this matches the host:port combination that is being requested
  • HTTP/1\.[0-1]\" here we match both HTTP/1.0 and HTTP/1.1. We need to escape the . and the square brackets let us specify a range of values
  • 400 and finally the 400 error code

Now we have the regex we can test it with the fail2ban-regex command to make sure it finds what we expect.

# fail2ban-regex /var/log/nginx/access.log '^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400' 

Running tests
=============

Use   failregex line : ^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400
Use         log file : /var/log/nginx/access.log
Use         encoding : UTF-8


Results
=======

Failregex: 12819 total
|-  #) [# of hits] regular expression
|   1) [12819] ^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [526165] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 526165 lines, 0 ignored, 12819 matched, 513346 missed
[processed in 97.73 sec]

Missed line(s): too many to print.  Use --print-all-missed to print all 513346 lines

We can see that the regex matched 12819 lines out of 526165 which is in the right ball park. Now we have the right regex we can build the fail2ban config files.

First up is the filter file that will live in /etc/fail2ban/filter.d/nginx-connect.conf

# Fail2Ban filter to match HTTP Proxy attempts
#

[INCLUDES]

[Definition]

failregex = ^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400

ignoreregex = 

next is the jail file in /etc/fail2ban/jail.d/nginx-connect.conf, this tells fail2ban what to do if it finds log lines that match the filter and where to find the logs.

[nginx-connect]
port	= http,https
enabled	= true
logpath = %(nginx_access_log)s
action	= %(action_mwl)s

The port entry is which ports to block in the firewall, enabled & logpath should be obvious (it’s using a pre-configured macro for the path). And finally the action, there are a number of pre-configured actions that range from just logging the fact that there were hits in the log all the way through to this on action_mwl, which adds a firewall rule to block the <HOST>, then emails me that with information about the IP address and a selection of the matching log lines.

I’ll probably reduce the action to one that just adds the firewall rule after a day or two when I have a feel for how much of this sort of traffic I’m getting. But the info about each IP address normally include a lookup of the owner and that usually has an email address to report problems. It’s not often that reporting things actually stops it happening, but it’s worth giving it ago at the start to see what happens.

Over the course of the afternoon since it was deployed the filter has blocked 148 IP addresses.

My default rules are a 12 hour ban if the same IP address triggers more than 3 times in 10 mins with a follow up of getting a 2 week ban if you trip that more than 3 times in 3 days.

Next up is to create a filter to match the GET https://www.instagram.com requests that are also polluting the logs.

64bit Raspberry Pi OS build with USB-C Ethernet enabled

As a follow up to the 32bit version I built, using the script I described in my post about building custom SD card images, I have created a 64bit version based on the Raspberry Pi OS beta.

You can download the 64bit version from here. This is a full Raspberry Pi OS image, not a “lite” version as only full versions are available in beta.

This is a manual build (following these instructions) because at the moment the script won’t work with the 64bit version because DockerPi doesn’t support USB devices when emulating 64bit CPUs. USB is needed to attach the emulated Network Adapter. There are a set of patches for QUEMU pending that should enable this.

As soon as the patches become generally available I’ll update the project on github.

Raspberry Pi Streaming Camera

I talked about using a ONVIF camera to stream to a Chromecast earlier because they come with an open well documented interface for pulling video from them (as well as pan/tilt/zoom control if available).

If you don’t have a camera that supports ONVIF you can build something similar with a Raspberry Pi and the Camera module.

This should work with pretty much all of the currently available Raspberry Pi models (With the exception of the basic Pi Zero that doesn’t have Wifi)

  1. Flash a SD card with the Raspbian Lite image
  2. Insert the camera ribbon cable into both the camera module and the Pi
  3. Once the card has booted use the raspi-conf command to enable the Camera interface
  4. Install ffmpeg sudo apt-get install ffmpeg
  5. Create a script with the following content
#!/bin/sh

v4l2-ctl --set-ctrl video_bitrate=300000

ffmpeg -f video4linux2 -input_format h264 -video_size 640x360 -framerate 30 -i /dev/video0  -vcodec copy -an -f flv rtmp://192.168.1.96/show/pi
  • This script sets the max video bitrate to 30kps
  • If you need to rotate the video you can insert v4l2-ctl --set-ctrl=rotate=180 before ffmpeg to rotate 180 degrees
  • ffmpeg uses the videolinux2` driver to read from the attached camera (/dev/video0)
  • Takes h264 encoded feed at 640x360 and 30 frames per second and outputs it to the same nginx instance that I mentioned in my previous post. The feed is called pi

ffmpeg uses the on board hardware support for the video encoding so even a Pi Zero W runs at about 5% CPU load. This means that if you only have 1 camera you could probably run nginx on the same device, else you can have a multiple cameras all feeding to a central video streaming server.

If you want a kit that comes with the Pi Zero W, Camera and a case to mount it to a window have a look at the Pimoroni OctoCam.

The instructions should also work for pretty much any USB (or built in) camera attached to a Linux machine.

Streaming Camera to Chromecast

I have a little cheap WiFi camera I’ve been meaning to do something with for a while. The on board web access doesn’t really work any more because it only supports Flash or Java Applet to view the stream.

But camera supports the ONVIF standard so it offers a rtsp:// feed so I can point Linux apps like mplayer at it and see the stream.

The camera is currently sat on my window sill looking out over the valley which is a pretty good view.

View from upstairs window

I thought it would be interesting to stream the view to the TV in my Living room while I’m working from home at the moment. It is also a way to check the weather without having to get up in the morning and open the blind.

I have a Chromecast in the back back of both TVs so using this seamed like it would be the easiest option.

Chromecast

Chromecasts support a number of different media types but for video we have 2 common codec that will work across all the currently available device types.

  • H264
  • VP8

And we have 3 options to deliver the video stream

  • HLS
  • DASH
  • SmoothStreaming

These are all basically the same, they chop the video up into short segments and generate a play list that points to the segments in order and the consumer downloads each segment. When it reaches the end of the list it downloads the list again which will now hold the next list of segments.

There is a plugin for Nginx that supports both HLS and DASH which looked like a good place to start.

NGINX

I’m running this whole stack on a Raspberry Pi 4 running Raspbian Buster.

$ sudo apt-get install nginx libnginx-mod-rtmp ffmpeg

Once the packages are installed the following needs to be added to the end of the /etc/nginx/nginx.conf file. This sets up a rtmp listener that we can stream the video to which will then be turned into both a HLS and DASH stream to be consumed.

...
rtmp {
  server {
    listen 1935; # Listen on standard RTMP port
    chunk_size 4000;

    application show {
      live on;
      # Turn on HLS
      hls on;
      hls_type live;
      hls_path /var/www/html/hls/;
      hls_fragment 5s;
      hls_playlist_length 20s;
      
      # Turn on DASH      
      dash on;
      dash_path /var/www/html/dash/;
      dash_fragment 5s;
      dash_playlist_length 20s;

      # disable consuming the stream from nginx as rtmp
      deny play all;
    }
  }
}

The playlist and video segments get written to /var/www/html/hls and /var/www/html/dash respectively. Because they will be short lived and replaced very regularly it’s a bad idea to write these to an SD card as they will just cause excessive flash wear.

To get round this I’ve mounted tmpfs filesystems at those points with the following entries in /etc/fstab

tmpfs	/var/www/html/dash	tmpfs	defaults,noatime,size=50m
tmpfs	/var/www/html/hls	tmpfs	defaults,noatime,size=50m

Now we have the playlists and segments being generated in a sensible way we need to server them up. I added the following to the /etc/nginx/sites-enabled/default file

server {
  listen 8080;
  listen [::]:8080;

  sendfile off;
  tcp_nopush on;
  directio 512;
  default_type application/octet-stream;

  location / {
    add_header 'Cache-Control' 'no-cache';
    add_header 'Access-Control-Allow-Origin' '*' always;
    add_header 'Access-Control-Allow-Credentials' 'true';
    add_header 'Access-Control-Expose-Headers' 'Content-Length';

    if ($request_method = 'OPTIONS') {
      add_header 'Access-Control-Allow-Origin' '*';
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Max-Age' 1728000;
      add_header 'Content-Type' 'text/plain charset=UTF-8';
      add_header 'Content-Length' 0;
      return 204;
    }

    types {
      application/dash+xml mpd;
      application/vnd.apple.mpegurl m3u8;
      video/mp2t ts;
    }

    root /var/www/html/;
  }
}

Now we have the system to stream the content in an acceptable format we need to get the video from the camera into nginx. We can use ffmpeg to do this.

ffmpeg -re -rtsp_transport tcp -i rtsp://192.168.1.104:554/live/ch1 -vcodec libx264 -vprofile baseline -acodec aac -strict -2 -f flv rtmp://localhost/show/stream

This reads from the RTSP stream rtsp://192.168.1.104:554/live/ch1 and streams it into the rtmp://localhost/show/stream. The showpart is the name of the application declared in the rtmp section in the nginx.conf and the stream will be the name of the HLS or DASH stream. In this case the following:

  • HLS -> http://192.168.1.98/hls/stream.m38u
  • DASH -> http://192.168.1.98/dash/stream.mpd

If you change the end of the `rtmp://localhost/show/XXXX` URL you can create multiple streams from different sources with different names (just make sure the tmpfs mounts have enough space for all the streams).

Testing

Node-RED Chromecast flow

I’ve been using the Node-RED Chromecast node to test the streams. The DASH stream is working pretty well, but the HLS is a bit more fragile for some reason. Latency is currently about 20-30 seconds which is appears mainly to be determined by the size and number of chunks in the playlist used but if I wind the fragment size down any lower than 5s or the 20s for the playlist length.

Next

Now it’s basically working the next steps are to add support for Camera devices to my Google Assistant Node-RED service so I can request the stream via voice and have it show on my Google Home Hub as well. I’m also building a standalone Google Smart Home Action just for Camera feeds using an all Google stack just as a learning exercise in Firebase.

At the moment the stream is only available from inside my network, I’ll probably proxy it to my external web server as well and add on authentication. The Google Assistant can be given a Bearer Auth Token along with the URL which means I’ll be able to view the stream on my phone while out. While not important for this stream it would be for other Security camera type applications.

Building Custom Raspberry Pi SD Card Images

After my post about using a Raspberry Pi 4 as a USB gadget got linked to by a YouTuber who worked out it also worked with the iPad Pro it has been getting a lot of traffic.

Pi4 Gadget

Along with the traffic came a number of comments from people wanting help setting things up. While the instructions are reasonably complete, they do assume a certain amount of existing knowledge and access to a monitor/keyboard/network to complete everything. Also given the majority of the readers were Apple users they couldn’t mount the main partition of the SDCard as it is a ext4 file system.

The quick and dirty solution is for me to modify a standard image and host it somewhere. This is OK, but it means I have to,

  • Find somewhere to host a 500mb file
  • Keep it up to date when new versions are released
  • Provide some way for people to trust what changes I’ve made to the image

I can probably get round the first one pretty easily, bandwidth is a lot cheaper than it used to be. The second and third items are a little harder.

I started to look at a way to script the modifications to a standard Raspbian image, that way I could just host the script and people could provide their own starting image. On a Linux machine I could mount both partitions on the card and modify or add the required config files, but the problem was installing dnsmasq. This needs the image to actually be running for apt-get to run, which gets back to the chicken/egg problem of needing to boot the pi order to make the changes. That and it would only run on Linux, not a OSx or Windows machine.

What I need is a way to “run” a virtual Raspberry Pi and a way to run commands in that virtual machine. I know from working with services like Balena.io‘s build system that it is possible to emulate the ARM processor found in a Raspberry Pi on Intel based hardware. I found a couple of examples of people using Qemu to run virtual Pis and I was just about to set up the same when I came across dockerpi, which is a docker image with everything preconfigured. You can mount a SD card image

Virtual Raspberry Pi in a terminal window

When started you end up with what is basically the virtual machine console as a command line app. You can interact with it just like any other console application. I logged in as pi and then used sudo to run apt-get update and apt-get install dnsmasq.

That works for doing it manually, but I need to script this, so it’s time to break out some old school Linux foo and use expect.

Expect is a scripting tool that reads from stdin, and outputs to stdout, but will wait for a known output before sending a reply. It was used in the early days of the Internet to script dial up internet.

#!/usr/bin/expect -f
set timeout -1
set imageName [lindex $argv 0]
if {[string trimleft $imageName] eq ""} {
  puts "No Image file provided"
  exit
}
set cwd [file normalize [file dirname $argv0]]
set imagePath [file join $cwd $imageName]
spawn docker run -i --rm -v $imagePath:/sdcard/filesystem.img lukechilds/dockerpi:vm
expect "login: "
send "pi\n"
expect "Password: "
send "raspberry\n"
interact

This expect script takes the name of the SD Card image as an argument, starts the Docker container and then logs in with the default pi/raspberry username & password.

With a bit more work we can get all all the changes done including creating the extra files.

...
proc slurp {file} {
    set fh [open $file r]
    set ret [read $fh]
    close $fh
    return $ret
}
...
set file [slurp "etc/network/interfaces.d/usb0"]
expect "# "
send "cat <<EOF >> /etc/network/interfaces.d/usb0\n"
send "$file\n"
send "EOF\n"
...

I’ve checked all the files into a github repository here. I’ve tested the output with a Pi Zero and things look good. To run it for yourself, clone the repo, copy the Raspbian Lite image (unzip it first) into the directory and run ./create-image <image file name>

There is a version of the output from here.

I’ve still got to get round to trying to the RNDIS Ethernet device support working so it will load the right drivers on Windows. And I need to extend the script to build the Personal CA Appliance from my last post.

A Personal Offline Certificate Authority

I had a slight scare in the run up to Christmas. I went to use the VPN on my phone to connect into my home network and discovering that the certificate that identifies both my phone and the one for the server had expired the day before.

This wouldn’t have been a problem except I couldn’t find where I’d stashed the files that represent the CA I had used to create the certificates. There was a short panic until I got home that evening and found them on a old decommissioned server that luckily I hadn’t got round to scrapping properly yet.

This led me to think of a better place to store these files. I wanted to have a (relatively) secure offline place to store them, but also somewhere that could handle the actual signing of certificates and the rest of the admin (I normally end up Googling the instructions for openssl each time I need to do this).

A simple approach would be to just store the files on a encrypted USB Mass Storage device, but I wanted something a little bit more automated.

Hardware

Recycling my ever useful Raspberry Pi Zero as a USB Ethernet Gadget instructions again with a Raspberry Pi Zero (note not a Zero W) gets me a device that has no direct internet connection, but that can be plugged into nearly any machine and accessible via a local network connection.

RTC attached to a Raspberry Pi Zero in a Pimoroni case

One little niggle is that working with certificates requires an accurate clock on the device. Raspbian by defaults sets it’s clock via NTP over the network since there is no persistent clock on the pi. The fix for this is a i2c battery backed up hardware clock. You can pick these up from a number of places, but I grabbed one of these from amazon.

To enable the RTC you need to add the following to /boot/config.txt

dtoverlay=i2c-rtc,ds3231

And comment out the first if block in /lib/udev/hwclock-set

...
dev=$1
#if [ -e /run/systemd/system ] ; then
#    exit 0
#fi
...

Now we have a reliable system clock we can go about setting up the CA.

Software

The first version just requires me to ssh in to the pi and use openssl on the command line. This was enough to get me started again, but I was looking for something a bit more user friendly.

I had a look round for a web interface to openssl and found a few different options

But they all requires a whole bunch of other things like OpenLDAP, MySQL and Apache which is all a bit too heavy weight for a Pi Zero.

A web form collecting data for a certificate

So I decided to write my own, a bit of poking around and I found the node-openssl-cert module on npm which looked like it should be able to handle everything I need.

Combined with express and I now have a form I can fill in with the subject details and hit submit.

The page then downloads a PKCS12 format bundle which contains the CA cert, Client cert and Client key all protected by a supplied password. I can then use openssl to break out the parts I need or just import the whole thing into Android.

At the moment I’ve just copied the existing CA key & cert out of the existing CA directory structure to get this to work. I intend to update the code to make use of the serial number and index tracking so if needed I can generate a Certificate Revocation List if needed, also potentially allow the downloading of previously generated certs.

You can find the project on github here and I hope to find some time to write up some end to end instructions for setting it all up.

The interesting bit was how to download a file from a XMLHttpRequest, you can see that trick here.

Aside

I originally titled this as “A Secure Offline Certificate Authority”. I changed it because this isn’t really any more secure than a USB key you just keep the CA key & cert on, and probably less secure than if you encrypted that drive. It is true that the CA key cert are not accessible from the host machine without SSHing to the device, but the CA key & cert are still just stored on the Pi’s SDCard so if anybody has physical access to it then it’s game over.

I could look at i2c or SPI secure elements that could be used to store the private key but the real solution to this is an ASCI or FPGA combined with a secure element, but that is all overkill for what I needed here.

Getting out past the firewall

Ahhh, the joys of a IT departments that think everybody just uses Word/Excel/Outlook and just browses to Facebook at lunchtime.

Networks that transparently proxy HTTP/HTTPS (probably with man in the middle TLS CA certs deployed to all the machines, but that is an entirely different problem) and block everything else really do not work in the modern world where access to places like GitHub via SSH or devices connecting out via MQTT are needed.

One possible solution to the SSH problem is a bastion host. This is a machine that can be reached from the internal network but is also allowed to connect to the outside world. This allows you to use this machine as a jumping off point to reach services blocked by the firewall.

The simple way is to log into the bastion, and then from the shell connect on to your intended external host, this works for targets you want a shell on but not for things like cloning/updating git repositories. We also want to automate as much of this as possible.

The first step is to set up public/private key login for the bastion machine. To do this we first generate a key pair with the ssh-keygen command.

$ ssh-keygen -f ~/.ssh/bastion -t ecdsa -b 521
Generating public/private ecdsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in ~/.ssh/bastion.
Your public key has been saved in ~/.ssh/bastion.pub.
The key fingerprint is:
SHA256:3Cfr60QRNbkRHDt6LEUUcemhKFmonqDlgEETgZl+H8A hardillb@tiefighter
The key's randomart image is:
+---[ECDSA 521]---+
|oB+      ..+X*.. |
|= .E    . .o++o  |
|.o  .  . o..+= . |
|....o...o..=o..  |
|  .=.o..S.* +    |
|  . ..o  . *     |
|          o      |
|         o       |
|         .+.     |
+----[SHA256]-----+

In this case we want to leave the passphrase blank because we want to use this key as part of automation of other steps, normally you should use a passphrase to protect access should the keys be compromised.

Once generated you can copy it to the `~/.ssh/authorized_keys` file on the bastion machine using the ssh-copy-id command

$ ssh-copy-id -i ~/.ssh/bastion user@bastion

Once that is in place we should be able to use the key to log straight into the bastion machine. We can now use the `-J` option to specify the bastion as a jump point to reach a remote machine.

$ ssh -J user@bastion user@remote.machine

We can also add this as an entry in the `.ssh/config` file which is more useful for things like git where it’s harder to get at the actual ssh command line.

Host bastion
 HostName bastion
 User user
 IdentityFile ~/.ssh/bastion

Host github
 Hostname github.com
 User git
 IdentityFile ~/.ssh/github
 ProxyCommand ssh -W %h:%p bastion

This config will proxy all git commands working with remote repositories on github.com via the bastion machine, using the bastion key to authenticate with the bastion machine and the github key to authenticate with github. This is all totally transparent to git.

ssh also supports a bunch of other useful ticks, such as port forwarding from either end of the connection.

It also can proxy other protocols using the Socks tunnelling protocol which means it can be used as poor man’s VPN in some situations. To enable Socks proxying you can use -D option to give a local port number or the DynamixProxy directive in the ~/.ssh/config file. This option is really useful with web browser that supports Socks proxies as it means you can point the browser at a local port and have it surf the web as if it was the remote machine.

All of this still works if you are using a bastion machine.

Custom hardware

Combining all this really useful SSH capability with a the Raspberry Pi gadgets makes it possible to carry a bastion host with you. Using a Raspberry Pi Zero W or even a full sized Pi 4 that can be configured to join a more open WiFi network (e.g. a visitor or testing network) you can have a device you just plug into a spare USB port that will give you a jumping off point to the outside world while still being connected to the more restricted internal network with all that provides. Just don’t tell IT security ;-).

This works well because even the most locked down machine normally still allows USB network adapters to be used.