Creating Fail2Ban rules

I recently installed some log charting software to get a view NGINX logs for the instance that servers up this site. While looking at some of the output I found some strange results so went to have a closer look at the actual logs

Chart showing hits/visitor to site

There were a whole bunch of entries with with a similar pattern, something similar to the following:

94.130.58.174 - - [15/Jul/2020:11:43:39 +0100] "CONNECT rate-market.ru:443 HTTP/1.1" 400 173 "-" "-"

CONNECT is a HTTP verb that is used by HTTP Proxy clients to tell the proxy server to set up a tunnel to the requested machine. This is needed because is the client is trying to connect to a HTTP server that uses SSL/TLS then it needs to be able to do the SSL/TLS handshake directly with that server to ensure it has the correct certificates.

Since my instance of NGINX is not configured to be a HTTP Proxy server it is rightly rejecting these requests as can been seen by the 400 (Bad Request) after the request string. Even though the clients making these request are all getting the 400 error they are continuing to send the same request over and over again.

I wanted to filter these bogus request out for 2 reasons

  • They clog the logs up, making it harder to see real problems
  • They take up resources to process

The solution is to use a tool called fail2ban, which watches log files for signs of abuse and then sets up the correct firewall rules to prevent the remote clients from being able to connect.

I already run fail2ban monitoring the SSH logs for all my machines that face the internet as there are plenty of scripts out there running trying standard username/password combinations attempting to log into any machine listening on port 22.

Fail2ban comes with a bunch of log filters to work will a applications, but the ones bundled on raspbian (RaspberryOS) at this time didn’t include one that would match this particular pattern. Writing filters is not that hard but it does require writing regular expressions which is a bit of dark art. The trick is to get a regex that only matches what we want.

The one I came up with is this:

^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400

Let’s break that down into it’s specific parts

  • ^ this matches the start of the log line
  • <HOST> fail2ban has some special tags that match an IP address or Host name
  • - - this matches the 2 dashes in the log
  • \[.*\] this matches the date/timestamp in the square brackets, I could have matched the date more precisely, but we don’t need to in this case. We have to escape the [ as it has meaning in regex as we will see later.
  • \"CONNECT this matches the start of the request string and limits it to just CONNECT requests
  • .* this matches the host:port combination that is being requested
  • HTTP/1\.[0-1]\" here we match both HTTP/1.0 and HTTP/1.1. We need to escape the . and the square brackets let us specify a range of values
  • 400 and finally the 400 error code

Now we have the regex we can test it with the fail2ban-regex command to make sure it finds what we expect.

# fail2ban-regex /var/log/nginx/access.log '^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400' 

Running tests
=============

Use   failregex line : ^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400
Use         log file : /var/log/nginx/access.log
Use         encoding : UTF-8


Results
=======

Failregex: 12819 total
|-  #) [# of hits] regular expression
|   1) [12819] ^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [526165] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 526165 lines, 0 ignored, 12819 matched, 513346 missed
[processed in 97.73 sec]

Missed line(s): too many to print.  Use --print-all-missed to print all 513346 lines

We can see that the regex matched 12819 lines out of 526165 which is in the right ball park. Now we have the right regex we can build the fail2ban config files.

First up is the filter file that will live in /etc/fail2ban/filter.d/nginx-connect.conf

# Fail2Ban filter to match HTTP Proxy attempts
#

[INCLUDES]

[Definition]

failregex = ^<HOST> - - \[.*\] \"CONNECT .* HTTP/1\.[0-1]\" 400

ignoreregex = 

next is the jail file in /etc/fail2ban/jail.d/nginx-connect.conf, this tells fail2ban what to do if it finds log lines that match the filter and where to find the logs.

[nginx-connect]
port	= http,https
enabled	= true
logpath = %(nginx_access_log)s
action	= %(action_mwl)s

The port entry is which ports to block in the firewall, enabled & logpath should be obvious (it’s using a pre-configured macro for the path). And finally the action, there are a number of pre-configured actions that range from just logging the fact that there were hits in the log all the way through to this on action_mwl, which adds a firewall rule to block the <HOST>, then emails me that with information about the IP address and a selection of the matching log lines.

I’ll probably reduce the action to one that just adds the firewall rule after a day or two when I have a feel for how much of this sort of traffic I’m getting. But the info about each IP address normally include a lookup of the owner and that usually has an email address to report problems. It’s not often that reporting things actually stops it happening, but it’s worth giving it ago at the start to see what happens.

Over the course of the afternoon since it was deployed the filter has blocked 148 IP addresses.

My default rules are a 12 hour ban if the same IP address triggers more than 3 times in 10 mins with a follow up of getting a 2 week ban if you trip that more than 3 times in 3 days.

Next up is to create a filter to match the GET https://www.instagram.com requests that are also polluting the logs.

64bit Raspberry Pi OS build with USB-C Ethernet enabled

As a follow up to the 32bit version I built, using the script I described in my post about building custom SD card images, I have created a 64bit version based on the Raspberry Pi OS beta.

You can download the 64bit version from here. This is a full Raspberry Pi OS image, not a “lite” version as only full versions are available in beta.

This is a manual build (following these instructions) because at the moment the script won’t work with the 64bit version because DockerPi doesn’t support USB devices when emulating 64bit CPUs. USB is needed to attach the emulated Network Adapter. There are a set of patches for QUEMU pending that should enable this.

As soon as the patches become generally available I’ll update the project on github.

Raspberry Pi Streaming Camera

I talked about using a ONVIF camera to stream to a Chromecast earlier because they come with an open well documented interface for pulling video from them (as well as pan/tilt/zoom control if available).

If you don’t have a camera that supports ONVIF you can build something similar with a Raspberry Pi and the Camera module.

This should work with pretty much all of the currently available Raspberry Pi models (With the exception of the basic Pi Zero that doesn’t have Wifi)

  1. Flash a SD card with the Raspbian Lite image
  2. Insert the camera ribbon cable into both the camera module and the Pi
  3. Once the card has booted use the raspi-conf command to enable the Camera interface
  4. Install ffmpeg sudo apt-get install ffmpeg
  5. Create a script with the following content
#!/bin/sh

v4l2-ctl --set-ctrl video_bitrate=300000

ffmpeg -f video4linux2 -input_format h264 -video_size 640x360 -framerate 30 -i /dev/video0  -vcodec copy -an -f flv rtmp://192.168.1.96/show/pi
  • This script sets the max video bitrate to 30kps
  • If you need to rotate the video you can insert v4l2-ctl --set-ctrl=rotate=180 before ffmpeg to rotate 180 degrees
  • ffmpeg uses the videolinux2` driver to read from the attached camera (/dev/video0)
  • Takes h264 encoded feed at 640x360 and 30 frames per second and outputs it to the same nginx instance that I mentioned in my previous post. The feed is called pi

ffmpeg uses the on board hardware support for the video encoding so even a Pi Zero W runs at about 5% CPU load. This means that if you only have 1 camera you could probably run nginx on the same device, else you can have a multiple cameras all feeding to a central video streaming server.

If you want a kit that comes with the Pi Zero W, Camera and a case to mount it to a window have a look at the Pimoroni OctoCam.

The instructions should also work for pretty much any USB (or built in) camera attached to a Linux machine.

Streaming Camera to Chromecast

I have a little cheap WiFi camera I’ve been meaning to do something with for a while. The on board web access doesn’t really work any more because it only supports Flash or Java Applet to view the stream.

But camera supports the ONVIF standard so it offers a rtsp:// feed so I can point Linux apps like mplayer at it and see the stream.

The camera is currently sat on my window sill looking out over the valley which is a pretty good view.

View from upstairs window

I thought it would be interesting to stream the view to the TV in my Living room while I’m working from home at the moment. It is also a way to check the weather without having to get up in the morning and open the blind.

I have a Chromecast in the back back of both TVs so using this seamed like it would be the easiest option.

Chromecast

Chromecasts support a number of different media types but for video we have 2 common codec that will work across all the currently available device types.

  • H264
  • VP8

And we have 3 options to deliver the video stream

  • HLS
  • DASH
  • SmoothStreaming

These are all basically the same, they chop the video up into short segments and generate a play list that points to the segments in order and the consumer downloads each segment. When it reaches the end of the list it downloads the list again which will now hold the next list of segments.

There is a plugin for Nginx that supports both HLS and DASH which looked like a good place to start.

NGINX

I’m running this whole stack on a Raspberry Pi 4 running Raspbian Buster.

$ sudo apt-get install nginx libnginx-mod-rtmp ffmpeg

Once the packages are installed the following needs to be added to the end of the /etc/nginx/nginx.conf file. This sets up a rtmp listener that we can stream the video to which will then be turned into both a HLS and DASH stream to be consumed.

...
rtmp {
  server {
    listen 1935; # Listen on standard RTMP port
    chunk_size 4000;

    application show {
      live on;
      # Turn on HLS
      hls on;
      hls_type live;
      hls_path /var/www/html/hls/;
      hls_fragment 5s;
      hls_playlist_length 20s;
      
      # Turn on DASH      
      dash on;
      dash_path /var/www/html/dash/;
      dash_fragment 5s;
      dash_playlist_length 20s;

      # disable consuming the stream from nginx as rtmp
      deny play all;
    }
  }
}

The playlist and video segments get written to /var/www/html/hls and /var/www/html/dash respectively. Because they will be short lived and replaced very regularly it’s a bad idea to write these to an SD card as they will just cause excessive flash wear.

To get round this I’ve mounted tmpfs filesystems at those points with the following entries in /etc/fstab

tmpfs	/var/www/html/dash	tmpfs	defaults,noatime,size=50m
tmpfs	/var/www/html/hls	tmpfs	defaults,noatime,size=50m

Now we have the playlists and segments being generated in a sensible way we need to server them up. I added the following to the /etc/nginx/sites-enabled/default file

server {
  listen 8080;
  listen [::]:8080;

  sendfile off;
  tcp_nopush on;
  directio 512;
  default_type application/octet-stream;

  location / {
    add_header 'Cache-Control' 'no-cache';
    add_header 'Access-Control-Allow-Origin' '*' always;
    add_header 'Access-Control-Allow-Credentials' 'true';
    add_header 'Access-Control-Expose-Headers' 'Content-Length';

    if ($request_method = 'OPTIONS') {
      add_header 'Access-Control-Allow-Origin' '*';
      add_header 'Access-Control-Allow-Credentials' 'true';
      add_header 'Access-Control-Max-Age' 1728000;
      add_header 'Content-Type' 'text/plain charset=UTF-8';
      add_header 'Content-Length' 0;
      return 204;
    }

    types {
      application/dash+xml mpd;
      application/vnd.apple.mpegurl m3u8;
      video/mp2t ts;
    }

    root /var/www/html/;
  }
}

Now we have the system to stream the content in an acceptable format we need to get the video from the camera into nginx. We can use ffmpeg to do this.

ffmpeg -re -rtsp_transport tcp -i rtsp://192.168.1.104:554/live/ch1 -vcodec libx264 -vprofile baseline -acodec aac -strict -2 -f flv rtmp://localhost/show/stream

This reads from the RTSP stream rtsp://192.168.1.104:554/live/ch1 and streams it into the rtmp://localhost/show/stream. The showpart is the name of the application declared in the rtmp section in the nginx.conf and the stream will be the name of the HLS or DASH stream. In this case the following:

  • HLS -> http://192.168.1.98/hls/stream.m38u
  • DASH -> http://192.168.1.98/dash/stream.mpd

If you change the end of the `rtmp://localhost/show/XXXX` URL you can create multiple streams from different sources with different names (just make sure the tmpfs mounts have enough space for all the streams).

Testing

Node-RED Chromecast flow

I’ve been using the Node-RED Chromecast node to test the streams. The DASH stream is working pretty well, but the HLS is a bit more fragile for some reason. Latency is currently about 20-30 seconds which is appears mainly to be determined by the size and number of chunks in the playlist used but if I wind the fragment size down any lower than 5s or the 20s for the playlist length.

Next

Now it’s basically working the next steps are to add support for Camera devices to my Google Assistant Node-RED service so I can request the stream via voice and have it show on my Google Home Hub as well. I’m also building a standalone Google Smart Home Action just for Camera feeds using an all Google stack just as a learning exercise in Firebase.

At the moment the stream is only available from inside my network, I’ll probably proxy it to my external web server as well and add on authentication. The Google Assistant can be given a Bearer Auth Token along with the URL which means I’ll be able to view the stream on my phone while out. While not important for this stream it would be for other Security camera type applications.

Node-RED Google Home Smart Home Action Generally Available

I started this post back in November 2017, it’s been a long slog, but we are finally here. We had a false start back in January 2019 when I got the bulk of the code all working and I thought it would take a few weeks get it certified and released and again at the beginning of this year when the Action got approved but still required you to be a member of a Google Group to be able to sign up . Unfortunately that wasn’t the case. But all that is behind us now.

A Node-RED flow with lots of Google Home Assistant nodes

You can find the full docs for how to install and configure the Action here, but the short version is:

  • Create an account here
  • Create some devices using the wizard
  • Link the NR-GAB Action to your Google Account in the Google Home app, it should have the following icon.
  • Install the node-red-contrib-googlehome node in Node-RED
  • Drag the googlehome-in node on to the canvas and start building flows that can be triggered by the Google Assistant.

As always the doc probably needs a little bit more work, so I’ll keep updating it as folks run into issues with it. If you have any questions the best place to ask them is in the #google-home-assistant channel on the Node-RED Slack team.

Next steps are I have a working version of the Google Assistant Local Control API that I will release as soon as Google open that up for general availability. This sends commands directly to Node-RED from the Smart Speaker device which reduces the latency in triggering actions.

Contributing to Accel-PPP

I mentioned in a previous post about my desktop ISP project I was using the accel-ppp access concentrator. This provides the main components for allowing my “users” to connect to my network, it handles all the authentication and setting up of PPPoE connections.

While playing with it I wanted to look at settings different DNS servers for different users. For IPv4 this was already built in, you could put entries in the LDAP that get passed on via the Radius lookup when a user connects. But with my messing about with IPv6 only networks I also wanted to set custom IPv6 DNS servers on a per user basis.

You can set a global set of DNS servers in the config file

...
[ipv6-dns]
dns=fd12:3456:789a::1
dns=fd12:3456:789a::2
...

The Radius spec has an entry for holding IPv6 DNS server values (DNS-Server-IPv6-Address from RFC6911), but accel-ppp didn’t support using it. I mentioned it on the accel-ppp forum as a feature request and the developers seemed to think it would be a good idea, so I decided to have a go at implementing it.

First up adding the required bits to the LDAP for the test user, here I have one IPv4 DNS and one IPv6 DNS.

displayName: Ben Hardill
cn: Ben
sn: Hardill
mail: isp1@hardill.me.uk
uid: isp1
radiusReplyAttribute: MS-Primary-DNS-Server := "192.168.5.1"
radiusReplyAttribute: Reply-Message := "Hello World"
radiusReplyAttribute: Delegated-IPv6-Prefix := "fd12:3456:789a:2::/64"
radiusReplyAttribute: Framed-IPv6-Prefix := "fd12:3456:789a:0:192:168:5:2/128"
radiusReplyAttribute: DNS-Server-IPv6-Address := "fd12:3456:789a:ff64::1"
radiusFramedIPAddress: 192.168.5.2
radiusFramedIPNetmask: 255.255.255.0

My first pass was a little naive in that it ended up changing the DNS servers for all subsequent users by overwriting the global settings. It also only supported providing 1 DNS server when the DHCPv6 and RA both support up to 3. So it was back to the drawing board.

First up is to generate a linked list containing the DNS server addresses from the Radius response.

...
			case Framed_IPv6_Route:
				rad_add_framed_ipv6_route(attr->val.string, rpd);
				break;
			case DNS_Server_IPv6_Address:
				a = _malloc(sizeof(*a));
				memset(a, 0, sizeof(*a));
				a->addr = attr->val.ipv6addr;
				list_add_tail(&a->entry, &rpd->ipv6_dns.addr_list);
				break;
		}
	}
...

This gets bound to the users session object so it can be retrieved later. In this case when responding to a DHCPv6 request.

...
	if (!list_empty(&ses->ipv6_dns->addr_list)) {
		list_for_each_entry(dns, &ses->ipv6_dns->addr_list, entry) {
			j++;
		}
		if (j >= 3) {
			j = 3;
		}
		opt1 = dhcpv6_option_alloc(reply, D6_OPTION_DNS_SERVERS, j * sizeof(addr));
		addr_ptr = (struct in6_addr *)opt1->hdr->data;
		list_for_each_entry(dns, &ses->ipv6_dns->addr_list, entry) {
			if (k < j) {
				memcpy(addr_ptr, &dns->addr, sizeof(addr));
				k++;
				addr_ptr++;
			} else {
				break;
			}
		}
...

Now this is not as clean as I would like since we have to walk the list to know how many DNS servers there are in order to allocate the right size chunk of memory to hold the addresses and then walk it again to copy the addresses into the response. I might go back and add the list length to the session structure and update it when adding items to the list so we can avoid this step.

Testing

While working on these changes I wanted something a bit quicker than having to deploy the changes to my physical test rig (a collection of Raspberry Pi and switched hidden under my sofa). To help with this I spun up a VM with a copy of CORE. CORE comes from the NRL and allows you to build virtual networks and run code on nodes within the network.

CORE emulating a simple network

This means I can start and stop the access coordinator really quickly and spin up as many clients as I want. I can also easily drop in L2TP clients (which use the same Radius and PPP infrastructure in accel-ppp) to check it works there as well. It also lets you attach tools like tcpdump and wireshark to capture packets at any point in the network which can be difficult with home grade switches (enterprise switches tend to have admin interfaces and the ability to nominate ports as taps that can see all traffic).

I have submitted a pull request against the project, so now fingers crossed it gets accepted.

Building Custom Raspberry Pi SD Card Images

After my post about using a Raspberry Pi 4 as a USB gadget got linked to by a YouTuber who worked out it also worked with the iPad Pro it has been getting a lot of traffic.

Pi4 Gadget

Along with the traffic came a number of comments from people wanting help setting things up. While the instructions are reasonably complete, they do assume a certain amount of existing knowledge and access to a monitor/keyboard/network to complete everything. Also given the majority of the readers were Apple users they couldn’t mount the main partition of the SDCard as it is a ext4 file system.

The quick and dirty solution is for me to modify a standard image and host it somewhere. This is OK, but it means I have to,

  • Find somewhere to host a 500mb file
  • Keep it up to date when new versions are released
  • Provide some way for people to trust what changes I’ve made to the image

I can probably get round the first one pretty easily, bandwidth is a lot cheaper than it used to be. The second and third items are a little harder.

I started to look at a way to script the modifications to a standard Raspbian image, that way I could just host the script and people could provide their own starting image. On a Linux machine I could mount both partitions on the card and modify or add the required config files, but the problem was installing dnsmasq. This needs the image to actually be running for apt-get to run, which gets back to the chicken/egg problem of needing to boot the pi order to make the changes. That and it would only run on Linux, not a OSx or Windows machine.

What I need is a way to “run” a virtual Raspberry Pi and a way to run commands in that virtual machine. I know from working with services like Balena.io‘s build system that it is possible to emulate the ARM processor found in a Raspberry Pi on Intel based hardware. I found a couple of examples of people using Qemu to run virtual Pis and I was just about to set up the same when I came across dockerpi, which is a docker image with everything preconfigured. You can mount a SD card image

Virtual Raspberry Pi in a terminal window

When started you end up with what is basically the virtual machine console as a command line app. You can interact with it just like any other console application. I logged in as pi and then used sudo to run apt-get update and apt-get install dnsmasq.

That works for doing it manually, but I need to script this, so it’s time to break out some old school Linux foo and use expect.

Expect is a scripting tool that reads from stdin, and outputs to stdout, but will wait for a known output before sending a reply. It was used in the early days of the Internet to script dial up internet.

#!/usr/bin/expect -f
set timeout -1
set imageName [lindex $argv 0]
if {[string trimleft $imageName] eq ""} {
  puts "No Image file provided"
  exit
}
set cwd [file normalize [file dirname $argv0]]
set imagePath [file join $cwd $imageName]
spawn docker run -i --rm -v $imagePath:/sdcard/filesystem.img lukechilds/dockerpi:vm
expect "login: "
send "pi\n"
expect "Password: "
send "raspberry\n"
interact

This expect script takes the name of the SD Card image as an argument, starts the Docker container and then logs in with the default pi/raspberry username & password.

With a bit more work we can get all all the changes done including creating the extra files.

...
proc slurp {file} {
    set fh [open $file r]
    set ret [read $fh]
    close $fh
    return $ret
}
...
set file [slurp "etc/network/interfaces.d/usb0"]
expect "# "
send "cat <<EOF >> /etc/network/interfaces.d/usb0\n"
send "$file\n"
send "EOF\n"
...

I’ve checked all the files into a github repository here. I’ve tested the output with a Pi Zero and things look good. To run it for yourself, clone the repo, copy the Raspbian Lite image (unzip it first) into the directory and run ./create-image <image file name>

There is a version of the output from here.

I’ve still got to get round to trying to the RNDIS Ethernet device support working so it will load the right drivers on Windows. And I need to extend the script to build the Personal CA Appliance from my last post.

A Personal Offline Certificate Authority

I had a slight scare in the run up to Christmas. I went to use the VPN on my phone to connect into my home network and discovering that the certificate that identifies both my phone and the one for the server had expired the day before.

This wouldn’t have been a problem except I couldn’t find where I’d stashed the files that represent the CA I had used to create the certificates. There was a short panic until I got home that evening and found them on a old decommissioned server that luckily I hadn’t got round to scrapping properly yet.

This led me to think of a better place to store these files. I wanted to have a (relatively) secure offline place to store them, but also somewhere that could handle the actual signing of certificates and the rest of the admin (I normally end up Googling the instructions for openssl each time I need to do this).

A simple approach would be to just store the files on a encrypted USB Mass Storage device, but I wanted something a little bit more automated.

Hardware

Recycling my ever useful Raspberry Pi Zero as a USB Ethernet Gadget instructions again with a Raspberry Pi Zero (note not a Zero W) gets me a device that has no direct internet connection, but that can be plugged into nearly any machine and accessible via a local network connection.

RTC attached to a Raspberry Pi Zero in a Pimoroni case

One little niggle is that working with certificates requires an accurate clock on the device. Raspbian by defaults sets it’s clock via NTP over the network since there is no persistent clock on the pi. The fix for this is a i2c battery backed up hardware clock. You can pick these up from a number of places, but I grabbed one of these from amazon.

To enable the RTC you need to add the following to /boot/config.txt

dtoverlay=i2c-rtc,ds3231

And comment out the first if block in /lib/udev/hwclock-set

...
dev=$1
#if [ -e /run/systemd/system ] ; then
#    exit 0
#fi
...

Now we have a reliable system clock we can go about setting up the CA.

Software

The first version just requires me to ssh in to the pi and use openssl on the command line. This was enough to get me started again, but I was looking for something a bit more user friendly.

I had a look round for a web interface to openssl and found a few different options

But they all requires a whole bunch of other things like OpenLDAP, MySQL and Apache which is all a bit too heavy weight for a Pi Zero.

A web form collecting data for a certificate

So I decided to write my own, a bit of poking around and I found the node-openssl-cert module on npm which looked like it should be able to handle everything I need.

Combined with express and I now have a form I can fill in with the subject details and hit submit.

The page then downloads a PKCS12 format bundle which contains the CA cert, Client cert and Client key all protected by a supplied password. I can then use openssl to break out the parts I need or just import the whole thing into Android.

At the moment I’ve just copied the existing CA key & cert out of the existing CA directory structure to get this to work. I intend to update the code to make use of the serial number and index tracking so if needed I can generate a Certificate Revocation List if needed, also potentially allow the downloading of previously generated certs.

You can find the project on github here and I hope to find some time to write up some end to end instructions for setting it all up.

The interesting bit was how to download a file from a XMLHttpRequest, you can see that trick here.

Aside

I originally titled this as “A Secure Offline Certificate Authority”. I changed it because this isn’t really any more secure than a USB key you just keep the CA key & cert on, and probably less secure than if you encrypted that drive. It is true that the CA key cert are not accessible from the host machine without SSHing to the device, but the CA key & cert are still just stored on the Pi’s SDCard so if anybody has physical access to it then it’s game over.

I could look at i2c or SPI secure elements that could be used to store the private key but the real solution to this is an ASCI or FPGA combined with a secure element, but that is all overkill for what I needed here.

Problems with a IPv6 only network

In my last post I talked about running a pure IPv6 network, as part of my ISP building project, but still allowing access to resources on the internet currently only available via IPv4.

This works well assuming all the clients on the local network are IPv6 capable, unfortunately this is not always the case. There are legacy devices that do not understand IPv6.

This is a real problem with IoT devices that are either no longer being maintained or just that have hardware that is incapable of using anything other than IPv4. There is also a small problem that a IP cam with a IPv6 address is probably available to the world with out some firewall rules or a ACL limiting access to the local /64, but those are problems for another day…

Another issue is hard coded IPv4 addresses in legacy applications, this is a problem even if the OS/device supports both IPv4 & IPv6 but is only connected via IPv6.

There is are a few of solution to both these problems.

Dual Stack networks

The simplest is to just run a dual stack network supplying both IPv4 & IPv6 all the way from the end device to the edge of the ISPs network. While this works it either means using lots of IPv4 addresses in the ISPs internal network and probably 2 layers of NAT (one at the customers router and then CGNAT at the edge of the ISPs network) assuming the ISP is not handing out publicly routed IPv4 addresses directly to customers.

464XLAT

464XLAT -> 4 to 6 to 4 X transLATion and the X can be either a Client or Provider.

464CLAT comes in 2 main types

464CLAT – on host

464CLAT works by doing the conversion from IPv4 to IPv6 on the client. This works for sites that are only available via IPv4 and for hard coded IPv4 addresses. It all happens in the TCP/IP stack on the host OS. For this to work the OS needs a way to work out what the DNS64 prefix is.

This paper discusses methods for a client to determine the NAT64 prefix. It suggests that the simple method of looking up the IPv6 address of a hostname known to only have a IPv4 entry ipv4only.arpa and using returned address (This behaviour is described in rfc7050).

There is an important step to validate that the address is returned is valid, because DNS64 breaks DNSSEC it takes a few extra steps.

  • Make a IPv6 (AAAA) look up for ipv4only.arp with DNSSEC checking disabled
  • Make a reverse (PTR) lookup of the IPv6 address 64:ff9b::c000:ab
  • This should return a pointer to a fully qualified domain name for a host in the ISPs domain (nat64.hardill.me.uk).
  • Finally make a IPv6 (AAAA) lookup for this hostname, which should return the same IPv6 address and a valid DNSSEC signature (RRSIG AAAA) record signed by the ISPs domain keys.

All of the required parts for this are running on my test network, I just need to attach some client devices to see if it works.

464CLAT – on router (Stateless translation)

This is a form of dual stack network, but it limits the dual stack to just the customer side of their router.

In this case the client network runs both native IPv6 and a RFC1918 range IPv4 and the clients router does the conversion by adding the IPv6 prefix. As well as running DHCPv6/SLAAC for IPv6 addressing it also runs DHCPv4 and hands out IPv4 addresses.

This works for devices that can’t support IPv6 (e.g. cheap IoT devices) and you still get to run a IPv6 only network in the ISP network.

The router generates it’s own prefix from the publicly routed IPv6 range it’s been allocated for it’s client side network. It then uses this prefix to generate 1 to 1 IPv6 addresses for each IPv4 address it hands out.

Jool can also be used to handle this in SIIT mode, you have to tell it what the input IPv4 range is and the IPv6 prefix it can work with.

#!/bin/sh
modprobe jool_siit
jool_siit instance add "example" --iptables
jool_siit -i "example" eamt add fd12:3456:789a:ff46:2::/96 10.99.0.0/24
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv6.conf.ppp0.accept_ra=2
sysctl -w net.ipv6.conf.all.forwarding=1
ip6tables -t mangle -A PREROUTING -j JOOL_SIIT --instance "example"
iptables  -t mangle -A PREROUTING -j JOOL_SIIT --instance "example"

Here I’ve hardcoded the IPv6 prefix, but for a real deployment we would need to use a delegated IPv6 prefix that is routed to the gateway. This should be run using the script option for dhcp6c when the prefixes are delegated.

interface ppp0 {
	send ia-na 0;
	send ia-pd 0;
	script "/etc/dhcp6c/reply.sh";
};
id-assoc na {
};
id-assoc pd 0 {
  prefix-interface eth1 {
    sla-len 0;
    sla-id 1;
  };
};

This config takes the /64 prefix and assigns it to eth1 for clients to use as part of SLAAC. I’m still working out how to extract the second /96 prefix and pass it to the script.

We can then update the script that starts jool as follows and point to it in the dhcp6c.conf file so it gets run once the network is up.

#!/bin/sh
#IPv6=`ip -o -6 a show dev eth1 | awk '/global/ { print $4 }'`
#PREFIX=`subnetcalc $IPv6 -n | awk '/Network/ { print $3}'`
IPv4=`ip -4 -o a show dev eth1 | awk '/global/ {print $4}'`
SUBNET=`subnetcalc $IPv4 -n | awk '/Network/ {print $3}'`
#need to make sure this only runs once
modprobe jool_siit
jool_siit instance add "example" --iptables
jool_siit -i "example" eamt add fd12:3456:789a:ff46:2::/96 $SUBNET/24
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1
ip6tables -t mangle -A PREROUTING -j JOOL_SIIT --instance "example"
iptables  -t mangle -A PREROUTING -j JOOL_SIIT --instance "example"

464PLAT

This is usually just the NAT64 we talked about in the last post running in the ISPs network.