More MQTT VirtualHost Proxying

A really quick follow up to the earlier post about using TLS SNI to host multiple MQTT brokers on a single IP address.

In the previous post I used nginx to do the routing, but I have also worked out that the required input to Traefik would be.

The static config file looks like this

global:
  checkNewVersion: false
  sendAnonymousUsage: false
entryPoints:
  mqtts:
    address: ":1883"
api:
  dashboard: true
  insecure: true
providers:
  file:
    filename: config.yml
    directory: /config
    watch: true

And the dynamic config like this
tcp:
  services:
    test1:
      loadBalancer:
        servers:
          - address: "192.168.1.1:1883"
    test2:
      loadBalancer:
        servers:
          - address: "192.168.1.2:1883"
  routers:
    test1:
      entryPoints:
        - "mqtts"
      rule: "HostSNI(`test1.example.com`)"
      service: test1
      tls: {}
    test2:
      entryPoints:
        - "mqtts"
      rule: "HostSNI(`test2.example.com`)"
      service: test2
      tls: {}

tls:
  certificates:
    - certFile: /certs/test1-chain.crt
      keyFile: /certs/test1.key
    - certFile: /certs/test2-chain.crt
      keyFile: /certs/test2.key

Of course all the dymaic stuff can be generated via any of the Traefik providers.

Hostname Based Proxying with MQTT

An interesting question came up on Stack Overflow recently that I suggested a hypothetical answer for how to do hostname based proxying for MQTT.

In this post I’ll explore how to actually implement that hypothetical solution.

History

HTTP added the ability to do hostname based proxying when it instroduced the Host header in HTTP v1.1. This meant that a single IP address could be used for many sites and the server would decide which content to serve based on the his header. Front end reverse proxies (e.g. nginx) can use the same header to decide which backend server to forward the traffic to.

This works well until we need encrypt the traffic to the HTTP server using SSL/TLS where the headers are encrypted. The solution to this is to use the SNI header in the TLS handshake, this tells the server which hostname the client is trying to connect to. The front end proxy can then either use this information to find the right local copy of the certificate/key for that site if it’s terminating the encryption at the frontend or it can forward the whole connection directly to the correct backend server.

MQTT

Since the SNI header is in the initial TLS handshake and is nothing to do with the underlying protocol it can be used for ay protocol, in this case MQTT. This measn if we set up a frontend proxy that uses SNI to pick the correct backend server to connect to.

Here is a nginx configuration file that proxies for 2 different MQTT brokers based on the hostname the client uses to connect. It is doing the TLS termination at the proxy before forwarding the clear version to the backend.

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

stream {  
  map $ssl_server_name $targetBackend {
    test1.example.com  192.168.1.1:1883;
    test2.example.com  192.168.1.2:1883;
  }

  map $ssl_server_name $targetCert {
    test1.example.com /certs/test1-chain.crt;
    test2.example.com /certs/test2-chain.crt;
  }

  map $ssl_server_name $targetCertKey {
    test1.example.com /certs/test1.key;
    test2.example.com /certs/test2.key;
  }
  
  server {
    listen 1883         ssl; 
    ssl_protocols       TLSv1.2;
    ssl_certificate     $targetCert;
    ssl_certificate_key $targetCertKey;
        
    proxy_connect_timeout 1s;
    proxy_pass $targetBackend;
  } 
}

Assuming the the DNS entries for test1.example.com and test2.example.com both point to the host running nginx then we can test this with the mosquitto_sub command as follows

$ mosquitto_sub -v -h test1.example.com -t test --cafile ./ca-certs.pem

This will be proxied to the broker running on 192.168.1.1, where as

$ mosquitto_sub -v -h test2.example.com -t test --cafile ./ca-certs.pem

will be proxied to the broker on 192.168.1.2.

Cavets

The main drawback with this approach is that it requires that all the clients connect using TLS, but this is not a huge problem as nearly all devices are capable of this now and for any internet facing service it should be the default anyway.

Acknowledgment

How to do this was mainly informed by the following Gist

DNSSEC and Letsencrypt

A couple of tweets from a colleague over the Christmas period along with some jobs I’d been saving up made me have another look at the DNS and HTTPS set up for a couple of sites I look after.

DNSSEC

I’ve been meaning to play with DNSSEC for a while, especially since I run my own primary DNS and set up DMKIM to verify my mail server identity (yeah, I know in this day and age of cloud running all your own services is a little quaint, but I like to understand how every thing works).

This a good introduction to DNSSEC if you’re not up to speed. TL;DR DNSSEC allows you to tell when people have been messing with your DNS entries.

To set up DNSSEC you need to create 2 sets of keys, a zone signing key and a key signing key you can create them with the following commands respectively.

$ dnssec-keygen -a NSEC3RSASHA1 -b 2048 -n ZONE hardill.me.uk
Generating key pair..................+++ .............+++
Khardill.me.uk.+007+40400
$ dnssec-keygen -f KSK -a NSEC3RSASHA1 -b 4096 -n ZONE hardill.me.uk
Generating key pair....................................................................................................................................................................................................................................................++ ................................................................................++ 
Khardill.me.uk.+007+23880

Key generation requires a lot of random numbers and these are created from the /dev/random, the values for this are generated from the system entropy so can take a long time on a machine that isn’t doing very much, to help with this I can installed the haveged daemon.

Now I have the 2 sets of keys (public and private) I need to add them to the end of my zone file with the following lines:

$INCLUDE Khardill.me.uk.+007+43892.key
$INCLUDE Khardill.me.uk.+007+23880.key

Now we can use these keys to actually sign the zone with the dnssec-signzone command, the NSEC3 setup takes a salt to help with security. The $(head -c 1000 /dev/random | sha1sum | cut -b 1-16) generates a 16 character random string to act as the salt.

$ dnssec-signzone -A -3 $(head -c 1000 /dev/random | sha1sum | cut -b 1-16) -N INCREMENT -o hardill.me.uk -t hardill.me.uk.db
Verifying the zone using the following algorithms: NSEC3RSASHA1.
Zone fully signed:
Algorithm: NSEC3RSASHA1: KSKs: 1 active, 0 stand-by, 0 revoked
                         ZSKs: 1 active, 0 stand-by, 0 revoked
hardill.me.uk.db.signed
Signatures generated:                       25
Signatures retained:                         0
Signatures dropped:                          0
Signatures successfully verified:            0
Signatures unsuccessfully verified:          0
Signing time in seconds:                 1.129
Signatures per second:                  22.143
Runtime in seconds:                      1.274

This generates 2 files, the first is hardill.me.uk.db.signed which is an updated version of the zone file with the signed hashes included for each entry. The second is dsset-hardill.me.uk. which holds the DS hashes for my 2 keys. The DS entries are hosted by the layer above my domain in the DNS hierarchy so that anybody wanting to verify the data can walk from the Signed root zone up the tree checking the level above before moving on. To get the DS entries into the zone above you normally have to go through your Domain Name Registrar who would in this case ask Nominet (as the keep of the me.uk domain) to host them for me, unfortunately my registrar (I won’t name them here) claims unable to be able pass this request on to Nominet. I need to see if I can get Nominet to do it for me, but I’m not confident so I’m currently in the market for a new registrar, any recommendations welcome.

In the mean time I decided to test the rest of it out on the private TLD I run on my lan. I can get round the need for a DS record by telling Bind to trust my key explicitly using the trusted-keys directive in named.conf. To get this far I followed this set of instructions, which are the manual steps for DNSSEC, there are also instructions to get Bind to automatically sign zones, which is especially useful if you are doing Dynamic DNS updates, this page has instructions for that which I’ll be looking at once I get things sorted to have my DS records hosted properly.

Letsencrypt

The letsencrypt project has a goal to provide free SSL certificates for everybody that are signed by a CA in the collection commonly included in modern browsers. It had been in private beta most of last year, but went into public beta at the start of December so I could sign up. Letsencrypt will generate you a certificate for any domain you can prove you own, you do this using a protocol called ACME and they have written a client to help with this. ACME works over HTTP/HTTPs by placing a hash value at a known location. This can be via an existing HTTP server (e.g. Apache) or by a one built into the client. At home I run my own private CA as it allows me to issue certificates for names on my private TLD and for my IP addresses. I also issue client certificates to authenticate users and having them all with the same CA makes things a little easier. When I get some time I will probably move my domain over to a letsencrypt certificate and only use my CA for client certs. In the mean time I needed to set up access to my Dad’s work mail server so my Brother can send/receive email from his iPhone, this needed to be secure so everything needs to be protected by a certificate. Rather than mess about getting the root CA certs for my private CA on to his phone I decided to use letsencrypt. The mail server doesn’t run a webserver so I used the one built into the client.

$ letsencrypt-auto certonly --standalone  --email admin@example.com --agree-tos -d mail.example.com

The command line arguments are as follows

  • certonly – This tells the client to download the certificate (rather than download it and install it
  • –standalone – This tells the client to use it’s built in HTTP server
  • –email admin@example.com – This tells the client who to email if there is a problem (like a cert expires without being renewed)
  • –agree-tos – This stops the client showing the TOS and prompting you to agree to them
  • -d mail.example.com – This tells the client which host name to create the certificate for, you can specify multiple instances

Certificates and Keys are stored under /etc/letsencrypt/ with the current cert under live/[host name]. I configured Postfix and Dovecot to point to these so that they just need to be restarted to pick up the new certs.

Letsencrypt hand out certificates that are only valid for 90days, this is for a couple of reasons, but mainly it means that any compromised certs only expose people for a short time and they can upgrade the supported algorithms/key strength regularly to keep ahead of new vulnerabilities. The downside to this is that you need to renew the certificate regularly. The client is actually pretty good at letting you automate things using a very similar command to the original version. I’ve set up a cron job to run on the first of every second month that renews a new cert every 60ish day and then restart Postfix and Dovecot, this gives plenty of time to fix anything should there be a problem.

25 15 1 1,3,5,7,9,11 * /home/admin/renew-cert.sh
#!/bin/sh
/home/admin/letsencrypt/letsencrypt-auto certonly --standalone --renew-by-default --email admin@example.com --agree-tos -d mail.example.com
sudo service dovecot restart
sudo service postfix restart

I had to add the following to the sudoers file to get everything to work without prompting for passwords

admin ALL= NOPASSWD: /home/admin/letsencrypt/letsencrypt-auto 
admin ALL= NOPASSWD: /usr/bin/service postfix *
admin ALL= NOPASSWD: /usr/bin/service dovecot *