dns-to-mdns

Having built the nginx-proxy-avahi-helper container to expose proxied instances as mDNS CNAME entries on my LAN I also wanted a way to allow these containers to also be able to resolve other devices that are exposed via mDNS on my LAN.

By default the Docker DNS service does not resolve mDNS hostnames, it either takes the docker hosts DNS server settings or defaults to using Goolge’s 8.8.8.8 service.

The containers themselves can’t do mDNS resolution unless you set their network mode to host mode which isn’t really what I want.

You can pass a list of DNS servers to a docker container when it’s started with the --dns= command line argument which means that if I can run a bridge that will convert normal DNS requests into mDNS resquests on my LAN I should be able to get the containers to resolve the local devices.

I’ve played with writing DNS proxies before when I was looking at DoH so I had a reasonably good idea where to start.

const dgram = require('dgram')
const dnsPacket = require('dns-packet')
const mdnsResolver = require('mdns-resolver')

const port = process.env["DNS_PORT"] || 53
const server = dgram.createSocket('udp4')

server.on('listening', () => {
  console.log("listening")
})

server.on('message', (msg, remote) => {
  const packet = dnsPacket.decode(msg)
  var response = {
    type: "response",
    id: packet.id,
    questions: [ packet.questions[0] ],
    answers: []
  }

  mdnsResolver.resolve(packet.questions[0].name, packet.questions[0].type)
  .then(addr => {
    response.answers.push({
      type: packet.questions[0].type,
      class: packet.questions[0].class,
      name: packet.questions[0].name,
      ttl: 30,
      data: addr
    })
    server.send(dnsPacket.encode(response), remote.port)
  })
  .catch (err => {
    server.send(dnsPacket.encode(response), remote.port)
  })
})
server.bind(port)

This worked well but I ran into a small problem with the mdns-resolver library which wouldn’t resolve CNAMEs, but a small pull request soon fixed that.

The next spin of the code added support to send any request not for a .local domain to an upstream DNS server to resolve which means I don’t need to add as may DNS servers to each container.

All the code is on github here.

Bonus points

This will also allow Windows machines, which don’t have native mDNS support, to do local name resolution.

IPv6 only network with IPv4 access (DNS64/NAT64)

Continuing the theme of building my own ISP I started looking at running a IPv6 only network.

As IPv4 addresses become increasingly scarce it won’t be possible to hand out publicly routeable addresses to every user. The alternatives are things like CGNAT but that has a bunch of problems.

On the other hand the default suggested IPv6 allocation per user is a /48 subnet (which is 65,5536 /64 subnets each containing 18,446,744,073,709,551,616 addresses) which should be more than enough for anybody. More and more services slowly are making them selves available via IPv6.

So rather than run a dual stack IPv4/IPv6 network with a double NAT’d (at the home router and again at the ISP’s CGNAT link to the rest of the internet ) IPv4 address, we can run a pure IPv6 ISP and offer access to IPv4 via a NAT64 gateways to allow access to those services that are still IPv4 only.

DNS64

This is the part that converts IPv4 addresses to IPv6 addresses.

The local device looking to connect makes a DNS request for the hostname of the remote device. If there is no native AAAA (IPv6 address) entry for the hostname, the DNS64 server will generate one based on converting the IPv4 address to hex and prepending a IPv6 prefix. The prefix can be anything with at least /96 (allowing enough room for all IPv4 addresses) but there is a pre-defined address range of 64:ff9b::/96.

So if looking up the remote hostname returns 192.168.1.5 then the mapped IPv6 address would be 64:ff9b::c0a8:0105 (192.168.1.5 -> c0.a8.01.05 in hex)

As of version 9.8 of bind support for DN64 is built in and configured by adding the following:

...
dns64 64:ff9b::/96 {
  clients { any; };
  mapped { !10/8; any;};
  break-dnssec yes;
};
...

Running your own means you can control who can access the server (using the clients directive and control which IP address ranges are mapped or excluded).

All this IP address mapping does break DNSSEC but since most clients rely on their recursive DNS servers to validate DNSSEC record rather than doing it directly this is less of a problem.

There are also a number of public DNS servers that support DNS64 including one run by Google.

To test you can use dig and point to the DNS64 server. e.g. to get a IPv6 mapped address for www.bbc.co.uk (212.58.233.254 & 212.58.237.254)

$ dig @2001:4860:4860::6464 -t AAAA www.bbc.co.uk

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> @2001:4860:4860::6464 -t AAAA www.bbc.co.uk
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1043
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.bbc.co.uk.			IN	AAAA

;; ANSWER SECTION:
www.bbc.co.uk.		235	IN	CNAME	www.bbc.net.uk.
www.bbc.net.uk.		285	IN	AAAA	64:ff9b::d43a:edfe
www.bbc.net.uk.		285	IN	AAAA	64:ff9b::d43a:e9fe

;; Query time: 133 msec
;; SERVER: 2001:4860:4860::6464#53(2001:4860:4860::6464)
;; WHEN: Mon Feb 03 21:40:50 GMT 2020
;; MSG SIZE  rcvd: 124

We can see that 212.58.233.254 is d43ae9fe in hex and has been added to the 64:ff9b::/96 prefix to make 64:ff9b::d43a:e9fe

NAT64

This is the part that actually does the mapping between the IPv6 address of the initiating device and the IPv4 address of the target device.

There are a few different implementations of NAT64 for Linux

I decided to give Jool a go first based on a presentation I found.

I had to build Jool from source, but this wasn’t particularly tricky and once installed I followed the Stateful NAT64 instructions. There were 2 bits missing from this that caused a few problems.

The first was because my host machine has multiple IPv4 addresses I needed to add the right address to the `pool4`. When adding the the address you also need to specify a range of ports to use and these need to be excluded from ephemeral local port range.

The ephemeral range can be between 1024 and 65535 and you can check what the current range is set to with sysctrl

$ sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768	60999

For a proper deployment this range needs reducing so that you can commit enough ports for all the NAT64 connections that will pass through the gateway. You can also add multiple IPv4 addresses.

Quick aside: The whole point of this exercise is to reduce the number of publicly routable IPv4 addresses that we need. To make this work we are always going to need some, but we will share a small number at one point at the edge of the network to be used as the NAT64 egress point, but as more and more services move over to supporting IPv6 this number will decrease.

Because I’m only playing at the moment, I’m just going to use the 61000-65535 range and leave the ephemeral ports alone. I will still be able to host 4500 connections.

To make starting it all easier I wrote a short script that

  • Loads the module
  • Enables IPv4 and IPv6 routing
  • Sets up jool with the default IPv6 prefix
  • Adds the iptables entries to intercept the packets
  • Adds the IPv4 output address to the jool config with the port range for TCP, UDP and ICMP
#!/bin/sh
modprobe jool

sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.forwarding=1

jool instance add "example" --iptables  --pool6 64:ff9b::/96

ip6tables -t mangle -A PREROUTING -j JOOL --instance "example"
iptables -t mangle -A PREROUTING -j JOOL --instance "example"

jool -i "example" pool4 add -i 192.168.1.94 61000-65535
jool -i "example" pool4 add -t 192.168.1.94 61000-65535
jool -i "example" pool4 add -u 192.168.1.94 61000-65535

Putting it together

To make it all work I need to set the DNS server handed out to my ISP customers point to the DNS64 instance and to make sure that 64:ff9b::/96 gets routed via the gateway machine.

To test I pinged www.bbc.co.uk

$ ping6 -c 2 www.bbc.co.uk
PING www.bbc.co.uk (64:ff9b::d43a:e9fe) 56 data bytes
64 bytes from 64:ff9b::d43a:e9fe: icmp_seq=1 ttl=54 time=18.1 ms
64 bytes from 64:ff9b::d43a:e9fe: icmp_seq=2 ttl=54 time=23.0 ms

--- 64:ff9b::d43a:e9fe ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 3ms
rtt min/avg/max/mdev = 18.059/20.539/23.019/2.480 ms
Network diagram
Network layout

DoH Update and DNS over TLS

I’ve been updating my DoH code again. It should now match RFC8484 and can be found on github here.

  • DNS wire format requests are now on /dns-query rather than /query
  • Change Content-Type to applicaton/dns-message
  • JSON format requests are now on /resolve
  • Made the dns-to-https-proxy only listen on IPv4 as it was always replying on IPv6

Normally ISPs have rules about running open recursive DNS servers on consumer lines, this is mainly because they can be subject to UDP source forgery and used in DDoS attacks. Because DoH is all TCP based it does not pose the same problem. So I’m going to stand up a version publicly so I can set my phone to use it for a while. I’ll be using nginx to proxy and sticking the following config bock in the http section that serves my https traffic.

location /dns-query {
  proxy_pass https://127.0.0.1:3000/dns-query;
}

location /resolve {
  proxy_pass https://127.0.0.1:3000/resolve;
}

As well as DoH I’ve been looking at setting up DoT (RFC858) for my DNS server. Since bind doesn’t have native support for TLS, this will again be using nginx as a proxy to terminate the TLS connection and then proxy to the bind instance. The following configuration should export port 853 and forward to port 53.

stream {
    upstream dns {
        zone dns 64k;
        server 127.0.0.1:53;
    }

    server {
        listen 853 ssl;
        ssl_certificate /etc/letsencrypt/live/example.com/cert.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        proxy_pass dns;
        proxy_bind 127.0.0.2;
    }
}

nginx is running on the same machine as the as bind, but runs different views for internal and external clients based on the IP address of the request came from. The internal view includes 127.0.0.1 which is why the proxy_bind directive is used to make sure the request comes from 127.0.0.2 so it looks like and external address.

DNS-over-HTTPS update

My post on DNS-over-HTTPS from last year is getting a fair bit more traffic after a few UK news paper articles (mainly crying that the new UK Government  censoring won’t work if Google roll it out in Chrome… what a shame). The followning article has a good overview [nakedsecurity].

Anyway I tweeted a link to the old post and it started a bit of a discussion and the  question about the other side of system came up. Namely how to use a DNS resolver that pushed traffic over DNS-over-HTTPS rather than provide a HTTPS endpoint that supported queries. The idea being that at the moment only Firefox & Chrome can take advantage of the secure lookups.

I did a bit of poking around and found things like stubby which DNS-over-TLS (another approach to secure DNS lookups) and also Cloudflare have cloudflared which can proxy for DNS-over-HTTPS to Cloudflare’s DNS server (it also is used to set up the VPN tunnel to Cloudflare’s Argo service, which is also worth a good look at.)

Anyway, while there are existing solutions out there I thought I’d have a really quick go at writing my own, to go with the part I’d written last year, just to see how hard it could be.

It turned out a really basic first pass could be done in about 40 lines of Javascript:

const dgram = require('dgram')
const request = require('request')
const dnsPacket = require('dns-packet')

const port = process.env["DNS_PORT"] || 53
//https://cloudflare-dns.com/dns-query
const url = process.env["DNS_URL"] 
    || "https://dns.google.com/experimental" 
const allow_selfSigned = 
    (process.env["DNS_INSECURE"] == 1) 

const server = dgram.createSocket('udp6')

server.on('listening', function(){
  console.log("listening")
})

server.on('message', function(msg, remote){
  var packet = dnsPacket.decode(msg)
  var id = packet.id
  var options = {
    url: url,
    method: 'POST',
    body: msg,
    encoding: null,
    rejectUnauthorized: allow_selfSigned ? false : true,
    headers: {
      'Accept': 'application/dns-message',
      'Content-Type': 'application/dns-message'
    }
  }

  request(options, function(err, resp, body){
    if (!err &amp;&amp; resp.statusCode == 200) {
      var respPacket = dnsPacket.decode(body)
      respPacket.id = id
      server.send(body,remote.port)
    } else {
      console.log(err)
    }
  })

})

server.bind(port)

It really could do with some caching and some more error handling and I’d like to add support for Google JSON based lookups as well as the binary DNS format, but I’m going to add it to the github project with the other half and people can help extend it if they want.

The hardest part was working out I needed the encoding: null in the request options to stop it trying to turn the binary response into a string but leaving it as a Buffer.

I’m in the process of migrating my DNS setup to a new machine, I’ll be adding a DNS-over-TLS (using stunnel) & a DNS-over-HTTPS listeners for the public facing sides.

DNS-Over-HTTPS

I saw the recent announcements from Mozilla, Cloudflare and Google about running a trial to try and make DNS name resolution more secure.

The basic problem is that most users get their DNS server set via DHCP which is controlled by who ever runs the network (at home this tends to be their ISP, but when using public wifi this could be anybody). The first approach to help with this was Google’s 8.8.8.8 public DNS service (followed by the IBM’s 9.9.9.9 and Cloudflares 1.1.1.1). This helps if people are technically literate enough know how to change their OS’s DNS settings and fix them to one of these providers. Also DNS is UDP based protocol which makes it particularly easy for a bad actor on the network to spoof responses.

The approach the 3 companies are taking is to run DNS over an existing secure protocol, in this case HTTPS. From Firefox version 60 (currently in beta) it is possible to set it up to do name host name resolution via DNS-Over-HTTPS.

There are currently 2 competing specifications for how to actually implement DNS-Over-HTTPS.

DNS Wireformat

This uses exactly the same data structure as existing DNS. Requests can be made via a HTTP GET or POST. For a POST the body is the binary request and the Content-Type is set to application/dns-udpwireformat.

For GET requests the payload is BASE64 encoded and passed as the dns query parameter.

In both cases the response is the same binary payload as would be made by a normal DNS server.

This approach is currently covered by this draft RFC

JSON

For this approach the request are made as a HTTP GET request with the hostname (or IP address) being passed as the name and the query type being passed as the type query parameters.

A response looks like this:

{
    "Status": 0,
    "RA": true,
    "RD": true,
    "TC": false,
    "AD": false,
    "CD": true,
    "Additional": [],
    "Answer": [
        {
            "TTL": 86400,
            "data": "93.184.216.34",
            "name": "example.com",
            "type": 1
        }
    ],
    "Question": [
        {
            "name": "example.com",
            "type": 1
        }
    ]
}

With a Content-Type of application/dns-json

You can find the spec for this scheme from Google here and Cloudflare here.

Both of these schemes have been implemented by both Google and Cloudflare and either can be used with Firefox 60+.

Privacy Fears

There has already been a bit of a backlash against this idea, mainly around privacy fears. The idea of Google/CloudFlare being able to collect information about all the hosts your browser resolves scared some people. Mozilla has an agreement in place with CloudFlare about data retention for the initial trial.

Given these fears I wondered if people might still want to play with DNS-Over-HTTPS but not want to share data with Google/Cloudflare. With this in mind I thought I’d try and see how easy it would be to implement a DNS-Over-HTTPS server. Also people may want to try this out on closed networks (for things like performance testing or security testing).

It turned out not to be too difficult, I started with a simple ExpressJS based HTTP server and then started to add DNS support. Initially I tried a couple of different DNS NodeJS nodes to get all the require details and in the end settled on dns-packet and actually sending my own UDP packets to the DNS server.

I’ve put my code up on github here if anybody wants a play. The README.md should include details about how to set up Firefox to use an instance.

DNSSEC and Letsencrypt

A couple of tweets from a colleague over the Christmas period along with some jobs I’d been saving up made me have another look at the DNS and HTTPS set up for a couple of sites I look after.

DNSSEC

I’ve been meaning to play with DNSSEC for a while, especially since I run my own primary DNS and set up DMKIM to verify my mail server identity (yeah, I know in this day and age of cloud running all your own services is a little quaint, but I like to understand how every thing works).

This a good introduction to DNSSEC if you’re not up to speed. TL;DR DNSSEC allows you to tell when people have been messing with your DNS entries.

To set up DNSSEC you need to create 2 sets of keys, a zone signing key and a key signing key you can create them with the following commands respectively.

$ dnssec-keygen -a NSEC3RSASHA1 -b 2048 -n ZONE hardill.me.uk
Generating key pair..................+++ .............+++
Khardill.me.uk.+007+40400
$ dnssec-keygen -f KSK -a NSEC3RSASHA1 -b 4096 -n ZONE hardill.me.uk
Generating key pair....................................................................................................................................................................................................................................................++ ................................................................................++ 
Khardill.me.uk.+007+23880

Key generation requires a lot of random numbers and these are created from the /dev/random, the values for this are generated from the system entropy so can take a long time on a machine that isn’t doing very much, to help with this I can installed the haveged daemon.

Now I have the 2 sets of keys (public and private) I need to add them to the end of my zone file with the following lines:

$INCLUDE Khardill.me.uk.+007+43892.key
$INCLUDE Khardill.me.uk.+007+23880.key

Now we can use these keys to actually sign the zone with the dnssec-signzone command, the NSEC3 setup takes a salt to help with security. The $(head -c 1000 /dev/random | sha1sum | cut -b 1-16) generates a 16 character random string to act as the salt.

$ dnssec-signzone -A -3 $(head -c 1000 /dev/random | sha1sum | cut -b 1-16) -N INCREMENT -o hardill.me.uk -t hardill.me.uk.db
Verifying the zone using the following algorithms: NSEC3RSASHA1.
Zone fully signed:
Algorithm: NSEC3RSASHA1: KSKs: 1 active, 0 stand-by, 0 revoked
                         ZSKs: 1 active, 0 stand-by, 0 revoked
hardill.me.uk.db.signed
Signatures generated:                       25
Signatures retained:                         0
Signatures dropped:                          0
Signatures successfully verified:            0
Signatures unsuccessfully verified:          0
Signing time in seconds:                 1.129
Signatures per second:                  22.143
Runtime in seconds:                      1.274

This generates 2 files, the first is hardill.me.uk.db.signed which is an updated version of the zone file with the signed hashes included for each entry. The second is dsset-hardill.me.uk. which holds the DS hashes for my 2 keys. The DS entries are hosted by the layer above my domain in the DNS hierarchy so that anybody wanting to verify the data can walk from the Signed root zone up the tree checking the level above before moving on. To get the DS entries into the zone above you normally have to go through your Domain Name Registrar who would in this case ask Nominet (as the keep of the me.uk domain) to host them for me, unfortunately my registrar (I won’t name them here) claims unable to be able pass this request on to Nominet. I need to see if I can get Nominet to do it for me, but I’m not confident so I’m currently in the market for a new registrar, any recommendations welcome.

In the mean time I decided to test the rest of it out on the private TLD I run on my lan. I can get round the need for a DS record by telling Bind to trust my key explicitly using the trusted-keys directive in named.conf. To get this far I followed this set of instructions, which are the manual steps for DNSSEC, there are also instructions to get Bind to automatically sign zones, which is especially useful if you are doing Dynamic DNS updates, this page has instructions for that which I’ll be looking at once I get things sorted to have my DS records hosted properly.

Letsencrypt

The letsencrypt project has a goal to provide free SSL certificates for everybody that are signed by a CA in the collection commonly included in modern browsers. It had been in private beta most of last year, but went into public beta at the start of December so I could sign up. Letsencrypt will generate you a certificate for any domain you can prove you own, you do this using a protocol called ACME and they have written a client to help with this. ACME works over HTTP/HTTPs by placing a hash value at a known location. This can be via an existing HTTP server (e.g. Apache) or by a one built into the client. At home I run my own private CA as it allows me to issue certificates for names on my private TLD and for my IP addresses. I also issue client certificates to authenticate users and having them all with the same CA makes things a little easier. When I get some time I will probably move my domain over to a letsencrypt certificate and only use my CA for client certs. In the mean time I needed to set up access to my Dad’s work mail server so my Brother can send/receive email from his iPhone, this needed to be secure so everything needs to be protected by a certificate. Rather than mess about getting the root CA certs for my private CA on to his phone I decided to use letsencrypt. The mail server doesn’t run a webserver so I used the one built into the client.

$ letsencrypt-auto certonly --standalone  --email admin@example.com --agree-tos -d mail.example.com

The command line arguments are as follows

  • certonly – This tells the client to download the certificate (rather than download it and install it
  • –standalone – This tells the client to use it’s built in HTTP server
  • –email admin@example.com – This tells the client who to email if there is a problem (like a cert expires without being renewed)
  • –agree-tos – This stops the client showing the TOS and prompting you to agree to them
  • -d mail.example.com – This tells the client which host name to create the certificate for, you can specify multiple instances

Certificates and Keys are stored under /etc/letsencrypt/ with the current cert under live/[host name]. I configured Postfix and Dovecot to point to these so that they just need to be restarted to pick up the new certs.

Letsencrypt hand out certificates that are only valid for 90days, this is for a couple of reasons, but mainly it means that any compromised certs only expose people for a short time and they can upgrade the supported algorithms/key strength regularly to keep ahead of new vulnerabilities. The downside to this is that you need to renew the certificate regularly. The client is actually pretty good at letting you automate things using a very similar command to the original version. I’ve set up a cron job to run on the first of every second month that renews a new cert every 60ish day and then restart Postfix and Dovecot, this gives plenty of time to fix anything should there be a problem.

25 15 1 1,3,5,7,9,11 * /home/admin/renew-cert.sh
#!/bin/sh
/home/admin/letsencrypt/letsencrypt-auto certonly --standalone --renew-by-default --email admin@example.com --agree-tos -d mail.example.com
sudo service dovecot restart
sudo service postfix restart

I had to add the following to the sudoers file to get everything to work without prompting for passwords

admin ALL= NOPASSWD: /home/admin/letsencrypt/letsencrypt-auto 
admin ALL= NOPASSWD: /usr/bin/service postfix *
admin ALL= NOPASSWD: /usr/bin/service dovecot *