Category: Projects

nginx-proxy-avahi-helper

I’ve been playing with the jwilder/nginx-proxy docker container recently. It automatically generates reverse proxy entries for other containers.

It does this by monitoring when new containers are started and then inspecting the environment variables attached to the new container. If the container has the VIRTUAL_HOST variable is uses it’s value as the virtual host for nginx to proxy access to the container.

Normally for this type of setup you would set up a wildcard DNS that points to the docker host so all DNS lookups for a machine in the root domain will return the same IP address.

If all the virtual hosts are in the example.com domain e.g. foo.example.com and bar.example.com you would setup the *.example.com DNS to point to the docker hosts IP address.

When working on my home LAN I normally use mDNS to access local machines so there is no where to set up the wildcard DNS entry. Instead I have build a container to add CNAME mDNS entries for the docker host for each of the virtual hosts.

In my case Docker is running on a machine called docker-pi.local and I’m using that as the root domain. e.g. manager.docker-pi.local or registry.docker-pi.local.

The container is using the docker-gen application that use templates to generate configuration files based on the running containers. In this case it generates a list of virtual hosts and writes them to a file.

The file is read by a small python application that connects to d-bus to talk to the avahi daemon on the docker host and configures the CNAMEs.

Both Docker and d-bus use unix domain sockets as their transport protocol so you have to mount the sockets from the host into the container.

 docker run -d -v /run/dbus/system_bus_socket:/run/dbus/system_bus_socket -v /var/run/docker.sock:/tmp/docker.sock --name avahi-cname hardillb/nginx-proxy-avahi-helper

I’ve put the code on github here and the container is on docker hub here.

Back to Building the Linear Clock

A LONG time ago I started to work out how to build a linear clock using a strip of 60 LEDs. It’s where all my playing with using Pi Zeros and Pi 4s as USB devices started from.

I’ve had a version running on the desk with jumper wires hooking everything up, but I’ve always wanted to try and do something a bit neater. So I’ve been looking at building a custom Pi pHAT to link everything together.

The design should be pretty simple

  • Break out pin 18 on the Pi to drive the ws2812b LEDs.
  • Supply 5v to the LED strip.
  • Include an I2C RTC module so the Pi will keep accurate time, especially when using a Pi Zero (not a Pi Zero W) with no network connectivity.

I know that the Pi can supply enough power from the 5v pin in the 40 pin header to drive the number of LEDs that the clock should have lit at any one time.

Also technically the data input for the ws2812b should also be 5v but I know that at least with the strip that I have it will work with the 3.3v supplied from GPIO pin 18.

I had started to work on this with Eagle back before it got taken over by Autodesk, while there is still a free version for hobbyists I thought I’d try something different this time round and found librePCB.

Designing the Board

All the PCB board software looks to work in a similar way, you start by laying out the circuit logically before working out how to lay it out physically.

The component list is:

The RTC is going to need a battery to keep it up to date if the power goes out so I’m using the Keystone 500 holder which is for a 12mm coin cell. These are a lot smaller than the more common 20mm (cr2032) version of coin cells, so should take up lot less space on the board.

I also checked that the M41T81 has a RTC Linux kernel driver and it looks to be included in the rtc-m41t80 module, so should work.

Finally I’m using the terminal block because I don’t seem to be able to find a suitable board mountable connector for the JST SM connector that comes on most of the LED strips I’ve seen. The Wikipedia entry implies that the standard is only used for wire to wire connectors.

Circuit layout

LibrePCB has a number of component libraries that include one for the Raspberry Pi 40 pin header.

Physical and logical diagram of RTC component

But I had to create a local library for some of the other parts, especially the RTC and the battery holders. I will look at how to contribute these parts to the library once I’ve got the whole thing up and running properly.

Block circuit diagram

The Pi has built in pull up resistors on the I2C bus pins so I shouldn’t need to add any.

Board layout

Now I have all the components linked up it was time to layout the actual board.

View of PCB layout

The board dimensions are 65mm x 30mm which matches the Pi Zero and with the 40 pin header across the top edge.

The arrangement of the pins on the RTC mean that no matter which way round I mount it I always end up with 2 tracks having to cross which means I have one via to the underside of the board. Everything else fits into one layer. I’ve stuck the terminal block on the left hand edge so the strip can run straight out from there and the power connector for the Pi can come in to the bottom edge.

Possible Improvements

  • An I2C identifier IC – The Raspberry Pi HAT spec includes the option to use the second I2C bus on the Pi to hold the device tree information for the device.
  • A power supply connector – Since the LED strip can draw a lot of power when all are lit, it can make sense to add either a separate power supply for just the LEDs or a bigger power supply that can power the Pi via the 5v GPIO pins.
  • A 3.3v to 5v Level shifter – Because not all the LED strips will work with 3.3v GPIO input.
  • Find a light level sensor so I can adjust the brightness of the LED strip based on the room brightness.

Next Steps

The board design files are all up on GitHub here.

I now need to send off the design to get some boards made, order the components and try and put one together.

Getting boards made is getting cheaper and cheaper, an initial test order of 5 boards to test from JLC PCB came in at £1.53 + £3.90 shipping (this looks to include a 50% first order discount). This has a 12 day shipping time, but I’m not in a rush so this just feels incredibly cheap.

Soldering the surface mount RTC is going to challenge my skills a bit but I’m up for giving it a go. I think I might need to buy some solder paste and a magnifier.

I’ll do another post when the parts all come in and I’ve put one together.

Node-RED Google Home Smart Home Action Generally Available

I started this post back in November 2017, it’s been a long slog, but we are finally here. We had a false start back in January 2019 when I got the bulk of the code all working and I thought it would take a few weeks get it certified and released and again at the beginning of this year when the Action got approved but still required you to be a member of a Google Group to be able to sign up . Unfortunately that wasn’t the case. But all that is behind us now.

A Node-RED flow with lots of Google Home Assistant nodes

You can find the full docs for how to install and configure the Action here, but the short version is:

  • Create an account here
  • Create some devices using the wizard
  • Link the NR-GAB Action to your Google Account in the Google Home app, it should have the following icon.
  • Install the node-red-contrib-googlehome node in Node-RED
  • Drag the googlehome-in node on to the canvas and start building flows that can be triggered by the Google Assistant.

As always the doc probably needs a little bit more work, so I’ll keep updating it as folks run into issues with it. If you have any questions the best place to ask them is in the #google-home-assistant channel on the Node-RED Slack team.

Next steps are I have a working version of the Google Assistant Local Control API that I will release as soon as Google open that up for general availability. This sends commands directly to Node-RED from the Smart Speaker device which reduces the latency in triggering actions.

Contributing to Accel-PPP

I mentioned in a previous post about my desktop ISP project I was using the accel-ppp access concentrator. This provides the main components for allowing my “users” to connect to my network, it handles all the authentication and setting up of PPPoE connections.

While playing with it I wanted to look at settings different DNS servers for different users. For IPv4 this was already built in, you could put entries in the LDAP that get passed on via the Radius lookup when a user connects. But with my messing about with IPv6 only networks I also wanted to set custom IPv6 DNS servers on a per user basis.

You can set a global set of DNS servers in the config file

...
[ipv6-dns]
dns=fd12:3456:789a::1
dns=fd12:3456:789a::2
...

The Radius spec has an entry for holding IPv6 DNS server values (DNS-Server-IPv6-Address from RFC6911), but accel-ppp didn’t support using it. I mentioned it on the accel-ppp forum as a feature request and the developers seemed to think it would be a good idea, so I decided to have a go at implementing it.

First up adding the required bits to the LDAP for the test user, here I have one IPv4 DNS and one IPv6 DNS.

displayName: Ben Hardill
cn: Ben
sn: Hardill
mail: isp1@hardill.me.uk
uid: isp1
radiusReplyAttribute: MS-Primary-DNS-Server := "192.168.5.1"
radiusReplyAttribute: Reply-Message := "Hello World"
radiusReplyAttribute: Delegated-IPv6-Prefix := "fd12:3456:789a:2::/64"
radiusReplyAttribute: Framed-IPv6-Prefix := "fd12:3456:789a:0:192:168:5:2/128"
radiusReplyAttribute: DNS-Server-IPv6-Address := "fd12:3456:789a:ff64::1"
radiusFramedIPAddress: 192.168.5.2
radiusFramedIPNetmask: 255.255.255.0

My first pass was a little naive in that it ended up changing the DNS servers for all subsequent users by overwriting the global settings. It also only supported providing 1 DNS server when the DHCPv6 and RA both support up to 3. So it was back to the drawing board.

First up is to generate a linked list containing the DNS server addresses from the Radius response.

...
			case Framed_IPv6_Route:
				rad_add_framed_ipv6_route(attr->val.string, rpd);
				break;
			case DNS_Server_IPv6_Address:
				a = _malloc(sizeof(*a));
				memset(a, 0, sizeof(*a));
				a->addr = attr->val.ipv6addr;
				list_add_tail(&a->entry, &rpd->ipv6_dns.addr_list);
				break;
		}
	}
...

This gets bound to the users session object so it can be retrieved later. In this case when responding to a DHCPv6 request.

...
	if (!list_empty(&ses->ipv6_dns->addr_list)) {
		list_for_each_entry(dns, &ses->ipv6_dns->addr_list, entry) {
			j++;
		}
		if (j >= 3) {
			j = 3;
		}
		opt1 = dhcpv6_option_alloc(reply, D6_OPTION_DNS_SERVERS, j * sizeof(addr));
		addr_ptr = (struct in6_addr *)opt1->hdr->data;
		list_for_each_entry(dns, &ses->ipv6_dns->addr_list, entry) {
			if (k < j) {
				memcpy(addr_ptr, &dns->addr, sizeof(addr));
				k++;
				addr_ptr++;
			} else {
				break;
			}
		}
...

Now this is not as clean as I would like since we have to walk the list to know how many DNS servers there are in order to allocate the right size chunk of memory to hold the addresses and then walk it again to copy the addresses into the response. I might go back and add the list length to the session structure and update it when adding items to the list so we can avoid this step.

Testing

While working on these changes I wanted something a bit quicker than having to deploy the changes to my physical test rig (a collection of Raspberry Pi and switched hidden under my sofa). To help with this I spun up a VM with a copy of CORE. CORE comes from the NRL and allows you to build virtual networks and run code on nodes within the network.

CORE emulating a simple network

This means I can start and stop the access coordinator really quickly and spin up as many clients as I want. I can also easily drop in L2TP clients (which use the same Radius and PPP infrastructure in accel-ppp) to check it works there as well. It also lets you attach tools like tcpdump and wireshark to capture packets at any point in the network which can be difficult with home grade switches (enterprise switches tend to have admin interfaces and the ability to nominate ports as taps that can see all traffic).

I have submitted a pull request against the project, so now fingers crossed it gets accepted.

Open Source Rewards

This is a bit of a rambling piece around some things that have been rattling round in my brain for a while. I’ve written it out mainly to just get it out of my head.

There has been a of noise around Open Source projects and cloud companies making huge profits when running these as services. To the extent of some projects even changing the license to either prevent this, to force the cloud providers to publish all the supporting code that allows the projects to be run at scale or include new features under none Open Source licenses.

Most of the cases that have been making the news have been around projects that have an organisation that supports them e.g. Elastic search that also sell support and hosted versions of the project. While I’m sympathetic to the arguments of the projects I’m not sure the license changes work, and the cloud companies do tend to commit developers to the projects (OK, usually with an aim of getting the features they want implemented, but they do fix bugs as well). Steve O’Grady from Redmonk has a good piece about this here.

I’m less interested in the big firms fighting over this sort of thing, I’m more interested in the other end of the scale, the little “guys/gals”.

There have also been cases about single developers that have built some core components that under pin huge amounts of Open Source software. This is especially clear in the NodeJS world where hundreds of tiny npm modules come together to build thousands of more complex modules that then make up every bodies applications.

While a lot of OS developers do it for the love, or to practice their skills on things they enjoy. But when a project becomes popular the expectations start to stack up. There are literally too many stories of entitled users expecting the same levels of support/service that they would get from a large enterprise when paying for a support contract.

The problem here is when the developer at the bottom of the stack gets fed up with everybody that depends on their module raising bugs and not contributing fixes or just gets bored and walks away. We end up with what happened to the event-stream module. In this case the dev got bored and handy ownership over to somebody else (the first person who asked), that somebody later injected a bunch of code that would steal cryptocurrency private keys.

So what are the options to allow a loan developer to work on their projects and get the support they deserve.

Get employed by a somebody that depends on your code

If your project is of value to a medium/large organisation it can be in their interests to bring support for that project in house. I’ve seen this happen with projects I’ve been involved with (if only on the periphery) and it can work really well.

The thing that can possibly be tricky is balancing the needs of the community that has grown up around a project and the developers new “master” who may well have their own idea’s about what direction the project should take.

I’ve also seen work projects be open sourced and their developers continuing to drive them and get paid to do so.

Set up own business around the project

This is sort of the ultimate version of the previous one.

It can be done by selling support/services around a project, but it also can make some of the problems I mentioned earlier worse as now some will expect even more now they are paying for it.

Paypal donations or Github sponsorship/Patreon/Ko-Fi

Adding a link on the projects About page to a paypal account or a Patreon/Github sponsorship page can let people show their appreciation for a developers work.

Paypal links work well for one off payments, where as the Patreon/Github/Ko-Fi sponsorship model is a little bit more of a commitment but can be a good way to cover on going costs without needing to charge for a service directly. With a little work the developer can make use of the APIs these platforms provide bespoke content/services for users who choose to donate.

I have included a Paypal link the about page of some of my projects, I set have set the default amount to £5 with the suggestion that I will probably use it to buy myself a beer from time to time.

I have also recently signed up to the Githib sponsorship project to see how it works. Github lets you set different monthly amounts between $1 and $20000, at this time I only have 1 level set to $1.

Adverts/Affiliate links in projects

If you are building a mobile app or run a website then there is always the option of including adverts in the page/app. With this approach the user of the project doesn’t have to do anything apart from put up with some hopefully relevant adverts.

There is a balance that has to be struck with this as too many adverts or for irrelevant things can annoy users. I do occasionally post Amazon affiliate links in blog posts and I keep track of how much I’ve earned on the about page.

This is not just a valid model for open source projects, many (most) mobile games have adopted this sort of model, even if it is just as a starting tire before either allowing users to pay a fee to remove the adds or to buy in game content.

Amazon Wishlists

This is a slightly different approach is to publish a link to something like an Amazon wishlist. This allows users to buy developers a gift as a token of appreciation. The list allow the developer to get things they actually want and to select a range of items at different price points.

Back when Amazon was closer to it’s roots as a online book store (and people still read books to learn new things) it was a great way to get a book about a new subject to start a new project.

Other random thoughts

For another very interesting take on some of this please watch this video from Tom Scott for the Royal Institution about Science Communicating in the world of Youtube and social media. It has a section in middle about Parasocial Relationships which is really interesting (as it the rest of the video) in this context.

Conclusion

I don’t really have one at the moment, as I said at the start this is a bit of a stream of conciousness post.

I do think that there isn’t a one size fits all model, nor are the options I’ve listed above all of them, they were just the ones that came to mind as it typed.

If I come up with anything meaningful, I’ll do a follow up post, also if somebody want to sponsor me $20,000 a month on Github to come up with something, drop me a line ;-).

Alternate PPPoE Server (Access Concentrator)

Earlier this year I had a short series of posts where I walked through building a tiny (fantasy) ISP.

I’ve been using the Roaring Penguin version of the PPPoE Server that was available by default in Raspbian as I am running all of this on a Raspberry Pi4. It worked pretty well but I had to add the traffic shaping manually, at the time this was useful as it gave me an excuse to finally get round to learning how to do some of those things.

I’ve been looking for a better accounting solution, one that I can reset the data counters on at a regular interval without forcing the connection to drop. While digging around I found an alternative PPPoE implementation called accel-ppp.

accel-ppp supports a whole host of tunnelling protocols such as pptp, l2tp, sstp and ipoe as well as pppoe. It also has a built in traffic shaper module. It builds easily enough on Rasbian so I thought I’d give it a try.

The documentation for accel-ppp isn’t great, it’s not complete and a mix of English and Russian which is not the most helpful. But it is possible to patch enough together to get things working.

[modules]
log_file
pppoe
auth_pap
radius
shaper
#net-snmp
#logwtmp
#connlimit
ipv6_nd
#ipv6_dhcp
#ipv6pool

[core]
log-error=/var/log/accel-ppp/core.log
thread-count=4

[common]
#single-session=replace
#sid-case=upper
#sid-source=seq
#max-sessions=1000
check-ip=1

[ppp]
verbose=1
min-mtu=1280
mtu=1400
mru=1400
#accomp=deny
#pcomp=deny
#ccp=0
#mppe=require
ipv4=require
ipv6=allow
ipv6-intf-id=0:0:0:1
ipv6-peer-intf-id=0:0:0:2
ipv6-accept-peer-intf-id=1
lcp-echo-interval=20
#lcp-echo-failure=3
lcp-echo-timeout=120
unit-cache=1
#unit-preallocate=1

[auth]
#any-login=0
#noauth=0

[pppoe]
verbose=1
ac-name=PPPoE7
#service-name=PPPoE7
#pado-delay=0
#pado-delay=0,100:100,200:200,-1:500
called-sid=mac
#tr101=1
#padi-limit=0
#ip-pool=pppoe
#ifname=pppoe%d
#sid-uppercase=0
#vlan-mon=eth0,10-200
#vlan-timeout=60
#vlan-name=%I.%N
interface=eth0

[dns]
#dns1=172.16.0.1
#dns2=172.16.1.1

[radius]
dictionary=/usr/local/share/accel-ppp/radius/dictionary
nas-identifier=accel-ppp
nas-ip-address=127.0.0.1
gw-ip-address=192.168.5.1
server=127.0.0.1,testing123,auth-port=1812,acct-port=1813,req-limit=50,fail-timeout=0,max-fail=10,weight=1
#dae-server=127.0.0.1:3799,testing123
verbose=1
#timeout=3
#max-try=3
#acct-timeout=120
#acct-delay-time=0
#acct-on=0
#attr-tunnel-type=My-Tunnel-Type

[log]
log-file=/var/log/accel-ppp/accel-ppp.log
log-emerg=/var/log/accel-ppp/emerg.log
log-fail-file=/var/log/accel-ppp/auth-fail.log
copy=1
#color=1
#per-user-dir=per_user
#per-session-dir=per_session
#per-session=1
level=3

[shaper]
vendor=RoaringPenguin
attr-up=RP-Upstream-Speed-Limit
attr-down=RP-Downstream-Speed-Limit
#down-burst-factor=0.1
#up-burst-factor=1.0
#latency=50
#mpu=0
#mtu=0
#r2q=10
#quantum=1500
#moderate-quantum=1
#cburst=1534
#ifb=ifb0
up-limiter=police
down-limiter=tbf
#leaf-qdisc=sfq perturb 10
#leaf-qdisc=fq_codel [limit PACKETS] [flows NUMBER] [target TIME] [interval TIME] [quantum BYTES] [[no]ecn]
#rate-multiplier=1
#fwmark=1
#rate-limit=2048/1024
verbose=1

[cli]
verbose=1
telnet=127.0.0.1:2000
tcp=127.0.0.1:2001
#password=123
#sessions-columns=ifname,username,ip,ip6,ip6-dp,type,state,uptime,uptime-raw,calling-sid,called-sid,sid,comp,rx-bytes,tx-bytes,rx-bytes-raw,tx-bytes-raw,rx-pkts,tx-pkts

[snmp]
master=0
agent-name=accel-ppp

[connlimit]
limit=10/min
burst=3
timeout=60

To use the same Radius attributes as before I had to copy the Roaring Penguin dictionary to /usr/local/share/accel-ppp/radius/ and edit it to add in BEGIN-VENDOR and END-VENDOR tags.

With this configuration is a straight drop in replacement for the Roaring Penguin version, no need to change anything in the LDAP or Radius, but it doesn’t need the `ip-up` script to setup the traffic shaping.

I’m still playing with some of the extra features, like SNMP support and the command line/telnet support for sending management commands.

Basic traffic shaping

So, I thought this would be a lot harder than it ended up being1.

Over the last few posts I’ve been walking through the parts needed to build a super simple miniature ISP and one of the last bits I think (I know I’ll have forgotten something) we need is a way to limit the bandwidth available to the end users.

Normally this is mainly done by a step in the chain we have been missing out, that being the actual DSL link between the users house and the exchange. The length of the telephone line imposes some technical restrictions as well as the encoding scheme used by the DSL modems. In the case I’ve been taking about we don’t have any of that as it’s all running directly over Gigabit Ethernet.

Limiting bandwidth is called traffic shaping. One of the reasons to apply traffic shaping is to make sure all the users get a consistent experience, e.g. to stop one user maxing out all the backhual bandwidth (streaming many 4k Netflix episodes) and preventing all the other users from being able to even just browse basic pages.

Home broadband connections tend to have an asymmetric bandwidth profile, this is because most of what home users do is dominated by information being downloaded rather than uploaded, e.g. requesting a web page consists of a request (a small upload) followed by a much larger download (the content of the page). So as a starting point I will assume the backhaul for our ISP is going to be configured in a similar way and set each user up with similar asymmetric set up of 10mb down and 5mb up.

Initially I thought it might be just a case of setting a couple of variable in the RADIUS response. While looking at the dictionary for the RADIUS client I came across the dictionary.roaringpenguin file that includes the following two attribute types

  • RP-Upstream-Speed-Limit
  • RP-Downstream-Speed-Limt

Since Roaring Penguin is the name of the package that provided the pppoe-server I wondered if this meant it had bandwidth control built in. I updated the RADIUS configuration files to include these alongside where I’d set Acct-Interim-Interval so they are sent for every user.

post-auth {

	update reply {
		Acct-Interim-Interval = 300
		RP-Upstream-Speed-Limit = 5120
		RP-Downstream-Speed-Limit = 10240
	}
        ...
}

Unfortunately this didn’t have any noticeable effect so it was time to have a bit of a wider look.

Linux has a traffic shaping tool called tc. The definitive guide is included in a document called the Linux Advanced Routing and Traffic Control HowTo and it is incredibly powerful. Luckily for me what I want is relatively trivial so there is no need to dig into all of it’s intricacies.

Traffic shaping is normally applied to outbound traffic so we will deal with that first. In this case outbound is relative to the machine running the pppoe-server so we will be setting the limits for the user’s download speed. Section 9.2.2.2 has an example we can use.

# tc qdisc add dev ppp0 root tbf rate 220kbit latency 50ms burst 1540

This limits the out going connection on device ppp0 to 220kbit. We can adjust the values for the rate to 10240kbitor 1mbitto get the right speed.

Traffic coming into the device is controlled with ingress rules and is called policing. The tc-policing man page has example for limiting incoming traffic.

 # tc qdisc add dev eth0 handle ffff: ingress
 # tc filter add dev eth0 parent ffff: u32 \
                   match u32 0 0 \
                   police rate 1mbit burst 100k

We can change the device to ppp0 and the rate to 5mbit and we have what we are looking for.

Automation

Setting this up on the command line once the connection is up and running is easy enough, but it really needs to be done automatically when ever a user connects. The pppd daemon that gets started for each connection has a script that can be used to do this. The /etc/ppp/ip-up.sh script is called and in turn this calls all the scripts in /etc/ppp/ip-up.d so we can include a script in there to do the work.

The next trick is where to find the settings. When setting up the pppoe-server we added the plugin radattr.so line to the /etc/ppp/options file, this causes all the RADIUS attributes to be written to a file when the connection is created. The file is /var/run/radattr.ppp0 (with the prefix changing for each connection).

Framed-Protocol PPP
Framed-Compression Van-Jacobson-TCP-IP
Reply-Message Hello World
Framed-IP-Address 192.168.5.2
Framed-IP-Netmask 255.255.255.0
Acct-Interim-Interval 300
RP-Upstream-Speed-Limit 5120
RP-Downstream-Speed-Limit 10240

With a little bit of sed and awk magic we can tidy (environments can’t contain - & we need to wrap the string value in ") that up and turn it into environment variables and a script to set the traffic shaping.

#!/bin/sh

eval "$(sed 's/-/_/g; s/ /=/' /var/run/radattr.$PPP_IFACE | awk -F = '{if ($0  ~ /(.*)=(.* .*)/) {print $1 "=\"" $2  "\""} else {print $0}}')"

if [ -n "$RP_Upstream_Speed_Limit" ];
then

#down
tc qdisc add dev $PPP_IFACE root tbf rate ${RP_Upstream_Speed_Limit}kbit latency 50ms burst 1540

#up
tc qdisc add dev $PPP_IFACE handle ffff: ingress
tc filter add dev $PPP_IFACE parent ffff: u32 \
          match u32 0 0 \
          police rate ${RP_Downstream_Speed_Limit}kbit burst 100k

else
	echo "no rate info"
fi

Now when we test the bandwidth with iperf we see the the speeds limited to what we are looking for.

Advanced

1 This is a super simple version that probably has lots of problems I’ve not yet discovered and it would be good to try and set up something that would allow a single user to get bursts of speed above a simple total/number of users share of the bandwidth if nobody else is wanting to use it. So it’s back to reading the LARTC guide to dig out some of the more advanced options.

Depolying a TTN LoRa Gateway

I’ve been meaning to get round to this ever since the Pi Supply Kickstarter delivered my LoRa Gateway HAT and the LoRa Node pHAT.

They have been sat in their boxes waiting until I had some spare time (and I’d finally finished moving a few things around to free up a spare Pi).

LoRa Gateway on a Pi 3

LoRa is a long range, low bandwidth radio system that uses the unlicensed spectrum. When combined with the higher level LoRaWAN protocol it makes great IoT platform for low power devices that want to send low volumes of data in places where there is no WiFi coverage and can’t justify the cost of a cellular connection.

LoRaWAN allows you to deploy a collection of Gateway devices that can act as receivers for a large number of deployed devices. These gateways then forward on messages to central point for processing.

The Things Network

A group called The Things Network run a LoRaWAN deployment. They are aiming for as large a coverage area as possible. To do this they allow users to deploy their own gateways and join these to the network. By joining the network you get to use everybody elses gateways in exchange for letting other people use yours.

Setting up the Gateway

This was particularly easy. I just had to download an image and flash it to a SD card. Stick that into the pi along with an ethernet cable and some power.

After the pi boots up you point your browser at http://iotloragateway.local and fill in a couple of values generated when I registered the gateway on the TTN site and that was it. The gateway is now up and running and ready to send/receive packets from any devices in range.

Testing

In order to test the gateway I need to set up a Pi Zero with the LoRa Node pHAT. This was a little trickier, but not much.

Fist I had to disable the Linux serial console, this can be done using the raspi-config command. I also had to add dtoverlay=pi3-miniuart-bt /boot/config.txt.

That was all that was needed to get the hardware configured, as for the software there is a rak811 python package that supplies the api and utilities to work with pHAT.

I now needed to declare an application on The Things Network site, this is how messages get routed to be processes. Taking the values for this application I could now write the following helloWorld.py

#!/usr/bin/env python3
from rak811 import Mode, Rak811

lora = Rak811()
lora.hard_reset()
lora.mode = Mode.LoRaWan
lora.band = 'EU868'
lora.set_config(app_eui='xxxxxxxxx',
                app_key='xxxxxxxxxxxx')
lora.join_otaa()
lora.dr = 5
lora.send('Hello world')
lora.close()

Which can then be seen arriving in The Things Network console.

Data arriving and being displayed in The Things Network console.

And I can subscribe directly to that data feed via MQTT:

$ mosquitto_sub -h eu.thethings.network -u 'lora-app1-hardill-me-uk' -P 'xxxxxxxxx' -v -t '+/devices/+/up'
{
  "app_id": "lora-app1-hardill-me-uk",
  "dev_id": "test-lora-pi-zero",
  "hardware_serial": "323833356E387901",
  "port": 1,
  "counter": 0,
  "is_retry": true,
  "payload_raw": "SGVsbG8gd29ybGQ=",
  "metadata": {
    "time": "2019-08-10T15:45:07.568449769Z",
    "frequency": 867.5,
    "modulation": "LORA",
    "data_rate": "SF7BW125",
    "airtime": 61696000,
    "coding_rate": "4/5",
    "gateways": [
      {
        "gtw_id": "lora-gw1-hardill-me-uk",
        "gtw_trusted": true,
        "timestamp": 910757708,
        "time": "2019-08-10T15:45:07Z",
        "channel": 5,
        "rssi": -91,
        "snr": 7.75,
        "rf_chain": 0,
        "latitude": 51.678905,
        "longitude": -2.3549008,
        "location_source": "registry"
      }
    ]
  }
}

Next Steps

First up will be to get a better antenna for the gateway and to move the whole things up in the attic, from there it should get a good view north out towards the River Severn. After that I want to get a small battery powered LoRa/GPS board, like a TTGO T-Beam and ride round on my bike to get a feel for what the range/coverage actually is.

I’ll also be keeping an eye on the stats from the gateway to see if anybody else near by is deploying TTN LoRaWAN devices.

Tracks2Miles and Tracks2TitanXT removed from Play Store

I have removed both of the apps in the title from the play store.

This is for a number reasons:

  1. MyTracks removed the API required to get at the recorded tracks meta data which vastly reduced the capability of the app.
  2. DailyMile shut down, which rendered Tracks2Miles useless.
  3. Google flagged both apps as potentially having SQL injection attacks (while theoretically possible I couldn’t find a directly exploitable use case).
  4. Even before the shutdown I had completely moved all my activity tracking to dedicated Garmin devices and Strava.