TV Scrobbling

Having seen Dale Lane’s work with VDR to build something similar to Last.FM for the TV he’s been watching, I’ve been looking to see if I could do the same with MythTV.

MythTV 0.23 has shipped recently and with it came a new event system. Having had a quick look at the feature for this it looked like it would be a good starting point for this. The following events looked like they may be able to provide what I needed.

  • Playback started
  • Playback stopped
  • Playback changed

To use events you specify a command to run when each fires. You can set these up with the mythtvsetup command. As well as specifying a command you can also pass arguments, these arguments are passed using the same tokens used for the MythTV job system. The following is not the full list, but should be enough to do what I’m looking for.

  • %TITLE% – Title
  • %SUBTITLE% – Episode subtitle
  • %CHANID% – MythTV channel id
  • %DESCRIPTION% – The blurb from EPG
  • %PROGSTART% – The listed started time for the programme
  • %PROGEND% – The listed end time for the programme

There is also an option to add a command that fires on every event which you can also pass a %EVENTNAME% argument, this is very useful for debugging event ordering. Not all of the events support all of the arguments as the data is not relevant.

After a bit of playing I managed to get most of what I needed, but there where a few problems

  1. No way to tell the difference between watching recorded and live TV
  2. No way to tell when you change channel when watching live TV
  3. No way to tell when one live programme ends and a new one starts with live TV

I raised a bug in the MythTV tracker (#8388) to cover the first 2 (since the 3rd one didn’t occur to me until I got a bit further). Which the guys very quickly added. So there is now a LiveTV Started event fired when a frontend starts watching live tv followed by a Playback Started event with the details of the programme. I’ll raise a ticket for the last one and assuming I can get sign off from the boss I’ll see about submitting a patch to add it in. In the mean time with a little extra work I can infer the programme change from the Record Started and Record Finished events.

Acting on the events

So with all these events which may be reported on either the backend or frontend it’s looking good for another messaging solution. Luckily I already broker up and running for my home power monitoring and security setup. So now I just needed to run up a script or two to publish the events and a consumer with a state machine to act on them.


#!/bin/sh
#
# playStarted.sh %TITLE% %SUBTITLE% %DESCRIPTION% %CHANID%

/home/mythtv/bin/sendMessage TV/watching/start “$1/$2 ($3) on $4″

mythtvsetup events screen

Where sendMessage is a small script that publishes the second argument to the topic in the first. The other end is Java JMS application that keeps the state machine up to date and update the database when a show ends. Dale was kind enough to send me his database schema, so the data should look the same.

What next

Now I’ve got my data logging to a local database I need to come up with a front end to present it all. Next time I get 5mins to chat to Dale when we are both in the office I will see if I can borrow some of his code and if he wants to look at if we can build a site where we can all share what we’ve been watching?

Along with the events for the internal workings of the front and backends there are 10 user configurable events that can be bound to key presses, assuming I can find a spare button on the remote it should be possible to bind one of these to something like Favourite.


Resources

IPv6

It’s been coming for a while, but we really are getting close to running out of IPv4 addresses. The rate of growth of the internet continues to accelerate with not just more and more people getting online but items such as appliances and sensors. This internet of things has been talked about before (small corporate plug) here, all these things need an IP address in order to interact with the rest of the world.

There are a number of technologies that have been deployed try and help eek out the finite pool of IPv4 addresses such as CIDR and NAT. NAT works well when connections are initiated from behind the NAT gateway, but don’t work when the remote end needs to open the connection, e.g. FTP data connections.

The long term solution is to move to IPv6, this new iteration of the protocol has a much larger pool of addresses (capable of supplying 6.67 * 10^27 addresses per square meter of the planet) which should last a while longer.

Playing

Since we are all going to have to move to IPv6 at some point I thought I’d have a play. My ISP at home does not offer IPv6 support yet but there are companies that offer IPv6 over IPv4 tunnels. Wikipedia has a list here, I picked Hurricane Electric who offer free 6in4 tunnels and have multiple end points in Asia, Europe and the US.

Once you have signed up there is a “Create Regular Tunnel” link in the left hand side bar. To use a tunnel from Hurricane Electric you need a static IPv4 address that can be pinged from the internet. When you have entered your IPv4, the site will suggest the closest end point.

By default Hurricane hand out a IPv6 subnet with a /64 prefix, this means that the top 64bits of the address are considered the network mask and the rest of the address can be used for up to 18,446,744,073709,551,616 hosts. With that many addresses to go at I don’t think I’m likely to run out any time soon. It is possible to get /48 subnets assigned as well if for any reason you think that a /64 will not be enough (actually there are good reasons why you might want this which I’ll mention later).

Once you have completed the tunnel request form you should end up with a page which has information similar to this.

Account: hardillb
	Global Tunnel ID: 53560 	Local Tunnel ID: 1701
	Description:
	Registration Date: 	Tue, Apr 13, 2010
Tunnel Endpoints
	Server IPv4 address: 	216.66.80.26
	Server IPv6 address: 	2001:470:xxxx:xxx::1/64
	Client IPv4 address: 	93.97.xxx.xxx
	Client IPv6 address: 	2001:470:xxxx:xxx::2/64
Available DNS Resolvers
	Anycasted IPv6 Caching Nameserver:	2001:470:20::2
	Anycasted IPv4 Caching Nameserver:	74.82.42.42
Routed IPv6 Prefixes and rDNS Delegations
	Routed /48: 	Allocate /48
	Routed /64: 	2001:470:xxxx:xxx::/64
	RDNS Delegation NS1: 	none
	RDNS Delegation NS2: 	none
	RDNS Delegation NS3: 	none

Hurricane also provide helpful little feature at the bottom of the page that details the configuration details for a bunch of different operating systems. There are 2 different sets for Linux depending which tool chain you are using.


modprobe ipv6
ifconfig sit0 up
ifconfig sit0 inet6 tunnel ::216.66.80.26 # Server IPv4 address from above
ifconfig sit1 up
ifconfig sit1 inet6 add 2001:470:xxxx:xxx::2/64  # Client IPv6 address from above
route -A inet6 add ::/0 dev sit1

Configuration

The configuration hints on the Hurricane page are useful for testing but don’t match up with the various distros methods for establishing the tunnel at startup. The following instructions apply to Fedora 12

The first step is to enable IPv6, this is easily done by adding the last line to /etc/sysconfig/network file:


NETWORKING=yes
HOSTNAME=tiefighter
NETWORKING_IPV6=yes

Next the tunnel interface needs setting up. To do this create the following file as /etc/sysconfig/network-scripts/ifcfg-sit1.


DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6TUNNELIPV4=216.66.80.26       # the IPv4 addres of your ISP's tunnel server
IPV6TUNNELIPV4LOCAL=192.168.1.5     # your host's local IPv4 address
IPV6ADDR=2001:470:xxxx:xxx::2/64         # your host's public IPv6 address

Once these where set restarting the networking component brought up the tunnel. This now means that this machine can send and receive traffic via IPv6, but that doesn’t get me any further than I had with the static IPv4 address I already had. The next step is to enable the other machines on my network so they can route via IPv6 as well. IPv6 has support for automatic address configuration built in called Stateless address autoconfiguration, the specification allows hosts to generate their own addresses based on the MAC address of the network card that it will use to send the packets over. This generates a 64bit number which acts as the host part of the address, the network part is supplied by the local router using the router announce protocol, as long as network part of the address is larger than /64 then it all works fine. . This all works because of IPv6’s hierarchical routing means that all packets with my prefix will be will be directed to tunnel and from then on it becomes my networks job to route them to the end hosts.

Back to the comment earlier about a /64 network not being enough for some people, if you have multiple network segments behind your tunnel then having a /48 network then you can assign different /64 networks to each segment to allow you to use Stateless address autoconfiguration on each.

To make the tunnel machine act as a router for all the other machines on the network it needs to be configured to forward packets and to make router announcements so the other machine can form correct addresses. Setting up the packet forwarding is easy enough, it’s just a case of adding another line to /etc/sysconfig/network file.


NETWORKING=yes
HOSTNAME=tiefighter
NETWORKING_IPV6=yes
IPV6_ROUTER=yes

To enable Router Announce we need the radvd app, once installed edit the /etc/radvd.conf file


interface eth0 {
	AdvSendAdvert on;
	MinRtrAdvInterval 30;
	MaxRtrAdvInterval 100;
	prefix 2001:470:xxxx:xxx::/64
	{
		AdvOnLink on;
		AdvAutonomous on;
	};
};

The last thing that needs doing is assigning a IPv6 address to the eth0 interface by adding it to /etc/sysconfig/network-scripts/ifcfg-eth0.


DEVICE=eth0
TYPE=Ethernet
BOOTPROTO=none
HWADDR=00:1B:FC:10:0E:E5
ONBOOT=yes
DHCP_HOSTNAME=tiefighter
USERCTL=no
IPV6INIT=yes
PEERDNS=no
IPADDR=192.168.1.5
NETMASK=255.255.255.0
GATEWAY=192.168.1.254
DNS2=213.162.97.66
SEARCH=loc
DNS1=127.0.0.1
IPV6ADDR=2001:470:xxxx:xxx::3
NM_CONTROLLED=no

That should be it, I now have a fully functional IPv6 subnet at home. For Linux machines running NetworkManager it should just be a case of enabling IPv6 for the connection.

The only bit that is missing is DNS because remebering IPv6 addresses is even harder than IPv4 addresses, I’ll save that for the next post.


Resources

Off site backups

This long weekends task has been to set up some form of off site backup for my dads business. He has a relatively small set of data (total less than 2gb) that is not going to change a huge amount but it is going to continue to grow over time. Previously dad had been cutting backups to DVDs every night, but as well as being wasteful by generating large numbers of half full DVDs it was slow and required somebody to remember to put a fresh disk in the drive at the end of every day.

Dad has a couple of different machines that need backing up,

  • A ancient Novell 4.x machine running some legacy software
  • A new Windows 2008 Server running a database
  • A small Ubuntu Linux box acting as NAT gateway and mail server

So the plan was to come up with something that would meet the following criteria

  • Automatic
  • Simple
  • Support very varied sources

I looked at a couple of the companies offering hosted backup online but decided to roll my own due to not wanting to have to worry about a monthly bill and knowing that if something went wrong and the backup was actually needed then it would be quicker to be able to carry whatever held the back up to the office and restore from that rather than to wait for it all to download again.

Since there was such a big difference in the type of machine that needed backing up and that the Novell stuff is so out of date that nobody was likely to support it any more I decided the best bet was to mount the bits that needed backing up from the Windows and Novell machines on the Linux box and then back it up from there. Accessing the Windows share from was easy enough using samba and the CIFS filesystem mount options. Getting at the Novell mount was a little harder.

The old NetWare Novell server doesn’t talk IP like everything else on the network, it uses IPX instead. So the first task was to configure the Linux machine to join the IPX network. Detailed instructions for setting up IPX can be found here, but the basics are assuming your Linux distribution has IPX compiled in you should be able to just run the following command as root.

ipx_configure --auto_interface=on --auto_primary=on

Once the network is up and running the next job was to mount the share, for this I needed the NCPFS package. This package is very similar to Samba and allows you to search for and mount Novell NetWare shares. There are 2 useful commands, the first is slist which will list all the Novell servers that can be seen on the network. The second is ncpmount which will mount the share.

ncpmount -S <server> <mount point> -U <Novell user>

Now I had access to all the data I needed somewhere to send it. While dad has a desktop and a laptop at home neither is on all the time or have large amounts of spare disk space, so I needed to find something to store the backups on. I was looking for a low power NAS with rsync support. I had a hunt round and found several, but most of them where big multi disk systems intended for mid sized office use and cost several hundred pounds. A bit more searching turned up the Buffalo LinkStation Live 500GB, which is basically a 500gb disk with a mini ARM based Linux machine strapped to the side. There is a reasonable sized community online hacking these little boxes to do all sorts and there are detailed instructions on how to enable telnet and SSH access which in turn can allow rsync to be used. Peek power draw is about 20W and significantly less when idle so it’s not going to cost much to run.

So I had send dad off to order one of these LinikStations and it was waiting for me when I got home. I initially started off following this set of instructions but had no joy. It turns out with the latest version of the firmware (1.24 at time of writing) the APC commander application no longer works. I started to search round some of the forums and came up with this new set of instructions on how to patch the firmware to add SSH and allow console access. The instructions are good, but it still took 2 attempts to get the firmware to upload, but once it did it all worked like a dream.

At this point I had access to all the data that needed backing up on the Linux box at the office and the LinkStation was ready to go at home, it was time to set up a way to get the data from one to the other. Rsync is perfect for this as it allows you to only copy the differences between 2 directories, these directories don’t even have to be on the same machine and all the communication can be tunnelled over SSH. The basic starting point for rsync over SSH is:

rsync -az -e ssh /path/to/src user@machine:/path/to/destination

Lets just step through the options being used here.

  • -a – This tells rsync to “archive” the src directory, this basically means conserve all the file access permissions, timestamps, symlinks and recurse into directories
  • -z – This tells rsync to compress all the data passed back and forth to save bandwidth
  • -e ssh – This tells rysnc to use ssh to connect to the target machine

That’s ok to start with as it will copy anything new in the source directory to the destination directory but it will not delete things from the destination if they disappear from the source directory, this is fine for this situation since dad is not likely to be deleting anything on the src side. So I wrapped it up in a script to run as a cron job after everybody should have gone home for the night

#!/bin/sh
USER=foo
SERVER=dads-broadband.isp.net
SRC=/mnt/backup
DST=/mnt/disk1/share/backup

rsync -az -e ssh $SRC $USER@$SERVER:$DST

The following line in the crontab runs the backup at 5 past midnight on Tuesday through Saturday so to pick up the changes for the working days.

5 0 * * 2-6 /usr/local/backup


At the moment every time the script runs it will ask for the password for the SSH connection to the Linkstation at home. To get round this I set up an SSH key. There are lots of examples for creating SSH keys and using online so I’ll skip past that now.

So that is nearly that, except having set this all up I started to think about how to recover should a problem happen. What happens if a important file gets trashed at some point and nobody notices for a day or two. If the current script is running every day at midnight then the safe copy of the trashed file will get overwritten as soon as the script runs on the day it was damaged. To get round this I needed some sort of rolling incremental backup. Luckily rsync has this support as well and there is an example on the rsync page of how to set it up. I’ve modified it a little bit to give me this:

#!/bin/sh
USER=hardillb
SERVER=dads-broadband.isp.net
SRCDIR=/mnt/backup
DSTDIR=/mnt/disk1/share/backup
INCDIR=`date --date=yesterday +%A`
OPTS="--backup --backup-dir=$DSTDIR/$INCDIR -az -e ssh"

# the following line clears the last weeks incremental directory
[ -d $HOME/emptydir ] || mkdir $HOME/emptydir
rsync --delete -a $HOME/emptydir/ $USER@$SERVER:/$DSTDIR/$INCDIR/
rmdir $HOME/emptydir

rsync $OPTS $SRCDIR $USER@$SERVER:$DSTDIR/current


This will create a directory called “current” which will hold the most up to date version backed up, it will also create 7 other directories named after the day of the week holding the previous version of any files that changed that day. This should allow any changes in the last week to be reverted.

In order to prevent having to copy all the initial 2gb over the dads broadband I took the LinkStation to the office and set it up on the local network then ran the backup script just with the local IP address rather than the SERVER entry.

For the normal operations I also had to set up dads router at home to assign the LinkStation a static IP address and forward port 22 to it so it was visible to the outside world.

Using Emprex 3009URF III Vista MCE Remote with MythTV

I recently got my hands on a new Acer Aspire Revo R3610, the plan is to use this as a second MythTV frontend in the bedrooom (I’ve still got to find a suitable TV to hang it on the back of). To go with this I needed a new remote control so I don’t need to have a keyboard and mouse plugged in, I had a bit of a search round online and found the Emprex 3009URF on Amazon, it looked like it would be perfect for what I needed. As well as having all the buttons I wanted it also uses RF rather than IR so no need to have the USB dongle in line of sight.

When it arrived I plugged it in and it showed up as 2 separate keyboards, each with about half of the buttons. This wasn’t a major problem as both where seamlessly merged by the Linux HAL keyboard layer and I was planning on wrapping the input with LIRC anyway. What was a problem was that for a few of the buttons I was not getting any key events. The missing buttons where:

  • The Big Green Start Button
  • Most of the menu buttons (TV, DVD, Music, Photos …)
  • Teletex
  • Red, Green, Yellow & Blue

Kernel work

I could have lived with out most of these apart from the Teletext and the colour buttons as these are use by the DVB-T service here in the UK to access the interactive content.I started to have a bit of hunt round online and found that there was already code in the Linux Kernel to support similar remotes. I was just about to start writing my own extention to support this one when I came across a patch that had been submitted about 2 week earlier. Details of the patch can be found here. There is some discusion between the submitter (Wayne Thomas) and the maintainer of the HID code (Dmitry Torokhov) saying that Dmitry would like to remove all the code that provides support for these types of remote now it was possible to achive the same thing with configuration files and udev. I followed the link about how to set up udev, but was unable to get even the “unkown” responses when running the monitoring program.

Given that I could not get the alternative to work it looked like the quickest way to get full support for the remote was going to mean applying the patch. Since the HID code is built into the Kernel I couldn’t just build it as a module to load into an existing Fedora Kernel, I was going to have to rebuild the whole thing. Rather than mess about pulling a raw source tree I decided to let rpmbuild do the heavy lifting since this would mean I ended up with a kernel that was a close as possible to what I already had. I downloaded the src rpm with yumdownloader. Yumdownloader doesn’t actually install the src rpm package so you need to then actually install the package


# yumdownloader kernel
# rpm -ivh kernel-2.6.31.12-174.2.22.fc12.src.rpm

The install will create a rpmbuild directory tree. This is used to do the compiling and packaging when the rpm is built using the rpmbuild command. The tree looks like this:


	rpmbuild
	   |-BUILD
	   |-BUILDROOT
	   |-RPMS
	   |-SOURCES
	   |-SPECS
	   |-SRPMS

The source tar file and all the patch files end up in the SOURCE dir and the rpm spec file goes in the SPECS dir. Applying the patch is a 2 step process. Firstly we need to down load the patch file from patchwork site listed above,I saved that in to the SOURCE dir in a file called btc-remote.patch. Secondly we need to update the spec file to ensure
that the patch is applied. This requires adding 2 lines, one to identify the file with the patch and one to actually apply it.


...
Patch16570: futex-handle-user-space-corruption-gracefully.patch

Patch16580: partitions-use-sector-size-for-efi-gpt.patch

<span style="color: blue">Patch16590: btc-remote.patch</span>

%endif

BuildRoot: %{_tmppath}/kernel-%{KVERREL}-root
...
# enable IR receiver on Hauppauge HD PVR (v4l-dvb merge pending)
ApplyPatch hdpvr-ir-enable.patch
# tell usbhid to ignore all imon devices (sent upstream 2009.07.31)
ApplyPatch hid-ignore-all-recent-imon-devices.patch
<span style="color: blue">ApplyPatch btc-remote.patch</span>

# Add kernel KSM support
ApplyPatch linux-2.6-ksm.patch
ApplyPatch linux-2.6-ksm-updates.patch
...

I have created another patch to add these lines as I expect to have to apply this next time Fedora ships a new Kernel. I have submitted a Bugzilla to ask for this patch to be included in future releases. So with patch in the right place and the right bits in the spec file the rpms are built by executing the follow command from the rpmbuild dir


# rpmbuild -bb SPECS/kernel.spec

And remove the Fedora build package then install the new one with:


# yum erase kernel-2.6.31.12-174.2.22.fc12
# yum --nogpgcheck localinstall RPMS/kernel-2.6.31.12-174.2.22.fc12.i686.rpm

LIRC

After rebooting the system to the new Kernel all the buttons now work fine. Now rather than go throw all the applications mapping the keys to the required functions I chose to use LIRC to map the key presses to the functions. LIRC has a dev/input module that will take standard /dev/input/event devices as input. First we need to
stop the HAL layer from grabbing the devices and adding them to the normal keyboard inputs. Put the following in a file called 10-ignore-emprex in /etc/hal/fdi/preprobe


<?xml version="1.0" encoding="UTF-8"?>
<deviceinfo version="0.2">
<device>
 <match key="info.product" contains_ncase="'BTC USB MCE Cordless Remote">
    <merge key="info.ignore" type="bool">true</merge>
 </match>
</device>
</deviceinfo>

Now we need to make sure we can indentify the /dev/input/event entries that map to the remote even if the order changes on boot. To do this we can use the udev rules. Looking in /proc/bus/input/device for the “BTC USB MCE Cordless Remote”, there will 2 entries looking a bit like this:


I: Bus=0003 Vendor=046e Product=5578 Version=0111
N: Name="BTC USB MCE Cordless Remote Control."
P: Phys=usb-0000:00:04.0-6/input1
S: Sysfs=/devices/pci0000:00/0000:00:04.0/usb2/2-6/2-6:1.1/input/input7
U: Uniq=
H: Handlers=kbd event7
B: EV=1f
B: KEY=837fff 2ff2b7 bf004444 0 0 1 f84 8a37c000 667bfa d941dfed 9e0000 0 0 0
B: REL=40
B: ABS=1 0
B: MSC=10

From this we can see that currently one of the 2 event sources is bound to event7, using this information udevadm tool a signature for the device that can be used to map the event devices to unique names.


# udevadm info -a -p $(udevadm info -q path -n /dev/input/event7)
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.

  looking at device '/devices/pci0000:00/0000:00:04.0/usb2/2-6/2-6:1.1/input/input7/event7':
    KERNEL=="event7"
    SUBSYSTEM=="input"
    DRIVER==""

  looking at parent device '/devices/pci0000:00/0000:00:04.0/usb2/2-6/2-6:1.1/input/input7':
    KERNELS=="input7"
    SUBSYSTEMS=="input"
    DRIVERS==""
    ATTRS{name}=="BTC USB MCE Cordless Remote Control."
    ATTRS{phys}=="usb-0000:00:04.0-6/input1"
    ATTRS{uniq}==""
    ATTRS{modalias}=="input:b0003v046Ep5578e0111-e0,1,2,3,4,k71,72,73,74,77,80,82,83,85,86,87,88,89,
8A,8B,8C,8E,8F,90,96,98,9B,9C,9E,9F,A1,A3,A4,A5,A6,A7,A8,A9,AB,AC,AD,AE,B1,B2,B5,B6,CE,CF,D0,D1,D2,D5,
D9,DB,DF,E2,E7,E8,E9,EA,EB,100,162,166,16A,16E,178,179,17A,17B,17C,17D,17F,180,181,182,185,18C,18D,192,
193,195,1A0,1A1,1A2,1A3,1A4,1A5,1A6,1A7,1A8,1A9,1AA,1AB,1AC,1AD,1AE,1B0,1B1,1B7,r6,a20,m4,lsfw"
...

Using this and the instructions from the How to write udev rules I built the following file /etc/udev/rules.d/99-emprex.rules


KERNEL=="event[0-9]*", ATTRS{name}=="BTC USB MCE C*", ATTRS{phys}=="usb-0000:00:04.0-6/input0", SYMLINK+="input/rfremote0"
KERNEL=="event[0-9]*", ATTRS{name}=="BTC USB MCE C*", ATTRS{phys}=="usb-0000:00:04.0-6/input1", SYMLINK+="input/rfremote1"

This adds 2 symlinks to the /dev/input directory called rfremote0 and rfremote1. Using these 2 instances of lircd can be started to read both sets of inputs and pass them to the applications. To bind the lircd instance you need to start one with –listen and one with –connect as follows:


# lircd --driver=dev/input --device=/dev/input/rfremote0  --pidfile=/var/run/lirc/lircd0.pid --listen
# lircd --driver=dev/input --device=/dev/input/rfremote1  --pidfile=/var/run/lirc/lircd1.pid  --connect=localhost:8765

Once these 2 instances are up and running irw can be used to test the input and also to build the .lircrc file


# Play
begin
prog = mythtv
button = KEY_PLAY
config = Return
end

# Stop
begin
prog = mythtv
button = KEY_STOPCD
config = I
end

# Escape/Exit/Back
begin
prog = mythtv
button = KEY_BACK
config = Esc
end
...
begin
prog = xine
button = KEY_PLAY
repeat = 3
config = Play
end

begin
prog = xine
button = KEY_STOPCD
repeat = 3
config = Stop
end

begin
prog = xine
button = KEY_BACK
repeat = 3
config = Quit
end

Now we are all good to go.

Resources

A different kind of TV remote control

This was originally posted to the eightbar blog back in June 2009.

I got a new TV around Christmas last year and while unpacking it I noticed along with the HDMI, SCART and other sockets on the back it had a 9-pin socket labelled “RS232C IN CONTROL&SERVICE”. I didn’t think that much of it at the time, but I remembered it last week while thinking about a couple of problems that had come up.

Tidy TV setup The first of these was that I had got home twice recently to find I’d left the TV turned on while I was at work, this was mainly because I use MythTV and I’d left it at the menu screen rather than turning the screen off as well. This had left shadow on the menu on the screen for a day or so afterwards (luckily no permanent damage as would have happened with a plasma or CRT TV).

The other point was from when we all first got hold of our Current Cost meters, there had been a lot of thought about how to work out exactly what appliances were on at any given time. While spotting when things like an electric water heater turned on was relatively easy, it was proving difficult to spot some of the lower power devices.

A plan started to form and I ordered a null modem cable from Amazon (£2.18 with free shipping) and went looking for some documentation on the protocol. The manual that came with the TV while being nearly an inch thick just covers the basics of how to plug it in and turn it on, but there was a CD-ROM with a much more detailed PDF document. The version for my TV is here. While searching round I found manuals for several other LG LCD/plasma TVs and they all seem to use the same basic protocol.

The protocol is relatively simple

[cmd1][cmd2] [setid] [data]

Where the cmd1 & cmd2 are 1 letter code, setid is for if you have multiple TVs connected to the same cable, the default id is 01 but you can change if needed, using 00 will work for all connected TVs. And data is a hex value of the option to pass the command.

The response from the TV looks like this for a success

[cmd2] [setid] OK[data]x

and like this for a failure

[cmd2] [setid] NG[data]x

The command to turn the TV on and off is “ka” so sending

ka 00 1

turns the TV on and sending

ka 00 0

turns it off. Most of the commands will reply with the current status if they are passed ff as the data. So sending

ka 00 ff

gets the following when the TV is off

a 00 OK0x

So now I had a way to turn the TV on and off along with checking its current status. The next step was to surface this some way and given the fascination we all seem to have with messaging, MQTT seemed like a good idea. A little bit of Java and the Java COMM API later and I had 2 topics TV/Commands & TV/Status.

I already have a topic that publishes if my mobile phone is in the flat by pinging it with Bluetooth. Combining this with the two new topics I can ensure that the TV is turned off when I leave. I’m also wondering if I should start to log the amount of time the TV is on, but I think the results may scare me a little…

How to get personal CCTV MMS’d to your phone

I left the setup in the last post with a system that would email photos of the burgular off site and then an SMS message. This is a prety good solution but since nearly everybody has a phone capable of receiving picture messages it seamed like the next step is to not just email the photos off site, but to also send them as a picture message so they can be checked for false alarms even when I’m not at my computer.

I went back to searching the net for a package that would supply MMS capability using a cell phone attached to a computer, not a direct connection to a bulk messaging provider. There is project called Mbuni that is a fully functional MMS gateway and relaying service as run by the cell phone providers. Normally this would run on the providers network and or at a company providing paid for content via MMS. Hidden away in the CVS for the latest version there is a add on to one of the components which will allow the sending of MMS messages via a phone.

Kannel

In the last post I had discounted Kannel for sending SMS messages because of the complexity. But Mbuni prereqs it so it was time to have another look at the setup. Mbuni also request a specific level of Kannel (CVS 2008-07-28 download) so because I was planning on using some very new function in Mbuni I decided to build this Kannel version from source to make sure it all matched up.

tar -zxf kannel-snapshot.tar.gz
cd kannel-snapshot
./configure
make
su -c "make install"

Kannel is made up of a number of separate programs that provide different bits of functionality

  • bearerbox
  • smsbox
  • wapbox

In order to be able to send and receive SMS messages we are going to need the frist two on the list. Wapbox is only used if you want to provide a dial up WAPgateway.

Setting up Kannel is not hard, the docs are very good and can be found here and there is a copy of my config files as a guide in the resources section

Mbuni

There is some good documentation for setting up the full MMS gateway version of Mbuni, but because the cell phone plugin is stil only in the development stream there is only a small sample config file and the source code. I have tried to document what I have learned setting it up here.

Since this is a bleeding edge function you will need to build Mbuni from the src in cvs. There are instructions on how to do this on the web site here, but here is a short version

cvs -d:pserver:anonymous@mbuni.cvs.sourceforge.net:/cvsroot/mbuni login
cvs -z3 -d:pserver:anonymous@mbuni.cvs.sourceforge.net:/cvsroot/mbuni co -P
mbuni
cd mbuni
./bootstrap
./configure
make
cd extras/mmsbox-mm1
make
cd ../..
su -c "make install"

The change of directory to the extras/mmsbox-mm1 is to build the extra library needed to work with the phone. If you are running on a machine that has SELINUX enabled you will need to run the following command to allow the library to work

su -c "chcon -t texrel_shlib_t /usr/local/lib/libmmsbox_mm1.so"

Like Kannel, Mbuni is made up of a collection of applications

  • mmsc
  • mmsproxy
  • mmsrelay
  • mmsbox

To send MMS messages via a phone we only need mmsbox which is what is known as a VAS gateway. So we need to create a Mbuni config file, there is a sample file shipped with the src in the doc/examples directory. Here is my version modified to work with O2 UKs MMS service

group = core
log-file = /var/log/kannel/mmsbox.log
access-log = /var/log/kannel/mmsbox-access.log
log-level = 0

group = mbuni
storage-directory = /usr/local/var/spool/mbuni
max-send-threads = 5
maximum-send-attempts = 50
default-message-expiry = 360000
queue-run-interval = 5
send-attempt-back-off = 300
sendmms-port = 10001
sendsms-url = http://localhost:13013/cgi-bin/sendsms
sendsms-username = tester
sendsms-password = foobar

# Sample conf for MMSBox using a modem (MM1)
group = mmsc
id = modem
type = custom
custom-settings = "smsc-on=lynx -dump 'http://localhost:13000/start-smsc?
password=bar&smsc=w880i'; 
 smsc-off=lynx -dump 'http://localhost:13000/stop-smsc?password=bar&smsc=w880i'; 
 gprs-on=/usr/bin/pon o2; 
 gprs-pid=cat /var/run/ppp0.pid | head -1;port=13014;
 mmsc-url=http://mmsc.mms.o2.co.uk:8002;proxy=193.113.200.195:8080;msisdn=100"
mmsc-library = /usr/local/lib/libmmsbox_mm1.so

group = send-mms-user
username = tester
password = foobar
faked-sender = 100

The interesting bits are the sendsms-url and the custom-settings lines. The sendsms-url points to the bearerbox/smsbox URL from setting up Kannel earlier which Mbuni uses to send the notification about the new mail.

The custom-settings line is a lot more complicated, it is basically a mini config file all of it’s own. The two entries that start with smsc-on and smsc-off are commands that the custom library built earlier uses to stop and start the sms gateway while the MMS message is sent. gprs-on is used to start a PPP session via the phone. This can be either gprs or 3G. The code implies that this command should not return until it’s killed at shutdown, but using /usr/bin/pon on Ubuntu seams to work.

The next few bits depend on which cell phone provider your using. The mmsc-url and proxy are the addresses for the machines on their network you need to use to
send MMS messages. I found the following page has a good list of the settings for UK provider http://www.filesaveas.com/mms.html

SMIL

So now we’ve got the set up working we need some content to send. MMS messages are defined using SMIL markup. The following is the simple SMIL file I am using to send a short video clip and text caption. The first half divides the display in half, with the video in the top half and the text in the lower. The second section contains the details of the links to where Mbuni can find the content to
fill those areas and how long to display them. This is a very simple example, much more complex messages can be assembled with
SMIL.

<smil>
   <head>
      <layout>
         <root-layout />
         <region id="Image" top="0" left="0" height="50%" width="100%" fit="hidden" />

         <region id="Text" top="50%" left="0" height="50%" width="100%" fit="hidden" />
      </layout>
   </head>
   <body>
      <par dur="5000ms">
         <video src="http://tiefighter.loc/cam1/intruder.3gp" region="Image"></
         video>

         <text src="http://tiefighter.loc/cam1/message.txt" region="Text"></text>
      </par>
   </body>
</smil>

ffmpeg

So far we have been just sending static images, next comes converting the avis created by Motion to mpeg4 in a 3gp container that should be playable on any MMS capable phones. The following ffmpeg command will convert the file to the right format.

ffmpeg -i 07-20090916100019.avi -s qcif -vcodec h263 -y intruder.3gp

Where “-i 07-20090916100019.avi” is the file created by motion, “-s qcif” tells ffmpeg to use an output file that is 176×144 and conforms to a standard that all phones should understand, “-vcodec h263″ is the video codec to use. “-y intruder.3gp” tells ffmpeg to overwrite the file if it already exists.

Here is an example of the Motion output.

After transformation:

When viewed on a 2 inch screen the drop in quality is not noticeable and it is still possible to tell if it is somebody you know.

Actually sending the MMS

So now we have actually created the content for the MMS message we need to put it somewhere mbuni can find it. In this case I put the video and text files into the /cam1 directory being server up by http server. The URLs match the entries in the SMIL file created earlier.

Now we need to send the SMIL file to Mbuni along with the phone number to send it to. The following curl command will send the SMIL file and the rest of it.

curl  --data "username=tester&password=foobar&to=07703xxxxxx&subject=Possible+Intruder&from=07543xxxxxx" --data-urlencode "smil@/var/www/html/cam1/intruder.smil"  http://x-wing.loc:1000

In this case Mbuni is running on the a machine called x-wing and listening on port 10001 (as set with the sendmms-port in the config above). The frist half is the urlencoded version of the username, password, the senders and recipients numbers and the subject of the message. The second section, after the –data-urlencode loads the SMIL file and encodes it before sending it.

Putting it together

Now we need a to collect all of this up in a scrip to attach to the movie end action of motion. The following script first helps to prevent false alarms by ensuring that any video has at least 15 frames. Assuming that test is satisfied the orginal version of the video is emailed offsite for safe keeping, before converting the it to the 3gp format. It then adds the time and date to the message.txt before sending the SMIL to Mbuni.

#!/bin/sh

FRAME_COUNT=`/home/hardillb/bin/frameCounter $1`

if [ $FRAME_COUNT -gt 14 ]; then

    uuenview -a -m foobar@xxxxxx.xxx $1 <<EOF
Subject: Movement detected video $2 $3

EOF

    /home/hardillb/bin/sendMessage SMS/Outbound "TO: +447703xxxxxx MESSAGE: possible intruder"

    ffmpeg -i $1 -s qcif -vcodec h263 -y /var/www/html/cam1/intruder.3gp

    echo "$2 $3" > /var/www/html/cam1/message.txt

    curl  --data "username=tester&password=foobar&to=07703xxxxxx&subject=Possible+Intruder&from=07543xxxxxx" 
        --data-urlencode "smil@/var/www/html/cam1/intruder.smil"  http://x-wing.loc:10001
fi

The frameCounter was a script I had run up earlier for a different project, made sense to reuse it here.

#!/bin/sh
mplayer -v $1 -nosound -vo null -frames 0 2> /dev/null | grep "frames  t" | awk ' { print $3 } '

So that’s about it, I have a motion activated CCTV system that will log off site and send alerts to me anywhere in the world with enough detail to decided if it’s a false alarm

Resources

How to get a Txt Msg from your burglar

So this is a project I’ve been picking away at for some time now, adding new bits along the way. I was looking for a way to keep an eye on my flat while spending a lot of time on the road with work.

Image CC licensed by Dr_John2005 on Flickr

Motion

While hunting around online I found a reference to a small application called Motion which will monitor a web cam and can trigger alerts when the image changes. Along with the alerts it can capture the images or video of what has changed and has filters to detect the gradual changes as it gets light or dark during the day.

My motion config file is in the Resources section at the end.

Originally I had to build Motion from src, but it was recently added to the extras rpm repository I use with Fedora 11. Once installed it’s relatively easy to configure, initially I just had to update the config file (/etc/motion/motion.conf) point it at my web cam (/dev/video0) and a directory to save the images in.

Motion has a number of useful features, once running you can update all of the settings via web interface and it will also stream the video from the camera on a different port which is good for testing. There is also a REST style interface which can be used to turn the motion detection on and off, more on that later.

After that I set up Motion to email the images to a spare gmail account to ensure there was a off site copy of the images just in case the computer with the camera was stolen. The following script was attached to the “on_picture_save” event as follows:

...
on_picture_save /home/user/bin/intruder.sh %f %H:%M:%S %d-%m-%Y
...

Where %f is the path to the image and %H:%M%S %d-%m-%Y is a time and date stamp.


#!/bin/sh
uuenview -a -m foobar@xxxxxx.xxx $1 << EOF
Subject: Possible Intruder $2 $3

EOF

This sends a mail with the time and date in the subject and the image in the body.

So that was not a bad solution, I normally run Thunderbird or have the gmail manager plugin for Firefox running so I get updates about new emails relatively quickly. But then I ended up at a customer who would not let me hook my laptop up to internet while in the office so I started to look for a different notification mechanism to add on as well.

Gnokii

The next perfect solution seamed to be SMS messages. I had a spare mobile phone and pay-as-you-go sim cards are free. So back to searching the net for some software to send SMS messages. I found 2 different Linux packages that would do this. The first is called Gnokii which is part of a package that started out as a way to back up and interact with Nokia phones. Gnokki’s smsd works from two simple mysql tables, one for inbound and one for outbound messages. It polls the outbox table at a configuable interval (I’ve set mine to 5 seconds) and sends any unprocessed messages.


Image CC licensed by Eerko on Flickr

The second SMS package for Linux is called Kannel. Kannel is a lot more than just a program to send SMS messages, it is also a full WAP stack. It is very powerful but requires 2 processes and webserver to be able to send and receive messages.

So to start with I chose Gnokii as the set up was a lot simpler. You basically just need to get a serial connection to your phone, this can be either via a USB cable or you can set up a connection via Bluetooth using the rfcomm command.

I am using a Sony w880i as my phone and it comes with a USB cable which doubles as a charger which makes things easier. When I plugged it into my box it showed up as 2 Serial ports (/dev/ttyACM0 and /dev/ttyACM1) they both seam to behave the same.

I added a new script to the “on_movie_end” Motion trigger

...
on_movie_end /home/user/bin/sendSMS.sh %H:%M:%S %d-%m-%Y
...
#!/bin/sh
/usr/bin/mysql -u smsgw -p xxxxxx smsgw << SQL
insert into outbox (number,text)values ('+4477xxxxxxxx','Possible Intruder - $1 $2');
quit
SQL

There is a sample Gnokii config file in the Resources section

Anybody Home?

So that is a good start, we’ve got a system that will detect motion and raise the alarm. But at the moment it’s turned on all the time, which is no good. This is where Motion REST style control API comes in. If I had a way to automatically tell when I’m at home I can turn off the motion detection. I discovered a blog post by another guy at Hursley called Gareth Jones

Who seamed to have already solved this problem. Gareth had put together some Python that scans for Bluetooth devices with in range of the computer and sends alerts as devices arrive or leave. Since most people carry a Bluetooth device arround with them all the time in the shape of a mobile phone this seamed like a good idea.

At the time I was looking at this bit I didn’t have time to go digging in Python, so I hacked up a quick shell script verion that would just track one phone and then use the Motion REST API to pause the detection when ever I’m at home and turn it on again when I leave.

The detection is done using the l2ping command from the Bluez package and giving it the Bluetooth mac address of the phone I want to track.


#!/bin/sh

MAC=00:AA:BB:CC:DD:EE

while true; do
  if [[ `sudo l2ping -c 1 $MAC 2> /dev/null | grep -c "44 bytes from"` == "1" ]]
  then
    echo "home"
    if [[ `cat /home/user/home` != "YES" ]]
    then
      echo YES > /home/user/home
    fi
     lwp-request -C user:password http://127.0.0.1:8080/0/detection/pause > /dev/null
     sleep 90
  else
    echo "away"
    if [[ `cat /home/user/home` != "NO" ]]
    then
      echo NO > /home/user/home
    fi
    lwp-request -C user:password http://127.0.0.1:8080/0/detection/start > /dev/null
    sleep 15
  fi

done

MAC is the bluetooth address of my phone, you can discover this by running the following command “hcitool scan” which will produce a list of all the bluetooth devices your machine can currently see and their names.

So this is where I left the solution for quite a while because I couldn’t find a project that would allow me to send MMS messages from Linux without having an arrangement with a cell phone
network.

Resources