I’ve spent the last few days on site at a bespoke coach builder’s helping to fit out a truck, this all came about after a chat over lunch at the end of last year. Kevin mentioned a project he was involved in to put some our lab demo’s into a truck that will be touring round a number of universities as part of the Smarter Planet initiative.
As well as building portable versions of some demo’s there was a new bit. The plan was to have a sort of photo booth to take pictures of the visiting students and build a custom avatar for them to use at each station and upload to sites like Flickr and Facebook.
Since we were already doing some work for the truck we said we would have a look at doing this bit as well, about half an hour of playing that afternoon it looked like we should be able to put something together using a commodity webcam, Linux and some existing libraries.
First Pass
The first approach was to see if we could do what we needed in a reasonably simple web page. Using a streaming video server, the HTML 5 <video> tag and a bit of javascript we had a working prototype up and running very quickly.
The only problem was the lag introduced by the video encoding and the browser buffering, most of the time it was about 12 seconds, with a bit of tinkering we got it down to 5 seconds, but this was still far too long to ask somebody to pose for in order to just grab a single image.
Second Pass
So after getting so close with the last attempt I decided to have a look at a native solution that should remove the lag. I had a bit of a look round to see what options where available and I came across the following:
- Video4Linux
This is gives direct access to the video hardware connected to Linux - GStreamer
This is a framework that allows you to build flows for interacting with media sources. This can be audio or video and from files as well as hardware devices.
As powerful as the Video4Linux API is it’s seamed a bit too heavy weight for what I was looking for. While looking into the GStreamer code I found it had a pre-built package that would do pretty much exactly what I wanted called CameraBin.
With a little bit of python it is possible to use the CameraBin module to show a Viewfinder and then write an image on request.
self.camerabin = gst.element_factory_make("camerabin", "cam")
self.sink = gst.element_factory_make("xvimagesink", "sink")
src = gst.element_factory_make("v4l2src","src")
src.set_property("device","/dev/video0")
self.camerabin.set_property("viewfinder-sink", self.sink)
self.camerabin.set_property("video-source", src)
self.camerabin.set_property("flicker-mode", 1)
self.camerabin.connect("image-done",self.image_captured)
Where self.sink is a Glade drawing area to use as a view finder and self.image_captured is a call back to execute when the image has been captured. To set the filename to save the image to and start the viewfinder run the following code.
self.camerabin.set_property("filename", "foo.jpg")
self.camerabin.set_state(gst.STATE_PLAYING)
To take a photo call the self.camerabin.emit(“capture-start”) method
The plan was for the avatar to be a silhouette on a supplied background, to make generating the silhouette easier the students will be standing in front of a green screen
The Python Imaging Library makes manipulate the captured image and extract the silhouette and then build up the final image from the background, the silhouette and finally the text.
image = Image.open(path)
image2 = image.crop((150,80,460,450))
image3 = image2.convert("RGBA")
pixels = image3.load()
size = image2.size;
for y in range(size[1]):
for x in range(size[0]):
pixel = pixels[x,y]
if (pixel[1] > 135 and pixel[0] < 142 and pixel[2] < 152):
pixels[x,y] = (0, 255, 0, 0)
else:
pixels[x,y] = (0, 0, 0, 255)
image4 = image3.filter(ImageFilter.ModeFilter(7))
image5 = image4.resize((465,555))
background = Image.open('facebook-background.jpg')
background.paste(image5,(432,173,897,728),image5)
text = Image.open('facebook-text.png')
background.paste(text,(0,0),text)
The final result shown on one of the plasma screens in the truck.
As well as building Silhouette wall, ETS has provided a couple of other items to go on the truck
-
See It Sign It
This application is a text to sign language translation engine that uses 3D avatars to sign. There will be 2 laptops on the truck that can be used to have a signing conversation. There is a short video demonstration of the system hooked up to a voice to text system here: http://www.youtube.com/watch?v=RarMKnjqzZU
-
Smarter Office
This is an evolution of the Smarter Home section in the ETS Demo lab at Hursley. This uses a Current Cost power meter to monitor the energy used and feeds this to a Ambient Orb to visualise the information better. It also has a watch that can recognise different gestures which in turn can be used to turn things like the lamp and desk fan on and off and the amount of power used by these is reflected in the change in colour from the orb.
For details of where the truck will be visiting over the year, please visit the tours facebook page in the resources.
Resources
- Gstreamer – http://www.gstreamer.net/
- Gstreamer CameraBin – http://www.gstreamer.net/wiki/CameraBin
- Python Imaging Library – http://www.pythonware.com/products/pil/
- HTML 5 Video frame grabbing – http://www.sanraul.com/2009/12/17/using-html5-canvas-to-capture-frames-from-a-video/
- IBM Smarter Planet Comes to You – http://www.facebook.com/ibmskillstour