WIFI presence detection

Back in my very first post I talked about using Bluetooth to detect my presence at home in order to disable the CCTV system and control a few other things.

While this works well it does not scale well to multiple people as the Bluetooth layer 2 ping takes about 5 seconds to time out if the device in not in range. This means that at most 12 different phones can be checked in a minute.

A couple of recent chats with a few people at work (Vaibhavi Joshi & Dale Lane and Bharat Bedi) got me thinking about this again. Modern phones tend to have WIFI as well as Bluetooth and 3G radios these days so I thought that I’d have a look at seeing if this could be used to locate devices.

After bit of a poke around it looked like a package called Kisment should be able to do what I wanted.

Kismet

Kismet is a client server application, the backend server reads from the network card and decodes the packets and the UI which requests data from the server over a socket connection. This also means the backend can be on a different machine, in fact several different drone backends can be consolidated in a single master backend server and all the captured data presented to UI. This means you could distribute a number of drones over site and generate a map as devices move between areas covered by the different backends.

The default client is a ncurses based application that can list all the visible networks and a chart showing the incoming packet rates. It’s great for getting a view of what networks are active which can be very useful when you have to set up a new one and want to see which channels are free.

Rather than use the default client I decided to write my own to drive the backend the way I wanted it and to make exposing the data easier (I’m going to publish detected devices on a MQTT topic). But first I had a bit of a play using the netcat (nc) command. Netcat basically pipes stdin/stdout to and from a given socket, this is useful because the Kisment protocol is just a set of simple text commands. For example the following command will get the kismet backend to return a list of all the clients it has seen to date.


echo -e '!0 enable client MAC,manuf,signal_dbm,signal_rssi' | nc localhost 2501


Returns something that looks like this:

...
*CLIENT: 00:25:69:7D:53:D9 [0x01]SagemCommu[0x01] -71 0
...

The only tricky bit about the response is that any field that can contain a space is wrapped in characters with a value of 0x01, in this case the manufacture field could contain spaces so we need the following regexp to chop up the responses for each time a client is spotted.

I decided my client in Java (because the MQTT libraries are easy to use) so I chose to use a regular expression to split up the response

Pattern.compile("\*CLIENT: ([0-9A-F:]*) \x01(.*)\x01 (-?\d+) (\d) ");

By default Kismet cycles round all the available channels to try and get a full picture of all the WIFI traffic in range, but this means it can miss some packets and in turn miss clients that are not generating a lot of traffic. To help get round this I have locked Kismet to just listen on the same channel as my WIFI access point since all my devices are likely to try and connect to it as soon as it comes in range and there is less chance of me missing detecting my phone up front.

!1  HOPSOURCE cab63dc8-9916-11e0-b51a-0f04751ce201 LOCK 13

cab63dc8-9916-11e0-b51a-0f04751ce201 is the UUID assigned to the wifi card by kismet and the 13 is the channel I run my WIFI access point on. You can find the UUID by running the following command:

echo -e '!1234 enable source type,username,channel,uuid' | nc localhost 2501

Which returns a string that looks like this every time the back end hops to new channel.

*KISMET: 0.0.0 1308611701 [0x01]Kismet_2009[0x01] [0x01]alert[0x01] 0 
*PROTOCOLS: KISMET,ERROR,ACK,PROTOCOLS,CAPABILITY,TERMINATE,TIME,PACKET,STATUS, 
PLUGIN,SOURCE,ALERT,WEPKEY,STRING,GPS,BSSID,SSID,CLIENT,BSSIDSRC,CLISRC, 
NETTAG,CLITAG,REMOVE,CHANNEL,SPECTRUM,INFO,BATTERY 
*SOURCE: orinoco_cs test 3 30b9b5a4-9b93-11e0-acfe-ee054e2c7201 
*ACK: 1234 [0x01]OK[0x01]

Publishing the last seen time on the following topic /WIFIWatch/<mac> allows applications to register to see a specific device and also build up a list of all devices ever seen and when.

WIFI Watch topics

It’s not just phones that have WIFI adapters these days, net books, tablets even digital cameras (with things like eyefi) all have , also with multiple kismet nodes it might be possible to track devices as they move around an area.

Next is to look at the signal strength information to see if I can judge a relative distance from the detection adapter.

Resources:

Kismet

Tracks2Mile 1.1.4 – With automatic details import

After discovering that my patch for My Tracks had been pulled just before their last release (1.1.5) I’ve been looking for another way to get the information about time and distance.

After another closer look at the instructions 3rd party use instruction on the wiki and it’s possible to access the My Tracks database directly. Running a query that filters on the workout title, which I can get from the GPX files name, I can get hold of all the information. It is possible to have multiple workouts with the same name so I’m picking the last one added to the db since this has to be the most likely one you’ll be wanting to upload.

Since I’m reading direct from the database there is also the problem that if the My Tracks guys change the table layout then I could have a problem. The API provides a method that returns a Track object which holds the data but you can only use this if you know workout id, but this available at the moment.

I’ll probably stick a new patch in for My Tracks to pass the workout id so I don’t have to make this assumption, but it should be good enough for now.

Other fixes include:

  • One to stop caching photos in memory before writing them to the sd card to prevent OutOfMemoryErrors
  • Closed a couple of windows where the database was accessed after it’s been closed.

Grab it from the Android Market here

My Tracks patch pulled

So it looks like my patch to My Tracks to get it to expose the extra information needed for Tracks2Miles got pulled just before the last release because it broke the export to GMail (and dropbox).

It’s a shame it got spotted so close to the release of My Tracks 1.1.5 I didn’t get chance to look at before it went out.

It looks like Android resolves and unpacks all the Parcellable extras upfront even if they are not used by the receiving Activity or Service. So apps not expecting the Track object I was using fail with ClassNotFoundException because it’s not available.

I’ll try and find some time to come up with a new solution as soon as I get some time to think about it properly.

Tracks2Miles 1.1.0

The latest Tracks2Miles update

Added a timeline view to see what you friends have been up to.

*** Edit ***

I’ve just noticed that on some phones that after the update the launch icon will still open the Activity for publishing a new workout rather than the Timeline Activity. This is a “feature” of some of the launchers, you may need to restart your phone to get it to see the update.

*** Edit ***

Fixed up a couple of race condition type problems.

Updated the code to import from My Tracks to use the patch that has been accepted for the next release.

My Tracks Patched

The guys at My Tracks have integrated the changes I sent to enable more data to be passed with the GPX file. There are some slight changes needed to the currently released version of Tracks2Miles but I will push a new version in the next couple of days to be ready for the next My Tracks drop.

I don’t know when the next version of My Tracks will ship yet, but we will be ready,

In the meantime here are the details of the change so others can use it to build similar add-ons for any other site (why they would want to use anything but dailymile?).

As mentioned back in the first post on this subject your going to need an Intent filter that will trap when My Tracks is looking to export a GPX file (the changes to My Tracks apply to all file types, but I’m only going to talk about GPX here). The following is what I’ve been using for Tracks2Miles:

 <activity android:name=".Export"
      android:label="@string/app_name" android:exported="true">
   <intent-filter>
     <action android:name="android.intent.action.SEND" />
     <category android:name="android.intent.category.DEFAULT" />
     <data android:mimeType="application/gpx+xml" />
   </intent-filter>
 </activity>

This hooks the “android.intent.action.SEND” action and filters it for the mimeType “application/gpx+xml”. So in the Activity that gets called when an intent matching this is broadcast you will need a onCreate method that looks something like this.

public void onCreate(Bundle savedInstanceState) {
  Intent intent = getIntent();

  if (intent.getAction().equals(Intent.ACTION_SEND)) {
    if (intent.hasExtra("com.google.android.apps.mytracks.TRACK")) {
      Track track = (Track) intent.getParcelableExtra("com.google.android.apps.mytracks.TRACK");
      ...
    }

    Uri gpxURI = (Uri)intent.getExtras().get(Intent.EXTRA_STREAM);
    ...
  }
}

Where Track is an instance of com.google.android.apps.mytracks.content.Track. You will need to grab this and the 2 classes in the package com.google.android.apps.mytracks.stats so they can be on the classpath for getParcelableExtra to find them. From the Track object you can get hold of name, time and distance of the workout and also there is a Statistics field that has info on average/max speed, elevation information.

I’m planning on writing a My Tracks wiki page about this some time soon where I’ll go in to a little bit more detail but this should get most people started.

Tracks2Miles 1.0.4

A quick update about the 1.0.4 * update release of Tracks2Miles.

  • Added some better error path handling to make it a bit more graceful should it fall over.
  • Added a local DB to cache route information, this is for 2 reasons. Firstly to make it quicker to just pick a route without having to wait for the list to be downloaded and secondly as a work around for the missing distance info in routes uploaded via the API. One of the side effects of this is that previously uploaded routes with no distance will not be shown in the list. Look at the end of the post for a way to fix these routes.
  • Added a bit more validation before posting, this should make it harder to post routes without titles or time and distances. A dialogue will ask if you really want to do this before posting.

I’ve also found a way to fix the uploaded routes missing distances. If you view the routes on the dailymile website, then hit the edit link at the top it will open the route editor. At this point it will recalculate the distance and you can just hit the Update button in the bottom right hand corner. The route should now have a valid distance.

Based on Dan’s comments on the announcement entry, I’m planning on looking to see if I can identify an existing route before posting a new one from My Tracks, this will reduce upload time and allow for dailymile to track your best time. To keep things easy it will probably just be based on the name of the track and distance.

I have added the following short video to the Android Market entry.

* 1.0.2 & 1.0.3 wouldn’t work on 2.1 due to using a method in API v8, I’ll remember to check 2.1 in the emulator before uploading next time

Announcing Tracks2Miles

OK, so the last 3* posts have been about the development of this so I guess you all knew it was coming, but now after some kind folk (Thanks @davidgeorgeuk, @bodonovan, @TheMightyAl & @ragtag) beta tested it for me it’s time to release it to the world.

A Friday afternoon chat between myself, Karl Roche, Brian O’Donovan and Dale Lane about tracking exercise habits with the website dailymile led to wondering why there was no Android app for it yet. A quick look at the API suggested it shouldn’t be too hard run something up. A short weekend of hacking later the majority of Tracks2Miles was done.
Tracks2Miles Icon
Tracks2Miles is an Android application to take workouts recorded with My Tracks and uploads them dailymile.

Rather than reinvent the wheel and write a full GPS tracking app I decided to see how easy it would be to extend one of the existing apps. My Tracks was one of the first training apps released for Android so it seamed like a good place to start. My Tracks have released a library to allow embedding the GPS tracking into new apps, but even this looked like a lot of work, but My Tracks also has a feature to “Share with friends…” which uses the android.intent.action.SEND intent action to allow other apps to forward on the data in a number of formats, one of which is GPX. This made writing the Tracks2Miles very easy.

Current Features:

  • Export routes from My Tracks – use the “Share with friends…” menu option to share the current track
  • Post workouts without using My Tracks – Tracks2Miles can be launched directly to upload a workout without uploading a new track e.g. treadmill seasons
  • Existing route reuse – start Tracks2Miles from the launcher and hit the “Use existing route” button and you can use routes already uploaded to dailymile

Possible Upcoming Features:

  • Support other apps – add support for exporting from apps like Endomondo or Runkeeper if they can export GPX files
  • Export any GPX – search the file system and upload any GPX found
  • dailymile timeline – a way to see what everybody else has been up to and add comments and motivation
  • Offline caching – save on bandwidth and API calls
  • User preferences – store defaults for Rating and workout type type
  • Contact syncing – add your running buddies to your phones contact list
  • Retry on network loss – uploading long routes can take a while over slow networks so a bit more fault tolerance would be good
  • Widget – because every app needs a widget

Known issues:

  • Facebook login not working – When authorising Tracks2Miles using an account created using your Facebook account it says password is wrong. This looks like either a problem on the dailymile or Facebook side of things, I have reported it
  • Time and distance not filled in when exporting from My Tracks – a patch to make this happen have been submitted to My Tracks, hopefully it will make it into the next release.
  • Reused routes have zero length – Some routes returned by dailymile are shown to have zero length so the distance is not filled in when using them. I have reported this to dailymile.

I’ve tested this on a HTC Desire, a ZTE Blade (Orange San Francisco) (updated to 2.2), a Samsung Galaxy and a Motorola Defy devices along with in the emulator running 2.1, 2.2 and 2.3. It should run on Android 2.1 or newer.

The Android Market URL is https://market.android.com/details?id=uk.me.hardill.dailymile or us the following QR code:

*

  1. My Tracks to dailymile GPX route exporter – part 1
  2. My Tracks to dailymile GPX route exporter – part 2
  3. My Tracks to dailymile GPX route exporter – part 3

My Tracks to dailymile GPX route exporter – part 3

So I’ve got the oAuth 2.0 all sorted it’s time to write the code to do the actual publish to dailymile.

First up I need a UI to capture all the other info about the workout other than the route. Not all of the options on the website are available via the API, but there are enough make a useful post.

I wanted to fill in as many of the fields as possible, so the route name comes from the GPX file name. When My Tracks creates the intent to share the GPX file it adds 2 extras which map to the equivalent of an email subject and email some body text to go with the file, but none of this contains data like the distance covered or the time. This information isn’t explicitly in the default GPX files either.

I could parse the GPX file and generate this info, but it’s going to be a heavy weight process and the data is something My Tracks should already have. I had a quick look at some of the other formats that My Tracks can export and found a the track description field in the KML version which holds the info I’m looking for.


<desc><![CDATA[Created by <a href='http://mytracks.appspot.com'>My Tracks</a< on Android.<p>Total Distance: 223.32 km (138.8 mi)<br>Total Time: 1:57<br>Moving Time: 0:20<br>Average Speed: 6866.26 km/h (4266.5 mi/h)<br> Average Moving Speed: 40197.14 km/h (24977.3 mi/h)<br> Max Speed: 0.00 km/h (0.0 mi/h)<br>Min Elevation: 0 m (0 ft)<br>Max Elevation: 0 m (0 ft)<br>Elevation Gain: 0 m (0 ft)<br>Max Grade: 0 %<br>Min Grade: 0 %<br>Recorded: Tue Apr 12 18:53:08 GMT+00:00 2011<br>Activity type: -<br><img border="0" src="http://chart.apis.google.com/chart?&chs=600x350&cht=lxy&chtt=Elevation&chxt=x,y&chxr=0,0,223,37|1,0.0,100.0,25&chco=009A00&chm=B,00AA00,0,0,0&chg=100000,25.0,1,0&chd=e:,"/>]]></desc>

There is a similar field in the GPX file that is currently empty. A quick look at the code showed that a 5 line patch should populate that field with the same data. I’ve sent the patch off to the My Tracks guys, but in the mean time if the field is empty the user will have to fill in the time and distance by hand.

With the UI sorted, the publishing is done in a service as it’s a multi-stage process including uploading the GPX file which could take a while over mobile networks. It’s basically 3 dailymile API calls

  1. Create a new route – POST https://api.dailymile.com/routes.json
  2. Upload GPS data to form the route – PUT https://api.dailymile.com/entries/id/track.json
  3. Create new workout entry – POST https://api.dailymile.com/entries.json

Interspersing these with Notification updates so the user can see how far through the process it is.

 

private void upload() {
HttpResponse resp;
final HttpClient httpClient = new DefaultHttpClient();</code>

String ns = Context.NOTIFICATION_SERVICE;
NotificationManager notificationManager = (NotificationManager) getSystemService(ns);

Notification notification = new Notification(R.drawable.icon, "Uploading Workout", System.currentTimeMillis());
Context context = getApplicationContext();

Intent notificationIntent = new Intent();
PendingIntent contentIntent = PendingIntent.getActivity(this, 0, notificationIntent, 0);

notification.setLatestEventInfo(context, "Uploading Workout", "Creating new Route", contentIntent);
notification.flags |= Notification.FLAG_ONGOING_EVENT;
notification.flags |= Notification.FLAG_NO_CLEAR;
notificationManager.notify(1, notification);
Log.i("DailyMile",Constants.NEW_ROUTE_URL + authToken);
final HttpPost post = new HttpPost(Constants.NEW_ROUTE_URL + authToken);
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
nameValuePairs.add(new BasicNameValuePair("name", routeName));
nameValuePairs.add(new BasicNameValuePair("activity_type", routeType));
try {
post.setEntity(new UrlEncodedFormEntity(nameValuePairs));
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String routeID = "";
try {
resp = httpClient.execute(post);
if (resp.getStatusLine().getStatusCode() == 201) {
InputStream in = resp.getEntity().getContent();
StringBuffer buffer = new StringBuffer();
for (int i = in.read(); i != -1; i = in.read()) {
buffer.append((char)i);
}
JSONObject jsObject = new JSONObject(new JSONTokener(buffer.toString()));
routeID = jsObject.getString("id");
Log.i("DailyMile", routeID);
} else {
Log.i("DailyMile","" + resp.getStatusLine().getStatusCode());
notification.setLatestEventInfo(context, "Uploading Workout", "Failed to create a new route", contentIntent);
notification.flags |= Notification.FLAG_AUTO_CANCEL;
notificationManager.notify(1, notification);
//barf
return;
}
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
notification.setLatestEventInfo(context, "Uploading Workout", "Uploading Route", contentIntent);
notificationManager.notify(1, notification);
//{"id":622197,"activity_type":"Fitness","name":"test"}

String putURL = Constants.UPDATE_ROUTE_URL.replace("{id}", routeID) + authToken;
Log.i("DailyMile", putURL);
final HttpPut put = new HttpPut(putURL);
try {
ParcelFileDescriptor pfd = this.getContentResolver().openFileDescriptor(gpxURI, "r");
InputStream in = this.getContentResolver().openInputStream(gpxURI);
InputStreamEntity entity = new InputStreamEntity(in, pfd.getStatSize());
entity.setContentType("application/gpx+xml");
put.setEntity(entity);
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

try {
resp = httpClient.execute(put);
if (resp.getStatusLine().getStatusCode() != 201) {
Log.i("DailyMile","" + resp.getStatusLine().getStatusCode());
notification.setLatestEventInfo(context, "Uploading Workout", "Failed to upload new route", contentIntent);
notification.flags |= Notification.FLAG_AUTO_CANCEL;
notificationManager.notify(1, notification);
//barf
return;
}
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

notification.setLatestEventInfo(context, "Uploading Workout", "Adding new workout", contentIntent);
notificationManager.notify(1, notification);

final HttpPost postWorkOut = new HttpPost(Constants.NEW_ENTRY_URL+authToken);
StringEntity entity = null;
JSONObject json = new JSONObject();
JSONObject workout = new JSONObject();
JSONObject distance = new JSONObject();
try {
json.put("message",workoutMessage);
workout.put("duration",workoutDuration);
workout.put("activity_type", routeType);
workout.put("felt", workoutRating.toLowerCase());
workout.put("route_id",Integer.parseInt(routeID));
distance.put("units", workoutDistanceUnits);
distance.put("value",workoutDistance);
workout.put("distance", distance);
json.put("workout", workout);
Log.i("DailyMile",json.toString());
entity = new StringEntity(json.toString());
entity.setContentType("application/json");
} catch (JSONException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (UnsupportedEncodingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
postWorkOut.setEntity(entity);

try {
resp = httpClient.execute(postWorkOut);
if (resp.getStatusLine().getStatusCode() != 201) {
Log.i("DailyMile","" + resp.getStatusLine().getStatusCode());
notification.flags |= Notification.FLAG_AUTO_CANCEL;
notification.setLatestEventInfo(context, "Uploading Workout", "Failed to upload new workout", contentIntent);
notificationManager.notify(1, notification);
//barf
return;
}
} catch (ClientProtocolException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}

notification.setLatestEventInfo(context, "Uploading Workout", "All Done", contentIntent);
notification.flags |= Notification.FLAG_AUTO_CANCEL;
notificationManager.notify(1, notification);

}

 

So it’s all done now, just got to jump trough a few hopes to get it sign off to publish it in the Android Market and hopefully the My Tracks guys will include my patch in the next drop.

My Tracks to dailymile GPX route exporter – part 2

In my last post I talked about a project I’m working on to export GPS tracks from My Tracks directly to dailymile.

I was having problems getting the mobile client flow for oAuth 2.0 to work with dailymile. As a work around I decided to see if I could get the User-Agent flow to work for this application.

At the end of the previous post I suggested using an Intent filter to catch a redirect to a custom URI schema, while I got this to work it was messy as it left the browser app on the activity stack during the authentication. While looking for a way round this I found the documentation for the WebView widget and came up a much better solution. The WebView widget allows you to embed HTML content in your application so it should be possible to embed the web based authorisation set into a Activity.


<pre>
getWindow().requestFeature(Window.FEATURE_PROGRESS);
WebView webView = new WebView(this);
setContentView(webView);
final Activity activity = this;
webView.loadUrl(Constants.AUTH_URL);
</pre>

In the onCreate method of the Activity I have done away with a normal layout and just used a full screen WebView and asked it to load the authentication URL from the User-Agent Flow

https://api.dailymile.com/oauth/authorize?response_type=token&client_id=<CLIENT_ID>&redirect_uri=dm://locahost

Login Authorise

The callback redirect URI (dm://localhost) still has a custom schema similar to the one suggested in the previous post. Assuming the user allows access to their dailymile profile we can intercept the redirect by overriding the WebViewClient on the Webview which will look a bit like this:

dm://localhost#access_token=[ACCESS_TOKEN]

webView.setWebViewClient(new WebViewClient() {
  @Override
  public boolean shouldOverrideUrlLoading(WebView view, String url) {
    if (url.startsWith("dm://")) {
      String summary = "<html><body>Grabbing Account info</body></html>";
      view.loadData(summary, "text/html", "utf-8");
      String token = url.substring(url.lastIndexOf("=")+1);
      User user = Utilities.getAccountDetails(token, activity, handler);
      if (user != null) {
        user.setToken(token);
        finishLogin(user);
      }
      return false;
    }
    view.loadUrl(url);
    return true;
}});

Now I’ve got the AuthToken I can tuck it away in the Android AccountManager so I can grab it later when I want to upload a training session.