Comprehensive logging and log transmission

[Nuutti Kotivuori]

I am asking for ideas and comments on a project I'm going to be
doing. I will start from describing what my needs are, starting from
the basic stuff and going on to the blue sky stuff. After that, I will
describe my current thoughts on implementing this.

Problem space
=============

So, I am finally going to get a phone with GPS and am able to easily
write custom software for it.

The obvious thing that I want to do immediately is to just record
everything. Record the GPS position all day long, where ever I go,
always when there is a GPS positioning available. This data is
ofcourse cool to use on the phone, but what I really want to do is to
upload that data somewhere else, where I can do whatever I want with
it. Calculating how much time I spend in various places, measuring
time spent on work trips, live location updates on a web-page
somewhere, the works.

So, what I want to have is constant logging of GPS position at regular
intervals. In addition to that, I want the data automatically uploaded
to a remote site, whenever there is a network connection available. In
my case, I will have an always on GPRS connection, but the same idea
works even when intermittedly connected. Obviously GPS position should
be recorded with timestamps even when there is no network connection
at the moment.

But it doesn't stop there. I want to correlate normal phone events,
calls, SMSs and such to the position data. I want to know where I was
when I received a certain phone call. And not only the common stuff,
but I basically want to log what I did with the phone at a certain
time and place. For example, knowing when I put the phone on silent is
valuable information. Some of this data may already be there on the
phone (does the journal record location data as well?), but this idea
is all about correlating different data sources.

The big picture isn't just about the phone either. There are other
things I wish to place on the same timeline as well. I want to know
when my laptop was idle and when it wasn't - when the lid was closed
and when it was opened. Perhaps even logging the songs I listen
to. The possibilities are endless. One important addition to these is
that I want to be able to quickly jot down notes, which are added to
the timeline and carry position information.

What is done with the data is not really in scope here. I bet everyone
can come up with nice use cases if this data is available. For me, in
the beginning, it's going to be the novelty factor of allowing people
to know where I physically reside - and accountable time tracking
based location and activities without manually marking times down.

Implementation
==============

So, what is needed is so nice and clean way of logging various events
of differing types. This should probably be library and a command-line
tool to allow easy logging from various programs.

Then there should be a daemon or separate program to allow uploading
the events to some remote machine. In my case, the events would be
uploaded every 5 minutes - for some others, they might be transmitted
only when the USB cable is plugged in.

Logging library
---------------

I think I want to use SQLITE 3 as the underlying storage for the
events. It's a smallish library with no external dependencies and
handles concurrency issues decently. Other options would be atleast
Berkeley DB and HDF. I'm trying to avoid anything that would affect
the event-loop of the program or require socket communications.

Logging an event would be really straightforward. Just open the
database, insert one row with timestamp and data, and be done with it.

I will probably use a Python implementation initially, until the
features have stabilized somewhat.

Transmitter daemon
------------------

This would read the database periodically and upload all new events to
a remote server. The connection should probably be a simple TCP
connection outgoing from the phone to go through most networks. To be
really compatible, the entries could be uploaded over HTTP, since that
is better supported than other TCP connections. I think the data is
security critical enough to force the usage of SSL.

In the first phase, I think I'm going to implement the daemon with the
Twisted framework for Python. It can wake up periodically, see if
there's any unsent events, if so, open an SSL connection and upload
the events there. If there are problems in transmissions or no
connectivity, we just try again later.

Event format
------------

Well, obviously the event needs a timestamp. In addition to that, I
think it would be good to enumerate the events with an id number. I'd
rather not trust timestamps for any functionality since a clock can be
wrong.

Then the event needs to actually carry the message - but the format
there is still an open issue. There most likely needs to be a message
type in some form, but should the body be typed? A single string? Byte
array? Key/value mapping? I don't know yet. What I do know is that I'd
like it if it would be possible to avoid specially handling every
event type in the logging libraries and transmitter daemons - so that
the event types would be strictly between the logging application and
the actual system the events are transmitted to. This means that there
should be no changes to the event database or transmission protocol on
new event types.

One idea which I had would be to use the DBUS message encoding with
the signature. It would atleast be a rather compact message
representation. But it seems the libraries do not usually expose just
this packing functionality and all the byte-order hoopla and
everything seems a bit too complex for such a simple issue. A simple
way would be to just use XML for the message body, but I'm not sure if
I want to go for that.

In conclusion
=============

This whole thing has the possibility to grow really big when
considering all the possibilities. I'm trying to first create the
needed functionality on just a desktop computer and start logging some
trivial desktop events. After I get the phone, I want to get the GPS
logging done ASAP, so I will have the data from as early as possible -
I can think of different ways to process it later on.

So, please give me comments and feedback on this.

-- Naked