Comprehensive logging and log transmission

[Nuutti Kotivuori]

Sander van Grieken wrote:
>> Logging library
>> ---------------
>>
>> I think I want to use SQLITE 3 as the underlying storage for the
>> events. It's a smallish library with no external dependencies and
>> handles concurrency issues decently. Other options would be atleast
>> Berkeley DB and HDF. I'm trying to avoid anything that would affect
>> the event-loop of the program or require socket communications.
>>
>> Logging an event would be really straightforward. Just open the
>> database, insert one row with timestamp and data, and be done with it.
>>
>> I will probably use a Python implementation initially, until the
>> features have stabilized somewhat.
>
> What's wrong with plain old syslog?

The problem with syslog is that the lines are in a simple text format
requiring text parsing to extract any components. The parts would have
to be in some machine readable format, delimited by some strict
markers as parsing human readable text is going to be way too tedious
and error prone.

Also, since the only identifying characteristic of a syslog message is
the timestamp, aside from the line number it happens to be in the
certain file, it is difficult to determine which events should still
be sent to the other side.

>> Event format
>> ------------
>>
>> Well, obviously the event needs a timestamp. In addition to that, I
>> think it would be good to enumerate the events with an id number. I'd
>> rather not trust timestamps for any functionality since a clock can be
>> wrong.
>
> Well, for most uses relative time (between events) is more important than
> absolute time.

In my case, absolute time is really important - relative time can be
used to correlate events to get absolute time - but without absolute
time the data is nearly useless to me. And relative time can be
somewhat trusted only when there are no reboots - and even when there
are no reboots, shifting the clock to the correct time will cause
problems.

>> Then the event needs to actually carry the message - but the format
>> there is still an open issue. There most likely needs to be a message
>> type in some form, but should the body be typed? A single string? Byte
>> array? Key/value mapping? I don't know yet. What I do know is that I'd
>
> Why restrict yourself here? if the process name gets dumped in the log
> (which syslog usually does) you already know how the logging events are
> formatted and you can extract all the information your want.

Programs logging to syslog can put a "process name" there - many do -
but it no way is definite. And how does knowing the process name help
me in this case? The program can have many different types of events -
and I will still have to come up with a textual representation of
those events, with data such as coordinates, phone number, message
contents and everything. To pass those through syslog formats, I will
have to be really careful of not including newlines and being able to
extract the fields later.

> Besides, syslog has classification and severity markers, so you
> already have basic filtering capabilities on the device,
> e.g. sending some logging data ASAP over GPRS, and some logging data
> when a fatter connection is available.

Syslog has a facility and severity, both which are simple integers,
encoded together between <> at the start of the message. Nothing else
is fixed. And the timestamp cannot be specified - it needs to be
implied as current time when the log message is received. These
filtering capabilities are hugely inadequate for what I wish to do.

So, if I wanted to use syslog, I would *still* have to do:

* Custom syslog daemon - the standard syslog daemons lack
  configurability in filtering and log messages in a text
  format. Also, timestamps are in local timezone, which creates
  problems when timezone changes.

* Custom daemon for delivering the syslog messages remotely. The
  remote syslog capabilities of existing daemons are really crude -
  and do not handle properly collecting of messages until they can be
  sent to a remote host.

* My message representation would be constrained to a single line of
  text, no newlines and hopefully no special characters either. Now
  this is restrictive for encoding a myriad of data fields. I can
  ofcourse encode the whole message in base64, or something, but that
  gets really ugly.

What using syslog would really buy me is that messages could be logged
with a simple 'syslog' call, instead of a custom library. But that is
already the most trivial part of the problem.

Perhaps I was wrong to have the subject as 'comprehensive logging' -
since I don't want log messages as such. What I want is remote
collation of events and status data - such as position from GPS and
power management events.

-- Naked