State of FreeCalypso

[Spacefalcon the Outlaw]

Hello community,

This periodic post is a summary of the goals of the FreeCalypso family
of projects and the high-level status toward their achievement.

Goals
=====

The overall end goals of the project are, in no particular order:

1. Produce a standalone realization of Openmoko's modem.  I have had
   occasion to work with various GSM modems and phones acting as modems
   (presenting an AT command interface) since A.D. 2000, and the modem
   in the Freerunner is by far the nicest I've ever touched.  TI's
   implementation of the GSM specs is the richest in terms of
   functionality (contrast with the lack of CSD support in most 3G+
   USB "stick" modems), and thanks to the Leonardo semi-src find, we
   now have full visibility into its inner workings.

   But it's a shame that this awesome GSM+GPRS modem is currently
   tucked away in the guts of the Freerunner, inaccessible to anyone
   besides the tiny handful of active FR owners/users - and even when
   one does have a Freerunner, it is not possible to take the FR's AP
   subsystem out of the picture and use the modem directly from an
   external host; one has to go through the AP instead, severely
   limiting the ability to use this modem outside of the FR.

   Hence I would like to build a modem just like Om's, but brought out
   on a board by itself, with external connections for power and the
   two UARTs.  And throw in a quadband RFFE and a higher capacity
   flash+pSRAM chip while at it.

2. Produce a practically usable phone that runs practically free
   firmware, i.e., fw whose source every user is empowered to study
   and improve or otherwise modify.  Note the emphasis on practical
   usability.  I hear from FR owners that the practical usability of
   the FR as a phone is rather poor, and because there is absolutely
   nothing wrong with the modem (hw or fw), the defects in usability
   must be the result of some flaw(s) in the AP subsystem - a
   subsystem which from my PoV is nothing but unwanted complexity.
   And I would never be able to use my FR as a personal phone because
   it would require running something like QtMoko, and that stuff is
   far too complex for my old peasant mind.  Free software which is
   far too complex for me to understand and work with comfortably is
   little different from proprietary sw from the purely practical
   standpoint - it's a impenetrable black box in practical terms.

   Therefore, the only way for me to have a practically usable phone
   that runs practically free firmware is to produce a non-smart phone,
   a plain phone with no AP subsystem.  The long-term solution is to
   build my own handset hardware, but in the short term it would be OK
   to use not-quite-fitting but already existing hardware like Pirelli
   and Motorola phones.

3. Produce a FreeCalypso modem module that could be used in the place
   of off-the-shelf proprietary ones by free smartphone projects like
   Neo900.  I would like to buy a couple of Neo900 units for two of my
   family members, but cannot do so for as long as the product includes
   a modem module from an immoral vendor who withholds source code and
   documentation and imposes restricted boot barriers to alternative
   firmware implementations.

   To the person who emailed me off-list and asked if I could design
   my FreeCalypso modem in the form factor matching Gemalto's so it
   could be a drop-in replacement: yes, I still like that idea very
   much and would like to do it, but I'm unsure whether I can manage
   such a task by myself, so we may need to work on it together.  I
   also think that it would be easier if I prove my basic design first
   on a non-form-factor-constrained board, and then go through the
   form factor gymnastics as a second step.

So the above 3 are the overall goals of the FreeCalypso family of
projects.  Out of those, goal 2 (practically usable non-smart phone
running free fw) has been my main focus because it is the one that
would improve my own quality of life: I am sick and tired of dealing
with Pirelli's proprietary fw (I use a Pirelli DP-L10 as my personal
daily phone, running its original proprietary fw as nothing better
exists yet - better as in more free *and* practically usable), and I
really, really, really want to replace it with my own free firmware.

Firmware subproject
===================

The firmware subproject of FreeCalypso is leading up toward an
intermediate goal which is not listed among the end goals above:
producing a firmware version that can run on the GTA01/02 modem and
function no worse than the Windows-built ones (either official mokoN
or my own leo2moko), but builds from full source (no blobs) with gcc
instead of TI's proprietary compiler, under Unix/Linux instead of
Windows/Wine.  This task involves an absolutely arduous amount of work
because:

* There is no corresponding source for the GSM L1 and G23M protocol
  stack components which came as binary libs in the TCS211 version,
  hence these major pieces of the firmware (the main ones, really)
  have to be lifted from a different version.  We do have another
  version in full source form thankfully, but there is some non-
  trivial porting work to make it build and function in the desired
  configuration, matching TCS211.  On the positive side, this
  alternate source is a little newer than the binary libs version:
  20090327 instead of 20070608.  (The ACI component in the
  mostly-binary-libs version from 2007 came as source though, so we
  can compare the two ACI source versions and see how the software
  evolved - and I do see a few updates.)

* In order to build with gcc under Unix/Linux instead of TI's
  proprietary tools under Windows/Wine, the entire build system has to
  be rebuilt from scratch (excuse the pun), component by component,
  module by module, one C file at a time.

This task is so massive that it has occupied me on and off (unpaid
volunteer time is naturally limited) for a year and a half now, since
the fall of 2013 when I obtained the last missing piece needed to
start the project.  And this task is not even an end goal of the
overall project in itself: in terms of functionality, the latest
moko12 (aka leo2moko-r1) firmware from 2013-10 is already as good as
it will ever get, and while it would certainly be nice to have a
"freedom upgrade", the number of FR users is now so vanishingly small
and the practical usability of the FR as a whole (outside the modem)
is so low that spending years of effort just to produce a "freedom
upgrade" for one or two FR users is not very justifiable.

However, replacing the Windows-built firmware in the FR's modem with
the just-described fully free version is an important intermediate
step toward other goals of the project:

1. When we produce a standalone modem equivalent to Om's, a modem
   available to non-FR users and directly usable with a laptop or any
   other external host, it would be much nicer if this modem could run
   truly free fw as in full source compiled with free tools, rather
   than fw built under Wine from mostly binary libs.

2. For my practically free non-smart phone, whether I build my own hw
   or repurpose Motorola's or Pirelli's, having firmware built with
   gcc under Unix/Linux is a prerequisite.  Well, OK, the actual fw
   requirements are (1) to have the GSM protocol stack core in the
   desired configuration and (2) an easy ability to make large
   additions and modifications to the support code outside of the GSM
   PS core - but trying to satisfy these two requirements while
   staying within TI's nasty Windows-based environment would be such
   an incredible pita that I would rather get a full-source, gcc-built
   version running on the FR's modem first.  And besides, we will most
   certainly want our "dumbphone" firmware to be built from full source
   with gcc eventually anyway, so we might as well bite the bullet and
   do that work upfront.

The current status is that I now got almost all major chunks of "meat"
comprising TI's modem firmware suite reintegrated onto the FreeCalypso
firmware skeleton, and I am *almost* able to build a firmware image
with the GSM protocol stack and AT command interpreter included.  Right
now the (hopefully) last remaining stumbling block is the audio
subsystem.  The audio subsystem in the firmware of a GSM modem
subservient to an external host *ought* to be rather trivial: just
turn the vocoder on and off as needed (on during calls, off at all
other times), and route the call audio to the analog voice channel
(Iota) or the digital one (MCSI) as required.  But unfortunately for
the FC firmware bring-up effort, TI's implementation is not so simple.

TI's firmware architecture includes a rich and complex audio subsystem
that can play various beeps and ringtones, record and play back voice
memos and even recognize voice commands - all on the Calypso.  Most of
these functions are naturally needed for handsets in which the Calypso
drives the entirety of the phone (although even there some of the
features are optional frills), but it appears that TI's fw architecture
is designed to include all or most of that stuff whether it's needed
or not.

Believe it or not, all currently existing FR modem firmwares (both
mokoN and leo2moko) do include *all* of that Calypso audio subsystem:
the L1 audio code, the RiViera audio service, the audio functions
wrapper in Condat's "drivers" library and the audio-twiddling parts of
ACI - all of it.  Of course in an application like the FR's modem most
of that code is just dead weight that will never be used, but it's
there.  I was hoping that I could build a firmware version with this
entire subsystem excluded, but it now seems that such a trimming would
be no easy task.  It appears that the ACI layer always calls some audio
functions, and if the audio subsystem isn't there, the link fails with
undefined references (also known as unresolved externals).  At this
point I could proceed in two ways:

1. Dive deep into the ACI and try to understand why it needs to call
   audio functions, what purpose these calls serve and how they can be
   removed.

2. Bite the bullet, take a detour from the full GSM stack build,
   reintegrate the audio subsystem (perhaps it can even be tested
   standalone without the GSM protocol stack being there), and then
   come back to the GSM stack build with the audio subsystem included.

I am currently undecided as to which way I should go, but I'm leaning
toward option 2.  After all, running full-source, gcc-built firmware
on FR's modem is not an end goal of my project in itself, instead it's
an intermediate goal toward building firmware for non-smart phones;
the latter will most certainly need the audio subsystem, so I might as
well do that work now.

Hardware subproject
===================

The hardware articles I would like to build in the FC project are:

1. A standalone modem for use with laptops etc;
2. A Free Dumb Phone;
3. A module that could be used by free smartphone projects like Neo900.

Out of these three, (1) is clearly the simplest - or more precisely,
the simplest thing to build would be a preliminary version of (1) with
jacks for hooking up a lab bench power supply emulating a battery,
rather than USB power as I envision for the "final product" version of
gadget 1.  (Figuring out the optimal way to run the modem from USB 5V
power will require some extra research and thought.)

So while the article of most use to me personally would be (2), logic
says that it would be easier to build just a bare minimum board first,
consisting of just the core chipset, a hook-up for lab bench power, a
SIM socket, an antenna connector and headers for the UARTs, JTAG and
MCSI, rather than going directly to building all that *plus* an LCD, a
keypad, full set of analog audio circuits required in a handset, a
battery charging circuit etc.  Hence the first piece of Calypso hw I
seek to build is that bare minimum board.

Wishlist item: encryption
=========================

One feature that would be really, really nice to implement is end-to-
end encrypted voice calls, going over CSD when this service is
available, or as a lower quality fallback, over GPRS/IP when CSD is
not available.  I would really like to have this feature for my
family's own use (it would be awesome to be able to talk to my S.O.
without FBI/NSA/etc listening), and it would certainly help my Free
Dumb Phone sell. :)  But I am currently uncertain as to how and when I
might be able to implement this highly desirable feature.

If the underlying transport is CSD (which still works on T-Mobile USA,
or at least it did the last time I checked), then there is a slight
chance that it *might* be possible to implement end-to-end voice
encryption as a pure software feature without any extra hardware.  The
trick would be to insert the encryption function into the Calypso's
DSP pipeline.  The DSP part of the Calypso has most of its code in
mask ROM, but there is also a limited ability to extend or modify this
DSP code with downloadable patches.  These patches cannot replace the
entirety of the DSP's mask ROM code as the RAM into which these
patches go is smaller than the ROM with the main code, but there is
some ability to add new functionality not envisioned when the chip was
made with its mask ROM code.  I wonder if it might be possible to
sandwich end-to-end encryption in between the DSP's standard functions
of radio burst demodulation below (along with the pointless A5/1
encryption that protects nothing but is required by the network) and
the standard GSM voice codecs above.

However, implementing end-to-end voice encryption over CSD in this
manner would be very difficult, and may be impossible after all:

1. A *very* extensive reverse engineering of the (totally undocumented
   binary) DSP code would be required, unless I can find an ex-TI
   employee with a survived copy of the source for that DSP code, as
   well as a COFF file corresponding to what went into the silicon.

2. It is very uncertain whether or not the Calypso DSP will have
   enough horsepower to perform this end-to-end encryption in addition
   to the radio burst demodulation and GSM voice codec functions that
   would be required of it at the same time.

A different way to implement the end-to-end encryption feature that
would require building extra hardware into the phone but would be more
reliable in terms of not depending on anything in the Calypso DSP land
(and could also work with GPRS/IP in addition to CSD) would be to add
an extra chip (an FPGA would be the most powerful and flexible solution,
but probably too power-hungry, so a low-power DSP would probably be
more practical) to do the end-to-end voice encryption feature.  Keep
the power to this chip completely off except when an encrypted call is
in progress.  Connect this extra chip to the Calypso on one side (MCSI
or an extra UART), and cut into the voice ADC/DAC channel between
Calypso and Iota on the other side.  The firmware on the Calypso would
feed encrypted payloads to the extra chip without spending any Calypso
CPU cycles on encryption/decryption, the extra chip would do the
encryption/decryption and our chosen voice codec (may or may not be
the same as standard GSM, as it will be inside our end-to-end encrypted
pipe), and the decrypted voice samples would travel between our extra
chip and the ADC/DAC in the Iota ABB without any part of the Calypso
ever seeing them.  It would be a little messy and ugly in hardware
terms, but it would get the job done.

I imagine that quite a few people will value this feature highly enough
to tolerate a bit of extra hardware and some ugliness in the design -
so I anticipate that I *will* get to do this hack eventually.  The
question is when - the complexity involved is certainly not something
I feel like building into my first design...

What will I work on next?
=========================

Unfortunately there is only one of me, so I can only work on one task
at a time.  Sweet dreams like end-to-end voice encryption are still
too far in the future for now, so there really are only two practical
choices as to what I could work on next:

1. The next steps in the firmware subproject: reintegrate TI's audio
   code, and then come back to building a full fw image with the GSM
   protocol stack and AT command interpreter.

2. The first FC hardware design: a bare-minimum board corresponding to
   Om's GTA01/02 modem.

Ultimately it all comes down to economics: if you wish to influence
the direction of my work, you can do so by funding it.  The amount
that's been raised at the Indiegogo campaign so far will certainly
help relieve my and Shannon's current severe hardship a little bit,
but it is not enough to significantly influence the direction of the
project.  Therefore, unless the funding steps up significantly, I plan
on spending the next couple of months working on the firmware
subproject of FreeCalypso - simply because it's a little easier for me
to work on in my present circumstances.

VLR,
SF

Links:

FreeCalypso sw/fw subproject source repository:
https://bitbucket.org/falconian/freecalypso-sw

Crowdfunding campaign:
https://www.indiegogo.com/projects/free-software-cellular-baseband