GTA04 Block V4

[Werner Almesberger]

Andy Green wrote:
> This was actively rejected by folks in .tw last time this was live, but
> I expect we get another chance when it comes up again.

The brief history of this:

- we started with the observation that we always ended up with bits
  and pieces that needed discrete logic and/or board revisions, and
  that could be solved quickly and/or more elegantly if we had a
  separate MPU that runs in parallel to the main CPU or when the main
  CPU doesn't.

  In particular, PMU setup and various other power management issues
  would be in this domain. We didn't have a specific "must have"
  scenario, though, only the general observation that one will
  certainly develop over time.

- the search for a suitable MPU quickly centered on the TI MSP430
  family, which are very low-power 16 bit microcontrollers.

- since the general idea was to have the MPU ready to take care of
  problems wherever they may pop up, a fairly large (and expensive)
  chip from that family was chosen as the candidate.

  It seems that we didn't communicate this concept properly, so the
  MPU was considered by many as something that would also eat lots
  of space and be really expensive in the final product.

- in parallel, we also considered ways for simplifying and improving
  our "unbrickability" concept. In GTA01, the only unbrickability
  feature was the debug board. That debug board has had an enormous
  cost on terms of engineering effort and it still isn't something
  we'd consider "nice". A brief overview of the debug board story:

  - the debugv2 design and prototype went quite smoothly, but then
    we discovered stray EEPROM failures, which eventually (after
    weeks ? months ?) were tracked down to overly large capacitors
    causing the supply voltage to drop just at the wrong moment
    during reset.

  - the FPC is fragile and can break when bent

  - the FPC connectors (both) break easily. Furthermore, they're
    only rated for 10 insertions.

  - the FPC often had problems fitting into the connectors, either
    being too wide (so it wouldn't go in or only if forced) or too
    narrow (so it would go in without a fight, but you had no idea
    what kind of contacts it made).

  - our FPC source dried up and we had to find a new one, with a
    repeat performance (just worse) of the mechanical issues. You'd
    think an FPC maker could make a cable that fits into a standard
    connector in their sleep. Well, actually, perhaps that's what
    they tried to do ...

  - there are suspected signal integrity issues due to the long
    unshielded cable as well

  - it's a large piece of external circuitry, which can alter the
    way the phone behaves electronically, thus limiting its use
    when debugging hardware problems

  - we then made a debugv3 board, fixing various minor issues and
    the reset problem.

  - for GTA02, we investigated into replacing the FPC connector
    with something mechanically less dubious, but couldn't find any
    solution we could trust to be reliable, buildable, and that
    would fit into the existing case without causing new problems.

  - there is talk of trying again to find a mechanically more sound
    connector for the debug board in future devices.

  - the debug board is full of bare contacts, eagerly waiting for
    an opportunity to short or to pick up some surge. We considered
    making a case in the debugv3 era, but abandoned the idea.

  - we made another attempt to give the debug board a case earlier
    this year, and this actually went to the point of having a
    design proposal and a quote for manufacturing it, but this was
    cancelled for schedule and cost reasons.

  - besides all the problems with the implementation of the debug
    board, the general design is quite unsuitable for a mobile
    device. (One of the original ideas was that one could carry Neo
    and debug board around as a package. Hence the USB downstream
    ports.)

  So the bottom line is that we have very much a love-hate
  relationship with the debug board. It's undeniably useful, it
  fits squarely with all we believe in regarding openness, but the
  implementation is fraught with problems, and many of our attempts
  to control them were abysmal failures.

  To mitigate the debug board issues somewhat, we added the NOR in
  GTA02, so that one could recover from mere software problems without
  requiring a debug board. The NOR has its own set of issues, namely:

  - it's designed to be "write-once" (at the factory, with regular
    users not expected to change it, although they can with a debug
    board or a bit of soldering skills) and it contains a complex
    piece of software, namely u-boot.

    This means that there's the risk that we'll fix bugs or need new
    features (e.g., if we were to change the partitioning scheme),
    which thus cannot be deployed.

    This has actually happened already, but fortunately none of the
    issues encountered so far were catastrophic.

  - the NOR chip costs money and is hard to source

  - it loads the memory bus, limiting our top speed for accessing
    the RAM

  - perhaps worst of all, it eats PCB space

  For GTA03, we plan to drop the NOR and to basically use the whole
  NAND just for booting, along with a more flexible write protection
  mechanism. This will mitigate or solve all of the issues above,
  but it also means that we have a major architectural constraint
  and we still haven't solved the debug board problems when
  debugging.

  Since not everyone has a debug board and it's messy to attach, this
  also limits our ability to obtain detailed bug reports from users
  who wouldn't consider themselves kernel hackers.

  Given all the problems with the debug board, we came up with the
  idea of simply removing all the unnecessary parts and integrating
  it into the phone.

  This would solve almost all of the problems of the debug board and
  it would also reduce the need for having a hardware-protected
  alternate boot path for unbricking.

  This idea went through two iterations and in the end we had it down
  to just needing the FT2232 and a bit of glue logic. However, also
  this drew flak:

  - the FT2232 is fairly large, something our hardware team obviously
    strongly resents

  - we'd need another USB connector, which may confuse users, is
    hated by design purists, and may also violate one of the more
    obscure areas of USB OTG.

  - the FT2232 isn't cheap either

  So after some struggle, that idea was shot down as well.

- during discussion about the microcontroller some discomfort was
  felt about having another CPU architecture in our system, and in
  particular one that's fairly exotic to most Linux hackers. The
  suggestion was made to just add another ARM.

  Now this may sound crazy, but in recent years, small ARM-based
  systems have started to enter what was previously the domain of
  small microcontroller, such as 8051, PIC, AVR, or MSP430. There
  are still clear differences, for example in power consumption, but
  they're no longer necessarily a big issue.

- all that has been put aside for a while as more pressing issues in
  GTA02 and the work towards GTA03 surfaced

- meanwhile, we ran into new problems (regarding PMU bringup) which
  confirmed our expectation that an MPU able to act independently
  from the CPU would be something really useful to have. So there's
  hope that we'll be able to defend the MPU better when the GTA04
  discussion is opened again. 

- realizing that we had two very promising enhancements that would
  also solve recurring nightmares that are as old as Openmoko itself,
  which both were shut down for cost and space reasons, the idea came
  up to actually combine the two, thus halving the perceived penalty.

  In addition to this, it's tempting to also try to avoid complicating
  our development environment, and make everything ARM.

And this is what this ultimately is about: if we can have an MPU that
has USB device functionality, it could act as a JTAG interface and
serial console towards the CPU. Since we want to be able to also
change the firmware of the MPU, it has to be unbrickable as well,
which could be achieved by having a robust mechanism to download
new firmware over USB.

So far, we haven't found a solution that would be perfect, and there's
still the possibility of having an MPU that only helps with low-level
tasks, without getting involved with USB and such. But I'm hopeful
that we'll get there, and that we can close the book on that series of
engineering tragedies debug board and lack of MPU have already caused
us.

- Werner