Merging openmoko into OE

[Harald Welte]

On Thu, Mar 01, 2007 at 03:58:11PM +0100, Koen Kooi wrote:
> Hi,
> 
> A quick inspection shows that openmoko userspace in OE contains more
> recipes and bugfixes than the userspace in trunk/oe. 

That can very well be, but is a direct result due to mickey being
unavailable due to thesis, fosdem, illness and now holidays.

We don't have anyone else as involved and experienced in OE in our team
internally so far.  I'm also not sure whether we would need that, given
that he'll be back soon...

So I would rather consider this temporary than a desired situation.

Also, given the current status of the UI apps (as I perceive it),
they're not generally useful yet either, so building them is not a top
priority yet.

> Therefore I want to propose to start deprecating trunk/oe next week.

I'm all for it, but that's really one of mickeys key areas.

> What needs to be done:
> 
> * update openocd + deps into OE
> * update uboot in OE
> * update 2.6.20 in OE
> * merge sourcepkg.bbclass

are there any issues with this? It's unlikely that our bb recipies
change a lot, so if you can just use ours, and fix them before mergin
them to upstream OE, that should do the trick, right?

> * delete recipes from svn

I'm happy to do this if we don't get any more build failures than we
have with our current OE snapshot + svn overlay.  Mickey was very
careful about selecting revisions that work, and to not track OE
mainline due to the amount of flux in it.  I know the current situation
is messy.  But it needs somebody who understands both sides to resolve
this.  
 
> That only leaves the decision wether to lock down to the working revision for p0 and p1.
> My gut feeling is that p0 doesn't need a locked down revision, since a lot of bugs will
> get uncovered and fixed. For p1 I'm not sure what to advice, but p2 will need a snapshot
> or branch for sure. Keep in mind that maintaining a branch is almost a full-time job if
> you want to do it right.

we don't need locked down revision.  Definitely not for phase 0, and
also not for phase1, at least not  until we're getting seriously close
to phase2.  Any act of trying to introduce a "stable" version before
phase2 will just call for more work and trouble.  We need to focus on
development.  

In debian terms, I don't think we will have anything than "unstable" for
the nest coming months.

> What are the plans, and how can the community help with carrying those out?

Our current agenda is more focussed on "getting things working at all"
rather than "doing everything correct in the first run".  I wish it was
different, but we're really under a lot of pressure here.

As far as the buildsystem is addressed, there are some distinct features
which we're lacking currently.  I have hired an experienced python
hacker to add those things properly to bitbake and then talk to you guys
about how this code can be merged.

Our primary requirements, with regard to the build system, are:

1) do proper and reliable caching, dependency resolving, and decisions
   on when to build what, while staying in-sync with upstream
   repositories.  We're working on somce code that adds methods to the
   fetchers (so far: only svn) to determine the 'last changed date' of
   the upstream repo.  Then we create a string as concatentation of all
   source uri repositories, and use that as PR.  This gets rid of the
   bogus timestamps and neverending rebuilds of the same source code.

   Also, the tarballs in 'sources' (which get mirrored to
   downloads.openmoko.org/sources) will then contain e.g. the svn
   revision number, not the checkout date.

   This whole logic will be optional, and only be active for
   SRCDATE="now" packages.

   Using this method we can e.g. only rebuild the uboot package if
   either upstream git or our quilt patchset in svn have changed.

   I know this might conflict with where bitbake is heading, or what is
   currently in 1.7.x.  We will investigate this later, once we have
   something that fits our needs first.  I know this is very
   un-free-software-like, but we really are short of months of time [and
   staff] and have to compromise somewhere.

2) run automatized builds on one of our buildhosts, continuously,
   sending reports about build failures directly into bugzilla.

3) find a cleaner way for the deploy/images directory.  There need to be
   per-release symlinks so that an automatized flashing tool can always
   assume that the current kernel for one release/snapshot/branch has a
   fixed name (can be symlinks). Also, the statically linked native
   openocd, dfu-util and sjf2410 binaries need to go to a different
   directory.

4) find a way how to correctly clean up packages after a build.
   Something like: If I rebuild a kernel uImage, then only the most current
   one is supposed to be in 'images', older ones get moved to 'attic'.

5) find a way how to completely clean up a package, including all
   related download cache, bitbake rule cache, .ipk's and/or 'images'
   that it created

6) find how how we can automatically create a full 'snapshot' from a
   current build-directory, by just using the latest version of every
   kerne/uboot image and .ipk

7) related: reimplement ipkg backend to use some indexed database. It's
   just too slow

8) think about signing binary packages and feeds plus signature
   verification on the device

As for community help: I'm open for about anything.  We just have way
too many issues to resolve than to engage in lengthy discussion
processes at the moment.  So we might run into the wrong direction for
some time, getting back on track once we've don our homework :(

-- 
- Harald Welte <laforge at openmoko.org>          	        http://openmoko.org/
============================================================================
Software for the world's first truly open Free Software mobile phone