Tree build system and OM2007.2

[Rodolphe Ortalo]

Le mercredi 01 août 2007 à 22:00 +0200, Michael 'Mickey' Lauer a écrit :
> Hi Rodolphe,
> 
> > sorry to scratch an old itch but, shouldn't we take the opportunity of
> > opening OM2007.2 to think about the build system (esp. the qmake vs.
> > autotools thingy)? It's just that it makes me nervous for the project to
> > see 2 build systems available simultaneously... (Maybe I am just too
> > nervous.
> 
> Didn't you see me removing all .pro files? :)
> Well -- basically qmake support is gone. It was there mainly for my
> personal preference (I confess, I love qmake and prefer it to autotools).
> 
> > There is one thing in particular that I'd like to see in SVN: the
> > ability to re-conf the whole tree automatically after update and build
> > it without going (too much) in the individual components updates.
> > If possible, I'd like that starting at the "target/" subtree (or even
> > higher!).
> > I don't think such a system has a chance of being adequately maintained
> > if the main developers do not elect a "primary" build system.
> 
> See above, it's done. autotools is it.

Well, so we'll have to support the shortcomings of autotools then
instead of those from qmake. (Hmmm, I am nervous _and_ pessimistic this
evening... ;-)

> > As an example, personnally, I kept on maintaining such a thing for
> > myself alone for a few months (for OM-2007/ only), as I really find it
> > useful for tracking the tree. [1]
> 
> I agree. This is very useful!
> 
> > Of course OE/Mokomakefile etc. are available to do that too, but I do
> > most of the coding attempts on the PC itself in xoo as that's much
> > faster natively on the PC to remake the tree or install things.
> 
> > I do not want to turn this into an autotools vs qmake debate but I
> > wonder if we can avoid to do some comparison now that both have been
> > tried for a while. Mickey, especially, do you think qmake can be up to
> > the task for being the primary build system for everything?
> 
> Of course it could. Qmake can do everything autotools can do :D

In fact, the primary advantage of the autotools are probably their wide
usage. (Most of the advantages are a consequence of this wide audience.)

When I tried your qmake (even unsuccessfully in the long run) I was
especially surprised by the speed of execution and the small size
of .pro files. My main problems with it were due to install directories
issue for libraries (especially when building in separate directories),
possibly because I did not know how to give the correct parameters (and
did not know much in python).

Well, whatever the technical feelings, IMHO it would be useful to try to
compare the 2 systems. I cannot really do this as I only know the
autotools, and not qmake.
But I can speak of the former and list right now the difficulties I have
encountered while using the autotools myself with a recursive configure
inside OM-2007/. (I don't especially want to change the decision, but at
least to detail some impacts of such a choice.)

See the end of this mail for (too many) details.

> Anyways, since you already worked on that, you'd be welcome to add
> some top-level directory (src/target/OM-2007.2) autotools stuff that
> could build everything in one run. Would you like to do that?

Of course! But that's not difficult you know.
The top-level configure.ac is probably simpler than all the others...
Furthermore, I suspect someone with a wider access (like you) may have
to look into this too to get it going smoothly when details in a package
deeper in the source tree raise a problem.

At least, I can already show you how I did it inside OM-2007/; building
a patch against svn is tricky but make dist runs, so you can have a look
easily.
As an example, see the build system inside:
http://rodolphe.ortalo.free.fr/openmoko-blob-0.1.tar.gz
I am not sure that this builds due to bugzilla #543 but I suppose you
are more interested in reading than running.

You have already decided to limit yourself to the OM-2007.2/ tree? Well,
that's probably reasonable but then everyone will need to build libgsmd
before?

Anyway, thanks for your fast response (and for you attention if you read
me up to the end ;-)

Rodolphe


======== Main suffurings with the autotools ======

1) A recursive configure is not really fully "recursive" in practice.
That's probably possible but I don't think it would be wise to do such a
recursion over more than 2 levels... :-)
So, IMHO, you will have to decide now if a configure.ac should be put in OM-2007.2/ or in target/... Well, we can try a three-level recursion (target/, OM-2007.2/ and then the real components) but I never tried.

2) A recursive configure does not play well with "autogen.sh" scripts. Those should probably be eliminated. Devs will have to do the configure and build steps separately. (I think that this is more a feature than a problem - but many people seem to disagree with me apparently as autogen.sh scripts proliferate...)
This is a problem too with "gnome-autogen.sh" (I had to use the
"NOCONFIGURE=1 ... gnome-autogen.sh" variant.)

3) Offering the possibility to build against an uninstalled version of libraries is more or less mandatory. (If this is not possible, it is not possible to run a "distcheck" at the top level for example. So you lose a big advantage.)
(In practice, when using pkg-config it only means to define a separate -uninstalled.pc.in file so that's not very long - but reading the documentation to find how to write that pc file is much longer...)
But then, it may have an impact on the organization of the source tree in those libs (for the header files). Note one "bug" I filed a while ago related to this: #543 . AFAIK it seems to mean that you cannot put your headers files anywhere in the library source tree.
This is linked to 1): if we put a top level configure in target/ then all sublibs will have to support this: gsm/ agpsd/ etc. (with a potential small reorganization to do in those sub trees).

4) I have yet to understand how to manage (correctly) the content of the "artwork/" subdir. to install it in a proper place.

5) When new _directories_ get added to, e.g. an application, one needs to "autoreconf" in that dir, but also autoreconf at the top level and re-run the full configure. That's boring, especially due to 9...

6) The "autotools" is not one tool, it's: configure, autoheader, automake, libtool, pkg-config - so that's a lot of documentation and few have the time to read it while coding. (I was obliged to read it fully a long time ago and that probably explains my patience with the autotools.)

7) If one of the sub-directories build system fails, the whole thing fails. (Well, that could probably be solved with magic in the top-level configure, but I'd rather see some magic done in the phone than in the build system... :-)

8) The most tricky issues I had to deal with are related to things like:
gettextize vs glib-gettextize or gnome-autogen.sh
Don't ask for details, I have forgotten everything as soon as I found the workaround.

9) Finally, note the obvious "speed problem":

On an already fully built tree, an "empty" make iteration is pretty good (and building is pretty efficient too):

ortalo at hurricane:/space/openmoko-svn/svn.openmoko.org/trunk/src/target/OM-2007/build$ time nice make >/dev/null 2>&1
real    0m5.757s
user    0m4.348s
sys     0m1.240s

But:

$ time ../configure --prefix=/space/openmoko-svn/testinstall --config-cache --enable-maintainer-mode
[...]
real    2m35.405s
user    0m57.680s
sys     0m54.819s

If you forget to say "--config-cache" (ie: to use a centralized config checks cache for all the sub-configures), the time goes up to:
real    4m52.452s
user    2m0.496s
sys     2m5.000s

NB: The above figures are with an Athlon 2GHz, and 26 sub-configures listed in the AC_CONFIG_SUBDIRS directive of the top-level configure.ac). That's _only_ to configure the build tree of course, no compilation at all.
There you will regret qmake no?

Note that, when doing a "make distcheck" at the top level (pretty convenient to automate a full test rebuild/packaging), --config-cache is NOT used... :-)


Now some pros (IMO):
* As soon as "make distcheck" runs successully at the top-level, you easily spot a _lot_ of subsequent errors (like forgetting a header file or something like that). Running make distcheck before doing a svn commit becomes a good habit very fast (1 out of 2 it spots something).
* It will be possible to distribute easily a big .tar.gz that is nearly guaranteed to build (nice to attract new devs?).
* What I really like in the autotools is the flexibility to build in seperate directorie(s) and install in separate directorie(s) too.
* At the end, the programs still seem runnable. ;-))