Request for stable, automated build process

Sat May 3 00:12:41 CEST 2008

On Thursday 01 May 2008 18:01:38 Bobby Martin wrote:
> I sent this before to openmoko-devel and was greeted with a deafening
> silence, so I'm resending to a broader audience.  As evidenced by a recent
> post to openmoko-devel by saurabh gupta, one of the GSoC selectees, this
> really is a problem.  The longer it's put off, the more potential benefits
> (in the form of community contributors who give up rather than improve OM)
> are lost.

Well, the reason I didn't reply to the first mail is because it is touching 
way too many topics to answer all of them in depth and finding a time slot to 
do this.

I'm working on the april software update. I have my branch and it has a high 
probablity of always building. Everything we built for the 
openmoko-qtopia-x11-image (just for testing) is set to a fixed 
version/srcdate. I can rebuild the whole distribution without internet 
(because I don't have internet at home) and I have written a task for 
base.bbclass to help me to lock down the built to versions.

The obvious downside is: To upgrade certain components I have to spent close 
to a day to build, and test the result.

The good thing: I can make sure to have an always building image, which allows 
you to make phone calls, sms,...

> ---------------------------------------------------------------------------
>-----
>
> I really think OM is shortchanging themselves with the current build
> process.  As far as I can tell:
>
>     * there is no way to run a build that has a high probability of working
> all the way through

Not true for my org.openmoko.zecke.april-update branch because everything is 
locked down (I can build without internet if I have fetched all sources 
before)

>     * there is no way to identify the latest code that builds together

You know the SRCDATE, SRCREVS, tarballs you are going to use. The information 
for a whole system is massive though.

>     * there is no automated build process, nor even a stable script one can
> follow to do a complete build from source on one's local machine

Right, I have attached my Openembedded scripts and configuration I use to 
build on my hardy system. You see it is really basic/simple.

>     * it looks to me as if the code lives in at least three different kinds
> of repositories: svn, GIT, and mtn

Yes, we integrate a lot of different projects. They use tarballs on http/ftp, 
cvs, hg, p4, mtn, git, svn. There is hardly anything we can change about 
that.

>     * I'm sure that there is a way to identify the code that went into a
> particular snapshot found on the web, but I don't know what it is

The ipk files should point to the recipe used to build the package and has the 
SRCDATE/SRCREV in the same. So it is easy to find out.

>     * there is no way to identify which tasks definitely have someone
> reliable working on them, which are open issues & need attention they're
> not getting, and which have some casual developer looking at them

I agree, we don't have an bugmaster at Openmoko who could help on this. For 
the april software update I mostly track the issues myself and pester the 
people involved (oh already my 3rd task).

>
> I have done several very minor programs for my neo, which I would have much
> preferred to integrate into the real build process.  However, when I've
> tried the Mokomakefile process, it will run for hours and then die in some
> obscure way.  I spent a few hours over the course of several days debugging
> that process on my Ubuntu machine at home, before I realized there were
> version issues with the assembler that I didn't see any way to work around.

Three things:
	- Finding the balance between always stable and cutting edge, specially while 
in development is pretty hard. I have opted for locking everything down 
because I lack internet at home which is helpful for what you want as well.

	- You can ask some one of the distro team to add your software to the build 
process, we are happy to do so.

	- Most of us use debian/ubuntu systems so should be able to help you in 
getting it built.

> I just don't have the energy to spend a dozen hours debugging a process
> that will enable me to spend a few dozen hours doing development.  I would
> think that many other potential contributors fall in the same boat.
>
> I think that a few procedures could dramatically increase the development
> community's involvement in OM, resulting in faster bugfixes, more features
> added sooner, and greater visibility into the project status:

Yes, I agree. Ading poky's SDK management will help to easily build your 
software. If you use autotools in a proper way it should be easy to package 
them. I think our distro team is interested in helping here. The alternative 
would be to create a feed with user submitted packages but it has the 
downside of not easily recreating the package (e.g. due changing ABI, 
compilers, c libraries..., we have been through this with the Sharp Zaurus).

> There should also be a test harness for developers to easily insert tests
> for their code, and for which every bug (except those that require hardware
> to test) must have a test inserted for the bug to be considered fixed.  The
> test harness should likewise be runnable trivially, with one simple command
> (e.g. 'make test')

Yes, part of the upstream projects:
	E.g., WebKit has a 300 MB Layout regression test suite, my openmoko rss 
reader has a couple of tests as well. The thing is properly packaging the 
tests and executing them on the device ("make test" will not work...packaging 
the build dir will not work as well). The best thing I have seen so far is 
bazaars selftest (which fails on ubuntu systems).

>
>
> *Continuous integration* (CI)
> Someone needs to set up a server that on a periodic basis (one or more
> times per day) pulls the latest build process script, labels the code

what is "the code"? Again we integrate hundreds of different projects. How do 
you label a tarball? Feel free to join my multi-year project Utopia and work 
on my whole system regression testing and data mining system ideas.

> If the process succeeds, then the code labeled CI_CANDIDATE would be
> relabeled something like GOOD_BUILD_2008_04_23 and the CI_CANDIDATE label
> removed.  Further, the test harness should be run, and if it passes a label
> like TESTS_PASS_2008_04_23 should be added.

Yes, we can use mtn certs to attach to a revision to say if that built or 
didnt.

> *Task tracking*
> Maybe this is supposed to happen in bugzilla now, but it seems to me that
> we need the kind of bug tracking I was discussing above - a full list of
> tasks, priority (something as simple as High, Medium, Low would work fine),
> and a way to see if it is *really* being worked or not.  Having a project
> manager who is an OM employee for every task would be helpful, since we
> could then expect to get an answer if we email a query about the status.

Yup, we need a bug master. We need one inside Openmoko as well to help us to 
focus on what needs to be done for a product. For the April software update I 
kind of assume this position (4th role, but it is sadly mostly on paper and 
poking the devs directly).

> *Documentation*
> If the above were implemented, that documentation could be as simple as
> telling people:
>     * the script location and how to configure what labels are used for
> each subsystem
>     * the location to download the VM or chroot tarball if necessary
>     * the meaning of the continuous integration labels, and which ones to
> look for
>     * perhaps the place to look for CI build statuses (most recent
> successful build, etc)
>     * the location of the task tracker and the important fields (status,
> priority, assignee(s), project manager)
>
> Some or possibly even all of this may exist, but if so I couldn't find the
> documentation.  In particular, CI and a test harness make a HUGE difference
> in the likelihood of success of a project, and the ease of contributing to
> the project.

At the beginning I ran a tinderbox with build results and build failures, 
automatically filing bug reports. The reality is, no user looked at the build 
status before starting his build. What I learned from a friend called Laibsch 
is: There are so many cool things you could automate and write scripts for 
but there are other things non-technical solutions that help so much more. :)

>
> I realize that this would cost someone several days of effort, but the
> payback wouldn't just be for the community, the developers at OM would
> benefit, and the managers and executives would have much greater visibility
> into the status of the project.

What are your issues?
	a) Want to have an always building distribution.
	b) Want to be able to add your package to the distribution process
	c) Want to enable users to help fixing issues
	d) What did I miss.

My take on these:

a)
	-The qtopia-x11-image should always built. I think opkg was the last thing we 
forget to set to a certain revision.
	- You know which versions got built and can fetch the sources and rebuilt 
without internet.

b)
	- I'm against user submitted feeds as we have no control on the quality and 
we lack the possibility to rebuild packages
	- I think it is in the interest of the distro team to assist you in packaging 
your software. Maybe write distro-devel for a request of packaging?
	- With the qtopia-x11-image you should be able to easily build and package 
autotools based projects

c)
	- We need someone to summarize important issues, set the right priorities, 
collect information.
	- I think this can be done within the community.
        - The next thing is to actually get the source and fix it. I work on 
this with the merging of poky's SDK capabilities and also considering debian 
source packages (work from the mamona guys at indt)

d)
	- No idea

this is my take on this... I think we have most of what you want, on some 
parts I agree with you (issue tracking), on the automation side I do not 
fully agree, there are better things to do which require less effort and have 
a bigger impact/success (e.g. a user telling which mtn revision of the OM 
metadata works...).

	z.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: source.sh
Type: application/x-shellscript
Size: 114 bytes
Desc: not available
Url : http://lists.openmoko.org/pipermail/community/attachments/20080503/5932907b/attachment.bin 
-------------- next part --------------
BBFILES = "${HOME}/source/openmoko/org.openmoko.dev/packages/*/*.bb"
MACHINE = "om-gta02"
DISTRO = "openmoko"
BB_NUMBER_THREADS = "4"
PARALLEL_MAKE = "-j4"
ENABLE_BINARY_LOCALE_GENERATION="0"

DL_DIR ="${HOME}/source/sources"