Tinderbox

The concept of a tinderbox originated with the Mozilla project, where it is extensively used to continually monitor the state of the development tree on many platforms simultaneously.

It consists of a server and one or more clients. The server maintains the uploaded logs of builds and generates from a database the results of those builds on a constant basis, so a developer can see almost immediately if they have broken a build on some platform by a check-in. It is worth looking at the Mozilla tinderbox and understand all it can do. It may motivate you to help complete setting up tinderbox on freedesktop.org.

The tinderbox allows a developer to quickly:

• see if checked in changes break a build on some platform.
• examine the log of the build for errors; logs are uploaded to the server about every 30 seconds during a build.
• attach explanitory comments to a build.

If the tree did not build on a client, you'll see by the red of its box for a build, and if it's most recently completed build failed, the column will be "flaming": the tree is on fire.

Tinderbox is a set of perl scripts, and runs on Linux, Unix, !OS/X, and Microsoft Windows.

Etiquette

Breaking the build is unfriendly. And the tinderbox comes with an obligation to not break your co-collaborator's builds on other platforms.

Please perform significant check-ins early in the day whenever possible, and watch the tinderbox until you have "green" on all major platforms. It is unfriendly to do check-ins and leave town, or leave the source tree in an unbuildable state for any longer than absolutely necessary. Reverting changes that break builds is good practice if you can't get the problem straightened out in a timely fashion.

Freedesktop.org tinderboxes

New tinderboxes are easy to establish via the admin interface and tinderbox.anholt.net is the current tinderbox being used by the monolithic Xorg distribution testing. The trees currently supported are

• xorg - the Xorg modular code base
• XMonolithic - the Xorg monolithic code base
• cairo - the cairo cairo vector graphics library
• liboil - the liboil liboil optimized inner loops library.
• kaffe - the kaffe Java VM
• psas - Portland State Aerospace Society's open-source rocket flight computer

Let EricAnholt know if you think a tinderbox would be useful for your project.

CVS access to the tinderclient

You can either use your freedesktop.org account and SSH or use anonymous CVS with:

$ cvs -d :pserver:anoncvs@cvs.freedesktop.org:/cvs/sysadmin login
CVS password: <hit return>
$ cvs -d :pserver:anoncvs@cvs.freedesktop.org:/cvs/sysadmin co tinderclient

Browse CVS with View CVS.

Required software: xorg

• FreeBSD:
  • p5-libwww
  • ccache
  • freetype2
  • fontconfig
  • pkgconfig
  • gnu-automake
  • gnu-libtool
  • libpng
• debian:
  • build-essential
  • bison
  • flex
  • zlib1g-dev
  • libpam0g-dev
  • libfreetype6-dev
  • libpng12-dev
  • automake1.8
  • autoconf
  • pkg-config
  • libtool
  • ccache
  • libwww-perl
  • libfontconfig1-dev
  • cvs
  • netkit-rsh (for xsm)

Required software: XMonolithic

• FreeBSD:
  • p5-libwww
  • ccache (recommended)
• (debian:)
  • build-essential
  • bison
  • flex
  • zlib1g-dev
  • libpam0g-dev
  • libexpat-dev
  • libfreetype6-dev
  • libpng12-dev

Required software: cairo

• FreeBSD:
  • p5-libwww
  • ccache
  • fontconfig
  • freetype2
  • pkgconfig
  • gnu-automake
  • gnu-libtool
  • libpng
  • glitz (recommended)
  • librsvg (recommended)
  • poppler (recommended)
  • svg2png (recommended)
  • ghostscript-gnu (recommended)

Required software: liboil

• FreeBSD:
  • p5-libwww
  • ccache
  • pkgconfig
  • gnu-automake
  • gnu-libtool
  • gtk-doc

Required software: kaffe

• FreeBSD:
  • p5-libwww
  • gmake
  • libart2
  • esound
  • zip
  • libgmp
  • gtk20
  • jikes
  • ccache (recommended)

Required software: psas

• FreeBSD:
  • gmake
  • ccache (recommended)

Tinderclient Installation

Installing a client is very easy.

• find some machine you don't mind keeping busy with sufficient disk space.
• download the current tinderclient scripts from CVS. I make no guarantees about whether CVS is necessarily sane, though if you are going to work on the scripts themselves, you should work from the CVS version of the scripts.
• If using kaffe, comment the XMonolithic command line in run-client.sh and uncomment the kaffe one.
• choose a short machine id for the build (so you don't use too much column space) which is useful to identify the machine/OS version you are running it on. Please use the format name-architecture (for example, anholt-i386 --EricAnholt).
• run "run-client.sh machine-id >& /dev/null". By default, the tinderclient tee's its output to a tinderclient.log file and standard output, so once you are up and running, you probably don't want to have it blathering on the screen constantly.
• If you'd like to have your build use ccache or jikes if you have them installed, get EricAnholt to fix up your machine configuration. ccache cuts build times in approximately half. jikes cuts kaffe builds down even more.
• add an option to do a lndir'ed tree, to shorten the log of all the synthesized files.
• if installing on OS X/Darwin you will need to create a link from gnumake to gmake 'ln -s /usr/bin/gnumake /usr/bin/gmake'
• if tindering the xorg modular tree, you must copy xorg-macros.m4 from util/macros/ to /usr/share/aclocal, or your builds will all fail really really miserably.

The script will create a directory of the tree's name in whatever directory you are in when you run "run-client", and do a CVS checkout and build of the module specified in the tinderbox itself in a subdirectory of that. Once a day, it is supposed to do a "clobber" build, where the entire directory is deleted and the files checked out fresh, and performs a "make World". Once the initial build is complete, the script we use (confusingly also called !XMonolithic) will do just do a cvs update and another make, rather than a "make -k World".

Since CVS is so broken, the xorg and probably XMonolith trees now always clobber, and never cvs up. You probably want to keep a mirror: start keeping one with rsync, and bug EricAnholt to change your config; especially if you're running multiple clients.

One way to run this is to place them in a detached screen:

 # screen -d -m ./run-client.sh somebody-i386

You can then review and re-attach to them as follows:

# screen -list
There is a screen on:
20464.pts-5.cl012                (Detached)
1 Socket in /var/run/screen/S-root.

# screen -r 20464.pts-5.cl012

Please update your tinderclient scripts regularly when there are changes.

If the build succeeds (it did not detect any lines with "*" in them in the logs), then you get a "green" on that client's timeline.

The To Do list

There are a number of features not yet implemented in freedesktop's tinderbox:

• logins
• Regression tests of XMonolithic
• by use of bonsai (not yet operational on freedesktop.org) you can quickly see what changes have recently been checked in.
• the blame column lets you quickly identify who likely broke the build by its identification of who recently checked in code.
• be used to do binary builds that may be uploaded for distribution.
• patches can be applied for testing.
• other tinderboxes also need establishment: e.g. for the modular build.
• the tinderbox client script can (potentially) be automatically updated to clients.

People interested in helping set up all of this would be greatly appreciated; I've filed bugs in bugzilla against tinderbox on these topics to keep track of them. If you want to work on one of them, assign the bug to yourself so we know who is working on them.

Tinderbox version.

This version of tinderbox is based on the tinderbox3 work that Cliff White and Christian Reis have been doing on behalf of OSDL, the Open Source Development labs extending the orginal tinderbox3 work of John Kaiser. Mozilla is still running the original version of tinderbox, and tinderbox 2 is confusingly in Mozilla's CVS, with the most recent documents all about tinderbox 2. It appears Mozilla may eventually convert to tinderbox3, completely skipping tinderbox 2.

Our great thanks for all the help Cliff and Christian have provided and the funding that OSDL provided to them for working on Tinderbox 3.

John Dennis was very helpful getting the client XMonolithic.pm client into better shape.