-
+INTRO
+-----
This directory (nightly/) contains a simple, automatic build-and-test
-system for Valgrind, intended to be run by cron.
-
-Note (importantly) it doesn't test the sources in the tree of which
-this directory is a part (viz, nightly/..). Instead it checks out
-a complete new tree, builds and tests that independently of the
-existing tree.
-
-To use, choose a tag, probably a machine name, and run
-
- bin/nightly /path/to/valgrind/nightly/ <tag>
-
-and supply the following two config files:
-
-- conf/<tag>.conf: this is sourced by the 'nightly' script, and can define
- any or all of the following environment variables:
-
- ABT_DETAILS: describes the machine in more detail, eg. the OS. The default
- is empty.
- ABT_CONFIGURE_OPTIONS: gives extra configure options. The default is empty.
- ABT_EVAL: if provided, it must be the name of a shell script that executes
- the shell command $1 with arguments $2 .. ${$#}. Allows to compile and
- run the Valgrind regression tests on another system than the system the
- 'nightly' script runs on. It is assumed that the remote system shares the
- local filesystem tree through e.g. NFS. It is the responsibility of the
- shell script to set the remote working directory such that it matches the
- local current directory ($PWD).
- ABT_RUN_REGTEST: if provided, it must be the name of an argumentless shell
- function (also specified in the tag.conf file) that will be used to run
- the tests. If not specified, the usual "perl tests/vg_regtest --all"
- will be used.
- ABT_JOBS: allows parallel builds -- it's passed as the argument to "make
- -j" when building Valgrind and the tests. The default is 1.
- [XXX: the .NOTPARALLEL that currently resides in Makefile.all.am foils
- this!]
-
-- conf/<tag>.sendmail: this should be a script that sends an email to the
- desired recipient (eg. the valgrind-developers list). It takes three
- command line arguments. The first is the email subject line, the second
- is the name of the file containing the email's body (showing the tests
- that failed, and the difference between now and 24 hours ago), the third
- is the name of the file containing all the diffs (which can be made into
- an attachment, for example).
+system for Valgrind, intended to be run nightly by cron or a similar
+program.
+
+
+BASIC OPERATIONS
+----------------
+When run, the system checks out two trees: the SVN trunk from 24 hours ago
+and the SVN trunk from now. ("24 hours ago" and "now" are determined when
+the script starts running, so if any commits happen while the tests are
+running they will not be tested.)
+
+If the two trees are different (i.e. there have been commits in the past 24
+hours) it builds ("make"), installs ("make install") and runs the regression
+tests ("make regtest") in both, and compares the results. Note that the
+"make install" isn't necessary in order to run the tests because the
+regression tests use the code built (with "make") within the tree, but it's
+worth doing because it tests that "make install" isn't totally broken.
+After checking both trees, it emails a summary of the results to a
+recipient. All this typically takes something like 30 minutes.
+
+If the two trees are identical, the tests are not run and no results are
+emailed. This avoids spamming people with uninteresting results emails when
+no commits have happened recently.
+
+
+SETTING UP
+----------
+To set up nightly testing for a machine, do the following.
+
+(1) Check out just this directory from the repository, eg:
+
+ svn co svn://svn.valgrind.org/valgrind/trunk/nightly $DIR
+
+ where $DIR is the name of the directory you want it to be in.
+
+ Note that this doesn't check out the whole Valgrind tree, just the
+ directory containing the nightly testing stuff. This is possible
+ because the testing script doesn't check the code in the tree it belongs
+ to; rather it checks out new trees (within $DIR) and tests them
+ independently.
+
+(2) Choose a tag that identifies the test results. This is usually the
+ machine name. We'll call it $TAG in what follows.
+
+(3) Create a configuration file $DIR/conf/$TAG.conf. It is sourced by the
+ 'nightly' script, and can define any or all of the following environment
+ variables. (In most cases, only ABT_DETAILS is needed.)
+
+ - ABT_DETAILS: describes the machine in more detail, eg. the OS. The
+ default is empty, but you should define it. An example:
+
+ export ABT_DETAILS="Ubuntu 9.04, Intel x86-64"
+
+ You could also use some invocation of 'uname' or something similar
+ to generate this string. Eg. on Ubuntu Linux this works nicely:
+
+ export ABT_DETAILS="`cat /etc/issue.net`, `uname -m`"
+
+ And on Mac OS X this works nicely:
+
+ export ABT_DETAILS=`uname -mrs`
+
+ The advantage of doing it like this is that if you update the OS on
+ the test machine you won't have to update ABT_DETAILS manually.
+
+ - ABT_CONFIGURE_OPTIONS: gives extra configure options. The default is
+ empty.
+
+ - ABT_EVAL: if provided, it must be the name of a shell script that
+ executes the shell command $1 with arguments $2 .. ${$#}. Allows to
+ compile and run the Valgrind regression tests on another system than
+ the system the 'nightly' script runs on. It is assumed that the remote
+ system shares the local filesystem tree through e.g. NFS. It is the
+ responsibility of the shell script to set the remote working directory
+ such that it matches the local current directory ($PWD).
+
+ - ABT_RUN_REGTEST: if provided, it must be the name of an argumentless
+ shell function (also specified in the $TAG.conf file) that will be used
+ to run the tests. If not specified, the usual "make regtest" will be
+ used.
+
+ - ABT_JOBS: allows parallel builds -- it's passed as the argument to
+ "make -j" when building Valgrind and the tests. The default is 1.
+ [XXX: the .NOTPARALLEL that currently resides in Makefile.all.am foils
+ this!]
+
+ Note that the appropriate syntax to use in this file will depend on the
+ shell from which the $DIR/bin/nightly script is run (which in turn may
+ depend on what shell is used by cron or any similar program).
+
+(4) Create a mailer script $DIR/conf/$TAG.sendmail. It must be executable.
+ It's used to send email results to the desired recipient (e.g.
+ valgrind-developers@lists.sourceforge.net) It must handle three command
+ line arguments.
+
+ - The first argument is the email subject line. It contains
+ $ABT_DETAILS plus some other stuff.
+
+ - The second argument is the name of the file containing the email's
+ body (which shows the tests that failed, and the differences between now
+ and 24 hours ago).
+
+ - The third is the name of the file containing all the diffs from
+ failing tests. Depending on the test results you get, you could
+ inline this file into the email body, or attach it, or compress and
+ attach it, or even omit it. The right choice depends on how many
+ failures you typically get -- if you get few failures, inlining the
+ results make them easier to read; if you get many failures,
+ compressing might be a good idea to minimise the size of the emails.
+
+ The best way to do this depends on how mail is set up on your machine.
+ You might be able to use /usr/bin/mail, or you might need something more
+ elaborate like using Mutt to send mail via an external account.
+
+(5) To run the tests, execute:
+
+ $DIR/bin/nightly $DIR $TAG
+
+ You probably want to put this command into a cron file or equivalent
+ so it is run regularly (preferably every night).
+
+
+OUTPUT FILES
+------------
+If the tests are run, the following files are produced:
+
+- $DIR/old.verbose and $DIR/new.verbose contain full output of the whole
+ process for each of the two trees.
+
+- $DIR/old.short and $DIR/new.short contain summary output of the process
+ for each of the two trees. The diff between these two files goes in
+ $DIR/diff.short.
+
+- $DIR/final contains the overall summary, constructed from $DIR/old.short,
+ $DIR/new.short, $DIR/diff.short and some other bits and pieces. (The name
+ of this file is what's passed as the second argument to
+ $DIR/conf/$TAG.sendmail.)
+
+- $DIR/diffs holds the diffs from all the failing tests in the newer tree,
+ concatenated together; the diff from each failure is truncated at 100
+ lines to minimise possible size blow-outs. (The name of this file is
+ what's passed as the third argument to $DIR/conf/$TAG.sendmail.)
+
+- $DIR/sendmail.log contains the output (stdout and stderr) from
+ $DIR/conf/$TAG.sendmail goes in $DIR/sendmail.log.
+
+- $DIR/valgrind/ contains the tested tree (and $DIR/valgrind/Inst/ contains
+ the installed code). It is used for both trees and so after the tests
+ have completed it will contain the newer tree.
+
+If the tests aren't run, the following file is produced:
+
+- $DIR/unchanged.log is created only if no tests were run because the two
+ trees were identical. It will contain a short explanatory message.
+
+Each time the tests are run, all files from previous runs are deleted.
+
+
+TROUBLESHOOTING
+---------------
+If something goes wrong, looking at the output files can be useful. For
+example, if no email was sent but you expected one, check sendmail.log to
+see if the mailer script had a problem. Or check if unchanged.log exists.
+
+Occasionally the SVN server isn't available when the tests runs, for either
+or both trees. When this happens the email will be sent but it won't be
+very informative. Usually it's just a temporary server problem and it'll
+run fine the next time without you having to do anything.
+
+Note that the test suite is imperfect:
+- There are very few machines where all tests pass; that's why the old/new
+ diff is produced. Some of the tests may not be as portable as intended.
+- Some tests are non-deterministic, and so may pass one day and fail the
+ next.
+
+Improving the test suite to avoid these problems is a long-term goal but it
+isn't easy.
+
+
+MAINTENANCE
+-----------
+The scripts in the nightly/ directory occasionally get updated. If that
+happens, you can just "svn update" within $DIR to get the updated versions,
+which will then be used the next time the tests run. (It's possible that
+the scripts will be changed in a way that requires changes to the files in
+$DIR/conf/, but we try to avoid this.)
+
+If you want such updates to happen automatically, you could write a script
+that does all the steps in SETTING UP above, and instead run that script
+from cron.
+
+
projects / thirdparty / valgrind.git / commitdiff
