About the tests
---------------
- There are two kinds of tests in the Automake testsuite (both implemented
- as shell scripts). The scripts with the '.sh' suffix are "simple"
- tests, their outcome completely determined by their exit status. Those
- with the '.tap' suffix use the TAP protocol.
+ There are two kinds of tests in the Automake-NG testsuite (both
+ implemented as shell scripts). The scripts with the '.sh' suffix
+ are "simple" tests, their outcome completely determined by their
+ exit status. Those with the '.tap' suffix use the TAP protocol.
- If you want to run a test by hand, you can do so directly if it is
- a simple test:
- ./t/nogzip.sh
+ If you want to run a test by hand, you should be able to do so using the
+ 'runtest' script provided in the Automake distribution:
- (it will be verbose by default), while if it is a TAP test you can pass
- it to your preferred TAP runner, as in e.g.:
+ ./runtest t/nogzip.sh
+ ./runtest t/add-missing.tap
- prove --verbose --merge ./t/add-missing.tap
-
- The tests can also be run directly in a VPATH build, as with:
-
- /path/to/srcdir/t/nogzip.sh
- prove --verbose --merge /path/to/srcdir/t/add-missing.tap
+ This will run the test using the correct shell, and should also work in
+ VPATH builds. Note that, to run the TAP tests this way, you'll need to
+ have the prove(1) utility available in $PATH.
Supported shells
----------------
- By default, the tests are run by the $SHELL detected at configure
- time. They also take care to re-execute themselves with that shell,
- unless told not to. So, to run the tests with a different shell, say
- '/path/to/another/sh', the user must use:
-
- AM_TESTS_REEXEC=no /path/to/another/sh ./t/foo.sh
- AM_TESTS_REEXEC=no prove -v -e /path/to/another/sh ./t/bar.tap
+ By default, the tests are run by a proper shell detected at configure
+ time. Here is how you can run the tests with a different shell, say
+ '/bin/my-sh':
- to run a test directly, and:
+ # Running through the makefile test driver.
- make check AM_TEST_RUNNER_SHELL=/bin/my-sh (GNU make)
- AM_TEST_RUNNER_SHELL=/bin/my-sh make -e check (non-GNU make)
++ make check AM_TEST_RUNNER_SHELL=/bin/my-sh
- make check LOG_COMPILER=/path/to/sh
+ # Run a test directly from the command line.
+ AM_TEST_RUNNER_SHELL=/bin/my-sh ./t/ax/runtest t/foo.sh
- to run the test(s) through the makefile test driver.
- The test scripts are written with portability in mind, so that they
- should run with any decent Bourne-compatible shell. However, it is
- worth nothing that older versions of Zsh (pre-4.3) exhibited several
- bugs and incompatibilities with our uses, and are thus not supported
- for running Automake's test scripts.
+ The test scripts are written with portability in mind, and should run
+ with any decent POSIX shell. However, it is worth nothing that older
+ versions of Zsh (pre-4.3) exhibited several bugs and incompatibilities
+ with our uses, and are thus not supported for running Automake's test
+ scripts.
Reporting failures
rm -f *.run
- : An empty '$(TESTS)' or '$(TEST_LOGS)' means that no test should be run.
- for var in TESTS TEST_LOGS; do
- env "$var=" $MAKE -e recheck >stdout || { cat stdout; exit 1; }
- cat stdout
- count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
- test ! -e a.run
- test ! -e b.run
- test ! -e c.run
- done
- unset var
+ : An empty '$(TESTS)' means that no test should be run.
+ $MAKE TESTS= recheck >stdout || { cat stdout; exit 1; }
+ cat stdout
+ count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
+ test ! -e a.run
+ test ! -e b.run
+ test ! -e c.run
: a.test was successful the first time, no need to re-run it.
- env TESTS=a.test $MAKE -e recheck >stdout \
- || { cat stdout; exit 1; }
+ using_gmake || $sleep # Required by BSD make.
+ $MAKE TESTS=a.test recheck >stdout || { cat stdout; exit 1; }
cat stdout
count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -e a.run
test ! -e c.run
: b.test failed, it should be re-run. And make it pass this time.
+ using_gmake || $sleep # Required by BSD make.
echo OK > b.ok
- TEST_LOGS=b.log $MAKE -e recheck >stdout \
- || { cat stdout; exit 1; }
+ $MAKE TESTS=b recheck >stdout || { cat stdout; exit 1; }
cat stdout
test ! -e a.run
test -f b.run
rm -f *.run
: No need to re-run a.test or b.test anymore.
- TEST_LOGS=b.log $MAKE -e recheck >stdout \
- || { cat stdout; exit 1; }
+ using_gmake || $sleep # Required by BSD make.
+ $MAKE TESTS=b recheck >stdout || { cat stdout; exit 1; }
cat stdout
count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -e a.run
test ! -e b.run
test ! -e c.run
- TESTS='a.test b.test' $MAKE -e recheck >stdout \
- || { cat stdout; exit 1; }
+ using_gmake || $sleep # Required by BSD make.
+ $MAKE TESTS='a.test b.test' recheck >stdout || { cat stdout; exit 1; }
cat stdout
count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -e a.run
: c.test contained and hard error the last time, so it should be re-run.
: This time, make it pass
# Use 'echo', not ':'; see comments above for why.
+ using_gmake || $sleep # Required by BSD make.
echo dummy > c.ok
- env TESTS='c.test a.test' $MAKE -e recheck >stdout \
- || { cat stdout; exit 1; }
+ $MAKE TESTS='c.test a.test' recheck >stdout || { cat stdout; exit 1; }
cat stdout
count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
test ! -e a.run