2009-03-28 Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
+ Akim Demaille <akim@lrde.epita.fr>
+
+ parallel-tests: redo lazy checking: recheck and RECHECK_LOGS.
+ Replace the LAZY_TEST_SUITE API with a simpler yet more powerful
+ one: RECHECK_LOGS specifies those tests which are to be removed
+ in any case before testing. Provide a `recheck' convenience
+ target to set RECHECK_LOGS to all failed and unexpectedly passed
+ tests. Document several ways to limit the set of tests run.
+ * lib/am/check.am [PARALLEL_TESTS] (RECHECK_LOGS): New variable,
+ default to $(TESTS_LOGS).
+ (check-TESTS): Remove $(RECHECK_LOGS) not $(TEST_LOGS). Drop use
+ of LAZY_TEST_SUITE.
+ ($(TEST_SUITE_LOG)): Do not output note about lazy rerun, as
+ LAZY_TEST_SUITE is gone.
+ (recheck): New target.
+ (recheck-am, recheck-TESTS): New internal targets.
+ * doc/automake.texi (Tests): Update @vindex for TESTS and
+ TEST_LOGS. Replace description of LAZY_TEST_SUITE with a list
+ of ways the set of tests to be run can be modified. Document
+ RECHECK_LOGS and the recheck target.
+ * tests/defs.in: Unset RECHECK_LOGS not LAZY_TEST_SUITE.
+ * tests/parallel-tests.test: Adjust, replacing LAZY_TEST_SUITE
+ with corresponding RECHECK_LOGS settings.
+ * tests/parallel-tests9.test: New tests.
+ * tests/Makefile.am: Update.
+ Suggestion and different implementation by Akim Demaille.
parallel-tests: do not mark check-TESTS as `.MAKE'.
* lib/am/check.am [PARALLEL_TESTS] (.MAKE): Remove check-TESTS.
to satisfy additional portability requirements.
@vindex TEST_SUITE_LOG
-@vindex TEST_LOGS
+@vindex TESTS
The driver operates by defining a set of @command{make} rules to create
a summary log file, @code{TEST_SUITE_LOG}, which defaults to
@file{test-suite.log} and requires a @file{.log} suffix. This file
corresponding tests.
@vindex TEST_EXTENSIONS
+@vindex TEST_LOGS
Each log file is created when the corresponding test has completed.
The set of log files is listed in the read-only variable
@code{TEST_LOGS}, and defaults to @code{TESTS}, with the executable
error}. Unless the variable @code{DISABLE_HARD_ERRORS} is set to a
nonempty value, such tests will be counted as failed.
-@vindex LAZY_TEST_SUITE
-By default, all tests listed in @code{TESTS} are run upon @code{make
-check}. When @code{LAZY_TEST_SUITE} is nonempty, then log files of
-a previous run are not removed before starting the test suite, so only
-tests that have not yet been completed are run, as well as tests that
-have been modified after the previous run.
+By default, the test suite driver will run all tests, but there are
+several ways to limit the set of tests that are run:
+
+@itemize @bullet
+@item
+You can set the @code{TESTS} variable, similarly to how you can with
+the simple test driver from the previous section. For example, you can
+use a command like this to run only a subset of the tests:
+
+@example
+env TESTS="foo.test bar.test" make -e check
+@end example
+
+@item
+You can set the @code{TEST_LOGS} variable. By default, this variable is
+computed at @command{make} run time from the value of @code{TESTS} as
+described above. For example, you can use the following:
+
+@example
+set x subset*.log; shift
+env TEST_LOGS="foo.log $*" make -e check
+@end example
+
+@item
+@vindex RECHECK_LOGS
+@cindex lazy test execution
+By default, the test driver removes all old per-test log files before it
+starts running tests to regenerate them. The variable
+@code{RECHECK_LOGS} contains the set of log files which are removed.
+@code{RECHECK_LOGS} defaults to @code{TEST_LOGS}, which means all tests
+need to be rechecked. By overriding this variable, you can choose which
+tests need to be reconsidered. For example, you can lazily rerun only
+those tests which are outdated, i.e., older than their prerequisite test
+files, by setting this variable to the empty value:
+
+@example
+env RECHECK_LOGS= make -e check
+@end example
+
+@item
+@trindex recheck
+You can ensure that all tests are rerun which have failed or passed
+unexpectedly, by running @code{make recheck} in the test directory.
+This convenience target will set @code{RECHECK_LOGS} appropriately
+before invoking the main test driver.
+@end itemize
In order to guarantee an ordering between tests even with @code{make
-j@var{N}}, dependencies between the corresponding log files may be
else dir="$(srcdir)/"; fi; \
tst=$$dir$$f; log='$@'; __SAVED_TERM=$$TERM; \
$(TESTS_ENVIRONMENT)
+RECHECK_LOGS = $(TEST_LOGS)
TEST_SUITE_LOG = test-suite.log
TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html)
am__test_logs1 = $(TESTS:=.log)
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
- if test -n '$(LAZY_TEST_SUITE)'; then \
- msg="$${msg}(tests were rerun lazily). "; \
- fi; \
if test "$$failures" -ne 0; then \
msg="$${msg}See $(subdir)/$(TEST_SUITE_LOG). "; \
if test -n "$(PACKAGE_BUGREPORT)"; then \
# Run all the tests.
check-TESTS:
- @list='$(TEST_LOGS)'; if test -z '$(LAZY_TEST_SUITE)' \
- && test -n "$$list"; then \
- rm -f $$list; \
- fi
+ @list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
@set_logs=; if test "X$(TEST_LOGS)" = X.log; then \
set_logs=TEST_LOGS=; \
fi; \
$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) $$set_logs
+recheck-TESTS:
+ @list='$(TEST_LOGS)'; \
+ logs=`for f in $$list; do \
+ if read line < $$f; then \
+ case $$line in FAIL*|XPASS*) echo $$f;; esac; \
+ else echo $$f; fi; \
+ done | tr '\012\015' ' '`; \
+ $(MAKE) $(AM_MAKEFLAGS) check-TESTS RECHECK_LOGS="$$logs"
+
+recheck-am: recheck-TESTS
+recheck: recheck-am
.log.html:
@list='$(RST2HTML) $$RST2HTML rst2html rst2html.py'; \
uninstall-am:
-.MAKE: check-am check-html install-am install-strip
+.MAKE: check-am check-html install-am install-strip recheck-am
.PHONY: all all-am check check-TESTS check-am check-html clean \
clean-generic distclean distclean-generic distdir dvi dvi-am \
install-ps install-ps-am install-strip installcheck \
installcheck-am installdirs maintainer-clean \
maintainer-clean-generic mostlyclean mostlyclean-generic pdf \
- pdf-am ps ps-am uninstall uninstall-am
+ pdf-am ps ps-am recheck recheck-TESTS recheck-am uninstall \
+ uninstall-am
# Tell versions [3.59,3.63) of GNU make to not export all variables.
## It provides special support for "unit tests", that is to say,
## tests that (once run) no longer need to be re-compiled and
## re-run at each "make check", unless their sources changed. To
-## enable unit-test supports, define LAZY_TEST_SUITE. In such a
+## enable unit-test supports, set RECHECK_LOGS to empty. In such a
## setting, that heavily relies on correct dependencies, its users may
## prefer to define EXTRA_PROGRAMS instead of check_PROGRAMS, because
## it allows intertwined compilation and execution of the tests.
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
- if test -n '$(LAZY_TEST_SUITE)'; then \
- msg="$${msg}(tests were rerun lazily). "; \
- fi; \
if test "$$failures" -ne 0; then \
msg="$${msg}See $(subdir)/$(TEST_SUITE_LOG). "; \
if test -n "$(PACKAGE_BUGREPORT)"; then \
test x"$$VERBOSE" = x || $$exit || cat $(TEST_SUITE_LOG); \
$$exit
+RECHECK_LOGS = $(TEST_LOGS)
+
# Run all the tests.
check-TESTS:
-## Expand $(TEST_LOGS) only once, to avoid exceeding line length limits.
- @list='$(TEST_LOGS)'; if test -z '$(LAZY_TEST_SUITE)' \
- && test -n "$$list"; then \
- rm -f $$list; \
- fi
+## Expand $(RECHECK_LOGS) only once, to avoid exceeding line length limits.
+ @list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
## We always have to remove TEST_SUITE_LOG, to ensure its rule is run
## in any case even in lazy mode: otherwise, if no test needs rerunning,
## or a prior run plus reruns all happen within the same timestamp
fi; \
$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) $$set_logs
+## Rerun all FAILed or XPASSed tests (as well as all whose logs are out
+## of date or do not exist).
+recheck-TESTS:
+ @list='$(TEST_LOGS)'; \
+ logs=`for f in $$list; do \
+ if read line < $$f; then \
+ case $$line in FAIL*|XPASS*) echo $$f;; esac; \
+ else echo $$f; fi; \
+ done | tr '\012\015' ' '`; \
+ $(MAKE) $(AM_MAKEFLAGS) check-TESTS RECHECK_LOGS="$$logs"
+
+recheck-am: recheck-TESTS
+recheck: recheck-am
+.PHONY: recheck recheck-am recheck-TESTS
+.MAKE: recheck-am
## -------------- ##
## Produce HTML. ##
parallel-tests6.test \
parallel-tests7.test \
parallel-tests8.test \
+parallel-tests9.test \
parse.test \
percent.test \
percent2.test \
else dir="$(srcdir)/"; fi; \
tst=$$dir$$f; log='$@'; __SAVED_TERM=$$TERM; \
$(TESTS_ENVIRONMENT)
+RECHECK_LOGS = $(TEST_LOGS)
TEST_SUITE_LOG = test-suite.log
TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html)
TEST_EXTENSIONS = .test
parallel-tests6.test \
parallel-tests7.test \
parallel-tests8.test \
+parallel-tests9.test \
parse.test \
percent.test \
percent2.test \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
- if test -n '$(LAZY_TEST_SUITE)'; then \
- msg="$${msg}(tests were rerun lazily). "; \
- fi; \
if test "$$failures" -ne 0; then \
msg="$${msg}See $(subdir)/$(TEST_SUITE_LOG). "; \
if test -n "$(PACKAGE_BUGREPORT)"; then \
# Run all the tests.
check-TESTS:
- @list='$(TEST_LOGS)'; if test -z '$(LAZY_TEST_SUITE)' \
- && test -n "$$list"; then \
- rm -f $$list; \
- fi
+ @list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
@set_logs=; if test "X$(TEST_LOGS)" = X.log; then \
set_logs=TEST_LOGS=; \
fi; \
$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) $$set_logs
+recheck-TESTS:
+ @list='$(TEST_LOGS)'; \
+ logs=`for f in $$list; do \
+ if read line < $$f; then \
+ case $$line in FAIL*|XPASS*) echo $$f;; esac; \
+ else echo $$f; fi; \
+ done | tr '\012\015' ' '`; \
+ $(MAKE) $(AM_MAKEFLAGS) check-TESTS RECHECK_LOGS="$$logs"
+
+recheck-am: recheck-TESTS
+recheck: recheck-am
.log.html:
@list='$(RST2HTML) $$RST2HTML rst2html rst2html.py'; \
uninstall-am:
-.MAKE: check-am check-html install-am install-strip
+.MAKE: check-am check-html install-am install-strip recheck-am
.PHONY: all all-am check check-TESTS check-am check-html clean \
clean-generic clean-local distclean distclean-generic distdir \
install-pdf-am install-ps install-ps-am install-strip \
installcheck installcheck-am installdirs maintainer-clean \
maintainer-clean-generic mostlyclean mostlyclean-generic pdf \
- pdf-am ps ps-am uninstall uninstall-am
+ pdf-am ps ps-am recheck recheck-TESTS recheck-am uninstall \
+ uninstall-am
$(srcdir)/parallel-tests.am: gen-parallel-tests Makefile.am
# need to control (and test for) in some of the tests to ensure
# backward-compatible behavior.
unset DISABLE_HARD_ERRORS
-unset LAZY_TEST_SUITE
+unset RECHECK_LOGS
unset VERBOSE
echo "=== Running test $0"
# - TEST_SUITE_LOG
# - dependencies between tests
# - DISABLE_HARD_ERRORS
-# - LAZY_TEST_SUITE
+# - RECHECK_LOGS
. ./defs-p || Exit 1
# Note that the previous test and this one taken together expose the timing
# issue that requires the check-TESTS rule to always remove TEST_SUITE_LOG
# before running the tests lazily.
-env LAZY_TEST_SUITE=yes $MAKE -e check > stdout && { cat stdout; Exit 1; }
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
cat stdout
test -f foo.log
grep foo.test stdout
grep bar.test stdout && Exit 1
grep baz.test stdout && Exit 1
grep '2.*tests.*failed' stdout
-grep 'lazily' stdout
# Now, explicitly retry with all test logs already updated, and ensure
# that the summary is still displayed.
-env LAZY_TEST_SUITE=yes $MAKE -e check > stdout && { cat stdout; Exit 1; }
+env RECHECK_LOGS= $MAKE -e check > stdout && { cat stdout; Exit 1; }
cat stdout
grep foo.test stdout && Exit 1
grep bar.test stdout && Exit 1
grep baz.test stdout && Exit 1
grep '2.*tests.*failed' stdout
+# Lazily rerunning only foo should only rerun this one test.
+env RECHECK_LOGS=foo.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
+cat stdout
+grep foo.test stdout
+grep bar.test stdout && Exit 1
+grep baz.test stdout && Exit 1
+grep '2.*tests.*failed' stdout
+
# Test VERBOSE.
env VERBOSE=yes $MAKE -e check > stdout && { cat stdout; Exit 1; }
cat stdout
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2009 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features:
+# - recheck
+
+. ./defs-p || Exit 1
+
+set -e
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_SUITE_LOG = mylog.log
+TESTS = foo.test bar.test baz.test
+END
+
+cat >>foo.test <<'END'
+#! /bin/sh
+echo "this is $0"
+exit 0
+END
+cat >>bar.test <<'END'
+#! /bin/sh
+echo "this is $0"
+exit 99
+END
+cat >>baz.test <<'END'
+#! /bin/sh
+echo "this is $0"
+exit 1
+END
+chmod a+x foo.test bar.test baz.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+$MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+
+$MAKE recheck >stdout && { cat stdout; Exit 1; }
+cat stdout
+grep foo.test stdout && Exit 1
+grep bar.test stdout || Exit 1
+grep baz.test stdout || Exit 1
+
+# If we cannot read the log file, then redo it as well.
+chmod a-r foo.log
+$MAKE recheck >stdout && { cat stdout; Exit 1; }
+cat stdout
+grep foo.test stdout || Exit 1
+
+:
projects / thirdparty / automake.git / commitdiff
