From: Stefano Lattarini Date: Sun, 15 Apr 2012 12:00:52 +0000 (+0200) Subject: [ng] check: big refactoring with semantic changes in parallel-tests support X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=12711fcc0f3dc914b7e63345a25a186b8d4e401f;p=thirdparty%2Fautomake.git [ng] check: big refactoring with semantic changes in parallel-tests support In this change, we move some further processing related to the parallel testsuite harness support from Automake runtime to make runtime. A welcome collateral effect of this is that we are now able to cope with test scripts whose name ends with several concatenated suffixes. Another (less welcome) collateral effect of the present change is that the user now cannot override TEST_LOGS at make runtime anymore: # This won't work anymore! make check TESTS_LOGS="foo.log bar.log baz.log" This admittedly is a small(ish) regression. But it's not a real problem, since we now explicitly allow *and document* that TESTS can be overridden at runtime without bothering to add the test suffixes to its entries: # This will work even if complete names of the tests are (say) # "foo.test", "bar.test$(EXEEXT)" and "baz$(EXEEXT)". make check TESTS="foo bar baz" With such usage, the user don't have to bother knowing which the exact extensions of each tests are -- which was precisely the only advantage of overriding TEST_LOGS instead of TESTS. * NG-NEWS: Update. * doc/automake.texi: Likewise. * lib/am/header-vars.am (am__strip_suffixes_0, am__strip_suffixes): New internal macros ... * t/internals.tap: ... covered by new checks in this test, and used ... * lib/am/check.am (am__xfail_test_bases): ... to define this new internal variable in term of $(am__cooked_xfail_tests) (and thus eventually of $(XFAIL_TESTS)), and to ... (am__TEST_BASES): ... (re)define this pre-existing internal variable in terms of $(am__cooked_tests) (and thus eventually of $(TESTS)), rather than in terms of $(TEST_LOGS) as was done before. (am__TEST_RESULTS, am__TEST_LOGS): Define in terms of $(am__TEST_BASES). (TEST_LOGS): Define to $(am__TEST_LOGS), for backward-compatibility. ($(TEST_SUITE_LOG)): Depend on $(am__TEST_RESULTS) rather than on $(TEST_LOGS). (am__check_pre): Use $(am__xfail_test_bases) and new temporary shell variable $f2, instead of $(am__cooked_xfail_tests) and $f, to decide whether a test is expected to fail or not. Fix some imprecise or botched comments since we are at it. (check-TESTS, recheck): When re-invoking make recursively, override am__TEST_BASES rather than TEST_LOGS. Relate tweaks. * automke.in (handle_tests): Pass new transform %HANDLE-EXEEXT% when including check.am (used for the definitions of am__xfail_test_bases and am__TEST_BASES). Don't define the TEST_LOGS make variable, nor the auxiliary make variables am__test_logs1, am__test_logs2, etc. Put $(am__TEST_LOGS) and $(am__TEST_RESULTS) among the "mostlyclean" files (rather than "$(TEST_LOGS)" and "$(TEST_LOGS:.log=.trs)" as before). * t/parallel-tests-concatenated-suffix.sh: New test. * t/nodeps.sh: Delete, it was causing too much spurious failures. * t/color2.sh: Avoid a spurious failure by not calling make with the '-e' flag. * t/parallel-tests-exeext.sh: Extended to check the user is not forced to specify the test suffixes nor the $(EXEEXT) suffix when overriding TESTS on the command line. * t/check-concurrency-bug9245.sh: Adjust to the new semantics of "no TEST_LOGS overriding from the command line". * t/parallel-tests-cmdline-override.sh: Likewise. * t/parallel-tests-dynamic.sh: Likewise. * t/parallel-tests-fork-bomb.sh: Likewise. * t/parallel-tests-log-override-2.sh: Likewise. * t/tap-basic.sh: Likewise. * t/test-driver-custom-multitest-recheck2.sh: Likewise. * t/test-missing.sh: Likewise. * t/test-trs-basic.sh: Likewise. * t/test-trs-recover.sh: Likewise. * t/parallel-tests-empty-tests.sh: Likewise, and extended to check that an empty TESTS produces am__cooked_tests, am__TEST_RESULTS, am__TEST_BASES and am__TEST_LOGS that are empty as well. * t/parallel-tests.sh: Adjust to the new semantics of "no TEST_LOGS overriding from the command line". Remove a workaround required only by HP-UX make. Make some "FIXME"-style comments stick out. * t/serial-tests.sh: Adjust some checks that grep the Automake generated Makefile.in. Signed-off-by: Stefano Lattarini --- diff --git a/NG-NEWS b/NG-NEWS index 49c16ea34..1f638b6db 100644 --- a/NG-NEWS +++ b/NG-NEWS @@ -99,6 +99,20 @@ Parallel testsuite harness work as expected: TESTS = $(wildcard $(srcdir)/t[0-9][0-9]*.sh) +* It is not anymore possible to override TEST_LOGS at make runtime as a + mean to redefine the list of tests to be run. + + # This won't work anymore! + make check TESTS_LOGS="foo.log bar.log baz.log" + + But it's still possible do so overriding TESTS at make runtime; and + in this case (as was the case for TEST_LOGS overriding), it is not + necessary to specify the extensions of the tests to be run: + + # This will work even if complete names of the tests are (say) + # "foo.test", "bar.test$(EXEEXT)" and "baz$(EXEEXT)". + make check TESTS="foo bar baz" + Pattern rules and suffix rules ============================== diff --git a/automake.in b/automake.in index 11d1f0b15..1b8462ddc 100644 --- a/automake.in +++ b/automake.in @@ -4708,11 +4708,12 @@ sub handle_tests if (var ('TESTS')) { push (@check_tests, 'check-TESTS'); - my $check_deps = "@check"; $output_rules .= &file_contents ('check', new Automake::Location, COLOR => !! option 'color-tests', PARALLEL_TESTS => !! option 'parallel-tests', - CHECK_DEPS => $check_deps); + CHECK_DEPS => "@check", + 'HANDLE-EXEEXT' => (exists $configure_vars{'EXEEXT'} ? + 'TRUE' : 'FALSE')); if (my $parallel_tests = option 'parallel-tests') { @@ -4743,30 +4744,12 @@ sub handle_tests # an explicit 'all' target) rather than from the 'all.test' script, # thus causing all sort of mishaps and confusion. push @test_suffixes, ''; - - define_variable ('am__test_logs1', - '$(patsubst %,%.log,$(strip $(am__cooked_tests)))', - INTERNAL); - my $nhelper = 1; foreach my $test_suffix (@test_suffixes) { handle_per_suffix_test ($test_suffix); - my @cooked_test_suffixes = ($test_suffix); - push @cooked_test_suffixes, ($test_suffix . '$(EXEEXT)') - if exists $configure_vars{'EXEEXT'}; - foreach my $cooked_test_suffix (@cooked_test_suffixes) - { - my $old_var = 'am__test_logs' . $nhelper++; - my $cur_var = 'am__test_logs' . $nhelper; - define_variable ( - $cur_var, - "\$($old_var:$cooked_test_suffix.log=.log)", - INTERNAL); - } } - define_variable ('TEST_LOGS', "\$(am__test_logs$nhelper)", INTERNAL); - $clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN; - $clean_files{'$(TEST_LOGS:.log=.trs)'} = MOSTLY_CLEAN; + $clean_files{'$(am__TEST_LOGS)'} = MOSTLY_CLEAN; + $clean_files{'$(am__TEST_RESULTS)'} = MOSTLY_CLEAN; $clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN; } else diff --git a/doc/automake.texi b/doc/automake.texi index c5ac92db1..471c2a421 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -8996,14 +8996,13 @@ For best results, the tests should be verbose by default now. @vindex TEST_LOGS Each couple of @file{.log} and @file{.trs} files is created when the corresponding test has completed. The set of log files is listed in -the read-only variable @code{TEST_LOGS}, and defaults to @code{TESTS}, -with the executable extension if any (@pxref{EXEEXT}), as well as any -suffix listed in @code{TEST_EXTENSIONS} removed, and @file{.log} appended. -Results are undefined if a test file name ends in several concatenated -suffixes. @code{TEST_EXTENSIONS} defaults to @file{.test}; it can be -overridden by the user, in which case any extension listed in it must be -constituted by a dot, followed by a non-digit alphabetic character, -followed by any number of alphabetic characters. +the @emph{read-only} variable @code{TEST_LOGS}, and defaults to +@code{TESTS}, with the executable extension if any (@pxref{EXEEXT}), +as well as any suffix listed in @code{TEST_EXTENSIONS} removed, and +@file{.log} appended. @code{TEST_EXTENSIONS} defaults to @file{.test}; +it can be overridden by the user, in which case any extension listed +in it must be constituted by a dot, followed by a non-digit alphabetic +character, followed by any number of alphabetic characters. @c Keep in sync with test-extensions.sh For example, @samp{.sh}, @samp{.T} and @samp{.t1} are valid extensions, while @samp{.x-y}, @samp{.6c} and @samp{.t.1} are not. @@ -9075,10 +9074,17 @@ You can set the @code{TESTS} variable. For example, you can use a command like this to run only a subset of the tests: @example -make check TESTS="foo.test bar.test" +make check TESTS="foo.test bar.sh" @end example -Note however that the command above will unconditionally overwrite the +If @code{.test} and @code{.sh} are in @code{$(TEST_EXTENSIONS)}, the +command above can also be shortened as: + +@example +make check TESTS="foo bar" +@end example + +Note however that the commands above will unconditionally overwrite the @file{test-suite.log} file, thus clobbering the recorded results of any previous testsuite run. This might be undesirable for packages whose testsuite takes long time to execute. Luckily, this problem can @@ -9094,19 +9100,8 @@ will write the result of the partial testsuite runs to the @file{partial.log}, without touching @file{test-suite.log}. @item -You can set the @code{TEST_LOGS} variable. By default, this variable is -computed at @command{make} run time from the value of @code{TESTS} as -described above. For example, you can use the following: - -@example -set x subset*.log; shift -make check TEST_LOGS="foo.log $*" -@end example - -The comments made above about @code{TEST_SUITE_LOG} overriding applies -here too. - -@item +@c FIXME: this should become "RECHECK_TESTS", for consistency. +@c FIXME: this will imply a change to the API, of course. @vindex RECHECK_LOGS @cindex lazy test execution By default, the test harness removes all old per-test @file{.log} and @@ -9217,7 +9212,7 @@ parallel harness remains in place; this includes: @itemize @item list of test scripts defined in @code{TESTS}, and overridable at -runtime through the redefinition of @code{TESTS} or @code{TEST_LOGS}; +runtime through the redefinition of @code{TESTS}; @item concurrency through the use of @command{make}'s option @option{-j}; @item diff --git a/lib/am/check.am b/lib/am/check.am index 8d5aa403e..ae40c0099 100644 --- a/lib/am/check.am +++ b/lib/am/check.am @@ -202,12 +202,10 @@ am__common_driver_flags = \ --expect-failure "$$am__expect_failure" # To be inserted before the command running the test. Creates the -# directory for the log if needed. Stores in $dir the directory -# containing $f, in $tst the test, in $log the log. Executes the -# developer- defined test setup AM_TESTS_ENVIRONMENT (if any), and -# passes TESTS_ENVIRONMENT. Set up options for the wrapper that -# will run the test scripts (or their associated LOG_COMPILER, if -# thy have one). +# directory for the log if needed. Executes the developer-defined +# test setup AM_TESTS_ENVIRONMENT (if any), and passes TESTS_ENVIRONMENT. +# Set up options for the wrapper that will run the test scripts (or their +# associated LOG_COMPILER, if they have one). am__check_pre = \ $(am__sh_e_setup); \ $(am__tty_colors); \ @@ -225,24 +223,34 @@ if test -n '$(DISABLE_HARD_ERRORS)'; then \ else \ am__enable_hard_errors=yes; \ fi; \ -case " $(am__cooked_xfail_tests) " in \ - *" $$f "*) am__expect_failure=yes;; \ - *) am__expect_failure=no;; \ +f2='$(patsubst $(srcdir)/%,%,$*)'; \ +case " $(am__xfail_test_bases) " in \ + *" $$f2 "*) am__expect_failure=yes;; \ + *) am__expect_failure=no;; \ esac; \ $(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT) # The names of the tests scripts with any registered extension removed -# (i.e., equivalently, the names of the test logs, with the '.log' suffix -# stripped), and the name of the test result files (i.e., equivalently, the -# names of the test logs, with the '.log' suffix substituted by the '.trs' -# suffix). This honors runtime overriding of TESTS and TEST_LOGS. It -# also works around the GNU make 3.80 bug where trailing whitespace in -# "TESTS = foo.test $(empty)" causes $(TESTS_LOGS) to erroneously expand -# to "foo.log .log". -am__TEST_BASES = $(patsubst %.log,%,$(strip $(TEST_LOGS))) -am__TEST_RESULTS = $(patsubst %.log,%.trs,$(strip $(TEST_LOGS))) +# This honors runtime overriding of TESTS, and takes into account +# $(EXEEXT) appending. +am__TEST_BASES = $(call am__strip_suffixes, $(TEST_EXTENSIONS), \ +?!HANDLE-EXEEXT?$(am__cooked_tests)) +?HANDLE-EXEEXT?$(patsubst %$(EXEEXT),%,$(am__cooked_tests))) +# Likewise for the xfailing tests. +am__xfail_test_bases = $(call am__strip_suffixes, $(TEST_EXTENSIONS), \ +?!HANDLE-EXEEXT?$(am__cooked_xfail_tests)) +?HANDLE-EXEEXT?$(patsubst %$(EXEEXT),%,$(am__cooked_xfail_tests))) -$(TEST_SUITE_LOG): $(TEST_LOGS) $(am__TEST_RESULTS) +# The $(strip) is to work around the GNU make 3.80 bug where trailing +# whitespace in "TESTS = foo.test $(empty)" causes $(TESTS_LOGS) to +# erroneously expand to "foo.log .log". +am__TEST_RESULTS = $(addsuffix .trs,$(strip $(am__TEST_BASES))) +am__TEST_LOGS = $(addsuffix .log,$(strip $(am__TEST_BASES))) + +# $(TEST_LOGS) is a published interface. +TEST_LOGS = $(am__TEST_LOGS) + +$(TEST_SUITE_LOG): $(am__TEST_LOGS) $(am__TEST_RESULTS) @$(am__sh_e_setup); $(am__tty_colors); \ fatal () { echo "fatal: making $@: $$*" >&2; exit 1; }; \ ## Detect a possible circular dependency, and error out. @@ -394,6 +402,8 @@ check-TESTS recheck: ## unexpected outcome (FAIL or XPASS) in the earlier run. bases=`for i in $$bases; do echo $$i; done \ | $(am__list_recheck_tests)` || exit 1; \ +## Remove newlines and normalize whitespace. + bases=`echo $$bases`; \ fi; \ log_list=`for i in $$bases; do echo $$i.log; done`; \ trs_list=`for i in $$bases; do echo $$i.trs; done`; \ @@ -407,7 +417,7 @@ check-TESTS recheck: test -z "$$log_list" || rm -f $$log_list; \ test -z "$$trs_list" || rm -f $$trs_list; \ fi; \ - $(MAKE) $(TEST_SUITE_LOG) TEST_LOGS="$$log_list" + $(MAKE) $(TEST_SUITE_LOG) am__TEST_BASES="$$bases" ## Recheck must depend on $(check_SCRIPTS), $(check_PROGRAMS), etc. ## It must also depend on the 'all' target. See automake bug#11252. diff --git a/lib/am/header-vars.am b/lib/am/header-vars.am index d620c8948..f6cb43708 100644 --- a/lib/am/header-vars.am +++ b/lib/am/header-vars.am @@ -98,6 +98,26 @@ am__uniq = $(strip \ am__memoize = $(or $(am__memoized_value/$1),$(strip \ $(eval am__memoized_value/$1 := $$2))$(am__memoized_value/$1)) +## $(call am__strip_suffixes_0, SUFFIXES, WORD) +## -------------------------------------------- +## Strip any of the SUFFIXES from WORD. Even if WORD terminates with +## several suffixes, only one is stripped: the first one that matches. +am__strip_suffixes_0 = $(strip \ + $(if $(strip $(1)), \ + $(if $(filter %$(firstword $(1)), $(2)), \ + $(patsubst %$(firstword $(1)), %, $(2)), \ + $(call am__strip_suffixes_0, \ + $(call am__strip_firstword, $(1)), $(2))), \ + $(2))) + +## $(call am__strip_suffixes, SUFFIXES, LIST) +## ------------------------------------------ +## Strip any of the SUFFIXES from each of the entries of LIST. Even if an +## entry of LIST terminates with several suffixes, only one is stripped: +## the first one that matches. +am__strip_suffixes = \ + $(foreach am__i,$(2),$(call am__strip_suffixes_0,$(1),$(am__i))) + ## Some derived variables that have been found to be useful. pkgdatadir = $(datadir)/@PACKAGE@ pkgincludedir = $(includedir)/@PACKAGE@ diff --git a/t/check-concurrency-bug9245.sh b/t/check-concurrency-bug9245.sh index c2d3d53a3..87fcadece 100755 --- a/t/check-concurrency-bug9245.sh +++ b/t/check-concurrency-bug9245.sh @@ -49,7 +49,7 @@ for j in '' -j1 -j2; do $MAKE $j TESTS=foo.test check && Exit 1 if test x"$am_parallel_tests" = x"yes"; then $MAKE $j recheck && Exit 1 - $MAKE $j TEST_LOGS=foo.log check && Exit 1 + $MAKE $j TESTS=foo check && Exit 1 rm -f test-suite.log $MAKE $j test-suite.log && Exit 1 test -f test-suite.log || Exit 1 diff --git a/t/color2.sh b/t/color2.sh index d03c6a20e..4e6862a9d 100755 --- a/t/color2.sh +++ b/t/color2.sh @@ -107,7 +107,7 @@ test_no_color () } cat >expect-make <<'END' -eval spawn $env(MAKE) -e check +eval spawn $env(MAKE) check expect eof END diff --git a/t/internals.tap b/t/internals.tap index 6efc96f5e..6d6916603 100755 --- a/t/internals.tap +++ b/t/internals.tap @@ -19,7 +19,7 @@ am_create_testdir=empty . ./defs || Exit 1 -plan_ 3 +plan_ 5 cp "$am_amdir"/header-vars.am . \ || fatal_ "fetching makefile fragment headers-vars.am" @@ -29,6 +29,8 @@ cp "$am_amdir"/header-vars.am . \ LC_ALL=C $EGREP -v '(^##|=.*@[a-zA-Z0-9_]+@)' header-vars.am > defn.mk rm -f header-vars.am +# WARNING: there are a lot of embedded tabs in this makefile. +# DO NOT "NORMALIZE" THEM TO SPACES! cat > Makefile << 'END' include ./defn.mk @@ -72,10 +74,60 @@ test-uniq: test '$(call am__uniq,3 1 1 4 1 4 1 1)' = '3 1 4' test '$(call am__uniq, 1 3 1 )' = '1 3' +.PHONY: test-strip-suffixes0 +test-strip-suffixes0: + test '$(call am__strip_suffixes_0,,)' = '' + test '$(call am__strip_suffixes_0, ,)' = '' + test '$(call am__strip_suffixes_0,, )' = '' + test '$(call am__strip_suffixes_0, , )' = '' + test '$(call am__strip_suffixes_0,x,)' = '' + test '$(call am__strip_suffixes_0,x, )' = '' + test '$(call am__strip_suffixes_0,x y, )' = '' + test '$(call am__strip_suffixes_0,,x)' = 'x' + test '$(call am__strip_suffixes_0, ,x)' = 'x' + test '$(call am__strip_suffixes_0,.c,foo.c)' = 'foo' + test '$(call am__strip_suffixes_0,.c .c++, bar.c++)' = 'bar' + test '$(call am__strip_suffixes_0,.c .c++, bar.cxx)' = 'bar.cxx' + test '$(call am__strip_suffixes_0,x,ax)' = 'a' + test '$(call am__strip_suffixes_0,x,xa)' = 'xa' + test '$(call am__strip_suffixes_0,x,xx)' = 'x' + test '$(call am__strip_suffixes_0,x,xux)' = 'xu' + test '$(call am__strip_suffixes_0, .a .b, x.a.a)' = 'x.a' + test '$(call am__strip_suffixes_0, .a .b, x.a.b)' = 'x.a' + test '$(call am__strip_suffixes_0, .a .b, x.b.a)' = 'x.b' + test '$(call am__strip_suffixes_0, .a .b, x.b.b)' = 'x.b' + # Corner cases: the *first* matched suffix is stripped + test '$(call am__strip_suffixes_0, .a .b.a, foo.b.a)' = 'foo.b' + test '$(call am__strip_suffixes_0, .b.a .a, foo.b.a)' = 'foo' + +.PHONY: test-strip-suffixes +test-strip-suffixes: + test '$(call am__strip_suffixes,,)' = '' + test '$(call am__strip_suffixes, ,)' = '' + test '$(call am__strip_suffixes,, )' = '' + test '$(call am__strip_suffixes, , )' = '' + test '$(call am__strip_suffixes,x,)' = '' + test '$(call am__strip_suffixes,x y, )' = '' + test '$(call am__strip_suffixes,,x y)' = 'x y' + test '$(call am__strip_suffixes, ,x y)' = 'x y' + test '$(call am__strip_suffixes,.c,foo.c)' = 'foo' + test '$(call am__strip_suffixes,.foo,a.foo b.foo)' = 'a b' + test '$(call am__strip_suffixes, x y, ax ay ax)' = 'a a a' + test '$(call am__strip_suffixes, .c .c++, \ + foo.c bar.c++ baz.cxx zap.c)' = 'foo bar baz.cxx zap' + test '$(call am__strip_suffixes, .a .b, \ + 1.a.a 2.a.b 3.b.a 4.b.b)' = '1.a 2.a 3.b 4.b' + # Corner cases: the *first* matched suffix is stripped + test '$(call am__strip_suffixes, .a .b.a, foo.b.a bar.a)' \ + = 'foo.b bar' + test '$(call am__strip_suffixes, .b.a .a, foo.b.a bar.a)' \ + = 'foo bar' END -command_ok_ "am__strip_firstword" $MAKE test-strip-firstword -command_ok_ "am__strip_lastword" $MAKE test-strip-lastword -command_ok_ "am__uniq" $MAKE test-uniq +command_ok_ am__strip_firstword $MAKE test-strip-firstword +command_ok_ am__strip_lastword $MAKE test-strip-lastword +command_ok_ am__uniq $MAKE test-uniq +command_ok_ am__test_strip_suffixes_0 $MAKE test-strip-suffixes0 +command_ok_ am__test_strip_suffixes $MAKE test-strip-suffixes : diff --git a/t/parallel-tests-cmdline-override.sh b/t/parallel-tests-cmdline-override.sh index 7d219e9b9..5dcd93ad0 100755 --- a/t/parallel-tests-cmdline-override.sh +++ b/t/parallel-tests-cmdline-override.sh @@ -14,8 +14,8 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# Check that we can use indirections when overriding TESTS and -# TEST_LOGS from the command line. +# Check that we can use indirections when overriding TESTS from +# the command line. am_parallel_tests=yes . ./defs || Exit 1 @@ -73,12 +73,12 @@ do_check () return $st } -tests='a.t $(var1) $(var3:.d=.t) $(var4:=.test)' -test_logs='a.log $(var1:.test=.log) $(var3:.d=.log) $(var4:=.log)' +tests1='a.t $(var1) $(var3:.d=.t) $(var4:=.test)' +tests2='a $(var1:.test=) $(var3:.d=) $(var4)' touch a.t b.test c.test d.t e.test -do_check TESTS="$tests" -do_check TEST_LOGS="$test_logs" +do_check TESTS="$tests1" +do_check TESTS="$tests2" : diff --git a/t/parallel-tests-concatenated-suffix.sh b/t/parallel-tests-concatenated-suffix.sh new file mode 100755 index 000000000..079122226 --- /dev/null +++ b/t/parallel-tests-concatenated-suffix.sh @@ -0,0 +1,76 @@ +#! /bin/sh +# Copyright (C) 2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# The parallel-tests driver should be able to cope with test scripts +# whose names end with several concatenated suffixes. + +am_parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.ac << 'END' +AC_OUTPUT +END + +tests='foo.sh foo.t.sh foo.sh.t foo.x.x foo.x.t.sh foo.t.x.sh foo.sh.t.x' + +for t in $tests; do + (echo '#!/bin/sh' && echo 'echo == /$0 ==') > $t + chmod a+x $t +done + +cat > Makefile.am <stdout + $MAKE $j check >>stdout || { cat stdout; Exit 1; } + cat stdout + count_test_results total=7 pass=7 fail=0 skip=0 xfail=0 xpass=0 error=0 + for t in $tests; do grep "^PASS: $t *$" stdout; done + + grep '== .*/foo\.sh ==' foo.log + grep '== .*/foo\.t\.sh ==' foo.t.log + grep '== .*/foo\.sh\.t ==' foo.sh.log + grep '== .*/foo\.x\.x ==' foo.x.log + grep '== .*/foo\.x\.t\.sh ==' foo.x.t.log + grep '== .*/foo\.t\.x\.sh ==' foo.t.x.log + grep '== .*/foo\.sh\.t\.x ==' foo.sh.t.log + + $MAKE $j clean + test ! -f foo.log + test ! -f foo.t.log + test ! -f foo.sh.log + test ! -f foo.x.log + test ! -f foo.x.t.log + test ! -f foo.t.x.log + test ! -f foo.sh.t.log + +done + +: diff --git a/t/parallel-tests-dynamic.sh b/t/parallel-tests-dynamic.sh index 3b4fcb983..a18662c7a 100755 --- a/t/parallel-tests-dynamic.sh +++ b/t/parallel-tests-dynamic.sh @@ -14,8 +14,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# Check that dynamic content for $(TESTS) and $(TEST_LOGS) is -# supported. +# Check that dynamic content for $(TESTS) is supported. am_parallel_tests=yes . ./defs || Exit 1 @@ -124,16 +123,15 @@ grep '^XFAIL: t98S\.sh' stdout $MAKE mostlyclean test "`find . -name *.log`" = ./config.log -# We can also override $(TEST_LOGS) dynamically. -$MAKE check TEST_LOGS='$(shell echo t00 | sed "s/$$/-foo.log/") t99.log' +$MAKE check TESTS='$(shell echo t00 | sed "s/$$/-foo/") t99' test -f t00-foo.log test -f t99.log # A little tricky in that we rely on the .log files created by # the previous run to be present. -$MAKE check TEST_LOGS="\ - \$(wildcard t[0-9]*.log) \ - \$(call my_add_dirprefix, t, nosuffix).log \ +$MAKE check TESTS="\ + \$(patsubst %.log,%,\$(wildcard t[0-9]*.log)) \ + \$(call my_add_dirprefix, t, nosuffix) \ " > stdout || { cat stdout; Exit 1; } cat stdout diff --git a/t/parallel-tests-empty-testlogs.sh b/t/parallel-tests-empty-tests.sh similarity index 90% rename from t/parallel-tests-empty-testlogs.sh rename to t/parallel-tests-empty-tests.sh index c49a41205..018cce0c7 100755 --- a/t/parallel-tests-empty-testlogs.sh +++ b/t/parallel-tests-empty-tests.sh @@ -14,9 +14,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# Check parallel-tests features: -# - empty TESTS -# - empty TEST_LOGS +# Check parallel-tests features: empty TESTS am_parallel_tests=yes . ./defs || Exit 1 @@ -35,7 +33,9 @@ mkdir sub1 sub2 cat > sub1/Makefile.am << 'END' TESTS = check-local: - echo $(TEST_LOGS) | grep . && exit 1; exit 0 + echo $(am__cooked_tests) $(am__TEST_LOGS) \ + $(am__TEST_RESULTS) $(am__TEST_BASES) \ + | grep . && exit 1; exit 0 END cat > sub2/Makefile.am << 'END' @@ -74,8 +74,6 @@ for vpath in : false; do cd ../sub2 $MAKE check VERBOSE=yes TESTS='' no_test_has_run - $MAKE check VERBOSE=yes TEST_LOGS='' - no_test_has_run cd .. $MAKE check cat sub2/foo.log diff --git a/t/parallel-tests-exeext.sh b/t/parallel-tests-exeext.sh index ac94533c5..14cb753f2 100755 --- a/t/parallel-tests-exeext.sh +++ b/t/parallel-tests-exeext.sh @@ -61,4 +61,15 @@ test ! -r y.bin.log test -f b.log test ! -r b.test.log +# Opportunistically check that we are not forced to specify +# the test suffixes nor the $(EXEEXT) suffix when overriding +# TESTS on the command line. +rm -f *.log *.trs + +$MAKE check TESTS='y a b' +ls -l # For debugging. +test -f a.log +test -f b.log +test -f y.log + : diff --git a/t/parallel-tests-fork-bomb.sh b/t/parallel-tests-fork-bomb.sh index 193a6e754..5029d2ce1 100755 --- a/t/parallel-tests-fork-bomb.sh +++ b/t/parallel-tests-fork-bomb.sh @@ -75,8 +75,7 @@ rm -f *.log *.test : > 2.test : > 3.test : > foobar.test -do_check foobar.log TEST_LOGS='0.log 1.log foobar.log 2.log 3.log' \ - TEST_SUITE_LOG=foobar.log +do_check foobar.log TESTS='0 1 foobar 2 3' TEST_SUITE_LOG=foobar.log rm -f *.log *.test : diff --git a/t/parallel-tests-log-override-2.sh b/t/parallel-tests-log-override-2.sh index 3c11c9cc0..ab839439f 100755 --- a/t/parallel-tests-log-override-2.sh +++ b/t/parallel-tests-log-override-2.sh @@ -14,9 +14,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# Check parallel-tests features: runtime redefinition of: -# - $(TEST_SUITE_LOG) and $(TESTS) -# - $(TEST_SUITE_LOG) and $(TEST_LOGS) +# parallel-tests: runtime redefinition of $(TEST_SUITE_LOG) and $(TESTS) am_parallel_tests=yes . ./defs || Exit 1 @@ -60,8 +58,8 @@ $AUTOMAKE -a # Extra quoting for the sake of Solaris sh bugs. for test_list_override in \ 'TESTS=pass.test skip.test' \ - 'TEST_LOGS=pass.log skip.log' -do + 'TESTS=pass skip' \ +; do eval "\$MAKE TEST_SUITE_LOG=partial.log '$test_list_override'" \ check >stdout || { cat stdout; Exit 1; } cat stdout diff --git a/t/parallel-tests.sh b/t/parallel-tests.sh index 4861ff54f..63949a2fb 100755 --- a/t/parallel-tests.sh +++ b/t/parallel-tests.sh @@ -19,8 +19,7 @@ # - log files, and what goes in 'test-suite.log' # - make clean # - dependencies between tests -# - TESTS redefinition at runtime -# - TEST_LOGS redefinition at runtime +# - TESTS redefinition at runtime (with and without test suffixes) # - RECHECK_LOGS redefinition at runtime am_parallel_tests=yes @@ -37,19 +36,14 @@ foo.log: bar.log bar.log: baz.log END -# foo.test and bar.test sleep to ensure their logs are always strictly newer -# than the logs of their prerequisites, for HP-UX make. The quoting pleases -# maintainer-check. cat > foo.test <<'END' #! /bin/sh echo "this is $0" -sleep '1' exit 0 END cat > bar.test <<'END' #! /bin/sh echo "this is $0" -sleep '1' exit 99 END cat > baz.test <<'END' @@ -85,9 +79,9 @@ test ! -f test-suite.log # Check dependencies: baz.test needs to run before bar.test, # but foo.test is not needed. -# Note that this usage has a problem: the summary will only -# take bar.log into account, because the $(TEST_SUITE_LOG) rule -# does not "see" baz.log. Hmm. +# FIXME: Note that this usage has a problem: the summary will only +# FIXME: take bar.log into account, because the $(TEST_SUITE_LOG) +# FIXME: rule does not "see" baz.log. Hmm. $MAKE check TESTS=bar.test >stdout && { cat stdout; Exit 1; } cat stdout grep '^FAIL: baz\.test$' stdout @@ -136,7 +130,7 @@ grep '^# FAIL: *1$' stdout grep '^# ERROR: *1$' stdout $MAKE clean -$MAKE check TEST_LOGS=baz.log > stdout && { cat stdout; Exit 1; } +$MAKE check TESTS=baz > stdout && { cat stdout; Exit 1; } cat stdout grep foo.test stdout && Exit 1 grep bar.test stdout && Exit 1 diff --git a/t/serial-tests.sh b/t/serial-tests.sh index 2661afa04..7e406e72b 100755 --- a/t/serial-tests.sh +++ b/t/serial-tests.sh @@ -23,7 +23,7 @@ am_create_testdir=empty hasnt_parallel_tests () { - $EGREP 'TEST_SUITE_LOG|TEST_LOGS|\.log.*:' $1 && Exit 1 + $EGREP 'TEST_SUITE_LOG|TEST_(LOGS|BASES)|\.log.*:' $1 && Exit 1 grep 'recheck.*:' $1 && Exit 1 grep '^check-TESTS: \$(TESTS)$' $1 } @@ -32,7 +32,7 @@ has_parallel_tests () { $EGREP '(^| )check-TESTS.*:' $1 $EGREP '(^| )recheck.*:' $1 - $EGREP '^\$\(TEST_SUITE_LOG\):.* \$\(TEST_LOGS\)( |$)' $1 + $EGREP '^\$\(TEST_SUITE_LOG\):.* \$\(am__TEST_LOGS\)( |$)' $1 grep '^%.log %.trs *:.*%\.test' $1 } diff --git a/t/tap-basic.sh b/t/tap-basic.sh index 417c24817..a7fcd0788 100755 --- a/t/tap-basic.sh +++ b/t/tap-basic.sh @@ -18,7 +18,7 @@ # - LOG_COMPILER support; # - basic support for TODO and SKIP directives, and "Bail out!" magic; # - testsuite progress output on console; -# - runtime overriding of TESTS and TEST_LOGS; +# - runtime overriding of TESTS; # - correct counts of test results (both in summary and in progress # output on console). # Note that some of the features checked here are checked in other @@ -144,12 +144,12 @@ grep '^PASS:' stdout && Exit 1 test `$FGREP -c ': bail.test' stdout` -eq 1 $FGREP 'success.test' stdout && Exit 1 -# Override TEST_LOGS from the command line, making it point to a test +# Override TESTS from the command line, making it point to a test # (ok.test) that has to be generated at make time. rm -f *.log *.test -$MAKE TEST_LOGS=ok.log check >stdout || { cat stdout; Exit 1; } +$MAKE TESTS=ok check >stdout || { cat stdout; Exit 1; } cat stdout count_test_results total=3 pass=1 fail=0 xpass=0 xfail=1 skip=1 error=0 diff --git a/t/test-driver-custom-multitest-recheck2.sh b/t/test-driver-custom-multitest-recheck2.sh index eef1043fa..1387f129e 100755 --- a/t/test-driver-custom-multitest-recheck2.sh +++ b/t/test-driver-custom-multitest-recheck2.sh @@ -16,8 +16,7 @@ # Custom test drivers: try the "recheck" functionality with test protocols # that allow multiple testcases in a single test script. In particular, -# check that this still works when we override $(TESTS) and $(TEST_LOGS) -# at make runtime. +# check that this still works when we override $(TESTS) at make runtime. # See also related tests 'test-driver-custom-multitest-recheck.test' and # 'parallel-tests-recheck-override.test'. @@ -94,15 +93,13 @@ for vpath in : false; do rm -f *.run - : An empty '$(TESTS)' or '$(TEST_LOGS)' means that no test should be run. - for var in TESTS TEST_LOGS; do - $MAKE "$var=" recheck >stdout || { cat stdout; Exit 1; } - cat stdout - count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0 - test ! -r a.run - test ! -r b.run - test ! -r c.run - done + : An empty '$(TESTS)' means that no test should be run. + $MAKE TESTS= recheck >stdout || { cat stdout; Exit 1; } + cat stdout + count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0 + test ! -r a.run + test ! -r b.run + test ! -r c.run unset var : a.test was successful the first time, no need to re-run it. @@ -115,7 +112,7 @@ for vpath in : false; do : b.test failed, it should be re-run. And make it pass this time. echo OK > b.ok - $MAKE TEST_LOGS=b.log recheck >stdout || { cat stdout; Exit 1; } + $MAKE TESTS=b recheck >stdout || { cat stdout; Exit 1; } cat stdout test ! -r a.run test -f b.run @@ -125,7 +122,7 @@ for vpath in : false; do rm -f *.run : No need to re-run a.test or b.test anymore. - $MAKE TEST_LOGS=b.log recheck >stdout || { cat stdout; Exit 1; } + $MAKE TESTS=b recheck >stdout || { cat stdout; Exit 1; } cat stdout count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0 test ! -r a.run @@ -144,7 +141,7 @@ for vpath in : false; do # Use 'echo' here, since Solaris 10 /bin/sh would try to optimize # a ':' away after the first iteration, even if it is redirected. echo dummy > c.err - $MAKE TEST_LOGS='a.log c.log' recheck >stdout && { cat stdout; Exit 1; } + $MAKE TESTS='a.test c' recheck >stdout && { cat stdout; Exit 1; } cat stdout count_test_results total=1 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=1 test ! -r a.run diff --git a/t/test-missing.sh b/t/test-missing.sh index 1547962e6..7ede22425 100755 --- a/t/test-missing.sh +++ b/t/test-missing.sh @@ -14,9 +14,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# parallel-tests: -# - non-existent scripts listed in TESTS get diagnosed -# See also related test 'test-missing2.test'. +# parallel-tests: non-existent scripts listed in TESTS get diagnosed. am_parallel_tests=yes . ./defs || Exit 1 @@ -50,7 +48,7 @@ cat stderr $FGREP 'zardoz2.log' stderr test ! -f test-suite.log -$MAKE TEST_LOGS='zardoz3.log' check 2>stderr && { cat stderr >&2; Exit 1; } +$MAKE TESTS='zardoz3' check 2>stderr && { cat stderr >&2; Exit 1; } cat stderr >&2 $FGREP 'zardoz3.log' stderr test ! -f test-suite.log diff --git a/t/test-trs-basic.sh b/t/test-trs-basic.sh index c98840028..4b5e19899 100755 --- a/t/test-trs-basic.sh +++ b/t/test-trs-basic.sh @@ -114,7 +114,7 @@ test -f unrelated.trs test -f sub/foo.trs # -# Try with a subset of TESTS. +# Try with a subset of TESTS, complete with test extensions. # $MAKE TESTS=foo.test check @@ -135,22 +135,22 @@ test ! -f bar.trs $MAKE clean # -# Try with a subset of TEST_LOGS. +# Try with a subset of TESTS, omitting the test extensions. # -$MAKE TEST_LOGS=sub/zardoz.log check +$MAKE TESTS=sub/zardoz check test ! -f foo.trs test ! -f bar.trs test -f sub/zardoz.trs $MAKE clean test ! -f sub/zardoz.trs -$MAKE TEST_LOGS='foo.log bar.log' check +$MAKE TESTS='foo bar' check test -f foo.trs test -f bar.trs test ! -f sub/zardoz.trs -# "make clean" shouldn't remove '.trs' files for tests whose log -# is not in $(TEST_LOGS). -$MAKE TEST_LOGS=foo.log clean +# "make clean" shouldn't remove '.trs' files for tests that are not +# in is not in $(TESTS). +$MAKE TESTS=foo clean test ! -f foo.trs test -f bar.trs test ! -f sub/zardoz.trs diff --git a/t/test-trs-recover.sh b/t/test-trs-recover.sh index 8acd270c0..de3b77191 100755 --- a/t/test-trs-recover.sh +++ b/t/test-trs-recover.sh @@ -102,9 +102,9 @@ test -f foo.trs test ! -f bar.trs test ! -f baz.trs -: Recreate with a "make check" with redefined TEST_LOGS. +: Recreate with a "make check" with redefined suffix-less TESTS. rm -f foo.trs bar.trs baz.trs -$MAKE TEST_LOGS=bar.log check +$MAKE TESTS=bar check test ! -f foo.trs test -f bar.trs test ! -f baz.trs