From 53b45c7187a1031f9c07cbd93e6e0a8c90e579bd Mon Sep 17 00:00:00 2001 From: Stefano Lattarini Date: Fri, 20 Apr 2012 21:04:36 +0200 Subject: [PATCH] [ng] parallel-tests: simplify automake-time preprocessing Our new pattern rules, introduced in recent commit 'v1.11b-124-gaeaba3f', are smart enough to allow us to ditch most of our cumbersome automake-time analysis and rewriting of the $(TESTS) variable when the parallel-tests driver is used. This also makes other related simplifications possible. This change causes a little regression in that we won't be able anymore to diagnose at automake time test cases that start with the '$(srcdir)' or '$(top_srcdir)' component, but that's acceptable, since a later commit will make the former case actually work, and the second case isn't important enough to worry to much about. This change also causes a small behavioural change in automake, in that it will also need the 'LOG_DRIVER' variable to be defined in each Makefile.am defining the 'TESTS' variable; and if it's not defined, automake will define it itself, and require the presence of the 'test-driver' auxiliary script in the process (or bing it in if the "--add-missing" option is in use). See the changes to 'NG-NEWS' for more details, and for workarounds. * automake.in (handle_tests): Don't walk nor rewrite $(TESTS) when the 'parallel-tests' option is active. Other related minor simplifications. * t/exeext4.sh: Define '$am_parallel_tests' to "no", so that the test won't be run with the 'parallel-tests' option enabled (as the semantics in that case have been changed in a way that would make this test fail). * t/check7.sh: When '$am_parallel_tests' is "yes", skip checks that don't work anymore with the 'parallel-tests' option is enabled. * t/test-driver-custom-no-extra-driver.sh: Adjust to avoid spurious failures to to the "LOG_DRIVER must be defined somehow" semantic change. * t/tap-common-setup.sh: Likewise. * t/tap-recheck.sh: Likewise. * t/tap-basic.sh: Likewise. * t/tap-diagnostic-custom.sh: Likewise. * t/tap-more.sh: Likewise. * t/tap-recheck.sh: Likewise. * t/tap-signal.tap: Likewise. * t/test-driver-custom-multitest.sh: Likewise. * t/test-driver-custom-multitest-recheck.sh: Likewise. * t/test-driver-custom-multitest-recheck2.sh: Likewise. * t/test-driver-custom-xfail-tests.sh: Likewise. * t/test-driver-fail.sh: Likewise. * t/test-driver-strip-vpath.sh: Likewise. * t/test-metadata-global-log.sh: Likewise. * t/test-metadata-global-results.sh: Likewise, and enhanced a little since we are at it. * t/test-metadata-results.sh: Likewise. * t/tap-bad-prog.tap: Likewise. * t/parallel-tests8.sh: Move the now-failing checks verifying that automake diagnoses test cases that start with the '$(srcdir)' or '$(top_srcdir)'... * t/parallel-tests-srcdir-in-test-name.sh: ... this new xfailing test. * t/spy-pattern-rules.sh: New "spy" test, verify our new assumptions on the precedence of pattern rules truly hold. * t/parallel-tests-longest-stem.sh: New test. * t/list-of-tests.mk: Update. * doc/automake.texi: Document that $(TESTS) is rewritten for $(EXEEXT) appending only when the 'parallel-tests' option is *not* in use. Add a "FIXME" comment about the fact that automake does not diagnose test cases that start with the '$(srcdir)' or '$(top_srcdir)' anymore. * NG-NEWS: Update. Signed-off-by: Stefano Lattarini --- NG-NEWS | 22 +++++ automake.in | 108 +++++++-------------- doc/automake.texi | 6 +- t/check7.sh | 9 +- t/exeext4.sh | 1 + t/list-of-tests.mk | 4 + t/parallel-tests-longest-stem.sh | 66 +++++++++++++ t/parallel-tests-srcdir-in-test-name.sh | 41 ++++++++ t/parallel-tests8.sh | 18 +--- t/spy-pattern-rules.sh | 54 +++++++++++ t/tap-bad-prog.tap | 13 +-- t/tap-basic.sh | 2 + t/tap-common-setup.sh | 8 +- t/tap-diagnostic-custom.sh | 1 + t/tap-more.sh | 3 +- t/tap-recheck.sh | 3 +- t/tap-signal.tap | 2 +- t/test-driver-custom-multitest-recheck.sh | 2 +- t/test-driver-custom-multitest-recheck2.sh | 2 +- t/test-driver-custom-multitest.sh | 2 + t/test-driver-custom-no-extra-driver.sh | 5 +- t/test-driver-custom-xfail-tests.sh | 2 +- t/test-driver-fail.sh | 8 +- t/test-driver-strip-vpath.sh | 11 ++- t/test-metadata-global-log.sh | 1 + t/test-metadata-global-result.sh | 45 ++++++++- t/test-metadata-results.sh | 25 ++--- 27 files changed, 334 insertions(+), 130 deletions(-) create mode 100755 t/parallel-tests-longest-stem.sh create mode 100755 t/parallel-tests-srcdir-in-test-name.sh create mode 100755 t/spy-pattern-rules.sh diff --git a/NG-NEWS b/NG-NEWS index 152feccc9..62c666af3 100644 --- a/NG-NEWS +++ b/NG-NEWS @@ -71,6 +71,28 @@ Parallel testsuite harness variables anymore. These have always been meant for internal use only anyway. +* The $(TESTS) and $(XFAIL_TESTS) variables are not anymore rewritten for + $(EXEEXT) appending. The use of compiled programs in $(TESTS) still + works as before though, even on systems where $(EXEEXT) is non-empty, + thanks to a careful use of pattern rules. + +* The new implementation of the parallel-tests harness tries to defer + as much as possible processing on the $(TESTS) variable at make, + rather than doing it at automake time. This as some positive effects + (FIXME: describe and quantify them), but also entails a minor annoyance. + When automake runs, it cannot know anymore whether $(TESTS) contains + or not any entry with no recognized test suffix; as a consequence, it + always need the '$(LOG_DRIVER)' variable to be defined. If it's not, + automake will define it to a proper invocation of the 'test-driver' + auxiliary script, and require that script to be present in the + config-aux directory (or bring it in if "--add-missing" is in use). + If you know you package won't make use of suffix-less tests (thus, in + particular, no use of compiled programs in $(TESTS)), you can avoid + requiring the 'test-driver' script by adding this line to your + configure.ac: + AC_SUBST([LOG_DRIVER], ['unused but required by automake']) + + Miscellaneous ============= diff --git a/automake.in b/automake.in index 64ec53d16..6b9cb9454 100644 --- a/automake.in +++ b/automake.in @@ -4781,27 +4781,12 @@ sub handle_tests PARALLEL_TESTS => !! option 'parallel-tests', CHECK_DEPS => $check_deps); - # Tests that are known programs should have $(EXEEXT) appended. - # For matching purposes, we need to adjust XFAIL_TESTS as well. - append_exeext { exists $known_programs{$_[0]} } 'TESTS'; - append_exeext { exists $known_programs{$_[0]} } 'XFAIL_TESTS' - if (var ('XFAIL_TESTS')); - if (my $parallel_tests = option 'parallel-tests') { define_variable ('TEST_SUITE_LOG', 'test-suite.log', INTERNAL); my $suff = '.test'; - my $at_exeext = ''; - my $handle_exeext = exists $configure_vars{'EXEEXT'}; - if ($handle_exeext) - { - $at_exeext = subst ('EXEEXT'); - $suff = $at_exeext . ' ' . $suff; - } - if (! var 'TEST_EXTENSIONS') - { - define_variable ('TEST_EXTENSIONS', $suff, INTERNAL); - } + define_variable ('TEST_EXTENSIONS', '.test', INTERNAL) + if (! var 'TEST_EXTENSIONS'); my $var = var 'TEST_EXTENSIONS'; # Currently, we are not able to deal with conditional contents # in TEST_EXTENSIONS. @@ -4818,68 +4803,47 @@ sub handle_tests "invalid test extensions: @invalid_test_suffixes"; } @test_suffixes = grep { is_valid_test_extension $_ } @test_suffixes; - if ($handle_exeext) - { - unshift (@test_suffixes, $at_exeext) - unless $test_suffixes[0] eq $at_exeext; - } - unshift (@test_suffixes, ''); - - transform_variable_recursively - ('TESTS', 'TEST_LOGS', 'am__testlogs', 1, INTERNAL, - sub { - my ($subvar, $val, $cond, $full_cond) = @_; - my $obj = $val; - return $obj - if $val =~ /^\@.*\@$/; - $obj =~ s/\$\(EXEEXT\)$//o; - - if ($val =~ /(\$\((top_)?srcdir\))\//o) - { - msg ('error', $subvar->rdef ($cond)->location, - "parallel-tests: using '$1' in TESTS is currently broken: '$val'"); - } - - foreach my $test_suffix (@test_suffixes) - { - next - if $test_suffix eq $at_exeext || $test_suffix eq ''; - return substr ($obj, 0, length ($obj) - length ($test_suffix)) . '.log' - if substr ($obj, - length ($test_suffix)) eq $test_suffix; - } - handle_per_suffix_test (''); - return "$obj.log"; - }); - - my $nhelper=1; - my $prev = 'TESTS'; - my $post = ''; - my $last_suffix = $test_suffixes[$#test_suffixes]; - my $cur = ''; + # It is *imperative* that the empty 'suffix goes last. + # Otherwise, a declaration like "TESTS = all.test" would cause GNU + # make to mistakenly try to build the 'all.log' and 'all.trs' files + # from a non-existent 'all' program (because the Makefile contains + # an explicit 'all' target) rather than from the 'all.test' script, + # thus causing all sort of mishaps and confusion. + push @test_suffixes, ''; + + define_variable ('am__test_logs1', + '$(patsubst %,%.log,$(strip $(TESTS)))', + INTERNAL); + my $nhelper = 1; foreach my $test_suffix (@test_suffixes) { - if ($test_suffix eq $last_suffix) - { - $cur = 'TEST_LOGS'; - } - else - { - $cur = 'am__test_logs' . $nhelper; - } - define_variable ($cur, - '$(' . $prev . ':' . $test_suffix . $post . '=.log)', INTERNAL); - $post = '.log'; - $prev = $cur; - $nhelper++; - if ($test_suffix ne $at_exeext && $test_suffix ne '') - { - handle_per_suffix_test ($test_suffix); - } + handle_per_suffix_test ($test_suffix); + my @cooked_test_suffixes = ($test_suffix); + push @cooked_test_suffixes, ($test_suffix . '$(EXEEXT)') + if exists $configure_vars{'EXEEXT'}; + foreach my $cooked_test_suffix (@cooked_test_suffixes) + { + my $old_var = 'am__test_logs' . $nhelper++; + my $cur_var = 'am__test_logs' . $nhelper; + define_variable ( + $cur_var, + "\$($old_var:$cooked_test_suffix.log=.log)", + INTERNAL); + } } + define_variable ('TEST_LOGS', "\$(am__test_logs$nhelper)", INTERNAL); $clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN; $clean_files{'$(TEST_LOGS:.log=.trs)'} = MOSTLY_CLEAN; $clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN; } + else + { + # Tests that are known programs should have $(EXEEXT) appended. + # For matching purposes, we need to adjust XFAIL_TESTS as well. + append_exeext { exists $known_programs{$_[0]} } 'TESTS'; + append_exeext { exists $known_programs{$_[0]} } 'XFAIL_TESTS' + if (var ('XFAIL_TESTS')); + } } } diff --git a/doc/automake.texi b/doc/automake.texi index 2661af537..d1e076547 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -6938,8 +6938,9 @@ bin_PROGRAMS = liver$(EXEEXT) The targets Automake generates are likewise given the @samp{$(EXEEXT)} extension. -The variables @code{TESTS} and @code{XFAIL_TESTS} (@pxref{Simple Tests}) -are also rewritten if they contain filenames that have been declared as +When the @option{parallel-tests} option is in use, the variables +@code{TESTS} and @code{XFAIL_TESTS} (@pxref{Simple Tests}) are also +rewritten if they contain filenames that have been declared as programs in the same @file{Makefile}. (This is mostly useful when some programs from @code{check_PROGRAMS} are listed in @code{TESTS}.) @@ -9202,6 +9203,7 @@ the inference rules generated by @command{automake} can apply. For literal test names, @command{automake} can generate per-target rules to avoid this limitation. +@c FIXME: Automake does not warn about these anymore! Please note that it is currently not possible to use @code{$(srcdir)/} or @code{$(top_srcdir)/} in the @code{TESTS} variable. This technical limitation is necessary to avoid generating test logs in the source tree. diff --git a/t/check7.sh b/t/check7.sh index 1003b5507..962cbe082 100755 --- a/t/check7.sh +++ b/t/check7.sh @@ -59,9 +59,12 @@ $AUTOMAKE -a ./configure $MAKE check -$MAKE EXEEXT=.bin print-xfail-tests >stdout || { cat stdout; Exit 1; } -cat stdout -$FGREP 'BEG: a.bin b c.bin d.bin :END' stdout + +if test x"$am_parallel_tests" != x"yes"; then + $MAKE EXEEXT=.bin print-xfail-tests >stdout || { cat stdout; Exit 1; } + cat stdout + $FGREP 'BEG: a.bin b c.bin d.bin :END' stdout +fi $MAKE distcheck diff --git a/t/exeext4.sh b/t/exeext4.sh index 298879dee..53c37dfe6 100755 --- a/t/exeext4.sh +++ b/t/exeext4.sh @@ -17,6 +17,7 @@ # Make sure $(EXEEXT) is appended to programs and to tests that are # programs, but not to @substitutions@. +am_parallel_tests=no . ./defs || Exit 1 cat >> configure.ac << 'END' diff --git a/t/list-of-tests.mk b/t/list-of-tests.mk index 436e401b9..7d12d0f89 100644 --- a/t/list-of-tests.mk +++ b/t/list-of-tests.mk @@ -40,6 +40,7 @@ t/instdir-cond2.sh \ t/interp3.sh \ t/java-nobase.sh \ t/objext-pr10128.sh \ +t/parallel-tests-srcdir-in-test-name.sh \ t/parallel-tests-many.sh \ t/pr8365-remake-timing.sh \ t/remake-am-pr10111.sh \ @@ -738,6 +739,8 @@ t/parallel-tests8.sh \ t/parallel-tests9.sh \ t/parallel-tests10.sh \ t/parallel-tests-no-repeat.sh \ +t/parallel-tests-longest-stem.sh \ +t/parallel-tests-srcdir-in-test-name.sh \ t/parallel-tests-recheck-depends-on-all.sh \ t/parallel-tests-exeext.sh \ t/parallel-tests-suffix.sh \ @@ -981,6 +984,7 @@ t/spell3.sh \ t/spelling.sh \ t/spy-autovars.sh \ t/spy-double-colon.sh \ +t/spy-pattern-rules.sh \ t/spy-phony.sh \ t/spy-rm.tap \ t/spy-vpath-rewrite.sh \ diff --git a/t/parallel-tests-longest-stem.sh b/t/parallel-tests-longest-stem.sh new file mode 100755 index 000000000..242d78fd6 --- /dev/null +++ b/t/parallel-tests-longest-stem.sh @@ -0,0 +1,66 @@ +#! /bin/sh +# Copyright (C) 2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# The parallel-tests driver must prefer tests with an extension to +# extension-less tests. This is required to allow the user to have +# a, say, 'all.test' test case even in the face of the 'all' target. + +am_parallel_tests=yes +. ./defs || Exit 1 + +echo AC_OUTPUT >> configure.ac + +cat > foo <<'END' +#!/bin/sh +echo "foo without suffix run" >&2 +exit 99 +END + +cat > foo.test <<'END' +#!/bin/sh +echo "$0 has been run" +END +chmod a+x foo.test + +cp foo.test all.test +cp foo.test dist.test +cp foo.test install.test +cp foo.test bad-target.test + + +cat > Makefile.am << 'END' +bad-target: + @echo $@ has been run >&2; exit 1 +install-data-local: + @echo $@ has been run >&2; exit 1 +TESTS = foo.test all.test install.test dist.test bad-target.test +EXTRA_DIST = oops-this-does-not-exist +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure --prefix="`pwd`/inst" + +$MAKE check +ls -l # For debugging. +test ! -d inst +for t in foo all install dist bad-target; do + grep "$t\.test has been run" $t.log +done + +: diff --git a/t/parallel-tests-srcdir-in-test-name.sh b/t/parallel-tests-srcdir-in-test-name.sh new file mode 100755 index 000000000..01042a535 --- /dev/null +++ b/t/parallel-tests-srcdir-in-test-name.sh @@ -0,0 +1,41 @@ +#! /bin/sh +# Copyright (C) 2009-2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check parallel-tests features: +# - listing $(srcdir)/ or $(top_srcdir)/ in TESTS doesn't work ATM, +# and is thus diagnosed. + +# TODO: this test should also ensure that the 'make' implementation +# properly adheres to rules in all cases. See the Autoconf +# manual for the ugliness in this area, when VPATH comes into +# play. :-/ + +am_parallel_tests=yes +. ./defs || Exit 1 + +echo AC_OUTPUT >> configure.ac + +cat > Makefile.am << 'END' +TESTS = $(srcdir)/bar.test $(top_srcdir)/baz.test +END + +$ACLOCAL +$AUTOCONF +AUTOMAKE_fails -a +grep '$(srcdir).*TESTS.*bar\.test' stderr +grep '$(top_srcdir).*TESTS.*baz\.test' stderr + +: diff --git a/t/parallel-tests8.sh b/t/parallel-tests8.sh index 69efc5124..a36cea76f 100755 --- a/t/parallel-tests8.sh +++ b/t/parallel-tests8.sh @@ -14,15 +14,7 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# Check parallel-tests features: -# - generated distributed tests. -# - listing $(srcdir)/ or $(top_srcdir)/ in TESTS doesn't work ATM, -# and is thus diagnosed. - -# TODO: this test should also ensure that the 'make' implementation -# properly adheres to rules in all cases. See the Autoconf -# manual for the ugliness in this area, when VPATH comes into -# play. :-/ +# Check parallel-tests features: generated distributed tests. am_parallel_tests=yes . ./defs || Exit 1 @@ -33,8 +25,6 @@ END cat > Makefile.am << 'END' TESTS = foo.test -## the next line will cause automake to error out: -TESTS += $(srcdir)/bar.test $(top_srcdir)/baz.test .in.test: cp $< $@ && chmod +x $@ check_SCRIPTS = $(TESTS) @@ -50,12 +40,6 @@ END $ACLOCAL $AUTOCONF -AUTOMAKE_fails -a -grep '$(srcdir).*TESTS.*bar\.test' stderr -grep '$(top_srcdir).*TESTS.*baz\.test' stderr - -sed '/^TESTS +=.*srcdir/d' < Makefile.am > t -mv -f t Makefile.am $AUTOMAKE -a ./configure diff --git a/t/spy-pattern-rules.sh b/t/spy-pattern-rules.sh new file mode 100755 index 000000000..b2bb25d96 --- /dev/null +++ b/t/spy-pattern-rules.sh @@ -0,0 +1,54 @@ +#! /bin/sh +# Copyright (C) 2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check that, if we have two pattern rules from which the same file (or +# set of files) can be built, and both are applicable, the first one wins. +# This is used at least in our 'parallel-tests' support. + +am_create_testdir=empty +. ./defs || Exit 1 + +cat > Makefile <<'END' +default: + +%.foo: % + cp $< $@ +%.foo: %.x + cp $< $@ + +%.bar: %.x + cp $< $@ +%.bar: % + +%.mu %.fu: %.1 + cp $< $*.mu && cp $< $*.fu +%.mu %.fu: %.2 + cp $< $*.mu && cp $< $*.fu +END + +echo one > all +echo two > all.x +$MAKE all.foo all.bar +diff all all.foo +diff all.x all.bar + +echo one > x.1 +echo two > x.2 +$MAKE x.mu +diff x.mu x.1 +diff x.fu x.1 + +: diff --git a/t/tap-bad-prog.tap b/t/tap-bad-prog.tap index 580e7d31c..5eb1c56b8 100755 --- a/t/tap-bad-prog.tap +++ b/t/tap-bad-prog.tap @@ -30,8 +30,9 @@ AC_OUTPUT END cat > Makefile.am << 'END' -TEST_LOG_DRIVER = $(srcdir)/tap-driver -TESTS = none.test noread.test noexec.test +LOG_DRIVER = $(srcdir)/tap-driver +TEST_LOG_DRIVER = $(LOG_DRIVER) +TESTS = none.test noread noexec.test none.test: END @@ -41,8 +42,8 @@ echo 1..1 echo ok 1 END -cp noexec.test noread.test -chmod a-r noread.test +cp noexec.test noread +chmod a-r noread $ACLOCAL $AUTOCONF @@ -72,10 +73,10 @@ else fi desc="non-readable test is reported" -if test -r noread.test; then +if test -r noread; then skip_ -r "any file is readable" "$desc" else - command_ok_ "$desc" -- grep '^ERROR: noread\.test' stdout + command_ok_ "$desc" -- grep '^ERROR: noread' stdout fi # Check that no spurious test result is reported. This is lower-priority diff --git a/t/tap-basic.sh b/t/tap-basic.sh index 96578e59d..417c24817 100755 --- a/t/tap-basic.sh +++ b/t/tap-basic.sh @@ -47,6 +47,8 @@ ok.test: echo 'ok 3 # SKIP' >>$@-t cat $@-t ;: For debugging. mv -f $@-t $@ + +LOG_DRIVER = false # Dummy but required. END cat > success.test << 'END' diff --git a/t/tap-common-setup.sh b/t/tap-common-setup.sh index b14099a5f..cd0a14d62 100755 --- a/t/tap-common-setup.sh +++ b/t/tap-common-setup.sh @@ -19,7 +19,12 @@ am_parallel_tests=yes . ./defs || Exit 1 -cat >> configure.ac << END +cat >> configure.ac << 'END' +# FIXME: must define this otherwise automake will require the presence +# FIXME: of the 'test-driver' script. This issue should be documented +# FIXME: in the manual ... +AC_SUBST([LOG_DRIVER], + ['$(error LOG_DRIVER should be never used) false']) AC_OUTPUT END @@ -32,6 +37,7 @@ END $ACLOCAL $AUTOCONF $AUTOMAKE +test ! -f test-driver ./configure diff --git a/t/tap-diagnostic-custom.sh b/t/tap-diagnostic-custom.sh index d3f28a577..bb189fd03 100755 --- a/t/tap-diagnostic-custom.sh +++ b/t/tap-diagnostic-custom.sh @@ -32,6 +32,7 @@ my_log_driver = $(srcdir)/tap-driver my_log_compiler = cat TEST_EXTENSIONS = TESTS = +LOG_DRIVER = $(error LOG_DRIVER is dummy, required, never extended) END : > later.mk diff --git a/t/tap-more.sh b/t/tap-more.sh index 00abbf28f..14181b646 100755 --- a/t/tap-more.sh +++ b/t/tap-more.sh @@ -29,7 +29,8 @@ am_parallel_tests=yes fetch_tap_driver -cat >> configure.ac <> configure.ac << 'END' +AC_SUBST([LOG_DRIVER], ["dummy but required"]) AC_SUBST([AM_TEST_LOG_DRIVER_FLAGS], ['--comments']) AC_OUTPUT END diff --git a/t/tap-recheck.sh b/t/tap-recheck.sh index 4b120a529..a2af0676c 100755 --- a/t/tap-recheck.sh +++ b/t/tap-recheck.sh @@ -22,11 +22,12 @@ am_parallel_tests=yes fetch_tap_driver -cat >> configure.ac <> configure.ac << 'END' AC_OUTPUT END cat > Makefile.am << 'END' +LOG_DRIVER = $(error LOG_DRIVER should never be expanded) false TEST_LOG_DRIVER = $(srcdir)/tap-driver TESTS = a.test b.test c.test d.test END diff --git a/t/tap-signal.tap b/t/tap-signal.tap index ba17bd402..867b762cf 100755 --- a/t/tap-signal.tap +++ b/t/tap-signal.tap @@ -67,7 +67,7 @@ chmod a+x *.test $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a ./configure diff --git a/t/test-driver-custom-multitest-recheck.sh b/t/test-driver-custom-multitest-recheck.sh index 9b4f5dfa3..af3d8518a 100755 --- a/t/test-driver-custom-multitest-recheck.sh +++ b/t/test-driver-custom-multitest-recheck.sh @@ -82,7 +82,7 @@ chmod a+x *.test $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a do_recheck () { diff --git a/t/test-driver-custom-multitest-recheck2.sh b/t/test-driver-custom-multitest-recheck2.sh index cf97135e3..eef1043fa 100755 --- a/t/test-driver-custom-multitest-recheck2.sh +++ b/t/test-driver-custom-multitest-recheck2.sh @@ -70,7 +70,7 @@ chmod a+x *.test $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a for vpath in : false; do if $vpath; then diff --git a/t/test-driver-custom-multitest.sh b/t/test-driver-custom-multitest.sh index d5e2cf56b..caf6a079a 100755 --- a/t/test-driver-custom-multitest.sh +++ b/t/test-driver-custom-multitest.sh @@ -30,6 +30,8 @@ AC_OUTPUT END cat > Makefile.am << 'END' +LOG_DRIVER = dummy but required + TEST_EXTENSIONS = .t T_LOG_DRIVER = $(SHELL) $(srcdir)/trivial-test-driver diff --git a/t/test-driver-custom-no-extra-driver.sh b/t/test-driver-custom-no-extra-driver.sh index 31000aff1..5f66f6dde 100755 --- a/t/test-driver-custom-no-extra-driver.sh +++ b/t/test-driver-custom-no-extra-driver.sh @@ -24,7 +24,7 @@ am_parallel_tests=yes cat >> configure.ac <<'END' AC_PROG_CC AC_CONFIG_FILES([sub1/Makefile sub2/Makefile]) -AC_SUBST([X_LOG_DRIVER], [none]) +AC_SUBST([LOG_DRIVER], ['unused but required by automake']) AC_OUTPUT END @@ -32,7 +32,6 @@ mkdir sub1 sub2 cat > Makefile.am <<'END' SUBDIRS = sub1 sub2 -LOG_DRIVER = TEST_LOG_DRIVER = : TESTS = foo bar.test END @@ -41,12 +40,12 @@ cat > sub1/Makefile.am <<'END' TEST_EXTENSIONS = .x .sh .pl SH_LOG_DRIVER = dummy1 PL_LOG_DRIVER = dummy2 +X_LOG_DRIVER = dummy3 TESTS = a.pl b.sh c.x END cat > sub2/Makefile.am <<'END' TEST_EXTENSIONS = .bar -LOG_DRIVER = x BAR_LOG_DRIVER = y TESTS = 1 2.bar 3.test 4.t 5.tt $(check_PROGRAMS) check_PROGRAMS = p1 p2$(EXEEXT) p3.bar p4.suf diff --git a/t/test-driver-custom-xfail-tests.sh b/t/test-driver-custom-xfail-tests.sh index 745772fb3..d883b73e3 100755 --- a/t/test-driver-custom-xfail-tests.sh +++ b/t/test-driver-custom-xfail-tests.sh @@ -136,7 +136,7 @@ chmod a+x td $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE --add-missing ./configure diff --git a/t/test-driver-fail.sh b/t/test-driver-fail.sh index 0eb226800..f1c3df80f 100755 --- a/t/test-driver-fail.sh +++ b/t/test-driver-fail.sh @@ -30,14 +30,16 @@ AC_OUTPUT END cat > Makefile.am <<'END' -TEST_LOG_DRIVER = ./oops -TESTS = foo.test +TEST_EXTENSIONS = +LOG_DRIVER = ./oops +TESTS = foo END -cat > foo.test <<'END' +cat > foo <<'END' #! /bin/sh exit 0 END +chmod a+x foo $ACLOCAL $AUTOCONF diff --git a/t/test-driver-strip-vpath.sh b/t/test-driver-strip-vpath.sh index 967018854..f7adb674c 100755 --- a/t/test-driver-strip-vpath.sh +++ b/t/test-driver-strip-vpath.sh @@ -39,9 +39,10 @@ cat > Makefile.am << 'END' # The $(empty) are for eliciting VPATH rewrites on make implementations # that support it (e.g., Solaris make), to improve coverage. empty = -TESTS = $(empty) foo.test src/bar.test ./src/baz.test $(empty) +TESTS = $(empty) foo.test src/bar.test ./src/baz $(empty) $(TESTS): -TEST_LOG_DRIVER = $(srcdir)/checkstrip-driver +LOG_DRIVER = $(srcdir)/checkstrip-driver +TEST_LOG_DRIVER = $(LOG_DRIVER) EXTRA_DIST = checkstrip-driver END @@ -61,8 +62,10 @@ while test $# -gt 0; do done echo "test name: $test_name" # For debugging. case $test_name in - foo.test|./foo.test|src/ba[rz].test|./src/ba[rz].test);; - *) exit 1;; + foo.test|./foo.test) ;; + src/bar.test|./src/bar.test) ;; + src/baz|./src/baz) ;; + *) exit 1 ;; esac echo dummy > "$log_file" echo dummy > "$trs_file" diff --git a/t/test-metadata-global-log.sh b/t/test-metadata-global-log.sh index b10f481d2..5d02e7793 100755 --- a/t/test-metadata-global-log.sh +++ b/t/test-metadata-global-log.sh @@ -24,6 +24,7 @@ am_parallel_tests=yes . ./defs || Exit 1 cat >> configure.ac << 'END' +AC_SUBST([LOG_DRIVER], ['ignored but required by autoamke, sigh!']) AC_OUTPUT END diff --git a/t/test-metadata-global-result.sh b/t/test-metadata-global-result.sh index b2474449c..9c0fd481c 100755 --- a/t/test-metadata-global-result.sh +++ b/t/test-metadata-global-result.sh @@ -29,7 +29,8 @@ cat > Makefile.am << 'END' TEST_EXTENSIONS = .test .x TEST_LOG_DRIVER = ./dummy-driver X_LOG_DRIVER = ./dummy-driver -TESTS = foo.test zar-doz.test +LOG_DRIVER = ./dummy-driver +TESTS = foo.test zar-doz.test mu END cat > dummy-driver <<'END' @@ -76,6 +77,7 @@ $AUTOMAKE : Basic checks. echo :global-test-result: PASS > foo.test +echo :global-test-result: PASS > mu echo :global-test-result: ERROR > zar-doz.x $MAKE check @@ -86,6 +88,11 @@ PASS: foo ========= END +have_result < foo.test echo :global-test-result: ALMOST PASSED > zar-doz.x +echo :global-test-result: HU? > mu $MAKE check cat test-suite.log @@ -109,6 +117,11 @@ ALMOST PASSED: zar-doz ====================== END +have_result < foo.test << 'END' @@ -123,6 +136,13 @@ cat > zar-doz.x << 'END' :test-result: XFAIL END +cat > mu << 'END' +:test-result: PASS +:global-test-result: XFAIL +:test-result: PASS +:test-result: PASS +END + $MAKE check && Exit 1 cat test-suite.log @@ -136,6 +156,11 @@ FAIL: zar-doz ============= END +have_result < foo.test << 'END' @@ -143,6 +168,7 @@ cat > foo.test << 'END' :test-result: ERROR END : > zar-doz.x +: > mu $MAKE check && Exit 1 cat test-suite.log @@ -157,10 +183,16 @@ RUN: zar-doz ============ END +have_result < foo.test echo ":global-test-result:${tab} ${tab}XFAIL ${tab} " > zar-doz.x +echo ":global-test-result: ${tab} PASS${tab} ${tab}${tab}" > mu $MAKE check cat test-suite.log @@ -175,10 +207,16 @@ XFAIL: zar-doz ============== END +have_result < foo.test echo "${tab}${tab}:global-test-result:${tab} ${tab}SKIP" > zar-doz.x +echo " :global-test-result:${tab}SKIP" > mu $MAKE check cat test-suite.log @@ -193,4 +231,9 @@ SKIP: zar-doz ============= END +have_result < Makefile.am << 'END' -TEST_LOG_DRIVER = ./dummy-driver -TESTS = foo.test bar.test +LOG_DRIVER = ./dummy-driver +TEST_LOG_DRIVER = ${LOG_DRIVER} +TESTS = foo.test bar END cat > dummy-driver <<'END' @@ -88,7 +89,7 @@ $AUTOMAKE # "old-style" directives with format "RESULT: test-name" are now ignored. : > foo.test -echo blah blah blah > bar.test +echo blah blah blah > bar mk_check count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0 @@ -96,7 +97,7 @@ cat > foo.test < bar.test < bar < bar.test +echo ERROR: bar > bar mk_check count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0 @@ -116,7 +117,7 @@ cat > foo.test < bar.test < bar < foo.test < bar.test < bar < foo.test < bar.test +: > bar mk_check && Exit 1 count_test_results total=6 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=1 -cp foo.test bar.test +cp foo.test bar mk_check && Exit 1 count_test_results total=12 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=2 # Check that we are liberal w.r.t. whitespace use. : > foo.test -: > bar.test +: > bar for RESULT in PASS FAIL XPASS XFAIL SKIP ERROR; do sed -e 's/^ *//' -e 's/|//g' >> foo.test <> bar.test + echo " $tab $tab$tab :test-result: $RESULT" >> bar done cat foo.test # For debugging. -cat bar.test # Likewise. +cat bar # Likewise. mk_check && Exit 1 count_test_results total=30 pass=5 fail=5 xpass=5 xfail=5 skip=5 error=5 -- 2.47.2