Our new pattern rules, introduced in recent commit '
v1.11b-124-gaeaba3f',
are smart enough to allow us to ditch most of our cumbersome automake-time
analysis and rewriting of the $(TESTS) variable when the parallel-tests
driver is used. This also makes other related simplifications possible.
This change causes a little regression in that we won't be able anymore to
diagnose at automake time test cases that start with the '$(srcdir)' or
'$(top_srcdir)' component, but that's acceptable, since a later commit will
make the former case actually work, and the second case isn't important
enough to worry to much about.
This change also causes a small behavioural change in automake, in that it
will also need the 'LOG_DRIVER' variable to be defined in each Makefile.am
defining the 'TESTS' variable; and if it's not defined, automake will
define it itself, and require the presence of the 'test-driver' auxiliary
script in the process (or bing it in if the "--add-missing" option is in
use). See the changes to 'NG-NEWS' for more details, and for workarounds.
* automake.in (handle_tests): Don't walk nor rewrite $(TESTS) when the
'parallel-tests' option is active. Other related minor simplifications.
* t/exeext4.sh: Define '$am_parallel_tests' to "no", so that the test
won't be run with the 'parallel-tests' option enabled (as the semantics
in that case have been changed in a way that would make this test fail).
* t/check7.sh: When '$am_parallel_tests' is "yes", skip checks that
don't work anymore with the 'parallel-tests' option is enabled.
* t/test-driver-custom-no-extra-driver.sh: Adjust to avoid spurious
failures to to the "LOG_DRIVER must be defined somehow" semantic change.
* t/tap-common-setup.sh: Likewise.
* t/tap-recheck.sh: Likewise.
* t/tap-basic.sh: Likewise.
* t/tap-diagnostic-custom.sh: Likewise.
* t/tap-more.sh: Likewise.
* t/tap-recheck.sh: Likewise.
* t/tap-signal.tap: Likewise.
* t/test-driver-custom-multitest.sh: Likewise.
* t/test-driver-custom-multitest-recheck.sh: Likewise.
* t/test-driver-custom-multitest-recheck2.sh: Likewise.
* t/test-driver-custom-xfail-tests.sh: Likewise.
* t/test-driver-fail.sh: Likewise.
* t/test-driver-strip-vpath.sh: Likewise.
* t/test-metadata-global-log.sh: Likewise.
* t/test-metadata-global-results.sh: Likewise, and enhanced a little since
we are at it.
* t/test-metadata-results.sh: Likewise.
* t/tap-bad-prog.tap: Likewise.
* t/parallel-tests8.sh: Move the now-failing checks verifying that automake
diagnoses test cases that start with the '$(srcdir)' or '$(top_srcdir)'...
* t/parallel-tests-srcdir-in-test-name.sh: ... this new xfailing test.
* t/spy-pattern-rules.sh: New "spy" test, verify our new assumptions on the
precedence of pattern rules truly hold.
* t/parallel-tests-longest-stem.sh: New test.
* t/list-of-tests.mk: Update.
* doc/automake.texi: Document that $(TESTS) is rewritten for $(EXEEXT)
appending only when the 'parallel-tests' option is *not* in use.
Add a "FIXME" comment about the fact that automake does not diagnose test
cases that start with the '$(srcdir)' or '$(top_srcdir)' anymore.
* NG-NEWS: Update.
Signed-off-by: Stefano Lattarini <stefano.lattarini@gmail.com>
variables anymore. These have always been meant for internal use only
anyway.
+* The $(TESTS) and $(XFAIL_TESTS) variables are not anymore rewritten for
+ $(EXEEXT) appending. The use of compiled programs in $(TESTS) still
+ works as before though, even on systems where $(EXEEXT) is non-empty,
+ thanks to a careful use of pattern rules.
+
+* The new implementation of the parallel-tests harness tries to defer
+ as much as possible processing on the $(TESTS) variable at make,
+ rather than doing it at automake time. This as some positive effects
+ (FIXME: describe and quantify them), but also entails a minor annoyance.
+ When automake runs, it cannot know anymore whether $(TESTS) contains
+ or not any entry with no recognized test suffix; as a consequence, it
+ always need the '$(LOG_DRIVER)' variable to be defined. If it's not,
+ automake will define it to a proper invocation of the 'test-driver'
+ auxiliary script, and require that script to be present in the
+ config-aux directory (or bring it in if "--add-missing" is in use).
+ If you know you package won't make use of suffix-less tests (thus, in
+ particular, no use of compiled programs in $(TESTS)), you can avoid
+ requiring the 'test-driver' script by adding this line to your
+ configure.ac:
+ AC_SUBST([LOG_DRIVER], ['unused but required by automake'])
+
+
Miscellaneous
=============
PARALLEL_TESTS => !! option 'parallel-tests',
CHECK_DEPS => $check_deps);
- # Tests that are known programs should have $(EXEEXT) appended.
- # For matching purposes, we need to adjust XFAIL_TESTS as well.
- append_exeext { exists $known_programs{$_[0]} } 'TESTS';
- append_exeext { exists $known_programs{$_[0]} } 'XFAIL_TESTS'
- if (var ('XFAIL_TESTS'));
-
if (my $parallel_tests = option 'parallel-tests')
{
define_variable ('TEST_SUITE_LOG', 'test-suite.log', INTERNAL);
my $suff = '.test';
- my $at_exeext = '';
- my $handle_exeext = exists $configure_vars{'EXEEXT'};
- if ($handle_exeext)
- {
- $at_exeext = subst ('EXEEXT');
- $suff = $at_exeext . ' ' . $suff;
- }
- if (! var 'TEST_EXTENSIONS')
- {
- define_variable ('TEST_EXTENSIONS', $suff, INTERNAL);
- }
+ define_variable ('TEST_EXTENSIONS', '.test', INTERNAL)
+ if (! var 'TEST_EXTENSIONS');
my $var = var 'TEST_EXTENSIONS';
# Currently, we are not able to deal with conditional contents
# in TEST_EXTENSIONS.
"invalid test extensions: @invalid_test_suffixes";
}
@test_suffixes = grep { is_valid_test_extension $_ } @test_suffixes;
- if ($handle_exeext)
- {
- unshift (@test_suffixes, $at_exeext)
- unless $test_suffixes[0] eq $at_exeext;
- }
- unshift (@test_suffixes, '');
-
- transform_variable_recursively
- ('TESTS', 'TEST_LOGS', 'am__testlogs', 1, INTERNAL,
- sub {
- my ($subvar, $val, $cond, $full_cond) = @_;
- my $obj = $val;
- return $obj
- if $val =~ /^\@.*\@$/;
- $obj =~ s/\$\(EXEEXT\)$//o;
-
- if ($val =~ /(\$\((top_)?srcdir\))\//o)
- {
- msg ('error', $subvar->rdef ($cond)->location,
- "parallel-tests: using '$1' in TESTS is currently broken: '$val'");
- }
-
- foreach my $test_suffix (@test_suffixes)
- {
- next
- if $test_suffix eq $at_exeext || $test_suffix eq '';
- return substr ($obj, 0, length ($obj) - length ($test_suffix)) . '.log'
- if substr ($obj, - length ($test_suffix)) eq $test_suffix;
- }
- handle_per_suffix_test ('');
- return "$obj.log";
- });
-
- my $nhelper=1;
- my $prev = 'TESTS';
- my $post = '';
- my $last_suffix = $test_suffixes[$#test_suffixes];
- my $cur = '';
+ # It is *imperative* that the empty 'suffix goes last.
+ # Otherwise, a declaration like "TESTS = all.test" would cause GNU
+ # make to mistakenly try to build the 'all.log' and 'all.trs' files
+ # from a non-existent 'all' program (because the Makefile contains
+ # an explicit 'all' target) rather than from the 'all.test' script,
+ # thus causing all sort of mishaps and confusion.
+ push @test_suffixes, '';
+
+ define_variable ('am__test_logs1',
+ '$(patsubst %,%.log,$(strip $(TESTS)))',
+ INTERNAL);
+ my $nhelper = 1;
foreach my $test_suffix (@test_suffixes)
{
- if ($test_suffix eq $last_suffix)
- {
- $cur = 'TEST_LOGS';
- }
- else
- {
- $cur = 'am__test_logs' . $nhelper;
- }
- define_variable ($cur,
- '$(' . $prev . ':' . $test_suffix . $post . '=.log)', INTERNAL);
- $post = '.log';
- $prev = $cur;
- $nhelper++;
- if ($test_suffix ne $at_exeext && $test_suffix ne '')
- {
- handle_per_suffix_test ($test_suffix);
- }
+ handle_per_suffix_test ($test_suffix);
+ my @cooked_test_suffixes = ($test_suffix);
+ push @cooked_test_suffixes, ($test_suffix . '$(EXEEXT)')
+ if exists $configure_vars{'EXEEXT'};
+ foreach my $cooked_test_suffix (@cooked_test_suffixes)
+ {
+ my $old_var = 'am__test_logs' . $nhelper++;
+ my $cur_var = 'am__test_logs' . $nhelper;
+ define_variable (
+ $cur_var,
+ "\$($old_var:$cooked_test_suffix.log=.log)",
+ INTERNAL);
+ }
}
+ define_variable ('TEST_LOGS', "\$(am__test_logs$nhelper)", INTERNAL);
$clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_LOGS:.log=.trs)'} = MOSTLY_CLEAN;
$clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN;
}
+ else
+ {
+ # Tests that are known programs should have $(EXEEXT) appended.
+ # For matching purposes, we need to adjust XFAIL_TESTS as well.
+ append_exeext { exists $known_programs{$_[0]} } 'TESTS';
+ append_exeext { exists $known_programs{$_[0]} } 'XFAIL_TESTS'
+ if (var ('XFAIL_TESTS'));
+ }
}
}
The targets Automake generates are likewise given the @samp{$(EXEEXT)}
extension.
-The variables @code{TESTS} and @code{XFAIL_TESTS} (@pxref{Simple Tests})
-are also rewritten if they contain filenames that have been declared as
+When the @option{parallel-tests} option is in use, the variables
+@code{TESTS} and @code{XFAIL_TESTS} (@pxref{Simple Tests}) are also
+rewritten if they contain filenames that have been declared as
programs in the same @file{Makefile}. (This is mostly useful when some
programs from @code{check_PROGRAMS} are listed in @code{TESTS}.)
literal test names, @command{automake} can generate per-target rules
to avoid this limitation.
+@c FIXME: Automake does not warn about these anymore!
Please note that it is currently not possible to use @code{$(srcdir)/}
or @code{$(top_srcdir)/} in the @code{TESTS} variable. This technical
limitation is necessary to avoid generating test logs in the source tree.
./configure
$MAKE check
-$MAKE EXEEXT=.bin print-xfail-tests >stdout || { cat stdout; Exit 1; }
-cat stdout
-$FGREP 'BEG: a.bin b c.bin d.bin :END' stdout
+
+if test x"$am_parallel_tests" != x"yes"; then
+ $MAKE EXEEXT=.bin print-xfail-tests >stdout || { cat stdout; Exit 1; }
+ cat stdout
+ $FGREP 'BEG: a.bin b c.bin d.bin :END' stdout
+fi
$MAKE distcheck
# Make sure $(EXEEXT) is appended to programs and to tests that are
# programs, but not to @substitutions@.
+am_parallel_tests=no
. ./defs || Exit 1
cat >> configure.ac << 'END'
t/interp3.sh \
t/java-nobase.sh \
t/objext-pr10128.sh \
+t/parallel-tests-srcdir-in-test-name.sh \
t/parallel-tests-many.sh \
t/pr8365-remake-timing.sh \
t/remake-am-pr10111.sh \
t/parallel-tests9.sh \
t/parallel-tests10.sh \
t/parallel-tests-no-repeat.sh \
+t/parallel-tests-longest-stem.sh \
+t/parallel-tests-srcdir-in-test-name.sh \
t/parallel-tests-recheck-depends-on-all.sh \
t/parallel-tests-exeext.sh \
t/parallel-tests-suffix.sh \
t/spelling.sh \
t/spy-autovars.sh \
t/spy-double-colon.sh \
+t/spy-pattern-rules.sh \
t/spy-phony.sh \
t/spy-rm.tap \
t/spy-vpath-rewrite.sh \
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2012 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# The parallel-tests driver must prefer tests with an extension to
+# extension-less tests. This is required to allow the user to have
+# a, say, 'all.test' test case even in the face of the 'all' target.
+
+am_parallel_tests=yes
+. ./defs || Exit 1
+
+echo AC_OUTPUT >> configure.ac
+
+cat > foo <<'END'
+#!/bin/sh
+echo "foo without suffix run" >&2
+exit 99
+END
+
+cat > foo.test <<'END'
+#!/bin/sh
+echo "$0 has been run"
+END
+chmod a+x foo.test
+
+cp foo.test all.test
+cp foo.test dist.test
+cp foo.test install.test
+cp foo.test bad-target.test
+
+
+cat > Makefile.am << 'END'
+bad-target:
+ @echo $@ has been run >&2; exit 1
+install-data-local:
+ @echo $@ has been run >&2; exit 1
+TESTS = foo.test all.test install.test dist.test bad-target.test
+EXTRA_DIST = oops-this-does-not-exist
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure --prefix="`pwd`/inst"
+
+$MAKE check
+ls -l # For debugging.
+test ! -d inst
+for t in foo all install dist bad-target; do
+ grep "$t\.test has been run" $t.log
+done
+
+:
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2009-2012 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features:
+# - listing $(srcdir)/ or $(top_srcdir)/ in TESTS doesn't work ATM,
+# and is thus diagnosed.
+
+# TODO: this test should also ensure that the 'make' implementation
+# properly adheres to rules in all cases. See the Autoconf
+# manual for the ugliness in this area, when VPATH comes into
+# play. :-/
+
+am_parallel_tests=yes
+. ./defs || Exit 1
+
+echo AC_OUTPUT >> configure.ac
+
+cat > Makefile.am << 'END'
+TESTS = $(srcdir)/bar.test $(top_srcdir)/baz.test
+END
+
+$ACLOCAL
+$AUTOCONF
+AUTOMAKE_fails -a
+grep '$(srcdir).*TESTS.*bar\.test' stderr
+grep '$(top_srcdir).*TESTS.*baz\.test' stderr
+
+:
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check parallel-tests features:
-# - generated distributed tests.
-# - listing $(srcdir)/ or $(top_srcdir)/ in TESTS doesn't work ATM,
-# and is thus diagnosed.
-
-# TODO: this test should also ensure that the 'make' implementation
-# properly adheres to rules in all cases. See the Autoconf
-# manual for the ugliness in this area, when VPATH comes into
-# play. :-/
+# Check parallel-tests features: generated distributed tests.
am_parallel_tests=yes
. ./defs || Exit 1
cat > Makefile.am << 'END'
TESTS = foo.test
-## the next line will cause automake to error out:
-TESTS += $(srcdir)/bar.test $(top_srcdir)/baz.test
.in.test:
cp $< $@ && chmod +x $@
check_SCRIPTS = $(TESTS)
$ACLOCAL
$AUTOCONF
-AUTOMAKE_fails -a
-grep '$(srcdir).*TESTS.*bar\.test' stderr
-grep '$(top_srcdir).*TESTS.*baz\.test' stderr
-
-sed '/^TESTS +=.*srcdir/d' < Makefile.am > t
-mv -f t Makefile.am
$AUTOMAKE -a
./configure
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2012 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check that, if we have two pattern rules from which the same file (or
+# set of files) can be built, and both are applicable, the first one wins.
+# This is used at least in our 'parallel-tests' support.
+
+am_create_testdir=empty
+. ./defs || Exit 1
+
+cat > Makefile <<'END'
+default:
+
+%.foo: %
+ cp $< $@
+%.foo: %.x
+ cp $< $@
+
+%.bar: %.x
+ cp $< $@
+%.bar: %
+
+%.mu %.fu: %.1
+ cp $< $*.mu && cp $< $*.fu
+%.mu %.fu: %.2
+ cp $< $*.mu && cp $< $*.fu
+END
+
+echo one > all
+echo two > all.x
+$MAKE all.foo all.bar
+diff all all.foo
+diff all.x all.bar
+
+echo one > x.1
+echo two > x.2
+$MAKE x.mu
+diff x.mu x.1
+diff x.fu x.1
+
+:
END
cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = $(srcdir)/tap-driver
-TESTS = none.test noread.test noexec.test
+LOG_DRIVER = $(srcdir)/tap-driver
+TEST_LOG_DRIVER = $(LOG_DRIVER)
+TESTS = none.test noread noexec.test
none.test:
END
echo ok 1
END
-cp noexec.test noread.test
-chmod a-r noread.test
+cp noexec.test noread
+chmod a-r noread
$ACLOCAL
$AUTOCONF
fi
desc="non-readable test is reported"
-if test -r noread.test; then
+if test -r noread; then
skip_ -r "any file is readable" "$desc"
else
- command_ok_ "$desc" -- grep '^ERROR: noread\.test' stdout
+ command_ok_ "$desc" -- grep '^ERROR: noread' stdout
fi
# Check that no spurious test result is reported. This is lower-priority
echo 'ok 3 # SKIP' >>$@-t
cat $@-t ;: For debugging.
mv -f $@-t $@
+
+LOG_DRIVER = false # Dummy but required.
END
cat > success.test << 'END'
am_parallel_tests=yes
. ./defs || Exit 1
-cat >> configure.ac << END
+cat >> configure.ac << 'END'
+# FIXME: must define this otherwise automake will require the presence
+# FIXME: of the 'test-driver' script. This issue should be documented
+# FIXME: in the manual ...
+AC_SUBST([LOG_DRIVER],
+ ['$(error LOG_DRIVER should be never used) false'])
AC_OUTPUT
END
$ACLOCAL
$AUTOCONF
$AUTOMAKE
+test ! -f test-driver
./configure
my_log_compiler = cat
TEST_EXTENSIONS =
TESTS =
+LOG_DRIVER = $(error LOG_DRIVER is dummy, required, never extended)
END
: > later.mk
fetch_tap_driver
-cat >> configure.ac <<END
+cat >> configure.ac << 'END'
+AC_SUBST([LOG_DRIVER], ["dummy but required"])
AC_SUBST([AM_TEST_LOG_DRIVER_FLAGS], ['--comments'])
AC_OUTPUT
END
fetch_tap_driver
-cat >> configure.ac <<END
+cat >> configure.ac << 'END'
AC_OUTPUT
END
cat > Makefile.am << 'END'
+LOG_DRIVER = $(error LOG_DRIVER should never be expanded) false
TEST_LOG_DRIVER = $(srcdir)/tap-driver
TESTS = a.test b.test c.test d.test
END
$ACLOCAL
$AUTOCONF
-$AUTOMAKE
+$AUTOMAKE -a
./configure
$ACLOCAL
$AUTOCONF
-$AUTOMAKE
+$AUTOMAKE -a
do_recheck ()
{
$ACLOCAL
$AUTOCONF
-$AUTOMAKE
+$AUTOMAKE -a
for vpath in : false; do
if $vpath; then
END
cat > Makefile.am << 'END'
+LOG_DRIVER = dummy but required
+
TEST_EXTENSIONS = .t
T_LOG_DRIVER = $(SHELL) $(srcdir)/trivial-test-driver
cat >> configure.ac <<'END'
AC_PROG_CC
AC_CONFIG_FILES([sub1/Makefile sub2/Makefile])
-AC_SUBST([X_LOG_DRIVER], [none])
+AC_SUBST([LOG_DRIVER], ['unused but required by automake'])
AC_OUTPUT
END
cat > Makefile.am <<'END'
SUBDIRS = sub1 sub2
-LOG_DRIVER =
TEST_LOG_DRIVER = :
TESTS = foo bar.test
END
TEST_EXTENSIONS = .x .sh .pl
SH_LOG_DRIVER = dummy1
PL_LOG_DRIVER = dummy2
+X_LOG_DRIVER = dummy3
TESTS = a.pl b.sh c.x
END
cat > sub2/Makefile.am <<'END'
TEST_EXTENSIONS = .bar
-LOG_DRIVER = x
BAR_LOG_DRIVER = y
TESTS = 1 2.bar 3.test 4.t 5.tt $(check_PROGRAMS)
check_PROGRAMS = p1 p2$(EXEEXT) p3.bar p4.suf
$ACLOCAL
$AUTOCONF
-$AUTOMAKE
+$AUTOMAKE --add-missing
./configure
END
cat > Makefile.am <<'END'
-TEST_LOG_DRIVER = ./oops
-TESTS = foo.test
+TEST_EXTENSIONS =
+LOG_DRIVER = ./oops
+TESTS = foo
END
-cat > foo.test <<'END'
+cat > foo <<'END'
#! /bin/sh
exit 0
END
+chmod a+x foo
$ACLOCAL
$AUTOCONF
# The $(empty) are for eliciting VPATH rewrites on make implementations
# that support it (e.g., Solaris make), to improve coverage.
empty =
-TESTS = $(empty) foo.test src/bar.test ./src/baz.test $(empty)
+TESTS = $(empty) foo.test src/bar.test ./src/baz $(empty)
$(TESTS):
-TEST_LOG_DRIVER = $(srcdir)/checkstrip-driver
+LOG_DRIVER = $(srcdir)/checkstrip-driver
+TEST_LOG_DRIVER = $(LOG_DRIVER)
EXTRA_DIST = checkstrip-driver
END
done
echo "test name: $test_name" # For debugging.
case $test_name in
- foo.test|./foo.test|src/ba[rz].test|./src/ba[rz].test);;
- *) exit 1;;
+ foo.test|./foo.test) ;;
+ src/bar.test|./src/bar.test) ;;
+ src/baz|./src/baz) ;;
+ *) exit 1 ;;
esac
echo dummy > "$log_file"
echo dummy > "$trs_file"
. ./defs || Exit 1
cat >> configure.ac << 'END'
+AC_SUBST([LOG_DRIVER], ['ignored but required by autoamke, sigh!'])
AC_OUTPUT
END
TEST_EXTENSIONS = .test .x
TEST_LOG_DRIVER = ./dummy-driver
X_LOG_DRIVER = ./dummy-driver
-TESTS = foo.test zar-doz.test
+LOG_DRIVER = ./dummy-driver
+TESTS = foo.test zar-doz.test mu
END
cat > dummy-driver <<'END'
: Basic checks.
echo :global-test-result: PASS > foo.test
+echo :global-test-result: PASS > mu
echo :global-test-result: ERROR > zar-doz.x
$MAKE check
=========
END
+have_result <<END
+PASS: mu
+========
+END
+
have_result <<END
ERROR: zar-doz
==============
echo :global-test-result: PASS/SKIP > foo.test
echo :global-test-result: ALMOST PASSED > zar-doz.x
+echo :global-test-result: HU? > mu
$MAKE check
cat test-suite.log
======================
END
+have_result <<END
+HU?: mu
+=======
+END
+
: Fields ':test-result:' does not interfere with the global test result.
cat > foo.test << 'END'
:test-result: XFAIL
END
+cat > mu << 'END'
+:test-result: PASS
+:global-test-result: XFAIL
+:test-result: PASS
+:test-result: PASS
+END
+
$MAKE check && Exit 1
cat test-suite.log
=============
END
+have_result <<END
+XFAIL: mu
+=========
+END
+
: What happens when ':global-test-result:' is absent.
cat > foo.test << 'END'
:test-result: ERROR
END
: > zar-doz.x
+: > mu
$MAKE check && Exit 1
cat test-suite.log
============
END
+have_result <<END
+RUN: mu
+=======
+END
+
# Leading and trailing whitespace gets eaten/normalized.
echo ":global-test-result:SKIP${tab} ${tab}${tab}" > foo.test
echo ":global-test-result:${tab} ${tab}XFAIL ${tab} " > zar-doz.x
+echo ":global-test-result: ${tab} PASS${tab} ${tab}${tab}" > mu
$MAKE check
cat test-suite.log
==============
END
+have_result <<END
+PASS: mu
+========
+END
+
# Whitespaces before and after ':global-test-result:' are handled OK.
echo " $tab:global-test-result:PASS" > foo.test
echo "${tab}${tab}:global-test-result:${tab} ${tab}SKIP" > zar-doz.x
+echo " :global-test-result:${tab}SKIP" > mu
$MAKE check
cat test-suite.log
=============
END
+have_result <<END
+SKIP: mu
+========
+END
+
:
END
cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = ./dummy-driver
-TESTS = foo.test bar.test
+LOG_DRIVER = ./dummy-driver
+TEST_LOG_DRIVER = ${LOG_DRIVER}
+TESTS = foo.test bar
END
cat > dummy-driver <<'END'
# "old-style" directives with format "RESULT: test-name" are now ignored.
: > foo.test
-echo blah blah blah > bar.test
+echo blah blah blah > bar
mk_check
count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0
:test-global-result: PASS
:test-result: FAIL
END
-cat > bar.test <<END
+cat > bar <<END
:test-result: SKIP
:test-global-result: ERROR
END
:test-result: PASS
:test-global-result: XPASS
END
-echo ERROR: bar.test > bar.test
+echo ERROR: bar > bar
mk_check
count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0
:test-global-result: SKIP
:test-result: FAIL
END
-cat > bar.test <<END
+cat > bar <<END
:test-global-result: PASS
END
mk_check && Exit 1
:test-result: PASS
:test-result: SKIP
END
-cat > bar.test <<END
+cat > bar <<END
:test-result: SKIP
:test-result: PASS
:test-result: SKIP
:test-result: XPASS
:test-result: ERROR
END
-: > bar.test
+: > bar
mk_check && Exit 1
count_test_results total=6 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=1
-cp foo.test bar.test
+cp foo.test bar
mk_check && Exit 1
count_test_results total=12 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=2
# Check that we are liberal w.r.t. whitespace use.
: > foo.test
-: > bar.test
+: > bar
for RESULT in PASS FAIL XPASS XFAIL SKIP ERROR; do
sed -e 's/^ *//' -e 's/|//g' >> foo.test <<END
|:test-result:$RESULT|
|:test-result:$RESULT $tab|
|:test-result:$tab$tab $RESULT$tab $tab |
END
- echo " $tab $tab$tab :test-result: $RESULT" >> bar.test
+ echo " $tab $tab$tab :test-result: $RESULT" >> bar
done
cat foo.test # For debugging.
-cat bar.test # Likewise.
+cat bar # Likewise.
mk_check && Exit 1
count_test_results total=30 pass=5 fail=5 xpass=5 xfail=5 skip=5 error=5
projects / thirdparty / automake.git / commitdiff
