console message about bug reporting on failure has a bit more detail.
(bug#68746)
+ - When using the (default) "parallel" test driver, you can now omit the
+ output of skipped tests from test-suite.log by defining the
+ variable IGNORE_SKIPPED_LOGS to a non-empty value. (bug#71422)
+
* Bugs fixed
- Generated file timestamp checks handle filesystems with sub-second
The first possibility (the historical practice) is to list all your
macros in @file{acinclude.m4}. This file will be included in
@file{aclocal.m4} when you run @command{aclocal}, and its macro(s) will
-henceforth be visible to @command{autoconf}. However if it contains
+henceforth be visible to @command{autoconf}. However, if it contains
numerous macros, it will rapidly become difficult to maintain, and it
will be almost impossible to share macros between packages.
@node Parallel Test Harness
@subsection Parallel Test Harness
-By default, Automake generated a parallel (concurrent) test harness. It
+By default, Automake generates a parallel (concurrent) test harness. It
features automatic collection of the test scripts output in @file{.log}
files, concurrent execution of tests with @code{make -j}, specification
of inter-test dependencies, lazy reruns of tests that have not completed
in a prior run, and hard errors for exceptional failures.
@anchor{Basics of test metadata}
-@vindex TEST_SUITE_LOG
@vindex TESTS
@cindex @file{.log} files
@cindex @file{.trs} files
contain, among the other things, the results of the test cases run by
the script.
+@vindex TEST_SUITE_LOG
+@vindex IGNORE_SKIPPED_LOGS
The parallel test harness will also create a summary log file,
@code{TEST_SUITE_LOG}, which defaults to @file{test-suite.log} and requires
a @file{.log} suffix. This file depends upon all the @file{.log} and
@file{.trs} files created for the test scripts listed in @code{TESTS}.
+It contains the output of all tests that failed, encountered a hard error,
+succeeded unexpectedly, or---unless the variable @code{IGNORE_SKIPPED_LOGS}
+is set to a non-empty value---were skipped.
@vindex VERBOSE
As with the serial harness above, by default one status line is printed
Whether the console output should be colorized or not (@pxref{Simple
tests and color-tests}, to learn when this option gets activated and
when it doesn't).
+@item --collect-skipped-logs @{yes|no@}
+Whether to include the logs of skipped tests in the global
+@file{test-suite.log} file.
@item --expect-failure @{yes|no@}
Whether the tested program is expected to fail.
@item --enable-hard-errors @{yes|no@}
be run, and all the following ones are command-line options and arguments
for this program.
-Note that the exact semantics attached to the @option{--color-tests},
-@option{--expect-failure} and @option{--enable-hard-errors} options are
-left up to the individual test drivers. Still, having a behavior
-compatible or at least similar to that provided by the default driver
-is advised, as that would offer a better consistency and a more pleasant
-user experience.
+Exact semantics attached to the @option{--color-tests},
+@option{--collect-skipped-logs}, @option{--expect-failure}, and
+@option{--enable-hard-errors} options are left up to the individual
+test drivers. Still, having a behavior compatible or at least similar
+to that provided by the default driver is advised, as that results
+in better consistency and thus a more pleasant user experience.
@node Log files generation and test results recording
@subsubsection Log files generation and test results recording
# Default flags passed to test drivers.
am__common_driver_flags = \
--color-tests "$$am__color_tests" \
+ $$am__collect_skipped_logs \
--enable-hard-errors "$$am__enable_hard_errors" \
--expect-failure "$$am__expect_failure"
elif test -f "$$f"; then dir=; \
else dir="$(srcdir)/"; fi; \
tst=$$dir$$f; log='$@'; \
+if test -n '$(IGNORE_SKIPPED_LOGS)'; then \
+ am__collect_skipped_logs='--collect-skipped-logs no'; \
+else \
+ am__collect_skipped_logs=''; \
+fi; \
if test -n '$(DISABLE_HARD_ERRORS)'; then \
am__enable_hard_errors=no; \
else \
#! /bin/sh
# test-driver - basic testsuite driver script.
-scriptversion=2018-03-07.03; # UTC
+scriptversion=2024-06-07.11; # UTC
# Copyright (C) 2011-2024 Free Software Foundation, Inc.
#
Usage:
test-driver --test-name NAME --log-file PATH --trs-file PATH
[--expect-failure {yes|no}] [--color-tests {yes|no}]
+ [--collect-skipped-logs {yes|no}]
[--enable-hard-errors {yes|no}] [--]
TEST-SCRIPT [TEST-SCRIPT-ARGUMENTS]
trs_file= # Where to save the metadata of the test run.
expect_failure=no
color_tests=no
+collect_skipped_logs=yes
enable_hard_errors=yes
while test $# -gt 0; do
case $1 in
--log-file) log_file=$2; shift;;
--trs-file) trs_file=$2; shift;;
--color-tests) color_tests=$2; shift;;
+ --collect-skipped-logs) collect_skipped_logs=$2; shift;;
--expect-failure) expect_failure=$2; shift;;
--enable-hard-errors) enable_hard_errors=$2; shift;;
--) shift; break;;
case $tweaked_estatus:$expect_failure in
0:yes) col=$red res=XPASS recheck=yes gcopy=yes;;
0:*) col=$grn res=PASS recheck=no gcopy=no;;
- 77:*) col=$blu res=SKIP recheck=no gcopy=yes;;
+ 77:*) col=$blu res=SKIP recheck=no gcopy=$collect_skipped_logs;;
99:*) col=$mgn res=ERROR recheck=yes gcopy=yes;;
*:yes) col=$lgn res=XFAIL recheck=no gcopy=yes;;
*:*) col=$red res=FAIL recheck=yes gcopy=yes;;
t/parallel-tests-empty-testlogs.sh \
t/parallel-tests-driver-install.sh \
t/parallel-tests-no-color-in-log.sh \
+t/parallel-tests-without-skipped-logs.sh \
t/parallel-tests-no-spurious-summary.sh \
t/parallel-tests-exit-statuses.sh \
t/parallel-tests-console-output.sh \
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2024 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+# If IGNORE_SKIPPED_LOGS is set, skipped tests should not be in test-suite.log.
+
+. test-init.sh
+
+cat >>configure.ac <<END
+AC_OUTPUT
+END
+
+cat >Makefile.am <<'END'
+LOG_COMPILER = $(SHELL)
+TESTS = pass fail skiptest xpass xfail error
+XFAIL_TESTS = xpass xfail
+END
+
+echo 'exit 0' > pass
+echo 'exit 0' > xpass
+echo 'exit 1' > fail
+echo 'exit 1' > xfail
+echo 'exit 77' > skiptest # unique name for grep
+echo 'exit 99' > error
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE --add-missing
+
+# Check that IGNORE_SKIPPED_LOGS works when given on the command line.
+
+./configure
+run_make -e FAIL check IGNORE_SKIPPED_LOGS=true
+grep skiptest test-suite.log && exit 1
+
+rm -f test-suite.log
+
+# Check that IGNORE_SKIPPED_LOGS works when given in the Makefile.
+
+cat >>Makefile.am <<'END'
+IGNORE_SKIPPED_LOGS = true
+END
+
+$AUTOMAKE --add-missing
+
+./configure
+run_make -e FAIL check
+grep skiptest test-suite.log && exit 1
+
+:
projects / thirdparty / automake.git / commitdiff
