+2011-07-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: allow more metadata in log files
+ This change reworks and improves the parallel test harness to use
+ more specialized reStructuredText fields in the log files (instead
+ of relying on specially-placed of "magic lines" and more indirect
+ semantical formatting); the new fields are the following:
+ - ":recheck:": tell whether the associated test will have to be
+ re-run by "make recheck";
+ - ":copy-in-global-log:": tell whether the content of the log
+ file should be copied in the "global log" `test-suite.log';
+ - ":end-metadata:", which inhibits the scanning of the rest of
+ the log file (for what concerns test metadata).
+ Also, the special `:test-result:' value "END" has been removed,
+ superseded by the new `:end-metadata:' field.
+ * doc/automake.texi (Log files generation and test results
+ recording): Document the new API and semantics. Remove or fix
+ some obsolete comments.
+ * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html):
+ Adjust comments and code.
+ * lib/tap-driver (finish): Adjust, with the help of ...
+ (must_recheck, copy_in_global_log): ... these new functions.
+ * lib/test-driver (Main code): Adjust, with the help of ...
+ ($recheck, $gcopy): ... these new variables.
+ * tests/trivial-test-driver: Update to obey the new APIs.
+ * tests/test-driver-recheck.test: Likewise.
+ * tests/test-driver-global-log.test: Likewise.
+ * tests/tap-passthrough.test: Relax the test, avoiding to check
+ what is written in `test-suite.log'; such check has been moved ...
+ * tests/tap-global-log.test: ... in this new test, and extended.
+ * tests/test-driver-metadata-no-leading-space.test: New test.
+ * tests/test-driver-end-test-results.test: Removed, it checked
+ the old APIs; superseded by ...
+ * tests/test-driver-end-metadata.test: ... this new test.
+ * tests/tap-log.test: Improve syncing with ...
+ * tests/test-log.test: ... this new test.
+ * tests/parallel-tests.test: Remove some duplication w.r.t. this
+ last new test. Updated heading comments.
+ * tests/Makefile.am (TESTS): Update.
+
2011-07-27 Stefano Lattarini <stefano.lattarini@gmail.com>
testsuite: fix a spurious failure with non-bash shells
The test driver must correctly generate the file specified by the
@option{--log-file} option (of course even when the tested program
-fails or crashes). This file is quite free-form, but still it has
-to conform to the following conventions, in order to work with the
-@option{parallel-tests} harness and take advantage of its features.
+fails or crashes). This file, while being quite free-form, @emph{must}
+register some metadata through the use of custom reStructuredText fields,
+in order to work with the @option{parallel-tests} harness and take
+advantage of its features. Note that such fields will be recognized only
+when used @emph{exactly} at the beginning of a line -- not even leading
+whitespace is permitted.
-@itemize @bullet
-@item
-@cindex Global test script result
-@cindex Test result, global
-The @emph{global test result} for the whole test script must be placed
-in the first line of the @file{.log} file.
-@c The following semantics is checked by tests 'test-driver-recheck.test'
-@c and 'test-driver-global-log.test'. Please keep them in sync whenever
-@c the following is changed.
-If this line does not begin with either @code{PASS:}, @code{XFAIL:} or
-@code{SKIP:}, the test script will be considered failed, and it will be
-re-run by @code{make recheck}. Also, if this line does not begin with
-either @code{PASS:} or @code{XFAIL:}, the content of the @file{.log} file
-will be copied into the global @file{test-suite.log} (in order to help in
-debugging and bug-report analysis).
+@table @code
-@item
-@cindex Single test case result
-@cindex Test result, single
-@cindex @code{:test-result:} reStructuredText field
-@cindex reStructuredText field, @code{:test-result:}
-One of the main features of the new testsuite harness is the ability to
-support test protocols that allow a single test script to run more
-test cases, @emph{each with its distinct result}. In order for the
-testsuite summary to be correct in this case, the test driver must
-register @emph{each} such result in the generated @file{.log} file, using
-the @code{:test-result:} reStructuredText field. Otherwise, only the
-global test result will be considered (as it is the case of the default
-@option{parallel-tests} driver). Note that such a field will be
-recognized only when used @emph{exactly} at the beginning of a line:
-leading whitespace will not be ignored.
+@item :test-result:
+@cindex Register test result
+@cindex Register test case result
+@cindex Test result, registering
+@cindex Test case result, registering
+@cindex @code{:test-result:}
+@cindex reStructuredText field, @code{:test-result:}
+The test driver must use this filed to register the results of @emph{each}
+test case run by a test script file. Several @code{:test-result:} fields
+can be present in the same log file; this is done in order to support test
+protocols that allow a single test script to run more test cases.
@c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
The only recognized test results are currently @code{PASS}, @code{XFAIL},
text holding the name and/or a brief description of the corresponding
test; the @option{parallel-tests} harness will ignore such extra text when
generating @file{test-suite.log} and preparing the testsuite summary.
-Also, @code{:test-result:} can be used with a special ``pseudo-result''
-@code{END}, that will instruct the testsuite harness to stop scanning
-the rest of the log file (this feature is used internally by the default
-@option{parallel-tests} driver, and can be useful to avoid spurious
-testsuite results).
-Let's see a small example. If a @file{.log} file contains the following
-lines (exactly in this order, but possibly interleaved with other
-``non-@code{:test-results:}'' lines):
+@c Keep in sync with 'test-driver-recheck.test'.
+@item @code{:recheck:}
+@cindex :recheck:
+@cindex reStructuredText field, @code{:recheck:}
+If this field is present and defined to @code{no}, then the corresponding
+test script will @emph{not} be run upon a @command{make recheck}. What
+happens when two or more @code{:recheck:} fields are present in a
+@file{.log} file is undefined behaviour.
+
+@c Keep in sync with 'test-driver-global-log.test'.
+@item @code{:copy-in-global-log:}
+@cindex :copy-in-global-log:
+@cindex reStructuredText field, @code{:copy-in-global-log:}
+If this field is present and defined to @code{no}, then the content
+of the @file{.log} file will @emph{not} be copied into the global
+@file{test-suite.log}. We allow to forsake such copying because, while
+it can be useful in debugging and analysis of bug report, it can also be
+just a waste of space in normal situations, e.g., when a test script is
+successful. What happens when two or more @code{:copy-in-global-log:}
+fields are present in a @file{.log} file is undefined behaviour.
+
+@c Keep in sync with 'test-driver-end-metadata.test'.
+@item @code{:end-metadata:}
+@cindex :end-metadata:
+@cindex reStructuredText field, @code{:end-metadata:}
+This special field causes all the following text in the @file{.log}
+file not to be scanned by the testsuite harness in search for other
+test metadata. This feature is used internally by the default
+@option{parallel-tests} driver, and can be useful to e.g., avoid
+spurious testsuite results.
+
+@end table
+
+@noindent
+Let's see a small example. Assume a @file{.log} file contains the
+following lines (in this order, but possibly interleaved with other
+``non-metadata'' lines):
@example
:test-result: PASS server starts
:test-result: PASS HTTP/1.1 request
:test-result: FAIL HTTP/1.0 request
+:recheck: yes
:test-result: SKIP HTTPS request (TLS library wasn't available)
:test-result: PASS server stops
-:test-result: END
+:global-log-copy: no
+:metadata-end:
:test-result: XPASS (this shouldn't be parsed as a test result)
@end example
@noindent
-it means that the corresponding test script contributes with @emph{five}
-test results to the testsuite summary (three of these tests being
-successful, one failed, and one skipped).
-
-@c FIXME: currently, the @code{:test-result:} field is recognized anywhere
-@c in the @file{.log} file; this is possibly prone to generating spurious
-@c results, in case of verbose tests. We should allow a special
-@c @code{:test-result:} that stops any further parsing (maybe a line
-@c @code{:test-result:END} will do?).
-
-@end itemize
-
-Finally, note that the declaration of the global test result in the first
-line, apart from being needed for backward-compatibility with the default
-@option{parallel-tests} driver, can be useful also for test protocols
-that allow more test results per test script: using it, a custom test
-driver is allowed to decide what the global outcome of the test script
-is in case of conflicting test results within the script. For example,
-if a test script reports 8 successful test cases and 2 skipped test
-cases, some drivers might report that globally as a SKIP, while others
-as a PASS.
+Then the corresponding test script will be re-run by @command{make check},
+will contribute with @emph{five} test results to the testsuite summary
+(three of these tests being successful, one failed, and one skipped), and
+the @file{.log} file content will @emph{not} be copied in the global log
+file @file{test-suite.log}.
@node Testsuite progress output
@subsubsection Testsuite progress output
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- rst_magic=":test-result:"; \
+ tab=' '; \
nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
+ IFS=" $$tab$$nl"; oIFS=$$IFS; \
+ ws="[ $$tab]"; \
list='$(TEST_LOGS)'; \
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
exec 5<&0; \
for f in $$list2; do \
exec 0<$$f; \
+ IFS=''; \
while read line; do \
- case $$line in $$rst_magic*);; *) continue;; esac; \
- res=`expr "x$$line" : "x$$rst_magic[ ]*\\\\(.*\\\\)$$"`; \
- case $$res in END*) break;; esac; \
- results2="$$results2$$nl$$res"; \
+ IFS=$$oIFS; \
+ case $$line in \
+ :end-metadata:*) \
+ break;; \
+ :test-result:*) \
+ res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
+ results2="$$results2$$nl$$res";; \
+ esac; \
+ IFS=''; \
done; \
+ IFS=$$oIFS; \
exec 0<&5; \
done; \
exec 5<&-; \
echo ".. contents:: :depth: 2"; \
echo; \
for f in $$list; do \
- test -r $$f && read line < $$f || line=; \
- case $$line in \
- PASS:*|XFAIL:*);; \
- *) echo; cat $$f;; \
- esac; \
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
+ || { echo && cat $$f; } \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4; \
exit $$rv
recheck recheck-html:
- @target=`echo $@ | sed 's,^re,,'`; \
+ @ws='[ ]'; \
+ target=`echo $@ | sed 's,^re,,'`; \
list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- if test -r $$f && read line < $$f; then \
- case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
- else echo $$f; fi; \
+ list=`for f in $$list; do \
+ test -f $$f || continue; \
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
+ || echo $$f; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"'
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
-## The custom reStructuredText field used to register the outcome of
-## a test. This is for supporting test protocols, such as TAP and
-## SubUnit, where a single test script can run multiple tests, each
-## with its own outcome.
- rst_magic=":test-result:"; \
+ tab=' '; \
## The following line will define the `$nl' shell variable to a literal
## newline. Idiom taken from the Autoconf manual.
nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
+## Sanitize internal field separator for the shell: "space tab newline".
+ IFS=" $$tab$$nl"; oIFS=$$IFS; \
+## Regular expression to match a whitespace.
+ ws="[ $$tab]"; \
## All test logs.
list='$(TEST_LOGS)'; \
## Readable test logs.
exec 5<&0; \
for f in $$list2; do \
exec 0<$$f; \
+## We want to support special reStructuredText fields only when they're
+## placed at the beginning of a line (as documented in the manual), so
+## we have to temporarily clear the $IFS variable to prevent automatic
+## whitespace stripping by the shell.
+ IFS=''; \
while read line; do \
-## Ignore lines that do not declare test results.
- case $$line in $$rst_magic*);; *) continue;; esac; \
-## The test result declared by this line goes in $res.
- res=`expr "x$$line" : "x$$rst_magic[ ]*\\\\(.*\\\\)$$"`; \
-## A special result of "END" means that the rest of the log is not to be
-## scanned. Be a little lax in matching such special result, to allow
-## for e.g., trailing whitespace or punctuation.
- case $$res in END*) break;; esac; \
-## Otherwise register the result.
- results2="$$results2$$nl$$res"; \
+ IFS=$$oIFS; \
+ case $$line in \
+## Special field that tells not to scan the rest of the log file.
+ :end-metadata:*) \
+ break;; \
+## A test test result -- register it.
+ :test-result:*) \
+ res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
+ results2="$$results2$$nl$$res";; \
+ esac; \
+## For the next loop.
+ IFS=''; \
done; \
-## Restore original standard input.
+## Restore original standard input and internal field separator.
+ IFS=$$oIFS; \
exec 0<&5; \
done; \
## We don't need this anymore.
echo; \
echo ".. contents:: :depth: 2"; \
echo; \
-## Here we assume that the test driver writes a proper "summarizing
-## result" for the test script on the first line.
for f in $$list; do \
- test -r $$f && read line < $$f || line=; \
- case $$line in \
- PASS:*|XFAIL:*);; \
- *) echo; cat $$f;; \
- esac; \
+## FIXME: two forks per test -- this is horrendously inefficient!
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
+ || { echo && cat $$f; } \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
## Rechecking failures. ##
## -------------------- ##
-## Rerun all FAILed or XPASSed tests.
+## Rerun all tests that experienced an error or an unexpected failure.
recheck recheck-html:
- @target=`echo $@ | sed 's,^re,,'`; \
+ @ws='[ ]'; \
+ target=`echo $@ | sed 's,^re,,'`; \
list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- if test -r $$f && read line < $$f; then \
-## Here we assume that the test driver writes a proper summary for the
-## test script on the first line. See the comments in the rules of
-## $(TEST_SUITE_LOG) above for why we consider this acceptable and even
-## advisable.
- case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
-## A test whose log is unreadable is to be considered failed.
- else echo $$f; fi; \
+ list=`for f in $$list; do \
+ test -f $$f || continue; \
+## FIXME: two forks per test -- this is horrendously inefficient!
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
+ || echo $$f; \
done | tr '\012\015' ' '`; \
## This apparently useless munging helps to avoid a nasty bug (a
## segmentation fault!) on Solaris XPG4 make.
return @test_results;
}
+ # Whether the test script should be re-run by "make recheck".
+ sub must_recheck ()
+ {
+ return grep { !/^(?:XFAIL|PASS|SKIP)$/ } (keys %test_results);
+ }
+
+ # Whether the content of the log file associated to this test should
+ # be copied into the "global" test-suite.log.
+ sub copy_in_global_log ()
+ {
+ return grep { not $_ eq "PASS" } (keys %test_results);
+ }
+
# FIXME: this can certainly be improved ...
sub get_global_test_result ()
{
{
open LOG, ">", $log_file or die "opening $log_file: $!\n";
- # We need to declare a global test result in order to have
- # "make recheck" working.
+ # Some extra trailing empty lines outputted below are required
+ # to support reStructuredText -> HTML conversion.
+
my $global_result = get_global_test_result;
my $global_result_line = "$global_result: $test_script_name";
print LOG "$global_result_line\n";
print LOG "=" x length ($global_result_line) . "\n";
print LOG "\n";
+ print LOG ":recheck: " . (must_recheck ? "yes" : "no") . "\n";
+ print LOG "\n";
+
+ print LOG ":copy-in-global-log: " .
+ (copy_in_global_log ? "yes" : "no") . "\n";
+ print LOG "\n";
+
foreach (get_test_results)
{
print LOG ":test-result: $_\n";
- # Add extra trailing empty lines to support reStructuredText -> HTML
- # conversion.
print LOG "\n";
}
- # So that the output from the test script won't be parsed for test
- # results (which would potentially expose us to false positives).
- print LOG ":test-result: END\n";
+ # So that the output from the test script won't be parsed for
+ # testsuite metadata.
+ print LOG ":end-metadata:\n";
print LOG "\n";
close TMP or die "closing $log_file-t: $!\n";
#! /bin/sh
# test-driver - basic driver script for the `parallel-tests' mode.
-scriptversion=2011-06-29.13; # UTC
+scriptversion=2011-07-27.12; # UTC
# Copyright (C) 2011 Free Software Foundation, Inc.
#
fi
case $estatus:$expect_failure in
- 0:yes) col=$red; res=XPASS;;
- 0:*) col=$grn; res=PASS ;;
- 77:*) col=$blu; res=SKIP ;;
- 99:*) col=$mgn; res=ERROR;;
- *:yes) col=$lgn; res=XFAIL;;
- *:*) col=$red; res=FAIL ;;
+ 0:yes) col=$red res=XPASS recheck=yes gcopy=yes;;
+ 0:*) col=$grn res=PASS recheck=no gcopy=no;;
+ 77:*) col=$blu res=SKIP recheck=no gcopy=yes;;
+ 99:*) col=$mgn res=ERROR recheck=yes gcopy=yes;;
+ *:yes) col=$lgn res=XFAIL recheck=no gcopy=yes;;
+ *:*) col=$red res=FAIL recheck=yes gcopy=yes;;
esac
# Report outcome to console.
# Now write log file.
{
echo "$res: $test_name (exit: $estatus)" | rst_section
+ # Blank lines required for having the output be valid RST.
+ echo
echo ":test-result: $res (exit status: $estatus)"
echo
- echo ":test-result:END"
+ echo ":recheck: $recheck"
+ echo
+ echo ":copy-in-global-log: $gcopy"
+ echo
+ # So that the output from the test script won't be parsed for
+ # testsuite metadata.
+ echo ":end-metadata:"
echo
# Use a reStructuredText transition to better separate the test
- # outcome report from its registered output.
+ # metadata the test registered output.
echo ------------
echo
cat $tmpfile
testsuite-summary-count.test \
testsuite-summary-count-many.test \
testsuite-summary-reference-log.test \
-test-driver-end-test-results.test \
+test-driver-end-metadata.test \
+test-driver-metadata-no-leading-space.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
test-driver-custom-xfail-tests.test \
test-driver-recheck.test \
test-driver-strip-vpath.test \
test-driver-fail.test \
+test-log.test \
parse.test \
percent.test \
percent2.test \
tap-signal.test \
tap-fancy.test \
tap-fancy2.test \
+tap-global-log.test \
tap-global-result.test \
tap-html.test \
tap-log.test \
testsuite-summary-count.test \
testsuite-summary-count-many.test \
testsuite-summary-reference-log.test \
-test-driver-end-test-results.test \
+test-driver-end-metadata.test \
+test-driver-metadata-no-leading-space.test \
test-driver-custom-no-extra-driver.test \
test-driver-custom.test \
test-driver-custom-xfail-tests.test \
test-driver-recheck.test \
test-driver-strip-vpath.test \
test-driver-fail.test \
+test-log.test \
parse.test \
percent.test \
percent2.test \
tap-signal.test \
tap-fancy.test \
tap-fancy2.test \
+tap-global-log.test \
tap-global-result.test \
tap-html.test \
tap-log.test \
$(TEST_SUITE_LOG): $(TEST_LOGS)
@$(am__sh_e_setup); $(am__tty_colors); \
- rst_magic=":test-result:"; \
+ tab=' '; \
nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \
+ IFS=" $$tab$$nl"; oIFS=$$IFS; \
+ ws="[ $$tab]"; \
list='$(TEST_LOGS)'; \
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
exec 5<&0; \
for f in $$list2; do \
exec 0<$$f; \
+ IFS=''; \
while read line; do \
- case $$line in $$rst_magic*);; *) continue;; esac; \
- res=`expr "x$$line" : "x$$rst_magic[ ]*\\\\(.*\\\\)$$"`; \
- case $$res in END*) break;; esac; \
- results2="$$results2$$nl$$res"; \
+ IFS=$$oIFS; \
+ case $$line in \
+ :end-metadata:*) \
+ break;; \
+ :test-result:*) \
+ res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \
+ results2="$$results2$$nl$$res";; \
+ esac; \
+ IFS=''; \
done; \
+ IFS=$$oIFS; \
exec 0<&5; \
done; \
exec 5<&-; \
echo ".. contents:: :depth: 2"; \
echo; \
for f in $$list; do \
- test -r $$f && read line < $$f || line=; \
- case $$line in \
- PASS:*|XFAIL:*);; \
- *) echo; cat $$f;; \
- esac; \
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \
+ || { echo && cat $$f; } \
done; \
} >$(TEST_SUITE_LOG).tmp; \
mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \
$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4; \
exit $$rv
recheck recheck-html:
- @target=`echo $@ | sed 's,^re,,'`; \
+ @ws='[ ]'; \
+ target=`echo $@ | sed 's,^re,,'`; \
list='$(TEST_LOGS)'; \
- list=`for f in $$list; do \
- test -f $$f || continue; \
- if test -r $$f && read line < $$f; then \
- case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \
- else echo $$f; fi; \
+ list=`for f in $$list; do \
+ test -f $$f || continue; \
+ sed -e '/^:end-metadata:/,$$d' $$f | \
+ grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \
+ || echo $$f; \
done | tr '\012\015' ' '`; \
list=`echo "$$list" | sed 's/ *$$//'`; \
$(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"'
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check parallel-tests features:
-# - VERBOSE
-# - clean
-# - TEST_SUITE_LOG
-# - dependencies between tests
-# - TESTS
-# - TEST_LOGS
-# - RECHECK_LOGS
+# Basic checks on parallel-tests support:
+# - console output
+# - log files, and what goes in 'test-suite.log'
+# - make clean
+# - dependencies between tests
+# - TESTS redefinition at runtime
+# - TEST_LOGS redefinition at runtime
+# - RECHECK_LOGS redefinition at runtime
parallel_tests=yes
. ./defs || Exit 1
END
cat > Makefile.am << 'END'
-TEST_SUITE_LOG = mylog.log
TESTS = foo.test bar.test baz.test
XFAIL_TESTS = bar.test
foo.log: bar.log
$MAKE check >stdout && { cat stdout; Exit 1; }
cat stdout
count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1
-test -f mylog.log
-cat mylog.log
-test `grep -c '^FAIL:' mylog.log` -eq 1
-test `grep -c '^ERROR:' mylog.log` -eq 1
-$EGREP '^(X?PASS|XFAIL|SKIP)' mylog.log && Exit 1
+test -f test-suite.log
+cat test-suite.log
+test `grep -c '^FAIL:' test-suite.log` -eq 1
+test `grep -c '^ERROR:' test-suite.log` -eq 1
+$EGREP '^(X?PASS|XFAIL|SKIP)' test-suite.log && Exit 1
test -f baz.log
test -f bar.log
test -f foo.log
-# clean should remove all log files (but not more).
-: > unrelated.log
$MAKE clean
test ! -f baz.log
test ! -f bar.log
test ! -f foo.log
-test ! -f mylog.log
-test -f unrelated.log
+test ! -f test-suite.log
# Check dependencies: baz.test needs to run before bar.test,
# but foo.test is not needed.
test -f baz.log
test -f bar.log
test ! -f foo.log
-test -f mylog.log
+test -f test-suite.log
# Upon a lazy rerun, foo.test should be run, but the others shouldn't.
# Note that the lazy rerun still exits with a failure, due to the previous
grep '^# FAIL: *1$' stdout
grep '^# ERROR: *1$' stdout
-# Test VERBOSE.
-env VERBOSE=yes $MAKE -e check > stdout && { cat stdout; Exit 1; }
-cat stdout
-grep 'this is.*bar.test' stdout
-grep 'this is.*baz.test' stdout
-
$MAKE clean
env TEST_LOGS=baz.log $MAKE -e check > stdout && { cat stdout; Exit 1; }
cat stdout
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# TAP support:
+# - which log files get copied in the global log?
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cp "$top_testsrcdir"/lib/tap-driver . \
+ || fatal_ "failed to fetch auxiliary script tap-driver"
+
+cat >> configure.in <<END
+AC_SUBST([PERL], ['$PERL'])
+AC_OUTPUT
+END
+
+cat > ok.test << 'END'
+1..5
+ok 1
+ok 2
+ok 3
+not seen in global log
+ok 4
+ok 5
+END
+
+cat > top << 'END'
+1..6
+ok 1
+ok 2
+ok 3
+END
+
+cat > bot << 'END'
+ok 5
+ok 6
+END
+
+cat top - bot > skip.test << 'END'
+ok # SKIP
+::skip::
+END
+
+cat top - bot > todo.test << 'END'
+not ok # TODO
+::todo::
+END
+
+cat top - bot > fail.test << 'END'
+not ok
+::fail::
+END
+
+cat top - bot > xpass.test << 'END'
+ok # TODO
+::xpass::
+END
+
+cat top - bot > bail.test << 'END'
+::bail::
+Bail out!
+END
+
+cat top - bot > error.test << 'END'
+::error::
+1..7
+END
+
+# Created with "dd if=/dev/urandom count=1 | base64-encode"
+cat > hodgepodge <<'END'
+1+0 records in
+1+0 records out
+512 bytes (512 B) copied, 0.000241092 s, 2.1 MB/s
+gdivnV4VhL4DOzhE3zULJuun3PwqqQqMdATVcZbIQkNgyRvNBoqqHMBQEs7QsjDbp2nK+Szz
+EcelGyvXmHrW7yImaom6Yrg95k31VCmp/pGDRnTDwDrcOPJiv9jDReEmTAQkPuqLO+mFNly+
+DDHM9fNbzGumstsQ3wq3DOXz1pCV3JXwhjeaHSboeEbmr55bX0XHLSKaecctA0rXDXEyZWZ/
+ODlawSrAXzw0H7a+xBwjnNXZ3zYiwk3x+WQrPqNjb+qXiLLTxAKzx2/KnaFhxkPlte5jPRNB
+FciDolL+H/10LsCdSzLOrGnY2zH6vL2JMZfxjnb73zWFcdWWE01LTD7wpN5O1MP3+N47lcVe
+lWbkD04cJvhwxLElYSO24B743GG5EyGYt9SeZRE6xbgwq3fVOS8KqjwGxwi4adSBTtw0CV8W
+S/6n8dck1vBvjA+qpk0zMSYSqc3+jzW9UiGTmTEIwfw80p/lGNsfjcNBJ86nFkWUnHmrsi8k
+Dv57sK70mTg239g08f5Uvdga/5UreMBSgB0hUj5sbq57r7B1fsVr0Kag468la8zKy3ZEZ0gX
+++sbaJ9WGHhnKvjooeH+4Y6HwAFsdINde++FlCmp4EuNKKEEuXbSKLaOTy3+6pJ2DYdvRCL+
+frZwxH4hcrw8qh+8IakB02viewZS/qT57v4=
+END
+
+exec 5>misc.test
+echo 'ok # SKIP' >&5
+cat hodgepodge >&5
+echo 'not ok # TODO' >&5
+echo 'ok' >&5
+exec 5>&-
+
+cat > skipall.test << 'END'
+1..0 # SKIP all
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver
+TEST_LOG_COMPILER = cat
+END
+echo TESTS = *.test >> Makefile.am
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+# We don't care about the exit status of "make check" here.
+$MAKE check || :
+cat test-suite.log
+
+grep 'ok\.test|not seen' test-suite.log && Exit 1
+
+for s in skip todo fail xpass bail error; do
+ $FGREP "::$s::" test-suite.log
+done
+
+grep '^1\.\.0 # SKIP all$' test-suite.log
+
+case `cat test-suite.log` in
+ *"`cat hodgepodge`"*) ;;
+ *) Exit 1;;
+esac
+
+:
# TAP support:
# - log file creation
# - log file removal
-# - stdout and stderr of a TAP-generating test script go in its log file
+# - stdout and stderr of a script go in its log file
# - TEST_SUITE_LOG redefinition, at either automake or make time
# - VERBOSE environment variable support
+# Keep in sync with 'test-log.test'.
parallel_tests=yes
. ./defs || Exit 1
$ACLOCAL
$AUTOCONF
-$AUTOMAKE
+$AUTOMAKE -a
./configure
done
test $st -eq 0 || Exit 1
cat my.log # For debugging.
-for result in fail xpass skip error; do
+for result in xfail fail xpass skip error; do
cat $result.log # For debugging.
$FGREP "$pmarker $result $pmarker" my.log || st=1
$FGREP "$cmarker $result $cmarker" my.log || st=1
done
-test `$FGREP -c "$pmarker" my.log` -eq 4
-test `$FGREP -c "$cmarker" my.log` -eq 4
+test `$FGREP -c "$pmarker" my.log` -eq 5
+test `$FGREP -c "$cmarker" my.log` -eq 5
-# Successful test scripts shouldn't be mentioned in the global log.
-$EGREP '(^pass|[^x]pass|xfail)\.test' my.log && Exit 1
-# But failing and skipped one should.
+# Passed test scripts shouldn't be mentioned in the global log.
+$EGREP '(^pass|[^x]pass)\.test' my.log && Exit 1
+# But failing (expectedly or not) and skipped ones should.
+$FGREP 'xfail.test' my.log
$FGREP 'skip.test' my.log
$FGREP 'fail.test' my.log
$FGREP 'xpass.test' my.log
test ! -f xpass.log
test ! -f skip.log
test ! -f error.log
+# "make mostlyclean" shouldn't remove unrelated log files.
test -f error2.log
test -f test-suite.log
test -f global.log
case $out in *"$log"*) ;; *) Exit 1;; esac
touch error2.log test-suite.log my.log
-$MAKE mostlyclean
+$MAKE clean
ls -l # For debugging.
test ! -f global.log
test ! -f pass.log
test ! -f xpass.log
test ! -f skip.log
test ! -f error.log
+# "make clean" shouldn't remove unrelated log files.
test -f error2.log
test -f test-suite.log
test -f my.log
rm -f *.log
-
:
# some diagnostic${tab}
not ok # TODO low priority
ok # SKIP who cares?
-This is not in test-suite.log, since ok.test does not fail.
$weirdchars
END
-st=0
-TESTS=ok.test $MAKE -e check || st=$?
+TESTS=ok.test $MAKE -e check || { cat ok.log; Exit 1; }
cat ok.log
-cat test-suite.log
-test $st -eq 0 || Exit 1
for rx in \
'1\.\.6' \
"# some diagnostic${tab}" \
'not ok # TODO low priority' \
'ok # SKIP who cares?' \
- 'This is not in test-suite\.log, since ok\.test does not fail\.' \
; do
grep "^$rx$" ok.log
done
$FGREP "$weirdchars" ok.log
-$FGREP 'ok.test' test-suite.log && Exit 1
-$FGREP 'not in test-suite.log' test-suite.log && Exit 1
-
#
# Mixed failing/successful tests.
#
cat > ok.test <<END
1..1
ok
-local only
+only one success here
END
cat > ko.test <<END
cat bail.log
cat skip.log
cat err.log
-cat test-suite.log
test $st -gt 0 || Exit 1
grep '^1\.\.1$' tiny.log
grep '^ok$' tiny.log
-
-grep '^local only$' ok.log
-grep 'local only' test-suite.log && Exit 1
+grep '^only one success here$' ok.log
for rx in \
'1\.\.5' \
'ok # TODO dunno' \
; do
grep "^$rx$" ko.log
- grep "^$rx$" test-suite.log
done
$FGREP "$weirdchars" ko.log
-$FGREP "$weirdchars" test-suite.log
-for log in bail.log test-suite.log; do
- grep '^Bail out! Test is taking too long!$' $log
-done
-for log in skip.log test-suite.log; do
- grep '^1\.\.0 # Skipped: WWW::Mechanize not installed$' $log
-done
+grep '^Bail out! Test is taking too long!$' bail.log
+grep '^1\.\.0 # Skipped: WWW::Mechanize not installed$' skip.log
for rx in \
'^1\.\.3$' \
'^ERROR:.* err\.test 23 .*OUT[ -]OF[ -]ORDER.*expecting 2' \
; do
grep "$rx" err.log
- grep "$rx" test-suite.log
done
:
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers and parallel-tests harness: check the documented
+# semantics for inhibiting the scanning of the remaining portion of a
+# `.log' file (for what concerns metadata). Currently, this is done
+# with the use of the reStructuredText field `:end-metadata:'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./passthrough-driver
+TESTS = foo.test bar.test baz.test
+END
+
+cat > passthrough-driver <<'END'
+#!/bin/sh
+set -e; set -u;
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "$test_name: RUN"
+cp $1 $log_file
+END
+chmod a+x passthrough-driver
+
+cat > foo.test <<END
+:end-metadata:
+:test-result: FAIL
+:copy-in-global-log: no
+:recheck: no
+foo foo foo
+END
+
+cat > bar.test <<END
+:test-result: PASS
+:recheck: no
+bar bar bar
+:copy-in-global-log: no
+:end-metadata: $tab
+:test-result: ERROR
+:end-metadata:
+END
+
+# `:end-metadata:' directives in other files shouldn't influence this one.
+cat > baz.test <<END
+:test-result: PASS
+:recheck: no
+:copy-in-global-log: no
+baz baz baz
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+st=0
+$MAKE check >stdout || st=1
+# For debugging.
+cat stdout
+cat foo.log
+cat bar.log
+cat test-suite.log
+
+# Check that only the `:test-result:' in baz.test and the first one in
+# foo.test have been considered.
+test $st -eq 0
+grep '^# TOTAL: *2$' stdout
+grep '^# PASS: *2$' stdout
+for result in SKIP XPASS FAIL XFAIL ERROR; do
+ grep "^# $result: *0$" stdout
+done
+
+# Check that a `:end-metadata:' directive does not prevent following text
+# to be copied into the log file.
+$FGREP 'foo foo foo' foo.log
+$FGREP 'bar bar bar' bar.log
+$FGREP 'baz baz baz' baz.log # More of a sanity check really.
+
+# Check that only the `:copy-in-global-log:' in bar.test has been
+# considered.
+$FGREP 'foo foo foo' test-suite.log
+$FGREP 'bar bar bar' test-suite.log && Exit 1
+$FGREP 'baz baz baz' test-suite.log && Exit 1
+
+# Check that only the `:recheck:' in bar.test has been
+# considered.
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+grep '^foo\.test: RUN$' stdout
+grep 'ba[rz]\.test.*RUN' stdout && Exit 1
+
+:
+++ /dev/null
-#! /bin/sh
-# Copyright (C) 2011 Free Software Foundation, Inc.
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2, or (at your option)
-# any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-# parallel-tests harness: check that a special ":test-result:END"
-# directive prevents the code generating the testsuite summary and
-# the test-suite.log from parsing the rest of the log file.
-
-parallel_tests=yes
-. ./defs || Exit 1
-
-cat >> configure.in << 'END'
-AC_OUTPUT
-END
-
-cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = ./dummy-driver
-TESTS = a.test b.test c.test d.test
-END
-
-cat > dummy-driver <<'END'
-#!/bin/sh
-set -e
-while test $# -gt 0; do
- case $1 in
- --log-file) log_file=$2; shift;;
- --test-name|--expect-failure|--color-tests|--enable-hard-errors) shift;;
- --) shift; break;;
- *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
- esac
- shift
-done
-cp $1 $log_file
-END
-chmod a+x dummy-driver
-
-cat > a.test <<'END'
-PASS: a.test
-:test-result:END
-:test-result:FAIL
-END
-
-cat > b.test <<END
-PASS: b.test
-:test-result:PASS
-:test-result:END
-:test-result:FAIL
-:test-result:ERROR
-END
-
-cat > c.test <<END
-PASS: c.test
-:test-result:SKIP
-:test-result: END$tab $tab$tab
-:test-result:XPASS
-END
-
-cat > d.test <<END
-PASS: d.test
-:test-result:PASS
-:test-result:PASS
-:test-result: $tab END.
-:test-result:FAIL
-END
-
-$ACLOCAL
-$AUTOCONF
-$AUTOMAKE
-
-./configure
-
-$MAKE check >output 2>&1 || { cat output; Exit 1; }
-cat output
-cat a.log; cat b.log; cat c.log; cat d.log # For debugging.
-
-grep '^# TOTAL: *4$' output
-grep '^# PASS: *3$' output
-grep '^# SKIP: *1$' output
-for result in XPASS FAIL XFAIL ERROR; do
- grep "^# $result: *0$" output
-done
-
-:
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Check that the output of testcases having a global outcome that is not
-# "PASS" or "XFAIL" is copied in the global 'test-suite.log'.
+# Custom test drivers and parallel-tests harness: check the documented
+# semantics for deciding when the content of a test log file should be
+# copied in the global test-suite.log file. Currently, this is done
+# with the use of the reStructuredText field `:copy-in-global-log:'.
parallel_tests=yes
. ./defs || Exit 1
AC_OUTPUT
END
-# The :test-result: fields should be irrelevant for the decision
-# of whether a test output is to be copied in the test-suite.log.
-cat > pass.test <<END
-PASS: pass.test
+# The `:test-result:' and `:recheck:' fields and the first line of the
+# log file should be be irrelevant for the decision of whether a test
+# output is to be copied in the `test-suite.log'.
+
+cat > no-1.test <<END
+FAIL: no-1.test
:test-result: SKIP
+:copy-in-global-log: no
:test-result: FAIL
-:test-result: XFAIL
:test-result: XPASS
-:test-result: ERROR
-not seen (pass)
+not seen 1
+END
+
+for RES in XPASS FAIL XFAIL SKIP ERROR UNKNOWN; do
+ unindent > $RES.test <<END
+ $RES: $RES.test
+ :test-result: $RES
+ :copy-in-global-log: no
+ not seen $RES
END
+done
-cat > xfail.test <<'END'
-XFAIL: xfail.test
-not seen (xfail)
+# In the last line, with leading and trailing whitespace.
+cat > no-2.test <<END
+:test-result: FAIL
+not seen 2
+:recheck: yes
+:copy-in-global-log:$tab $tab no $tab
END
-echo SKIP: skip.test > skip.test
-echo FAIL: fail.test > fail.test
-echo XPASS: xpass.test > xpass.test
-echo ERROR: error.test > error.test
-echo :test-result: PASS > fake-pass.test
-echo "$tab $tab$tab" > empty.test
-
-cat > Makefile.am << 'END'
-TEST_LOG_DRIVER = ./dummy-driver
-TESTS = pass.test skip.test fail.test xfail.test xpass.test error.test \
- fake-pass.test empty.test
+# In the first line, with no whitespace.
+cat > no-3.test <<END
+:copy-in-global-log:no
+not seen 3
+:test-result: FAIL
END
+cat > yes-1.test <<END
+PASS: yes-1.test
+:test-result: PASS
+:copy-in-global-log: yes
+seen yes 1
+END
+
+# A lacking `:copy-in-global-log:' implies that the content of
+# the log file should be copied.
+cat > yes-2.test <<END
+PASS: y.test
+:test-result: PASS
+seen yes 2
+END
+
+# Corner cases.
+echo ':copy-in-global-log:' > corn-1.test
+echo " $tab $tab$tab" > corn-2.test
+cat > corn-3.test <<'END'
+seen corn 31
+:copy-in-global-log:#@%!
+seen corn 32
+END
+
+echo TEST_LOG_DRIVER = ./dummy-driver > Makefile.am
+echo TESTS = *.test >> Makefile.am
cat > dummy-driver <<'END'
#!/bin/sh
./configure
-$MAKE check && { cat test-suite.log; Exit 1; }
+# We don't care about the exit status of "make check" here, that
+# should be checked in other tests.
+$MAKE check || :
cat test-suite.log
-grep '^SKIP: skip\.test$' test-suite.log
-grep '^FAIL: fail.test$' test-suite.log
-grep '^XPASS: xpass.test$' test-suite.log
-grep '^ERROR: error.test$' test-suite.log
-grep '^:test-result: PASS$' test-suite.log
-grep "^$tab $tab$tab$" test-suite.log
-$EGREP 'not seen' test-suite.log && Exit 1
-test `grep -c ':test-result:' test-suite.log` -eq 1
-
-cat > pass.test <<END
-PASS: pass.test
-:test-result: SKIP
-:test-result: XFAIL
-not seen (pass)
-END
-cat > xfail.test <<END
-XFAIL: xfail.test
-:test-result: SKIP
-:test-result: PASS
-not seen (xfail)
-END
-
-TESTS='pass.test skip.test xfail.test' $MAKE -e check \
- || { cat test-suite.log; Exit 1; }
-cat test-suite.log
-grep '^SKIP: skip\.test$' test-suite.log
-$EGREP ':test-result:|not seen' test-suite.log && Exit 1
+grep '^seen yes 1$' test-suite.log
+grep '^seen yes 2$' test-suite.log
+grep '^seen corn 31$' test-suite.log
+grep '^seen corn 32$' test-suite.log
+grep '^:copy-in-global-log:$' test-suite.log
+grep "^ $tab $tab$tab$" test-suite.log
+$FGREP 'not seen' test-suite.log && Exit 1
+$EGREP '^(XPASS|FAIL|SKIP|ERROR|UNKNOWN)' test-suite.log && Exit 1
+test `grep -c ':test-result:' test-suite.log` -eq 2
:
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Custom test drivers and parallel-tests harness: check that, in the
+# log files, the reStructuredText fields for metadata declaration are
+# recognized only when placed at the beginning of a line.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in << 'END'
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TEST_LOG_DRIVER = ./passthrough-driver
+TESTS = foo.test
+END
+
+cat > passthrough-driver <<'END'
+#!/bin/sh
+set -e; set -u;
+while test $# -gt 0; do
+ case $1 in
+ --log-file) log_file=$2; shift;;
+ --test-name) test_name=$2; shift;;
+ --expect-failure|--color-tests|--enable-hard-errors) shift;;
+ --) shift; break;;
+ *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;;
+ esac
+ shift
+done
+echo "$test_name: RUN"
+cp $1 $log_file
+END
+chmod a+x passthrough-driver
+
+cat > foo.test <<END
+ $tab :test-result: FAIL
+ $tab :end-metadata:
+:test-result: PASS
+ $tab :copy-in-global-log: no
+ $tab :recheck: no
+foo foo foo
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE
+
+./configure
+
+st=0
+$MAKE check >stdout || st=1
+# For debugging.
+cat stdout
+cat foo.log
+cat test-suite.log
+
+# Check that only the second `:test-result:' has been processed.
+test $st -eq 0
+grep '^# TOTAL: *1$' stdout
+grep '^# PASS: *1$' stdout
+for result in SKIP XPASS FAIL XFAIL ERROR; do
+ grep "^# $result: *0$" stdout
+done
+
+# Check that `:copy-in-global-log:' hasn't been processed.
+grep 'foo foo foo' test-suite.log
+
+# Check that `:recheck:' hasn't been processed.
+: > older
+$sleep
+$MAKE recheck >stdout || { cat stdout; Exit 1; }
+cat stdout
+grep '^foo\.test: RUN$' stdout
+grep 'foo foo foo' test-suite.log
+is_newest test-suite.log older
+is_newest foo.log older
+
+:
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
-# Test the "make recheck" semantics for custom test driver, as
-# documented in detail in the Automake manual.
+# Test the "make recheck" semantics for custom test drivers, as documented
+# in the Automake manual. Currently, that is based on the reStructuredText
+# field `:recheck:'.
+
parallel_tests=yes
. ./defs || Exit 1
TESTS =
END
-rechecked=''
-notrechecked=''
-
-for R in PASS XFAIL SKIP; do
- echo $R: > $R-1
- echo " $R:" > $R-2
- echo $R:foo > $R-3
- echo "$tab $R: bar baz" > $R-4
- # The :test-result: fields should be irrelevant for the decision of
- # whether "make recheck" should or should not re-run a test.
- unindent > $R-5 <<END
- $R:
- :test-result: FAIL
- :test-result: XPASS
- :test-result: ERROR
+#
+# Tests to be re-run by "make recheck"
+#
+
+: > y-1
+echo "foo bar" > y-2
+echo ":recheck:" > y-3
+echo ":recheck:yes" > y-4
+echo ":recheck: who cares" > y-5
+echo ":recheck: $tab y" > y-6
+echo ":recheck: yeah!$tab$tab " > y-7
+cat > y-10 <<END
+foo
+:recheck: ???
END
- notrechecked="$notrechecked $R-1 $R-2 $R-3 $R-4 $R-5"
- echo $R > BAD-$R-1
- echo $R. > BAD-$R-2
- echo :$R: > BAD-$R-3
- echo $R asd > BAD-$R-4
- rechecked="$rechecked BAD-$R-1 BAD-$R-2 BAD-$R-3 BAD-$R-4"
-done
+cat > y-11 <<END
+:recheck: YES
+foo
+END
+cat > y-12 <<END
+foo
+:recheck:yes
-for R in FAIL XPASS ERROR UNKNOWN; do
- echo $R: > $R-1
- echo $R:foo > $R-2
- echo $R: bar baz > $R-3
- echo $R > $R-4
- echo $R asd > $R-5
- # The :test-result: fields should be irrelevant for the decision of
- # whether "make recheck" should or should not re-run a test.
- unindent > $R-6 <<END
- :test-result: PASS
- :test-result: SKIP
- :test-result: XFAIL
+bar
+zardoz
END
- (echo $R: && cat $R-6) > $R-7
- rechecked="$rechecked $R-1 $R-2 $R-3 $R-4 $R-5 $R-6 $R-7"
-done
-: > EMPTY
-echo " $tab $tab" > WHITE
-rechecked="$rechecked EMPTY WHITE"
+# The :test-result: fields and the fist line of the log should be
+# irrelevant for the decision of whether "make recheck" should or
+# should not re-run a test.
+
+echo ":test-result: PASS" > y-100
+
+echo "PASS: y-101"
+
+cat > y-102 <<END
+PASS: y-102
+===========
+
+:test-result: PASS
+END
+
+#
+# Tests *not* to be re-run by "make recheck"
+#
+
+echo ":recheck:no" > n-1
+echo ":recheck: no " > n-2
+echo ":recheck: $tab no" > n-3
+echo ":recheck: no $tab$tab " > n-4
+cat > n-5 <<END
+foo
+:recheck:no
+END
+cat > n-6 <<END
+:recheck: no
+foo
+END
+cat > n-7 <<END
+foo
+:recheck: no$tab$tab
+
+bar
+zardoz
+END
+
+# The :test-result: fields should be irrelevant for the decision of
+# whether "make recheck" should or should not re-run a test.
+cat > n-100 <<END
+:test-result: FAIL
+:test-result: XPASS
+:test-result: ERROR
+:test-result: UNKNOWN
+:recheck: no
+END
-tests="$rechecked $notrechecked"
+rechecked=`echo y-[0-9]*`
-for t in $tests; do echo $t; done | sed 's/.*/TESTS += &/' >> Makefile.am
+for t in [yn]-[0-9]*; do echo $t; done \
+ | sed 's/.*/TESTS += &/' >> Makefile.am
cat Makefile.am # For debugging.
$MAKE recheck
ls -l
for t in $rechecked; do test -f $t.run; done
- for t in $notrechecked; do test ! -r $t.run; done
- rm -f *.run
+ find . -name 'n-*.run' | grep . && Exit 1
done
:
--- /dev/null
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Check parallel-tests features:
+# - log file creation
+# - log file removal
+# - stdout and stderr of a test script go in its log file
+# - TEST_SUITE_LOG redefinition, at either automake or make time
+# - VERBOSE environment variable support
+# Keep in sync with 'tap-log.test'.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat >> configure.in <<END
+AC_OUTPUT
+END
+
+cat > Makefile.am << 'END'
+TESTS = pass.test skip.test xfail.test fail.test xpass.test error.test
+XFAIL_TESTS = xpass.test xfail.test
+TEST_SUITE_LOG = global.log
+END
+
+# Custom markers, for use in grepping checks.
+cmarker=::: # comment marker
+pmarker=%%% # plain maker
+
+cat > pass.test <<END
+#! /bin/sh
+echo "$pmarker pass $pmarker" >&2
+echo "# $cmarker pass $cmarker" >&2
+exit 0
+END
+
+cat > skip.test <<END
+#! /bin/sh
+echo "$pmarker skip $pmarker"
+echo "# $cmarker skip $cmarker"
+exit 1
+END
+
+cat > xfail.test <<END
+#! /bin/sh
+echo "$pmarker xfail $pmarker" >&2
+echo "# $cmarker xfail $cmarker" >&2
+exit 1
+END
+
+cat > fail.test <<END
+#! /bin/sh
+echo 1..1
+echo "$pmarker fail $pmarker"
+echo "# $cmarker fail $cmarker"
+exit 1
+END
+
+cat > xpass.test <<END
+#! /bin/sh
+echo "$pmarker xpass $pmarker" >&2
+echo "# $cmarker xpass $cmarker" >&2
+exit 0
+END
+
+cat > error.test <<END
+#! /bin/sh
+echo "$pmarker error $pmarker"
+echo "# $cmarker error $cmarker"
+exit 99
+END
+
+chmod a+x *.test
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+
+TEST_SUITE_LOG=my.log $MAKE -e check && Exit 1
+ls -l # For debugging.
+test ! -f test-suite.log
+test ! -f global.log
+test -f my.log
+st=0
+for result in pass fail xfail xpass skip error; do
+ cat $result.log # For debugging.
+ $FGREP "$pmarker $result $pmarker" $result.log || st=1
+ $FGREP "$cmarker $result $cmarker" $result.log || st=1
+done
+test $st -eq 0 || Exit 1
+cat my.log # For debugging.
+for result in xfail fail xpass skip error; do
+ cat $result.log # For debugging.
+ $FGREP "$pmarker $result $pmarker" my.log || st=1
+ $FGREP "$cmarker $result $cmarker" my.log || st=1
+done
+test `$FGREP -c "$pmarker" my.log` -eq 5
+test `$FGREP -c "$cmarker" my.log` -eq 5
+
+# Passed test scripts shouldn't be mentioned in the global log.
+$EGREP '(^pass|[^x]pass)\.test' my.log && Exit 1
+# But failing (expectedly or not) and skipped ones should.
+$FGREP 'xfail.test' my.log
+$FGREP 'skip.test' my.log
+$FGREP 'fail.test' my.log
+$FGREP 'xpass.test' my.log
+$FGREP 'error.test' my.log
+
+touch error2.log test-suite.log global.log
+TEST_SUITE_LOG=my.log $MAKE -e mostlyclean
+ls -l # For debugging.
+test ! -f my.log
+test ! -f pass.log
+test ! -f fail.log
+test ! -f xfail.log
+test ! -f xpass.log
+test ! -f skip.log
+test ! -f error.log
+# "make mostlyclean" shouldn't remove unrelated log files.
+test -f error2.log
+test -f test-suite.log
+test -f global.log
+
+rm -f *.log
+
+VERBOSE=yes $MAKE check >stdout && { cat stdout; Exit 1; }
+cat stdout
+cat global.log
+test ! -f my.log
+test ! -f test-suite.log
+# Check that VERBOSE causes the global testsuite log to be
+# emitted on stdout.
+out=`cat stdout`
+log=`cat global.log`
+case $out in *"$log"*) ;; *) Exit 1;; esac
+
+touch error2.log test-suite.log my.log
+$MAKE clean
+ls -l # For debugging.
+test ! -f global.log
+test ! -f pass.log
+test ! -f fail.log
+test ! -f xfail.log
+test ! -f xpass.log
+test ! -f skip.log
+test ! -f error.log
+# "make clean" shouldn't remove unrelated log files.
+test -f error2.log
+test -f test-suite.log
+test -f my.log
+
+rm -f *.log
+
+:
if test ! -s $tmp_status; then
global_result=PASS
+ recheck=no
+ copy_in_global_log=no
else
global_result=FAIL
+ recheck=yes
+ copy_in_global_log=yes
fi
## Write the log file.
{
echo "$global_result: $test_name"
echo "$global_result: $test_name" | sed 's/./=/g'
+ # Blank lines required for having the output be valid RST.
+ echo
+ echo ":recheck: $recheck"
+ echo
+ echo ":copy-in-global-log: $copy_in_global_log"
echo
cat $tmp_results
echo
projects / thirdparty / automake.git / commitdiff
