From: Stefano Lattarini Date: Tue, 26 Jul 2011 12:51:20 +0000 (+0200) Subject: test harness: allow more metadata in log files X-Git-Tag: ng-0.5a~89^2~155^2~12 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=4014b502e2180c1c1ea501ef77e5f97c186d4cbe;p=thirdparty%2Fautomake.git test harness: allow more metadata in log files This change reworks and improves the parallel test harness to use more specialized reStructuredText fields in the log files (instead of relying on specially-placed of "magic lines" and more indirect semantical formatting); the new fields are the following: - ":recheck:": tell whether the associated test will have to be re-run by "make recheck"; - ":copy-in-global-log:": tell whether the content of the log file should be copied in the "global log" `test-suite.log'; - ":end-metadata:", which inhibits the scanning of the rest of the log file (for what concerns test metadata). Also, the special `:test-result:' value "END" has been removed, superseded by the new `:end-metadata:' field. * doc/automake.texi (Log files generation and test results recording): Document the new API and semantics. Remove or fix some obsolete comments. * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html): Adjust comments and code. * lib/tap-driver (finish): Adjust, with the help of ... (must_recheck, copy_in_global_log): ... these new functions. * lib/test-driver (Main code): Adjust, with the help of ... ($recheck, $gcopy): ... these new variables. * tests/trivial-test-driver: Update to obey the new APIs. * tests/test-driver-recheck.test: Likewise. * tests/test-driver-global-log.test: Likewise. * tests/tap-passthrough.test: Relax the test, avoiding to check what is written in `test-suite.log'; such check has been moved ... * tests/tap-global-log.test: ... in this new test, and extended. * tests/test-driver-metadata-no-leading-space.test: New test. * tests/test-driver-end-test-results.test: Removed, it checked the old APIs; superseded by ... * tests/test-driver-end-metadata.test: ... this new test. * tests/tap-log.test: Improve syncing with ... * tests/test-log.test: ... this new test. * tests/parallel-tests.test: Remove some duplication w.r.t. this last new test. Updated heading comments. * tests/Makefile.am (TESTS): Update. --- diff --git a/ChangeLog b/ChangeLog index 89b5059db..afc41347d 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,43 @@ +2011-07-27 Stefano Lattarini + + test harness: allow more metadata in log files + This change reworks and improves the parallel test harness to use + more specialized reStructuredText fields in the log files (instead + of relying on specially-placed of "magic lines" and more indirect + semantical formatting); the new fields are the following: + - ":recheck:": tell whether the associated test will have to be + re-run by "make recheck"; + - ":copy-in-global-log:": tell whether the content of the log + file should be copied in the "global log" `test-suite.log'; + - ":end-metadata:", which inhibits the scanning of the rest of + the log file (for what concerns test metadata). + Also, the special `:test-result:' value "END" has been removed, + superseded by the new `:end-metadata:' field. + * doc/automake.texi (Log files generation and test results + recording): Document the new API and semantics. Remove or fix + some obsolete comments. + * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html): + Adjust comments and code. + * lib/tap-driver (finish): Adjust, with the help of ... + (must_recheck, copy_in_global_log): ... these new functions. + * lib/test-driver (Main code): Adjust, with the help of ... + ($recheck, $gcopy): ... these new variables. + * tests/trivial-test-driver: Update to obey the new APIs. + * tests/test-driver-recheck.test: Likewise. + * tests/test-driver-global-log.test: Likewise. + * tests/tap-passthrough.test: Relax the test, avoiding to check + what is written in `test-suite.log'; such check has been moved ... + * tests/tap-global-log.test: ... in this new test, and extended. + * tests/test-driver-metadata-no-leading-space.test: New test. + * tests/test-driver-end-test-results.test: Removed, it checked + the old APIs; superseded by ... + * tests/test-driver-end-metadata.test: ... this new test. + * tests/tap-log.test: Improve syncing with ... + * tests/test-log.test: ... this new test. + * tests/parallel-tests.test: Remove some duplication w.r.t. this + last new test. Updated heading comments. + * tests/Makefile.am (TESTS): Update. + 2011-07-27 Stefano Lattarini testsuite: fix a spurious failure with non-bash shells diff --git a/doc/automake.texi b/doc/automake.texi index e1a1ca320..2e5e2bad7 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -9372,41 +9372,26 @@ consistency and a more pleasant user experience. The test driver must correctly generate the file specified by the @option{--log-file} option (of course even when the tested program -fails or crashes). This file is quite free-form, but still it has -to conform to the following conventions, in order to work with the -@option{parallel-tests} harness and take advantage of its features. +fails or crashes). This file, while being quite free-form, @emph{must} +register some metadata through the use of custom reStructuredText fields, +in order to work with the @option{parallel-tests} harness and take +advantage of its features. Note that such fields will be recognized only +when used @emph{exactly} at the beginning of a line -- not even leading +whitespace is permitted. -@itemize @bullet -@item -@cindex Global test script result -@cindex Test result, global -The @emph{global test result} for the whole test script must be placed -in the first line of the @file{.log} file. -@c The following semantics is checked by tests 'test-driver-recheck.test' -@c and 'test-driver-global-log.test'. Please keep them in sync whenever -@c the following is changed. -If this line does not begin with either @code{PASS:}, @code{XFAIL:} or -@code{SKIP:}, the test script will be considered failed, and it will be -re-run by @code{make recheck}. Also, if this line does not begin with -either @code{PASS:} or @code{XFAIL:}, the content of the @file{.log} file -will be copied into the global @file{test-suite.log} (in order to help in -debugging and bug-report analysis). +@table @code -@item -@cindex Single test case result -@cindex Test result, single -@cindex @code{:test-result:} reStructuredText field -@cindex reStructuredText field, @code{:test-result:} -One of the main features of the new testsuite harness is the ability to -support test protocols that allow a single test script to run more -test cases, @emph{each with its distinct result}. In order for the -testsuite summary to be correct in this case, the test driver must -register @emph{each} such result in the generated @file{.log} file, using -the @code{:test-result:} reStructuredText field. Otherwise, only the -global test result will be considered (as it is the case of the default -@option{parallel-tests} driver). Note that such a field will be -recognized only when used @emph{exactly} at the beginning of a line: -leading whitespace will not be ignored. +@item :test-result: +@cindex Register test result +@cindex Register test case result +@cindex Test result, registering +@cindex Test case result, registering +@cindex @code{:test-result:} +@cindex reStructuredText field, @code{:test-result:} +The test driver must use this filed to register the results of @emph{each} +test case run by a test script file. Several @code{:test-result:} fields +can be present in the same log file; this is done in order to support test +protocols that allow a single test script to run more test cases. @c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG). The only recognized test results are currently @code{PASS}, @code{XFAIL}, @@ -9415,48 +9400,63 @@ when declared with @code{:test-result:}, can be optionally followed by text holding the name and/or a brief description of the corresponding test; the @option{parallel-tests} harness will ignore such extra text when generating @file{test-suite.log} and preparing the testsuite summary. -Also, @code{:test-result:} can be used with a special ``pseudo-result'' -@code{END}, that will instruct the testsuite harness to stop scanning -the rest of the log file (this feature is used internally by the default -@option{parallel-tests} driver, and can be useful to avoid spurious -testsuite results). -Let's see a small example. If a @file{.log} file contains the following -lines (exactly in this order, but possibly interleaved with other -``non-@code{:test-results:}'' lines): +@c Keep in sync with 'test-driver-recheck.test'. +@item @code{:recheck:} +@cindex :recheck: +@cindex reStructuredText field, @code{:recheck:} +If this field is present and defined to @code{no}, then the corresponding +test script will @emph{not} be run upon a @command{make recheck}. What +happens when two or more @code{:recheck:} fields are present in a +@file{.log} file is undefined behaviour. + +@c Keep in sync with 'test-driver-global-log.test'. +@item @code{:copy-in-global-log:} +@cindex :copy-in-global-log: +@cindex reStructuredText field, @code{:copy-in-global-log:} +If this field is present and defined to @code{no}, then the content +of the @file{.log} file will @emph{not} be copied into the global +@file{test-suite.log}. We allow to forsake such copying because, while +it can be useful in debugging and analysis of bug report, it can also be +just a waste of space in normal situations, e.g., when a test script is +successful. What happens when two or more @code{:copy-in-global-log:} +fields are present in a @file{.log} file is undefined behaviour. + +@c Keep in sync with 'test-driver-end-metadata.test'. +@item @code{:end-metadata:} +@cindex :end-metadata: +@cindex reStructuredText field, @code{:end-metadata:} +This special field causes all the following text in the @file{.log} +file not to be scanned by the testsuite harness in search for other +test metadata. This feature is used internally by the default +@option{parallel-tests} driver, and can be useful to e.g., avoid +spurious testsuite results. + +@end table + +@noindent +Let's see a small example. Assume a @file{.log} file contains the +following lines (in this order, but possibly interleaved with other +``non-metadata'' lines): @example :test-result: PASS server starts :test-result: PASS HTTP/1.1 request :test-result: FAIL HTTP/1.0 request +:recheck: yes :test-result: SKIP HTTPS request (TLS library wasn't available) :test-result: PASS server stops -:test-result: END +:global-log-copy: no +:metadata-end: :test-result: XPASS (this shouldn't be parsed as a test result) @end example @noindent -it means that the corresponding test script contributes with @emph{five} -test results to the testsuite summary (three of these tests being -successful, one failed, and one skipped). - -@c FIXME: currently, the @code{:test-result:} field is recognized anywhere -@c in the @file{.log} file; this is possibly prone to generating spurious -@c results, in case of verbose tests. We should allow a special -@c @code{:test-result:} that stops any further parsing (maybe a line -@c @code{:test-result:END} will do?). - -@end itemize - -Finally, note that the declaration of the global test result in the first -line, apart from being needed for backward-compatibility with the default -@option{parallel-tests} driver, can be useful also for test protocols -that allow more test results per test script: using it, a custom test -driver is allowed to decide what the global outcome of the test script -is in case of conflicting test results within the script. For example, -if a test script reports 8 successful test cases and 2 skipped test -cases, some drivers might report that globally as a SKIP, while others -as a PASS. +Then the corresponding test script will be re-run by @command{make check}, +will contribute with @emph{five} test results to the testsuite summary +(three of these tests being successful, one failed, and one skipped), and +the @file{.log} file content will @emph{not} be copied in the global log +file @file{test-suite.log}. @node Testsuite progress output @subsubsection Testsuite progress output diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in index 3777b5d96..9b45aa23e 100644 --- a/lib/Automake/tests/Makefile.in +++ b/lib/Automake/tests/Makefile.in @@ -331,8 +331,10 @@ cscope cscopelist: $(TEST_SUITE_LOG): $(TEST_LOGS) @$(am__sh_e_setup); $(am__tty_colors); \ - rst_magic=":test-result:"; \ + tab=' '; \ nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \ + IFS=" $$tab$$nl"; oIFS=$$IFS; \ + ws="[ $$tab]"; \ list='$(TEST_LOGS)'; \ list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \ results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \ @@ -340,12 +342,19 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) exec 5<&0; \ for f in $$list2; do \ exec 0<$$f; \ + IFS=''; \ while read line; do \ - case $$line in $$rst_magic*);; *) continue;; esac; \ - res=`expr "x$$line" : "x$$rst_magic[ ]*\\\\(.*\\\\)$$"`; \ - case $$res in END*) break;; esac; \ - results2="$$results2$$nl$$res"; \ + IFS=$$oIFS; \ + case $$line in \ + :end-metadata:*) \ + break;; \ + :test-result:*) \ + res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \ + results2="$$results2$$nl$$res";; \ + esac; \ + IFS=''; \ done; \ + IFS=$$oIFS; \ exec 0<&5; \ done; \ exec 5<&-; \ @@ -399,11 +408,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) echo ".. contents:: :depth: 2"; \ echo; \ for f in $$list; do \ - test -r $$f && read line < $$f || line=; \ - case $$line in \ - PASS:*|XFAIL:*);; \ - *) echo; cat $$f;; \ - esac; \ + sed -e '/^:end-metadata:/,$$d' $$f | \ + grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \ + || { echo && cat $$f; } \ done; \ } >$(TEST_SUITE_LOG).tmp; \ mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \ @@ -462,13 +469,14 @@ check-html: $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4; \ exit $$rv recheck recheck-html: - @target=`echo $@ | sed 's,^re,,'`; \ + @ws='[ ]'; \ + target=`echo $@ | sed 's,^re,,'`; \ list='$(TEST_LOGS)'; \ - list=`for f in $$list; do \ - test -f $$f || continue; \ - if test -r $$f && read line < $$f; then \ - case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \ - else echo $$f; fi; \ + list=`for f in $$list; do \ + test -f $$f || continue; \ + sed -e '/^:end-metadata:/,$$d' $$f | \ + grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \ + || echo $$f; \ done | tr '\012\015' ' '`; \ list=`echo "$$list" | sed 's/ *$$//'`; \ $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"' diff --git a/lib/am/check.am b/lib/am/check.am index ffb70bab1..272fc0a85 100644 --- a/lib/am/check.am +++ b/lib/am/check.am @@ -126,14 +126,14 @@ $(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT) $(TEST_SUITE_LOG): $(TEST_LOGS) @$(am__sh_e_setup); $(am__tty_colors); \ -## The custom reStructuredText field used to register the outcome of -## a test. This is for supporting test protocols, such as TAP and -## SubUnit, where a single test script can run multiple tests, each -## with its own outcome. - rst_magic=":test-result:"; \ + tab=' '; \ ## The following line will define the `$nl' shell variable to a literal ## newline. Idiom taken from the Autoconf manual. nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \ +## Sanitize internal field separator for the shell: "space tab newline". + IFS=" $$tab$$nl"; oIFS=$$IFS; \ +## Regular expression to match a whitespace. + ws="[ $$tab]"; \ ## All test logs. list='$(TEST_LOGS)'; \ ## Readable test logs. @@ -148,19 +148,27 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) exec 5<&0; \ for f in $$list2; do \ exec 0<$$f; \ +## We want to support special reStructuredText fields only when they're +## placed at the beginning of a line (as documented in the manual), so +## we have to temporarily clear the $IFS variable to prevent automatic +## whitespace stripping by the shell. + IFS=''; \ while read line; do \ -## Ignore lines that do not declare test results. - case $$line in $$rst_magic*);; *) continue;; esac; \ -## The test result declared by this line goes in $res. - res=`expr "x$$line" : "x$$rst_magic[ ]*\\\\(.*\\\\)$$"`; \ -## A special result of "END" means that the rest of the log is not to be -## scanned. Be a little lax in matching such special result, to allow -## for e.g., trailing whitespace or punctuation. - case $$res in END*) break;; esac; \ -## Otherwise register the result. - results2="$$results2$$nl$$res"; \ + IFS=$$oIFS; \ + case $$line in \ +## Special field that tells not to scan the rest of the log file. + :end-metadata:*) \ + break;; \ +## A test test result -- register it. + :test-result:*) \ + res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \ + results2="$$results2$$nl$$res";; \ + esac; \ +## For the next loop. + IFS=''; \ done; \ -## Restore original standard input. +## Restore original standard input and internal field separator. + IFS=$$oIFS; \ exec 0<&5; \ done; \ ## We don't need this anymore. @@ -225,14 +233,11 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) echo; \ echo ".. contents:: :depth: 2"; \ echo; \ -## Here we assume that the test driver writes a proper "summarizing -## result" for the test script on the first line. for f in $$list; do \ - test -r $$f && read line < $$f || line=; \ - case $$line in \ - PASS:*|XFAIL:*);; \ - *) echo; cat $$f;; \ - esac; \ +## FIXME: two forks per test -- this is horrendously inefficient! + sed -e '/^:end-metadata:/,$$d' $$f | \ + grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \ + || { echo && cat $$f; } \ done; \ } >$(TEST_SUITE_LOG).tmp; \ mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \ @@ -326,20 +331,17 @@ AM_RECURSIVE_TARGETS += check-html ## Rechecking failures. ## ## -------------------- ## -## Rerun all FAILed or XPASSed tests. +## Rerun all tests that experienced an error or an unexpected failure. recheck recheck-html: - @target=`echo $@ | sed 's,^re,,'`; \ + @ws='[ ]'; \ + target=`echo $@ | sed 's,^re,,'`; \ list='$(TEST_LOGS)'; \ - list=`for f in $$list; do \ - test -f $$f || continue; \ - if test -r $$f && read line < $$f; then \ -## Here we assume that the test driver writes a proper summary for the -## test script on the first line. See the comments in the rules of -## $(TEST_SUITE_LOG) above for why we consider this acceptable and even -## advisable. - case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \ -## A test whose log is unreadable is to be considered failed. - else echo $$f; fi; \ + list=`for f in $$list; do \ + test -f $$f || continue; \ +## FIXME: two forks per test -- this is horrendously inefficient! + sed -e '/^:end-metadata:/,$$d' $$f | \ + grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \ + || echo $$f; \ done | tr '\012\015' ' '`; \ ## This apparently useless munging helps to avoid a nasty bug (a ## segmentation fault!) on Solaris XPG4 make. diff --git a/lib/tap-driver b/lib/tap-driver index 69533b495..9075d35ce 100755 --- a/lib/tap-driver +++ b/lib/tap-driver @@ -138,6 +138,19 @@ TEST_RESULTS : return @test_results; } + # Whether the test script should be re-run by "make recheck". + sub must_recheck () + { + return grep { !/^(?:XFAIL|PASS|SKIP)$/ } (keys %test_results); + } + + # Whether the content of the log file associated to this test should + # be copied into the "global" test-suite.log. + sub copy_in_global_log () + { + return grep { not $_ eq "PASS" } (keys %test_results); + } + # FIXME: this can certainly be improved ... sub get_global_test_result () { @@ -168,25 +181,31 @@ sub finish () { open LOG, ">", $log_file or die "opening $log_file: $!\n"; - # We need to declare a global test result in order to have - # "make recheck" working. + # Some extra trailing empty lines outputted below are required + # to support reStructuredText -> HTML conversion. + my $global_result = get_global_test_result; my $global_result_line = "$global_result: $test_script_name"; print LOG "$global_result_line\n"; print LOG "=" x length ($global_result_line) . "\n"; print LOG "\n"; + print LOG ":recheck: " . (must_recheck ? "yes" : "no") . "\n"; + print LOG "\n"; + + print LOG ":copy-in-global-log: " . + (copy_in_global_log ? "yes" : "no") . "\n"; + print LOG "\n"; + foreach (get_test_results) { print LOG ":test-result: $_\n"; - # Add extra trailing empty lines to support reStructuredText -> HTML - # conversion. print LOG "\n"; } - # So that the output from the test script won't be parsed for test - # results (which would potentially expose us to false positives). - print LOG ":test-result: END\n"; + # So that the output from the test script won't be parsed for + # testsuite metadata. + print LOG ":end-metadata:\n"; print LOG "\n"; close TMP or die "closing $log_file-t: $!\n"; diff --git a/lib/test-driver b/lib/test-driver index aa39c5be6..45b3afe37 100755 --- a/lib/test-driver +++ b/lib/test-driver @@ -1,7 +1,7 @@ #! /bin/sh # test-driver - basic driver script for the `parallel-tests' mode. -scriptversion=2011-06-29.13; # UTC +scriptversion=2011-07-27.12; # UTC # Copyright (C) 2011 Free Software Foundation, Inc. # @@ -107,12 +107,12 @@ if test $enable_hard_errors = no && test $estatus -eq 99; then fi case $estatus:$expect_failure in - 0:yes) col=$red; res=XPASS;; - 0:*) col=$grn; res=PASS ;; - 77:*) col=$blu; res=SKIP ;; - 99:*) col=$mgn; res=ERROR;; - *:yes) col=$lgn; res=XFAIL;; - *:*) col=$red; res=FAIL ;; + 0:yes) col=$red res=XPASS recheck=yes gcopy=yes;; + 0:*) col=$grn res=PASS recheck=no gcopy=no;; + 77:*) col=$blu res=SKIP recheck=no gcopy=yes;; + 99:*) col=$mgn res=ERROR recheck=yes gcopy=yes;; + *:yes) col=$lgn res=XFAIL recheck=no gcopy=yes;; + *:*) col=$red res=FAIL recheck=yes gcopy=yes;; esac # Report outcome to console. @@ -121,12 +121,20 @@ echo "${col}${res}${std}: $test_name" # Now write log file. { echo "$res: $test_name (exit: $estatus)" | rst_section + # Blank lines required for having the output be valid RST. + echo echo ":test-result: $res (exit status: $estatus)" echo - echo ":test-result:END" + echo ":recheck: $recheck" + echo + echo ":copy-in-global-log: $gcopy" + echo + # So that the output from the test script won't be parsed for + # testsuite metadata. + echo ":end-metadata:" echo # Use a reStructuredText transition to better separate the test - # outcome report from its registered output. + # metadata the test registered output. echo ------------ echo cat $tmpfile diff --git a/tests/Makefile.am b/tests/Makefile.am index f32205405..9a0c22279 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -743,7 +743,8 @@ testsuite-summary-color.test \ testsuite-summary-count.test \ testsuite-summary-count-many.test \ testsuite-summary-reference-log.test \ -test-driver-end-test-results.test \ +test-driver-end-metadata.test \ +test-driver-metadata-no-leading-space.test \ test-driver-custom-no-extra-driver.test \ test-driver-custom.test \ test-driver-custom-xfail-tests.test \ @@ -757,6 +758,7 @@ test-driver-global-log.test \ test-driver-recheck.test \ test-driver-strip-vpath.test \ test-driver-fail.test \ +test-log.test \ parse.test \ percent.test \ percent2.test \ @@ -1001,6 +1003,7 @@ tap-exit.test \ tap-signal.test \ tap-fancy.test \ tap-fancy2.test \ +tap-global-log.test \ tap-global-result.test \ tap-html.test \ tap-log.test \ diff --git a/tests/Makefile.in b/tests/Makefile.in index 0ce3dd906..13e850cba 100644 --- a/tests/Makefile.in +++ b/tests/Makefile.in @@ -988,7 +988,8 @@ testsuite-summary-color.test \ testsuite-summary-count.test \ testsuite-summary-count-many.test \ testsuite-summary-reference-log.test \ -test-driver-end-test-results.test \ +test-driver-end-metadata.test \ +test-driver-metadata-no-leading-space.test \ test-driver-custom-no-extra-driver.test \ test-driver-custom.test \ test-driver-custom-xfail-tests.test \ @@ -1002,6 +1003,7 @@ test-driver-global-log.test \ test-driver-recheck.test \ test-driver-strip-vpath.test \ test-driver-fail.test \ +test-log.test \ parse.test \ percent.test \ percent2.test \ @@ -1246,6 +1248,7 @@ tap-exit.test \ tap-signal.test \ tap-fancy.test \ tap-fancy2.test \ +tap-global-log.test \ tap-global-result.test \ tap-html.test \ tap-log.test \ @@ -1441,8 +1444,10 @@ cscope cscopelist: $(TEST_SUITE_LOG): $(TEST_LOGS) @$(am__sh_e_setup); $(am__tty_colors); \ - rst_magic=":test-result:"; \ + tab=' '; \ nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \ + IFS=" $$tab$$nl"; oIFS=$$IFS; \ + ws="[ $$tab]"; \ list='$(TEST_LOGS)'; \ list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \ results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \ @@ -1450,12 +1455,19 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) exec 5<&0; \ for f in $$list2; do \ exec 0<$$f; \ + IFS=''; \ while read line; do \ - case $$line in $$rst_magic*);; *) continue;; esac; \ - res=`expr "x$$line" : "x$$rst_magic[ ]*\\\\(.*\\\\)$$"`; \ - case $$res in END*) break;; esac; \ - results2="$$results2$$nl$$res"; \ + IFS=$$oIFS; \ + case $$line in \ + :end-metadata:*) \ + break;; \ + :test-result:*) \ + res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \ + results2="$$results2$$nl$$res";; \ + esac; \ + IFS=''; \ done; \ + IFS=$$oIFS; \ exec 0<&5; \ done; \ exec 5<&-; \ @@ -1509,11 +1521,9 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) echo ".. contents:: :depth: 2"; \ echo; \ for f in $$list; do \ - test -r $$f && read line < $$f || line=; \ - case $$line in \ - PASS:*|XFAIL:*);; \ - *) echo; cat $$f;; \ - esac; \ + sed -e '/^:end-metadata:/,$$d' $$f | \ + grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \ + || { echo && cat $$f; } \ done; \ } >$(TEST_SUITE_LOG).tmp; \ mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \ @@ -1572,13 +1582,14 @@ check-html: $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_HTML) || exit 4; \ exit $$rv recheck recheck-html: - @target=`echo $@ | sed 's,^re,,'`; \ + @ws='[ ]'; \ + target=`echo $@ | sed 's,^re,,'`; \ list='$(TEST_LOGS)'; \ - list=`for f in $$list; do \ - test -f $$f || continue; \ - if test -r $$f && read line < $$f; then \ - case $$line in PASS:*|XFAIL:*|SKIP:*);; *) echo $$f;; esac; \ - else echo $$f; fi; \ + list=`for f in $$list; do \ + test -f $$f || continue; \ + sed -e '/^:end-metadata:/,$$d' $$f | \ + grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \ + || echo $$f; \ done | tr '\012\015' ' '`; \ list=`echo "$$list" | sed 's/ *$$//'`; \ $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"' diff --git a/tests/parallel-tests.test b/tests/parallel-tests.test index b67922f71..2d9e2a225 100755 --- a/tests/parallel-tests.test +++ b/tests/parallel-tests.test @@ -14,14 +14,14 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# Check parallel-tests features: -# - VERBOSE -# - clean -# - TEST_SUITE_LOG -# - dependencies between tests -# - TESTS -# - TEST_LOGS -# - RECHECK_LOGS +# Basic checks on parallel-tests support: +# - console output +# - log files, and what goes in 'test-suite.log' +# - make clean +# - dependencies between tests +# - TESTS redefinition at runtime +# - TEST_LOGS redefinition at runtime +# - RECHECK_LOGS redefinition at runtime parallel_tests=yes . ./defs || Exit 1 @@ -31,7 +31,6 @@ AC_OUTPUT END cat > Makefile.am << 'END' -TEST_SUITE_LOG = mylog.log TESTS = foo.test bar.test baz.test XFAIL_TESTS = bar.test foo.log: bar.log @@ -69,23 +68,20 @@ $AUTOMAKE -a $MAKE check >stdout && { cat stdout; Exit 1; } cat stdout count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1 -test -f mylog.log -cat mylog.log -test `grep -c '^FAIL:' mylog.log` -eq 1 -test `grep -c '^ERROR:' mylog.log` -eq 1 -$EGREP '^(X?PASS|XFAIL|SKIP)' mylog.log && Exit 1 +test -f test-suite.log +cat test-suite.log +test `grep -c '^FAIL:' test-suite.log` -eq 1 +test `grep -c '^ERROR:' test-suite.log` -eq 1 +$EGREP '^(X?PASS|XFAIL|SKIP)' test-suite.log && Exit 1 test -f baz.log test -f bar.log test -f foo.log -# clean should remove all log files (but not more). -: > unrelated.log $MAKE clean test ! -f baz.log test ! -f bar.log test ! -f foo.log -test ! -f mylog.log -test -f unrelated.log +test ! -f test-suite.log # Check dependencies: baz.test needs to run before bar.test, # but foo.test is not needed. @@ -100,7 +96,7 @@ grep '^ERROR: bar\.test$' stdout test -f baz.log test -f bar.log test ! -f foo.log -test -f mylog.log +test -f test-suite.log # Upon a lazy rerun, foo.test should be run, but the others shouldn't. # Note that the lazy rerun still exits with a failure, due to the previous @@ -139,12 +135,6 @@ grep '^# PASS: *1$' stdout grep '^# FAIL: *1$' stdout grep '^# ERROR: *1$' stdout -# Test VERBOSE. -env VERBOSE=yes $MAKE -e check > stdout && { cat stdout; Exit 1; } -cat stdout -grep 'this is.*bar.test' stdout -grep 'this is.*baz.test' stdout - $MAKE clean env TEST_LOGS=baz.log $MAKE -e check > stdout && { cat stdout; Exit 1; } cat stdout diff --git a/tests/tap-global-log.test b/tests/tap-global-log.test new file mode 100755 index 000000000..da5aeb8bd --- /dev/null +++ b/tests/tap-global-log.test @@ -0,0 +1,140 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# TAP support: +# - which log files get copied in the global log? + +parallel_tests=yes +. ./defs || Exit 1 + +cp "$top_testsrcdir"/lib/tap-driver . \ + || fatal_ "failed to fetch auxiliary script tap-driver" + +cat >> configure.in < ok.test << 'END' +1..5 +ok 1 +ok 2 +ok 3 +not seen in global log +ok 4 +ok 5 +END + +cat > top << 'END' +1..6 +ok 1 +ok 2 +ok 3 +END + +cat > bot << 'END' +ok 5 +ok 6 +END + +cat top - bot > skip.test << 'END' +ok # SKIP +::skip:: +END + +cat top - bot > todo.test << 'END' +not ok # TODO +::todo:: +END + +cat top - bot > fail.test << 'END' +not ok +::fail:: +END + +cat top - bot > xpass.test << 'END' +ok # TODO +::xpass:: +END + +cat top - bot > bail.test << 'END' +::bail:: +Bail out! +END + +cat top - bot > error.test << 'END' +::error:: +1..7 +END + +# Created with "dd if=/dev/urandom count=1 | base64-encode" +cat > hodgepodge <<'END' +1+0 records in +1+0 records out +512 bytes (512 B) copied, 0.000241092 s, 2.1 MB/s +gdivnV4VhL4DOzhE3zULJuun3PwqqQqMdATVcZbIQkNgyRvNBoqqHMBQEs7QsjDbp2nK+Szz +EcelGyvXmHrW7yImaom6Yrg95k31VCmp/pGDRnTDwDrcOPJiv9jDReEmTAQkPuqLO+mFNly+ +DDHM9fNbzGumstsQ3wq3DOXz1pCV3JXwhjeaHSboeEbmr55bX0XHLSKaecctA0rXDXEyZWZ/ +ODlawSrAXzw0H7a+xBwjnNXZ3zYiwk3x+WQrPqNjb+qXiLLTxAKzx2/KnaFhxkPlte5jPRNB +FciDolL+H/10LsCdSzLOrGnY2zH6vL2JMZfxjnb73zWFcdWWE01LTD7wpN5O1MP3+N47lcVe +lWbkD04cJvhwxLElYSO24B743GG5EyGYt9SeZRE6xbgwq3fVOS8KqjwGxwi4adSBTtw0CV8W +S/6n8dck1vBvjA+qpk0zMSYSqc3+jzW9UiGTmTEIwfw80p/lGNsfjcNBJ86nFkWUnHmrsi8k +Dv57sK70mTg239g08f5Uvdga/5UreMBSgB0hUj5sbq57r7B1fsVr0Kag468la8zKy3ZEZ0gX +++sbaJ9WGHhnKvjooeH+4Y6HwAFsdINde++FlCmp4EuNKKEEuXbSKLaOTy3+6pJ2DYdvRCL+ +frZwxH4hcrw8qh+8IakB02viewZS/qT57v4= +END + +exec 5>misc.test +echo 'ok # SKIP' >&5 +cat hodgepodge >&5 +echo 'not ok # TODO' >&5 +echo 'ok' >&5 +exec 5>&- + +cat > skipall.test << 'END' +1..0 # SKIP all +END + +cat > Makefile.am << 'END' +TEST_LOG_DRIVER = $(PERL) $(srcdir)/tap-driver +TEST_LOG_COMPILER = cat +END +echo TESTS = *.test >> Makefile.am + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +# We don't care about the exit status of "make check" here. +$MAKE check || : +cat test-suite.log + +grep 'ok\.test|not seen' test-suite.log && Exit 1 + +for s in skip todo fail xpass bail error; do + $FGREP "::$s::" test-suite.log +done + +grep '^1\.\.0 # SKIP all$' test-suite.log + +case `cat test-suite.log` in + *"`cat hodgepodge`"*) ;; + *) Exit 1;; +esac + +: diff --git a/tests/tap-log.test b/tests/tap-log.test index 37b5956ad..e5bab2f47 100755 --- a/tests/tap-log.test +++ b/tests/tap-log.test @@ -17,9 +17,10 @@ # TAP support: # - log file creation # - log file removal -# - stdout and stderr of a TAP-generating test script go in its log file +# - stdout and stderr of a script go in its log file # - TEST_SUITE_LOG redefinition, at either automake or make time # - VERBOSE environment variable support +# Keep in sync with 'test-log.test'. parallel_tests=yes . ./defs || Exit 1 @@ -94,7 +95,7 @@ chmod a+x *.test $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a ./configure @@ -111,17 +112,18 @@ for result in pass fail xfail xpass skip error; do done test $st -eq 0 || Exit 1 cat my.log # For debugging. -for result in fail xpass skip error; do +for result in xfail fail xpass skip error; do cat $result.log # For debugging. $FGREP "$pmarker $result $pmarker" my.log || st=1 $FGREP "$cmarker $result $cmarker" my.log || st=1 done -test `$FGREP -c "$pmarker" my.log` -eq 4 -test `$FGREP -c "$cmarker" my.log` -eq 4 +test `$FGREP -c "$pmarker" my.log` -eq 5 +test `$FGREP -c "$cmarker" my.log` -eq 5 -# Successful test scripts shouldn't be mentioned in the global log. -$EGREP '(^pass|[^x]pass|xfail)\.test' my.log && Exit 1 -# But failing and skipped one should. +# Passed test scripts shouldn't be mentioned in the global log. +$EGREP '(^pass|[^x]pass)\.test' my.log && Exit 1 +# But failing (expectedly or not) and skipped ones should. +$FGREP 'xfail.test' my.log $FGREP 'skip.test' my.log $FGREP 'fail.test' my.log $FGREP 'xpass.test' my.log @@ -137,6 +139,7 @@ test ! -f xfail.log test ! -f xpass.log test ! -f skip.log test ! -f error.log +# "make mostlyclean" shouldn't remove unrelated log files. test -f error2.log test -f test-suite.log test -f global.log @@ -155,7 +158,7 @@ log=`cat global.log` case $out in *"$log"*) ;; *) Exit 1;; esac touch error2.log test-suite.log my.log -$MAKE mostlyclean +$MAKE clean ls -l # For debugging. test ! -f global.log test ! -f pass.log @@ -164,11 +167,11 @@ test ! -f xfail.log test ! -f xpass.log test ! -f skip.log test ! -f error.log +# "make clean" shouldn't remove unrelated log files. test -f error2.log test -f test-suite.log test -f my.log rm -f *.log - : diff --git a/tests/tap-passthrough.test b/tests/tap-passthrough.test index ab9b3e55a..b4a7c127a 100755 --- a/tests/tap-passthrough.test +++ b/tests/tap-passthrough.test @@ -62,15 +62,11 @@ ok 4 - x # some diagnostic${tab} not ok # TODO low priority ok # SKIP who cares? -This is not in test-suite.log, since ok.test does not fail. $weirdchars END -st=0 -TESTS=ok.test $MAKE -e check || st=$? +TESTS=ok.test $MAKE -e check || { cat ok.log; Exit 1; } cat ok.log -cat test-suite.log -test $st -eq 0 || Exit 1 for rx in \ '1\.\.6' \ @@ -83,15 +79,11 @@ for rx in \ "# some diagnostic${tab}" \ 'not ok # TODO low priority' \ 'ok # SKIP who cares?' \ - 'This is not in test-suite\.log, since ok\.test does not fail\.' \ ; do grep "^$rx$" ok.log done $FGREP "$weirdchars" ok.log -$FGREP 'ok.test' test-suite.log && Exit 1 -$FGREP 'not in test-suite.log' test-suite.log && Exit 1 - # # Mixed failing/successful tests. # @@ -104,7 +96,7 @@ END cat > ok.test < ko.test <. + +# Custom test drivers and parallel-tests harness: check the documented +# semantics for inhibiting the scanning of the remaining portion of a +# `.log' file (for what concerns metadata). Currently, this is done +# with the use of the reStructuredText field `:end-metadata:'. + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in << 'END' +AC_OUTPUT +END + +cat > Makefile.am << 'END' +TEST_LOG_DRIVER = ./passthrough-driver +TESTS = foo.test bar.test baz.test +END + +cat > passthrough-driver <<'END' +#!/bin/sh +set -e; set -u; +while test $# -gt 0; do + case $1 in + --log-file) log_file=$2; shift;; + --test-name) test_name=$2; shift;; + --expect-failure|--color-tests|--enable-hard-errors) shift;; + --) shift; break;; + *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;; + esac + shift +done +echo "$test_name: RUN" +cp $1 $log_file +END +chmod a+x passthrough-driver + +cat > foo.test < bar.test < baz.test <stdout || st=1 +# For debugging. +cat stdout +cat foo.log +cat bar.log +cat test-suite.log + +# Check that only the `:test-result:' in baz.test and the first one in +# foo.test have been considered. +test $st -eq 0 +grep '^# TOTAL: *2$' stdout +grep '^# PASS: *2$' stdout +for result in SKIP XPASS FAIL XFAIL ERROR; do + grep "^# $result: *0$" stdout +done + +# Check that a `:end-metadata:' directive does not prevent following text +# to be copied into the log file. +$FGREP 'foo foo foo' foo.log +$FGREP 'bar bar bar' bar.log +$FGREP 'baz baz baz' baz.log # More of a sanity check really. + +# Check that only the `:copy-in-global-log:' in bar.test has been +# considered. +$FGREP 'foo foo foo' test-suite.log +$FGREP 'bar bar bar' test-suite.log && Exit 1 +$FGREP 'baz baz baz' test-suite.log && Exit 1 + +# Check that only the `:recheck:' in bar.test has been +# considered. +$MAKE recheck >stdout || { cat stdout; Exit 1; } +cat stdout +grep '^foo\.test: RUN$' stdout +grep 'ba[rz]\.test.*RUN' stdout && Exit 1 + +: diff --git a/tests/test-driver-end-test-results.test b/tests/test-driver-end-test-results.test deleted file mode 100755 index 700870086..000000000 --- a/tests/test-driver-end-test-results.test +++ /dev/null @@ -1,95 +0,0 @@ -#! /bin/sh -# Copyright (C) 2011 Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2, or (at your option) -# any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -# parallel-tests harness: check that a special ":test-result:END" -# directive prevents the code generating the testsuite summary and -# the test-suite.log from parsing the rest of the log file. - -parallel_tests=yes -. ./defs || Exit 1 - -cat >> configure.in << 'END' -AC_OUTPUT -END - -cat > Makefile.am << 'END' -TEST_LOG_DRIVER = ./dummy-driver -TESTS = a.test b.test c.test d.test -END - -cat > dummy-driver <<'END' -#!/bin/sh -set -e -while test $# -gt 0; do - case $1 in - --log-file) log_file=$2; shift;; - --test-name|--expect-failure|--color-tests|--enable-hard-errors) shift;; - --) shift; break;; - *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;; - esac - shift -done -cp $1 $log_file -END -chmod a+x dummy-driver - -cat > a.test <<'END' -PASS: a.test -:test-result:END -:test-result:FAIL -END - -cat > b.test < c.test < d.test <output 2>&1 || { cat output; Exit 1; } -cat output -cat a.log; cat b.log; cat c.log; cat d.log # For debugging. - -grep '^# TOTAL: *4$' output -grep '^# PASS: *3$' output -grep '^# SKIP: *1$' output -for result in XPASS FAIL XFAIL ERROR; do - grep "^# $result: *0$" output -done - -: diff --git a/tests/test-driver-global-log.test b/tests/test-driver-global-log.test index 35b6fe13c..7596e943f 100755 --- a/tests/test-driver-global-log.test +++ b/tests/test-driver-global-log.test @@ -14,8 +14,10 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# Check that the output of testcases having a global outcome that is not -# "PASS" or "XFAIL" is copied in the global 'test-suite.log'. +# Custom test drivers and parallel-tests harness: check the documented +# semantics for deciding when the content of a test log file should be +# copied in the global test-suite.log file. Currently, this is done +# with the use of the reStructuredText field `:copy-in-global-log:'. parallel_tests=yes . ./defs || Exit 1 @@ -24,36 +26,69 @@ cat >> configure.in << 'END' AC_OUTPUT END -# The :test-result: fields should be irrelevant for the decision -# of whether a test output is to be copied in the test-suite.log. -cat > pass.test < no-1.test < $RES.test < xfail.test <<'END' -XFAIL: xfail.test -not seen (xfail) +# In the last line, with leading and trailing whitespace. +cat > no-2.test < skip.test -echo FAIL: fail.test > fail.test -echo XPASS: xpass.test > xpass.test -echo ERROR: error.test > error.test -echo :test-result: PASS > fake-pass.test -echo "$tab $tab$tab" > empty.test - -cat > Makefile.am << 'END' -TEST_LOG_DRIVER = ./dummy-driver -TESTS = pass.test skip.test fail.test xfail.test xpass.test error.test \ - fake-pass.test empty.test +# In the first line, with no whitespace. +cat > no-3.test < yes-1.test < yes-2.test < corn-1.test +echo " $tab $tab$tab" > corn-2.test +cat > corn-3.test <<'END' +seen corn 31 +:copy-in-global-log:#@%! +seen corn 32 +END + +echo TEST_LOG_DRIVER = ./dummy-driver > Makefile.am +echo TESTS = *.test >> Makefile.am cat > dummy-driver <<'END' #!/bin/sh @@ -78,35 +113,19 @@ $AUTOMAKE ./configure -$MAKE check && { cat test-suite.log; Exit 1; } +# We don't care about the exit status of "make check" here, that +# should be checked in other tests. +$MAKE check || : cat test-suite.log -grep '^SKIP: skip\.test$' test-suite.log -grep '^FAIL: fail.test$' test-suite.log -grep '^XPASS: xpass.test$' test-suite.log -grep '^ERROR: error.test$' test-suite.log -grep '^:test-result: PASS$' test-suite.log -grep "^$tab $tab$tab$" test-suite.log -$EGREP 'not seen' test-suite.log && Exit 1 -test `grep -c ':test-result:' test-suite.log` -eq 1 - -cat > pass.test < xfail.test <. + +# Custom test drivers and parallel-tests harness: check that, in the +# log files, the reStructuredText fields for metadata declaration are +# recognized only when placed at the beginning of a line. + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in << 'END' +AC_OUTPUT +END + +cat > Makefile.am << 'END' +TEST_LOG_DRIVER = ./passthrough-driver +TESTS = foo.test +END + +cat > passthrough-driver <<'END' +#!/bin/sh +set -e; set -u; +while test $# -gt 0; do + case $1 in + --log-file) log_file=$2; shift;; + --test-name) test_name=$2; shift;; + --expect-failure|--color-tests|--enable-hard-errors) shift;; + --) shift; break;; + *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;; + esac + shift +done +echo "$test_name: RUN" +cp $1 $log_file +END +chmod a+x passthrough-driver + +cat > foo.test <stdout || st=1 +# For debugging. +cat stdout +cat foo.log +cat test-suite.log + +# Check that only the second `:test-result:' has been processed. +test $st -eq 0 +grep '^# TOTAL: *1$' stdout +grep '^# PASS: *1$' stdout +for result in SKIP XPASS FAIL XFAIL ERROR; do + grep "^# $result: *0$" stdout +done + +# Check that `:copy-in-global-log:' hasn't been processed. +grep 'foo foo foo' test-suite.log + +# Check that `:recheck:' hasn't been processed. +: > older +$sleep +$MAKE recheck >stdout || { cat stdout; Exit 1; } +cat stdout +grep '^foo\.test: RUN$' stdout +grep 'foo foo foo' test-suite.log +is_newest test-suite.log older +is_newest foo.log older + +: diff --git a/tests/test-driver-recheck.test b/tests/test-driver-recheck.test index f9482b187..a8654c1dc 100755 --- a/tests/test-driver-recheck.test +++ b/tests/test-driver-recheck.test @@ -14,8 +14,10 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# Test the "make recheck" semantics for custom test driver, as -# documented in detail in the Automake manual. +# Test the "make recheck" semantics for custom test drivers, as documented +# in the Automake manual. Currently, that is based on the reStructuredText +# field `:recheck:'. + parallel_tests=yes . ./defs || Exit 1 @@ -30,54 +32,86 @@ TEST_EXTENSIONS = TESTS = END -rechecked='' -notrechecked='' - -for R in PASS XFAIL SKIP; do - echo $R: > $R-1 - echo " $R:" > $R-2 - echo $R:foo > $R-3 - echo "$tab $R: bar baz" > $R-4 - # The :test-result: fields should be irrelevant for the decision of - # whether "make recheck" should or should not re-run a test. - unindent > $R-5 < y-1 +echo "foo bar" > y-2 +echo ":recheck:" > y-3 +echo ":recheck:yes" > y-4 +echo ":recheck: who cares" > y-5 +echo ":recheck: $tab y" > y-6 +echo ":recheck: yeah!$tab$tab " > y-7 +cat > y-10 < BAD-$R-1 - echo $R. > BAD-$R-2 - echo :$R: > BAD-$R-3 - echo $R asd > BAD-$R-4 - rechecked="$rechecked BAD-$R-1 BAD-$R-2 BAD-$R-3 BAD-$R-4" -done +cat > y-11 < y-12 < $R-1 - echo $R:foo > $R-2 - echo $R: bar baz > $R-3 - echo $R > $R-4 - echo $R asd > $R-5 - # The :test-result: fields should be irrelevant for the decision of - # whether "make recheck" should or should not re-run a test. - unindent > $R-6 < $R-7 - rechecked="$rechecked $R-1 $R-2 $R-3 $R-4 $R-5 $R-6 $R-7" -done -: > EMPTY -echo " $tab $tab" > WHITE -rechecked="$rechecked EMPTY WHITE" +# The :test-result: fields and the fist line of the log should be +# irrelevant for the decision of whether "make recheck" should or +# should not re-run a test. + +echo ":test-result: PASS" > y-100 + +echo "PASS: y-101" + +cat > y-102 < n-1 +echo ":recheck: no " > n-2 +echo ":recheck: $tab no" > n-3 +echo ":recheck: no $tab$tab " > n-4 +cat > n-5 < n-6 < n-7 < n-100 <> Makefile.am +for t in [yn]-[0-9]*; do echo $t; done \ + | sed 's/.*/TESTS += &/' >> Makefile.am cat Makefile.am # For debugging. @@ -119,8 +153,7 @@ for iteration in 1 2; do $MAKE recheck ls -l for t in $rechecked; do test -f $t.run; done - for t in $notrechecked; do test ! -r $t.run; done - rm -f *.run + find . -name 'n-*.run' | grep . && Exit 1 done : diff --git a/tests/test-log.test b/tests/test-log.test new file mode 100755 index 000000000..1385386de --- /dev/null +++ b/tests/test-log.test @@ -0,0 +1,168 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check parallel-tests features: +# - log file creation +# - log file removal +# - stdout and stderr of a test script go in its log file +# - TEST_SUITE_LOG redefinition, at either automake or make time +# - VERBOSE environment variable support +# Keep in sync with 'tap-log.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in < Makefile.am << 'END' +TESTS = pass.test skip.test xfail.test fail.test xpass.test error.test +XFAIL_TESTS = xpass.test xfail.test +TEST_SUITE_LOG = global.log +END + +# Custom markers, for use in grepping checks. +cmarker=::: # comment marker +pmarker=%%% # plain maker + +cat > pass.test <&2 +echo "# $cmarker pass $cmarker" >&2 +exit 0 +END + +cat > skip.test < xfail.test <&2 +echo "# $cmarker xfail $cmarker" >&2 +exit 1 +END + +cat > fail.test < xpass.test <&2 +echo "# $cmarker xpass $cmarker" >&2 +exit 0 +END + +cat > error.test <stdout && { cat stdout; Exit 1; } +cat stdout +cat global.log +test ! -f my.log +test ! -f test-suite.log +# Check that VERBOSE causes the global testsuite log to be +# emitted on stdout. +out=`cat stdout` +log=`cat global.log` +case $out in *"$log"*) ;; *) Exit 1;; esac + +touch error2.log test-suite.log my.log +$MAKE clean +ls -l # For debugging. +test ! -f global.log +test ! -f pass.log +test ! -f fail.log +test ! -f xfail.log +test ! -f xpass.log +test ! -f skip.log +test ! -f error.log +# "make clean" shouldn't remove unrelated log files. +test -f error2.log +test -f test-suite.log +test -f my.log + +rm -f *.log + +: diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver index 4b4350688..d334a9365 100644 --- a/tests/trivial-test-driver +++ b/tests/trivial-test-driver @@ -86,8 +86,12 @@ tmp_status=$log_file-status.tmp if test ! -s $tmp_status; then global_result=PASS + recheck=no + copy_in_global_log=no else global_result=FAIL + recheck=yes + copy_in_global_log=yes fi ## Write the log file. @@ -95,6 +99,11 @@ fi { echo "$global_result: $test_name" echo "$global_result: $test_name" | sed 's/./=/g' + # Blank lines required for having the output be valid RST. + echo + echo ":recheck: $recheck" + echo + echo ":copy-in-global-log: $copy_in_global_log" echo cat $tmp_results echo