From 0c81b43f711fb861f04227ced8dba889596d9c43 Mon Sep 17 00:00:00 2001 From: Stefano Lattarini Date: Thu, 28 Jul 2011 12:25:22 +0200 Subject: [PATCH] test harness: use new `.trs' files to hold test metadata With this change, the test harness will keep test metadata in dedicated `.trs' files, instead of having them embedded into the `.log' files. This allows for easier forward-compatibility and extension of test metadata, and for more flexibility in the format of the `.log' files. Note that this change makes the `:end-metadata:' field obsolete. * doc/automake.texi (Parallel Test Harness, Log files generation and test results recording): Document the new APIs and behaviour; some related minor rewordings and fixlets. * NEWS: Update. * automake.in (handle_tests): When bringing in the content of `check2.am', substitute %BASE% with the basename of the `.log' file being created by a rule. Add the generated `.trs' files to the list of files to be cleaned by "make mostlyclean". * lib/am/check.am (am__test_driver_flags): Rename ... (am__common_driver_flags): ... to this, and remove the flags `--test-name' and `--log-file' from it: they are now define in the proper rules in `check2.am'. (am__TEST_BASES): New internal variable, holding the names of the tests, with any registered extension removed. (am__stealth_MAKE): New internal variable, can be used instead of $(MAKE) in recipes requiring a recursive call to make, but which are not intended to be executed by "make -n". (.log.trs): New suffix rule, to recover from deletion of `.trs' files. ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the new API of "test logs in `.log' files, test metadata in `.trs' files". It goes to some length to work correctly in face of unreadable or missing `.log' and `.trs' files, and to error out with proper error messages when this is not possible. [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale" `.trs' files (in addition to `.log files) before remaking the $(TEST_SUITE_LOG). (recheck, recheck-html): Look for the `:recheck:' field in the `.trs' files, not in the `.log' files. * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust the call to the test driver, in particularly passing the new option `--trs-file'. [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise. * lib/tap-driver ($USAGE): Adjust the help screen. (Getopt::Long::GetOptions): Handle the `--trs-file' option, through the use of ... ($trs_file): ... this new global variable. (finish): Write metadata for the test run to `$trs_file' rather then to `$log_file', through the use of ... (write_test_results): ... this new function. * lib/test-driver (print_usage): Update the help screen. (Option parsing): Handle the `--trs-file' option, through the use of ... ($resfile): ... this new global variable. (Main code): Write metadata for the test run to `$trsfile' rather than to `$logfile'. Minor related adjustments to comments. * tests/.gitignore: Ignore `*.trs' files. * tests/parallel-tests-unreadable-log.test: Moved ... * tests/parallel-tests-unreadable.test: ... to this, and extended to also check the semantics for unreadable `.trs' files. * tests/test-driver-end-metadata.test: Deleted as obsolete. * tests/test-driver-metadata-no-leading-space.test: Likewise. * tests/test-driver-global-log.test: Renamed ... * tests/test-metadata-global-log.test: ... to this, and modified as to verify the new APIs and semantics. * tests/test-driver-recheck: Renamed ... * tests/test-metadata-recheck.test: ... to this, and modified likewise. * tests/parallel-tests-once.test: New test. * tests/parallel-tests-make-n.test: Likewise. * test-metadata-results.test: Likewise. * test-missing.test: Likewise. * test-missing2.test: Likewise. * test-trs-basic.test: Likewise. * test-trs-recover.test: Likewise. * test-trs-recover2.test: Likewise. * tests/Makefile.am (TESTS): Update. --- ChangeLog | 78 ++++++++ NEWS | 5 + automake.in | 4 + doc/automake.texi | 175 ++++++++++-------- lib/Automake/tests/Makefile.in | 118 +++++++----- lib/am/check.am | 155 +++++++++------- lib/am/check2.am | 16 +- lib/tap-driver | 42 ++--- lib/test-driver | 30 ++- tests/.gitignore | 1 + tests/Makefile.am | 16 +- tests/Makefile.in | 134 ++++++++------ tests/parallel-tests-make-n.test | 107 +++++++++++ ...able-log.test => parallel-tests-once.test} | 32 +--- tests/parallel-tests-unreadable.test | 75 ++++++++ tests/parallel-tests9.test | 9 - tests/test-driver-create-log-dir.test | 19 +- tests/test-driver-custom-multitest.test | 1 + tests/test-driver-custom-no-html.test | 4 +- tests/test-driver-custom-xfail-tests.test | 36 ++-- tests/test-driver-custom.test | 4 + tests/test-driver-end-metadata.test | 120 ------------ ...test-driver-metadata-no-leading-space.test | 94 ---------- tests/test-driver-strip-vpath.test | 2 + tests/test-driver-trs-suffix-registered.test | 58 ++++++ ...log.test => test-metadata-global-log.test} | 132 +++++++------ ...echeck.test => test-metadata-recheck.test} | 22 ++- tests/test-metadata-results.test | 169 +++++++++++++++++ tests/test-missing.test | 72 +++++++ tests/test-missing2.test | 54 ++++++ tests/test-trs-basic.test | 138 ++++++++++++++ tests/test-trs-recover.test | 166 +++++++++++++++++ tests/test-trs-recover2.test | 133 +++++++++++++ tests/trivial-test-driver | 58 +++--- 34 files changed, 1598 insertions(+), 681 deletions(-) create mode 100755 tests/parallel-tests-make-n.test rename tests/{parallel-tests-unreadable-log.test => parallel-tests-once.test} (59%) create mode 100755 tests/parallel-tests-unreadable.test delete mode 100755 tests/test-driver-end-metadata.test delete mode 100755 tests/test-driver-metadata-no-leading-space.test create mode 100755 tests/test-driver-trs-suffix-registered.test rename tests/{test-driver-global-log.test => test-metadata-global-log.test} (62%) rename tests/{test-driver-recheck.test => test-metadata-recheck.test} (93%) create mode 100755 tests/test-metadata-results.test create mode 100755 tests/test-missing.test create mode 100755 tests/test-missing2.test create mode 100755 tests/test-trs-basic.test create mode 100755 tests/test-trs-recover.test create mode 100755 tests/test-trs-recover2.test diff --git a/ChangeLog b/ChangeLog index afc41347d..356c7636c 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,81 @@ +2011-08-01 Stefano Lattarini + + test harness: use new `.trs' files to hold test metadata + With this change, the test harness will keep test metadata in + dedicated `.trs' files, instead of having them embedded into the + `.log' files. This allows for easier forward-compatibility and + extension of test metadata, and for more flexibility in the + format of the `.log' files. Note that this change makes the + `:end-metadata:' field obsolete. + * doc/automake.texi (Parallel Test Harness, Log files generation + and test results recording): Document the new APIs and behaviour; + some related minor rewordings and fixlets. + * NEWS: Update. + * automake.in (handle_tests): When bringing in the content of + `check2.am', substitute %BASE% with the basename of the `.log' + file being created by a rule. Add the generated `.trs' files + to the list of files to be cleaned by "make mostlyclean". + * lib/am/check.am (am__test_driver_flags): Rename ... + (am__common_driver_flags): ... to this, and remove the flags + `--test-name' and `--log-file' from it: they are now define in + the proper rules in `check2.am'. + (am__TEST_BASES): New internal variable, holding the names of + the tests, with any registered extension removed. + (am__stealth_MAKE): New internal variable, can be used instead of + $(MAKE) in recipes requiring a recursive call to make, but which + are not intended to be executed by "make -n". + (.log.trs): New suffix rule, to recover from deletion of `.trs' + files. + ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the + new API of "test logs in `.log' files, test metadata in `.trs' + files". It goes to some length to work correctly in face of + unreadable or missing `.log' and `.trs' files, and to error out + with proper error messages when this is not possible. + [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale" + `.trs' files (in addition to `.log files) before remaking the + $(TEST_SUITE_LOG). + (recheck, recheck-html): Look for the `:recheck:' field in the + `.trs' files, not in the `.log' files. + * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust + the call to the test driver, in particularly passing the new + option `--trs-file'. + [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise. + * lib/tap-driver ($USAGE): Adjust the help screen. + (Getopt::Long::GetOptions): Handle the `--trs-file' option, + through the use of ... + ($trs_file): ... this new global variable. + (finish): Write metadata for the test run to `$trs_file' rather + then to `$log_file', through the use of ... + (write_test_results): ... this new function. + * lib/test-driver (print_usage): Update the help screen. + (Option parsing): Handle the `--trs-file' option, through the + use of ... + ($resfile): ... this new global variable. + (Main code): Write metadata for the test run to `$trsfile' rather + than to `$logfile'. + Minor related adjustments to comments. + * tests/.gitignore: Ignore `*.trs' files. + * tests/parallel-tests-unreadable-log.test: Moved ... + * tests/parallel-tests-unreadable.test: ... to this, and extended + to also check the semantics for unreadable `.trs' files. + * tests/test-driver-end-metadata.test: Deleted as obsolete. + * tests/test-driver-metadata-no-leading-space.test: Likewise. + * tests/test-driver-global-log.test: Renamed ... + * tests/test-metadata-global-log.test: ... to this, and modified + as to verify the new APIs and semantics. + * tests/test-driver-recheck: Renamed ... + * tests/test-metadata-recheck.test: ... to this, and modified + likewise. + * tests/parallel-tests-once.test: New test. + * tests/parallel-tests-make-n.test: Likewise. + * test-metadata-results.test: Likewise. + * test-missing.test: Likewise. + * test-missing2.test: Likewise. + * test-trs-basic.test: Likewise. + * test-trs-recover.test: Likewise. + * test-trs-recover2.test: Likewise. + * tests/Makefile.am (TESTS): Update. + 2011-07-27 Stefano Lattarini test harness: allow more metadata in log files diff --git a/NEWS b/NEWS index 620fe2ec7..96d3126c0 100644 --- a/NEWS +++ b/NEWS @@ -75,6 +75,11 @@ New in 1.11a: - A new developer-reserved variable `AM_TESTS_FD_REDIRECT' can be used to redirect/define file descriptors used by the test scripts. + - The parallel-tests harness generates now, in addition the `.log' files + holding the output produced by the test scripts, a new set of `.trs' + files, holding "metadata" derived by the execution of the test scripts; + among such metadata are the outcomes of the test cases run by a script. + * WARNING: Future backward-incompatibilities! - The Automake support for automatic de-ANSI-fication will be removed in diff --git a/automake.in b/automake.in index 2ad8dc2d8..ad58f064e 100644 --- a/automake.in +++ b/automake.in @@ -5033,6 +5033,7 @@ sub handle_tests return substr ($obj, 0, length ($obj) - length ($test_suffix)) . '.log' if substr ($obj, - length ($test_suffix)) eq $test_suffix; } + my $base = $obj; $obj .= '.log'; # The "test driver" program, deputed to handle tests protocol used by # test scripts. By default, it's assumed that no protocol is used, @@ -5054,6 +5055,7 @@ sub handle_tests $output_rules .= file_contents ('check2', new Automake::Location, GENERIC => 0, OBJ => $obj, + BASE => $base, SOURCE => $val, DRIVER => $driver, DRIVER_FLAGS => $driver_flags, @@ -5107,6 +5109,7 @@ sub handle_tests $output_rules .= file_contents ('check2', new Automake::Location, GENERIC => 1, OBJ => '', + BASE => '$*', SOURCE => '$<', DRIVER => $driver, DRIVER_FLAGS => $driver_flags, @@ -5122,6 +5125,7 @@ sub handle_tests $clean_files{'$(TEST_LOGS_TMP)'} = MOSTLY_CLEAN; $clean_files{'$(TEST_LOGS)'} = MOSTLY_CLEAN; + $clean_files{'$(am__TEST_BASES:=.trs)'} = MOSTLY_CLEAN; $clean_files{'$(TEST_SUITE_LOG)'} = MOSTLY_CLEAN; $clean_files{'$(TEST_SUITE_HTML)'} = MOSTLY_CLEAN; } diff --git a/doc/automake.texi b/doc/automake.texi index 2e5e2bad7..d02e6ca8b 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -8975,12 +8975,27 @@ order to satisfy additional portability requirements. @vindex TEST_SUITE_LOG @vindex TESTS -The parallel harness operates by defining a set of @command{make} rules -to create a summary log file, @code{TEST_SUITE_LOG}, which defaults to -@file{test-suite.log} and requires a @file{.log} suffix. This file -depends upon log files created for each single test program listed in -@code{TESTS}, which in turn contain all output produced by the -corresponding tests. +@cindex @file{.log} files +@cindex @file{.trs} files +@cindex test metadata +The parallel test harness operates by defining a set of @command{make} +rules that run the test scripts listed in @code{TESTS}, and, for each +such script, save its output in a corresponding @file{.log} file and +its results (and other ``metadata'', @pxref{API for Custom Test Drivers}) +in a corresponding @file{.trs} (as in @b{T}est @b{R}e@b{S}ults) file. +@c We choose the `.trs' extension also because, at the time of writing, +@c it isn't already used for other significant purposes; see e.g.: +@c - http://filext.com/file-extension/trs +@c - http://www.file-extensions.org/search/?searchstring=trs +The @file{.log} file will contain all the output emitted by the test on +its standard output and its standard error. The @file{.trs} file will +contain, among the other things, the results of the test cases run by +the script. + +The parallel test harness will also create a summary log file, +@code{TEST_SUITE_LOG}, which defaults to @file{test-suite.log} and requires +a @file{.log} suffix. This file depends upon all the @file{.log} and +@file{.trs} files created for the test scripts listed in @code{TESTS}. @vindex VERBOSE As with the serial harness above, by default one status line is printed @@ -8989,16 +9004,17 @@ However, standard output and standard error of the test are redirected to a per-test log file, so that parallel execution does not produce intermingled output. The output from failed tests is collected in the @file{test-suite.log} file. If the variable @samp{VERBOSE} is set, this -file is output after the summary. For best results, the tests should be -verbose by default now. +file is output after the summary. +@c FIXME: we should be clearer about what we mean exactly here ... +For best results, the tests should be verbose by default now. @vindex TEST_EXTENSIONS @vindex TEST_LOGS -Each log file is created when the corresponding test has completed. -The set of log files is listed in the read-only variable -@code{TEST_LOGS}, and defaults to @code{TESTS}, with the executable -extension if any (@pxref{EXEEXT}), as well as any suffix listed in -@code{TEST_EXTENSIONS} removed, and @file{.log} appended. +Each couple of @file{.log} and @file{.trs} files is created when the +corresponding test has completed. The set of log files is listed in +the read-only variable @code{TEST_LOGS}, and defaults to @code{TESTS}, +with the executable extension if any (@pxref{EXEEXT}), as well as any +suffix listed in @code{TEST_EXTENSIONS} removed, and @file{.log} appended. @code{TEST_EXTENSIONS} defaults to @file{.test}. Results are undefined if a test file name ends in several concatenated suffixes. @@ -9036,7 +9052,9 @@ AM_LOG_FLAGS = -d @noindent will invoke @samp{$(PERL) -w foo.pl}, @samp{$(PYTHON) -v bar.py}, and @samp{./wrapper-script -d baz} to produce @file{foo.log}, -@file{bar.log}, and @file{baz.log}, respectively. +@file{bar.log}, and @file{baz.log}, respectively. The @file{foo.trs}, +@file{bar.trs} and @file{baz.trs} files will be automatically produced +as a side-effect. It's important to note that, differently from what we've seen for the serial test harness (@pxref{Parallel Test Harness}), the @@ -9110,14 +9128,15 @@ here too. @item @vindex RECHECK_LOGS @cindex lazy test execution -By default, the test harness removes all old per-test log files before -it starts running tests to regenerate them. The variable -@code{RECHECK_LOGS} contains the set of log files which are removed. -@code{RECHECK_LOGS} defaults to @code{TEST_LOGS}, which means all tests -need to be rechecked. By overriding this variable, you can choose which -tests need to be reconsidered. For example, you can lazily rerun only -those tests which are outdated, i.e., older than their prerequisite test -files, by setting this variable to the empty value: +By default, the test harness removes all old per-test @file{.log} and +@file{.trs} files before it starts running tests to regenerate them. The +variable @code{RECHECK_LOGS} contains the set of @file{.log} (and, by +implication, @file{.trs}) files which are removed. @code{RECHECK_LOGS} +defaults to @code{TEST_LOGS}, which means all tests need to be rechecked. +By overriding this variable, you can choose which tests need to be +reconsidered. For example, you can lazily rerun only those tests which +are outdated, i.e., older than their prerequisite test files, by setting +this variable to the empty value: @example env RECHECK_LOGS= make -e check @@ -9130,14 +9149,14 @@ You can ensure that all tests are rerun which have failed or passed unexpectedly, by running @code{make recheck} in the test directory. This convenience target will set @code{RECHECK_LOGS} appropriately before invoking the main test harness. The @code{recheck-html} target -does the same as @code{recheck} but again converts the resulting log -file in HTML format, like the @code{check-html} target. +does the same as @code{recheck} but again converts the resulting +@file{.log} file in HTML format, like the @code{check-html} target. @end itemize In order to guarantee an ordering between tests even with @code{make --j@var{N}}, dependencies between the corresponding log files may be -specified through usual @command{make} dependencies. For example, the -following snippet lets the test named @file{foo-execute.test} depend +-j@var{N}}, dependencies between the corresponding @file{.log} files +may be specified through usual @command{make} dependencies. For example, +the following snippet lets the test named @file{foo-execute.test} depend upon completion of the test @file{foo-compile.test}: @example @@ -9201,12 +9220,12 @@ the package authors to use third-party custom test drivers, in case the default ones are inadequate for their purposes, or do not support their testing protocol of choice. -A custom test driver is expected to properly run the test programs -passed to it, including the command-line arguments passed to it, -to analyze their execution and outcome, to create the @file{.log} -files associated to these test runs, and to display the test results -on the console. It is responsibility of the author of the test driver -to ensure that it implements all the above steps meaningfully and +A custom test driver is expected to properly run the test programs passed +to it (including the command-line arguments passed to those programs, if +any), to analyze their execution and outcome, to create the @file{.log} +and @file{.trs} files associated to these test runs, and to display the test +results on the console. It is responsibility of the author of the test +driver to ensure that it implements all the above steps meaningfully and correctly; Automake isn't and can't be of any help here. On the other hand, the Automake-provided code for testsuite summary generation offers support for test drivers allowing several test results per test script, @@ -9233,8 +9252,8 @@ runtime through the redefinition of @code{TESTS} or @code{TEST_LOGS}; @item concurrency through the use of @command{make}'s option @option{-j}; @item -per-test @file{.log} files, and generation of a summary @file{.log} -file from them; +per-test @file{.log} and @file{.trs} files, and generation of a summary +@file{.log} file from them; @item @code{recheck} target, @code{RECHECK_LOGS} variable, and lazy reruns of tests; @@ -9337,10 +9356,14 @@ The name of the test, with VPATH prefix (if any) removed. This can have a suffix and a directory component (as in e.g., @file{sub/foo.test}), and is mostly meant to be used in console reports about testsuite advancements and results (@pxref{Testsuite progress output}). -@item --log-file=@var{PATH} -The log file the test driver must create. If it has a directory component -(as in e.g., @file{sub/foo.log}), the test harness will ensure that such -directory exist @emph{before} the test driver is called. +@item --log-file=@file{@var{PATH}.log} +The @file{.log} file the test driver must create. If it has a directory +component (as in e.g., @file{sub/foo.log}), the test harness will ensure +that such directory exist @emph{before} the test driver is called. +@item --trs-file=@file{@var{PATH}.trs} +The @file{.trs} file the test driver must create. If it has a directory +component (as in e.g., @file{sub/foo.trs}), the test harness will ensure +that such directory exist @emph{before} the test driver is called. @item --color-tests=@{yes|no@} Whether the console output should be colorized or not (@pxref{Simple tests and color-tests}, to learn when this option gets activated and @@ -9370,14 +9393,24 @@ consistency and a more pleasant user experience. @node Log files generation and test results recording @subsubsection Log files generation and test results recording -The test driver must correctly generate the file specified by the -@option{--log-file} option (of course even when the tested program -fails or crashes). This file, while being quite free-form, @emph{must} -register some metadata through the use of custom reStructuredText fields, -in order to work with the @option{parallel-tests} harness and take -advantage of its features. Note that such fields will be recognized only -when used @emph{exactly} at the beginning of a line -- not even leading -whitespace is permitted. +The test driver must correctly generate the files specified by the +@option{--log-file} and @option{--trs-file} option (even when the tested +program fails or crashes). + +The @file{.log} file should ideally contain all the output produced by the +tested program, plus optionally other information that might facilitate +debugging or analysis of bug reports. Apart from that, its format is +basically free; the only limitation being that it must parse validly as +reStructuredText if the @file{.log} -> @file{.html} conversion is to be +supported. + +The @file{.trs} file is used to register some metadata through the use +of custom reStructuredText fields. This metadata is expected to be +employed in various ways by the parallel test harness; for example, to +count the test results when printing the testsuite summary, or to decide +which tests to re-run upon @command{make reheck}. Unrecognized metadata +in a @file{.trs} file is currently ignored by the harness, but this might +change in the future. The list of currently recognized metadata follows. @table @code @@ -9388,10 +9421,11 @@ whitespace is permitted. @cindex Test case result, registering @cindex @code{:test-result:} @cindex reStructuredText field, @code{:test-result:} -The test driver must use this filed to register the results of @emph{each} +The test driver must use this field to register the results of @emph{each} test case run by a test script file. Several @code{:test-result:} fields -can be present in the same log file; this is done in order to support test -protocols that allow a single test script to run more test cases. +can be present in the same @file{.trs} file; this is done in order to +support test protocols that allow a single test script to run more test +cases. @c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG). The only recognized test results are currently @code{PASS}, @code{XFAIL}, @@ -9401,16 +9435,16 @@ text holding the name and/or a brief description of the corresponding test; the @option{parallel-tests} harness will ignore such extra text when generating @file{test-suite.log} and preparing the testsuite summary. -@c Keep in sync with 'test-driver-recheck.test'. +@c Keep in sync with 'test-metadata-recheck.test'. @item @code{:recheck:} @cindex :recheck: @cindex reStructuredText field, @code{:recheck:} If this field is present and defined to @code{no}, then the corresponding test script will @emph{not} be run upon a @command{make recheck}. What -happens when two or more @code{:recheck:} fields are present in a -@file{.log} file is undefined behaviour. +happens when two or more @code{:recheck:} fields are present in the same +@file{.trs} file is undefined behaviour. -@c Keep in sync with 'test-driver-global-log.test'. +@c Keep in sync with 'test-metadata-global-log.test'. @item @code{:copy-in-global-log:} @cindex :copy-in-global-log: @cindex reStructuredText field, @code{:copy-in-global-log:} @@ -9420,43 +9454,30 @@ of the @file{.log} file will @emph{not} be copied into the global it can be useful in debugging and analysis of bug report, it can also be just a waste of space in normal situations, e.g., when a test script is successful. What happens when two or more @code{:copy-in-global-log:} -fields are present in a @file{.log} file is undefined behaviour. - -@c Keep in sync with 'test-driver-end-metadata.test'. -@item @code{:end-metadata:} -@cindex :end-metadata: -@cindex reStructuredText field, @code{:end-metadata:} -This special field causes all the following text in the @file{.log} -file not to be scanned by the testsuite harness in search for other -test metadata. This feature is used internally by the default -@option{parallel-tests} driver, and can be useful to e.g., avoid -spurious testsuite results. +fields are present in the same @file{.trs} file is undefined behaviour. @end table @noindent -Let's see a small example. Assume a @file{.log} file contains the -following lines (in this order, but possibly interleaved with other -``non-metadata'' lines): +Let's see a small example. Assume a @file{.trs} file contains the +following lines: @example :test-result: PASS server starts +:global-log-copy: no :test-result: PASS HTTP/1.1 request :test-result: FAIL HTTP/1.0 request :recheck: yes :test-result: SKIP HTTPS request (TLS library wasn't available) :test-result: PASS server stops -:global-log-copy: no -:metadata-end: -:test-result: XPASS (this shouldn't be parsed as a test result) @end example @noindent Then the corresponding test script will be re-run by @command{make check}, will contribute with @emph{five} test results to the testsuite summary (three of these tests being successful, one failed, and one skipped), and -the @file{.log} file content will @emph{not} be copied in the global log -file @file{test-suite.log}. +the content of the corresponding @file{.log} file will @emph{not} be +copied in the global log file @file{test-suite.log}. @node Testsuite progress output @subsubsection Testsuite progress output @@ -9492,11 +9513,9 @@ implementations' output might be a good idea too. If HTML testsuite output (with @code{check-html}) is to be supported, the generated @file{.log} files must contain syntactically valid -reStructuredText (among the other things, this means that every -@code{:test-result:} line must be followed by a blank line). If this -is not the case, the HTML generation will not work, although all the -other functionalities of the Automake testsuite harness should remain -untouched, and continue to work correctly. +reStructuredText. If this is not the case, the HTML generation will not +work, although all the other functionalities of the Automake testsuite +harness should remain untouched, and continue to work correctly. @node Using the TAP test protocol @section Using the TAP test protocol diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in index 9b45aa23e..78a7e4aae 100644 --- a/lib/Automake/tests/Makefile.in +++ b/lib/Automake/tests/Makefile.in @@ -119,10 +119,8 @@ am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//' # pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it # by disabling -e (using the XSI extension "set +e") if it's set. am__sh_e_setup = case $$- in *e*) set +e;; esac -# Default flags passed to all log compiler wrappers. -am__test_driver_flags = \ - --test-name "$$f" \ - --log-file '$@' \ +# Default flags passed to test drivers. +am__common_driver_flags = \ --color-tests "$$am__color_tests" \ --enable-hard-errors "$$am__enable_hard_errors" \ --expect-failure "$$am__expect_failure" @@ -156,6 +154,14 @@ case " $(XFAIL_TESTS) " in \ am__expect_failure=no;; \ esac; \ $(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT) +# The names of the tests, with any registered extension removed. Or +# equivalently, the names of the test logs, with the `.log' extension +# renoved. This honours runtime overriding of TESTS and TEST_LOGS. +am__TEST_BASES = $(TEST_LOGS:.log=) +# This can be used instead of $(MAKE) in recipes requiring a recursive call +# to make, but which are not intended to be executed by "make -n". See the +# GNU make manual for more details. +am__stealth_MAKE = $(MAKE) RECHECK_LOGS = $(TEST_LOGS) AM_RECURSIVE_TARGETS = check check-html recheck recheck-html TEST_SUITE_LOG = test-suite.log @@ -289,7 +295,7 @@ EXTRA_DIST = $(TESTS) all: all-am .SUFFIXES: -.SUFFIXES: .html .log .pl +.SUFFIXES: .html .log .pl .trs $(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(am__configure_deps) @for dep in $?; do \ case '$(am__configure_deps)' in \ @@ -329,43 +335,46 @@ CTAGS: cscope cscopelist: +# Recover from deleted `.trs' file; this should ensure that +# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create +# both `foo.log' and `foo.trs'. +.log.trs: + rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $< + $(TEST_SUITE_LOG): $(TEST_LOGS) @$(am__sh_e_setup); $(am__tty_colors); \ - tab=' '; \ - nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \ - IFS=" $$tab$$nl"; oIFS=$$IFS; \ - ws="[ $$tab]"; \ - list='$(TEST_LOGS)'; \ - list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \ - results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \ - results2=''; \ - exec 5<&0; \ - for f in $$list2; do \ - exec 0<$$f; \ - IFS=''; \ - while read line; do \ - IFS=$$oIFS; \ - case $$line in \ - :end-metadata:*) \ - break;; \ - :test-result:*) \ - res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \ - results2="$$results2$$nl$$res";; \ - esac; \ - IFS=''; \ + bases='$(am__TEST_BASES)'; \ + ws='[ ]'; \ + redo_bases=`for b in $$bases; do \ + test -f $$b.trs && test -r $$b.trs \ + && test -f $$b.log && test -r $$b.log \ + || echo $$b; \ + done`; \ + if test -n "$$redo_bases"; then \ + redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \ + redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \ + rm -f $$redo_logs && rm -f $$redo_results \ + && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \ + fi; \ + st=0; \ + for b in $$redo_bases; do \ + for e in trs log; do \ + if test ! -f $$b.$$e || test ! -r $$b.$$e; then \ + echo "fatal: making $@: failed to create $$b.$$e" >&2; \ + st=1; \ + fi; \ done; \ - IFS=$$oIFS; \ - exec 0<&5; \ done; \ - exec 5<&-; \ - results=`echo "$$results1" && echo "$$results2"`; \ - all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[ ]*//'`; \ - fail=`echo "$$results" | grep -c '^FAIL'`; \ - pass=`echo "$$results" | grep -c '^PASS'`; \ - skip=`echo "$$results" | grep -c '^SKIP'`; \ - xfail=`echo "$$results" | grep -c '^XFAIL'`; \ - xpass=`echo "$$results" | grep -c '^XPASS'`; \ - error=`echo "$$results" | grep -c '^ERROR'`; \ + test $$st -eq 0 || exit 1; \ + results=`for b in $$bases; do echo $$b.trs; done`; \ + test -n "$$results" || results=/dev/null; \ + all=` grep "^$$ws*:test-result:" $$results | wc -l`; \ + pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \ + fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \ + skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \ + xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \ + xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \ + error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \ if test `expr $$fail + $$xpass + $$error` -eq 0; then \ success=true; \ else \ @@ -407,10 +416,15 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) echo; \ echo ".. contents:: :depth: 2"; \ echo; \ - for f in $$list; do \ - sed -e '/^:end-metadata:/,$$d' $$f | \ - grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \ - || { echo && cat $$f; } \ + for b in $$bases; do \ + if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \ + >/dev/null; then :; \ + elif test ! -r $$b.log; then \ + echo "ERROR: cannot read $$b.log" >&2; \ + echo && echo "WARNING: could not read $$b.log!"; \ + else \ + echo && cat $$b.log; \ + fi; \ done; \ } >$(TEST_SUITE_LOG).tmp; \ mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \ @@ -437,6 +451,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) # Run all the tests. check-TESTS: @list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list + @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list @test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG) @list='$(TEST_LOGS)'; \ list=`for f in $$list; do \ @@ -471,19 +486,19 @@ check-html: recheck recheck-html: @ws='[ ]'; \ target=`echo $@ | sed 's,^re,,'`; \ - list='$(TEST_LOGS)'; \ - list=`for f in $$list; do \ - test -f $$f || continue; \ - sed -e '/^:end-metadata:/,$$d' $$f | \ - grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \ - || echo $$f; \ + bases='$(am__TEST_BASES)'; \ + list=`for b in $$bases; do \ + test ! -f $$b.trs && test ! -f $$b.log && continue; \ + grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \ + >/dev/null 2>&1 || echo $$b.log; \ done | tr '\012\015' ' '`; \ list=`echo "$$list" | sed 's/ *$$//'`; \ $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"' .pl.log: - @p='$<'; $(am__check_pre) \ - $(PL_LOG_DRIVER) $(am__test_driver_flags) $(AM_PL_LOG_DRIVER_FLAGS) $(PL_LOG_DRIVER_FLAGS) -- \ - $(PL_LOG_COMPILE) "$$tst" $(AM_TESTS_FD_REDIRECT) + @p='$<'; $(am__check_pre) $(PL_LOG_DRIVER) --test-name "$$f" \ + --log-file '$*.log' --trs-file '$*.trs' \ + $(am__common_driver_flags) $(AM_PL_LOG_DRIVER_FLAGS) $(PL_LOG_DRIVER_FLAGS) -- $(PL_LOG_COMPILE) "$$tst" \ + $(AM_TESTS_FD_REDIRECT) distdir: $(DISTFILES) @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ @@ -544,6 +559,7 @@ mostlyclean-generic: -test -z "$(TEST_LOGS_TMP)" || rm -f $(TEST_LOGS_TMP) -test -z "$(TEST_SUITE_HTML)" || rm -f $(TEST_SUITE_HTML) -test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG) + -test -z "$(am__TEST_BASES:=.trs)" || rm -f $(am__TEST_BASES:=.trs) clean-generic: diff --git a/lib/am/check.am b/lib/am/check.am index 272fc0a85..c406e9f27 100644 --- a/lib/am/check.am +++ b/lib/am/check.am @@ -83,10 +83,8 @@ am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//' # by disabling -e (using the XSI extension "set +e") if it's set. am__sh_e_setup = case $$- in *e*) set +e;; esac -# Default flags passed to all log compiler wrappers. -am__test_driver_flags = \ - --test-name "$$f" \ - --log-file '$@' \ +# Default flags passed to test drivers. +am__common_driver_flags = \ --color-tests "$$am__color_tests" \ --enable-hard-errors "$$am__enable_hard_errors" \ --expect-failure "$$am__expect_failure" @@ -124,64 +122,71 @@ case " $(XFAIL_TESTS) " in \ esac; \ $(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT) +# The names of the tests, with any registered extension removed. Or +# equivalently, the names of the test logs, with the `.log' extension +# renoved. This honours runtime overriding of TESTS and TEST_LOGS. +am__TEST_BASES = $(TEST_LOGS:.log=) + +# This can be used instead of $(MAKE) in recipes requiring a recursive call +# to make, but which are not intended to be executed by "make -n". See the +# GNU make manual for more details. +am__stealth_MAKE = $(MAKE) + +# Recover from deleted `.trs' file; this should ensure that +# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create +# both `foo.log' and `foo.trs'. +.log.trs: + rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $< + $(TEST_SUITE_LOG): $(TEST_LOGS) @$(am__sh_e_setup); $(am__tty_colors); \ - tab=' '; \ -## The following line will define the `$nl' shell variable to a literal -## newline. Idiom taken from the Autoconf manual. - nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \ -## Sanitize internal field separator for the shell: "space tab newline". - IFS=" $$tab$$nl"; oIFS=$$IFS; \ -## Regular expression to match a whitespace. - ws="[ $$tab]"; \ -## All test logs. - list='$(TEST_LOGS)'; \ -## Readable test logs. - list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \ -## Each unreadable test log counts as a failed test. - results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \ -## Now we're going to extract the outcome of all the testcases from the -## test logs. - results2=''; \ -## Can't do the redirections directly in the "for" loop below, otherwise -## some shells would run the loop body in a subshell. - exec 5<&0; \ - for f in $$list2; do \ - exec 0<$$f; \ -## We want to support special reStructuredText fields only when they're -## placed at the beginning of a line (as documented in the manual), so -## we have to temporarily clear the $IFS variable to prevent automatic -## whitespace stripping by the shell. - IFS=''; \ - while read line; do \ - IFS=$$oIFS; \ - case $$line in \ -## Special field that tells not to scan the rest of the log file. - :end-metadata:*) \ - break;; \ -## A test test result -- register it. - :test-result:*) \ - res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \ - results2="$$results2$$nl$$res";; \ - esac; \ -## For the next loop. - IFS=''; \ + bases='$(am__TEST_BASES)'; \ + ws='[ ]'; \ +## We need to ensures that all the required `.trs' and `.log' files will +## be present and readable. The direct dependencies of $(TEST_SUITE_LOG) +## only ensure that all the `.log' files exists; they don't ensure that +## the `.log' files are readable, and worse, they don't ensure that the +## `.trs' files even exist. + redo_bases=`for b in $$bases; do \ + test -f $$b.trs && test -r $$b.trs \ + && test -f $$b.log && test -r $$b.log \ + || echo $$b; \ + done`; \ + if test -n "$$redo_bases"; then \ +## Uh-oh, either some `.log' files were unreadable, or some `.trs' files +## were missing (or unreadable). We need to re-run the corresponding +## tests in order to re-create them. + redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \ + redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \ + rm -f $$redo_logs && rm -f $$redo_results \ + && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \ + fi; \ +## Sanity check: each unreadable or non-existent test result file should +## has been properly remade at this point, as should the corresponding log +## file. + st=0; \ + for b in $$redo_bases; do \ + for e in trs log; do \ + if test ! -f $$b.$$e || test ! -r $$b.$$e; then \ + echo "fatal: making $@: failed to create $$b.$$e" >&2; \ + st=1; \ + fi; \ done; \ -## Restore original standard input and internal field separator. - IFS=$$oIFS; \ - exec 0<&5; \ done; \ -## We don't need this anymore. - exec 5<&-; \ -## Prepare data for the test suite summary. - results=`echo "$$results1" && echo "$$results2"`; \ - all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[ ]*//'`; \ - fail=`echo "$$results" | grep -c '^FAIL'`; \ - pass=`echo "$$results" | grep -c '^PASS'`; \ - skip=`echo "$$results" | grep -c '^SKIP'`; \ - xfail=`echo "$$results" | grep -c '^XFAIL'`; \ - xpass=`echo "$$results" | grep -c '^XPASS'`; \ - error=`echo "$$results" | grep -c '^ERROR'`; \ + test $$st -eq 0 || exit 1; \ +## List of test result files. + results=`for b in $$bases; do echo $$b.trs; done`; \ + test -n "$$results" || results=/dev/null; \ +## Prepare data for the test suite summary. These do not take into account +## unreadable test results, but they'll be appropriately updated later if +## needed. + all=` grep "^$$ws*:test-result:" $$results | wc -l`; \ + pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \ + fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \ + skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \ + xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \ + xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \ + error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \ ## Whether the testsuite was successful or not. if test `expr $$fail + $$xpass + $$error` -eq 0; then \ success=true; \ @@ -233,11 +238,19 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) echo; \ echo ".. contents:: :depth: 2"; \ echo; \ - for f in $$list; do \ -## FIXME: two forks per test -- this is horrendously inefficient! - sed -e '/^:end-metadata:/,$$d' $$f | \ - grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \ - || { echo && cat $$f; } \ + for b in $$bases; do \ +## FIXME: one fork per test -- this is horrendously inefficient! + if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \ + >/dev/null; then :; \ +## If we cannot read the .log file of a test ... + elif test ! -r $$b.log; then \ +## ... print an error message to the console ... + echo "ERROR: cannot read $$b.log" >&2; \ +## ... and a warning in the `test-suite.log' file. + echo && echo "WARNING: could not read $$b.log!"; \ + else \ + echo && cat $$b.log; \ + fi; \ done; \ } >$(TEST_SUITE_LOG).tmp; \ mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \ @@ -272,6 +285,7 @@ RECHECK_LOGS = $(TEST_LOGS) check-TESTS: ## Expand $(RECHECK_LOGS) only once, to avoid exceeding line length limits. @list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list + @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list ## We always have to remove TEST_SUITE_LOG, to ensure its rule is run ## in any case even in lazy mode: otherwise, if no test needs rerunning, ## or a prior run plus reruns all happen within the same timestamp @@ -335,13 +349,14 @@ AM_RECURSIVE_TARGETS += check-html recheck recheck-html: @ws='[ ]'; \ target=`echo $@ | sed 's,^re,,'`; \ - list='$(TEST_LOGS)'; \ - list=`for f in $$list; do \ - test -f $$f || continue; \ -## FIXME: two forks per test -- this is horrendously inefficient! - sed -e '/^:end-metadata:/,$$d' $$f | \ - grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \ - || echo $$f; \ + bases='$(am__TEST_BASES)'; \ + list=`for b in $$bases; do \ +## Skip tests that haven't been run, but recover gracefully from deleted +## `.trs' files. + test ! -f $$b.trs && test ! -f $$b.log && continue; \ +## FIXME: one fork per test -- this is horrendously inefficient! + grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \ + >/dev/null 2>&1 || echo $$b.log; \ done | tr '\012\015' ' '`; \ ## This apparently useless munging helps to avoid a nasty bug (a ## segmentation fault!) on Solaris XPG4 make. diff --git a/lib/am/check2.am b/lib/am/check2.am index f4fb3c918..ad0a4aa36 100644 --- a/lib/am/check2.am +++ b/lib/am/check2.am @@ -14,12 +14,13 @@ ## You should have received a copy of the GNU General Public License ## along with this program. If not, see . -## From a test file to a log file. +## From a test file to a .log and .trs file. ?GENERIC?%EXT%.log: ?!GENERIC?%OBJ%: %SOURCE% - @p='%SOURCE%'; $(am__check_pre) \ - %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \ - %COMPILE% "$$tst" $(AM_TESTS_FD_REDIRECT) + @p='%SOURCE%'; $(am__check_pre) %DRIVER% --test-name "$$f" \ + --log-file '%BASE%.log' --trs-file '%BASE%.trs' \ + $(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% "$$tst" \ + $(AM_TESTS_FD_REDIRECT) ## If no programs are built in this package, then this rule is removed ## at automake time. Otherwise, %am__EXEEXT% expands to a configure time @@ -27,7 +28,8 @@ ## conflict with the previous one. if %am__EXEEXT% ?GENERIC?%EXT%$(EXEEXT).log: - @p='%SOURCE%'; $(am__check_pre) \ - %DRIVER% $(am__test_driver_flags) %DRIVER_FLAGS% -- \ - %COMPILE% "$$tst" $(AM_TESTS_FD_REDIRECT) + @p='%SOURCE%'; $(am__check_pre) %DRIVER% --test-name "$$f" \ + --log-file '%BASE%.log' --trs-file '%BASE%.trs' \ + $(am__common_driver_flags) %DRIVER_FLAGS% -- %COMPILE% "$$tst" \ + $(AM_TESTS_FD_REDIRECT) endif %am__EXEEXT% diff --git a/lib/tap-driver b/lib/tap-driver index 9075d35ce..8237fecb2 100755 --- a/lib/tap-driver +++ b/lib/tap-driver @@ -15,7 +15,7 @@ my $ME = "tap-driver"; my $USAGE = <<'END'; Usage: - tap-driver [--help|--version] --test-name=NAME --log-file=PATH + tap-driver --test-name=NAME --log-file=PATH --trs-file=PATH [--expect-failure={yes|no}] [--color-tests={yes|no}] [--enable-hard-errors={yes|no}] [--merge|--no-merge] [--ignore-exit] [--comments|--no-comments] [--] TEST-COMMAND @@ -62,12 +62,14 @@ my %cfg = ( my $test_script_name = undef; my $log_file = undef; +my $trs_file = undef; Getopt::Long::GetOptions ( 'help' => sub { print $HELP; exit 0; }, 'version' => sub { print "$ME $VERSION"; exit 0; }, 'test-name=s' => \$test_script_name, 'log-file=s' => \$log_file, + 'res-file=s' => \$trs_file, 'color-tests=s' => \&bool_opt, 'expect-failure=s' => \&bool_opt, 'enable-hard-errors=s' => \&bool_opt, @@ -164,6 +166,20 @@ TEST_RESULTS : } +sub write_test_results () +{ + open RES, ">", $trs_file or die "opening $trs_file: $!\n"; + print RES ":recheck: " . + (must_recheck ? "yes" : "no") . "\n"; + print RES ":copy-in-global-log: " . + (copy_in_global_log ? "yes" : "no") . "\n"; + foreach my $result (get_test_results) + { + print RES ":test-result: $result\n"; + } + close RES or die "closing $trs_file: $!\n"; +} + sub start (@) { # Redirect stderr and stdout to a temporary log file. Save the @@ -179,35 +195,15 @@ sub start (@) sub finish () { - open LOG, ">", $log_file or die "opening $log_file: $!\n"; - - # Some extra trailing empty lines outputted below are required - # to support reStructuredText -> HTML conversion. + write_test_results; + open LOG, ">", $log_file or die "opening $log_file: $!\n"; my $global_result = get_global_test_result; my $global_result_line = "$global_result: $test_script_name"; print LOG "$global_result_line\n"; print LOG "=" x length ($global_result_line) . "\n"; print LOG "\n"; - print LOG ":recheck: " . (must_recheck ? "yes" : "no") . "\n"; - print LOG "\n"; - - print LOG ":copy-in-global-log: " . - (copy_in_global_log ? "yes" : "no") . "\n"; - print LOG "\n"; - - foreach (get_test_results) - { - print LOG ":test-result: $_\n"; - print LOG "\n"; - } - - # So that the output from the test script won't be parsed for - # testsuite metadata. - print LOG ":end-metadata:\n"; - print LOG "\n"; - close TMP or die "closing $log_file-t: $!\n"; # FIXME: remove this hack! my $test_output = `cat $log_file-t && rm -f $log_file-t`; diff --git a/lib/test-driver b/lib/test-driver index 45b3afe37..472ba4925 100755 --- a/lib/test-driver +++ b/lib/test-driver @@ -1,7 +1,7 @@ #! /bin/sh # test-driver - basic driver script for the `parallel-tests' mode. -scriptversion=2011-07-27.12; # UTC +scriptversion=2011-08-01.21; # UTC # Copyright (C) 2011 Free Software Foundation, Inc. # @@ -48,7 +48,7 @@ print_usage () { cat < $trsfile +echo ":recheck: $recheck" >> $trsfile +echo ":copy-in-global-log: $gcopy" >> $trsfile + # Now write log file. { echo "$res: $test_name (exit: $estatus)" | rst_section - # Blank lines required for having the output be valid RST. - echo - echo ":test-result: $res (exit status: $estatus)" - echo - echo ":recheck: $recheck" - echo - echo ":copy-in-global-log: $gcopy" - echo - # So that the output from the test script won't be parsed for - # testsuite metadata. - echo ":end-metadata:" - echo - # Use a reStructuredText transition to better separate the test - # metadata the test registered output. - echo ------------ echo cat $tmpfile } > $logfile diff --git a/tests/.gitignore b/tests/.gitignore index 026e261e4..0d5a032b5 100644 --- a/tests/.gitignore +++ b/tests/.gitignore @@ -5,6 +5,7 @@ instspc-tests.am parallel-tests.am *.dir *.log +*.trs *.log-t *-p.test instspc-*-build.test diff --git a/tests/Makefile.am b/tests/Makefile.am index 9a0c22279..104f70be8 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -724,9 +724,10 @@ parallel-tests7.test \ parallel-tests8.test \ parallel-tests9.test \ parallel-tests10.test \ +parallel-tests-make-n.test \ parallel-tests-fd-redirect.test \ parallel-tests-am_tests_environment.test \ -parallel-tests-unreadable-log.test \ +parallel-tests-unreadable.test \ parallel-tests-subdir.test \ parallel-tests-interrupt.test \ parallel-tests-reset-term.test \ @@ -739,12 +740,11 @@ parallel-test-driver-install.test \ parallel-tests-no-spurious-summary.test \ parallel-tests-exit-statuses.test \ parallel-tests-console-output.test \ +parallel-tests-once.test \ testsuite-summary-color.test \ testsuite-summary-count.test \ testsuite-summary-count-many.test \ testsuite-summary-reference-log.test \ -test-driver-end-metadata.test \ -test-driver-metadata-no-leading-space.test \ test-driver-custom-no-extra-driver.test \ test-driver-custom.test \ test-driver-custom-xfail-tests.test \ @@ -754,11 +754,17 @@ test-driver-custom-multitest-recheck2.test \ test-driver-custom-html.test \ test-driver-custom-no-html.test \ test-driver-create-log-dir.test \ -test-driver-global-log.test \ -test-driver-recheck.test \ test-driver-strip-vpath.test \ test-driver-fail.test \ test-log.test \ +test-metadata-global-log.test \ +test-metadata-recheck.test \ +test-metadata-results.test \ +test-missing.test \ +test-missing2.test \ +test-trs-basic.test \ +test-trs-recover.test \ +test-trs-recover2.test \ parse.test \ percent.test \ percent2.test \ diff --git a/tests/Makefile.in b/tests/Makefile.in index 13e850cba..1732555d9 100644 --- a/tests/Makefile.in +++ b/tests/Makefile.in @@ -124,10 +124,8 @@ am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//' # pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it # by disabling -e (using the XSI extension "set +e") if it's set. am__sh_e_setup = case $$- in *e*) set +e;; esac -# Default flags passed to all log compiler wrappers. -am__test_driver_flags = \ - --test-name "$$f" \ - --log-file '$@' \ +# Default flags passed to test drivers. +am__common_driver_flags = \ --color-tests "$$am__color_tests" \ --enable-hard-errors "$$am__enable_hard_errors" \ --expect-failure "$$am__expect_failure" @@ -161,6 +159,14 @@ case " $(XFAIL_TESTS) " in \ am__expect_failure=no;; \ esac; \ $(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT) +# The names of the tests, with any registered extension removed. Or +# equivalently, the names of the test logs, with the `.log' extension +# renoved. This honours runtime overriding of TESTS and TEST_LOGS. +am__TEST_BASES = $(TEST_LOGS:.log=) +# This can be used instead of $(MAKE) in recipes requiring a recursive call +# to make, but which are not intended to be executed by "make -n". See the +# GNU make manual for more details. +am__stealth_MAKE = $(MAKE) RECHECK_LOGS = $(TEST_LOGS) AM_RECURSIVE_TARGETS = check check-html recheck recheck-html TEST_SUITE_LOG = test-suite.log @@ -969,9 +975,10 @@ parallel-tests7.test \ parallel-tests8.test \ parallel-tests9.test \ parallel-tests10.test \ +parallel-tests-make-n.test \ parallel-tests-fd-redirect.test \ parallel-tests-am_tests_environment.test \ -parallel-tests-unreadable-log.test \ +parallel-tests-unreadable.test \ parallel-tests-subdir.test \ parallel-tests-interrupt.test \ parallel-tests-reset-term.test \ @@ -984,12 +991,11 @@ parallel-test-driver-install.test \ parallel-tests-no-spurious-summary.test \ parallel-tests-exit-statuses.test \ parallel-tests-console-output.test \ +parallel-tests-once.test \ testsuite-summary-color.test \ testsuite-summary-count.test \ testsuite-summary-count-many.test \ testsuite-summary-reference-log.test \ -test-driver-end-metadata.test \ -test-driver-metadata-no-leading-space.test \ test-driver-custom-no-extra-driver.test \ test-driver-custom.test \ test-driver-custom-xfail-tests.test \ @@ -999,11 +1005,17 @@ test-driver-custom-multitest-recheck2.test \ test-driver-custom-html.test \ test-driver-custom-no-html.test \ test-driver-create-log-dir.test \ -test-driver-global-log.test \ -test-driver-recheck.test \ test-driver-strip-vpath.test \ test-driver-fail.test \ test-log.test \ +test-metadata-global-log.test \ +test-metadata-recheck.test \ +test-metadata-results.test \ +test-missing.test \ +test-missing2.test \ +test-trs-basic.test \ +test-trs-recover.test \ +test-trs-recover2.test \ parse.test \ percent.test \ percent2.test \ @@ -1396,7 +1408,7 @@ $(parallel_tests) all: all-am .SUFFIXES: -.SUFFIXES: .html .log .test +.SUFFIXES: .html .log .test .trs $(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am $(srcdir)/instspc-tests.am $(am__configure_deps) @for dep in $?; do \ case '$(am__configure_deps)' in \ @@ -1442,43 +1454,46 @@ CTAGS: cscope cscopelist: +# Recover from deleted `.trs' file; this should ensure that +# "rm -f foo.log; make foo.trs" re-run `foo.test', and re-create +# both `foo.log' and `foo.trs'. +.log.trs: + rm -f $< $@ && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $< + $(TEST_SUITE_LOG): $(TEST_LOGS) @$(am__sh_e_setup); $(am__tty_colors); \ - tab=' '; \ - nlinit=`echo 'nl="'; echo '"'`; eval "$$nlinit"; unset nlinit; \ - IFS=" $$tab$$nl"; oIFS=$$IFS; \ - ws="[ $$tab]"; \ - list='$(TEST_LOGS)'; \ - list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \ - results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \ - results2=''; \ - exec 5<&0; \ - for f in $$list2; do \ - exec 0<$$f; \ - IFS=''; \ - while read line; do \ - IFS=$$oIFS; \ - case $$line in \ - :end-metadata:*) \ - break;; \ - :test-result:*) \ - res=`expr "x$$line" : "x:test-result:$$ws*\\\\(.*\\\\)$$"`; \ - results2="$$results2$$nl$$res";; \ - esac; \ - IFS=''; \ + bases='$(am__TEST_BASES)'; \ + ws='[ ]'; \ + redo_bases=`for b in $$bases; do \ + test -f $$b.trs && test -r $$b.trs \ + && test -f $$b.log && test -r $$b.log \ + || echo $$b; \ + done`; \ + if test -n "$$redo_bases"; then \ + redo_logs=`for i in $$redo_bases; do echo $$i.log; done`; \ + redo_results=`for i in $$redo_bases; do echo $$i.trs; done`; \ + rm -f $$redo_logs && rm -f $$redo_results \ + && $(am__stealth_MAKE) $(AM_MAKEFLAGS) $$redo_logs; \ + fi; \ + st=0; \ + for b in $$redo_bases; do \ + for e in trs log; do \ + if test ! -f $$b.$$e || test ! -r $$b.$$e; then \ + echo "fatal: making $@: failed to create $$b.$$e" >&2; \ + st=1; \ + fi; \ done; \ - IFS=$$oIFS; \ - exec 0<&5; \ done; \ - exec 5<&-; \ - results=`echo "$$results1" && echo "$$results2"`; \ - all=`echo "$$results" | sed '/^$$/d' | wc -l | sed -e 's/^[ ]*//'`; \ - fail=`echo "$$results" | grep -c '^FAIL'`; \ - pass=`echo "$$results" | grep -c '^PASS'`; \ - skip=`echo "$$results" | grep -c '^SKIP'`; \ - xfail=`echo "$$results" | grep -c '^XFAIL'`; \ - xpass=`echo "$$results" | grep -c '^XPASS'`; \ - error=`echo "$$results" | grep -c '^ERROR'`; \ + test $$st -eq 0 || exit 1; \ + results=`for b in $$bases; do echo $$b.trs; done`; \ + test -n "$$results" || results=/dev/null; \ + all=` grep "^$$ws*:test-result:" $$results | wc -l`; \ + pass=` grep "^$$ws*:test-result:$$ws*PASS" $$results | wc -l`; \ + fail=` grep "^$$ws*:test-result:$$ws*FAIL" $$results | wc -l`; \ + skip=` grep "^$$ws*:test-result:$$ws*SKIP" $$results | wc -l`; \ + xfail=`grep "^$$ws*:test-result:$$ws*XFAIL" $$results | wc -l`; \ + xpass=`grep "^$$ws*:test-result:$$ws*XPASS" $$results | wc -l`; \ + error=`grep "^$$ws*:test-result:$$ws*ERROR" $$results | wc -l`; \ if test `expr $$fail + $$xpass + $$error` -eq 0; then \ success=true; \ else \ @@ -1520,10 +1535,15 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) echo; \ echo ".. contents:: :depth: 2"; \ echo; \ - for f in $$list; do \ - sed -e '/^:end-metadata:/,$$d' $$f | \ - grep "^:copy-in-global-log:$$ws*no$$ws*$$" >/dev/null \ - || { echo && cat $$f; } \ + for b in $$bases; do \ + if grep "^$$ws*:copy-in-global-log:$$ws*no$$ws*$$" $$b.trs \ + >/dev/null; then :; \ + elif test ! -r $$b.log; then \ + echo "ERROR: cannot read $$b.log" >&2; \ + echo && echo "WARNING: could not read $$b.log!"; \ + else \ + echo && cat $$b.log; \ + fi; \ done; \ } >$(TEST_SUITE_LOG).tmp; \ mv $(TEST_SUITE_LOG).tmp $(TEST_SUITE_LOG); \ @@ -1550,6 +1570,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) # Run all the tests. check-TESTS: @list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list + @list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list @test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG) @list='$(TEST_LOGS)'; \ list=`for f in $$list; do \ @@ -1584,19 +1605,19 @@ check-html: recheck recheck-html: @ws='[ ]'; \ target=`echo $@ | sed 's,^re,,'`; \ - list='$(TEST_LOGS)'; \ - list=`for f in $$list; do \ - test -f $$f || continue; \ - sed -e '/^:end-metadata:/,$$d' $$f | \ - grep "^:recheck:$$ws*no$$ws*$$" >/dev/null \ - || echo $$f; \ + bases='$(am__TEST_BASES)'; \ + list=`for b in $$bases; do \ + test ! -f $$b.trs && test ! -f $$b.log && continue; \ + grep "^$$ws*:recheck:$$ws*no$$ws*$$" $$b.trs \ + >/dev/null 2>&1 || echo $$b.log; \ done | tr '\012\015' ' '`; \ list=`echo "$$list" | sed 's/ *$$//'`; \ $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"' .test.log: - @p='$<'; $(am__check_pre) \ - $(TEST_LOG_DRIVER) $(am__test_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS) $(TEST_LOG_DRIVER_FLAGS) -- \ - $(TEST_LOG_COMPILE) "$$tst" $(AM_TESTS_FD_REDIRECT) + @p='$<'; $(am__check_pre) $(TEST_LOG_DRIVER) --test-name "$$f" \ + --log-file '$*.log' --trs-file '$*.trs' \ + $(am__common_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS) $(TEST_LOG_DRIVER_FLAGS) -- $(TEST_LOG_COMPILE) "$$tst" \ + $(AM_TESTS_FD_REDIRECT) distdir: $(DISTFILES) @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ @@ -1657,6 +1678,7 @@ mostlyclean-generic: -test -z "$(TEST_LOGS_TMP)" || rm -f $(TEST_LOGS_TMP) -test -z "$(TEST_SUITE_HTML)" || rm -f $(TEST_SUITE_HTML) -test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG) + -test -z "$(am__TEST_BASES:=.trs)" || rm -f $(am__TEST_BASES:=.trs) clean-generic: diff --git a/tests/parallel-tests-make-n.test b/tests/parallel-tests-make-n.test new file mode 100755 index 000000000..451f7829a --- /dev/null +++ b/tests/parallel-tests-make-n.test @@ -0,0 +1,107 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check `make -n' for testsuite-related targets, when `parallel-tests' +# is in use. + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in << 'END' +AC_OUTPUT +END + +cat > Makefile.am <<'END' +TESTS = foo.test bar.test +TEST_LOG_COMPILER = $(SHELL) +END + +$ACLOCAL +$AUTOMAKE -a +$AUTOCONF + +./configure + +make_n_ () +{ + st=0 + $MAKE -n "$@" >output 2>&1 || { cat output; ls -l; Exit 1; } + cat output + # Look out for possible errors from common tools used by recipes. + $EGREP -i ' (exist|permission|denied|no .*(such|file))' output && Exit 1 + $EGREP '(mv|cp|rm|cat|grep|sed|awk): ' output && Exit 1 + : +} + +: > output + +files='foo.log bar.log foo.trs bar.trs' + +for target in check recheck test-suite.log; do + test ! -f foo.log + test ! -f foo.trs + test ! -f bar.log + test ! -f bar.trs + test ! -f test-suite.log +done + +echo 'exit 0' > foo.test +echo 'exit 1' > bar.test + +$MAKE check && Exit 1 + +chmod a-w . + +make_n_ clean +test -f foo.log +test -f foo.trs +test -f foo.log +test -f bar.trs + +echo 'echo this is bad; exit 1' > foo.test +echo 'exit 0' > bar.test + +for target in check recheck test-suite.log; do + make_n_ $target + grep '^:test-result: *FAIL' bar.trs + grep 'this is bad' foo.log test-suite.log && Exit 1 +done + +chmod a-rw $files +if test -r foo.log; then + : You can still read unreadable files! Skip these checks. +else + for target in check recheck test-suite.log; do + make_n_ $target + for f in $files; do + test -f $f && test ! -r $f || Exit 1 + done + done +fi +chmod u+r $files + +chmod u+w . +rm -f foo.log bar.trs +chmod a-w . +for target in check recheck test-suite.log $files; do + make_n_ $target + test ! -f foo.log + test -f foo.trs + test ! -f bar.trs + test -f bar.log +done + +: diff --git a/tests/parallel-tests-unreadable-log.test b/tests/parallel-tests-once.test similarity index 59% rename from tests/parallel-tests-unreadable-log.test rename to tests/parallel-tests-once.test index e8e101b9b..d676a128f 100755 --- a/tests/parallel-tests-unreadable-log.test +++ b/tests/parallel-tests-once.test @@ -14,7 +14,9 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . -# Check that the testsuite driver copes well with unreadable test logs. +# Tests shouldn't be run multiple times by a simple "make check" in a +# clean directory. An early implementation of the `.trs' intermediate +# files incurred a similar problem. parallel_tests=yes . ./defs || Exit 1 @@ -24,37 +26,23 @@ AC_OUTPUT END cat > Makefile.am << 'END' -TESTS = foo.test bar.test -XFAIL_TESTS = bar.test +TESTS = foo.test END -cat > foo.test << 'END' +cat > foo.test <<'END' #! /bin/sh -exit 0 +test -f foo.run && Exit 1 +: > foo.run END -cat > bar.test << 'END' -#! /bin/sh -exit 1 -END -chmod a+x foo.test bar.test +chmod a+x foo.test $ACLOCAL $AUTOCONF $AUTOMAKE -a -# The testsuite driver will use this variable, so ensure it sanitizes -# it and do not allow in spurious values from the environment. -line=PASS; export line - ./configure -$MAKE foo.log -$MAKE bar.log -chmod a-r foo.log bar.log -test ! -r foo.log || Exit 77 -$MAKE test-suite.log >stdout && { cat stdout; Exit 1; } -cat stdout -grep '^# ERROR: *2$' stdout -grep '^# ERROR: *2$' test-suite.log +$MAKE check +test -f foo.run # Sanity check. : diff --git a/tests/parallel-tests-unreadable.test b/tests/parallel-tests-unreadable.test new file mode 100755 index 000000000..47edc9a77 --- /dev/null +++ b/tests/parallel-tests-unreadable.test @@ -0,0 +1,75 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check that the testsuite driver copes well with unreadable `.log' +# and `.trs' files. + +parallel_tests=yes +. ./defs || Exit 1 + +: > t +chmod a-r t && test ! -r t || skip_ "you can still read unreadable files" +rm -f t + +cat >> configure.in << 'END' +AC_OUTPUT +END + +cat > Makefile.am << 'END' +TESTS = foo.test bar.test +END + +cat > foo.test << 'END' +#! /bin/sh +echo foofoofoo +exit 0 +END + +cat > bar.test << 'END' +#! /bin/sh +echo barbarbar +exit 77 +END + +chmod a+x foo.test bar.test + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure + +for files in \ + 'foo.log bar.log' \ + 'foo.trs bar.trs' \ + 'foo.trs bar.log' \ + 'foo.log bar.trs' \ +; do + $MAKE check + rm -f test-suite.log + chmod a-r $files + $MAKE test-suite.log || { ls -l; Exit 1; } + ls -l + grep '^foofoofoo$' foo.log + grep '^:test-result: PASS' foo.trs + grep '^barbarbar$' bar.log + grep '^:test-result: SKIP' bar.trs + grep '^SKIP: bar.test' test-suite.log + grep '^barbarbar$' test-suite.log + $EGREP 'foo\.test|foofoofoo' test-suite.log && Exit 1 +done + +: diff --git a/tests/parallel-tests9.test b/tests/parallel-tests9.test index 68fd1d447..f86562ae1 100755 --- a/tests/parallel-tests9.test +++ b/tests/parallel-tests9.test @@ -67,15 +67,6 @@ grep 'foo\.test' stdout && Exit 1 grep '^ERROR: bar\.test$' stdout grep '^FAIL: baz\.test$' stdout -# If we cannot read the log file, then redo it as well. -chmod a-r foo.log -if test ! -r foo.log; then - $MAKE recheck >stdout && { cat stdout; Exit 1; } - cat stdout - count_test_results total=3 pass=1 fail=1 skip=0 xfail=0 xpass=0 error=1 - grep '^PASS: foo\.test$' stdout || Exit 1 -fi - # Ensure that recheck builds check_SCRIPTS, and that # recheck reruns nothing if check has not been run. $MAKE clean diff --git a/tests/test-driver-create-log-dir.test b/tests/test-driver-create-log-dir.test index 5713b92be..c55479321 100755 --- a/tests/test-driver-create-log-dir.test +++ b/tests/test-driver-create-log-dir.test @@ -40,14 +40,18 @@ check-local: $(TEST_SUITE_LOG) test -f sub/foo.log test -f sub/subsub/bar.log test -f sub1/baz.log + test -f sub/foo.trs + test -f sub/subsub/bar.trs + test -f sub1/baz.trs END cat > checkdir-driver <<'END' #! /bin/sh -set -e +set -e; set -u while test $# -gt 0; do case $1 in --log-file) log_file=$2; shift;; + --trs-file) trs_file=$2; shift;; --test-name|--expect-failure|--color-tests|--enable-hard-errors) shift;; --) shift; break;; *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;; @@ -55,11 +59,16 @@ while test $# -gt 0; do shift done echo "log: $log_file" # For debugging. +echo "trs: $trs_file" # Ditto. case $log_file in */*);; *) exit 1;; esac -dir=`expr "$log_file" : '\(.*\)/[^/]*'` -echo "dir: $dir" # For debugging. -test -d "$dir" || exit 1 -echo dummy > "$log_file" +dir_log=`expr "$log_file" : '\(.*\)/[^/]*'` +dir_trs=`expr "$trs_file" : '\(.*\)/[^/]*'` +echo "dir_log: $dir_log" # For debugging. +echo "dir_trs: $dir_trs" # Likewise. +test x"$dir_trs" = x"$dir_log" || exit 1 +test -d "$dir_log" || exit 1 +echo dummy1 > "$log_file" +echo dummy2 > "$trs_file" END chmod a+x checkdir-driver diff --git a/tests/test-driver-custom-multitest.test b/tests/test-driver-custom-multitest.test index df0568c8e..4fb023c73 100755 --- a/tests/test-driver-custom-multitest.test +++ b/tests/test-driver-custom-multitest.test @@ -127,6 +127,7 @@ for vpath in : false; do # `trivial-test-driver' script is changed. $FGREP INVALID.NAME stdout test-suite.log && Exit 1 test -f BAD.LOG && Exit 1 + test -f BAD.TRS && Exit 1 # These log files must all have been created by the testsuite. cat pass.log cat fail.log diff --git a/tests/test-driver-custom-no-html.test b/tests/test-driver-custom-no-html.test index 683659b5f..1a2b72637 100755 --- a/tests/test-driver-custom-no-html.test +++ b/tests/test-driver-custom-no-html.test @@ -35,14 +35,14 @@ END cat > no-rst <<'END' #! /bin/sh +echo ':test-result: SKIP' > foo.trs +echo ':copy-in-global-log: yes' >> foo.trs # The genereted log file is deliberately syntactically invalid # reStructuredText. cat > foo.log <<'EoL' SKIP: FooBar ============= -:test-result: SKIP - -------------- dummy title EoL diff --git a/tests/test-driver-custom-xfail-tests.test b/tests/test-driver-custom-xfail-tests.test index 4f8bb83ea..e68e04b34 100755 --- a/tests/test-driver-custom-xfail-tests.test +++ b/tests/test-driver-custom-xfail-tests.test @@ -88,15 +88,17 @@ done cat > td <<'END' #! /bin/sh -set -e +set -e; set -u test_name=INVALID log_file=/dev/null +trs_file=/dev/null expect_failure=no while test $# -gt 0; do case $1 in --test-name) test_name=$2; shift;; --expect-failure) expect_failure=$2; shift;; --log-file) log_file=$2; shift;; + --trs-file) trs_file=$2; shift;; # Ignored. --color-tests) shift;; --enable-hard-errors) shift;; @@ -109,23 +111,22 @@ while test $# -gt 0; do done st=0 "$@" || st=$? -rm -f "$log_file" case $st,$expect_failure in 0,no) echo "PASS: $test_name" | tee "$log_file" - echo ":test-result: PASS" >> "$log_file" + echo ":test-result: PASS" > "$trs_file" ;; 1,no) echo "FAIL: $test_name" | tee "$log_file" - echo ":test-result: FAIL" >> "$log_file" + echo ":test-result: FAIL" > "$trs_file" ;; 0,yes) echo "XPASS: $test_name" | tee "$log_file" - echo ":test-result: XPASS" >> "$log_file" + echo ":test-result: XPASS" > "$trs_file" ;; 1,yes) echo "XFAIL: $test_name" | tee "$log_file" - echo ":test-result: XFAIL" >> "$log_file" + echo ":test-result: XFAIL" > "$trs_file" ;; *) echo "INTERNAL ERROR" >&2 @@ -141,21 +142,24 @@ $AUTOMAKE ./configure -VERBOSE=yes $MAKE check -test -f test-suite.log -test -f sub1/test-suite.log -test -f sub2/test-suite.log +$MAKE check >stdout || { cat stdout; Exit 1; } +cat stdout +test `grep -c '^PASS:' stdout` -eq 3 +test `grep -c '^XFAIL:' stdout` -eq 13 for dir in sub1 sub2; do cd $dir cp pass.test x1.test cp x2.test pass.test - $MAKE check && { cat test-suite.log; Exit 1; } - cat test-suite.log - grep '^FAIL: pass\.test$' test-suite.log - grep '^XPASS: x1\.test$' test-suite.log - test `grep -c '^FAIL' test-suite.log` -eq 1 - test `grep -c '^XPASS' test-suite.log` -eq 1 + $MAKE check >stdout && { cat stdout; Exit 1; } + cat stdout + test "`cat pass.trs`" = ":test-result: FAIL" + test "`cat x1.trs`" = ":test-result: XPASS" + test "`cat x2.trs`" = ":test-result: XFAIL" + grep '^FAIL: pass\.test$' stdout + grep '^XPASS: x1\.test$' stdout + grep '^XFAIL: x2\.test$' stdout + count_test_results total=7 pass=0 xpass=1 fail=1 xfail=5 skip=0 error=0 cd .. done diff --git a/tests/test-driver-custom.test b/tests/test-driver-custom.test index b6a33d2fc..830504753 100755 --- a/tests/test-driver-custom.test +++ b/tests/test-driver-custom.test @@ -59,11 +59,13 @@ log_wflags='@log_wflags@' test_name=INVALID log_file=BAD.log +trs_file=BAD.trs extra_opts= while test $# -gt 0; do case $1 in --test-name) test_name=$2; shift;; --log-file) log_file=$2; shift;; + --trs-file) trs_file=$2; shift;; # Ignored. --expect-failure) shift;; --color-tests) shift;; @@ -79,6 +81,7 @@ while test $# -gt 0; do done echo "$me" "$test_name" $extra_opts > "$log_file" +: > "$trs_file" exec "$@" exit 127 @@ -123,6 +126,7 @@ VERBOSE=yes $MAKE check ls -l . sub test ! -r BAD.log +test ! -r BAD.trs echo 'chk-wrapper 1.chk --am-chk --chk' > 1.exp echo 'test-wrapper 2.test -am-test -test' > 2.exp diff --git a/tests/test-driver-end-metadata.test b/tests/test-driver-end-metadata.test deleted file mode 100755 index 42e7e57ad..000000000 --- a/tests/test-driver-end-metadata.test +++ /dev/null @@ -1,120 +0,0 @@ -#! /bin/sh -# Copyright (C) 2011 Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2, or (at your option) -# any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -# Custom test drivers and parallel-tests harness: check the documented -# semantics for inhibiting the scanning of the remaining portion of a -# `.log' file (for what concerns metadata). Currently, this is done -# with the use of the reStructuredText field `:end-metadata:'. - -parallel_tests=yes -. ./defs || Exit 1 - -cat >> configure.in << 'END' -AC_OUTPUT -END - -cat > Makefile.am << 'END' -TEST_LOG_DRIVER = ./passthrough-driver -TESTS = foo.test bar.test baz.test -END - -cat > passthrough-driver <<'END' -#!/bin/sh -set -e; set -u; -while test $# -gt 0; do - case $1 in - --log-file) log_file=$2; shift;; - --test-name) test_name=$2; shift;; - --expect-failure|--color-tests|--enable-hard-errors) shift;; - --) shift; break;; - *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;; - esac - shift -done -echo "$test_name: RUN" -cp $1 $log_file -END -chmod a+x passthrough-driver - -cat > foo.test < bar.test < baz.test <stdout || st=1 -# For debugging. -cat stdout -cat foo.log -cat bar.log -cat test-suite.log - -# Check that only the `:test-result:' in baz.test and the first one in -# foo.test have been considered. -test $st -eq 0 -grep '^# TOTAL: *2$' stdout -grep '^# PASS: *2$' stdout -for result in SKIP XPASS FAIL XFAIL ERROR; do - grep "^# $result: *0$" stdout -done - -# Check that a `:end-metadata:' directive does not prevent following text -# to be copied into the log file. -$FGREP 'foo foo foo' foo.log -$FGREP 'bar bar bar' bar.log -$FGREP 'baz baz baz' baz.log # More of a sanity check really. - -# Check that only the `:copy-in-global-log:' in bar.test has been -# considered. -$FGREP 'foo foo foo' test-suite.log -$FGREP 'bar bar bar' test-suite.log && Exit 1 -$FGREP 'baz baz baz' test-suite.log && Exit 1 - -# Check that only the `:recheck:' in bar.test has been -# considered. -$MAKE recheck >stdout || { cat stdout; Exit 1; } -cat stdout -grep '^foo\.test: RUN$' stdout -grep 'ba[rz]\.test.*RUN' stdout && Exit 1 - -: diff --git a/tests/test-driver-metadata-no-leading-space.test b/tests/test-driver-metadata-no-leading-space.test deleted file mode 100755 index 4c3f02669..000000000 --- a/tests/test-driver-metadata-no-leading-space.test +++ /dev/null @@ -1,94 +0,0 @@ -#! /bin/sh -# Copyright (C) 2011 Free Software Foundation, Inc. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2, or (at your option) -# any later version. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -# Custom test drivers and parallel-tests harness: check that, in the -# log files, the reStructuredText fields for metadata declaration are -# recognized only when placed at the beginning of a line. - -parallel_tests=yes -. ./defs || Exit 1 - -cat >> configure.in << 'END' -AC_OUTPUT -END - -cat > Makefile.am << 'END' -TEST_LOG_DRIVER = ./passthrough-driver -TESTS = foo.test -END - -cat > passthrough-driver <<'END' -#!/bin/sh -set -e; set -u; -while test $# -gt 0; do - case $1 in - --log-file) log_file=$2; shift;; - --test-name) test_name=$2; shift;; - --expect-failure|--color-tests|--enable-hard-errors) shift;; - --) shift; break;; - *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;; - esac - shift -done -echo "$test_name: RUN" -cp $1 $log_file -END -chmod a+x passthrough-driver - -cat > foo.test <stdout || st=1 -# For debugging. -cat stdout -cat foo.log -cat test-suite.log - -# Check that only the second `:test-result:' has been processed. -test $st -eq 0 -grep '^# TOTAL: *1$' stdout -grep '^# PASS: *1$' stdout -for result in SKIP XPASS FAIL XFAIL ERROR; do - grep "^# $result: *0$" stdout -done - -# Check that `:copy-in-global-log:' hasn't been processed. -grep 'foo foo foo' test-suite.log - -# Check that `:recheck:' hasn't been processed. -: > older -$sleep -$MAKE recheck >stdout || { cat stdout; Exit 1; } -cat stdout -grep '^foo\.test: RUN$' stdout -grep 'foo foo foo' test-suite.log -is_newest test-suite.log older -is_newest foo.log older - -: diff --git a/tests/test-driver-strip-vpath.test b/tests/test-driver-strip-vpath.test index f1b316c5f..5b21cc0ff 100755 --- a/tests/test-driver-strip-vpath.test +++ b/tests/test-driver-strip-vpath.test @@ -51,6 +51,7 @@ set -e while test $# -gt 0; do case $1 in --log-file) log_file=$2; shift;; + --trs-file) trs_file=$2; shift;; --test-name) test_name=$2; shift;; --expect-failure|--color-tests|--enable-hard-errors) shift;; --) shift; break;; @@ -64,6 +65,7 @@ case $test_name in *) exit 1;; esac echo dummy > "$log_file" +echo dummy > "$trs_file" END chmod a+x checkstrip-driver diff --git a/tests/test-driver-trs-suffix-registered.test b/tests/test-driver-trs-suffix-registered.test new file mode 100755 index 000000000..7f6522a5e --- /dev/null +++ b/tests/test-driver-trs-suffix-registered.test @@ -0,0 +1,58 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# parallel-tests support: the following are registered with `.SUFFIXES': +# - .log +# - .trs (used by files that store test results and metadata) +# - .test if $(TEST_EXTENSIONS) is not defined +# - stuff in $(TEST_EXTENSIONS) otherwise + +parallel_tests=yes +. ./defs || Exit 1 + +: > Makefile.am + +cat > 1.am << 'END' +TESTS = +END + +cat > 2.am << 'END' +TEST_EXTENSIONS = .SH .abcdef +TESTS = +END + +: > test-driver + +$ACLOCAL + +$AUTOMAKE 1 +$AUTOMAKE 2 + +sed -e 's/$/ /' 1.in > mk.1 +sed -e 's/$/ /' 2.in > mk.2 + +grep '^\.SUFFIXES:' mk.1 +grep '^\.SUFFIXES:' mk.2 + +for suf in test log trs; do + grep "^\\.SUFFIXES:.* \\.$suf " mk.1 +done + +for suf in SH abcdef log trs; do + grep "^\\.SUFFIXES:.* \\.$suf " mk.2 +done + +: diff --git a/tests/test-driver-global-log.test b/tests/test-metadata-global-log.test similarity index 62% rename from tests/test-driver-global-log.test rename to tests/test-metadata-global-log.test index 7596e943f..a6f09991d 100755 --- a/tests/test-driver-global-log.test +++ b/tests/test-metadata-global-log.test @@ -17,7 +17,8 @@ # Custom test drivers and parallel-tests harness: check the documented # semantics for deciding when the content of a test log file should be # copied in the global test-suite.log file. Currently, this is done -# with the use of the reStructuredText field `:copy-in-global-log:'. +# with the use of the reStructuredText field `:copy-in-global-log:' in +# the associated `.trs' files. parallel_tests=yes . ./defs || Exit 1 @@ -26,86 +27,104 @@ cat >> configure.in << 'END' AC_OUTPUT END +cat > Makefile.am << 'END' +TEST_LOG_DRIVER = ./passthrough-driver +TEST_LOG_COMPILER = $(SHELL) -e +END + +cat > passthrough-driver <<'END' +#!/bin/sh +set -e; set -u; +while test $# -gt 0; do + case $1 in + --log-file) log_file=$2; shift;; + --trs-file) trs_file=$2; shift;; + --test-name) test_name=$2; shift;; + --expect-failure|--color-tests|--enable-hard-errors) shift;; + --) shift; break;; + *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;; + esac + shift +done +echo "$test_name: RUN" +"$@" >$log_file 2>&1 5>$trs_file +END +chmod a+x passthrough-driver + # The `:test-result:' and `:recheck:' fields and the first line of the # log file should be be irrelevant for the decision of whether a test # output is to be copied in the `test-suite.log'. cat > no-1.test <&5 +echo :copy-in-global-log: no >&5 +echo :test-result: FAIL >&5 +echo :test-result: XPASS >&5 +echo not seen 1 +END + +# In the last line, with leading and trailing whitespace in the value. +cat > no-2.test <&5 +echo "not seen 2" +echo ":recheck: yes" >&5 +echo ":copy-in-global-log:$tab $tab no $tab" >&5 END for RES in XPASS FAIL XFAIL SKIP ERROR UNKNOWN; do unindent > $RES.test <&5 + echo :copy-in-global-log: no >&5 + echo not seen $RES END done -# In the last line, with leading and trailing whitespace. -cat > no-2.test < no-3.test <&5 +echo ":test-result: FAIL" >&5 +echo "not seen 3" +END + +# Leading whitespace before the field. +cat > no-4.test <&5 +echo " $tab $tab$tab :copy-in-global-log: no" >&5 +echo "not seen 4" END cat > yes-1.test <&5 +echo :copy-in-global-log: yes >&5 +echo seen yes 1 END # A lacking `:copy-in-global-log:' implies that the content of # the log file should be copied. cat > yes-2.test <&5 +echo seen yes 2 END -# Corner cases. -echo ':copy-in-global-log:' > corn-1.test -echo " $tab $tab$tab" > corn-2.test -cat > corn-3.test <<'END' -seen corn 31 -:copy-in-global-log:#@%! -seen corn 32 +# Three corner cases. + +cat > corn-1.test <&5 END -echo TEST_LOG_DRIVER = ./dummy-driver > Makefile.am -echo TESTS = *.test >> Makefile.am +cat > corn-2.test <&5 +END -cat > dummy-driver <<'END' -#!/bin/sh -set -e -while test $# -gt 0; do - case $1 in - --log-file) log_file=$2; shift;; - --test-name) test_name=$2; shift;; - --expect-failure|--color-tests|--enable-hard-errors) shift;; - --) shift; break;; - *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;; - esac - shift -done -cp $1 $log_file +cat > corn-3.test <<'END' +echo seen corn 31 +echo ':copy-in-global-log:#@%!' >&5 +echo seen corn 32 END -chmod a+x dummy-driver + +echo TESTS = *.test >> Makefile.am $ACLOCAL $AUTOCONF @@ -117,15 +136,12 @@ $AUTOMAKE # should be checked in other tests. $MAKE check || : cat test-suite.log - grep '^seen yes 1$' test-suite.log grep '^seen yes 2$' test-suite.log +grep '^seen corn 1$' test-suite.log +grep '^seen corn 2$' test-suite.log grep '^seen corn 31$' test-suite.log grep '^seen corn 32$' test-suite.log -grep '^:copy-in-global-log:$' test-suite.log -grep "^ $tab $tab$tab$" test-suite.log $FGREP 'not seen' test-suite.log && Exit 1 -$EGREP '^(XPASS|FAIL|SKIP|ERROR|UNKNOWN)' test-suite.log && Exit 1 -test `grep -c ':test-result:' test-suite.log` -eq 2 : diff --git a/tests/test-driver-recheck.test b/tests/test-metadata-recheck.test similarity index 93% rename from tests/test-driver-recheck.test rename to tests/test-metadata-recheck.test index a8654c1dc..6a372f39a 100755 --- a/tests/test-driver-recheck.test +++ b/tests/test-metadata-recheck.test @@ -15,9 +15,7 @@ # along with this program. If not, see . # Test the "make recheck" semantics for custom test drivers, as documented -# in the Automake manual. Currently, that is based on the reStructuredText -# field `:recheck:'. - +# in the Automake manual. parallel_tests=yes . ./defs || Exit 1 @@ -44,12 +42,12 @@ echo ":recheck: who cares" > y-5 echo ":recheck: $tab y" > y-6 echo ":recheck: yeah!$tab$tab " > y-7 cat > y-10 < y-11 < y-12 < y-8 + # The :test-result: fields and the fist line of the log should be # irrelevant for the decision of whether "make recheck" should or # should not re-run a test. @@ -83,12 +83,12 @@ echo ":recheck: no " > n-2 echo ":recheck: $tab no" > n-3 echo ":recheck: no $tab$tab " > n-4 cat > n-5 < n-6 < n-7 < n-8 + # The :test-result: fields should be irrelevant for the decision of # whether "make recheck" should or should not re-run a test. cat > n-100 < dummy-driver <<'END' #!/bin/sh -set -e +set -e; set -u while test $# -gt 0; do case $1 in --log-file) log_file=$2; shift;; + --trs-file) trs_file=$2; shift;; --test-name) test_name=$2; shift;; --expect-failure|--color-tests|--enable-hard-errors) shift;; --) shift; break;; @@ -129,7 +132,8 @@ while test $# -gt 0; do shift done : > $test_name.run -cp $1 $log_file +: > $log_file +cp $1 $trs_file END chmod a+x dummy-driver diff --git a/tests/test-metadata-results.test b/tests/test-metadata-results.test new file mode 100755 index 000000000..effa4881a --- /dev/null +++ b/tests/test-metadata-results.test @@ -0,0 +1,169 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Parallel testsuite harness: check APIs for the registering of test +# results in `*.trs' files, as documented in the automake manual. + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in << 'END' +AC_OUTPUT +END + +cat > Makefile.am << 'END' +TEST_LOG_DRIVER = ./dummy-driver +TESTS = foo.test bar.test +END + +cat > dummy-driver <<'END' +#! /bin/sh +set -e; set -u +while test $# -gt 0; do + case $1 in + --log-file) log_file=$2; shift;; + --trs-file) trs_file=$2; shift;; + --test-name) test_name=$2; shift;; + --expect-failure|--color-tests|--enable-hard-errors) shift;; + --) shift; break;; + *) echo "$0: invalid option/argument: '$1'" >&2; exit 2;; + esac + shift +done +: > $log_file +cp $1 $trs_file +END +chmod a+x dummy-driver + +mk_check () +{ + st=0 + $MAKE check >stdout || st=$? + cat stdout + # Our dummy driver make no testsuite progress report. + grep ': .*\.test' stdout && Exit 1 + # Nor it writes to the log files. + test -s foo.log && Exit 1 + test -s bar.log && Exit 1 + return $st +} + +# This must be different from the one defined in `test/defs', as that +# assumes that the driver does proper testsuite progress reporting. +count_test_results () +{ + total=ERR pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR + eval "$@" + st=0 + grep "^# TOTAL: *$total$" stdout || rc=1 + grep "^# PASS: *$pass$" stdout || rc=1 + grep "^# XFAIL: *$xfail$" stdout || rc=1 + grep "^# SKIP: *$skip$" stdout || rc=1 + grep "^# FAIL: *$fail$" stdout || rc=1 + grep "^# XPASS: *$xpass$" stdout || rc=1 + grep "^# ERROR: *$error$" stdout || rc=1 + test $st -eq 0 || Exit 1 +} + +$ACLOCAL +$AUTOCONF +$AUTOMAKE + +./configure + +# Basic checks. Also that that "old-style" directives with format +# "RESULT: test-name" are be ignored now. + +cat > foo.test < bar.test +mk_check +count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0 + +cat > foo.test < bar.test +: > bar.test +mk_check && Exit 1 +count_test_results total=1 pass=0 fail=1 xpass=0 xfail=0 skip=0 error=0 + +cat > foo.test < bar.test < foo.test < bar.test < foo.test < bar.test +mk_check && Exit 1 +count_test_results total=6 pass=1 fail=1 xpass=1 xfail=1 skip=1 error=1 + +cp foo.test bar.test +mk_check && Exit 1 +count_test_results total=12 pass=2 fail=2 xpass=2 xfail=2 skip=2 error=2 + +# Check that we are liberal w.r.t. whitespace use. + +: > foo.test +: > bar.test +for RESULT in PASS FAIL XPASS XFAIL SKIP ERROR; do + sed -e 's/^ *//' -e 's/|//g' >> foo.test <> bar.test +done +cat foo.test # For debugging. +cat bar.test # Likewise. +mk_check && Exit 1 +count_test_results total=30 pass=5 fail=5 xpass=5 xfail=5 skip=5 error=5 + +: diff --git a/tests/test-missing.test b/tests/test-missing.test new file mode 100755 index 000000000..a4ac179c8 --- /dev/null +++ b/tests/test-missing.test @@ -0,0 +1,72 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# parallel-tests: +# - non-existent scripts listed in TESTS get diagnosed +# See also related test 'test-missing2.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in << 'END' +AC_OUTPUT +END + +cat > Makefile.am << 'END' +TESTS = ok.test zardoz.test +TEST_LOG_COMPILER = true +END + +: > ok.test + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure + +$MAKE check >output 2>&1 && { cat output; Exit 1; } +cat output +test -f ok.log +grep '^PASS: ok\.test' output +$FGREP 'zardoz.log' output +test ! -f test-suite.log + +TESTS='zardoz2.test' $MAKE -e check >output 2>&1 \ + && { cat output; Exit 1; } +cat output +$FGREP 'zardoz2.log' output +test ! -f test-suite.log + +TEST_LOGS='zardoz3.log' $MAKE -e check >output 2>&1 \ + && { cat output; Exit 1; } +cat output +$FGREP 'zardoz3.log' output +test ! -f test-suite.log + +# The errors should persist even after `test-suite.log' +# has been created. + +: > zardoz.test +$MAKE check +rm -f zardoz.test + +$MAKE check >output 2>&1 && { cat output; Exit 1; } +cat output +$FGREP 'zardoz.log' output +test ! -f test-suite.log + +: diff --git a/tests/test-missing2.test b/tests/test-missing2.test new file mode 100755 index 000000000..bc2979af7 --- /dev/null +++ b/tests/test-missing2.test @@ -0,0 +1,54 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# parallel-tests: +# - non-existent scripts listed in TESTS get diagnosed, even when +# all the $(TEST_LOGS) have a dummy dependency. +# See also related test 'test-missing.test'. + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in << 'END' +AC_OUTPUT +END + +cat > Makefile.am << 'END' +TESTS = foobar1.test foobar2.test +$(TEST_LOGS): +END + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure + +$MAKE foobar1.log foobar2.log || Exit 99 +test ! -f foobar1.log || Exit 99 +test ! -f foobar1.trs || Exit 99 +test ! -f foobar2.log || Exit 99 +test ! -f foobar2.trs || Exit 99 + +$MAKE check >output 2>&1 && { cat output; Exit 1; } +cat output +grep 'test-suite\.log.*foobar1\.log' output +grep 'test-suite\.log.*foobar1\.trs' output +grep 'test-suite\.log.*foobar2\.log' output +grep 'test-suite\.log.*foobar2\.trs' output +test ! -f test-suite.log + +: diff --git a/tests/test-trs-basic.test b/tests/test-trs-basic.test new file mode 100755 index 000000000..33367d34b --- /dev/null +++ b/tests/test-trs-basic.test @@ -0,0 +1,138 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check parallel harness features: +# - creation and removal of `.trs' auxiliary files +# - check some internals regarding the use of `.trs' files. + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in < Makefile.am << 'END' +TEST_EXTENSIONS = .sh .test +TESTS = foo.test bar.sh sub/zardoz.test +TEST_LOG_COMPILER = $(SHELL) +SH_LOG_COMPILER = $(SHELL) + +## Used to check some internal details. +tb: + echo $(am__TEST_BASES) > $@ +END + +mkdir sub +echo 'exit $FOO_STATUS' > foo.test +: > bar.sh +: > sub/zardoz.test + +FOO_STATUS=0; export FOO_STATUS + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure + +# +# Check some internal details first. +# + +$MAKE tb +test x"`cat tb`" = x"foo bar sub/zardoz" +rm -f tb +TESTS='foo.test foo2.sh foo3.exe foo4' $MAKE -e tb +test x"`cat tb`" = x"foo foo2 foo3.exe foo4" +rm -f tb + +# +# The `test-suite.stamp' file and the `.trs' files get created by +# "make check" and removed by "make clean" and "make mostlyclean". +# + +: > unrelated.trs +: > sub/foo.trs + +$MAKE check +test -f foo.trs +test -f bar.trs +test -f sub/zardoz.trs +$MAKE clean +test ! -f foo.trs +test ! -f bar.trs +test ! -f sub/zardoz.trs +# Unrelated `.trs' files shouldn't be removed. +test -f unrelated.trs +test -f sub/foo.trs + +# The files should be properly created in case of testsuite failure too. +FOO_STATUS=1 $MAKE check && Exit 1 +test -f foo.trs +test -f bar.trs +test -f sub/zardoz.trs +$MAKE mostlyclean +test ! -f foo.trs +test ! -f bar.trs +test ! -f sub/zardoz.trs +# Unrelated `.trs' files shouldn't be removed. +test -f unrelated.trs +test -f sub/foo.trs + +# +# Try with a subset of TESTS. +# + +TESTS=foo.test $MAKE -e check +test -f foo.trs +test ! -f bar.trs +test ! -f sub/zardoz.trs +$MAKE clean +test ! -f foo.trs +TESTS='foo.test bar.sh' $MAKE -e check +test -f foo.trs +test -f bar.trs +test ! -f sub/zardoz.trs +# "make clean" shouldn't remove `.trs' files for tests not in $(TESTS). +TESTS=bar.sh $MAKE -e clean +test -f foo.trs +test ! -f bar.trs + +$MAKE clean + +# +# Try with a subset of TEST_LOGS. +# + +TEST_LOGS=sub/zardoz.log $MAKE -e check +test ! -f foo.trs +test ! -f bar.trs +test -f sub/zardoz.trs +$MAKE clean +test ! -f sub/zardoz.trs +TEST_LOGS='foo.log bar.log' $MAKE -e check +test -f foo.trs +test -f bar.trs +test ! -f sub/zardoz.trs +# "make clean" shouldn't remove `.trs' files for tests whose log +# is not in $(TEST_LOGS). +TEST_LOGS=foo.log $MAKE -e clean +test ! -f foo.trs +test -f bar.trs +test ! -f sub/zardoz.trs + +: diff --git a/tests/test-trs-recover.test b/tests/test-trs-recover.test new file mode 100755 index 000000000..7fff3ae5c --- /dev/null +++ b/tests/test-trs-recover.test @@ -0,0 +1,166 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check parallel harness features: +# - recovery from deleted `.trs' files, in various scenarios +# This test is complex and tricky, but that's acceptable since we are +# testing semantics that are potentially complex and tricky. + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in < Makefile.am << 'END' +TESTS = foo.test bar.test baz.test +TEST_LOG_COMPILER = $(SHELL) +END + +echo 'exit $TEST_STATUS' > foo.test +echo 'exit $TEST_STATUS' > bar.test +: > baz.test + +TEST_STATUS=0; export TEST_STATUS + +# Slower and possible overkill in some situations, but also clearer +# and safer. +update_stamp () { $sleep && touch stamp && $sleep; } + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure + +: Create the required log files. +$MAKE check + +: Recreate by hand. +rm -f foo.trs bar.trs baz.trs +$MAKE foo.trs +test -f foo.trs +test ! -f bar.trs +test ! -f baz.trs + +: Recreate by hand, several at the same time. +rm -f foo.trs bar.trs baz.trs +$MAKE foo.trs bar.trs +test -f foo.trs +test -f bar.trs +test ! -f baz.trs + +: Recreate by hand, with a failing test. +rm -f foo.trs bar.trs +TEST_STATUS=1 $MAKE bar.trs baz.trs >stdout || { cat stdout; Exit 1; } +cat stdout +test ! -f foo.trs +test -f bar.trs +test -f baz.trs +grep '^FAIL: bar\.test' stdout +$EGREP '^(baz|foo)\.test' stdout && Exit 1 + +: Recreate with a sweeping "make check", and ensure that also up-to-date +: '.trs' files are remade. +update_stamp +rm -f foo.trs bar.trs +$MAKE check +test -f foo.trs +test -f bar.trs +is_newest baz.trs stamp + +: Recreate with a sweeping "make check" with failing tests. Again, +: ensure that also up-to-date '.trs' files are remade -- this time we +: grep the "make check" output verify that. +rm -f foo.trs bar.trs +TEST_STATUS=1 $MAKE check >stdout && { cat stdout; Exit 1; } +test -f foo.trs +test -f bar.trs +grep '^FAIL: foo\.test' stdout +grep '^FAIL: bar\.test' stdout +grep '^PASS: baz\.test' stdout + +: Recreate with a "make check" with redefined TESTS. +rm -f foo.trs bar.trs baz.trs +TESTS=foo.test $MAKE -e check +test -f foo.trs +test ! -f bar.trs +test ! -f baz.trs + +: Recreate with a "make check" with redefined TEST_LOGS. +rm -f foo.trs bar.trs baz.trs +TEST_LOGS=bar.log $MAKE -e check +test ! -f foo.trs +test -f bar.trs +test ! -f baz.trs + +: Interactions with "make recheck" are OK. +rm -f foo.trs bar.trs baz.log baz.trs +$MAKE recheck >stdout || { cat stdout; Exit 1; } +cat stdout +test -f foo.trs +test -f bar.trs +test ! -f baz.trs +test ! -f baz.log +grep '^PASS: foo\.test' stdout +grep '^PASS: bar\.test' stdout +grep 'baz\.test' stdout && Exit 1 +count_test_results total=2 pass=2 fail=0 xpass=0 xfail=0 skip=0 error=0 + +: Setup for the next check. +$MAKE check +test -f foo.trs +test -f bar.trs +test -f baz.trs + +: Recreate by remaking the global test log, and ensure that up-to-date +: '.trs' files are *not* remade. +update_stamp +rm -f foo.trs bar.trs test-suite.log +$MAKE test-suite.log >stdout || { cat stdout; Exit 1; } +cat stdout +grep '^PASS: foo\.test' stdout +grep '^PASS: bar\.test' stdout +grep 'baz\.test' stdout && Exit 1 +stat *.trs *.log stamp || : # For debugging. +# Check that make has updated what it needed to, but no more. +test -f foo.trs +test -f bar.trs +is_newest stamp baz.trs +is_newest test-suite.log foo.trs bar.trs + +: Setup for the next check. +$MAKE check +test -f foo.trs +test -f bar.trs +test -f baz.trs + +: Interactions with lazy test reruns are OK. +rm -f foo.trs +update_stamp +touch bar.test +RECHECK_LOGS= $MAKE -e check >stdout || { cat stdout; Exit 1; } +cat stdout +# Check that make has updated what it needed to, but no more. +test -f foo.trs +is_newest bar.trs bar.test +is_newest stamp baz.trs +grep '^PASS: foo\.test' stdout +grep '^PASS: bar\.test' stdout +grep 'baz\.test' stdout && Exit 1 + +: diff --git a/tests/test-trs-recover2.test b/tests/test-trs-recover2.test new file mode 100755 index 000000000..bc94b3b7c --- /dev/null +++ b/tests/test-trs-recover2.test @@ -0,0 +1,133 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check parallel harness features: +# - recovery from unreadable `.trs' files, in various scenarios + +parallel_tests=yes +. ./defs || Exit 1 + +cat >> configure.in < Makefile.am << 'END' +TESTS = foo.test bar.test +TEST_LOG_COMPILER = true +END + +: > foo.test +: > bar.test + +TEST_STATUS=0; export TEST_STATUS + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure + +: > t +chmod a-r t +test ! -r t || Exit 77 +rm -f t + +: Create the required log files. +$MAKE check +test -f foo.trs +test -f bar.trs + +: Recreate with a sweeping "make check". +chmod a-r bar.trs +$MAKE check +test -f foo.trs +test -r foo.trs +test -f bar.trs +test -r bar.trs + +: Again, this time with one .trs file missing and the other +: one unreadable. +rm -f foo.trs +chmod a-r bar.trs +$MAKE check +test -f foo.trs +test -r foo.trs +test -f bar.trs +test -r bar.trs + +: Again, but using "make recheck" this time. +rm -f foo.trs +chmod a-r bar.trs +$MAKE recheck >stdout || { cat stdout; Exit 1; } +cat stdout +test -f foo.trs +test -r foo.trs +test -f bar.trs +test -r bar.trs +grep '^PASS: foo\.test' stdout +grep '^PASS: bar\.test' stdout + +: More complex interactions with "make recheck" are OK. +chmod a-r bar.log bar.trs +$MAKE recheck >stdout || { cat stdout; Exit 1; } +cat stdout +test -f bar.trs +test -r bar.trs +grep '^PASS: bar\.test' stdout +grep 'foo\.test' stdout && Exit 1 +count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0 + +: Recreate by remaking the global test log. +chmod a-r foo.trs +rm -f test-suite.log +$MAKE test-suite.log >stdout || { cat stdout; Exit 1; } +cat stdout +test -f foo.trs +test -r foo.trs +grep '^PASS: foo\.test' stdout +grep 'bar\.test' stdout && Exit 1 +# Also test that have only run before should be counted in the final +# testsuite summary. +grep '^# TOTAL: *2$' stdout + +: Setup for the next check. +: > baz.test +sed 's/^TESTS =.*/& baz.test/' Makefile > t +diff t Makefile && Exit 99 +mv -f t Makefile +$MAKE check +test -f foo.trs +test -f bar.trs +test -f baz.trs + +: Interactions with lazy test reruns are OK. +chmod a-r foo.trs +$sleep +touch stamp +$sleep +touch bar.test +RECHECK_LOGS= $MAKE -e check >stdout || { cat stdout; Exit 1; } +cat stdout +test -r foo.trs +is_newest bar.trs bar.test +grep '^PASS: foo\.test' stdout +grep '^PASS: bar\.test' stdout +grep 'baz\.test' stdout && Exit 1 +# Also test that have only run before should be counted in the final +# testsuite summary. +grep '^# TOTAL: *3$' stdout + +: diff --git a/tests/trivial-test-driver b/tests/trivial-test-driver index d334a9365..7b3ef10d5 100644 --- a/tests/trivial-test-driver +++ b/tests/trivial-test-driver @@ -36,10 +36,12 @@ set -u test_name=INVALID.NAME log_file=BAD.LOG +trs_file=BAD.TRS while test $# -gt 0; do case $1 in --test-name) test_name=$2; shift;; --log-file) log_file=$2; shift;; + --trs-file) trs_file=$2; shift;; # Ignored. --expect-failure) shift;; --color-tests) shift;; @@ -52,16 +54,18 @@ while test $# -gt 0; do shift done -## Run the test script, get test cases results, display them on console. +## Log file header. +{ + echo "RUN: $test_name" + echo "RUN: $test_name" | sed 's/./=/g' + echo +} > $log_file -tmp_output=$log_file-output.tmp -tmp_results=$log_file-results.tmp -tmp_status=$log_file-status.tmp +## Run the test script, get test cases results, display them on console. -"$@" 2>&1 | tee $tmp_output | ( +"$@" 2>&1 | tee -a $log_file | ( i=0 st=0 - exec 5> $tmp_results - : > $tmp_status + exec 5> $trs_file while read line; do result= case $line in @@ -81,38 +85,18 @@ tmp_status=$log_file-status.tmp echo >&5 fi done - test $st -eq 0 || echo fail > $tmp_status + if test $st -eq 0; then + recheck=no + copy_in_global_log=no + else + recheck=yes + copy_in_global_log=yes + fi + echo ":recheck: $recheck" >&5 + echo ":copy-in-global-log: $copy_in_global_log" >&5 + exec 5>&- ) | awk '{ print $0 ", testcase " NR }' -if test ! -s $tmp_status; then - global_result=PASS - recheck=no - copy_in_global_log=no -else - global_result=FAIL - recheck=yes - copy_in_global_log=yes -fi - -## Write the log file. - -{ - echo "$global_result: $test_name" - echo "$global_result: $test_name" | sed 's/./=/g' - # Blank lines required for having the output be valid RST. - echo - echo ":recheck: $recheck" - echo - echo ":copy-in-global-log: $copy_in_global_log" - echo - cat $tmp_results - echo - echo -------------------- - echo - cat $tmp_output -} > $log_file -rm -f $tmp_output $tmp_results $tmp_status - ## And we're done. exit 0 -- 2.47.2