From: Stefano Lattarini Date: Wed, 3 Aug 2011 13:46:02 +0000 (+0200) Subject: Merge branch 'test-protocols' into tap-testsuite-work X-Git-Tag: ng-0.5a~89^2~155 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=dffe171f1da9706598d650bf989eb845a9cf275f;p=thirdparty%2Fautomake.git Merge branch 'test-protocols' into tap-testsuite-work --- dffe171f1da9706598d650bf989eb845a9cf275f diff --cc ChangeLog index 33cd8ccb7,5d2757311..785b0e89a --- a/ChangeLog +++ b/ChangeLog @@@ -1,256 -1,647 +1,900 @@@ + 2011-08-03 Stefano Lattarini + + testsuite: one more use of TAP in our own tests + * tests/tap-bad-prog.test, tests/tap-bad-prog2.test: Merged + into ... + * tests/tap-bad-prog.tap: ... this TAP-generating test. + * tests/Makefile.am (tap_other_tests): Remove plan-bad-prog.test + and plan-bad-prog2.test. + (XFAIL_TESTS): Remove plan-bad-prog2.test. + (plan-bad-prog2.log): Remove. + + 2011-08-03 Stefano Lattarini + + testsuite: scaffolding to allow use of TAP in our own tests + Now that Automake has initial support for the TAP test protocol, + we can start "eating our own dog food" and rewrite some tests to + use TAP; this should allow for better granularity, especially in + the control of skips and expected failures. With this change, we + set up the initial scaffolding required by the planned TAP tests, + and convert few older tests to use TAP, mostly in order to verify + that there are no obvious errors. The tests in our testsuite that + use the TAP protocol will be marked by the new `.tap' extension. + * tests/plain-functions.sh: New file containing definition of + helper shell functions used by the "protocol-less" tests of the + Automake testsuite. + * tests/tap-functions.sh: New file containing definition of + helper shell functions used by the TAP-producing tests of the + Automake testsuite. + * tests/defs: Remove definitions of some functions that are + now defined in `plain-functions.sh'. Instead, source one of + `plain-functions.sh' or `tap-functions.sh', depending on whether + `$use_tap' is set to "no" or "yes". + (exit trap): Call `late_plan_' if `$use_tap' is true. Also, + unset shell traces before issuing the latest commands, to avoid + confusing the tap driver with spurious output. + * tests/defs-static.in ($use_tap): New variable, by default set + to "yes" if the calling test script has a `.tap' suffix, and to + "no" otherwise. The individual scripts can override it though. + In code sanity-checking the environment, verify that `$use_tap' + is not exported. + * tests/self-check-env-sanitize.test: Update, and small related + reformatting. + * tests/self-check-tap.test: New very minimal self test. + * tests/acoutbs.test, tests/acoutbs2.test, tests/acoutnoq.test, + tests/acoutpt.test, tests/acoutpt2.test, tests/acoutqnl.test: + Removed, merged into ... + * tests/ac-output-old.tap: ... this new test, that uses TAP. + * tests/Makefile.am (TAP_LOG_DRIVER): Define to invoke our own + `tap-driver' script. + (TAP_LOG_DRIVER_FLAGS): Define to `--merge', so that the stdout + and stderr of the test scripts remains synced. + (TEST_EXTENSIONS): Define, to add the `.tap' suffix; also list + the `.test' suffix explicitly. + (EXTRA_DIST): Distribute the new files `plain-functions.sh' and + `tap-functions.sh'. + ($(TEST_LOGS)): Depends on them. + (AM_TESTS_ENVIRONMENT): Ensure that `use_tap' is not exported in + the environment of the tests. + (TESTS): Update. + + 2011-08-03 Stefano Lattarini + + * NEWS: Fix typo, and related reformatting. + + 2011-08-03 Stefano Lattarini + + * NEWS: add mention of new experimental TAP support + + 2011-08-02 Stefano Lattarini + + testsuite: rename some test scripts + * tests/parallel-tests-am_tests_environment.test: Renamed ... + * tests/am-tests-environment.test: ... to this. + * tests/check-tests_environment.test: Renamed ... + * tests/tests-environment.test: ... to this. + * tests/parallel-test-driver-install.test: Renamed ... + * tests/parallel-tests-driver-install.test: ... to this. + * tests/parallel-tests-make-n.test: Renamed ... + * tests/parallel-tests-dry-run.test: ... to this. + * tests/Makefile.am (TESTS): Updated. + + 2011-08-02 Stefano Lattarini + + coverage: parallel-tests and lazy dependencies on EXTRA_PROGRAMS + * tests/parallel-tests-extra-programs.test: New test. + * tests/Makefile.am (TESTS): Update. + * doc/automake.texi (Parallel Test Harness): Add a comment + pointing to the new test. + + 2011-08-02 Stefano Lattarini + + docs: improve, extend and fix documentation on TAP support + * doc/automake.texi ("Using the TAP test protocol"): Divide this + section into ... + ("Introduction to TAP", "Use TAP with the Automake test harness", + "Incompatibilities with other TAP parsers and drivers", "Links + and external resources"): ... these subsections, extend them by + adding more information and examples, and improve them by removing + incomplete and/or temporary wordings and TODO items. + ("Script-based Testsuites", "Parallel Test Harness"): Add a couple + of anchors to improve the granularity of cross-references. + * tests/tap-doc2.test: New test, verifying the correctness of the + new examples given in the manual. + * tests/Makefile.am (tap_other_tests): Add the new test. + + 2011-08-01 Stefano Lattarini + + testsuite: separate the only failing check of an xfailing test + * tests/plan-bad-prog.test: Move the only failing check of this + test (i.e., the one about the total number of "ERROR" outcomes) + into ... + * tests/plan-bad-prog2.test: ... this new test. + * tests/Makefile.am (XFAIL_TESTS): Remove `plan-bad-prog.test', + add `plan-bad-prog2.test'. + (tap_other_tests): Add `plan-bad-prog2.test'. + (plan-bad-prog2.log): Depend on `plan-bad-prog.test'. + + 2011-08-01 Stefano Lattarini + + testsuite: fix spurious errors in an xfailing test on TAP support + * tests/plan-bad-prog.test: Fix typo in the name of the test + being made unreadable. Correct the wording of the potential + skip message, and simplify the condition under which the test + is to be skipped. Escape literal dots in grep regexps. + + 2011-08-01 Stefano Lattarini + + testsuite: better granularity in a couple of tests on TAP support + * tests/tap-summary-aux.sh: New auxiliary script, filled with code + moved out from ... + * tests/tap-summary.test: ... this test, from which the checks on + colored testsuite have further been moved into ... + * tests/tap-summary-color.test: ... this new test, which in turn + * tests/Makefile.am (tap_other_tests): Add `tap-summary-color.test'. + (EXTRA_DIST): Distribute `tap-summary-aux.sh' + (tap-summary.log): Depend on `tap-summary-aux.sh'. + (tap-summary-color.log): Likewise. + + 2011-08-01 Stefano Lattarini + + testsuite: optimize tests on TAP for speed and against duplication + The creation and configuration of common files and data used by + many tests on TAP is, with this change, factored out into a new + dedicated auxiliary test, from which those tests will depend upon. + This reduces code duplication in tests and, more importantly, + offers a noticeable speedup in the involved tests (30-40%). + * tests/tap-common-setup.test: New test, setting up the common + files and data used by various tests on TAP support. + * tests/tap-setup.sh: New file, to be sourced by tests wanting to + bring in data generated by `tap-common-setup.test'. + * tests/tap-autonumber.test: Update to use the pre-computed data + files. + * tests/tap-bailout.test: Likewise. + * tests/tap-color.test: Likewise. + * tests/tap-deps.test: Likewise. + * tests/tap-diagnostic.test: Likewise. + * tests/tap-empty-diagnostic.test: Likewise. + * tests/tap-empty.test: Likewise. + * tests/tap-escape-directive.test: Likewise. + * tests/tap-exit.test: Likewise. + * tests/tap-signal.test: Likewise. + * tests/tap-fancy.test: Likewise. + * tests/tap-fancy2.test: Likewise. + * tests/tap-global-log.test: Likewise. + * tests/tap-global-result.test: Likewise. + * tests/tap-html.test: Likewise. + * tests/tap-log.test: Likewise. + * tests/tap-merge-stdout-stderr.test: Likewise. + * tests/tap-no-merge-stdout-stderr.test: Likewise. + * tests/tap-message-0.test: Likewise. + * tests/tap-no-disable-hard-error.test: Likewise. + * tests/tap-no-spurious-summary.test: Likewise. + * tests/tap-no-spurious.test: Likewise. + * tests/tap-not-ok-skip.test: Likewise. + * tests/tap-numeric-description.test: Likewise. + * tests/tap-out-of-order.test: Likewise. + * tests/tap-passthrough.test: Likewise. + * tests/tap-passthrough-exit.test: Likewise. + * tests/tap-plan.test: Likewise. + * tests/tap-plan-corner.test: Likewise. + * tests/tap-plan-corner2.test: Likewise. + * tests/tap-plan-errors.test: Likewise. + * tests/tap-realtime.test: Likewise. + * tests/tap-recheck-logs.test: Likewise. + * tests/tap-skip-whole.test: Likewise. + * tests/tap-todo-skip-together.test: Likewise. + * tests/tap-todo-skip-whitespace.test: Likewise. + * tests/tap-todo-skip.test: Likewise. + * tests/tap-unplanned.test: Likewise. + * tests/tap-with-and-without-number.test: Likewise. + * tests/tap-xfail-tests.test: Likewise. + * tests/tap-skip-whole-whitespace.test: Likewise, and remove + redundant definitions of `$sp' and `$tab' (they are already + defined in `tests/defs'). + * tests/tap-whitespace-normalization.test: Likewise. + * tests/Makefile.am (TESTS): Update. + (EXTRA_DIST): Distribute `tap-setup.sh'. + (tap_with_common_setup_tests, tap_with_common_setup_logs)): New + variables, holding respectively the list of tests using the files + pre-computed by `tap-common-setup.test', and the list of their + corresponding log files. + (tap_other_tests): New variable, holding the list of other tests + on TAP support. + ($(tap_with_common_setup_logs)): Depend on `tap-common-setup.log' + and `tap-setup.sh'. + + 2011-08-01 Stefano Lattarini + + test harness: use new `.trs' files to hold test metadata + With this change, the test harness will keep test metadata in + dedicated `.trs' files, instead of having them embedded into the + `.log' files. This allows for easier forward-compatibility and + extension of test metadata, and for more flexibility in the + format of the `.log' files. Note that this change makes the + `:end-metadata:' field obsolete. + * doc/automake.texi (Parallel Test Harness, Log files generation + and test results recording): Document the new APIs and behaviour; + some related minor rewordings and fixlets. + * NEWS: Update. + * automake.in (handle_tests): When bringing in the content of + `check2.am', substitute %BASE% with the basename of the `.log' + file being created by a rule. Add the generated `.trs' files + to the list of files to be cleaned by "make mostlyclean". + * lib/am/check.am (am__test_driver_flags): Rename ... + (am__common_driver_flags): ... to this, and remove the flags + `--test-name' and `--log-file' from it: they are now define in + the proper rules in `check2.am'. + (am__TEST_BASES): New internal variable, holding the names of + the tests, with any registered extension removed. + (am__stealth_MAKE): New internal variable, can be used instead of + $(MAKE) in recipes requiring a recursive call to make, but which + are not intended to be executed by "make -n". + (.log.trs): New suffix rule, to recover from deletion of `.trs' + files. + ($(TEST_SUITE_LOG)): Almost completely rewritten to follow the + new API of "test logs in `.log' files, test metadata in `.trs' + files". It goes to some length to work correctly in face of + unreadable or missing `.log' and `.trs' files, and to error out + with proper error messages when this is not possible. + [%?PARALLEL_TESTS%] (check-TESTS): Also remove relevant "stale" + `.trs' files (in addition to `.log files) before remaking the + $(TEST_SUITE_LOG). + (recheck, recheck-html): Look for the `:recheck:' field in the + `.trs' files, not in the `.log' files. + * lib/am/check2.am (?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Adjust + the call to the test driver, in particularly passing the new + option `--trs-file'. + [%am__EXEEXT%] (?GENERIC?%EXT%$(EXEEXT).log): Likewise. + * lib/tap-driver ($USAGE): Adjust the help screen. + (Getopt::Long::GetOptions): Handle the `--trs-file' option, + through the use of ... + ($trs_file): ... this new global variable. + (finish): Write metadata for the test run to `$trs_file' rather + then to `$log_file', through the use of ... + (write_test_results): ... this new function. + * lib/test-driver (print_usage): Update the help screen. + (Option parsing): Handle the `--trs-file' option, through the + use of ... + ($resfile): ... this new global variable. + (Main code): Write metadata for the test run to `$trsfile' rather + than to `$logfile'. + Minor related adjustments to comments. + * tests/.gitignore: Ignore `*.trs' files. + * tests/parallel-tests-unreadable-log.test: Moved ... + * tests/parallel-tests-unreadable.test: ... to this, and extended + to also check the semantics for unreadable `.trs' files. + * tests/test-driver-end-metadata.test: Deleted as obsolete. + * tests/test-driver-metadata-no-leading-space.test: Likewise. + * tests/test-driver-global-log.test: Renamed ... + * tests/test-metadata-global-log.test: ... to this, and modified + as to verify the new APIs and semantics. + * tests/test-driver-recheck: Renamed ... + * tests/test-metadata-recheck.test: ... to this, and modified + likewise. + * tests/parallel-tests-once.test: New test. + * tests/parallel-tests-make-n.test: Likewise. + * test-metadata-results.test: Likewise. + * test-missing.test: Likewise. + * test-missing2.test: Likewise. + * test-trs-basic.test: Likewise. + * test-trs-recover.test: Likewise. + * test-trs-recover2.test: Likewise. + * tests/Makefile.am (TESTS): Update. + + 2011-07-27 Stefano Lattarini + + test harness: allow more metadata in log files + This change reworks and improves the parallel test harness to use + more specialized reStructuredText fields in the log files (instead + of relying on specially-placed of "magic lines" and more indirect + semantical formatting); the new fields are the following: + - ":recheck:": tell whether the associated test will have to be + re-run by "make recheck"; + - ":copy-in-global-log:": tell whether the content of the log + file should be copied in the "global log" `test-suite.log'; + - ":end-metadata:", which inhibits the scanning of the rest of + the log file (for what concerns test metadata). + Also, the special `:test-result:' value "END" has been removed, + superseded by the new `:end-metadata:' field. + * doc/automake.texi (Log files generation and test results + recording): Document the new API and semantics. Remove or fix + some obsolete comments. + * lib/am/check.am ($(TEST_SUITE_LOG), recheck, recheck-html): + Adjust comments and code. + * lib/tap-driver (finish): Adjust, with the help of ... + (must_recheck, copy_in_global_log): ... these new functions. + * lib/test-driver (Main code): Adjust, with the help of ... + ($recheck, $gcopy): ... these new variables. + * tests/trivial-test-driver: Update to obey the new APIs. + * tests/test-driver-recheck.test: Likewise. + * tests/test-driver-global-log.test: Likewise. + * tests/tap-passthrough.test: Relax the test, avoiding to check + what is written in `test-suite.log'; such check has been moved ... + * tests/tap-global-log.test: ... in this new test, and extended. + * tests/test-driver-metadata-no-leading-space.test: New test. + * tests/test-driver-end-test-results.test: Removed, it checked + the old APIs; superseded by ... + * tests/test-driver-end-metadata.test: ... this new test. + * tests/tap-log.test: Improve syncing with ... + * tests/test-log.test: ... this new test. + * tests/parallel-tests.test: Remove some duplication w.r.t. this + last new test. Updated heading comments. + * tests/Makefile.am (TESTS): Update. + + 2011-07-27 Stefano Lattarini + + testsuite: fix a spurious failure with non-bash shells + * tests/tap-fancy2.test: Remove an unportable use of backslashes + with the `echo' builtin, which was causing some shells (among them + bash and Debian /bin/ksh) to print a `\\' string, while other + shells (among them zsh, dash, and Solaris /bin/sh and /bin/ksh) + were unexpectedly printing a single `\' character. Since we are + at it, add a sanity check to ensure that this issue does not + resurface. + + 2011-07-27 Stefano Lattarini + + testsuite: fix spurious failures with Solaris /bin/sh + * tests/tap-more.test: Use `echo > file', not `: > file', to + create empty files in the "for" loops; this is required since, + as documented in the autoconf manual, Solaris 10 /bin/sh + "optimizes" away the `:' command after the first iteration, + even if it is redirected. + * tests/test-driver-custom-multitest-recheck2.test: Likewise. + * tests/tap-numeric-description.test: Partial rewrite to avoid + using positional parameters from the 10th onward, which are + unportable to Solaris /bin/sh (using `${10}' causes the shell + to die with "bad substitution"). + + 2011-07-31 Stefano Lattarini + + simple tests: support developer-defined fd redirections + Motivated by coreutils bug#8846, and related discussions: + + + In those threads it has been shown how problematic it can be to do + portable file descriptor redirections in the testsuite when using + the Automake testsuite harness. This change should remedy to that + situation. + * lib/am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, + ?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Append redirection defined + in $(AM_TESTS_FD_REDIRECT) to the command-line invocations of the + test scripts. + * lib/am/check.am [!%?PARALLEL_TESTS%] $(check-TESTS): Likewise. + * NEWS: Update. + * doc/automake.texi (Script-based Testsuites): Document the new + feature. + * tests/check-fd-redirect.test: New test. + * tests/parallel-tests-fd-redirect.test: Likewise. + * tests/parallel-tests-am_tests_environment.test: Remove checks + about the use of redirections in AM_TESTS_ENVIRONMENT: they would + check deprecated (if not undefined) behaviour now. Strengthen a + couple of still valid checks, to keep the test more in sync with + the documentation. Improve debugging information. + * tests/Makefile.am (TESTS): Update. + + 2011-07-22 Stefano Lattarini + + docs: rework and extend documentation on testsuites support + * doc/automake.texi (Generalities about Testing): New section. + (Simple Tests using parallel-tests): Section removed, merged + into ... + (Simple Tests): ... this one, which on the other hand has been + subdivided into ... + (Script-based Testsuites, Serial Test Harness, Parallel Test + Harness): ... these new subsection. + (DejaGnu Tests): Minor adjustments. + Other related typofixes and rewordings throughout the manual; + in particular, avoid to use the term "test driver" for three + different concepts (and use instead "test harness" and "test + runner" where appropriate). + * tests/tap-doc.test: New test. + * tests/tap-no-disable-hard-error.test: Likewise. + * tests/Makefile.am (TESTS): Update. + + 2011-07-18 Stefano Lattarini + + tap: support colorization of testsuite progress output + * lib/tap-driver (%COLORS): New variable (definition extracted + from `lib/am/check.am:$(am__tty_colors)', with some obvious + adjustments. + (report): Adjust to colorize console output when required, + using ... + (decorate_result): ... this new function. + (colored): New function, used by the one above. + * tests/tap-summary.test: Also run the checks when `color-tests' + is in use. + * tests/Makefile.am (XFAIL_TESTS): Remove `tap-color.test'. + + 2011-07-18 Stefano Lattarini + + tap: some preparatory refactoring (2) + This is a follow-up simplification. + * lib/tap-driver (console_output): Renamed ... + (report): ... to this, and extended to appropriately register + the test results when needed. + (testsuite_error, handle_tap_comment, handle_tap_test, + handle_tap_plan): Adjusted accordingly. + + 2011-07-18 Stefano Lattarini + + tap: some preparatory refactoring (1) + This refactoring is aimed at simplifying the introduction of + colored console output for the TAP driver. + * lib/tap-driver (console_output): Now accept two arguments, the + first one indicating which kind of thing is to be displayed (for + now only a test result or a diagnostic comment), and the second + one (if present) the message associated to it. + (handle_tap_test, handle_tap_comment, handle_tap_plan, + testsuite_error): Adapt to the new `console_output' interface. + + 2011-07-18 Stefano Lattarini + + tap: add experimental TAP-aware driver + * doc/automake.texi (Using the TAP test protocol): New section. + (Overview of Custom Test Drivers Support): Minor updates. + * lib/tap-driver: New script, TAP-aware test driver for Automake; + implemented in perl and based on TAP::Parser. + * lib/Makefile.am (dist_script_DATA): Add it. + * tests/tap-autonumber.test: New test. + * tests/tap-bailout.test: Likewise. + * tests/tap-basic.test: Likewise. + * tests/tap-deps.test: Likewise. + * tests/tap-diagnostic.test: Likewise. + * tests/tap-empty.test: Likewise. + * tests/tap-empty-diagnostic.test: Likewise. + * tests/tap-escape-directive.test: Likewise. + * tests/tap-exit.test: Likewise. + * tests/tap-fancy.test: Likewise. + * tests/tap-fancy2.test: Likewise. + * tests/tap-global-result.test: Likewise. + * tests/tap-html.test: Likewise. + * tests/tap-log.test: Likewise. + * tests/tap-merge-stdout-stderr.test: Likewise. + * tests/tap-more.test: Likewise. + * tests/tap-more2.test: Likewise. + * tests/tap-no-merge-stdout-stderr.test: Likewise. + * tests/tap-no-spurious-summary.test: Likewise. + * tests/tap-no-spurious.test: Likewise. + * tests/tap-not-ok-skip.test: Likewise. + * tests/tap-numeric-description.test: Likewise. + * tests/tap-out-of-order.test: Likewise. + * tests/tap-passthrough.test: Likewise. + * tests/tap-plan.test: Likewise. + * tests/tap-plan-errors.test: Likewise. + * tests/tap-plan-corner.test: Likewise. + * tests/tap-realtime.test: Likewise. + * tests/tap-recheck-logs.test: Likewise. + * tests/tap-recheck.test: Likewise. + * tests/tap-skip-whole.test: Likewise. + * tests/tap-summary.test: Likewise. + * tests/tap-todo-skip.test: Likewise. + * tests/tap-todo-skip-together.test: Likewise. + * tests/tap-todo-skip-whitespace.test: Likewise. + * tests/tap-skipall-whitespace.test: Likewise. + * tests/tap-unplanned.test: Likewise. + * tests/tap-whitespace-normalization.test: Likewise. + * tests/tap-with-and-without-number.test: Likewise. + * tests/tap-xfail-tests.test: Likewise. + * tests/tap-bad-prog.test: New xfailing test. + * tests/tap-color.test: Likewise. + * tests/tap-plan-corner2.test: Likewise. + * tests/tap-message-0.test: Likewise. + * tests/tap-signal.test: Likewise. + * tests/Makefile.am (TESTS, XFAIL_TESTS): Update. + + 2011-07-07 Stefano Lattarini + + test defs: new auxiliary function 'count_test_results' + * tests/defs (count_test_results): New function. + * tests/check11.test: Use it. + * tests/test-driver-custom-multitest.test: Likewise. + * tests/test-driver-custom-multitest-recheck.test: Likewise. + * tests/test-driver-custom-multitest-recheck2.test: Likewise. + * tests/parallel-tests-log-override-recheck.test: Likewise. + * tests/parallel-tests-log-override-recheck.test: Likewise. + * tests/parallel-tests-no-spurious-summary.test: Likewise, and + slightly improve debugging output. + * tests/parallel-tests.test: Make use of `count_test_results'. + Also, make grepping of "make check" output slightly stricter + * tests/parallel-tests9.test: Likewise. + * tests/parallel-tests-log-override-2.test: Likewise, and throw + in a small optimization. + + 2011-07-07 Stefano Lattarini + + parallel-tests: simplify testsuite summary + Prefer a more deterministic, "tabular" format for the testsuite + summary, always listing the numbers of passed, failed, xfailed, + xpassed, skipped and errored tests, even when these numbers are + zero. This simplify the logic of testsuite summary creation, + makes it more easily machine-parseable, and will probably allow + for easier addition of new kinds of test results in the future. + * lib/am/check.am (am__tty_colors_dummy): New make variable, to + reduce code duplication. Extracted from previous versions of + $(am__tty_colors), and extended by defining two new variables + `$mgn' and `$brg'. + [%?COLOR%, %!?COLOR%] (am__tty_colors): Use that new variable. + (am__text_box): Delete, is not needed anymore. + ($(TEST_SUITE_LOG)): Rewrite associated rules to implement the + new testsuite summary format. + * NEWS: Update. + * tests/check10.test: Don't run with the parallel-tests harness + too, that makes no sense anymore. + * tests/color.test: Update and adjust. + * tests/color2.test: Likewise. + * tests/parallel-tests.test: Likewise. + * tests/parallel-tests3.test: Likewise. + * tests/parallel-tests6.test: Likewise. + * tests/parallel-tests9.test: Likewise. + * tests/parallel-tests-unreadable-log.test: Likewise. + * tests/parallel-tests-empty-testlogs.test: Likewise. + * tests/parallel-tests-log-override-recheck.test: Likewise. + * tests/parallel-tests-no-spurious-summary.test: Likewise. + * tests/test-driver-custom-multitest.test: Likewise. + * tests/test-driver-end-test-results.test: Likewise. + * tests/parallel-tests-no-color-in-log.test: New test. + * tests/testsuite-summary-color.test: Likewise. + * tests/testsuite-summary-count.test: Likewise. + * tests/testsuite-summary-count-many.test: Likewise. + * tests/testsuite-summary-reference-log.test: Likewise. + * tests/testsuite-summary-checks.sh: New auxiliary script, used + by the new tests above. + * tests/extract-testsuite-summary: Likewise. + * tests/trivial-test-driver: Optimize for speed when there are + lots of of tests. + * tests/Makefile.am (EXTRA_DIST): Distribute them. + (testsuite-summary-color.log, testsuite-summary-count.log): Depend + on them. + (testsuite-summary-count-many.log): Depend on the auxiliary scripts + 'trivial-test-driver' and 'extract-testsuite-summary'. + (TESTS): Update. + + 2011-07-07 Stefano Lattarini + + parallel-tests: new recognized test result 'ERROR' + * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result + `ERROR'. Use it when encountering unreadable test logs (previously + a simple `FAIL' was used in this situations). + * lib/test-driver: Set the global test result to `ERROR' when the + test exit status is 99. When doing colorized output, color `ERROR' + results in magenta. + * doc/automake.texi (Log files generation and test results + recording): Update by listing `ERROR' too among the list of valid + `:test-results:' arguments. + * NEWS: Update. + * tests/trivial-test-driver: Update. + * tests/parallel-tests.test: Likewise. + * tests/parallel-tests-harderror.test: Likewise. + * tests/parallel-tests-no-spurious-summary.test: Likewise. + * tests/test-driver-global-log.test: Likewise. + * tests/test-driver-recheck.test: Likewise. + * tests/test-driver-custom-multitest-recheck.test: Likewise. + * tests/test-driver-custom-multitest-recheck2.test: Likewise. + * tests/test-driver-custom-multitest.test: Likewise. + * tests/test-driver-custom-no-html.test: Likewise. + * tests/test-driver-end-test-results.test: Likewise. + * tests/color.test: Likewise. Also, make stricter, and also test + from VPATH. + * tests/color2.test: Likewise, and improve syncing with color.test. + * tests/parallel-tests-exit-statuses.test: New test. + * tests/parallel-tests-console-output.test: Likewise. + * tests/Makefile.am (TESTS): Update. + + 2011-07-07 Stefano Lattarini + + parallel-tests: make parsing of test results safer + The new code for parsing the testsuite-generated `.log' files, + as introduced in commit `v1.11-872-gc96b881', considers each + `:test-result:' field anywhere in a `.log' file as a declaration + of a test result, and accounts for it as such in the testsuite + summary. Unfortunately this could easily cause spurious test + failures being reported in the testsuite summary. This happened + in practice with the Automake's own testsuite; for example: + $ make check TESTS='check12-p.test'; echo exit: $? + ... + PASS: check12-p.test + ===================================== + 4 of 5 tests failed + See tests/test-suite.log + Please report to bug-automake@gnu.org + ===================================== + make[2]: *** [test-suite.log] Error 1 + make: *** [check-am] Error 2 + exit: 2 + This change introduces a new special `:test-result:' "END", that, + when seen, prevents the rest of the log file from being parsed. + For more information, refer to the thread: + + * lib/am/check.am ($(TEST_SUITE_LOG)): Stop the parsing of a log + file as soon as the special ":test-result:END" directive is seen. + Related changes and enhancements. + * lib/test-driver: Protect the rest of the log after the result + lined with a ":test-result:END" directive. + * doc/automake.texi (Log files generation and test results + recording): Update, and related improvements. + * tests/parallel-tests-no-spurious-summary.test: New test. + * tests/test-driver-end-test-results.test: Likewise. + * tests/Makefile.am (TESTS): Update. + + 2011-06-29 Stefano Lattarini + + docs: document custom test drivers and protocols + * doc/automake.texi (Simple Tests): Note that the TESTS_ENVIRONMENT + use suggested here is not portable to 'parallel-tests'. + (Simple Tests using parallel-tests): Document new restrictions on + the uses of TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT. + (Custom Test Drivers): New section and node. + (Overview of Custom Test Drivers Support): New subsection. + (Declaring Custom Test Drivers in @file{Makefile.am}): Likewise. + (APIs for Custom Test Drivers): Likewise. + (Options): Update description of color-tests. + * lib/am/check ($(TEST_SUITE_LOG)): Remove comments that have been + moved into the manual. + (recheck, recheck-html): Minor adjustments to better conform to the + documentation (this should cause no semantic changes w.r.t. the + former behaviour); minor improvements and extensions to existing + comments. + * tests/test-driver-create-log-dir.test: New test. + * tests/test-driver-strip-vpath.test: Likewise. + * tests/test-driver-global-log.test: Likewise. + * tests/test-driver-recheck.test: Likewise. + * tests/Makefile.am (TESTS): Update. + +2011-08-03 Bruno Haible + + docs: how to use '-I' option in AM_CPPFLAGS for best VPATH support + * doc/automake.texi (Program Variables): Recommend -I options to + both the build directory and the source directory when needed. + +2011-07-23 Stefano Lattarini + + tests: relax a test on amhello examples to cater to Solaris tar + * tests/amhello-binpkg.test: When the tar implementation in use + is not GNU tar, relax the tests on tar output, to avoid spurious + failures. For example, "tar cvf ..." with GNU tar can output + lines like "./usr/bin/hello" on the standard output, while with + Solaris tar it can output lines like "a ./usr/bin/hello 8K" on + standard output, and with Heirloom tar it can output lines like + "a ./usr/bin/hello 15 tape blocks" on standard error. + +2011-07-23 Stefano Lattarini + + test defs: function 'is_newest' now works also with directories + * tests/defs (is_newest): Call `find' with the `-prune' option, + so that it won't descend in the directories (which could cause + spurious results). + * tests/self-check-is_newest: Extend accordingly. + From a report by Jim Meyering, see automake bug#9147. + +2011-07-15 Benoit Sigoure + + docs: add references between the 2 sections on java support + * doc/automake.texi (Java Support, Java): Add cross-references. + +2011-07-20 Ralf Wildenhues + + Sync auxiliary files from upstream. + * INSTALL, lib/INSTALL, lib/config.guess, lib/config.sub, + lib/texinfo.tex: Sync from upstream. + +2011-07-24 Stefano Lattarini + + tests: small enhancement to 'primary3.test' + * tests/primary3.test: Also test that the program named `foo.la' + has been effectively created. + +2011-07-24 Stefano Lattarini + + tests: a small simplification in 'pr8365-remake-timing.test' + * tests/pr8365-remake-timing.test: Avoid redundant definition + and use of `$save_AUTOCONF' variable. + +2011-07-24 Stefano Lattarini + + tests: remove an extra leading blank line + * tests/override-conditional-1.test: Remove extra blank line + placed before the shebang line. + +2011-07-18 Stefano Lattarini + + tests: fix typos, grammaros and other blunders in comments + All affected files changed. + +2011-07-16 Stefano Lattarini + + test defs: new subroutine 'seq_', simulating GNU seq(1) + * tests/defs (seq_): New subroutine. + * tests/instmany.test: Use it. + * tests/instmany-mans.test: Likewise. + * tests/instmany-python.test: Likewise. + * tests/self-check-seq.test: New self test. + * tests/Makefile.am (TESTS): Update. + +2011-07-16 Stefano Lattarini + + tests: remove duplication about testing of config.* aux files + * tests/add-missing.test: Also check that the `AC_CANONICAL_SYSTEM' + autoconf macro causes the `config.sub' and `config.guess' scripts + to be installed by `automake --add-missing'. Since we are at it, + fix minor buglets and cosmetic issues in the `check_' function. + * tests/hosts.test: Removed, completely subsumed by the previous + test now. + * tests/Makefile.am (TESTS): Update. + +2011-07-16 Stefano Lattarini + + tests: more uses of the 'unindent' subroutine + * tests/backcompat.test: Prefer `unindent' over plain `cat' for + here documents created from within a loop of if/else body. + * tests/backcompat2.test: Likewise. + * tests/backcompat4.test: Likewise. + * tests/init.test: Likewise. + * tests/instmany.test: Likewise. + * tests/instmany-mans.test: Likewise. + * tests/instmany-python.test: Likewise. + * tests/missing5.test: Likewise. + * tests/parallel-am2.test : Likewise. + * tests/parallel-am3.test : Likewise. + * tests/pr307.test: Likewise. + * tests/depend4.test: Likewise. Since we are at it, normalize + other code formatting. + * tests/instfail-java.test: Prefer `echo' over `cat' + here-doc + inside a for loop. + +2011-07-16 Stefano Lattarini + + tests: avoid '##'-style comments inside recipe commands + * tests/autodist.test: Do not whitespace-indent `##' comments when + they are embedded in a makefile rule: having them indented is not + part of the Automake API, and might cause failures with e.g., Tru64 + make. + * tests/autodist-subdir.test: Likewise. + * tests/backcompat.test: Likewise. + * tests/backcompat6.test: Likewise. + * tests/colon7.test: Likewise. + * tests/posixsubst-scripts.test: Likewise. + * tests/posixsubst-sources.test: Likewise. + +2011-07-16 Stefano Lattarini + + tests: tweak, improve and extend tests on lisp support + * tests/lisp2.test: Prettify, and improve debugging output. Make + grepping of automake stderr stricter. + * tests/lisp7.test: Ensure verbose printing of captured make + output. Add trailing `:' command. + * tests/lisp8.test: Likewise. + * tests/lisp3.test: Likewise. Also, check that `.el' files and + compiled `.elc' files get installed by "make install", and + uninstalled by "make uninstall". + * tests/lisp4.test: Ensure installed `.el' files gets removed by + "make uninstall". Related changes. Add trailing `:' command. + * tests/lisp5.test: Likewise. + * tests/lisp6.test: Use proper m4 quoting in configure.in. Fix + use of blank lines, to improve clarity and symmetry. Fix typo in + comment. Add trailing `:' command. + +2011-07-16 Stefano Lattarini + + tests: drop useless requirement in 'license.test' + * tests/license.test: Drop useless requirement "makeinfo". Add + an extra blank line, for clarity. + +2011-07-08 Stefano Lattarini + + tests: fix weakness in 'tests-environment-backcompat.test' + * tests/tests-environment-backcompat.test: Do not override the + content of xfailing test `baz.test' with a "weaker" version that + fails unconditionally: the test must fail only when the 'strict' + pragma is in use, in order not to reduce coverage. + +2011-07-08 Stefano Lattarini + + docs, tests: synchronize examples on silent-rules from config.site + * doc/automake.texi (Automake silent-rules Option): Reference test + 'silent-configsite.test' in comments. + +2011-07-08 Stefano Lattarini + + tests defs: more uses of $top_testsrcdir + * tests/autodist-stamp-vti.test: Use `$top_testsrcdir' instead + of `$testsrcdir/..'. + * tests/repeated-options.test: Likewise. + * tests/suffix5.test: Likewise. + * tests/vtexi3.test: Likewise. + +2011-07-08 Stefano Lattarini + + self tests: fix typo in comment + * tests/self-check-exit.test: Fix typo in heading comments. + +2011-07-08 Stefano Lattarini + + tests: remove redundant settings of `errexit' shell flag + * tests/amhello-binpkg.test: Do not set the `errexit' shell + flag, as it is already set by `tests/defs'. + * tests/amhello-cflags.test: Likewise. + * tests/amhello-cross-compile.test: Likewise. + * tests/ansi2knr-deprecation.test: Likewise. + * tests/autodist-no-duplicate.test: Likewise. + * tests/distcheck-configure-flags-am.test: Likewise. + * tests/distcheck-configure-flags-subpkg.test: Likewise. + * tests/distcheck-configure-flags.test: Likewise. + * tests/distcheck-hook.test: Likewise. + * tests/distcheck-hook2.test: Likewise. + * tests/parallel-tests-am_tests_environment.test: Likewise. + * tests/parallel-tests-harderror.test: Likewise. + * tests/parallel-tests-log-compiler-example.test: Likewise. + * tests/parallel-tests-log-override-1.test: Likewise. + * tests/parallel-tests-log-override-2.test: Likewise. + * tests/parallel-tests-log-override-recheck.test: Likewise. + * tests/primary-prefix-couples-documented-valid.test: Likewise. + * tests/primary-prefix-couples-force-valid.test: Likewise. + * tests/primary-prefix-invalid-couples.test: Likewise. + * tests/primary-prefix-valid-couples.test: Likewise. + * tests/remake-subdir-from-subdir.test: Likewise. + * tests/remake-subdir-gnu.test: Likewise. + * tests/remake-subdir-long-time.test: Likewise. + * tests/remake-subdir.test: Likewise. + * tests/remake-subdir2.test: Likewise. + * tests/silent-configsite.test: Likewise. + * tests/tests-environment-backcompat.test: Likewise. + +2011-07-08 Stefano Lattarini + + tests: portability fixes in tests on amhello examples + * tests/amhello-binpkg.test: Don't use "tar xzf too.tag.gz" to + extract a gzip-compressed tarball, that's unportable to some + tar implementations; use the "gzip -dc fo.tar.gz | tar xf -" + idiom instead. + * tests/amhello-cflags.test: Likewise. + * tests/amhello-cross-compile.test: Likewise. + Suggestion from Ralf Wildenhues. + +2011-07-04 Stefano Lattarini + + remake: add test ensuring that slower remakes don't hang + * tests/remake-subdir-long-time.test: New test. + * tests/Makefile.am (TESTS): Update. + Suggestion by Ralf Wildenhues. + +2011-07-04 Stefano Lattarini + + remake: fix outdated comment in configure.am + * lib/am/configure.am: Fix comment falsified by changes in + commit `v1.11-366-gbee9871'. + Suggestion by Ralf Wildenhues. + +2011-07-01 Stefano Lattarini + + docs, tests: synchronize examples from docs to tests + * tests/README (Writing test cases): Give suggestions on how to + keep test cases and examples in the documentation synchronized. + * doc/automake.texi: Improve or fix existing testcase-referencing + comments, and add many new ones. + * HACKING (Administrivia): Suggest to test complex examples and + idioms from the manual. + * tests/specflg8.test: Improve synchronization with the example + in the manual. + * tests/output11.test:Likewise. + * tests/txinfo21.test:Likewise. + * tests/interp.test: Likewise. + * tests/amhello-cflags.test: New test. + * tests/amhello-cross-compile.test: Likewise. + * tests/amhello-binpkg.test: Likewise. + * tests/tests-environment-backcompat.test: Likewise. + * tests/parallel-tests-log-compiler-example.test: Likewise. + * tests/Makefile.am (TESTS): Update. + +2011-06-30 Stefano Lattarini + + coverage: new test on parallel-tests TESTS runtime overriding + * tests/parallel-tests-cmdline-override.test: New test, check that + we can use indirections when overriding TESTS and TEST_LOGS from + the command line. + * tests/Makefile.am (TESTS): Update. + 2011-06-29 Stefano Lattarini docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated diff --cc INSTALL index 81fd332ce,81fd332ce..a1e89e18a --- a/INSTALL +++ b/INSTALL @@@ -1,8 -1,8 +1,8 @@@ Installation Instructions ************************* --Copyright (C) 1994, 1995, 1996, 1999, 2000, 2001, 2002, 2004, 2005, --2006, 2007, 2008, 2009, 2010 Free Software Foundation, Inc. ++Copyright (C) 1994-1996, 1999-2002, 2004-2011 Free Software Foundation, ++Inc. Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright @@@ -226,6 -226,6 +226,11 @@@ order to use an ANSI C compiler and if that doesn't work, install pre-built binaries of GCC for HP-UX. ++ HP-UX `make' updates targets which have the same time stamps as ++their prerequisites, which makes it generally unusable when shipped ++generated files such as `configure' are involved. Use GNU `make' ++instead. ++ On OSF/1 a.k.a. Tru64, some versions of the default C compiler cannot parse its `' header file. The option `-nodtk' can be used as a workaround. If GNU CC is not installed, it is therefore recommended diff --cc doc/automake.texi index c982b401a,65a692d05..5eb40d5ab --- a/doc/automake.texi +++ b/doc/automake.texi @@@ -8708,41 -8868,59 +8906,60 @@@ The Autoconf Manual}), and in particula @vindex TESTS @vindex TESTS_ENVIRONMENT - The variable @code{TESTS_ENVIRONMENT} can be used to set environment - variables for the test run; the environment variable @env{srcdir} is - set in the rule. If all your test programs are scripts, you can also - set @code{TESTS_ENVIRONMENT} to an invocation of the shell (e.g. - @samp{$(SHELL) -x} can be useful for debugging the tests), or any other - interpreter. For instance, the following setup may be used to run tests - with Perl: + @vindex AM_TESTS_ENVIRONMENT + The @code{AM_TESTS_ENVIRONMENT} and @code{TESTS_ENVIRONMENT} variables can + be used to run initialization code and set environment variables for the + test scripts. The former variable is developer-reserved, and can be + defined in the @file{Makefile.am}, while the latter is reserved for the + user, which can employ it to extend or override the settings in the + former; for this to work portably, however, the contents of a non-empty + @code{AM_TESTS_ENVIRONMENT} @emph{must} be terminated by a semicolon. + + @vindex AM_TESTS_FD_REDIRECT + The @code{AM_TESTS_FD_REDIRECT} variable can be used to define file + descriptor redirections for the test scripts. One might think that + @code{AM_TESTS_ENVIRONMENT} could be used for this purpose, but experience + has shown that doing so portably is practically impossible. The main + hurdle is constituted by Korn shells, which usually set the close-on-exec + flag on file descriptors opened with the @command{exec} builtin, thus + rendering an idiom like @code{AM_TESTS_ENVIRONMENT = exec 9>&2;} + ineffectual. This issue also affects some Bourne shells, such as the + HP-UX's @command{/bin/sh}, + @c FIXME: should we offer a link to the relevant discussions on the + @c bug-autoconf list? +@c Keep in sync with tests-environment-backcompat.test. @example - TESTS_ENVIRONMENT = $(PERL) -Mstrict -w - TESTS = foo.pl bar.pl baz.pl + AM_TESTS_ENVIRONMENT = \ + ## Some environment initializations are kept in a separate shell file + ## `tests-env.sh', which can make it easier to also run tests from + ## the command line. + . $(srcdir)/tests-env.sh; \ + ## On Solaris, prefer more POSIX-compliant versions of the standard + ## tools by default. + if test -d /usr/xpg4/bin; then \ + PATH=/usr/xpg4/bin:$$PATH; export PATH; \ + fi; + @c $$ restore font-lock + ## With this, the test scripts will be able to print diagnostic messages + ## to the original standard error stream, even if the test driver + ## redirects the stderr of the test scripts to a log file before executing + ## them. + AM_TESTS_FD_REDIRECT = 9>&2 @end example - Note that the @option{parallel-tests} driver provides a more elegant - way to achieve the same effect, freeing the @code{TESTS_ENVIRONMENT} - variable for the user to override (@pxref{Simple Tests using - parallel-tests}). - - - @cindex Tests, expected failure - @cindex Expected test failure - - @vindex XFAIL_TESTS - You may define the variable @code{XFAIL_TESTS} to a list of tests - (usually a subset of @code{TESTS}) that are expected to fail. This will - reverse the result of those tests. + @noindent + Note however that @code{AM_TESTS_ENVIRONMENT} is, for historical and + implementation reasons, @emph{not} supported by the serial harness + (@pxref{Serial Test Harness}). Automake ensures that each file listed in @code{TESTS} is built before - any tests are run; you can list both source and derived programs (or - scripts) in @code{TESTS}; the generated rule will look both in - @code{srcdir} and @file{.}. For instance, you might want to run a C - program as a test. To do this you would list its name in @code{TESTS} - and also in @code{check_PROGRAMS}, and then specify it as you would - any other program. + it is run; you can list both source and derived programs (or scripts) + in @code{TESTS}; the generated rule will look both in @code{srcdir} and + @file{.}. For instance, you might want to run a C program as a test. + To do this you would list its name in @code{TESTS} and also in + @code{check_PROGRAMS}, and then specify it as you would any other + program. Programs listed in @code{check_PROGRAMS} (and @code{check_LIBRARIES}, @code{check_LTLIBRARIES}...) are only built during @code{make check}, @@@ -8816,15 -9040,14 +9079,15 @@@ if a test file name ends in several con @vindex AM_@var{ext}_LOG_FLAGS @vindex AM_LOG_FLAGS For tests that match an extension @code{.@var{ext}} listed in - @code{TEST_EXTENSIONS}, you can provide a test driver using the variable - @code{@var{ext}_LOG_COMPILER} (note the upper-case extension) and pass - options in @code{AM_@var{ext}_LOG_FLAGS} and allow the user to pass - options in @code{@var{ext}_LOG_FLAGS}. It will cause all tests with - this extension to be called with this driver. For all tests without a - registered extension, the variables @code{LOG_COMPILER}, + @code{TEST_EXTENSIONS}, you can provide a custom ``test runner'' using + the variable @code{@var{ext}_LOG_COMPILER} (note the upper-case + extension) and pass options in @code{AM_@var{ext}_LOG_FLAGS} and allow + the user to pass options in @code{@var{ext}_LOG_FLAGS}. It will cause + all tests with this extension to be called with this runner. For all + tests without a registered extension, the variables @code{LOG_COMPILER}, @code{AM_LOG_FLAGS}, and @code{LOG_FLAGS} may be used. For example, +@c Keep in sync with parallel-tests-log-compiler-example.test. @example TESTS = foo.pl bar.py baz TEST_EXTENSIONS = .pl .py diff --cc lib/Automake/tests/Makefile.in index 18308b0d0,78a7e4aae..f375e3c18 --- a/lib/Automake/tests/Makefile.in +++ b/lib/Automake/tests/Makefile.in @@@ -325,8 -295,8 +305,8 @@@ am__tdf = diff-in-tests-lists.tm all: all-am .SUFFIXES: - .SUFFIXES: .html .log .pl + .SUFFIXES: .html .log .pl .trs -$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(am__configure_deps) +$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(top_srcdir)/CheckListOfTests.am $(am__configure_deps) @for dep in $?; do \ case '$(am__configure_deps)' in \ *$$dep*) \ diff --cc tests/.gitignore index a6d0cce61,0d5a032b5..430b25ff8 --- a/tests/.gitignore +++ b/tests/.gitignore @@@ -3,10 -3,9 +3,11 @@@ automake- defs-static instspc-tests.am parallel-tests.am +config-shell-tests.am +depmod-tests.am *.dir *.log + *.trs *.log-t *-p.test instspc-*-build.test diff --cc tests/Makefile.am index 925c2ca81,43ada56a1..2ba3abcff --- a/tests/Makefile.am +++ b/tests/Makefile.am @@@ -19,10 -23,6 +23,9 @@@ TAP_LOG_DRIVER_FLAGS = --merg MAINTAINERCLEANFILES = EXTRA_DIST = ChangeLog-old - TEST_EXTENSIONS = .test +# Run the tests with the shell detected at configure time. +TEST_LOG_COMPILER = $(SHELL) + XFAIL_TESTS = \ all.test \ auxdir2.test \ @@@ -128,23 -110,12 +134,24 @@@ $(config_shell_tests) AM_TESTS_ENVIRONMENT = \ test x"$$me" = x || unset me; \ test x"$$required" = x || unset required; \ + test x"$$use_tap" = x || unset use_tap; \ test x"$$parallel_tests" = x || unset parallel_tests; \ + test x"$$test_prefer_config_shell" || unset test_prefer_config_shell; \ test x"$$original_AUTOMAKE" = x || unset original_AUTOMAKE; \ - test x"$$original_ACLOCAL" = x || unset original_ACLOCAL; + test x"$$original_ACLOCAL" = x || unset original_ACLOCAL; \ + AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC; \ + stderr_fileno_=9; export stderr_fileno_; 9>&2 TESTS = \ + $(handwritten_tests) \ + $(config_shell_tests) \ + $(parallel_tests) \ + $(depmod_tests) \ + $(instspc_tests) + +EXTRA_DIST += $(handwritten_tests) + +handwritten_tests = \ aclocal.test \ aclocal3.test \ aclocal4.test \ @@@ -273,12 -235,11 +275,12 @@@ check8.test check10.test \ check11.test \ check12.test \ +check-subst.test \ +check-subst-prog.test \ check-exported-srcdir.test \ + check-fd-redirect.test \ check-tests-in-builddir.test \ - check-tests_environment.test \ - tests-environment-backcompat.test \ + check-no-test-driver.test \ checkall.test \ clean.test \ clean2.test \ @@@ -760,12 -721,11 +762,14 @@@ parallel-tests6.test parallel-tests8.test \ parallel-tests9.test \ parallel-tests10.test \ +parallel-tests-suffix.test \ +parallel-tests-suffix-prog.test \ +parallel-tests-ext-driver.test \ +parallel-tests-ext-driver-prog.test \ - parallel-tests-am_tests_environment.test \ - parallel-tests-unreadable-log.test \ + parallel-tests-dry-run.test \ + parallel-tests-fd-redirect.test \ + parallel-tests-extra-programs.test \ + parallel-tests-unreadable.test \ parallel-tests-subdir.test \ parallel-tests-interrupt.test \ parallel-tests-reset-term.test \ @@@ -773,8 -733,38 +777,41 @@@ parallel-tests-harderror.test parallel-tests-log-override-1.test \ parallel-tests-log-override-2.test \ parallel-tests-log-override-recheck.test \ - parallel-tests-cmdline-override.test \ +parallel-tests-log-compiler-example.test \ ++parallel-tests-cmdline-override.test \ + parallel-tests-empty-testlogs.test \ + parallel-tests-driver-install.test \ + parallel-tests-no-spurious-summary.test \ + parallel-tests-exit-statuses.test \ + parallel-tests-console-output.test \ + parallel-tests-once.test \ + tests-environment.test \ + am-tests-environment.test \ ++tests-environment-backcompat.test \ + testsuite-summary-color.test \ + testsuite-summary-count.test \ + testsuite-summary-count-many.test \ + testsuite-summary-reference-log.test \ + test-driver-custom-no-extra-driver.test \ + test-driver-custom.test \ + test-driver-custom-xfail-tests.test \ + test-driver-custom-multitest.test \ + test-driver-custom-multitest-recheck.test \ + test-driver-custom-multitest-recheck2.test \ + test-driver-custom-html.test \ + test-driver-custom-no-html.test \ + test-driver-create-log-dir.test \ + test-driver-strip-vpath.test \ + test-driver-fail.test \ + test-log.test \ + test-metadata-global-log.test \ + test-metadata-recheck.test \ + test-metadata-results.test \ + test-missing.test \ + test-missing2.test \ + test-trs-basic.test \ + test-trs-recover.test \ + test-trs-recover2.test \ parse.test \ percent.test \ percent2.test \ @@@ -897,13 -883,11 +934,14 @@@ self-check-cleanup.test self-check-dir.test \ self-check-env-sanitize.test \ self-check-exit.test \ +self-check-explicit-skips.test \ self-check-is_newest.test \ self-check-me.test \ +self-check-reexec.test \ self-check-report.test \ self-check-sanity.test \ +self-check-seq.test \ + self-check-tap.test \ self-check-unindent.test \ sanity.test \ scripts.test \ @@@ -1120,16 -1105,104 +1160,106 @@@ yflags-conditional.test yflags-d-false-positives.test \ yflags-force-override.test \ yflags-force-conditional.test \ -yflags-var-expand.test \ -$(parallel_tests) - -EXTRA_DIST += $(TESTS) +yflags-var-expand.test + ## Files containing auxiliary functions used by our test cases. + EXTRA_DIST += tap-functions.sh plain-functions.sh + + test-driver-custom-multitest.log: trivial-test-driver + test-driver-custom-multitest-recheck.log: trivial-test-driver + test-driver-custom-multitest-recheck2.log: trivial-test-driver + test-driver-custom-html.log: trivial-test-driver + EXTRA_DIST += trivial-test-driver + + testsuite-summary-color.log testsuite-summary-count.log: \ + testsuite-summary-checks.sh extract-testsuite-summary + EXTRA_DIST += testsuite-summary-checks.sh + EXTRA_DIST += extract-testsuite-summary + + testsuite-summary-count-many.log: trivial-test-driver + testsuite-summary-count-many.log: extract-testsuite-summary + + # List of tests on TAP support that use the files pre-computed by + # `tap-common-setup.test', and sources the `tap-setup.sh' helper + # script. + tap_with_common_setup_tests = \ + tap-autonumber.test \ + tap-bailout.test \ + tap-color.test \ + tap-deps.test \ + tap-diagnostic.test \ + tap-empty-diagnostic.test \ + tap-empty.test \ + tap-escape-directive.test \ + tap-exit.test \ + tap-signal.test \ + tap-fancy.test \ + tap-fancy2.test \ + tap-global-log.test \ + tap-global-result.test \ + tap-html.test \ + tap-log.test \ + tap-merge-stdout-stderr.test \ + tap-no-merge-stdout-stderr.test \ + tap-message-0.test \ + tap-no-disable-hard-error.test \ + tap-no-spurious-summary.test \ + tap-no-spurious.test \ + tap-not-ok-skip.test \ + tap-numeric-description.test \ + tap-out-of-order.test \ + tap-passthrough.test \ + tap-passthrough-exit.test \ + tap-plan.test \ + tap-plan-corner.test \ + tap-plan-corner2.test \ + tap-plan-errors.test \ + tap-realtime.test \ + tap-recheck-logs.test \ + tap-skip-whole-whitespace.test \ + tap-skip-whole.test \ + tap-todo-skip-together.test \ + tap-todo-skip-whitespace.test \ + tap-todo-skip.test \ + tap-unplanned.test \ + tap-whitespace-normalization.test \ + tap-with-and-without-number.test \ + tap-xfail-tests.test + + # Their log files. + tap_with_common_setup_logs = $(tap_with_common_setup_tests:.test=.log) + + # Their dependencies. + $(tap_with_common_setup_logs): tap-common-setup.log tap-setup.sh + EXTRA_DIST += tap-setup.sh + + # Other tests on TAP support. + tap_other_tests = \ + tap-common-setup.test \ + tap-bad-prog.tap \ + tap-basic.test \ + tap-doc.test \ + tap-doc2.test \ + tap-empty.test \ + tap-more.test \ + tap-more2.test \ + tap-recheck.test \ + tap-summary.test \ + tap-summary-color.test + + tap-summary.log tap-summary-color.log: tap-summary-aux.sh + EXTRA_DIST += tap-summary-aux.sh - ## Checking the test of tests. - checked_test_extensions = .test ++## Checking the list of tests. ++checked_test_extensions = .test .tap +expected_list_of_tests = $(handwritten_tests) +include $(top_srcdir)/CheckListOfTests.am + # Dependencies valid for each test case. $(TEST_LOGS): defs defs-static aclocal-$(APIVERSION) automake-$(APIVERSION) + # FIXME: this should be made more granular once we have a cleaner + # subdivision of the tests. + $(TEST_LOGS): plain-functions.sh tap-functions.sh clean-local: clean-local-check .PHONY: clean-local-check diff --cc tests/Makefile.in index f5aea3b26,9052c93cb..afe3b4c00 --- a/tests/Makefile.in +++ b/tests/Makefile.in @@@ -197,20 -173,12 +176,28 @@@ TEST_SUITE_LOG = test-suite.lo TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html) am__test_logs1 = $(TESTS:=.log) am__test_logs2 = $(am__test_logs1:.test.log=.log) + TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver TEST_LOG_COMPILE = $(TEST_LOG_COMPILER) $(AM_TEST_LOG_FLAGS) \ $(TEST_LOG_FLAGS) - am__test_logs3 = $(am__test_logs2:.ptest.log=.log) -TEST_LOGS = $(am__test_logs2:.tap.log=.log) ++am__test_logs3 = $(am__test_logs2:.tap.log=.log) + TAP_LOG_COMPILE = $(TAP_LOG_COMPILER) $(AM_TAP_LOG_FLAGS) \ + $(TAP_LOG_FLAGS) ++am__test_logs4 = $(am__test_logs3:.ptest.log=.log) ++PTEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver +PTEST_LOG_COMPILE = $(PTEST_LOG_COMPILER) $(AM_PTEST_LOG_FLAGS) \ + $(PTEST_LOG_FLAGS) - am__test_logs4 = $(am__test_logs3:.instspc.log=.log) ++am__test_logs5 = $(am__test_logs4:.instspc.log=.log) ++INSTSPC_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver +INSTSPC_LOG_COMPILE = $(INSTSPC_LOG_COMPILER) $(AM_INSTSPC_LOG_FLAGS) \ + $(INSTSPC_LOG_FLAGS) - am__test_logs5 = $(am__test_logs4:.depmod.log=.log) ++am__test_logs6 = $(am__test_logs5:.depmod.log=.log) ++DEPMOD_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver +DEPMOD_LOG_COMPILE = $(DEPMOD_LOG_COMPILER) $(AM_DEPMOD_LOG_FLAGS) \ + $(DEPMOD_LOG_FLAGS) - TEST_LOGS = $(am__test_logs5:.shtst.log=.log) ++TEST_LOGS = $(am__test_logs6:.shtst.log=.log) ++SHTST_LOG_DRIVER = $(SHELL) $(top_srcdir)/lib/test-driver +SHTST_LOG_COMPILE = $(SHTST_LOG_COMPILER) $(AM_SHTST_LOG_FLAGS) \ + $(SHTST_LOG_FLAGS) TEST_LOGS_TMP = $(TEST_LOGS:.log=.log-t) DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) ACLOCAL = @ACLOCAL@ @@@ -311,94 -275,89 +298,101 @@@ target_alias = @target_alias top_build_prefix = @top_build_prefix@ top_builddir = @top_builddir@ top_srcdir = @top_srcdir@ -TEST_EXTENSIONS = .test .tap ++TEST_EXTENSIONS = .test .tap .ptest .instspc .depmod .shtst + TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver + TAP_LOG_DRIVER_FLAGS = --merge -MAINTAINERCLEANFILES = $(parallel_tests) $(instspc_tests) -EXTRA_DIST = ChangeLog-old gen-parallel-tests instspc-tests.sh \ - $(TESTS) tap-functions.sh plain-functions.sh \ - trivial-test-driver testsuite-summary-checks.sh \ - extract-testsuite-summary tap-setup.sh tap-summary-aux.sh +MAINTAINERCLEANFILES = +EXTRA_DIST = ChangeLog-old gen-parallel-tests parallel-tests.sh \ + instspc-tests.sh depmod-tests.sh gen-config-shell-tests \ - config-shell-tests.sh $(handwritten_tests) - TEST_EXTENSIONS = .test .ptest .instspc .depmod .shtst ++ config-shell-tests.sh $(handwritten_tests) tap-functions.sh \ ++ plain-functions.sh trivial-test-driver \ ++ testsuite-summary-checks.sh extract-testsuite-summary \ ++ tap-setup.sh tap-summary-aux.sh ++ +# Run the tests with the shell detected at configure time. +TEST_LOG_COMPILER = $(SHELL) XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \ override-conditional-2.test pr8365-remake-timing.test \ - yacc-dist-nobuild-subdir.test txinfo5.test \ + yacc-dist-nobuild-subdir.test tap-plan-corner2.test \ + tap-message-0.test tap-signal.test txinfo5.test \ $(instspc_xfail_tests) -parallel_tests = backcompat5-p.test check-exported-srcdir-p.test \ - check-fd-redirect-p.test check-tests-in-builddir-p.test \ - check-p.test check11-p.test check12-p.test check2-p.test \ - check3-p.test check4-p.test check5-p.test check6-p.test \ - check7-p.test check8-p.test check9-p.test color-p.test \ - color2-p.test comment9-p.test dejagnu-p.test exeext4-p.test \ - maken3-p.test maken4-p.test posixsubst-tests-p.test \ - tests-environment-p.test -instspc_tests = instspc-squote-build.test instspc-squote-install.test \ - instspc-dquote-build.test instspc-dquote-install.test \ - instspc-bquote-build.test instspc-bquote-install.test \ - instspc-sharp-build.test instspc-sharp-install.test \ - instspc-dollar-build.test instspc-dollar-install.test \ - instspc-bang-build.test instspc-bang-install.test \ - instspc-bslash-build.test instspc-bslash-install.test \ - instspc-ampersand-build.test instspc-ampersand-install.test \ - instspc-percent-build.test instspc-percent-install.test \ - instspc-leftpar-build.test instspc-leftpar-install.test \ - instspc-rightpar-build.test instspc-rightpar-install.test \ - instspc-pipe-build.test instspc-pipe-install.test \ - instspc-caret-build.test instspc-caret-install.test \ - instspc-tilde-build.test instspc-tilde-install.test \ - instspc-qmark-build.test instspc-qmark-install.test \ - instspc-star-build.test instspc-star-install.test \ - instspc-plus-build.test instspc-plus-install.test \ - instspc-minus-build.test instspc-minus-install.test \ - instspc-comma-build.test instspc-comma-install.test \ - instspc-colon-build.test instspc-colon-install.test \ - instspc-semicol-build.test instspc-semicol-install.test \ - instspc-equal-build.test instspc-equal-install.test \ - instspc-less-build.test instspc-less-install.test \ - instspc-more-build.test instspc-more-install.test \ - instspc-at-build.test instspc-at-install.test \ - instspc-lqbrack-build.test instspc-lqbrack-install.test \ - instspc-rqbrack-build.test instspc-rqbrack-install.test \ - instspc-lcbrack-build.test instspc-lcbrack-install.test \ - instspc-rcbrack-build.test instspc-rcbrack-install.test \ - instspc-space-build.test instspc-space-install.test \ - instspc-tab-build.test instspc-tab-install.test \ - instspc-linefeed-build.test instspc-linefeed-install.test \ - instspc-backspace-build.test instspc-backspace-install.test \ - instspc-formfeed-build.test instspc-formfeed-install.test \ - instspc-carriageret-build.test \ - instspc-carriageret-install.test \ - instspc-quadrigraph0-build.test \ - instspc-quadrigraph0-install.test \ - instspc-quadrigraph1-build.test \ - instspc-quadrigraph1-install.test \ - instspc-quadrigraph2-build.test \ - instspc-quadrigraph2-install.test \ - instspc-quadrigraph3-build.test \ - instspc-quadrigraph3-install.test \ - instspc-quadrigraph4-build.test \ - instspc-quadrigraph4-install.test instspc-a_b-build.test \ - instspc-a_b-install.test instspc-a__b-build.test \ - instspc-a__b-install.test instspc-a_lf_b-build.test \ - instspc-a_lf_b-install.test instspc-dotdotdot-build.test \ - instspc-dotdotdot-install.test instspc-dosdrive-build.test \ - instspc-dosdrive-install.test instspc-miscglob1-build.test \ - instspc-miscglob1-install.test instspc-miscglob2-build.test \ - instspc-miscglob2-install.test -instspc_xfail_tests = instspc-squote-build.test \ - instspc-dquote-build.test instspc-bquote-build.test \ - instspc-sharp-build.test instspc-dollar-build.test \ - instspc-bslash-build.test instspc-ampersand-build.test \ - instspc-linefeed-build.test instspc-quadrigraph0-build.test \ - instspc-a_lf_b-build.test instspc-squote-install.test \ - instspc-dquote-install.test instspc-bquote-install.test \ - instspc-sharp-install.test instspc-dollar-install.test \ - instspc-linefeed-install.test instspc-a_lf_b-install.test +parallel_tests = backcompat5-p.ptest check-exported-srcdir-p.ptest \ - check-subst-prog-p.ptest check-subst-p.ptest \ - check-tests-in-builddir-p.ptest \ - check-tests_environment-p.ptest check-p.ptest check10-p.ptest \ - check11-p.ptest check12-p.ptest check2-p.ptest check3-p.ptest \ - check4-p.ptest check5-p.ptest check6-p.ptest check7-p.ptest \ - check8-p.ptest color-p.ptest color2-p.ptest comment9-p.ptest \ - dejagnu-p.ptest exeext4-p.ptest maken3-p.ptest maken4-p.ptest \ - posixsubst-tests-p.ptest ++ check-fd-redirect-p.ptest check-subst-prog-p.ptest \ ++ check-subst-p.ptest check-tests-in-builddir-p.ptest \ ++ check-p.ptest check11-p.ptest check12-p.ptest check2-p.ptest \ ++ check3-p.ptest check4-p.ptest check5-p.ptest check6-p.ptest \ ++ check7-p.ptest check8-p.ptest color-p.ptest color2-p.ptest \ ++ comment9-p.ptest dejagnu-p.ptest exeext4-p.ptest \ ++ maken3-p.ptest maken4-p.ptest posixsubst-tests-p.ptest \ ++ tests-environment-p.ptest +PTEST_LOG_COMPILER = $(SHELL) $(srcdir)/parallel-tests.sh +instspc_tests = build-squote.instspc install-squote.instspc \ + build-dquote.instspc install-dquote.instspc \ + build-bquote.instspc install-bquote.instspc \ + build-sharp.instspc install-sharp.instspc build-dollar.instspc \ + install-dollar.instspc build-bang.instspc install-bang.instspc \ + build-bslash.instspc install-bslash.instspc \ + build-ampersand.instspc install-ampersand.instspc \ + build-percent.instspc install-percent.instspc \ + build-leftpar.instspc install-leftpar.instspc \ + build-rightpar.instspc install-rightpar.instspc \ + build-pipe.instspc install-pipe.instspc build-caret.instspc \ + install-caret.instspc build-tilde.instspc \ + install-tilde.instspc build-qmark.instspc \ + install-qmark.instspc build-star.instspc install-star.instspc \ + build-plus.instspc install-plus.instspc build-minus.instspc \ + install-minus.instspc build-comma.instspc \ + install-comma.instspc build-colon.instspc \ + install-colon.instspc build-semicol.instspc \ + install-semicol.instspc build-equal.instspc \ + install-equal.instspc build-less.instspc install-less.instspc \ + build-more.instspc install-more.instspc build-at.instspc \ + install-at.instspc build-lqbrack.instspc \ + install-lqbrack.instspc build-rqbrack.instspc \ + install-rqbrack.instspc build-lcbrack.instspc \ + install-lcbrack.instspc build-rcbrack.instspc \ + install-rcbrack.instspc build-space.instspc \ + install-space.instspc build-tab.instspc install-tab.instspc \ + build-linefeed.instspc install-linefeed.instspc \ + build-backspace.instspc install-backspace.instspc \ + build-formfeed.instspc install-formfeed.instspc \ + build-carriageret.instspc install-carriageret.instspc \ + build-quadrigraph0.instspc install-quadrigraph0.instspc \ + build-quadrigraph1.instspc install-quadrigraph1.instspc \ + build-quadrigraph2.instspc install-quadrigraph2.instspc \ + build-quadrigraph3.instspc install-quadrigraph3.instspc \ + build-quadrigraph4.instspc install-quadrigraph4.instspc \ + build-a_b.instspc install-a_b.instspc build-a__b.instspc \ + install-a__b.instspc build-a_lf_b.instspc \ + install-a_lf_b.instspc build-dotdotdot.instspc \ + install-dotdotdot.instspc build-dosdrive.instspc \ + install-dosdrive.instspc build-miscglob1.instspc \ + install-miscglob1.instspc build-miscglob2.instspc \ + install-miscglob2.instspc +instspc_xfail_tests = build-squote.instspc build-dquote.instspc \ + build-bquote.instspc build-sharp.instspc build-dollar.instspc \ + build-bslash.instspc build-ampersand.instspc \ + build-linefeed.instspc build-quadrigraph0.instspc \ + build-a_lf_b.instspc install-squote.instspc \ + install-dquote.instspc install-bquote.instspc \ + install-sharp.instspc install-dollar.instspc \ + install-linefeed.instspc install-a_lf_b.instspc +INSTSPC_LOG_COMPILER = $(SHELL) $(srcdir)/instspc-tests.sh +depmod_tests = depcomp-gcc3.depmod depcomp-gcc.depmod \ + depcomp-hp.depmod depcomp-sgi.depmod depcomp-aix.depmod \ + depcomp-icc.depmod depcomp-hp2.depmod depcomp-tru64.depmod \ + depcomp-msvc7.depmod depcomp-msvc7msys.depmod \ + depcomp-nosideeffect.depmod depcomp-dashmstdout.depmod \ + depcomp-dashXmstdout.depmod depcomp-makedepend.depmod \ + depcomp-cpp.depmod depcomp-msvisualcpp.depmod \ + depcomp-msvcmsys.depmod +DEPMOD_LOG_COMPILER = $(SHELL) $(srcdir)/depmod-tests.sh +config_shell_tests = ar-lib-w.shtst compile-w.shtst compile2-w.shtst \ + compile3-w.shtst compile4-w.shtst compile5-w.shtst \ + compile6-w.shtst instsh2-w.shtst instsh3-w.shtst \ + mdate5-w.shtst mdate6-w.shtst missing-w.shtst missing2-w.shtst \ + missing3-w.shtst missing5-w.shtst mkinst3-w.shtst +SHTST_LOG_COMPILER = $(SHELL) $(srcdir)/config-shell-tests.sh # Some testsuite-influential variables should be overridable from the # test scripts, but not from the environment. @@@ -418,21 -364,12 +412,22 @@@ AM_TESTS_ENVIRONMENT = \ test x"$$me" = x || unset me; \ test x"$$required" = x || unset required; \ + test x"$$use_tap" = x || unset use_tap; \ test x"$$parallel_tests" = x || unset parallel_tests; \ + test x"$$test_prefer_config_shell" || unset test_prefer_config_shell; \ test x"$$original_AUTOMAKE" = x || unset original_AUTOMAKE; \ - test x"$$original_ACLOCAL" = x || unset original_ACLOCAL; + test x"$$original_ACLOCAL" = x || unset original_ACLOCAL; \ + AM_TESTS_REEXEC=no; export AM_TESTS_REEXEC; \ + stderr_fileno_=9; export stderr_fileno_; 9>&2 TESTS = \ + $(handwritten_tests) \ + $(config_shell_tests) \ + $(parallel_tests) \ + $(depmod_tests) \ + $(instspc_tests) + +handwritten_tests = \ aclocal.test \ aclocal3.test \ aclocal4.test \ @@@ -561,12 -489,11 +551,12 @@@ check8.test check10.test \ check11.test \ check12.test \ +check-subst.test \ +check-subst-prog.test \ check-exported-srcdir.test \ + check-fd-redirect.test \ check-tests-in-builddir.test \ - check-tests_environment.test \ - tests-environment-backcompat.test \ + check-no-test-driver.test \ checkall.test \ clean.test \ clean2.test \ @@@ -1048,12 -975,11 +1038,14 @@@ parallel-tests6.test parallel-tests8.test \ parallel-tests9.test \ parallel-tests10.test \ +parallel-tests-suffix.test \ +parallel-tests-suffix-prog.test \ +parallel-tests-ext-driver.test \ +parallel-tests-ext-driver-prog.test \ - parallel-tests-am_tests_environment.test \ - parallel-tests-unreadable-log.test \ + parallel-tests-dry-run.test \ + parallel-tests-fd-redirect.test \ + parallel-tests-extra-programs.test \ + parallel-tests-unreadable.test \ parallel-tests-subdir.test \ parallel-tests-interrupt.test \ parallel-tests-reset-term.test \ @@@ -1061,8 -987,38 +1053,41 @@@ parallel-tests-harderror.test parallel-tests-log-override-1.test \ parallel-tests-log-override-2.test \ parallel-tests-log-override-recheck.test \ - parallel-tests-cmdline-override.test \ +parallel-tests-log-compiler-example.test \ ++parallel-tests-cmdline-override.test \ + parallel-tests-empty-testlogs.test \ + parallel-tests-driver-install.test \ + parallel-tests-no-spurious-summary.test \ + parallel-tests-exit-statuses.test \ + parallel-tests-console-output.test \ + parallel-tests-once.test \ + tests-environment.test \ + am-tests-environment.test \ ++tests-environment-backcompat.test \ + testsuite-summary-color.test \ + testsuite-summary-count.test \ + testsuite-summary-count-many.test \ + testsuite-summary-reference-log.test \ + test-driver-custom-no-extra-driver.test \ + test-driver-custom.test \ + test-driver-custom-xfail-tests.test \ + test-driver-custom-multitest.test \ + test-driver-custom-multitest-recheck.test \ + test-driver-custom-multitest-recheck2.test \ + test-driver-custom-html.test \ + test-driver-custom-no-html.test \ + test-driver-create-log-dir.test \ + test-driver-strip-vpath.test \ + test-driver-fail.test \ + test-log.test \ + test-metadata-global-log.test \ + test-metadata-recheck.test \ + test-metadata-results.test \ + test-missing.test \ + test-missing2.test \ + test-trs-basic.test \ + test-trs-recover.test \ + test-trs-recover2.test \ parse.test \ percent.test \ percent2.test \ @@@ -1185,13 -1137,11 +1210,14 @@@ self-check-cleanup.test self-check-dir.test \ self-check-env-sanitize.test \ self-check-exit.test \ +self-check-explicit-skips.test \ self-check-is_newest.test \ self-check-me.test \ +self-check-reexec.test \ self-check-report.test \ self-check-sanity.test \ +self-check-seq.test \ + self-check-tap.test \ self-check-unindent.test \ sanity.test \ scripts.test \ @@@ -1408,18 -1359,80 +1436,84 @@@ yflags-conditional.test yflags-d-false-positives.test \ yflags-force-override.test \ yflags-force-conditional.test \ -yflags-var-expand.test \ -$(parallel_tests) +yflags-var-expand.test - checked_test_extensions = .test + + # List of tests on TAP support that use the files pre-computed by + # `tap-common-setup.test', and sources the `tap-setup.sh' helper + # script. + tap_with_common_setup_tests = \ + tap-autonumber.test \ + tap-bailout.test \ + tap-color.test \ + tap-deps.test \ + tap-diagnostic.test \ + tap-empty-diagnostic.test \ + tap-empty.test \ + tap-escape-directive.test \ + tap-exit.test \ + tap-signal.test \ + tap-fancy.test \ + tap-fancy2.test \ + tap-global-log.test \ + tap-global-result.test \ + tap-html.test \ + tap-log.test \ + tap-merge-stdout-stderr.test \ + tap-no-merge-stdout-stderr.test \ + tap-message-0.test \ + tap-no-disable-hard-error.test \ + tap-no-spurious-summary.test \ + tap-no-spurious.test \ + tap-not-ok-skip.test \ + tap-numeric-description.test \ + tap-out-of-order.test \ + tap-passthrough.test \ + tap-passthrough-exit.test \ + tap-plan.test \ + tap-plan-corner.test \ + tap-plan-corner2.test \ + tap-plan-errors.test \ + tap-realtime.test \ + tap-recheck-logs.test \ + tap-skip-whole-whitespace.test \ + tap-skip-whole.test \ + tap-todo-skip-together.test \ + tap-todo-skip-whitespace.test \ + tap-todo-skip.test \ + tap-unplanned.test \ + tap-whitespace-normalization.test \ + tap-with-and-without-number.test \ + tap-xfail-tests.test + + + # Their log files. + tap_with_common_setup_logs = $(tap_with_common_setup_tests:.test=.log) + + # Other tests on TAP support. + tap_other_tests = \ + tap-common-setup.test \ + tap-bad-prog.tap \ + tap-basic.test \ + tap-doc.test \ + tap-doc2.test \ + tap-empty.test \ + tap-more.test \ + tap-more2.test \ + tap-recheck.test \ + tap-summary.test \ + tap-summary-color.test + ++checked_test_extensions = .test .tap +expected_list_of_tests = $(handwritten_tests) +am__tmk = tests-in-makefile-list.tmp +am__tfs = tests-on-filesystem-list.tmp +am__tdf = diff-in-tests-lists.tmp all: all-am .SUFFIXES: - .SUFFIXES: .depmod .html .instspc .log .ptest .shtst .test -.SUFFIXES: .html .log .tap .test .trs -$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am $(srcdir)/instspc-tests.am $(am__configure_deps) ++.SUFFIXES: .depmod .html .instspc .log .ptest .shtst .tap .test .trs +$(srcdir)/Makefile.in: $(srcdir)/Makefile.am $(srcdir)/parallel-tests.am $(srcdir)/instspc-tests.am $(srcdir)/depmod-tests.am $(srcdir)/config-shell-tests.am $(top_srcdir)/CheckListOfTests.am $(am__configure_deps) @for dep in $?; do \ case '$(am__configure_deps)' in \ *$$dep*) \ @@@ -1588,15 -1637,15 +1718,35 @@@ recheck recheck-html list=`echo "$$list" | sed 's/ *$$//'`; \ $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"' .test.log: - @p='$<'; $(am__check_pre) $(TEST_LOG_COMPILE) "$$tst" $(am__check_post) + @p='$<'; $(am__check_pre) $(TEST_LOG_DRIVER) --test-name "$$f" \ + --log-file '$*.log' --trs-file '$*.trs' \ + $(am__common_driver_flags) $(AM_TEST_LOG_DRIVER_FLAGS) $(TEST_LOG_DRIVER_FLAGS) -- $(TEST_LOG_COMPILE) "$$tst" \ + $(AM_TESTS_FD_REDIRECT) + .tap.log: + @p='$<'; $(am__check_pre) $(TAP_LOG_DRIVER) --test-name "$$f" \ + --log-file '$*.log' --trs-file '$*.trs' \ + $(am__common_driver_flags) $(AM_TAP_LOG_DRIVER_FLAGS) $(TAP_LOG_DRIVER_FLAGS) -- $(TAP_LOG_COMPILE) "$$tst" \ + $(AM_TESTS_FD_REDIRECT) +.ptest.log: - @p='$<'; $(am__check_pre) $(PTEST_LOG_COMPILE) "$$tst" $(am__check_post) ++ @p='$<'; $(am__check_pre) $(PTEST_LOG_DRIVER) --test-name "$$f" \ ++ --log-file '$*.log' --trs-file '$*.trs' \ ++ $(am__common_driver_flags) $(AM_PTEST_LOG_DRIVER_FLAGS) $(PTEST_LOG_DRIVER_FLAGS) -- $(PTEST_LOG_COMPILE) "$$tst" \ ++ $(AM_TESTS_FD_REDIRECT) +.instspc.log: - @p='$<'; $(am__check_pre) $(INSTSPC_LOG_COMPILE) "$$tst" $(am__check_post) ++ @p='$<'; $(am__check_pre) $(INSTSPC_LOG_DRIVER) --test-name "$$f" \ ++ --log-file '$*.log' --trs-file '$*.trs' \ ++ $(am__common_driver_flags) $(AM_INSTSPC_LOG_DRIVER_FLAGS) $(INSTSPC_LOG_DRIVER_FLAGS) -- $(INSTSPC_LOG_COMPILE) "$$tst" \ ++ $(AM_TESTS_FD_REDIRECT) +.depmod.log: - @p='$<'; $(am__check_pre) $(DEPMOD_LOG_COMPILE) "$$tst" $(am__check_post) ++ @p='$<'; $(am__check_pre) $(DEPMOD_LOG_DRIVER) --test-name "$$f" \ ++ --log-file '$*.log' --trs-file '$*.trs' \ ++ $(am__common_driver_flags) $(AM_DEPMOD_LOG_DRIVER_FLAGS) $(DEPMOD_LOG_DRIVER_FLAGS) -- $(DEPMOD_LOG_COMPILE) "$$tst" \ ++ $(AM_TESTS_FD_REDIRECT) +.shtst.log: - @p='$<'; $(am__check_pre) $(SHTST_LOG_COMPILE) "$$tst" $(am__check_post) ++ @p='$<'; $(am__check_pre) $(SHTST_LOG_DRIVER) --test-name "$$f" \ ++ --log-file '$*.log' --trs-file '$*.trs' \ ++ $(am__common_driver_flags) $(AM_SHTST_LOG_DRIVER_FLAGS) $(SHTST_LOG_DRIVER_FLAGS) -- $(SHTST_LOG_COMPILE) "$$tst" \ ++ $(AM_TESTS_FD_REDIRECT) distdir: $(DISTFILES) @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ @@@ -1750,12 -1800,9 +1901,11 @@@ uninstall-am backcompat5-p.log: backcompat5.test check-exported-srcdir-p.log: check-exported-srcdir.test + check-fd-redirect-p.log: check-fd-redirect.test +check-subst-prog-p.log: check-subst-prog.test +check-subst-p.log: check-subst.test check-tests-in-builddir-p.log: check-tests-in-builddir.test - check-tests_environment-p.log: check-tests_environment.test check-p.log: check.test - check10-p.log: check10.test check11-p.log: check11.test check12-p.log: check12.test check2-p.log: check2.test @@@ -1787,98 -1852,53 +1938,117 @@@ $(srcdir)/instspc-tests.am: instspc-tes $(AM_V_GEN)($(am__cd) $(srcdir) \ && $(SHELL) ./instspc-tests.sh --generate-makefile) >$@ -$(instspc_tests): Makefile.am - $(AM_V_at)rm -f $@ $@-t - $(AM_V_GEN) :; \ - base=`expr 'x/$@' : 'x.*/instspc-\(.*\)\.test$$'`; \ - name=`expr x"$$base" : x'\(.*\)-'`; \ - action=`expr x"$$base" : x'.*-\(.*\)'`; \ - { \ - echo '#!/bin/sh'; \ - echo '# DO NOT EDIT! GENERATED AUTOMATICALLY!'; \ - echo; \ - echo '# Ensure proper definition of $$testsrcdir.'; \ - echo '. ./defs-static || exit 99'; \ - echo 'test -n "$$testsrcdir" || exit 99 # sanity check'; \ - echo; \ - echo "instspc_test_name='$$name'"; \ - echo "instspc_action='test-$$action'"; \ - echo ". \$$testsrcdir/instspc-tests.sh"; \ - } > $@-t - $(AM_V_at)chmod a+rx $@-t && mv -f $@-t $@ - -# All instspc-*.test tests work by sourcing the `instspc-tests.sh' -# script. Also, they all use shared data generated by the helper -# test `instspc-data.test', for reasons of speed. +# All `*.instspc' tests work by calling the script `instspc-tests.sh'. +# Also, they all use shared data generated by the `instspc-data.test' +# helper test (for reasons of speed). instspc-data.log: instspc-tests.sh -$(instspc_tests:.test=.log): instspc-tests.sh instspc-data.log +$(instspc_tests:.instspc=.log): instspc-tests.sh instspc-data.log +# This dummy declaration is required to have make actually produce +# expected log files from the `.instspc.log' suffix rule. +$(instspc_tests): + +$(srcdir)/depmod-tests.am: depmod-tests.sh Makefile.am + $(AM_V_GEN)($(am__cd) $(srcdir) \ + && $(SHELL) ./depmod-tests.sh --generate-makefile) >$@ + +# All `*.depmod' tests work by calling the script `depmod-tests.sh'. +# Also, they all use shared data generated by the `depmod-data.test' +# helper test (for reasons of speed). +depmod-data.log: depmod-tests.sh +$(depmod_tests:.depmod=.log): depmod-tests.sh depmod-data.log +# This dummy declaration is required to have make actually produce +# expected log files from the `.depmod.log' suffix rule. +$(depmod_tests): +ar-lib-w.log: ar-lib.test +compile-w.log: compile.test +compile2-w.log: compile2.test +compile3-w.log: compile3.test +compile4-w.log: compile4.test +compile5-w.log: compile5.test +compile6-w.log: compile6.test +instsh2-w.log: instsh2.test +instsh3-w.log: instsh3.test +mdate5-w.log: mdate5.test +mdate6-w.log: mdate6.test +missing-w.log: missing.test +missing2-w.log: missing2.test +missing3-w.log: missing3.test +missing5-w.log: missing5.test +mkinst3-w.log: mkinst3.test + +$(srcdir)/config-shell-tests.am: gen-config-shell-tests Makefile.am + $(AM_V_GEN)($(am__cd) $(srcdir) && $(SHELL) ./gen-config-shell-tests) >$@ + +# All `*-w.shtst' tests work by sourcing the script `config-shell-tests.sh'. +$(config_shell_tests:.shtst=.log): config-shell-tests.sh +# This dummy declaration is required to have make actually produce +# expected log files from the `.shtst.log' suffix rule. +$(config_shell_tests): + test-driver-custom-multitest.log: trivial-test-driver + test-driver-custom-multitest-recheck.log: trivial-test-driver + test-driver-custom-multitest-recheck2.log: trivial-test-driver + test-driver-custom-html.log: trivial-test-driver + + testsuite-summary-color.log testsuite-summary-count.log: \ + testsuite-summary-checks.sh extract-testsuite-summary + + testsuite-summary-count-many.log: trivial-test-driver + testsuite-summary-count-many.log: extract-testsuite-summary + + # Their dependencies. + $(tap_with_common_setup_logs): tap-common-setup.log tap-setup.sh + + tap-summary.log tap-summary-color.log: tap-summary-aux.sh + +.PHONY: maintainer-check-list-of-tests +maintainer-check-list-of-tests: + @if diff -u /dev/null /dev/null >/dev/null 2>&1; then \ + diff='diff -u'; \ + else \ + diff='diff'; \ + fi; \ + set X $(checked_test_extensions); shift; \ + if test $$# -eq 0; then \ + set X $(TEST_EXTENSIONS); shift; \ + fi; \ + if test $$# -eq 0; then \ + echo '$@: internal error: $$(checked_test_extensions) and' \ + '$$(TEST_EXTENSIONS) are both undefined or empty' >&2; \ + exit 255; \ + fi; \ + lst='$(expected_list_of_tests)'; \ + test -n "$$lst" || lst='$(TESTS)'; \ + for t in $$lst; do \ + echo "$$t"; \ + done | sort >$(am__tmk); \ + for ext in $$*; do \ + ls *$$ext 2>/dev/null; \ + if test $(srcdir) != $(builddir); then \ + (cd $(srcdir) && ls *$$ext 2>/dev/null); \ + fi; \ + done | sort | uniq >$(am__tfs); \ + if $$diff $(am__tmk) $(am__tfs) >$(am__tdf); then \ + result=0; \ + else \ + echo '$@: list of tests in Makefile an on filesystem differ' >&2; \ + echo "+ $$diff in-makefile on-filesystem" >&2; \ + cat $(am__tdf) >&2; \ + result=1; \ + fi; \ + rm -f $(am__tmk) $(am__tfs) $(am__tdf); \ + exit $$result; + +.PHONY: clean-maintcheck-testslist-tmp +clean-local: clean-maintcheck-testslist-tmp +clean-maintcheck-testslist-tmp: + rm -f $(am__tmk) $(am__tfs) $(am__tdf) + # Dependencies valid for each test case. $(TEST_LOGS): defs defs-static aclocal-$(APIVERSION) automake-$(APIVERSION) + # FIXME: this should be made more granular once we have a cleaner + # subdivision of the tests. + $(TEST_LOGS): plain-functions.sh tap-functions.sh clean-local: clean-local-check .PHONY: clean-local-check diff --cc tests/check11.test index 5305c96c2,acd05b0a2..bd6be3420 --- a/tests/check11.test +++ b/tests/check11.test @@@ -38,12 -38,23 +38,23 @@@ $AUTOCON $AUTOMAKE -a ./configure + -env TESTS=skip $MAKE -e check >stdout +env TESTS=skip $MAKE -e check >stdout || { cat stdout; Exit 1; } cat stdout - grep '1.*passed' stdout && Exit 1 + if test x"$parallel_tests" = x"yes"; then + count_test_results total=1 pass=0 fail=0 skip=1 xfail=0 xpass=0 error=0 + else + grep '1.*passed' stdout && Exit 1 + : For shells with buggy 'set -e'. + fi -env TESTS="skip skip2" $MAKE -e check >stdout +env TESTS="skip skip2" $MAKE -e check >stdout || { cat stdout; Exit 1; } cat stdout - grep '2.*passed' stdout && Exit 1 + if test x"$parallel_tests" = x"yes"; then + count_test_results total=2 pass=0 fail=0 skip=2 xfail=0 xpass=0 error=0 + else + grep '2.*passed' stdout && Exit 1 + : For shells with buggy 'set -e'. + fi : diff --cc tests/check3.test index b4f4f9d2e,ccfb9236d..4f3273706 --- a/tests/check3.test +++ b/tests/check3.test @@@ -54,18 -54,16 +54,18 @@@ EN $ACLOCAL $AUTOCONF - $AUTOMAKE + $AUTOMAKE -a ./configure --prefix "`pwd`/inst" -$MAKE check >stdout + +$MAKE check >stdout || { cat stdout; Exit 1; } cat stdout -grep 'PASS: subrun.sh' stdout +grep '^PASS: subrun\.sh *$' stdout +grep 'PASS.*echo\.sh' stdout && Exit 1 -# check should depend directly on check-am (similar tests are -# in check.test and check2.test). -$FGREP 'check: $(BUILT_SOURCES)' Makefile.in -$FGREP 'check: $(BUILT_SOURCES)' dir/Makefile.in +# check should depend directly on $(BUILT_SOURCES) (similar tests +# are in check.test and check2.test). +$EGREP '^check:.* \$\(BUILT_SOURCES\)( |$)' Makefile.in +$EGREP '^check:.* \$\(BUILT_SOURCES\)( |$)' dir/Makefile.in $MAKE clean # Sanity checks diff --cc tests/defs index 5f3108e4e,2dca4cfb6..da4fd7456 --- a/tests/defs +++ b/tests/defs @@@ -164,26 -113,23 +164,24 @@@ am__test_skipped=n Exit () { set +e - (exit $1) - exit $1 + # See comments in the exit trap for the reason we do this. + test 77 = $1 && am__test_skipped=yes + (exit $1); exit $1 } - # Print warnings (e.g., about skipped and failed tests) to this file - # number. Override by putting, say: - # stderr_fileno_=9; export stderr_fileno_; 9>&2 - # at the *end* (yes, this is mandatory; see comments in Makefile.am) of - # the definition of AM_TESTS_ENVIRONMENT. - # This is useful when using automake's parallel tests mode, to print the - # reason for skip/failure to console, rather than to the *.log files. - : ${stderr_fileno_=2} - - # Copied from Gnulib's `tests/init.sh'. - warn_ () { echo "$@" 1>&$stderr_fileno_; } - fail_ () { warn_ "$me: failed test: $@"; Exit 1; } - skip_ () { warn_ "$me: skipped test: $@"; Exit 77; } - fatal_ () { warn_ "$me: hard error: $@"; Exit 99; } - framework_failure_ () { warn_ "$me: set-up failure: $@"; Exit 99; } + if test $using_tap = yes; then + funcs_file_=tap-functions.sh + else + funcs_file_=plain-functions.sh + fi + + if test -f "$testsrcdir/$funcs_file_"; then + . "$testsrcdir/$funcs_file_" + else + echo "$me: $testsrcdir/$funcs_file_ not found, check \$testsrcdir" >&2 + Exit 99 + fi + unset funcs_file_ # cross_compiling # --------------- @@@ -255,44 -201,44 +253,81 @@@ using_gmake ( Exit 99;; esac } +am__using_gmake="" # Avoid interferences from the environment. + +# seq_ - print a sequence of numbers +# ---------------------------------- +# This function simulates GNU seq(1) portably. Valid usages: +# - seq LAST +# - seq FIRST LAST +# - seq FIRST INCREMENT LAST +seq_ () +{ + case $# in + 0) fatal_ "seq_: missing argument";; + 1) seq_first=1 seq_incr=1 seq_last=$1;; + 2) seq_first=$1 seq_incr=1 seq_last=$2;; + 3) seq_first=$1 seq_incr=$2 seq_last=$3;; + *) fatal_ "seq_: too many arguments";; + esac + # Try to avoid forks if possible. + case "$BASH_VERSION" in + ""|[12].*) + : Not bash, or a too old bash version. ;; + *) + # Use eval to protect dumber shells from parsing errors. + eval 'for ((i = seq_first; i <= seq_last; i += seq_incr)); do + echo $i + done' + return 0;; + esac + # Else, use GNU seq if available. + seq "$@" && return 0 + # Otherwise revert to a slower loop using expr(1). + i=$seq_first + while test $i -le $seq_last; do + echo $i + i=`expr $i + $seq_incr` + done +} + # count_test_results total=N pass=N fail=N xpass=N xfail=N skip=N error=N + # ----------------------------------------------------------------------- + # Check that a testsuite run driven by the parallel-tests harness has + # had the specified numbers of test results (specified by kind). + # This function assumes that the output of "make check" or "make recheck" + # has been saved in the `stdout' file in the current directory, and its + # log in the `test-suite.log' file. + count_test_results () + { + # Use a subshell so that we won't pollute the script namespace. + ( + # TODO: Do proper checks on the arguments? + total=ERR pass=ERR fail=ERR xpass=ERR xfail=ERR skip=ERR error=ERR + eval "$@" + # For debugging. + $EGREP -i '(total|x?pass|x?fail|skip|error)' stdout || : + rc=0 + # Avoid spurious failures with shells with "overly sensible" + # `errexit' shell flag, such as e.g., Solaris /bin/sh. + set +e + test `grep -c '^PASS:' stdout` -eq $pass || rc=1 + test `grep -c '^XFAIL:' stdout` -eq $xfail || rc=1 + test `grep -c '^SKIP:' stdout` -eq $skip || rc=1 + test `grep -c '^FAIL:' stdout` -eq $fail || rc=1 + test `grep -c '^XPASS:' stdout` -eq $xpass || rc=1 + test `grep -c '^ERROR:' stdout` -eq $error || rc=1 + grep "^# TOTAL: *$total$" stdout || rc=1 + grep "^# PASS: *$pass$" stdout || rc=1 + grep "^# XFAIL: *$xfail$" stdout || rc=1 + grep "^# SKIP: *$skip$" stdout || rc=1 + grep "^# FAIL: *$fail$" stdout || rc=1 + grep "^# XPASS: *$xpass$" stdout || rc=1 + grep "^# ERROR: *$error$" stdout || rc=1 + test $rc -eq 0 + ) + } + commented_sed_unindent_prog=' /^$/b # Nothing to do for empty lines. x # Get x into pattern space. diff --cc tests/defs-static.in index 17ed28493,f1294c638..407060d30 --- a/tests/defs-static.in +++ b/tests/defs-static.in @@@ -70,8 -66,8 +70,8 @@@ f # Check that the environment is properly sanitized. # Having variables exported to the empty string is OK, since our code # treats such variables as if they were unset. - for var in me required parallel_tests test_prefer_config_shell \ - original_AUTOMAKE original_ACLOCAL -for var in me required use_tap parallel_tests \ ++for var in me required use_tap parallel_tests test_prefer_config_shell \ + original_AUTOMAKE original_ACLOCAL; do if eval "test x\"\$$var\" != x" && env | grep "^$var=" >/dev/null; then echo "$argv0: variable \`$var' is set in the environment:" \ diff --cc tests/parallel-tests-ext-driver-prog.test index e92d26274,40fc4a5bf..eb0fcabf2 --- a/tests/parallel-tests-ext-driver-prog.test +++ b/tests/parallel-tests-ext-driver-prog.test @@@ -81,14 -77,12 +81,16 @@@ $AUTOCON $AUTOMAKE -a ./configure ++ + $MAKE -$MAKE check +$MAKE check || { cat test-suite.log; Exit 1; } +ls -l . sub +cat test-suite.log - grep 'chk-driver *1 *2 ' foo.log - grep 'test-driver *3 *4 ' bar.log - grep 'noext-driver *5 *6 ' baz.log - grep 'test-driver *3 *4 ' bla.log - grep 'noext-driver *5 *6 ' bli.suff.log - grep 'noext-driver *5 *6 ' sub/test.log + grep 'chk-compiler *1 *2' foo.log + grep 'test-compiler *3 *4' bar.log + grep 'noext-compiler *5 *6' baz.log + grep 'test-compiler *3 *4' bla.log + grep 'noext-compiler *5 *6' bli.suff.log + grep 'noext-compiler *5 *6' sub/test.log + : diff --cc tests/self-check-env-sanitize.test index 9dc97bef9,1f7dba2b3..e2d8f2a65 --- a/tests/self-check-env-sanitize.test +++ b/tests/self-check-env-sanitize.test @@@ -23,12 -23,20 +23,21 @@@ set -x exec 5>&1 - for var in me parallel_tests required original_AUTOMAKE original_ACLOCAL \ - test_prefer_config_shell - do + vars=' + me + parallel_tests + required + use_tap ++ test_prefer_config_shell + original_AUTOMAKE + original_ACLOCAL + ' + + for var in $vars; do env "$var=foo" $SHELL -c '. ./defs' foo.test && exit 1 env "$var=foo" $SHELL -c '. ./defs' foo.test 2>&1 1>&5 \ - | grep "foo\.test:.* variable \`$var'.* in the environment.*unsafe" || exit 1 + | grep "foo\.test:.* variable \`$var'.* in the environment.*unsafe" \ + || exit 1 done :