From: Stefano Lattarini Date: Sun, 7 Aug 2011 23:43:34 +0000 (+0200) Subject: Merge branch 'test-protocols' into tap-testsuite-work X-Git-Tag: ng-0.5a~89^2~132 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=cc98b9c624d0a07d703d6cf5eabd06fd0bbc34e8;p=thirdparty%2Fautomake.git Merge branch 'test-protocols' into tap-testsuite-work * test-protocols: fix: regenerate Makefiles test harness: work around a VPATH rewrite issue parallel-tests: work around Solaris XPG4 make segfault testsuite: fix weird spurious failure with Solaris /bin/sh testsuite: fix another spurious failure on Solaris make testsuite: fix two spurious failures on Solaris make tap driver: refactor and remove dead code testsuite: fix spurious failure in a test on TAP support tap: plan location is more liberal w.r.t. non-TAP lines testsuite: remove now-passing test from XFAIL_TESTS --- cc98b9c624d0a07d703d6cf5eabd06fd0bbc34e8 diff --cc ChangeLog index 4bff03b7e,533029622..30eb489d3 --- a/ChangeLog +++ b/ChangeLog @@@ -1,106 -1,112 +1,215 @@@ + 2011-08-08 Stefano Lattarini + + test harness: work around a VPATH rewrite issue + * lib/am/check.am (am__TEST_BASES): Removed, it's role taken + over by ... + (am__set_TESTS_bases): ... these new variable. + ($(TEST_SUITE_LOG): Use it, to avoid VPATH rewrite issues. + * automake.in (handle_tests): Update the code for the cleanup + of the `.trs' file to use `$(TEST_LOGS)' instead of + `$(am__TEST_BASES)'. + * tests/test-harness-vpath-rewrite.test: New test, better + exposing the problem fixed by this change + * tests/Makefile.am (TESTS): Add it. + * tests/test-trs-basic.test: Update and extend. + + 2011-08-07 Stefano Lattarini + + parallel-tests: work around Solaris XPG4 make segfault + Issue exposed by test `posixsubst-tests-p.test', and similar to + the problem solved by commit `v1.11-159-ge7aa360'. + * lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Trim trailing + whitespace from $list, to avoid triggering a nasty bug (potential + segfault) on Solaris XPG4 make and Heirloom make. + + 2011-08-07 Stefano Lattarini + + testsuite: fix weird spurious failure with Solaris /bin/sh + Solaris /bin/sh, when killed with a SIGTERM or SIGINT signal, can + apparently end up exiting with exit status 208, instead of leaving + the correct wide exit status to the parent. See: + + Work around this incompatibility. + * tap-signal.test: Look for the above Solaris bug. + (signal_caught): Adapt to handle it. + + 2011-08-07 Stefano Lattarini + + testsuite: fix another spurious failure on Solaris make + * tests/parallel-tests-log-override-recheck.test: Filter make + output before grepping it, for make implementations that, like + Solaris' one, print the whole of the failed recipe on failure. + + 2011-08-07 Stefano Lattarini + + testsuite: fix two spurious failures on Solaris make + * tests/tests-environment-and-log-compiler.test: Do not fail if + "make -n check" fails, as that is issued only for debugging, its + semantics being tested in other test scripts. + * tests/test-driver-strip-vpath.test: Likewise. + + 2011-08-07 Stefano Lattarini + + tap driver: refactor and remove dead code + * lib/tap-driver (get_tap_line): Removed, ... + (all callers): ... updated to use $parser->next directly instead. + (peek_tap_line, unget_tap_line, @tap_lines): Remove, they're not + used anymore. + (TAP_PEEKING): Block deleted, all its subroutines and variables + having been removed. + + 2011-08-07 Stefano Lattarini + + testsuite: fix spurious failure in a test on TAP support + * tests/tap-passthrough-exit.test: When the `--ignore-exit' flag + of the TAP driver is used, don't look for a message reporting the + non-zero exit statuses of tests in the log files; such message is + not expected to be there anymore. Related simplifications. + + 2011-08-07 Stefano Lattarini + + tap: plan location is more liberal w.r.t. non-TAP lines + With this change, only lines that are TAP results will matter + w.r.t. the position of the TAP plan in the input; for example, + this input: + this is a non-TAP line + # and this a TAP diagnostic line + 1..1 + ok 1 + was considered to be an error, diagnosed with a message "test + plan in middle of output"; as effect of the current change, such + input is now valid. This is more consistent with the behaviour + of the `prove' utility. + * lib/tap-driver ($lineno): Removed, no more needed. + ($tap_stopped): New global variable. + (stringify_test_result): Return "ERROR" if a TAP result is found + when `$tap_stopped' is set to true. + (handle_tap_test): Diagnose TAP results that comes after a late + plan. Add a couple of blank lines, for clarity. + (handle_tap_plan): Set `$tap_stopped' to true after a late plan + is encountered. Do not complain anymore for extra non-TAP lines + preceding or following the plan. Adjust comments. + (main): Don't increment $lineno anymore. + * tests/tap-plan.test: Extend a bit, and remove stale comment. + * tests/tap-color.test: Adjust. + * tests/tap-passthrough.test: Likewise. + * tests/tap-plan-corner.test: Adjust and extend. + * tests/tap-plan-errors.test: Likewise. + * tests/tap-plan-middle.test: New test. + * tests/tap-plan-corner2.test: Delete, it's obsolete now. + * tests/Makefile.am (XFAIL_TESTS): Remove it. + (tap_with_common_setup_tests): Likewise, and add + `tap-plan-corner.test'. + + 2011-08-07 Stefano Lattarini + + testsuite: remove now-passing test from XFAIL_TESTS + * tests/Makefile.am (XFAIL_TESTS): Remove `tap-signal.test', + which is passing since previous commit `v1.11-974-gc7fa872'. + +2011-08-07 Stefano Lattarini + + test defs: get name of TAP-using tests correctly + This change fixes the names of the temporary directories used + by the TAP-based tests (e.g., test `foo.tap' now uses `foo.dir' + rather than `foo.tap.dir'); more importantly, this make the + variable `$destdir' defined in `tests/defs' usable also in the + TAP tests. + ($me): Also strip the any `.tap' suffix from the name of the + test. + +2011-08-07 Stefano Lattarini + + testsuite: TAP tests properly decide when to remove tempdirs + Before this change, the TAP tests in the Automake testsuite were + removing the temporary test directory even when they failed or + were skipped, thus making debugging more difficult. + * tests/tap-functions.sh (incr_tap_count): Removed, superseded + by ... + (incr_): ... this function, which can increment the value of any + variable passed to it. + (result_): Updated to use `incr_' instead of the now-removed + `incr_tap_count_'. Keep count of failures, xfailures, xpasses, + and skips, using ... + ($tap_skip_count_, $tap_bad_count, _$tap_xfail_count_): ... + these new variables. + * tests/defs (trap): Try to use their values to decide whether + the temporary directory being used by the test script should be + removed or not. Other code reorganizations. And move the code + for the removal of the temporary directory out to ... + (rm_rf_): ... this new subroutine. + (Main code): Use that instead of duplicating the code. + +2011-08-07 Stefano Lattarini + + testsuite: improve and refactor our custom TAP shell library + This change is mostly done in preparation for the next one. + * tests/tap-functions.sh (result_with_directive_): Merge into ... + (result_): ... this function, which has been greatly extended and + improved in various ways. + (ok_, not_ok_, skip_, xfail_, xpass_): Adjust. + Adjust comments. Remove an extra empty line. + * tests/instspc.tap: Use the new improved interface of `result_'. + * tests/tap-bad-prog.tap: Likewise. + +2011-08-07 Stefano Lattarini + + testsuite: update "user interface" description in tests/README + * tests/README (Subsection "Interpretation"): Mention the "ERROR" + outcome. + (Subsection "Getting details from failures"): Don't tell that the + test scripts can be run directly, this is not 100% true; anymore; + instead ... + (Subsection "About the tests"): ... use this new subsection to + explain how to run the tests (either simple or TAP-based) by hand. + (Subsection "Supported shells"): Extend to refer to TAP tests too. + +2011-08-07 Stefano Lattarini + + testsuite: TAP tests can have early plan now + * tests/defs: When using TAP, allow the user to specify the number + of tests through the `$planned' variable. + (trap): Do not print the auto-generated plan if an explicit one + has already been printed. + * tests/defs-static.in: Complain and bail out if the `$planned' + variable is exported in the environment. Reformat the code to + facilitate future additions. + * tests/Makefile.am (TESTS_ENVIRONMENT): Neutralize `$planned'. + * tests/self-check-env-sanitize.test: Update. + * tests/ac-output-old.tap: Give an explicit TAP plan, by defining + `$planned' just before including `./defs'. + * tests/depmode.tap: Likewise. + * tests/automake-cmdline.tap: Likewise. + * tests/tap-bad-prog.tap: Likewise. + * tests/instspc.tap: Likewise, and related adjustments. + * tests/add-missing.tap: Make it explicit that we have no a-priori + plan, and tell why. + +2011-08-07 Stefano Lattarini + + testsuite: run autogenerated tests with $(LOG_COMPILER) too + This change enhances consistency in the testsuite. + * tests/Makefile.am: Some cosmetic reordering, to keep the + definition of FOO_LOG_COMPILER near to the place where `.foo' + is added to $(TEST_EXTENSIONS). + (PTEST_LOG_COMPILER): Run `$(srcdir)/parallel-tests.sh' with + `$(LOG_COMPILER)', not with `$(SHELL)'. + (SHTST_LOG_COMPILER): Likewise. + +2011-08-07 Stefano Lattarini + + testsuite: run TAP and "plain" tests with the same $(LOG_COMPILER) + With this change, it will be easier to override, at make time, the + shell used to run the tests. + * tests/Makefile.am (LOG_COMPILER): Define to `$(SHELL)'. + (TEST_LOG_COMPILER): Re-define to `$(LOG_COMPILER)'. + (TAP_LOG_COMPILER): Likewise. + +2011-08-06 Stefano Lattarini + + testsuite: run TAP-based tests explicitly with $(SHELL) + * tests/Makefile.am (TAP_LOG_COMPILER): Define to `$(SHELL)'. + 2011-08-06 Stefano Lattarini tap driver: handle signals received by the tests being run diff --cc tests/Makefile.in index cd30b562f,1372a0fa9..ee729c370 --- a/tests/Makefile.in +++ b/tests/Makefile.in @@@ -289,50 -282,88 +296,48 @@@ target_alias = @target_alias top_build_prefix = @top_build_prefix@ top_builddir = @top_builddir@ top_srcdir = @top_srcdir@ -TEST_EXTENSIONS = .test .tap + +# Run the tests with the shell detected at configure time. +LOG_COMPILER = $(SHELL) +TEST_EXTENSIONS = .test .tap .ptest .shtst +TEST_LOG_COMPILER = $(LOG_COMPILER) +TAP_LOG_COMPILER = $(LOG_COMPILER) TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver -TAP_LOG_DRIVER_FLAGS = --merge -MAINTAINERCLEANFILES = $(parallel_tests) $(instspc_tests) -EXTRA_DIST = ChangeLog-old gen-parallel-tests instspc-tests.sh \ - $(TESTS) tap-functions.sh plain-functions.sh \ +TAP_LOG_DRIVER_FLAGS = --merge --comments --diagnostic-string `printf \ + '\043%%\043\n'` +MAINTAINERCLEANFILES = +EXTRA_DIST = ChangeLog-old gen-parallel-tests parallel-tests.sh \ + gen-config-shell-tests config-shell-tests.sh \ + $(handwritten_tests) tap-functions.sh plain-functions.sh \ trivial-test-driver testsuite-summary-checks.sh \ extract-testsuite-summary tap-setup.sh tap-summary-aux.sh -XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \ - override-conditional-2.test pr8365-remake-timing.test \ - yacc-dist-nobuild-subdir.test tap-message-0.test txinfo5.test \ - $(instspc_xfail_tests) -parallel_tests = backcompat5-p.test check-exported-srcdir-p.test \ - check-fd-redirect-p.test check-tests-in-builddir-p.test \ - check-p.test check11-p.test check12-p.test check2-p.test \ - check3-p.test check4-p.test check5-p.test check6-p.test \ - check7-p.test check8-p.test check9-p.test color-p.test \ - color2-p.test comment9-p.test dejagnu-p.test exeext4-p.test \ - maken3-p.test maken4-p.test posixsubst-tests-p.test \ - tests-environment-p.test -instspc_tests = instspc-squote-build.test instspc-squote-install.test \ - instspc-dquote-build.test instspc-dquote-install.test \ - instspc-bquote-build.test instspc-bquote-install.test \ - instspc-sharp-build.test instspc-sharp-install.test \ - instspc-dollar-build.test instspc-dollar-install.test \ - instspc-bang-build.test instspc-bang-install.test \ - instspc-bslash-build.test instspc-bslash-install.test \ - instspc-ampersand-build.test instspc-ampersand-install.test \ - instspc-percent-build.test instspc-percent-install.test \ - instspc-leftpar-build.test instspc-leftpar-install.test \ - instspc-rightpar-build.test instspc-rightpar-install.test \ - instspc-pipe-build.test instspc-pipe-install.test \ - instspc-caret-build.test instspc-caret-install.test \ - instspc-tilde-build.test instspc-tilde-install.test \ - instspc-qmark-build.test instspc-qmark-install.test \ - instspc-star-build.test instspc-star-install.test \ - instspc-plus-build.test instspc-plus-install.test \ - instspc-minus-build.test instspc-minus-install.test \ - instspc-comma-build.test instspc-comma-install.test \ - instspc-colon-build.test instspc-colon-install.test \ - instspc-semicol-build.test instspc-semicol-install.test \ - instspc-equal-build.test instspc-equal-install.test \ - instspc-less-build.test instspc-less-install.test \ - instspc-more-build.test instspc-more-install.test \ - instspc-at-build.test instspc-at-install.test \ - instspc-lqbrack-build.test instspc-lqbrack-install.test \ - instspc-rqbrack-build.test instspc-rqbrack-install.test \ - instspc-lcbrack-build.test instspc-lcbrack-install.test \ - instspc-rcbrack-build.test instspc-rcbrack-install.test \ - instspc-space-build.test instspc-space-install.test \ - instspc-tab-build.test instspc-tab-install.test \ - instspc-linefeed-build.test instspc-linefeed-install.test \ - instspc-backspace-build.test instspc-backspace-install.test \ - instspc-formfeed-build.test instspc-formfeed-install.test \ - instspc-carriageret-build.test \ - instspc-carriageret-install.test \ - instspc-quadrigraph0-build.test \ - instspc-quadrigraph0-install.test \ - instspc-quadrigraph1-build.test \ - instspc-quadrigraph1-install.test \ - instspc-quadrigraph2-build.test \ - instspc-quadrigraph2-install.test \ - instspc-quadrigraph3-build.test \ - instspc-quadrigraph3-install.test \ - instspc-quadrigraph4-build.test \ - instspc-quadrigraph4-install.test instspc-a_b-build.test \ - instspc-a_b-install.test instspc-a__b-build.test \ - instspc-a__b-install.test instspc-a_lf_b-build.test \ - instspc-a_lf_b-install.test instspc-dotdotdot-build.test \ - instspc-dotdotdot-install.test instspc-dosdrive-build.test \ - instspc-dosdrive-install.test instspc-miscglob1-build.test \ - instspc-miscglob1-install.test instspc-miscglob2-build.test \ - instspc-miscglob2-install.test -instspc_xfail_tests = instspc-squote-build.test \ - instspc-dquote-build.test instspc-bquote-build.test \ - instspc-sharp-build.test instspc-dollar-build.test \ - instspc-bslash-build.test instspc-ampersand-build.test \ - instspc-linefeed-build.test instspc-quadrigraph0-build.test \ - instspc-a_lf_b-build.test instspc-squote-install.test \ - instspc-dquote-install.test instspc-bquote-install.test \ - instspc-sharp-install.test instspc-dollar-install.test \ - instspc-linefeed-install.test instspc-a_lf_b-install.test +XFAIL_TESTS = \ +all.test \ +auxdir2.test \ +cond17.test \ +gcj6.test \ +override-conditional-2.test \ +pr8365-remake-timing.test \ +yacc-dist-nobuild-subdir.test \ - tap-plan-corner2.test \ +tap-message-0.test \ - tap-signal.test \ +txinfo5.test + +parallel_tests = backcompat5-p.ptest check-exported-srcdir-p.ptest \ + check-fd-redirect-p.ptest check-subst-prog-p.ptest \ + check-subst-p.ptest check-tests-in-builddir-p.ptest \ + check-p.ptest check11-p.ptest check12-p.ptest check2-p.ptest \ + check3-p.ptest check4-p.ptest check5-p.ptest check6-p.ptest \ + check7-p.ptest check8-p.ptest color-p.ptest color2-p.ptest \ + comment9-p.ptest dejagnu-p.ptest exeext4-p.ptest \ + maken3-p.ptest maken4-p.ptest posixsubst-tests-p.ptest \ + tests-environment-p.ptest +PTEST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/parallel-tests.sh +config_shell_tests = ar-lib-w.shtst compile-w.shtst compile2-w.shtst \ + compile3-w.shtst compile4-w.shtst compile5-w.shtst \ + compile6-w.shtst instsh2-w.shtst instsh3-w.shtst \ + mdate5-w.shtst mdate6-w.shtst missing-w.shtst missing2-w.shtst \ + missing3-w.shtst missing5-w.shtst mkinst3-w.shtst +SHTST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/config-shell-tests.sh # Some testsuite-influential variables should be overridable from the # test scripts, but not from the environment.