+ 2011-08-08 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test harness: work around a VPATH rewrite issue
+ * lib/am/check.am (am__TEST_BASES): Removed, it's role taken
+ over by ...
+ (am__set_TESTS_bases): ... these new variable.
+ ($(TEST_SUITE_LOG): Use it, to avoid VPATH rewrite issues.
+ * automake.in (handle_tests): Update the code for the cleanup
+ of the `.trs' file to use `$(TEST_LOGS)' instead of
+ `$(am__TEST_BASES)'.
+ * tests/test-harness-vpath-rewrite.test: New test, better
+ exposing the problem fixed by this change
+ * tests/Makefile.am (TESTS): Add it.
+ * tests/test-trs-basic.test: Update and extend.
+
+ 2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ parallel-tests: work around Solaris XPG4 make segfault
+ Issue exposed by test `posixsubst-tests-p.test', and similar to
+ the problem solved by commit `v1.11-159-ge7aa360'.
+ * lib/am/check.am [%?PARALLEL_TESTS%] (check-TESTS): Trim trailing
+ whitespace from $list, to avoid triggering a nasty bug (potential
+ segfault) on Solaris XPG4 make and Heirloom make.
+
+ 2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix weird spurious failure with Solaris /bin/sh
+ Solaris /bin/sh, when killed with a SIGTERM or SIGINT signal, can
+ apparently end up exiting with exit status 208, instead of leaving
+ the correct wide exit status to the parent. See:
+ <http://dbaspot.com/shell/396118-bourne-shell-exit-code-term.html>
+ Work around this incompatibility.
+ * tap-signal.test: Look for the above Solaris bug.
+ (signal_caught): Adapt to handle it.
+
+ 2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix another spurious failure on Solaris make
+ * tests/parallel-tests-log-override-recheck.test: Filter make
+ output before grepping it, for make implementations that, like
+ Solaris' one, print the whole of the failed recipe on failure.
+
+ 2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix two spurious failures on Solaris make
+ * tests/tests-environment-and-log-compiler.test: Do not fail if
+ "make -n check" fails, as that is issued only for debugging, its
+ semantics being tested in other test scripts.
+ * tests/test-driver-strip-vpath.test: Likewise.
+
+ 2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap driver: refactor and remove dead code
+ * lib/tap-driver (get_tap_line): Removed, ...
+ (all callers): ... updated to use $parser->next directly instead.
+ (peek_tap_line, unget_tap_line, @tap_lines): Remove, they're not
+ used anymore.
+ (TAP_PEEKING): Block deleted, all its subroutines and variables
+ having been removed.
+
+ 2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: fix spurious failure in a test on TAP support
+ * tests/tap-passthrough-exit.test: When the `--ignore-exit' flag
+ of the TAP driver is used, don't look for a message reporting the
+ non-zero exit statuses of tests in the log files; such message is
+ not expected to be there anymore. Related simplifications.
+
+ 2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tap: plan location is more liberal w.r.t. non-TAP lines
+ With this change, only lines that are TAP results will matter
+ w.r.t. the position of the TAP plan in the input; for example,
+ this input:
+ this is a non-TAP line
+ # and this a TAP diagnostic line
+ 1..1
+ ok 1
+ was considered to be an error, diagnosed with a message "test
+ plan in middle of output"; as effect of the current change, such
+ input is now valid. This is more consistent with the behaviour
+ of the `prove' utility.
+ * lib/tap-driver ($lineno): Removed, no more needed.
+ ($tap_stopped): New global variable.
+ (stringify_test_result): Return "ERROR" if a TAP result is found
+ when `$tap_stopped' is set to true.
+ (handle_tap_test): Diagnose TAP results that comes after a late
+ plan. Add a couple of blank lines, for clarity.
+ (handle_tap_plan): Set `$tap_stopped' to true after a late plan
+ is encountered. Do not complain anymore for extra non-TAP lines
+ preceding or following the plan. Adjust comments.
+ (main): Don't increment $lineno anymore.
+ * tests/tap-plan.test: Extend a bit, and remove stale comment.
+ * tests/tap-color.test: Adjust.
+ * tests/tap-passthrough.test: Likewise.
+ * tests/tap-plan-corner.test: Adjust and extend.
+ * tests/tap-plan-errors.test: Likewise.
+ * tests/tap-plan-middle.test: New test.
+ * tests/tap-plan-corner2.test: Delete, it's obsolete now.
+ * tests/Makefile.am (XFAIL_TESTS): Remove it.
+ (tap_with_common_setup_tests): Likewise, and add
+ `tap-plan-corner.test'.
+
+ 2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: remove now-passing test from XFAIL_TESTS
+ * tests/Makefile.am (XFAIL_TESTS): Remove `tap-signal.test',
+ which is passing since previous commit `v1.11-974-gc7fa872'.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: get name of TAP-using tests correctly
+ This change fixes the names of the temporary directories used
+ by the TAP-based tests (e.g., test `foo.tap' now uses `foo.dir'
+ rather than `foo.tap.dir'); more importantly, this make the
+ variable `$destdir' defined in `tests/defs' usable also in the
+ TAP tests.
+ ($me): Also strip the any `.tap' suffix from the name of the
+ test.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: TAP tests properly decide when to remove tempdirs
+ Before this change, the TAP tests in the Automake testsuite were
+ removing the temporary test directory even when they failed or
+ were skipped, thus making debugging more difficult.
+ * tests/tap-functions.sh (incr_tap_count): Removed, superseded
+ by ...
+ (incr_): ... this function, which can increment the value of any
+ variable passed to it.
+ (result_): Updated to use `incr_' instead of the now-removed
+ `incr_tap_count_'. Keep count of failures, xfailures, xpasses,
+ and skips, using ...
+ ($tap_skip_count_, $tap_bad_count, _$tap_xfail_count_): ...
+ these new variables.
+ * tests/defs (trap): Try to use their values to decide whether
+ the temporary directory being used by the test script should be
+ removed or not. Other code reorganizations. And move the code
+ for the removal of the temporary directory out to ...
+ (rm_rf_): ... this new subroutine.
+ (Main code): Use that instead of duplicating the code.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: improve and refactor our custom TAP shell library
+ This change is mostly done in preparation for the next one.
+ * tests/tap-functions.sh (result_with_directive_): Merge into ...
+ (result_): ... this function, which has been greatly extended and
+ improved in various ways.
+ (ok_, not_ok_, skip_, xfail_, xpass_): Adjust.
+ Adjust comments. Remove an extra empty line.
+ * tests/instspc.tap: Use the new improved interface of `result_'.
+ * tests/tap-bad-prog.tap: Likewise.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: update "user interface" description in tests/README
+ * tests/README (Subsection "Interpretation"): Mention the "ERROR"
+ outcome.
+ (Subsection "Getting details from failures"): Don't tell that the
+ test scripts can be run directly, this is not 100% true; anymore;
+ instead ...
+ (Subsection "About the tests"): ... use this new subsection to
+ explain how to run the tests (either simple or TAP-based) by hand.
+ (Subsection "Supported shells"): Extend to refer to TAP tests too.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: TAP tests can have early plan now
+ * tests/defs: When using TAP, allow the user to specify the number
+ of tests through the `$planned' variable.
+ (trap): Do not print the auto-generated plan if an explicit one
+ has already been printed.
+ * tests/defs-static.in: Complain and bail out if the `$planned'
+ variable is exported in the environment. Reformat the code to
+ facilitate future additions.
+ * tests/Makefile.am (TESTS_ENVIRONMENT): Neutralize `$planned'.
+ * tests/self-check-env-sanitize.test: Update.
+ * tests/ac-output-old.tap: Give an explicit TAP plan, by defining
+ `$planned' just before including `./defs'.
+ * tests/depmode.tap: Likewise.
+ * tests/automake-cmdline.tap: Likewise.
+ * tests/tap-bad-prog.tap: Likewise.
+ * tests/instspc.tap: Likewise, and related adjustments.
+ * tests/add-missing.tap: Make it explicit that we have no a-priori
+ plan, and tell why.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: run autogenerated tests with $(LOG_COMPILER) too
+ This change enhances consistency in the testsuite.
+ * tests/Makefile.am: Some cosmetic reordering, to keep the
+ definition of FOO_LOG_COMPILER near to the place where `.foo'
+ is added to $(TEST_EXTENSIONS).
+ (PTEST_LOG_COMPILER): Run `$(srcdir)/parallel-tests.sh' with
+ `$(LOG_COMPILER)', not with `$(SHELL)'.
+ (SHTST_LOG_COMPILER): Likewise.
+
+2011-08-07 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: run TAP and "plain" tests with the same $(LOG_COMPILER)
+ With this change, it will be easier to override, at make time, the
+ shell used to run the tests.
+ * tests/Makefile.am (LOG_COMPILER): Define to `$(SHELL)'.
+ (TEST_LOG_COMPILER): Re-define to `$(LOG_COMPILER)'.
+ (TAP_LOG_COMPILER): Likewise.
+
+2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: run TAP-based tests explicitly with $(SHELL)
+ * tests/Makefile.am (TAP_LOG_COMPILER): Define to `$(SHELL)'.
+
2011-08-06 Stefano Lattarini <stefano.lattarini@gmail.com>
tap driver: handle signals received by the tests being run
top_build_prefix = @top_build_prefix@
top_builddir = @top_builddir@
top_srcdir = @top_srcdir@
-TEST_EXTENSIONS = .test .tap
+
+# Run the tests with the shell detected at configure time.
+LOG_COMPILER = $(SHELL)
+TEST_EXTENSIONS = .test .tap .ptest .shtst
+TEST_LOG_COMPILER = $(LOG_COMPILER)
+TAP_LOG_COMPILER = $(LOG_COMPILER)
TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver
-TAP_LOG_DRIVER_FLAGS = --merge
-MAINTAINERCLEANFILES = $(parallel_tests) $(instspc_tests)
-EXTRA_DIST = ChangeLog-old gen-parallel-tests instspc-tests.sh \
- $(TESTS) tap-functions.sh plain-functions.sh \
+TAP_LOG_DRIVER_FLAGS = --merge --comments --diagnostic-string `printf \
+ '\043%%\043\n'`
+MAINTAINERCLEANFILES =
+EXTRA_DIST = ChangeLog-old gen-parallel-tests parallel-tests.sh \
+ gen-config-shell-tests config-shell-tests.sh \
+ $(handwritten_tests) tap-functions.sh plain-functions.sh \
trivial-test-driver testsuite-summary-checks.sh \
extract-testsuite-summary tap-setup.sh tap-summary-aux.sh
-XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \
- override-conditional-2.test pr8365-remake-timing.test \
- yacc-dist-nobuild-subdir.test tap-message-0.test txinfo5.test \
- $(instspc_xfail_tests)
-parallel_tests = backcompat5-p.test check-exported-srcdir-p.test \
- check-fd-redirect-p.test check-tests-in-builddir-p.test \
- check-p.test check11-p.test check12-p.test check2-p.test \
- check3-p.test check4-p.test check5-p.test check6-p.test \
- check7-p.test check8-p.test check9-p.test color-p.test \
- color2-p.test comment9-p.test dejagnu-p.test exeext4-p.test \
- maken3-p.test maken4-p.test posixsubst-tests-p.test \
- tests-environment-p.test
-instspc_tests = instspc-squote-build.test instspc-squote-install.test \
- instspc-dquote-build.test instspc-dquote-install.test \
- instspc-bquote-build.test instspc-bquote-install.test \
- instspc-sharp-build.test instspc-sharp-install.test \
- instspc-dollar-build.test instspc-dollar-install.test \
- instspc-bang-build.test instspc-bang-install.test \
- instspc-bslash-build.test instspc-bslash-install.test \
- instspc-ampersand-build.test instspc-ampersand-install.test \
- instspc-percent-build.test instspc-percent-install.test \
- instspc-leftpar-build.test instspc-leftpar-install.test \
- instspc-rightpar-build.test instspc-rightpar-install.test \
- instspc-pipe-build.test instspc-pipe-install.test \
- instspc-caret-build.test instspc-caret-install.test \
- instspc-tilde-build.test instspc-tilde-install.test \
- instspc-qmark-build.test instspc-qmark-install.test \
- instspc-star-build.test instspc-star-install.test \
- instspc-plus-build.test instspc-plus-install.test \
- instspc-minus-build.test instspc-minus-install.test \
- instspc-comma-build.test instspc-comma-install.test \
- instspc-colon-build.test instspc-colon-install.test \
- instspc-semicol-build.test instspc-semicol-install.test \
- instspc-equal-build.test instspc-equal-install.test \
- instspc-less-build.test instspc-less-install.test \
- instspc-more-build.test instspc-more-install.test \
- instspc-at-build.test instspc-at-install.test \
- instspc-lqbrack-build.test instspc-lqbrack-install.test \
- instspc-rqbrack-build.test instspc-rqbrack-install.test \
- instspc-lcbrack-build.test instspc-lcbrack-install.test \
- instspc-rcbrack-build.test instspc-rcbrack-install.test \
- instspc-space-build.test instspc-space-install.test \
- instspc-tab-build.test instspc-tab-install.test \
- instspc-linefeed-build.test instspc-linefeed-install.test \
- instspc-backspace-build.test instspc-backspace-install.test \
- instspc-formfeed-build.test instspc-formfeed-install.test \
- instspc-carriageret-build.test \
- instspc-carriageret-install.test \
- instspc-quadrigraph0-build.test \
- instspc-quadrigraph0-install.test \
- instspc-quadrigraph1-build.test \
- instspc-quadrigraph1-install.test \
- instspc-quadrigraph2-build.test \
- instspc-quadrigraph2-install.test \
- instspc-quadrigraph3-build.test \
- instspc-quadrigraph3-install.test \
- instspc-quadrigraph4-build.test \
- instspc-quadrigraph4-install.test instspc-a_b-build.test \
- instspc-a_b-install.test instspc-a__b-build.test \
- instspc-a__b-install.test instspc-a_lf_b-build.test \
- instspc-a_lf_b-install.test instspc-dotdotdot-build.test \
- instspc-dotdotdot-install.test instspc-dosdrive-build.test \
- instspc-dosdrive-install.test instspc-miscglob1-build.test \
- instspc-miscglob1-install.test instspc-miscglob2-build.test \
- instspc-miscglob2-install.test
-instspc_xfail_tests = instspc-squote-build.test \
- instspc-dquote-build.test instspc-bquote-build.test \
- instspc-sharp-build.test instspc-dollar-build.test \
- instspc-bslash-build.test instspc-ampersand-build.test \
- instspc-linefeed-build.test instspc-quadrigraph0-build.test \
- instspc-a_lf_b-build.test instspc-squote-install.test \
- instspc-dquote-install.test instspc-bquote-install.test \
- instspc-sharp-install.test instspc-dollar-install.test \
- instspc-linefeed-install.test instspc-a_lf_b-install.test
+XFAIL_TESTS = \
+all.test \
+auxdir2.test \
+cond17.test \
+gcj6.test \
+override-conditional-2.test \
+pr8365-remake-timing.test \
+yacc-dist-nobuild-subdir.test \
- tap-plan-corner2.test \
+tap-message-0.test \
- tap-signal.test \
+txinfo5.test
+
+parallel_tests = backcompat5-p.ptest check-exported-srcdir-p.ptest \
+ check-fd-redirect-p.ptest check-subst-prog-p.ptest \
+ check-subst-p.ptest check-tests-in-builddir-p.ptest \
+ check-p.ptest check11-p.ptest check12-p.ptest check2-p.ptest \
+ check3-p.ptest check4-p.ptest check5-p.ptest check6-p.ptest \
+ check7-p.ptest check8-p.ptest color-p.ptest color2-p.ptest \
+ comment9-p.ptest dejagnu-p.ptest exeext4-p.ptest \
+ maken3-p.ptest maken4-p.ptest posixsubst-tests-p.ptest \
+ tests-environment-p.ptest
+PTEST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/parallel-tests.sh
+config_shell_tests = ar-lib-w.shtst compile-w.shtst compile2-w.shtst \
+ compile3-w.shtst compile4-w.shtst compile5-w.shtst \
+ compile6-w.shtst instsh2-w.shtst instsh3-w.shtst \
+ mdate5-w.shtst mdate6-w.shtst missing-w.shtst missing2-w.shtst \
+ missing3-w.shtst missing5-w.shtst mkinst3-w.shtst
+SHTST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/config-shell-tests.sh
# Some testsuite-influential variables should be overridable from the
# test scripts, but not from the environment.