From: Stefano Lattarini Date: Mon, 22 Aug 2011 13:48:15 +0000 (+0200) Subject: Merge branch 'test-protocols' into testsuite-work X-Git-Tag: ng-0.5a~89^2~97 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=fcab6cddec5f183c7180beb566e83ded67eabf59;p=thirdparty%2Fautomake.git Merge branch 'test-protocols' into testsuite-work * test-protocols: testsuite: use the new awk+shell TAP driver in our own testsuite tap/awk: handle non-zero exit status from the test command tap/awk: prepare to fetch the exit status of the test command tap/awk: refactor for future changes testsuite: avoid spurious failure if rst2html program is missing tap/awk: support Solaris /usr/xpg4/bin/awk tap/awk: support Solaris /usr/bin/nawk tap/awk: correctly extract message from SKIP plans tap/awk: support colorized output tap: real (but still incomplete) awk implementation of TAP driver testsuite: fix spurious failure due to Solaris 'wc' incompatibility coverage: keyword "SKIP" in TAP plan is case-insensitive coverage: ambiguous use of TAP "TODO"/"SKIP" directives coverage: non-directive comments in TAP results are preserved coverage: TAP result number followed by non-whitespace word boundary coverage: a TAP result numbered "0" is always out-of-order cosmetics: fix botched indentation in perl TAP driver fix: regenerate tests/Makefile.in coverage: some more tests on corner cases of TAP support --- fcab6cddec5f183c7180beb566e83ded67eabf59 diff --cc ChangeLog index fb6bc35da,7cfe648eb..84fa082aa --- a/ChangeLog +++ b/ChangeLog @@@ -1,47 -1,172 +1,216 @@@ + 2011-08-22 Stefano Lattarini + + testsuite: use the new awk+shell TAP driver in our own testsuite + * tests/Makefile.am (TAP_LOG_DRIVER): Redefine to use the new + awk+shell tap driver, which should be complete enough for use in + a "controlled" production environment. + + 2011-08-22 Stefano Lattarini + + tap/awk: handle non-zero exit status from the test command + * lib/tap-driver.sh (get_test_exit_message): New function in the + awk script, used to extract the exit status of the test program, + or at least a good guess of it. + (write_test_results): Use it, reporting an ERROR result if it + detects that the test program exited with a non-zero status. + * tests/tap-signal.test: Account for the differences in the + error messages generated by the awk and perl TAP drivers. Fix + an unrelated typo in comments since we are at it. + + 2011-08-22 Stefano Lattarini + + tap/awk: prepare to fetch the exit status of the test command + * lib/tap-driver.sh: Rewrite some logic in the main parsing loop, + to make it possible to read the exit status of the test command + from the last line of the input stream. + + 2011-08-22 Stefano Lattarini + + tap/awk: refactor for future changes + * lib/tap-driver.sh: Rewrite the awk script so that the parsing + of the input stream is implemented as a hand-rolled loop in the + BEGIN block, using the `getline' builtin. + + 2011-08-22 Stefano Lattarini + + testsuite: avoid spurious failure if rst2html program is missing + * tests/tap-html.test ($required): Require `rst2html'. + + 2011-08-22 Stefano Lattarini + + tap/awk: support Solaris /usr/xpg4/bin/awk + * lib/tap-driver.sh: Use `\t', not `\\t', to represent tabulation + character in regexps, even when inside double-quoted strings; that + seems to be more portable to Solaris 10 XPG4 awk. + + 2011-08-22 Stefano Lattarini + + tap/awk: support Solaris /usr/bin/nawk + * lib/tap-driver.sh (close_or_die): Remove, the use of the `close' + builtin in there was confusing /usr/bin/nawk from Solaris 10 into + a syntax error. + (write_test_results): Simply use `close' on the `*.trs' file, that + should be enough. + + 2011-08-21 Stefano Lattarini + + tap/awk: correctly extract message from SKIP plans + * lib/tap-driver.sh (/^1\.\.0+[ \t]*#/): Add a forgotten `?' to + the regex used to extract the skip message from a "1..0" TAP plan. + * tests/planskip.test: Check for the just-fixed bug. + * tests/tap-planskip-unplanned-corner.test: Don't be too strict + w.r.t. the formatting of the "skip whole test" message, to avoid + failing due to bugs irrelevant for this test (like the one just + fixed). + + 2011-08-21 Stefano Lattarini + + tap/awk: support colorized output + * lib/tap-driver.sh: Pass definition of colors to the BEGIN block + of the awk script, using ... + ($init_colors): ... this new variable, which supersedes ... + ($red, $grn, $lgn, $blu, $mgn, $std): ... these variables, which + have been consequently removed. + ($color_tests): We don't have to pass this anymore to the awk + script, so do not convert it to a numeric boolean. + (decorate_result): Implement for real. + With this change, the test 'tap-color.test' now passes. + + 2011-08-21 Stefano Lattarini + + tap: real (but still incomplete) awk implementation of TAP driver + * lib/tap-driver.sh: Add an incomplete, but mostly working, + implementation of a TAP parser and driver in awk. It doesn't yet + support colorized output, fetching of exit status from test + programs, nor a way to escape TAP directives in TAP result lines, + but passes all the tests of TAP support in the automake testsuite, + apart from the following ones: + - tap-color.test + - tap-escape-directive.test + - tap-exit.test + - tap-missing-plan-and-bad-exit.test + - tap-passthrough-exit.test + - tap-planskip-badexit.test + - tap-planskip-unplanned-corner.test + - tap-signal.test + Tested on Debian GNU/Linux with GNU awk 3.1.7 and 3.0.2, "original + awk" 2010-05-23-1, and mawk 1.3.3-15 (with which also the test + `tap-realtime.test' fails, in addition to those listed above). + + 2011-08-21 Stefano Lattarini + + testsuite: fix spurious failure due to Solaris 'wc' incompatibility + * tests/tap-number-wordboundary.test ($planned): Strip any extra + whitespace from the output of `wc', to cater to Solaris 10. + + 2011-08-21 Stefano Lattarini + + coverage: keyword "SKIP" in TAP plan is case-insensitive + * tests/tap-planskip-case-insensitive.test: New test. + * tests/tap-planskip.test: Remove now-duplicated checks. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-21 Stefano Lattarini + + coverage: ambiguous use of TAP "TODO"/"SKIP" directives + * tests/tap-ambiguous-directive.test: New test. + * tests/tap-todo-skip-together.test: Reference it in heading + comments. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-21 Stefano Lattarini + + coverage: non-directive comments in TAP results are preserved + * tests/tap-result-comment.test: New test. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-20 Stefano Lattarini + + coverage: TAP result number followed by non-whitespace word boundary + * tests/tap-number-wordboundary.test: New test. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-20 Stefano Lattarini + + coverage: a TAP result numbered "0" is always out-of-order + * tests/tap-test-number-0.test: New, checks that a test result + numbered as 0 is to be considered out-of-order; this is + consistent with the behaviour of the `prove' utility. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-20 Stefano Lattarini + + cosmetics: fix botched indentation in perl TAP driver + * lib/tap-driver.pl (main, Getopt::Long::GetOptions): Fix + improper or botched indentation. + + 2011-08-20 Stefano Lattarini + + coverage: some more tests on corner cases of TAP support + * tests/tap-no-spurious.test: Extend checks verifying that a line + matching, say, the regex "^ok[a-zA-Z0-9_]" is *not* considered a + TAP result. + * tests/tap-no-spurious-numbers.test: New test, checking that our + TAP driver doesn't spuriously recognize as TAP result numbers what + is not (even if it seems pretty close). + * tests/tap-negative-numbers.test: New test, checking that our + TAP driver doesn't spuriously recognize negative TAP result + numbers, but correctly interprets them as test descriptions + instead. + * tests/tap-plan-leading-zero.test, tap-numbers-leading-zero.test: + New tests, checking how our driver fares in recognizing numbers + with leading zeroes in TAP results or TAP plans. + * tests/tap-planskip-malformed.test: New test, checking that a + malformed TAP plan is not recognized as a valid plan. + * tests/tap-plan-whitespace.test: New test, checking that a TAP + plan line with trailing whitespace is recognized and handled + correctly. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + +2011-08-19 Stefano Lattarini + + testsuite: various fixlets and tweakings + * tests/seenc.test: Make grepping of automake stderr stricter. + Add a trailing `:' command. + * tests/symlink.test: Likewise. Also, define `$am_create_testdir' + to "empty" to avoid bringing in unused auxiliary files (only to + have to remove them right away), and use proper m4 quoting in + `configure.in'. + * tests/vpath.test: Make grepping of generated `Makefile.in' + slightly stricter. Prefer trailing `:' over trailing `Exit 0'. + * tests/suffix6c.test: Unset OBJEXT to avoid interferences from + the environment. + * tests/suffix12.test: Do not whitespace-indent `##' comments + when they are embedded in a makefile rule: having them indented + is not part of the Automake API, and might cause failures with + e.g., Tru64 make. + * tests/syntax.test: Simplify the `Makefile.am' to ensure that + automake doesn't fail for the wrong reasons. Make grepping of + automake stderr slightly stricter. + * tests/test-harness-vpath-rewrite.test: Remove useless variable + definition from `Makefile.am'. + * tests/test-driver-custom-multitest.test: Fix typo in comments. + * tests/self-check-me.tap: Fix minor typo in test description. + * tests/vars3.test: Make grepping of automake stderr stricter + and safer. + * tests/version6.test: Add sanity check, verifying that the + version number extracted from `automake --version' output seems + legit. + * tests/auxdir2.test: Renamed ... + * tests/auxdir-compauted.tap: ... to this, and converted to the + use of TAP. + * tests/auxdir4.test: Renamed ... + * tests/auxdir-unportable.tap: ... to this, and converted to the + use of TAP. + * tests/auxdir3.test: Renamed ... + * tests/auxdir-misplaced.test: ... to this. + * tests/auxdir5.test: Renamed ... + * tests/auxdir-nonexistent.test: ... to this. + * tests/auxdir9.test: Renamed ... + * tests/auxdir-autodetect.test: ... to this. + * tests/Makefile.am (TESTS): Update. + (XFAIL_TESTS): Remove `auxdir2.test'. + 2011-08-19 Stefano Lattarini testsuite: fix some redundant autotools calls in tests on TAP diff --cc tests/Makefile.am index 337f8c684,4c8f3423f..c75e5d4eb --- a/tests/Makefile.am +++ b/tests/Makefile.am @@@ -16,24 -16,9 +16,24 @@@ # You should have received a copy of the GNU General Public License # along with this program. If not, see . +# Run the tests with the shell detected at configure time. +LOG_COMPILER = $(SHELL) + TEST_EXTENSIONS = .test .tap +TEST_LOG_COMPILER = $(LOG_COMPILER) +TAP_LOG_COMPILER = $(LOG_COMPILER) + - TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver.pl + TAP_LOG_DRIVER = AM_TAP_AWK='$(AWK)' $(SHELL) $(top_srcdir)/lib/tap-driver.sh -TAP_LOG_DRIVER_FLAGS = --merge +TAP_LOG_DRIVER_FLAGS = --merge --comments +## Avoid spurious TAP diagnostic. We use an awful indirection with a +## shell command substitution here, since the string needs to contain +## `#' characters, and these can't be escaped portably in a make macro +## definition. +## FIXME: this is quite inefficient though, as it adds one extra fork +## FIXME: per TAP test script; this is OK for the moment, since we have +## FIXME: few such scripts, but might become an issue in the future. +## Keep this in sync with the definition of $diag_string_ in tests/defs. +TAP_LOG_DRIVER_FLAGS += --diagnostic-string `printf '\043%%\043\n'` MAINTAINERCLEANFILES = EXTRA_DIST = ChangeLog-old diff --cc tests/Makefile.in index beada3f78,d9fafa794..1f2565af5 --- a/tests/Makefile.in +++ b/tests/Makefile.in @@@ -292,46 -278,88 +292,46 @@@ target_alias = @target_alias top_build_prefix = @top_build_prefix@ top_builddir = @top_builddir@ top_srcdir = @top_srcdir@ -TEST_EXTENSIONS = .test .tap + +# Run the tests with the shell detected at configure time. +LOG_COMPILER = $(SHELL) +TEST_EXTENSIONS = .test .tap .ptest .shtst +TEST_LOG_COMPILER = $(LOG_COMPILER) +TAP_LOG_COMPILER = $(LOG_COMPILER) - TAP_LOG_DRIVER = $(PERL) $(top_srcdir)/lib/tap-driver.pl + TAP_LOG_DRIVER = AM_TAP_AWK='$(AWK)' $(SHELL) $(top_srcdir)/lib/tap-driver.sh -TAP_LOG_DRIVER_FLAGS = --merge -MAINTAINERCLEANFILES = $(parallel_tests) $(instspc_tests) -EXTRA_DIST = ChangeLog-old gen-parallel-tests instspc-tests.sh \ - $(TESTS) tap-functions.sh plain-functions.sh \ +TAP_LOG_DRIVER_FLAGS = --merge --comments --diagnostic-string `printf \ + '\043%%\043\n'` +MAINTAINERCLEANFILES = +EXTRA_DIST = ChangeLog-old gen-parallel-tests parallel-tests.sh \ + gen-config-shell-tests config-shell-tests.sh \ + $(handwritten_tests) tap-functions.sh plain-functions.sh \ trivial-test-driver testsuite-summary-checks.sh \ extract-testsuite-summary tap-setup.sh tap-summary-aux.sh -XFAIL_TESTS = all.test auxdir2.test cond17.test gcj6.test \ - override-conditional-2.test pr8365-remake-timing.test \ - yacc-dist-nobuild-subdir.test txinfo5.test \ - $(instspc_xfail_tests) -parallel_tests = backcompat5-p.test check-concurrency-bug9245-p.test \ - check-exported-srcdir-p.test check-fd-redirect-p.test \ - check-tests-in-builddir-p.test check-p.test check11-p.test \ - check12-p.test check2-p.test check3-p.test check4-p.test \ - check5-p.test check6-p.test check7-p.test check8-p.test \ - check9-p.test color-p.test color2-p.test comment9-p.test \ - dejagnu-p.test exeext4-p.test maken3-p.test maken4-p.test \ - posixsubst-tests-p.test tests-environment-p.test -instspc_tests = instspc-squote-build.test instspc-squote-install.test \ - instspc-dquote-build.test instspc-dquote-install.test \ - instspc-bquote-build.test instspc-bquote-install.test \ - instspc-sharp-build.test instspc-sharp-install.test \ - instspc-dollar-build.test instspc-dollar-install.test \ - instspc-bang-build.test instspc-bang-install.test \ - instspc-bslash-build.test instspc-bslash-install.test \ - instspc-ampersand-build.test instspc-ampersand-install.test \ - instspc-percent-build.test instspc-percent-install.test \ - instspc-leftpar-build.test instspc-leftpar-install.test \ - instspc-rightpar-build.test instspc-rightpar-install.test \ - instspc-pipe-build.test instspc-pipe-install.test \ - instspc-caret-build.test instspc-caret-install.test \ - instspc-tilde-build.test instspc-tilde-install.test \ - instspc-qmark-build.test instspc-qmark-install.test \ - instspc-star-build.test instspc-star-install.test \ - instspc-plus-build.test instspc-plus-install.test \ - instspc-minus-build.test instspc-minus-install.test \ - instspc-comma-build.test instspc-comma-install.test \ - instspc-colon-build.test instspc-colon-install.test \ - instspc-semicol-build.test instspc-semicol-install.test \ - instspc-equal-build.test instspc-equal-install.test \ - instspc-less-build.test instspc-less-install.test \ - instspc-more-build.test instspc-more-install.test \ - instspc-at-build.test instspc-at-install.test \ - instspc-lqbrack-build.test instspc-lqbrack-install.test \ - instspc-rqbrack-build.test instspc-rqbrack-install.test \ - instspc-lcbrack-build.test instspc-lcbrack-install.test \ - instspc-rcbrack-build.test instspc-rcbrack-install.test \ - instspc-space-build.test instspc-space-install.test \ - instspc-tab-build.test instspc-tab-install.test \ - instspc-linefeed-build.test instspc-linefeed-install.test \ - instspc-backspace-build.test instspc-backspace-install.test \ - instspc-formfeed-build.test instspc-formfeed-install.test \ - instspc-carriageret-build.test \ - instspc-carriageret-install.test \ - instspc-quadrigraph0-build.test \ - instspc-quadrigraph0-install.test \ - instspc-quadrigraph1-build.test \ - instspc-quadrigraph1-install.test \ - instspc-quadrigraph2-build.test \ - instspc-quadrigraph2-install.test \ - instspc-quadrigraph3-build.test \ - instspc-quadrigraph3-install.test \ - instspc-quadrigraph4-build.test \ - instspc-quadrigraph4-install.test instspc-a_b-build.test \ - instspc-a_b-install.test instspc-a__b-build.test \ - instspc-a__b-install.test instspc-a_lf_b-build.test \ - instspc-a_lf_b-install.test instspc-dotdotdot-build.test \ - instspc-dotdotdot-install.test instspc-dosdrive-build.test \ - instspc-dosdrive-install.test instspc-miscglob1-build.test \ - instspc-miscglob1-install.test instspc-miscglob2-build.test \ - instspc-miscglob2-install.test -instspc_xfail_tests = instspc-squote-build.test \ - instspc-dquote-build.test instspc-bquote-build.test \ - instspc-sharp-build.test instspc-dollar-build.test \ - instspc-bslash-build.test instspc-ampersand-build.test \ - instspc-linefeed-build.test instspc-quadrigraph0-build.test \ - instspc-a_lf_b-build.test instspc-squote-install.test \ - instspc-dquote-install.test instspc-bquote-install.test \ - instspc-sharp-install.test instspc-dollar-install.test \ - instspc-linefeed-install.test instspc-a_lf_b-install.test +XFAIL_TESTS = \ +all.test \ +cond17.test \ +gcj6.test \ +override-conditional-2.test \ +pr8365-remake-timing.test \ +yacc-dist-nobuild-subdir.test \ +txinfo5.test + +parallel_tests = backcompat5-p.ptest check-concurrency-bug9245-p.ptest \ + check-exported-srcdir-p.ptest check-fd-redirect-p.ptest \ + check-subst-prog-p.ptest check-subst-p.ptest \ + check-tests-in-builddir-p.ptest check-p.ptest check11-p.ptest \ + check12-p.ptest check2-p.ptest check3-p.ptest check4-p.ptest \ + check5-p.ptest check6-p.ptest check7-p.ptest check8-p.ptest \ + color-p.ptest color2-p.ptest comment9-p.ptest dejagnu-p.ptest \ + exeext4-p.ptest maken3-p.ptest maken4-p.ptest \ + posixsubst-tests-p.ptest tests-environment-p.ptest +PTEST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/parallel-tests.sh +config_shell_tests = ar-lib-w.shtst compile-w.shtst compile2-w.shtst \ + compile3-w.shtst compile4-w.shtst compile5-w.shtst \ + compile6-w.shtst instsh2-w.shtst instsh3-w.shtst \ + mdate5-w.shtst mdate6-w.shtst missing-w.shtst missing2-w.shtst \ + missing3-w.shtst missing5-w.shtst mkinst3-w.shtst +SHTST_LOG_COMPILER = $(LOG_COMPILER) $(srcdir)/config-shell-tests.sh # Some testsuite-influential variables should be overridable from the # test scripts, but not from the environment.