From: Stefano Lattarini Date: Sun, 14 Aug 2011 15:12:37 +0000 (+0200) Subject: Merge branch 'test-protocols' into testsuite-work X-Git-Tag: ng-0.5a~89^2~101 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=66dc614311f2010c6346a66a41dff14c1c0a8a65;p=thirdparty%2Fautomake.git Merge branch 'test-protocols' into testsuite-work * test-protocols: coverage: missing tap plan and non-zero exit status testsuite: fix an unportable use of sed testsuite: avoid spurious failure if rst2html program is missing fix: two recent ChangeLog entries tap: improve diagnosing and reporting of plan mismatches tap: log all TAP stream, even after a "Bail out!" coverage: TAP diagnostics after "Bail out!" aren't reported coverage: TAP errors after a "Bail out!" directive aren't reported testsuite: reorganize tests on TAP plans coverage: more TAP driver consistency w.r.t. 'prove' tap: improve some comments in the TAP driver tap: non-zero exit status after "Bail out!" should not be reported tap: a "plan with SKIP" after test results is an error test harness: avoid possible fork bomb coverage: possible infinite recursion in the test harness coverage: possible infinite recursion in the test harness test defs: yet more environment cleanup --- 66dc614311f2010c6346a66a41dff14c1c0a8a65 diff --cc ChangeLog index fc233edc0,2cb53d466..3477d6125 --- a/ChangeLog +++ b/ChangeLog @@@ -1,62 -1,178 +1,237 @@@ + 2011-08-14 Stefano Lattarini + + coverage: missing tap plan and non-zero exit status + * tests/tap-missing-plan-and-bad-exit.test: New test. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-12 Stefano Lattarini + + testsuite: fix an unportable use of sed + * tests/tap-todo-skip-whitespace.test: Alternation with "\|" in + sed regular expressions is not portable to (at least) FreeBSD + sed; do not use it. + + 2011-08-12 Stefano Lattarini + + testsuite: avoid spurious failure if rst2html program is missing + * tests/test-driver-custom-no-html.test: Account for the + possibility of a lacking `rst2html' program in our final + grepping of make output. + + 2011-08-12 Stefano Lattarini + + tap: improve diagnosing and reporting of plan mismatches + Problem exposed by a failure in the test 'tap-plan-errors.test'. + * lib/tap-driver (NO_PLAN, EARLY_PLAN, LATE_PLAN): New constants. + (Throughout the file): Use more complex semantics for `$plan_seen', + which now also remember whether the plan that has been seen is an + "early" or "late" plan; in turn, this makes ... + ($tap_stopped): ... this variable redundant; remove it. + (handle_tap_plan): Adjust to avoid producing spurious or confusing + error messages about extra or mismatched "late" TAP plans. + * tests/tap-plan-corned.test: Adjust and extend. + * tests/tap-color.test: Likewise. + + 2011-08-12 Stefano Lattarini + + tap: log all TAP stream, even after a "Bail out!" + * lib/tap-driver ($bailed_out): New global boolean variable, + telling whether a "Bail out!" directive has been seen or not. + (handle_tap_bailout): This function does not anymore stop the + reading from TAP stream; instead, it sets `$bailed_out' to a + true value, so that only the subsequent parsing of the input + TAP stream is stopped. + (finish): Remove, no more needed, its contents inlined into ... + (main): ... this function, with related adjustments in the code + flow. + (get_test_exit_message): Do not "flush" the input TAP stream + to fetch the exit status of test script, it is not anymore + required. Add a sanity check. + * tests/tap-bailout-and-logging.test: New test. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-12 Stefano Lattarini + + coverage: TAP diagnostics after "Bail out!" aren't reported + This is compatible with the behaviour of the `prove' utility. + * tests/tap-bailout-suppress-later-diagnostic.test: New test. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-12 Stefano Lattarini + + coverage: TAP errors after a "Bail out!" directive aren't reported + This is compatible with the behaviour of the `prove' utility. + * tests/tap-bailout-suppress-later-errors.test: New test. + * tests/tap-bailout-badexit.test: Renamed ... + * tests/tap-bailout-suppress-badexit.test: ... to this, for + consistency. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-12 Stefano Lattarini + + testsuite: reorganize tests on TAP plans + * tests/tap-skip-whole.test: Rename ... + * tests/tap-planskip.test: ... to this + * tests/tap-skip-whole-badexit.test: Rename ... + * tests/tap-planskip-badexit.test: ... to this. + * tests/tap-skip-whole-bailout.test: Rename ... + * tests/tap-planskip-bailout.test: ... to this. + * tests/tap-skip-whole-bailout.test: Rename ... + * tests/tap-planskip-later-errors.test: ... to this. + * tests/tap-skip-whole-unplanned.test: Rename ... + * tests/tap-planskip-unplanned.test: ... to this. + * tests/tap-skip-whole-whitespace.test: Rename ... + * tests/tap-planskip-whitespace.test: ... to this. + * tests/tap-skip-whole-continue.test: Rename ... + * tests/tap-planskip-and-logging.test: ... to this, extend to try + more cases, and fix heading comments. + * tests/tap-skip-whole-lastline.test: Rename ... + * tests/tap-planskip-late.test: ... to this, fix heading comments, + and remove some checks that are just a duplication of those already + performed in the previous script. + * tests/tap-results-with-planskip.test: Rename to ... + * tests/tap-planskip-unplanned-corner.test: ... this, and improve + hading comments. + * tests/tap-planskip-later-errors.test: New test. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-12 Stefano Lattarini + + coverage: more TAP driver consistency w.r.t. 'prove' + * tests/skip-results-with-planskip.test: New test, verifies that + test results seen in a TAP stream that has a "plan with SKIP" are + flagged as errors. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-12 Stefano Lattarini + + tap: improve some comments in the TAP driver + * lib/tap-driver (handle_tap_plan): Improve comments describing + possible errors in a "plan with SKIP" directive. + + 2011-08-12 Stefano Lattarini + + tap: non-zero exit status after "Bail out!" should not be reported + This new behaviour is more consistent with that of the `prove' + utility. Also, experience with the use of TAP in the Automake's + own test suite has shown that reporting non-zero exit status from + a script after it has issued a "Bail out!" directive is mostly + redundant, more introducing noise rather than conveying useful + information. + * lib/tap-driver (finish): Inline the part of it that was getting + the script exit status ... + (main): ... in here. + * tests/tap-bailout-badexit.test: New test. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-11 Stefano Lattarini + + tap: a "plan with SKIP" after test results is an error + This new behaviour is both more natural and more consistent with + that of the `prove' utility. + * lib/tap-driver (handle_tap_plan): Do not stop TAP parsing when + a "plan with SKIP" line (e.g., "1..0 # SKIP") is encountered, and + do not print a "SKIP" line if some TAP result has already been + seen. + * tests/tap-skip-whole.test: Adapt. + * tests/tap-skip-whole-lastline.test: Likewise. + * tests/tap-global-result.test: Adapt and extend. + * tests/tap-skip-plan-errors.test: Likewise, and fix an obsolete + small part of a comment. + * tests/tap-skip-whole-bailout.test: New test. + * tests/tap-skip-whole-unplanned.test: Likewise. + * tests/tap-skip-whole-continue.test: Likewise. + * tests/Makefile.am (tap_with_common_setup_tests): Update. + + 2011-08-09 Stefano Lattarini + + test harness: avoid possible fork bomb + This fixes a regression w.r.t. the master branch, exposed by + test 'parallel-tests-fork-bomb.test'. + * lib/am/check.am (am--redo-logs): Detect possible infinite + recursion due to a test log in $(TEST_LOGS) being the same + as $(TEST_SUITE_LOG). + + 2011-08-09 Stefano Lattarini + + coverage: enhance test against infinite recursion in test harness + * tests/parallel-tests-fork-bomb.test: Enhance and extend a + little now, for soon-to-be-implemented more explicit and safer + semantics. + + 2011-08-09 Stefano Lattarini + + coverage: possible infinite recursion in the test harness + Motivated by a regression in the 'test-protocols' branch. + * tests/parallel-tests-fork-bomb.test: New test, checking that + if $(TEST_SUITE_LOG) is in $(TEST_LOGS), we obtain a diagnosed + error rather than a make hang or a fork bomb. + * tests/Makefile.am (TESTS): Update. + + 2011-08-09 Stefano Lattarini + + test defs: yet more environment cleanup + * tests/defs: Also unset the TEST_SUITE_LOG variable. + +2011-08-14 Stefano Lattarini + + self tests: some minor refactoring + * tests/self-check-exit.tap: Disable the `errexit' shell flag; + this allow to remove some boilerplate idioms and to make some + code slightly less cumbersome. + * tests/self-check-me.tap: Likewise. + +2011-08-14 Stefano Lattarini + + maintcheck: look for TAP-based tests without TAP plan + * Makefile.am (sc_tests_tap_plan): New maintainer check. + (syntax_check_rules): Add it. + * tests/suffix10.tap: Declare a TAP plan. + * tests/add-missing.tap: Likewise. + +2011-08-14 Stefano Lattarini + + testsuite: avoid confusing SKIP messages for errored tests + * tests/defs [$use_tap] (exit trap): Don't print the test plan + in case the client script exited with a non-zero status. This + avoids the display of redundant "SKIP" messages for tests with + lazy plans that error out before any test is run. + +2011-08-13 Stefano Lattarini + + testsuite: keep more debugging info around in 'add-missing.tap' + * tests/add-missing.tap: Use a different directory for each test, + so that in case of failure more information is kept around. + Since we are at it, remove redundant TAP diagnostic, and define + `$am_create_testdir' to "empty" to avoid bringing in unused + auxiliary files. + +2011-08-13 Stefano Lattarini + + testsuite: more granular count of test results in our TAP library + * tests/tap-functions.sh ($tap_bad_count_): Removed, superseded + by ... + ($tap_fail_count_, $tap_xpass_count_): ... these new variables, + which keep more granular counts. + ($tap_pass_count_: New variable. + * tests/defs (exit trap): Adjust and simplify accordingly. + +2011-08-13 Stefano Lattarini + + testsuite: fixlets and improvements in two long TAP-based tests + * tests/depmod.tap: Clean up the subdirectories used by tests that + passed, to avoid ending up with a too big test directory. This is + especially important since, on each and every system, at least some + tests in this script are expected to be skipped (which ones exactly + depends on the system), thus causing the test directory not to be + removed when the script terminates. + * tests/instspc.tap: Likewise (with the difference that tests here + are not expected to be skipped, but to xfail). Give more explicit + error message in case of setup failure. Skip the proper number of + tests when required, to avoid spurious errors due to mismatched + plan. Also, really avoid to run the unsupported test cases when + the need to skip them is detected. + 2011-08-09 Stefano Lattarini test harness: recursive make invocations must use $(AM_MAKEFLAGS) diff --cc tests/Makefile.am index f2e9c2dcf,59a6d142a..0db204e14 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@@ -740,8 -734,9 +740,9 @@@ parallel-tests-harderror.test parallel-tests-log-override-1.test \ parallel-tests-log-override-2.test \ parallel-tests-log-override-recheck.test \ -parallel-tests-cmdline-override.test \ parallel-tests-log-compiler-example.test \ +parallel-tests-cmdline-override.test \ + parallel-tests-fork-bomb.test \ parallel-tests-empty-testlogs.test \ parallel-tests-driver-install.test \ parallel-tests-no-color-in-log.test \ diff --cc tests/Makefile.in index 2821858bf,e294596dd..5f50e7d0b --- a/tests/Makefile.in +++ b/tests/Makefile.in @@@ -983,8 -992,9 +983,9 @@@ parallel-tests-harderror.test parallel-tests-log-override-1.test \ parallel-tests-log-override-2.test \ parallel-tests-log-override-recheck.test \ -parallel-tests-cmdline-override.test \ parallel-tests-log-compiler-example.test \ +parallel-tests-cmdline-override.test \ + parallel-tests-fork-bomb.test \ parallel-tests-empty-testlogs.test \ parallel-tests-driver-install.test \ parallel-tests-no-color-in-log.test \ @@@ -1409,12 -1417,17 +1414,18 @@@ tap-plan.test tap-plan-corner.test \ tap-plan-errors.test \ tap-plan-middle.test \ ++tap-missing-plan-and-bad-exit.test \ + tap-planskip.test \ + tap-planskip-late.test \ + tap-planskip-and-logging.test \ + tap-planskip-unplanned.test \ + tap-planskip-unplanned-corner.test \ + tap-planskip-whitespace.test \ + tap-planskip-badexit.test \ + tap-planskip-bailout.test \ + tap-planskip-later-errors.test \ tap-realtime.test \ tap-recheck-logs.test \ - tap-skip-whole-whitespace.test \ - tap-skip-whole.test \ - tap-skip-whole-lastline.test \ - tap-skip-whole-badexit.test \ tap-todo-skip-together.test \ tap-todo-skip-whitespace.test \ tap-todo-skip.test \