tests: list of tests placed in a separate Makefile fragment
This change allows us to easily use make instead of by-hand (and
more fragile) grepping to extract the list of hand-written tests
from our Makefiles.
* bootstrap (BOOTSTRAP_MAKE): New variable.
Use it to run make in order to extract the list of hand-written
tests from ...
* tests/list-of-tests.mk: ... this new file, where they are
listed in ...
(handwritten_TESTS, tap_with_common_setup_TESTS, tap_other_TESTS):
... these variables, extracted respectively from ...
* tests/Makefile.am (handwritten_tests, tap_other_tests,
tap_with_common_setup_tests): ... these older variables, now
removed.
(EXTRA_DIST, expected_list_of_tests): Update w.r.t. the variable
names' changes.
(include $(srcdir)/list-of-tests.mk): New inclusion.
Remove an overkill indirection, and with it ...
(tap_with_common_setup_logs): ... this variable.
($(srcdir)/parallel-tests.am, $(srcdir)/config-shell-tests.am):
Update to match the new interface of ...
(gen-parallel-tests, gen-config-shell-tests): ... these scripts,
that now read the list of tests from the standard input.
* test-protocols:
tests: prefer AM_... variables to user-reserved ones in Makefile
tests: some minor fixlets w.r.t. TAP-based tests
tests: avoid re-running a test with the 'parallel-tests' option
tests: defaults to awk+shell driver in tests on TAP support
With minor manual edits to 'tests/Makefile.am' to complete the
TAP_LOG_DRIVER_FLAGS -> AM_TAP_LOG_DRIVER_FLAGS rename.
The variable that is used by our test scripts to decide whether
they are "plain" or TAP-based is `using_tap'; however, some
peripheral sanity checks and sanitization code in our testsuite
were erroneously using the `use_tap' variable instead. Fix that.
* tests/Makefile.am (AM_TESTS_ENVIRONMENT): Nullify `using_tap',
not `use_tap'.
* tests/defs-static.in: Check that `using_tap', not `use_tap',
is not exported.
* tests/self-check-env-sanitize.test: Update.
tests: avoid re-running a test with the 'parallel-tests' option
* tests/backcompat5.test (parallel_tests): Define to "no", to
prevent the generation of a sibling test script that uses the
'parallel-tests' driver. That would be useless anyway, since
we manually override the contents of configure.in, thus clobbering
the `parallel-tests' option in AM_INIT_AUTOMAKE.
Fix a typo in comments since we are at it.
testsuite: different test scripts cannot share the same basename
If two test scripts have the same basename, they will end up
sharing the same log file, leading to all sort of undefined and
undesired behaviours. Add a Makefile check verifying that this
is not the case.
* tests/test-logs-repeated.test: New test, verifying previously
unspecified semantics (in the definition of $(TEST_LOGS) from
$(TESTS)) that we use in our new check.
* tests/Makefile.am (TESTS): Update.
(check-no-repeated-test-name): New .PHONY check.
(check-local): Depend on it.
* test-protocols:
tests: fix spurious failure for awk implementation of TAP driver
tap/awk: allow escaping of TAP directives
coverage: more about escaping of TAP directives
tap: improve syncing between awk+shell and perl implementations
tap/perl: don't redirect perl warnings/errors to log files
tap/perl: add copyright notice, version string, and emacs stuff
tap/awk: don't redirect awk stderr to log files
tap/awk: remove obsolete "FIXME" comment
tests: fix spurious failure for awk implementation of TAP driver
* tests/tap-realtime.test: The awk+shell implementation of the TAP
driver must "read ahead" of one line in order to catch the exit
status of the test script it runs. So, in the TAP-producer script
here, be sure to echo one "dummy" line after each TAP result line
in order not to cause false positives.
* tests/Makefile.am (TESTS): Also list ...
* tests/java-empty-classpath.test: ... this test. Since we are
at it, remove trailing whitespace from a couple of lines.
* lib/tap-driver.sh (setup_result_obj): Handle escaping of TAP
directives in a way tat is (mostly) compatible by what is done
by the TAP::Parser module.
With this change, the tests `tap-escape-directive.test' and
`tap-escape-directive-2.test' now also pass with the shell/awk
implementation of the TAP driver.
* tests/tap-escape-directive.test: Make grepping of testsuite
progress output slightly stricter.
* tests/tap-escape-directive-2.test: New test (failing with the
awk implementation of the TAP driver).
* tests/Makefile.am (tap_with_common_setup_tests): Update.
tap: improve syncing between awk+shell and perl implementations
* lib/tap-driver.pl (stringify_test_result): Renamed ...
(stringify_result_obj): ... to this. Break up a clause in the
long "if/elsif/.../else" construct to avoid unaesthetic line
breaks and to be more synced with the sibling function in
`tap-driver.sh'. Rename the `$result', `$PASS' and `$FAIL'
variables to respectively `$result_obj', `$COOKED_PASS' and
`$COOKED_FAIL', for clarity and better syncing.
(handle_tap_test): Renamed ...
(handle_tap_result): ... to this, and change the name of the
`$test' local variable to `$result_obj'.
(extract_comment): Reimplement using the simpler `index' and
`substr' builtins, rather than with more advanced uses of
regular expressions.
(%test_results, @test_results): Renamed respectively ...
(%test_results_seen, @test_results_list): ... to these, and
related adjustments throughout the `TEST_RESULTS' block.
(main, get_global_test_result): Refactor and do some cosmetic
changes to make these functions clearer and better synced with
sibling code in `tap-driver.sh'.
Other minor cosmetic and typo fixes.
* lib/tap-driver.sh (extract_tap_comment): Remove outdated
"FIXME" comments.
(get_global_test_result): Small reordering to make it better
synced with its sibling function in `tap-driver.pl'.
(stringify_result_obj): Consistently use `result_obj' as the
parameter name.
Other minor cosmetic and typo fixes.
tap/perl: don't redirect perl warnings/errors to log files
With this change, the test `tap-driver-stderr.test' also passes
with the perl implementation of the TAP driver.
* lib/tap-driver.pl (start): Save the original stderr into the
`OLDERR' file handle, and call ...
(trap_perl_warnings_and_errors): ... this new function, trapping
the `__WARN__' and `__DIE__' "pseudo-signals" to ensure that the
warning and error messages goes to that original stderr stream.
Since we are at it, be sure to prepend all possible "die" message
with the name of the script as given by the `$ME' variable.
tap/perl: add copyright notice, version string, and emacs stuff
* lib/tap-driver.pl: Add proper copyright notice and bug reporting
address. Remove obsolete heading comments. Add trailing comments
for emacs w.r.t. code-formatting rules and automatic updating of
the date-based script version saved into the `$VERSION' variable.
($VERSION): Define to the current UTC date.
* lib/tap-driver.sh: Handle the file descriptors redirections
more smartly, to avoid sending error messages from awk (about
e.g. syntax or I/O errors) to the log files instead that to the
console.
* tests/tap-driver-stderr.test: New test, verifying the improved
driver behaviour. Notice that this test still fails when using
the perl implementation of the TAP driver.
* tests/Makefile.am (tap_other_tests): Update.
* lib/tap-driver.sh: Remove obsolete "FIXME" comment about our
driver losing the exit status of the tested program; this issue
has been solved in precedent commit `v1.11-1052-gd630a0d'. Fix
an unrelated typo in comments since we are at it.
* tests/tap-signal.test (signal_caught): Fix a regression from
commit `v1.11-1052-gd630a0d' that was causing this test to fail
spuriously on Solaris (due to an already worked-around Solaris
/bin/sh bug, see commit `v1.11-981-g0ef3ef5').
* test-protocols:
testsuite: use the new awk+shell TAP driver in our own testsuite
tap/awk: handle non-zero exit status from the test command
tap/awk: prepare to fetch the exit status of the test command
tap/awk: refactor for future changes
testsuite: avoid spurious failure if rst2html program is missing
tap/awk: support Solaris /usr/xpg4/bin/awk
tap/awk: support Solaris /usr/bin/nawk
tap/awk: correctly extract message from SKIP plans
tap/awk: support colorized output
tap: real (but still incomplete) awk implementation of TAP driver
testsuite: fix spurious failure due to Solaris 'wc' incompatibility
coverage: keyword "SKIP" in TAP plan is case-insensitive
coverage: ambiguous use of TAP "TODO"/"SKIP" directives
coverage: non-directive comments in TAP results are preserved
coverage: TAP result number followed by non-whitespace word boundary
coverage: a TAP result numbered "0" is always out-of-order
cosmetics: fix botched indentation in perl TAP driver
fix: regenerate tests/Makefile.in
coverage: some more tests on corner cases of TAP support
testsuite: use the new awk+shell TAP driver in our own testsuite
* tests/Makefile.am (TAP_LOG_DRIVER): Redefine to use the new
awk+shell tap driver, which should be complete enough for use in
a "controlled" production environment.
tap/awk: handle non-zero exit status from the test command
* lib/tap-driver.sh (get_test_exit_message): New function in the
awk script, used to extract the exit status of the test program,
or at least a good guess of it.
(write_test_results): Use it, reporting an ERROR result if it
detects that the test program exited with a non-zero status.
* tests/tap-signal.test: Account for the differences in the
error messages generated by the awk and perl TAP drivers. Fix
an unrelated typo in comments since we are at it.
tap/awk: prepare to fetch the exit status of the test command
* lib/tap-driver.sh: Rewrite some logic in the main parsing loop,
to make it possible to read the exit status of the test command
from the last line of the input stream.
* lib/tap-driver.sh: Rewrite the awk script so that the parsing
of the input stream is implemented as a hand-rolled loop in the
BEGIN block, using the `getline' builtin.
* lib/tap-driver.sh: Use `\t', not `\\t', to represent tabulation
character in regexps, even when inside double-quoted strings; that
seems to be more portable to Solaris 10 XPG4 awk.
* lib/tap-driver.sh (close_or_die): Remove, the use of the `close'
builtin in there was confusing /usr/bin/nawk from Solaris 10 into
a syntax error.
(write_test_results): Simply use `close' on the `*.trs' file, that
should be enough.
tap/awk: correctly extract message from SKIP plans
* lib/tap-driver.sh (/^1\.\.0+[ \t]*#/): Add a forgotten `?' to
the regex used to extract the skip message from a "1..0" TAP plan.
* tests/planskip.test: Check for the just-fixed bug.
* tests/tap-planskip-unplanned-corner.test: Don't be too strict
w.r.t. the formatting of the "skip whole test" message, to avoid
failing due to bugs irrelevant for this test (like the one just
fixed).
* lib/tap-driver.sh: Pass definition of colors to the BEGIN block
of the awk script, using ...
($init_colors): ... this new variable, which supersedes ...
($red, $grn, $lgn, $blu, $mgn, $std): ... these variables, which
have been consequently removed.
($color_tests): We don't have to pass this anymore to the awk
script, so do not convert it to a numeric boolean.
(decorate_result): Implement for real.
With this change, the test 'tap-color.test' now passes.
tap: real (but still incomplete) awk implementation of TAP driver
* lib/tap-driver.sh : Add an incomplete, but mostly working,
implementation of a TAP parser and driver in awk. It doesn't yet
support colorized output, fetching of exit status from test
programs, nor a way to escape TAP directives in TAP result lines,
but passes all the tests of TAP support in the automake testsuite,
apart from the following ones:
- tap-color.test
- tap-escape-directive.test
- tap-exit.test
- tap-missing-plan-and-bad-exit.test
- tap-passthrough-exit.test
- tap-planskip-badexit.test
- tap-planskip-unplanned-corner.test
- tap-signal.test
Tested on Debian GNU/Linux with GNU awk 3.1.7 and 3.0.2, "original
awk" 2010-05-23-1, and mawk 1.3.3-15 (with which also the test
`tap-realtime.test' fails, in addition to those listed above).
coverage: ambiguous use of TAP "TODO"/"SKIP" directives
* tests/tap-ambiguous-directive.test: New test.
* tests/tap-todo-skip-together.test: Reference it in heading
comments.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
coverage: a TAP result numbered "0" is always out-of-order
* tests/tap-test-number-0.test: New, checks that a test result
numbered as 0 is to be considered out-of-order; this is
consistent with the behaviour of the `prove' utility.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
coverage: some more tests on corner cases of TAP support
* tests/tap-no-spurious.test: Extend checks verifying that a line
matching, say, the regex "^ok[a-zA-Z0-9_]" is *not* considered a
TAP result.
* tests/tap-no-spurious-numbers.test: New test, checking that our
TAP driver doesn't spuriously recognize as TAP result numbers what
is not (even if it seems pretty close).
* tests/tap-negative-numbers.test: New test, checking that our
TAP driver doesn't spuriously recognize negative TAP result
numbers, but correctly interprets them as test descriptions
instead.
* tests/tap-plan-leading-zero.test, tap-numbers-leading-zero.test:
New tests, checking how our driver fares in recognizing numbers
with leading zeroes in TAP results or TAP plans.
* tests/tap-planskip-malformed.test: New test, checking that a
malformed TAP plan is not recognized as a valid plan.
* tests/tap-plan-whitespace.test: New test, checking that a TAP
plan line with trailing whitespace is recognized and handled
correctly.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
* tests/seenc.test: Make grepping of automake stderr stricter.
Add a trailing `:' command.
* tests/symlink.test: Likewise. Also, define `$am_create_testdir'
to "empty" to avoid bringing in unused auxiliary files (only to
have to remove them right away), and use proper m4 quoting in
`configure.in'.
* tests/vpath.test: Make grepping of generated `Makefile.in'
slightly stricter. Prefer trailing `:' over trailing `Exit 0'.
* tests/suffix6c.test: Unset OBJEXT to avoid interferences from
the environment.
* tests/suffix12.test: Do not whitespace-indent `##' comments
when they are embedded in a makefile rule: having them indented
is not part of the Automake API, and might cause failures with
e.g., Tru64 make.
* tests/syntax.test: Simplify the `Makefile.am' to ensure that
automake doesn't fail for the wrong reasons. Make grepping of
automake stderr slightly stricter.
* tests/test-harness-vpath-rewrite.test: Remove useless variable
definition from `Makefile.am'.
* tests/test-driver-custom-multitest.test: Fix typo in comments.
* tests/self-check-me.tap: Fix minor typo in test description.
* tests/vars3.test: Make grepping of automake stderr stricter
and safer.
* tests/version6.test: Add sanity check, verifying that the
version number extracted from `automake --version' output seems
legit.
* tests/auxdir2.test: Renamed ...
* tests/auxdir-compauted.tap: ... to this, and converted to the
use of TAP.
* tests/auxdir4.test: Renamed ...
* tests/auxdir-unportable.tap: ... to this, and converted to the
use of TAP.
* tests/auxdir3.test: Renamed ...
* tests/auxdir-misplaced.test: ... to this.
* tests/auxdir5.test: Renamed ...
* tests/auxdir-nonexistent.test: ... to this.
* tests/auxdir9.test: Renamed ...
* tests/auxdir-autodetect.test: ... to this.
* tests/Makefile.am (TESTS): Update.
(XFAIL_TESTS): Remove `auxdir2.test'.
* test-protocols:
testsuite: fix some redundant autotools calls in tests on TAP
* THANKS (Daniel Richard G.): Update e-mail address.
java: avoid compilation errors when CLASSPATH is empty
parallel-tests: no more spurious successes for FreeBSD make
tap: improve granularity for tests on problematic TAP messages
tap: correctly handle string "0" in TAP messages
tap: a minor simplification in the perl TAP driver
parallel-tests: fix help screen for test driver scripts
tap: add a dummy TAP driver script implemented in shell + awk
testsuite: refactor tests on TAP support in view of future changes
fix: regenerate tests/Makefile.in
hacking: we now require autoconf 2.68
testsuite: fix some redundant autotools calls in tests on TAP
* tests/tap-deps.test: Do not call autotools and configure; there
is no need to, since this script sources `tap-setup.sh'.
* tests/tap-empty-diagnostic.test: Likewise.
* tests/tap-escape-directive.test: Likewise.
* tests/tap-fancy2.test: Likewise.
* tests/tap-no-spurious.test: Likewise.
* tests/tap-out-of-order.test: Likewise.
* master:
* THANKS (Daniel Richard G.): Update e-mail address.
java: avoid compilation errors when CLASSPATH is empty
parallel-tests: no more spurious successes for FreeBSD make
hacking: we now require autoconf 2.68
* maint:
* THANKS (Daniel Richard G.): Update e-mail address.
java: avoid compilation errors when CLASSPATH is empty
parallel-tests: no more spurious successes for FreeBSD make
hacking: we now require autoconf 2.68
Daniel Richard G [Tue, 16 Aug 2011 13:19:14 +0000 (15:19 +0200)]
java: avoid compilation errors when CLASSPATH is empty
* lib/am/java.am (CLASSPATH_ENV): When redefining `$CLASSPATH',
do not append an empty component in case the previous value of
CLASSPATH is empty or unset.
* tests/java-empty-classpath.test: New test.
* tests/Makefile.am (TESTS): Update.
parallel-tests: no more spurious successes for FreeBSD make
Work around a bug of FreeBSD make bug that was causing the
automake-generated "check" target to complete with success
even if some tests failed; this happened only when FreeBSD
make was run in concurrent mode (as in, e.g., "make -j2
check"). The bug is not present in NetBSD make.
This change fixes automake bug#9245:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245>
See also FreeBSD PR bin/159730:
<http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>
* lib/am/check.am [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Use a
more "safe" (and apparently redundant) idiom to exit with error,
so that the non-zero exit status is picked up also by FreeBSD
make when it's running in concurrent mode.
* NEWS: Update.
* tests/check-concurrency-bug9245.test: New test.
* tests/Makefile.am (TESTS): Update.
tap: improve granularity for tests on problematic TAP messages
* tests/tap-message-0.test: Break up into ...
* tests/tap-msg0-result.test, tests/tap-msg0-directive.test,
tests/tap-msg0-planskip.test, tests/tap-msg0-misc.test,
tests/tap-msg0-bailout.test: ... these new tests, and extend
a little.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
* lib/tap-driver.pl (is_null_string): New function, can be used
to determine whether a given string variable is empty or undefined.
Useful to avoid pitfalls like:
if ($message) { print "$message\n"; }
which wouldn't print anything if $message is the literal "0".
(handle_tap_test, handle_tap_plan, handle_tap_bailout): Use it,
to avoid missing messages composed only by a literal "0" in TAP
result descriptions and in skip, todo and bailout explanations.
* tests/tap-message-0.test: Enhance.
* tests/Makefile.am (XFAIL_TESTS): Remove it, it passes now.
tap: add a dummy TAP driver script implemented in shell + awk
The user can also now decide which implementation of the TAP driver
to use in the testsuite by defining the `$am_tap_implementation'
variable to either "perl" or "shell". Future enhancements will
allow the testsuite to automatically run the test scripts on TAP
support with both the TAP driver implementations, to improve
coverage.
* tests/defs (fetch_tap_driver): Honor the `$am_tap_implementation'
variable to decide which implementation of the TAP driver to fetch.
($am_tap_implementation): Default to "perl".
* tests/tap-common-setup.test: Do not fetch the TAP driver, the
code in tap-setup.sh does that already (and respecting runtime
overriding of `$am_tap_implementation').
* lib/tap-driver: Renamed ...
* lib/tap-driver.pl: ... to this, and ...
($ME): ... adjusted this.
* doc/automake.texi: Adjust to the renaming.
* tests/Makefile.am (TAP_LOG_DRIVER): Likewise.
* tests/tap-doc2.test: Likewise.
* lib/tap-driver.sh: New script, still mostly dummy.
* lib/Makefile.am (dist_script_DATA): Update, and since we are at
it, rewrite it to make it easier to add new entries in the future.
testsuite: refactor tests on TAP support in view of future changes
* tests/defs (fetch_tap_driver): New subroutine; it fetches the
automake-provided TAP driver from the `lib/' directory into the
current directory, and edits its shebang line so that it will be
run with the perl interpreter determined at configure time.
* tests/tap-setup.sh: Use it.
* tests/tap-common-setup.test: There's no need to AC_SUBST `PERL'
anymore, nor to use it in the Makefile to run the TAP driver.
Also, use the `fetch_tap_driver' function instead of copying the
`tap-driver' auxiliary script directly.
* tests/tap-bad-prog.tap: Likewise.
* tests/tap-diagnostic-custom.test: Likewise.
* tests/tap-doc.test: Likewise.
* tests/tap-merge-stdout-stderr.test: Likewise.
* tests/tap-more.test: Likewise.
* tests/tap-more2.test: Likewise.
* tests/tap-recheck.test: Likewise.
* tests/tap-summary-aux.sh: Likewise.
* tests/tap-basic.test: Likewise, and fix a grammaro in comments
since we are at it.
* test-protocols:
coverage: missing tap plan and non-zero exit status
testsuite: fix an unportable use of sed
testsuite: avoid spurious failure if rst2html program is missing
fix: two recent ChangeLog entries
tap: improve diagnosing and reporting of plan mismatches
tap: log all TAP stream, even after a "Bail out!"
coverage: TAP diagnostics after "Bail out!" aren't reported
coverage: TAP errors after a "Bail out!" directive aren't reported
testsuite: reorganize tests on TAP plans
coverage: more TAP driver consistency w.r.t. 'prove'
tap: improve some comments in the TAP driver
tap: non-zero exit status after "Bail out!" should not be reported
tap: a "plan with SKIP" after test results is an error
test harness: avoid possible fork bomb
coverage: possible infinite recursion in the test harness
coverage: possible infinite recursion in the test harness
test defs: yet more environment cleanup
* tests/self-check-exit.tap: Disable the `errexit' shell flag;
this allow to remove some boilerplate idioms and to make some
code slightly less cumbersome.
* tests/self-check-me.tap: Likewise.
maintcheck: look for TAP-based tests without TAP plan
* Makefile.am (sc_tests_tap_plan): New maintainer check.
(syntax_check_rules): Add it.
* tests/suffix10.tap: Declare a TAP plan.
* tests/add-missing.tap: Likewise.
testsuite: avoid confusing SKIP messages for errored tests
* tests/defs [$use_tap] (exit trap): Don't print the test plan
in case the client script exited with a non-zero status. This
avoids the display of redundant "SKIP" messages for tests with
lazy plans that error out before any test is run.
testsuite: keep more debugging info around in 'add-missing.tap'
* tests/add-missing.tap: Use a different directory for each test,
so that in case of failure more information is kept around.
Since we are at it, remove redundant TAP diagnostic, and define
`$am_create_testdir' to "empty" to avoid bringing in unused
auxiliary files.
testsuite: more granular count of test results in our TAP library
* tests/tap-functions.sh ($tap_bad_count_): Removed, superseded
by ...
($tap_fail_count_, $tap_xpass_count_): ... these new variables,
which keep more granular counts.
($tap_pass_count_: New variable.
* tests/defs (exit trap): Adjust and simplify accordingly.
testsuite: fixlets and improvements in two long TAP-based tests
* tests/depmod.tap: Clean up the subdirectories used by tests that
passed, to avoid ending up with a too big test directory. This is
especially important since, on each and every system, at least some
tests in this script are expected to be skipped (which ones exactly
depends on the system), thus causing the test directory not to be
removed when the script terminates.
* tests/instspc.tap: Likewise (with the difference that tests here
are not expected to be skipped, but to xfail). Give more explicit
error message in case of setup failure. Skip the proper number of
tests when required, to avoid spurious errors due to mismatched
plan. Also, really avoid to run the unsupported test cases when
the need to skip them is detected.
tap: improve diagnosing and reporting of plan mismatches
Problem exposed by a failure in the test 'tap-plan-errors.test'.
* lib/tap-driver (NO_PLAN, EARLY_PLAN, LATE_PLAN): New constants.
(Throughout the file): Use more complex semantics for `$plan_seen',
which now also remember whether the plan that has been seen is an
"early" or "late" plan; in turn, this makes ...
($tap_stopped): ... this variable redundant; remove it.
(handle_tap_plan): Adjust to avoid producing spurious or confusing
error messages about extra or mismatched "late" TAP plans.
* tests/tap-plan-corned.test: Adjust and extend.
* tests/tap-color.test: Likewise.
* lib/tap-driver ($bailed_out): New global boolean variable,
telling whether a "Bail out!" directive has been seen or not.
(handle_tap_bailout): This function does not anymore stop the
reading from TAP stream; instead, it sets `$bailed_out' to a
true value, so that only the subsequent parsing of the input
TAP stream is stopped.
(finish): Remove, no more needed, its contents inlined into ...
(main): ... this function, with related adjustments in the code
flow.
(get_test_exit_message): Do not "flush" the input TAP stream
to fetch the exit status of test script, it is not anymore
required. Add a sanity check.
* tests/tap-bailout-and-logging.test: New test.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
* tests/tap-skip-whole.test: Rename ...
* tests/tap-planskip.test: ... to this
* tests/tap-skip-whole-badexit.test: Rename ...
* tests/tap-planskip-badexit.test: ... to this.
* tests/tap-skip-whole-bailout.test: Rename ...
* tests/tap-planskip-bailout.test: ... to this.
* tests/tap-skip-whole-bailout.test: Rename ...
* tests/tap-planskip-later-errors.test: ... to this.
* tests/tap-skip-whole-unplanned.test: Rename ...
* tests/tap-planskip-unplanned.test: ... to this.
* tests/tap-skip-whole-whitespace.test: Rename ...
* tests/tap-planskip-whitespace.test: ... to this.
* tests/tap-skip-whole-continue.test: Rename ...
* tests/tap-planskip-and-logging.test: ... to this, extend to try
more cases, and fix heading comments.
* tests/tap-skip-whole-lastline.test: Rename ...
* tests/tap-planskip-late.test: ... to this, fix heading comments,
and remove some checks that are just a duplication of those already
performed in the previous script.
* tests/tap-results-with-planskip.test: Rename to ...
* tests/tap-planskip-unplanned-corner.test: ... this, and improve
hading comments.
* tests/tap-planskip-later-errors.test: New test.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
coverage: more TAP driver consistency w.r.t. 'prove'
* tests/skip-results-with-planskip.test: New test, verifies that
test results seen in a TAP stream that has a "plan with SKIP" are
flagged as errors.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
tap: non-zero exit status after "Bail out!" should not be reported
This new behaviour is more consistent with that of the `prove'
utility. Also, experience with the use of TAP in the Automake's
own test suite has shown that reporting non-zero exit status from
a script after it has issued a "Bail out!" directive is mostly
redundant, more introducing noise rather than conveying useful
information.
* lib/tap-driver (finish): Inline the part of it that was getting
the script exit status ...
(main): ... in here.
* tests/tap-bailout-badexit.test: New test.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
tap: a "plan with SKIP" after test results is an error
This new behaviour is both more natural and more consistent with
that of the `prove' utility.
* lib/tap-driver (handle_tap_plan): Do not stop TAP parsing when
a "plan with SKIP" line (e.g., "1..0 # SKIP") is encountered, and
do not print a "SKIP" line if some TAP result has already been
seen.
* tests/tap-skip-whole.test: Adapt.
* tests/tap-skip-whole-lastline.test: Likewise.
* tests/tap-global-result.test: Adapt and extend.
* tests/tap-skip-plan-errors.test: Likewise, and fix an obsolete
small part of a comment.
* tests/tap-skip-whole-bailout.test: New test.
* tests/tap-skip-whole-unplanned.test: Likewise.
* tests/tap-skip-whole-continue.test: Likewise.
* tests/Makefile.am (tap_with_common_setup_tests): Update.
This fixes a regression w.r.t. the master branch, exposed by
test 'parallel-tests-fork-bomb.test'.
* lib/am/check.am (am--redo-logs): Detect possible infinite
recursion due to a test log in $(TEST_LOGS) being the same
as $(TEST_SUITE_LOG).
* tests/parallel-tests-fork-bomb.test: Enhance and extend a
little now that we have more explicit semantics.
coverage: possible infinite recursion in the test harness
Motivated by a regression in the 'test-protocols' branch.
* tests/parallel-tests-fork-bomb.test: New test, checking that
if $(TEST_SUITE_LOG) is in $(TEST_LOGS), we obtain a diagnosed
error rather than a make hang or a fork bomb.
* tests/Makefile.am (TESTS): Update.
coverage: possible infinite recursion in the test harness
Motivated by a regression in the 'test-protocols' branch.
* tests/parallel-tests-fork-bomb.test: New test, checking that
if $(TEST_SUITE_LOG) is in $(TEST_LOGS), we obtain a diagnosed
error rather than a make hang or a fork bomb.
* tests/Makefile.am (TESTS): Update.
tests: fix various blunders in 'suffix-chain.test'
* tests/suffix-chain.test: Fix various blunders that were causing
this test to fail spuriously: append to `configure.in', not to
`configure.ac'; add a `;' character after the `return' statement
in an input file; use tabs, not spaces, to indent Makefile recipes;
and use more uncommon suffixes that don't trigger built-in recipes
with FreeBSD make.
testsuite: improvements to our shell TAP library and APIs
These improvements impact only the Automake's own testsuite.
* tests/tap-functions.sh (plan_): Allow new special arguments
"unknown" a.k.a. "later" and "lazy" a.k.a. "now". Perform
sanity check the given arguments. Adjust incorrect comments.
(lazy_plan_): Removed, superseded by the new version of `plan_'.
(not): New function, run a command and inverts its exit status.
(ok_, not_ok_, skip_): These are just shorthands for common
`result_' usages.
(xfail_, xpass_, reset_test_count_): Removed, they are not used,
nor does a legitimate use for them seem in sight.
(not): New function, to run a command inverting its exit status.
(command_not_ok_): Removed, superseded by `command_ok_' + `not'.
(command_ok_): Extended to accept TAP directives.
* tests/defs (trap): Don't use `late_plan_' anymore, use
`plan_ "now"' instead.
* tests/ac-output-old.tap: Update to take advantage of the new
APIs.
* tests/self-check-env-sanitize.tap: Likewise.
* tests/suffix-chain.tap: Likewise.
* tests/suffix10.tap: Likewise.
* tests/suffix11.tap: Likewise.
* tests/suffix3.tap: Likewise.
* tests/suffix8.tap: Likewise.
* tests/tap-bad-prog.tap: Likewise.
* tests/self-check-exit.tap: Likewise, plus a small improvement
in the name of a variable.
This will allow us to explicitly declare some expected failures
involving Solaris make as such, instead of partly papering over
them with skips and partly letting them fail too loudly.
* tests/suffix-chain.test: Rename ...
* tests/suffix-chain.tap: ... to this, and convert to the use of
TAP. Also, fix some bugs since we are at it, and ...
($required): ... require `cc'.
* tests/suffix3.test: Rename ...
* tests/suffix3.tap: ... to this, and convert to the use of TAP.
Also, since we are at it, throw in some formatting fixes.
* tests/suffix8.test: Rename ...
* tests/suffix8.tap: ... to this, and convert to the use of TAP.
Also, since we are at it, throw in some formatting fixes.
* tests/suffix10.test: Rename ...
* tests/suffix10.tap: ... to this, and convert to the use of TAP.
* tests/suffix11.test: Rename ...
* tests/suffix11.tap: ... to this, and convert to the use of TAP.
* tests/Makefile.am (TESTS): Update.
tests: don't let a known Solaris make bug poison too many tests
See automake bug#7670 and bug#7824.
* tests/defs (make_can_chain_suffix_rules): New function, tell
whether the make implementation in use can chain suffix rules
automatically or not.
* tests/suffix3.test: Use it to avoid issuing calls to make that
are unportable to make implementations that are not smart enough
to chain suffix rules automatically.
* tests/suffix8.test: Use it to avoid issuing calls to make that
* tests/suffix10.test: Use it to avoid issuing calls to make that
* tests/suffix11.test: Use it to avoid issuing calls to make that
* tests/suffix-chain.test: New test, exposes the limitation that
we have papered over in the tests above.
* .gitignore: Anchor files that are intended to be ignored only
if found in the same directory of the `.gitignore' file, not also
in its subdirectories.
* doc/.gitignore, doc/amhello/.gitignore, lib/Automake/.gitignore,
lib/Automake/tests/.gitignore, tests/.gitignore: Likewise. Also,
where needed, add new entries that were once implied by the
non-anchored entries in the upper-level `.gitignore' files.
testsuite: avoid spurious failure of self checks on Solaris
This is just a quick workaround to prevent having too much noise
in the testsuite; more proper fixes have been committed into the
'testsuite-work' branch.
* tests/self-check-dir.test: Relax a test to avoid unimportant
failure on Solaris 10.
* tests/self-check-exit.test: Likewise.
projects / thirdparty / automake.git / log
