+ 2011-05-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: each test case depends on `defs-static'
+ * tests/Makefile.am ($(TEST_LOGS)): Depends on `defs-static' too.
+ Simplify comments.
+
+2011-05-27 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: use $SHELL to run tests which are shell scripts
+ * tests/Makefile.am (TEST_LOG_COMPILER): Define so that the
+ configure-time $SHELL is used to run the tests.
+ * tests/defs: Add code to re-execute by default the running
+ test script with configure-time $SHELL. Updated comments.
+ * configure.ac: Check that `set -e' is working for $SHELL,
+ not for /bin/sh.
+ * tests/defs-static.in: Update comments.
+ * tests/README (Supported shells): Updated.
+ (Getting details from failures): Don't tell that tests are
+ run by /bin/sh by default.
+
+2011-05-26 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: split 'subst2.test' to improve modularity and coverage
+ * tests/subst2.test: Test removed, split into ...
+ * tests/programs-primary-rewritten.test: ... this test ...
+ * tests/subst-no-trailing-empty-line.test: ... and this one ...
+ * tests/extra-programs-empty.test: ... and this one.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-05-26 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: tweak and improve tests on Automake conditionals
+ The "avoid the requirement of a working compiler" we refer about
+ below is obtained by disabling automatic dependency tracking and
+ defining `CC' (and `EXEEXT', `OBJEXT', `CXX', etc., if needed) to
+ dummy values, either directly in the `Makefile.am', or by AC_SUBST
+ in `configure.in'.
+ The "cosmetic changes" we refer about below consists in, e.g.,
+ adding a trailing `:' command to a script, using proper m4 quoting
+ in `configure.in' files, adding commands that offer more debugging
+ output, explicitly declaring phony targets as such in Makefiles,
+ making use of idioms more consistent with those employed in other
+ tests, following the GNU formatting standards more closely, etc.
+ * tests/cond.test: Cosmetic changes.
+ * tests/cond3.test: Likewise.
+ * tests/cond10.test: Likewise.
+ * tests/cond14.test: Likewise.
+ * tests/cond15.test: Likewise.
+ * tests/cond17.test: Likewise.
+ * tests/cond40.test: Likewise.
+ * tests/cond13.test: Likewise, and make grepping of the generated
+ `Makefile.in' slightly stricter.
+ * tests/cond6.test: Extend by also running ./configure, make and
+ "make install". Few cosmetic changes.
+ * tests/cond8.test: Likewise.
+ * tests/cond2.test: Do few cosmetic changes. Make grepping of
+ automake stderr stricter.
+ * tests/cond20.test: Likewise.
+ * tests/cond23.test: Likewise.
+ * tests/cond24.test: Likewise.
+ * tests/cond42.test: Likewise.
+ * tests/cond46.test: Likewise.
+ * tests/cond9.test: Move more checks in the `Makefile.am' instead
+ of grepping make's output.
+ * tests/cond38.test: Likewise.
+ * tests/cond11.test: Likewise. Avoid the requirement of a working
+ C compiler.
+ * tests/cond16.test: Likewise.
+ * tests/cond22.test: Likewise.
+ * tests/cond30.test: Likewise.
+ * tests/cond31.test: Likewise.
+ * tests/cond4.test: Likewise, and drop the now-useless GNU make
+ requirement as well.
+ * tests/cond18.test: Likewise.
+ * tests/cond19.test: Likewise.
+ * tests/cond22.test: Likewise.
+ * tests/cond25.test: Add trailing `:' command.
+ * tests/cond26.test: Likewise.
+ * tests/cond27.test: Likewise.
+ * tests/cond28.test: Likewise.
+ * tests/cond29.test: Use `unindent' function for better formatting.
+ Other minor cosmetic changes. Use a `timeout' program (if it's
+ available) to determine whether the script takes too long, instead
+ of just hoping that the user will notice an abnormally long test
+ execution time.
+ * tests/cond33.test: Extend a bit. Fix heading comments. Few more
+ cosmetic changes.
+ * tests/cond37.test: Ensure verbose printing of captured make
+ output. Minor cosmetic changes.
+ * tests/cond34.test: Likewise. Avoid the requirement of a working
+ C compiler. Make grepping of make output slightly stricter.
+ * tests/cond35.test: Quote literal dots in grep regexp. Add extra
+ debugging output. Minor cosmetic changes.
+ * tests/cond36.test: Likewise.
+ * tests/cond41.test: Do not simply check that aclocal fails, but
+ also grep its stderr for the expected error message.
+ * tests/cond43.test: Likewise (but the program being automake).
+ Few cosmetic changes.
+ * tests/condd.test: Avoid the requirement of a working C compiler.
+ * tests/condman3.test: Extend by using more man pages and more
+ manpage sections.
+ * tests/condman.test: Removed, it's completely superseded by
+ `condman3.test'.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-05-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ lex tests: avoid spurious failures when LEXLIB isn't found
+ The AC_PROG_LEX Autoconf macro does not diagnose a failure to find
+ the "lex library" expected to provide a `yywrap' function (function
+ which is required to link most lex-generated programs). On the
+ contrary, when all the link attempts (i.e., with `-ll' and `-lfl')
+ fail, configure declares that no lex library is needed, and simply
+ proceeds with the configuration process -- only for the build to
+ possibly fail later, at make time.
+ This behaviour might be (partly) intended; the Autoconf manual
+ reads:
+ ``You are encouraged to use Flex in your sources, since it is
+ both more pleasant to use than plain Lex and the C source
+ it produces is portable. In order to ensure portability,
+ however, you must either provide a function `yywrap' or, if
+ you don't use it (e.g., your scanner has no `#include'-like
+ feature), simply include a `%noyywrap' statement in the
+ scanner's source.''
+ This AC_PROG_LEX behaviour is causing some spurious failures of the
+ Automake testsuite in environments which lack a proper library
+ providing `yywrap' (this happens for example in Linux->MinGW cross
+ compilations). But at this point is clear that a proper workaround
+ is to simply provide a fall-back implementation of `yywrap' in our
+ lexers.
+ * tests/cond35.test: Provide a dummy `yywrap' function.
+ * tests/lex3.test: Likewise.
+ * tests/lexvpath.test: Likewise.
+ * tests/silent-many-gcc.test: Likewise.
+ * tests/silent-many-generic.test: Likewise.
+ * tests/silent-lex-gcc.test: Likewise, and a dummy `main' too.
+ * tests/silent-lex-generic.test: Likewise.
+ * tests/lex-lib.test: New test.
+ * tests/lex-libobj.test: New test.
+ * tests/lex-nowrap.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+ * THANKS: Update.
+ Thanks to Russ Allbery for the suggestion.
+
+2011-05-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: require C++ compiler explicitly in tests needing it
+ The list of the test scripts which needed a C++ compiler but
+ failed to require it explicitly has been found by running:
+ $ tests=`grep -lE 'CXX|\.c(c|\+\+|xx|pp)' *.test | tr '\n' ' '`
+ $ am_explicit_skips=yes CXX=false make check TESTS="$tests"
+ and looking for tests that reported FAIL instead of SKIP.
+ After the present change, all those previously failing tests
+ either pass or get correctly skipped.
+ * tests/subobj9.test: Require `c++', instead of explicitly
+ skipping on configure failure. Add excerpts and/or details from
+ the original bug report that prompted this tests to be written.
+ * tests/silentcxx-gcc.test: Require g++.
+ * tests/silentcxx.test: Require c++.
+ * tests/suffix3.test: Likewise.
+
+2011-05-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: don't require g++ where any C++ compiler is enough
+ * tests/specflg10.test ($required): Use `c++', not g++.
+ * tests/silent-many-generic.test: Likewise.
+
+2011-05-25 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: require C compiler explicitly in tests needing it
+ The list of the test scripts which needed a C compiler but
+ failed to require it explicitly has been found by running:
+ $ am_explicit_skips=yes CC=false make check
+ and looking for tests that reported FAIL instead of SKIP.
+ After the present change, all those previously failing tests
+ either pass or get correctly skipped.
+ * tests/aclocal4.test ($required): Add `cc'.
+ * tests/ansi.test: Likewise.
+ * tests/ansi10.test: Likewise.
+ * tests/ansi3.test: Likewise.
+ * tests/ansi3b.test: Likewise.
+ * tests/ansi5.test: Likewise.
+ * tests/ansi6.test: Likewise.
+ * tests/ansi7.test: Likewise.
+ * tests/ansi9.test: Likewise.
+ * tests/backcompat6.test: Likewise.
+ * tests/canon7.test: Likewise.
+ * tests/check5.test: Likewise.
+ * tests/check7.test: Likewise.
+ * tests/check8.test: Likewise.
+ * tests/cond4.test: Likewise.
+ * tests/cond11.test: Likewise.
+ * tests/cond16.test: Likewise.
+ * tests/cond18.test: Likewise.
+ * tests/cond19.test: Likewise.
+ * tests/cond22.test: Likewise.
+ * tests/cond30.test: Likewise.
+ * tests/cond31.test: Likewise.
+ * tests/cond32.test: Likewise.
+ * tests/cond34.test: Likewise.
+ * tests/cond35.test: Likewise.
+ * tests/cond39.test: Likewise.
+ * tests/condd.test: Likewise.
+ * tests/cygnus-dependency-tracking.test: Likewise.
+ * tests/depcomp2.test: Likewise.
+ * tests/depcomp6.test: Likewise.
+ * tests/depcomp7.test: Likewise.
+ * tests/depcomp8a.test: Likewise.
+ * tests/depcomp8b.test: Likewise.
+ * tests/depdist.test: Likewise.
+ * tests/depend2.test: Likewise.
+ * tests/depend5.test: Likewise.
+ * tests/distcleancheck.test: Likewise.
+ * tests/distname.test: Likewise.
+ * tests/exeext.test: Likewise.
+ * tests/exeext4.test: Likewise.
+ * tests/extradep.test: Likewise.
+ * tests/extradep2.test: Likewise.
+ * tests/gnits2.test: Likewise.
+ * tests/gnits3.test: Likewise.
+ * tests/instdir-ltlib.test: Likewise.
+ * tests/instdir-prog.test: Likewise.
+ * tests/instfail.test: Likewise.
+ * tests/instfail-libtool.test: Likewise.
+ * tests/lex3.test: Likewise.
+ * tests/lex5.test: Likewise.
+ * tests/lexvpath.test: Likewise.
+ * tests/lex-subobj-nodep.test: Likewise.
+ * tests/lflags.test: Likewise.
+ * tests/libobj-basic.test: Likewise.
+ * tests/libobj2.test: Likewise.
+ * tests/libobj7.test: Likewise.
+ * tests/libobj10.test: Likewise.
+ * tests/libobj16a.test: Likewise.
+ * tests/libobj16b.test: Likewise.
+ * tests/libobj17.test: Likewise.
+ * tests/libobj19.test: Likewise.
+ * tests/libtool2.test: Likewise.
+ * tests/libtool3.test: Likewise.
+ * tests/libtool7.test: Likewise.
+ * tests/libtool9.test: Likewise.
+ * tests/libtoo10.test: Likewise.
+ * tests/libtoo11.test: Likewise.
+ * tests/ltcond.test: Likewise.
+ * tests/ltcond2.test: Likewise.
+ * tests/ltconv.test: Likewise.
+ * tests/ltinit.test: Likewise.
+ * tests/ltlibsrc.test: Likewise.
+ * tests/ltorder.test: Likewise.
+ * tests/nobase.test: Likewise.
+ * tests/nobase-libtool.test: Likewise.
+ * tests/mmodely.test: Likewise.
+ * tests/parallel-tests5.test: Likewise.
+ * tests/parallel-tests-suffix-prog.test: Likewise.
+ * tests/parallel-tests-ext-driver-prog.test: Likewise.
+ * tests/posixsubst-ldadd.test: Likewise.
+ * tests/posixsubst-libraries.test: Likewise.
+ * tests/posixsubst-ltlibraries.test: Likewise.
+ * tests/posixsubst-programs.test: Likewise.
+ * tests/posixsubst-sources.test: Likewise.
+ * tests/pr87.test: Likewise.
+ * tests/pr204.test: Likewise.
+ * tests/pr224.test: Likewise.
+ * tests/pr300-lib.test: Likewise.
+ * tests/pr300-ltlib.test: Likewise.
+ * tests/pr300-prog.test: Likewise.
+ * tests/pr401.test: Likewise.
+ * tests/pr401b.test: Likewise.
+ * tests/pr401c.test: Likewise.
+ * tests/remake-gnulib-add-acsubst.test: Likewise.
+ * tests/remake-gnulib-add-header.test: Likewise.
+ * tests/regex.test: Likewise.
+ * tests/repeated-options.test: Likewise.
+ * tests/silent.test: Likewise.
+ * tests/silent3.test: Likewise.
+ * tests/silent9.test: Likewise.
+ * tests/silent-lex-generic.test: Likewise.
+ * tests/silent-many-generic.test: Likewise.
+ * tests/silent-yacc-generic.test: Likewise.
+ * tests/specflg7.test: Likewise.
+ * tests/specflg9.test: Likewise.
+ * tests/specflg10.test: Likewise.
+ * tests/stdinc.test: Likewise.
+ * tests/strip.test: Likewise.
+ * tests/strip2.test: Likewise.
+ * tests/strip3.test: Likewise.
+ * tests/subdirbuiltsources.test: Likewise.
+ * tests/subobj3.test: Likewise.
+ * tests/subobj6.test: Likewise.
+ * tests/subobj11a.test: Likewise.
+ * tests/subpkg.test: Likewise.
+ * tests/subst2.test: Likewise.
+ * tests/subst3.test: Likewise.
+ * tests/substref.test: Likewise.
+ * tests/substre2.test: Likewise.
+ * tests/suffix5.test: Likewise.
+ * tests/suffix8.test: Likewise.
+ * tests/suffix10.test: Likewise.
+ * tests/suffix11.test: Likewise.
+ * tests/suffix12.test: Likewise.
+ * tests/suffix13.test: Likewise.
+ * tests/target-cflags.test: Likewise.
+ * tests/transform.test: Likewise.
+ * tests/transform2.test: Likewise.
+ * tests/yacc-basic.test: Likewise.
+ * tests/yacc-d-basic.test: Likewise.
+ * tests/yacc-clean.test: Likewise.
+ * tests/yacc-dist-nobuild.test: Likewise.
+ * tests/yacc-nodist.test: Likewise.
+ * tests/yaccvpath.test: Likewise.
+ * tests/yacc-d-vpath.test: Likewise.
+ * tests/yacc4.test: Likewise.
+ * tests/yacc7.test: Likewise.
+ * tests/yacc8.test: Likewise.
+ * tests/yaccdry.test: Likewise.
+ * tests/yflags-cmdline-override.test: Likewise.
+ * tests/yflags-force-override.test: Likewise.
+ * tests/python-virtualenv.test: Likewise. Also, improve skip
+ messages.
+ * tests/subobj5.test ($required): Add `cc'.
+ (Makefile.am): Set `AUTOMAKE_OPTIONS' to `subdir-objects', and
+ add new checking rules `test-build' and `test-distdir'.
+ Extend the test by building and examining the distdir, the
+ program, and the object files.
+ * tests/postproc.test ($required): Add `cc'.
+ Avoid the explicit `|| Exit $?' after call to configure, which
+ is now either redundant (e.g., when `am_explicit_skips' is
+ unset), or counter-productive (e.g., when `am_explicit_skips'
+ is set to "yes").
+ * tests/pr243.test: Likewise.
+ * tests/pr266.test: Likewise.
+ * tests/pr220.test: Simplify so that it doesn't require a
+ C compiler anymore.
+ * tests/subdir5.test: Likewise.
+ * tests/subdir8.test: Likewise.
+ * tests/lflags.test: Likewise.
+ * tests/yflags.test: Likewise.
+ * tests/yflags-force-conditional.test: Likewise.
+ * tests/lflags2.test: Simplify so that it doesn't require a
+ C++ compiler anymore.
+ * tests/yflags2.test: Likewise.
+ * tests/autohdrdry.test (configure.in): Remove unneeded call
+ to `AC_PROG_CC'.
+ * tests/pr287.test: Likewise.
+ * tests/check6.test: Likewise.
+ * tests/cond21.test: Likewise, plus some cosmetic adjustments.
+ * tests/upc.test: Skip if configure fails with status `77'.
+ * tests/upc3.test: Likewise.
+ * tests/vala4.test: Likewise.
+ * tests/nostdinc.test: Likewise. Also, make grepping checks
+ on `Makefile.in'. Update heading comments.
+ * tests/compile5.test: When we must skip, skip explicitly,
+ and with a meaningful message.
+ * tests/instspc-tests.sh: When running in "test-build" or
+ "test-install" mode (as determined by the value of variable
+ `$instspc_action'), require `cc'. Adjust comments.
+ * tests/depmod-tests.sh: Likewise.
+
+2011-05-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: allow user to ask for stricter skip semantics
+ Given how the Automake testsuite is currently structured, if a
+ command in a test script fails with status `77', the global test
+ outcome is considered a SKIP, because the value of `77' for `$?'
+ is passed to the exit trap. Indeed, this happens in practice, as
+ an autoconf-generated configure script can exit with status `77'
+ if it fails to find, e.g., a required compiler.
+ While this behaviour is quite useful for avoiding spurious test
+ failures in the wild, it can also sometimes prevent the Automake
+ developers to easily see and declare the requirements of their
+ tests.
+ This change introduces a new variable `am_explicit_skips', meant to
+ be user-overridable, and which, when set to a "true" value (i.e.,
+ `yes' or `1'), require a direct call to `Exit 77' in order to make
+ the test outcome be considered a SKIP.
+ * tests/defs.in ($am__test_skipped): New variable, initialized
+ to `no'.
+ (Exit): Set `$am__test_skipped' to `yes' if passed an exit status
+ of 77.
+ (trap '...' 0): When `$am_explicit_skips' is set to a "true" value,
+ reset an exit status of `77' to `78' if $am__test_skipped is not
+ set to `yes'.
+ * tests/self-check-exit.test: Adjust: unset `am_explicit_skips'.
+ * tests/self-check-explicit-skips.test: New test.
+ * tests/Makefile.am (TESTS): Update.
+
+2011-05-24 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ testsuite: user can force skipping of compiler-requiring tests
+ * tests/defs (cc, c++, fortran, fortran77): Skip the test if
+ the relevant compiler is disabled by having the corresponding
+ variable (CC, CXX, FC and F77, respectively) set to "false".
+ (yacc): For consistency, skip the test when the YACC variable
+ is set to "false", not when it's set to "no". Since we are at
+ it, fix the skip message to be shorter and more consistent.
+
+2011-05-23 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests: tweak and improve tests on "Simple Tests" driver
+ * tests/check.test: Make grepping of generated Makefile.in
+ slightly stricter. Add trailing `:' command.
+ * tests/check2.test: Ensure verbose printing of captured make
+ output. Make grepping of generated Makefile.in and of make
+ output stricter. Add trailing `:' command.
+ * tests/check3.test: Likewise. Also, fix minor error in a
+ comment.
+ * tests/check4.test: Stricter grepping of make output. Use
+ `using_gmake' subroutine instead of copy&paste from defs.in
+ to determine whether $MAKE is GNU make. Add a trailing `:'
+ command.
+ * tests/check6.test: Add trailing `:' command.
+ * tests/check10.test: Prefer redirection `>' over `>>' when
+ creating new files. Ensure verbose printing of captured make
+ output.
+ * tests/check11.test: Likewise.
+ * tests/check7.test: Likewise. Make grepping of make output
+ stricter. Prefer "make distcheck" over a simple "make check"
+ in a VPATH build. Add trailing `:' command.
+ (a.c): Format following GNU coding standards.
+ (Makefile.am): Explicitly declare target `print-xfail-tests'
+ as PHONY. Distribute $(check_SCRIPTS), for distcheck.
+ * tests/parallel-tests.test: Prefer redirection `>' over `>>'
+ when creating new files.
+ * tests/parallel-tests2.test: Likewise.
+ * tests/parallel-tests9.test: Likewise.
+ * tests/parallel-tests8.test: Likewise. Make grepping of
+ automake stderr stricter.
+ * tests/check8.test: Likewise. Make formatting of created
+ *.c files more consistent with GNU Coding Standards.
+ * tests/check-subst-prog.test (configure.in): Break overly
+ long lines.
+ (pass-prog.c, xfail-prog.c): Format them more consistently
+ with GNU Coding Standards.
+ * tests/parallel-tests3.test: Use a shorter and clearer skip
+ message. Use the `unindent' subroutine to improve readability
+ of indented loop bodies. Simplify the hack used to please
+ maintainer-check.
+ * tests/parallel-tests5.test: Make formatting of created *.c
+ files more consistent with GNU Coding Standards. Use the
+ `unindent' subroutine to improve readability of indented loop
+ bodies.
+ * tests/parallel-tests10.test: Add blank line, for clarity.
+ Prefer trailing `:' over trailing `Exit 0', for consistency
+ with other tests.
+
+2011-05-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests/README: small consistency improvement
+ * tests/README (Section "User interface" subsection "Running the
+ tests"): In examples, be more consistent w.r.t. the differences
+ between GNU and non-GNU make.
+
+2011-05-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ tests/README: suggest how to run tests in cross-compile mode
+ * tests/README (Section "User interface" subsection "Running the
+ tests"): Briefly explain how to override 'host_alias' at runtime
+ to force the use of cross-compilers by the testsuite. Give an
+ example.
+
+2011-05-21 Stefano Lattarini <stefano.lattarini@gmail.com>
+
+ test defs: better detection of cross-compile mode
+ * configure.ac (AC_CANONICAL_HOST): New, probably not strictly
+ necessary, but useful to complements AC_CANONICAL_BUILD.
+ * tests/defs-static.in ($host_alias, $build_alias): New variables,
+ exported, user-overridable, defaults AC_SUBST'd by configure.
+ * tests/defs (cross_compiling): Consider cross-compilation mode
+ active only if `$host_alias' is not empty *and* it differs from
+ `$build_alias'. This is more faithful to the way configure
+ determines whether it is cross-compiling, but still allows the
+ user to easily force cross-compilation by overriding `host_alias'
+ and `build_alias' at runtime.
+ From a report by Ralf Wildenhues.
+
2011-05-20 Stefano Lattarini <stefano.lattarini@gmail.com>
testsuite: avoid re-running few tests with 'parallel-tests' option
projects / thirdparty / automake.git / commitdiff
