tests: more checks on portable fd redirection in TESTS_ENVIRONMENT
* tests/tests-environment-fd-redirect.test: Extend by also using
a perl script among the tests. Run the test shell script with
the `errexit' flag active. Export `VERBOSE' to yes when running
"make check", to give more debugging information in case of
failures. Look for a Korn Shell also in `/usr/bin', not on only
in `/bin'.
tests: check portable fd redirection in TESTS_ENVIRONMENT
* tests/tests-environment-fd-redirect.test: New test.
* tests/Makefile.am (TESTS): Update.
Motivated by coreutils bug#8846:
<http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8846>
See also following CC:ed thread on bug-autoconf list:
<http://lists.gnu.org/archive/html/bug-autoconf/2011-06/msg00002.html>
Before this patch, the only way offered by tests/defs to
properly signal a hard error was the `framework_failure_'
function. But the error message issued by that function,
as its name would suggest, refers to a set-up failure in the
testsuite, while hard errors can obviously also be due to
other reasons. The best way to fix this inconsistency is to
introduce a new function with a more general error message.
Inspired by a recent similar change to Gnulib's tests/init.sh.
* tests/defs.in (fatal_): New function.
* tests/README (Section "Writing test cases" subsection "Do"):
Suggest the use of `fatal_', not of `framework_failure_', for
generic hard errors. The latter should be reserved for "real"
set-up failures.
* tests/auxdir2.test (configure.in): Close m4 quoting in the
argument to AC_CONFIG_AUX_DIR. Without this, aclocal fails with
"ERROR: end of file in string". This problem hasn't been exposed
by the testsuite before because this test is in XFAIL_TESTS, so
its failure went unnoticed, even if it was due to a wrong cause.
* tests/instdir-ltlib.test: Use creative quoting to avoid
spuriously triggering the `sc_rm_minus_f' maintainer check.
* tests/instdir-prog.test: Likewise.
* tests/instspc-data.test: Use creative quoting to avoid
spuriously triggering the `sc_tests_Exit_not_exit' maintainer
check.
* Makefile.am (sc_diff_automake_in_automake): Update, as we
now expect 9 lines, not 8, to be changed from `automake.in'
to `automake'.
(sc_diff_aclocal_in_aclocal): New maintainer check, similar to
the above, and checking that only 10 lines are changed from
`aclocal.in' to `aclocal'.
(syntax_check_rules): Update.
(sc_tests_Exit_not_exit): Exempt self tests `self-check-*.test'
from this check, as they can legitimately use the bare `exit'
builtin in various places.
* doc/automake.texi (Python): Remove stray `@' from the end of
a line. Typo introduced in commit `v1.11-312-g5bf7af6'.
* tests/depcomp8a.test: Pass DISTCHECK_CONFIGURE_FLAGS to make
from the environment rather than from the command line, to
pacify the `sc_tests_overriding_macros_on_cmdline' maintainer
check.
* tests/depcomp8b.test: Likewise.
* maint:
automake, aclocal: honour configure-time AUTOCONF and AUTOM4TE
build: the user can override AUTOM4TE, AUTORECONF and AUTOUPDATE too
tests/README: fix example about `make -e' usage
automake, aclocal: honour configure-time AUTOCONF and AUTOM4TE
Currently, the Automake's own configure script allow definition
of AUTOCONF and AUTOM4TE, expected to point respectively to an
autoconf and autom4te programs. But while these definitions are
honoured in the Automake's build systems and test suite, they
were *not* honoured in the generated `automake' and `aclocal'
scripts. This behaviour, apart from being wrong in that it does
not allow the user enough freedom in choosing his tools, also
caused inconsistencies in the test suite, brining to spurious
failures.
Problem reported by Graham Reitz on the automake list; see thread:
<http://lists.gnu.org/archive/html/automake/2011-05/msg00022.html>
* automake.in ($traces): Use `@am_AUTOCONF', not simply `autoconf'.
* aclocal.in ($traces): Use `@am_AUTOM4TE', not simply `autom4te'.
* Makefile.am (do_subst): Substitute also `@am_AUTOCONF' and
`@am_AUTOM4TE'.
* NEWS: Update.
* THANKS: Update.
build: the user can override AUTOM4TE, AUTORECONF and AUTOUPDATE too
Our build system allows the user to override AUTOCONF and AUTOHEADER
at configure time, and honours these overrides in our testsuite.
But it didn't do the same with AUTOM4TE, AUTORECONF and AUTOUPDATE.
This change fixes that inconsistency.
* configure.ac (am_AUTOM4TE, am_AUTOUPDATE, am_AUTORECONF): New
AC_SUBSTitutions. Update comments.
* tests/defs.in ($AUTOUPDATE): Default to `@am_AUTOUPDATE@' now.
($AUTOM4TE): New variable, defaulting to `@am_AUTOM4TE@'.
($AUTORECONF): New variable, defaulting to `@am_AUTORECONF@'.
* doc/Makefile.am ($(srcdir)/amhello-1.0.tar.gz): Update.
remake: behave better with non-GNU make in subdirectories
Currently, with every decent make program, it is possible to
rebuild out-of-date autotools-generated files with a simple
"make Makefile" -- but for this to work reliably with non-GNU
make implementations, the command must be issued from the
top-level directory. This patch removes such limitation.
* lib/am/configure.am (am--refresh): Depend on `%MAKEFILE%'.
* tests/defs.in (using_gmake): New function, backported from the
`master' branch (and simplified).
* tests/remake-subdir.test: New test.
* tests/remake-subdir2.test: Likewise.
* tests/remake-subdir-gnu.test: Likewise.
* tests/remake-subdir-from-subdir.test: Likewise.
* tests/Makefile.am (TESTS): Update.
Ralf Wildenhues [Sun, 22 May 2011 17:02:27 +0000 (19:02 +0200)]
tests/README: fix example about `make -e' usage
* tests/README (Section "Writing test cases" subsection "Do"): When
some variable is never initialized in the Makefile, `-e' is not
necessary in order to override it. DESTDIR is such a variable: we
ensure that we do not ever initialize it. And as such, it is quite
portable to use:
$ make DESTDIR=/foo/bar install
and in fact, quite widely used.
So our example about when `make -e' is required, which references
the `DESTDIR' variable, is poorly chosen, if not downright wrong.
Rewrite it to use `prefix' as the overridden variable instead.
* maint:
testsuite: avoid re-running few tests with 'parallel-tests' option
testsuite: allow tests to avoid the use of 'parallel-tests' option
tests/README: update obsoleted advice
testsuite: avoid re-running few tests with 'parallel-tests' option
Some tests in our testsuite use the 'simple-tests' driver only
marginally, or simply as a mean to conveniently check unrelated
invariants. It makes little sense to force these tests to also
run with the 'parallel-tests' Automake option active, as doing so
offers no real gain in coverage, while often causing a measurable
overhead in execution time (for an already too-slow testsuite).
* tests/pr401.test (parallel_tests): Define to "no", to prevent
the generation of a sibling test script using the 'parallel-tests'
driver.
* tests/pr401b.test: Likewise.
* tests/pr401c.test: Likewise.
testsuite: allow tests to avoid the use of 'parallel-tests' option
* tests/gen-parallel-tests: Do not generate "siblings" for tests
that explicitly define the `parallel_tests', whether to "yes" or
to any other value. Extend heading comments to give a rationale
for this behaviour.
* tests/README: Update.
* tests/README (Section "Writing test cases" subsection "Do"):
Do not suggest to use the `*-p.test' pattern for the names of
hand-written tests which use the `parallel-tests' Automake option.
Not only is this not respected by the existing tests, but it is
more likely to cause conflicts with auto-generated tests.
So, suggest to *avoid* using the `*-p.test' pattern in names
of hand-written tests instead.
(Section "Writing test cases" subsection "Do not"): When
suggesting not to override Makefile variables using command
line arguments, do not use the badly outdated variables `U'
and 'ANSI2KNR' in the example; instead, use the more common
and typical `DESTDIR'.
* tests/defs.in (cross_compiling): New subroutine.
(am__tool_prefix): New internal variable.
(gcc, g++, gcj): Force the use of the correct "tool prefix"
when cross compiling.
(gfortran, g77, non-cross): New requirements.
tests: fix portability issues in 'repeated-options.test'
* tests/repeated-options.test: Do not assume that object files
have `.o' suffix and executables have no default suffix; instead,
use `$(OBJEXT)' and `$(EXEEXT)'.
tests: fix spurious failure of extradep.test on FreeBSD
* tests/extradep.test: When using `ls -t' to determine whether a
file has been updated, make sure to use as reference a file whose
timestamp is expected to be *strictly* older that that of the file
being checked. This is required because at least FreeBSD `ls' do
not sort files with the same timestamp in alphabetical order when
using the `-t' option.
* tests/extradep2.test: Likewise.
tests: fix spurious failure of txinfo21.test on FreeBSD
* tests/txinfo21.test: Use the `is_newest' subroutine instead of
the `ls -t' hack to to determine whether a file has been updated.
This is required because at least FreeBSD `ls' do not sort files
with the same timestamp in alphabetical order when using the `-t'
option.
With some edits to `tests/defs' and `ChangeLog', to reflect the
fact that in master we already had a `yacc' requirement, and no
more had an explicitly-recognized `bison' requirement.
tests defs: allow requirements for compilers (mostly dummy)
Most of the new requirements that are now accepted in `$required'
as consequence of this patch are still dummy. They are planned
to be implemented only in master (or in some derived branch), but
having them here (even just as no-op) will allow for an easier
integration/backporting of potential new testcases.
* tests/defs.in (cc, c++, fortran, fortran77): New requirements,
still dummy.
(flex): New requirement, picking LEX for configure.
(lex): New requirement, alias for `flex'. A more appropriate
implementation, looking for a generic `lex' program, will follow
in the future.
(yacc): New requirement, alias for `bison'. A more appropriate
implementation, looking for a generic `yacc' program, will follow
in the future.
* tests/Makefile.am (TESTS): Added test scripts man[678].test,
which have been present on the filesystem and in the repository,
but which had erroneously been left out from $(TESTS).
* tests/color2.test: If $MAKE contains command-line arguments (as
in e.g., "make -j2"), expect's directive "spawn $env(MAKE)" fails
spuriously, because it tries to run "$MAKE" as a single command.
Fix this with proper uses of the TCL `eval' builtin.
tests: fix self check spurious failure with older bash versions
* tests/self-check-cleanup.test: Add trailing `:' to the test code
passed to $SHELL, otherwise older versions of bash (at least 2.05
and 3.2.39) fail to correctly remove the temporary directory in
the exit trap.
* tests/defs: Move the checks verifying that the variables `me',
`parallel_tests' and `required' aren't set in the environment ...
* tests/defs-static.in: ... in here, with some optimizations to
avoid useless forks. Also, do the same checks for the variables
`original_AUTOMAKE' and `original_ACLOCAL' too.
* tests/self-check-env-sanitize.test: Update.
* tests/Makefile.am (TESTS_ENVIRONMENT): Unset also variables
`original_AUTOMAKE' and `original_ACLOCAL'.
tests: fix spurious failure in a test on TESTS (VPATH-related)
* check-tests-in-builddir.test: When not using the parallel-tests
option, do not check that VPATH components are not present in the
displayed test name, since the simple-tests driver do not try to
strip them.
Jim Meyering [Sun, 16 Jan 2011 14:36:07 +0000 (15:36 +0100)]
test defs: new subroutines for test skipping/failing.
* tests/defs.in (Exit): Move definition of this function earlier.
(warn_, skip_, fail_, framework_failure_): New functions, inspired
to the homonyms in gnulib's tests/init.sh.
($stderr_fileno_): New global variable, used by the new functions
above.
* tests/README: Updated.
* tests/mclean.test: Remove, it's a weak grepping test completely
superseded by the much more complete `maintclean.test'.
* tests/Makefile.am (TESTS): Update.
* tests/defs.in: Sanity check: abort if any of `parallel_tests'
or `required' is in the environment.
($sed_unindent_prog): Initialize to empty, to avoid interferences
from the environment.
* tests/self-check-me-in-env.test: Renamed to ...
* tests/self-check-env-sanitize.test: ... this, and extended.
* tests/Makefile.am (TESTS): Update.
(TESTS_ENVIRONMENT): Unset variables `parallel_tests' and
`required'. Adjust comments.
tests: don't allow `$me' to be overridden from the environment
* tests/defs.in: Sanity check: abort if $me is in the environment.
* tests/self-check-me-in-env.test: New test.
* tests/Makefile.am (TESTS_ENVIRONMENT): Unset variable `me'.
(TESTS): Update.
The AM_TESTS_SETUP naming was not a good one after all. It may
be technically more correct than AM_TESTS_ENVIRONMENT, but the
latter is a better one simply because it is easier to remember,
and even if you've never heard of it and only know the semantics
of TESTS_ENVIRONMENT, you can have a straightforward way to
figure out how AM_TESTS_ENVIRONMENT would work.
* tests/check.am (am__check_pre): Update.
* doc/automake.in (Simple Tests using parallel-tests): Update.
* tests/parallel-tests-am_tests_setup.test: Renamed ...
* tests/parallel-tests-am_tests_environment.test: ... to this,
and updated.
* tests/Makefile.am (TESTS): Update.
check: new developer-reserved AM_TESTS_SETUP variable
For reference, see the discussion at:
<http://lists.gnu.org/archive/html/automake-patches/2011-01/msg00213.html>
* lib/am/check.am [%?PARALLEL_TESTS%] (am__check_pre): Pass also
$(AM_TESTS_SETUP). Comments updated, and some typos fixed.
* doc/automake.texi (Simple Tests using parallel-tests): Document
AM_TESTS_SETUP. Reorder some of the existing documentation a bit.
* tests/parallel-tests-am_tests_setup.test: New test.
* tests/depcomp8b.test: Add the `--enable-dependency-tracking'
option to the ./configure invocation, so that slower dependency
extractors are not rejects.
* tests/depcomp8b.test: Likewise. Also ...
(foo.c): ... since we are at it, fix spacing to be consistent
with GNU coding standards.
The following additional changes has been made:
* m4/amversion.in: Bump serial number from `1' to `8', to avoid
potential future conflicts with branch-1.11.
* m4/amversion.m4: Likewise.
* m4/gcj.m4: Likewise.
* m4/python.m4: Likewise.
* m4/install-sh.m4: Likewise.
* tests/self-check-cleanup.test: Be laxer when grepping output
from `ls -l', to account for ACLs and SELinux-only files.
* tests/self-check-dir.test: Source `defs-static' to read in the
correct definition for $SHELL.
* tests/self-check-me.test: Likewise, and extend a bit.
maintainer check: quote literal `$' in Makefile rule
* Makefile.am (sc_tests_PATH_SEPARATOR): Escape literal `$'
character in double-quoted string. Fix a bug in which the rule
emitted an erroneously empty substring in its error message.
test defs: define default $distdir (help reducing duplication)
* tests/defs.in ($distdir): New variable, might be used in
testcases checking distribution-related features.
* tests/pr9.test: Use it.
* tests/subdir9.test: Likewise.
* tests/vtexi3.test: Add comment explaining why we redefine
$distdir in this test.
depcomp: fix bugs in tests and in the depcomp script
* lib/depcomp (gcc): Remove duplicated `-e' from sed invocation.
* tests/depcomp10.test: Make it executable. Fix a blunder that
has left part of an intended comment not prefixed by `#', thus
causing shell syntax errors. In this same comment, break a
too-long reported error message on multiple lines, for clarity.
Add reference to the relevant bug report. Add a comment which
explains why the test result 'skipped' if the first "make" call
fails. Add other useful comments.
* tests/depcomp9.test: Slightly improve comments.
* lib/depcomp: Be sure to remove VPATH-prefixed object from
dependency output when creating stub rule.
* tests/depcomp10.test: New test.
* tests/Makefile.am (TESTS): Update.
* NEWS: Update.
Report by Bruno Haible.
test defs: fix 'javac' requirement for older JDK versions
The Java compiler from JDK 1.5 (and presumably earlier versions)
cannot handle the `-version' option by itself; and while it does
print the version number, it then errors out with an usage error:
$ javac -version
javac 1.5.0_22
javac: no source files
Usage: javac <options> <source files>
...
Luckily, adding the `-help' option to the `javac' invocation
seems to fix this problem.
* tests/defs.in (javac): Pass also the `-help' option to the
`javac' program. Add a comment explaining why it is needed.