testsuite: Skip tests that use aclocal or automake if they’re too old.

Old versions of aclocal and/or automake can cause several tests to
fail, for two unrelated reasons:

 - when used with sufficiently new Perl they might print a “your code
   has a minor bug” message on every invocation, causing tests that
   check for specific output to choke
 - aclocal prior to 1.11.2 does not support --system-acdir

We already had code (in each individual test) to skip tests that depend
on aclocal and/or automake when those programs were *unavailable*, or
when buggy wrapper scripts were detected (apparently some BSDs once
shipped with a wrapper that required you to set an environment variable
before ‘automake’ would do anything at all).  Consolidate all of that
code into local.at and augment it to detect the above two problems.

Individual tests that require automake and/or aclocal should now
just say AT_REQUIRE_AUTOMAKE and/or AT_REQUIRE_ACLOCAL at the top.
Individual tests that run autoreconf but *don’t* need a real aclocal
should instead start with AT_SUPPRESS_ACLOCAL, which sets ACLOCAL=true
and creates a dummy aclocal.m4.

While I was at it I noticed that AT_TESTED malfunctions if you have a
shell variable in there that expands to more than one shell word, so I
removed the AT_TESTED line for $M4 $PERL etc and replaced it with a
custom report for the most important system-provided programs that the
testsuite uses.  That report now also includes automake and aclocal.

This should fix the problems reported by Sevan Janiyan in
<https://lists.gnu.org/archive/html/bug-autoconf/2023-12/msg00159.html>.
Tested on x86_64-linux against automake 1.11.1, 1.11.2, 1.13.0, and 1.16.5.

* tests/local.at (AT_PREPARE_TESTS): Consolidate code to detect
  availability of automake and aclocal here.  Also detect Perl
  warning messages that will interfere with matching on the output,
  and aclocal too old to understand --system-acdir.  Produce a custom
  report of the versions of system-provided programs we need:
  m4, perl, sh, make, and optionally automake and aclocal.
  (AT_TESTED): Only list programs that are part of this package.
  (AT_REQUIRE_AUTOMAKE, AT_REQUIRE_ACLOCAL, AT_SUPPRESS_ACLOCAL):
  New helper macros.
  * tests/fortran.at, tests/tools.at, tests/torture.at:
  Use AT_REQUIRE_AUTOMAKE, AT_REQUIRE_ACLOCAL, AT_SUPPRESS_ACLOCAL
  throughout, as appropriate.

diff --git a/tests/fortran.at b/tests/fortran.at
index 87eba8ff937a502dafeb5f002f50edcb4b4bef4c..7c98171bf7a32c0fa95bf3dc347096d6571fbea2 100644 (file)
--- a/tests/fortran.at
+++ b/tests/fortran.at
@@ -109,12 +109,7 @@ AT_DATA([foo.f],
       end
 ]])
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
-AT_DATA([aclocal.m4])
-ACLOCAL=true
-export ACLOCAL
-
+AT_SUPPRESS_ACLOCAL
 AT_CHECK([autoreconf -vi], [], [ignore], [ignore])
 AT_CHECK_CONFIGURE
 AT_CHECK_MAKE
@@ -154,12 +149,7 @@ AT_DATA([foo.f],
       end
 ]])
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
-AT_DATA([aclocal.m4])
-ACLOCAL=true
-export ACLOCAL
-
+AT_SUPPRESS_ACLOCAL
 AT_CHECK([autoreconf -vi], [], [ignore], [ignore])
 AT_CHECK_CONFIGURE
 AT_CHECK_MAKE

diff --git a/tests/local.at b/tests/local.at
index da1b42a66de23660de61f3bbf4339f4f0f978719..c8ce8163c427435a2a7b3e9cde36cf13dec391fd 100644 (file)
--- a/tests/local.at
+++ b/tests/local.at
@@ -24,8 +24,6 @@ m4_pattern_allow([^m4_(define|shift)$])
 # Programs this package provides
 AT_TESTED([autom4te autoconf autoheader autoupdate autoreconf ifnames])
 
-# System-provided programs that this package relies on
-AT_TESTED([$M4 $PERL ${CONFIG_SHELL-$SHELL} $MAKE])
 
 # Enable colored test output.
 AT_COLOR_TESTS
@@ -43,12 +41,63 @@ AS_IF([test -e nonexistent],
 CONFIG_SITE=`pwd`/nonexistent/config.site
 export CONFIG_SITE
 
-# Ensure MAKE is set to a useful value.  Unlike the above, we *do*
-# want to inherit this variable from the parent environment and/or
-# our command line.
+# Ensure MAKE, AUTOMAKE, and ACLOCAL are set to useful values.  Unlike
+# the above, we *do* want to inherit these variables from the parent
+# environment and/or our command line.  Also, detect now whether
+# automake and aclocal are unavailable or too old.
 : "${MAKE=make}"
 export MAKE
 
+# Some old versions of automake, when used with newer Perl interpreters,
+# will print a warning message about their own Perl code on every
+# invocation, which breaks various tests' expectations for output.
+# This also weeds out broken wrapper scripts; in the past some vendors
+# have shipped an 'automake' that didn't work without user configuration.
+: "${AUTOMAKE=automake}"
+at_automake_version="`$AUTOMAKE --version 2>&1 || echo not found`"
+at_automake_version_1="`AS_ECHO(["$at_automake_version"]) | sed 1q`"
+AS_CASE([$at_automake_version_1],
+  [[*GNU*\)\ [1-9]\.[0-9]*]], [],
+  [AUTOMAKE=false])
+export AUTOMAKE
+
+# Used in the code below that decides delay intervals.
+if test "$AUTOMAKE" == false
+then
+  # If automake is unavailable, then whether it supports subsecond
+  # mtime is moot.
+  at_automake_subsecond_mtime=:
+elif AS_ECHO(["$at_automake_version"]) |
+     grep 'Features: subsecond-mtime' > /dev/null 2>&1
+then
+  at_automake_subsecond_mtime=:
+else
+  at_automake_subsecond_mtime=false
+fi
+
+# The same Perl and wrapper issues exist with aclocal.  Also, we
+# require a version that understands --system-acdir and configure.ac.
+: "${ACLOCAL=aclocal}"
+at_aclocal_version="`$ACLOCAL --version 2>&1 || echo not found`"
+at_aclocal_version_1="`AS_ECHO(["$at_aclocal_version"]) | sed 1q`"
+AS_CASE([$at_aclocal_version_1],
+  [[*GNU*\)\ [1-9]\.[0-9]*]],
+  [mkdir at_scratch
+   (cd at_scratch &&
+    touch configure.ac &&
+    mkdir empty &&
+    $ACLOCAL --system-acdir=`pwd`/empty) || ACLOCAL=false
+   rm -rf at_scratch],
+  [ACLOCAL=false])
+export ACLOCAL
+
+# Prevent aclocal from reading third-party macros, in case they are buggy.
+# (AM_INIT_AUTOMAKE will still be available via the default --automake-acdir.)
+if test "$ACLOCAL" != false; then
+  test -d at_empty_dir || mkdir at_empty_dir
+  ACLOCAL="$ACLOCAL --system-acdir=`cd at_empty_dir && pwd`"
+fi
+
 # Determine how long we need to delay in between operations that might
 # modify autom4te.cache.  This depends on three factors: whether the
 # 'sleep' utility supports fractional seconds in its argument; what
@@ -76,14 +125,13 @@ export MAKE
 # Default to the coarsest case.
 at_ts_resolution=2
 
-# Only try to go finer than 1s if sleep, autom4te, and automake
+# Only try to go finer than 1s if sleep, autom4te, and automake (if present)
 # can all handle it.
 at_try_resolutions=1
 if sleep 0.001 2>/dev/null &&
    autom4te --version 2>&1 |
      grep 'Features:.*subsecond-mtime' > /dev/null 2>&1 &&
-   automake --version 2>&1 |
-     grep 'Features:.*subsecond-mtime' > /dev/null 2>&1
+     $at_automake_subsecond_mtime
 then
   at_try_resolutions="0.001 0.01 0.1 $at_try_resolutions"
 fi
@@ -121,8 +169,64 @@ for at_try_res in $at_try_resolutions; do
 done
 rm -f conftest.ts?
 
-AS_ECHO(["$at_srcdir/AT_LINE: using ${at_ts_resolution}s as timestamp resolution
-"]) >&AS_MESSAGE_LOG_FD
+# AT_TESTED doesn't support variables in its argument, particularly
+# when they might contain the name of a program along with some
+# arguments, or when they might evaluate to 'false'.
+# Do this in a subshell to contain the effects of 'set'.
+(status=0
+AS_BOX([System supplied programs that we depend on.])
+echo
+for at_program in \
+  "r m4 $M4" \
+  "r perl $PERL" \
+  "r sh ${CONFIG_SHELL-$SHELL}" \
+  "r make $MAKE" \
+  "o automake $AUTOMAKE" \
+  "o aclocal $ACLOCAL"
+do
+  # The first three words of $at_program are a code for whether
+  # it's required (r) or optional (o); the conventional name of the
+  # program (for error messages); and the name (possibly a pathname)
+  # we were told to use for it.
+  set dummy $at_program
+  at_required=$[]2
+  at_prog_name=$[]3
+  at_prog_use=$[]4
+  at_prog_use_=
+
+  AS_CASE(["$at_prog_use"],
+    [: | true | false], [],
+    [[[\\/]* | ?:[\\/]*]],
+      [test -f "$at_prog_use" && at_prog_use_=$at_prog_use],
+    [_AS_PATH_WALK([$PATH],
+      [if test -f "$as_dir$at_prog_use"; then
+         at_prog_use_=$as_dir$at_prog_use
+         break
+       fi])])
+
+  if test -n "$at_prog_use_"; then
+    {
+      AS_ECHO(["# $at_prog_name is $at_prog_use_"])
+      AS_ECHO(["$at_prog_use_ --version"])
+      "$at_prog_use_" --version </dev/null
+    } >&AS_MESSAGE_LOG_FD 2>&1
+  elif test $at_required = r; then
+    AS_ECHO(["*** Required program $at_prog_name is unavailable."])
+    status=1
+  else
+    AS_ECHO(["# $at_prog_name is unavailable"])
+  fi
+  echo
+done
+
+AS_ECHO(["$at_srcdir/AT_LINE: using ${at_ts_resolution}s as mtime resolution"])
+echo
+exit $status
+) >&AS_MESSAGE_LOG_FD 2>&1
+
+if test $? -ne 0; then
+  AS_ERROR([*** Some required programs were not found.])
+fi
 ])
 
 
@@ -145,6 +249,27 @@ m4_define([AT_CMP],
 m4_define([AT_MTIME_DELAY],
 [sleep $at_ts_resolution])
 
+# AT_REQUIRE_AUTOMAKE
+# -------------------
+# Skip this test if automake is unavailable or too old.
+m4_define([AT_REQUIRE_AUTOMAKE],
+[AT_SKIP_IF([test "$AUTOMAKE" = false])])
+
+# AT_REQUIRE_ACLOCAL
+# ------------------
+# Skip this test if aclocal is unavailable or too old.
+m4_define([AT_REQUIRE_ACLOCAL],
+[AT_SKIP_IF([test "$ACLOCAL" = false])])
+
+# AT_SUPPRESS_ACLOCAL
+# -------------------
+# Prevent autoreconf from running aclocal, which might not be available.
+# Use this instead of AT_REQUIRE_ACLOCAL in tests that run autoreconf
+# but don't need aclocal to do anything.
+m4_define([AT_SUPPRESS_ACLOCAL],
+[AT_DATA([aclocal.m4])
+ACLOCAL=true])
+
 ## ---------------- ##
 ## Testing syntax.  ##
 ## ---------------- ##

diff --git a/tests/tools.at b/tests/tools.at
index 5dcf5f29fd8527849c9104e5b9e92525ae52a477..84109d6ce4c7383e2b2da25b59a8d94e803ecc36 100644 (file)
--- a/tests/tools.at
+++ b/tests/tools.at
@@ -1461,13 +1461,7 @@ AT_CLEANUP
 # ----------------------------------------
 AT_SETUP([autoupdating with aclocal and m4@&t@_include])
 
-# We use aclocal.
-AT_CHECK([aclocal --version || exit 77], [], [ignore], [ignore])
-
-# Prevent aclocal from reading third-party macros, in case they are buggy.
-mkdir empty
-ACLOCAL="aclocal --system-acdir=`cd empty && pwd`"
-export ACLOCAL
+AT_REQUIRE_ACLOCAL
 
 mkdir m4 aclocal
 AT_DATA([configure.ac],
@@ -1495,16 +1489,12 @@ AT_CLEANUP
 # -----------------------------
 
 AT_SETUP([autom4te preselections])
-# We use aclocal and automake.  Skip broken automake wrappers.
-AT_CHECK([automake --version || exit 77], [], [stdout], [ignore])
-AT_CHECK([[grep '[1-9]\.[0-9]' stdout || exit 77]], [], [ignore])
-AT_CHECK([test ! -f $HOME/.autom4te.cfg || exit 77], [], [ignore], [ignore])
 
-# Prevent aclocal from reading third-party macros, in case they are buggy.
-# (AM_INIT_AUTOMAKE will still be available via the default --automake-acdir.)
-mkdir empty
-ACLOCAL="aclocal --system-acdir=`cd empty && pwd`"
-export ACLOCAL
+AT_REQUIRE_AUTOMAKE
+AT_REQUIRE_ACLOCAL
+
+# This user configuration file will interfere with this test.
+AT_CHECK([test ! -f $HOME/.autom4te.cfg || exit 77], [], [ignore], [ignore])
 
 AT_DATA([configure.ac],
 [[AC_INIT(GNU foo, 1.0)
@@ -1640,10 +1630,7 @@ AT_CLEANUP
 
 AT_SETUP([autotools and whitespace in file names])
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
-ACLOCAL=true
-export ACLOCAL
+AT_SUPPRESS_ACLOCAL
 
 x=
 export x
@@ -1686,8 +1673,6 @@ END
   AT_CHECK([autoscan --force -I "$dir"], [], [], [ignore])
   # autoreconf requires a sane input file name.
   mv -f "$file.ac" configure.ac
-  # Since aclocal is disabled, provide a stub aclocal.m4.
-  AT_DATA([aclocal.m4])
   AT_CHECK([autoreconf -B "$dir"])
   AT_CHECK([autoreconf --force -I "$dir"])
 
@@ -1707,13 +1692,7 @@ AT_CLEANUP
 
 AT_SETUP([autotools and relative TMPDIR])
 
-# We use aclocal.
-AT_CHECK([aclocal --version || exit 77], [], [ignore], [ignore])
-
-# Prevent aclocal from reading third-party macros, in case they are buggy.
-mkdir empty
-ACLOCAL="aclocal --system-acdir=`cd empty && pwd`"
-export ACLOCAL
+AT_REQUIRE_ACLOCAL
 
 mkdir _tmp
 TMPDIR=_tmp

diff --git a/tests/torture.at b/tests/torture.at
index ba52e12dc8c64d74ee5a89a7f1b5dee9e372e5d1..288a5a5590ac8ae3c6ea9b33dd457bccaf7cb82e 100644 (file)
--- a/tests/torture.at
+++ b/tests/torture.at
@@ -1456,17 +1456,7 @@ AT_BANNER([autoreconf.])
 AT_SETUP([Configuring subdirectories])
 AT_KEYWORDS(autoreconf)
 
-# We use aclocal (via autoreconf).
-AT_CHECK([aclocal --version || exit 77], [], [stdout], [ignore])
-AT_CHECK([[grep '[1-9]\.[0-9]' stdout || exit 77]], [], [ignore])
-
-# It should understand configure.ac.
-AT_CHECK([[grep '[^0-9]1\.[01234][^0-9]' stdout && exit 77]], [1], [ignore])
-
-# Prevent aclocal from reading third-party macros, in case they are buggy.
-mkdir empty
-ACLOCAL="aclocal --system-acdir=`cd empty && pwd`"
-export ACLOCAL
+AT_REQUIRE_ACLOCAL
 
 # The contents of 'inner/', and 'inner/innermost/'.
 AS_MKDIR_P([inner/innermost])
@@ -1584,13 +1574,7 @@ AT_CLEANUP
 AT_SETUP([Deep Package])
 AT_KEYWORDS(autoreconf)
 
-# We use aclocal (via autoreconf).
-AT_CHECK([aclocal --version || exit 77], [], [ignore], [ignore])
-
-# Prevent aclocal from reading third-party macros, in case they are buggy.
-mkdir empty
-ACLOCAL="aclocal --system-acdir=`cd empty && pwd`"
-export ACLOCAL
+AT_REQUIRE_ACLOCAL
 
 # The contents of '.'
 AT_DATA([install-sh], [])
@@ -1724,12 +1708,7 @@ AT_SETUP([Non-Autoconf AC_CONFIG_SUBDIRS])
 AT_KEYWORDS([autoreconf])
 
 # We use aclocal (via autoreconf).
-AT_CHECK([aclocal --version || exit 77], [], [ignore], [ignore])
-
-# Prevent aclocal from reading third-party macros, in case they are buggy.
-mkdir empty
-ACLOCAL="aclocal --system-acdir=`cd empty && pwd`"
-export ACLOCAL
+AT_REQUIRE_ACLOCAL
 
 AT_DATA([install-sh], [])
 AT_DATA([configure.ac],
@@ -1784,12 +1763,7 @@ AC_CONFIG_SUBDIRS([$my_subdirs])
 AC_OUTPUT
 ]])
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
-AT_DATA([aclocal.m4])
-ACLOCAL=true
-export ACLOCAL
-
+AT_SUPPRESS_ACLOCAL
 AS_MKDIR_P([foo])
 
 AT_DATA([foo/configure],
@@ -1830,12 +1804,7 @@ AC_CONFIG_SUBDIRS()
 AC_OUTPUT
 ]])
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
-AT_DATA([aclocal.m4])
-ACLOCAL=true
-export ACLOCAL
-
+AT_SUPPRESS_ACLOCAL
 # autoreconf should have no complaints, and the generated configure
 # script should run fine with or without --no-recursion.
 AT_CHECK([autoreconf -Werror], [0], [ignore])
@@ -1851,10 +1820,9 @@ AT_CLEANUP
 AT_SETUP([Empty directory])
 AT_KEYWORDS([autoreconf])
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
+# Prevent autoreconf from running aclocal, which might not be available.
+dnl We can't use AT_SUPPRESS_ACLOCAL here because it creates an aclocal.m4.
 ACLOCAL=true
-export ACLOCAL
 
 # The test group directory is not necessarily _empty_, but it does not contain
 # files meaningful to 'autoreconf'.
@@ -1874,15 +1842,8 @@ AT_CLEANUP
 AT_SETUP([Unusual Automake input files])
 AT_KEYWORDS([autoreconf])
 
-# We use aclocal and automake via autoreconf.
-AT_CHECK([automake --version || exit 77], [], [stdout], [ignore])
-AT_CHECK([[grep '[1-9]\.[0-9]' stdout || exit 77]], [], [ignore])
-
-# Prevent aclocal from reading third-party macros, in case they are buggy.
-# (AM_INIT_AUTOMAKE will still be available via the default --automake-acdir.)
-mkdir empty
-ACLOCAL="aclocal --system-acdir=`cd empty && pwd`"
-export ACLOCAL
+AT_REQUIRE_AUTOMAKE
+AT_REQUIRE_ACLOCAL
 
 AT_DATA([configure.ac],
 [[AC_INIT(GNU foo, 1.0)
@@ -1913,14 +1874,8 @@ AT_KEYWORDS([autoreconf])
 # --help output, that option should not cause errors, even if some
 # of the subsidiary programs don't support it.
 
-# We use aclocal and automake via autoreconf.
-AT_CHECK([automake --version || exit 77], [], [ignore], [ignore])
-
-# Prevent aclocal from reading third-party macros, in case they are buggy.
-# (AM_INIT_AUTOMAKE will still be available via the default --automake-acdir.)
-mkdir empty
-ACLOCAL="aclocal --system-acdir=`cd empty && pwd`"
-export ACLOCAL
+AT_REQUIRE_AUTOMAKE
+AT_REQUIRE_ACLOCAL
 
 AT_DATA([configure.ac],
 [[AC_INIT(GNU foo, 1.0)
@@ -1962,12 +1917,7 @@ AT_CLEANUP
 AT_SETUP([Missing auxiliary files (config.*)])
 AT_KEYWORDS([autoreconf])
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
-AT_DATA([aclocal.m4])
-ACLOCAL=true
-export ACLOCAL
-
+AT_SUPPRESS_ACLOCAL
 AT_DATA([configure.ac],
 [[AC_INIT([GNU foo], [1.0])
 AC_CONFIG_AUX_DIR([build-aux])
@@ -2028,12 +1978,7 @@ AT_KEYWORDS([autoreconf])
 # configure script _doesn't_ need config.{sub,guess} but does need
 # install-sh.
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
-AT_DATA([aclocal.m4])
-ACLOCAL=true
-export ACLOCAL
-
+AT_SUPPRESS_ACLOCAL
 AT_DATA([configure.ac],
 [[AC_INIT([GNU foo], [1.0])
 AC_CONFIG_AUX_DIR([build-aux])
@@ -2078,12 +2023,7 @@ AT_KEYWORDS([autoreconf])
 # usage in some automake recipes, but which was broken in autoconf
 # beta 2.69d (see https://savannah.gnu.org/support/?110363).
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
-AT_DATA([aclocal.m4])
-ACLOCAL=true
-export ACLOCAL
-
+AT_SUPPRESS_ACLOCAL
 AT_DATA([configure.ac],
 [[AC_INIT([GNU foo], [1.0])
 AC_CONFIG_AUX_DIR([build-aux])
@@ -2123,12 +2063,7 @@ AT_KEYWORDS([autoreconf])
 # Additional wrinkle: in case automake got to the files we install
 # first, we need to *not* overwrite a newer copy supplied by them.
 
-# Prevent autoreconf from running aclocal, which might not exist,
-# or could barf over warnings in third-party macro files.
-AT_DATA([aclocal.m4])
-ACLOCAL=true
-export ACLOCAL
-
+AT_SUPPRESS_ACLOCAL
 AT_DATA([configure.ac],
 [[AC_INIT([GNU foo], [1.0])
 AC_CONFIG_AUX_DIR([build-aux])