From 6b3735581b93ce179e300a66da1ebccac2736dfa Mon Sep 17 00:00:00 2001 From: Karl Berry Date: Mon, 22 Jun 2026 09:46:19 -0700 Subject: [PATCH] doc: split Parallel Test Harness node into subsubsections. * doc/automake.texi (Parallel Test Harness): break this too-long node into several Parallel Test ... subsubsections. --- doc/automake.texi | 76 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 55 insertions(+), 21 deletions(-) diff --git a/doc/automake.texi b/doc/automake.texi index a098e7e03..f2bf092b3 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -9967,11 +9967,23 @@ files, concurrent execution of tests with @code{make -j}, specification of inter-test dependencies, lazy reruns of tests that have not completed in a prior run, and hard errors for exceptional failures. +@menu +* Basics: Parallel Test Basics. @file{.trs}/@file{.log} file pairs. +* Runners: Parallel Test Runners. Specifying commands to run tests. +* Selection: Parallel Test Selection. Which tests to run. +* Dependencies: Parallel Test Dependencies. Serializing tests. +* Variables: Parallel Test Variables. Limitations on certain variables. +@end menu + +@node Parallel Test Basics +@subsubsection Parallel Test Basics + @anchor{Basics of test metadata} @vindex TESTS @cindex @file{.log} files @cindex @file{.trs} files @cindex test metadata +@cindex metadata, for tests The parallel test harness operates by defining a set of @command{make} rules that run the test scripts listed in @code{TESTS}, and, for each such script, save its output in a corresponding @file{.log} file and @@ -10007,7 +10019,7 @@ file is output after the summary. @vindex TEST_EXTENSIONS @vindex TEST_LOGS -Each couple of @file{.log} and @file{.trs} files is created when the +Each pair of @file{.log} and @file{.trs} files is created when the corresponding test has completed. The set of log files is listed in the read-only variable @code{TEST_LOGS}, and defaults to @code{TESTS}, with the executable extension if any (@pxref{EXEEXT}), as well as any @@ -10027,6 +10039,11 @@ lifted), configure substitutions in the definition of @code{TESTS} can only work if they will expand to a list of tests that have a suffix listed in @code{TEST_EXTENSIONS}. +@node Parallel Test Runners +@subsubsection Parallel Test Runners + +@cindex test runners +@cindex runners, custom for tests @vindex _LOG_COMPILE @vindex _LOG_COMPILER @vindex _LOG_FLAGS @@ -10085,6 +10102,12 @@ LOG_COMPILER = $(PERL) AM_LOG_FLAGS = -Mstrict -w @end example +@node Parallel Test Selection +@subsubsection Parallel Test Selection + +@cindex selecting tests to run +@cindex limiting tests that are run +@cindex test selection By default, the test suite harness will run all tests, but there are several ways to limit the set of tests that are run: @@ -10098,6 +10121,7 @@ env TESTS="foo.test bar.test" make -e check @end example @c https://bugs.gnu.org/42883 +@vindex SUBDIRS@r{, setting when selecting tests} If you're using a recursive make setup, you'll probably also need to override @code{SUBDIRS}: @@ -10162,6 +10186,12 @@ This convenience target will set @code{RECHECK_LOGS} appropriately before invoking the main test harness. @end itemize +@node Parallel Test Dependencies +@subsubsection Parallel Test Dependencies + +@cindex serialization of tests +@cindex tests, serialization of +@cindex parallel tests, serializing @noindent In order to guarantee an ordering between tests even with @code{make -j@var{N}}, dependencies between the corresponding @file{.log} files @@ -10178,22 +10208,25 @@ foo-execute.log: foo-compile.log Please note that this ordering ignores the @emph{results} of required tests, thus the test @file{foo-execute.test} is run even if the test @file{foo-compile.test} failed or was skipped beforehand. Further, -please note that specifying such dependencies currently works only for -tests that end in one of the suffixes listed in @code{TEST_EXTENSIONS}. +specifying such dependencies currently works only for tests that end +in one of the suffixes listed in @code{TEST_EXTENSIONS}. Tests without such specified dependencies may be run concurrently with parallel @command{make -j@var{N}}, so be sure they are prepared for concurrent execution. -@cindex Unit tests +@cindex unit tests @c Keep in sync with 'parallel-tests-extra-programs.sh'. -The combination of lazy test execution and correct dependencies between -tests and their sources may be exploited for efficient unit testing -during development. To further speed up the edit-compile-test cycle, it -may even be useful to specify compiled programs in @code{EXTRA_PROGRAMS} -instead of with @code{check_PROGRAMS}, as the former allows intertwined -compilation and test execution (but note that @code{EXTRA_PROGRAMS} are -not cleaned automatically; @pxref{Uniform}). +The combination of lazy test execution and correct dependencies +between tests and their sources may be exploited for efficient unit +testing during development. To further speed up the edit-compile-test +cycle, it may even be useful to specify compiled programs in +@code{EXTRA_PROGRAMS} instead of with @code{check_PROGRAMS}, as the +former allows intertwined compilation and test execution (but +@code{EXTRA_PROGRAMS} are not cleaned automatically; @pxref{Uniform}). + +@node Parallel Test Variables +@subsubsection Parallel Test Variables The variables @code{TESTS} and @code{XFAIL_TESTS} may contain conditional parts as well as configure substitutions. In the latter @@ -10203,16 +10236,17 @@ the inference rules generated by @command{automake} can apply. For literal test names, @command{automake} can generate per-target rules to avoid this limitation. -Please note that it is currently not possible to use @code{$(srcdir)/} -or @code{$(top_srcdir)/} in the @code{TESTS} variable. This technical -limitation is necessary to avoid generating test logs in the source tree -and has the unfortunate consequence that it is not possible to specify -distributed tests that are themselves generated by means of explicit -rules, in a way that is portable to all @command{make} implementations -(@pxref{Make Target Lookup,,, autoconf, The Autoconf Manual}, the -semantics of FreeBSD and OpenBSD @command{make} conflict with this). -In case of doubt you may want to require to use GNU @command{make}, -or work around the issue with inference rules to generate the tests. +It is currently not possible to use @code{$(srcdir)/} or +@code{$(top_srcdir)/} in the @code{TESTS} variable. This technical +limitation is necessary to avoid generating test logs in the source +tree and has the unfortunate consequence that it is not possible to +specify distributed tests that are themselves generated by means of +explicit rules, in a way that is portable to all @command{make} +implementations (@pxref{Make Target Lookup,,, autoconf, The Autoconf +Manual}; the semantics of FreeBSD and OpenBSD @command{make} conflict +with this). In case of doubt you may want to require to use GNU +@command{make}, or work around the issue with inference rules to +generate the tests. @node Custom Test Drivers @section Custom Test Drivers -- 2.47.3