of inter-test dependencies, lazy reruns of tests that have not completed
in a prior run, and hard errors for exceptional failures.
+@menu
+* Basics: Parallel Test Basics. @file{.trs}/@file{.log} file pairs.
+* Runners: Parallel Test Runners. Specifying commands to run tests.
+* Selection: Parallel Test Selection. Which tests to run.
+* Dependencies: Parallel Test Dependencies. Serializing tests.
+* Variables: Parallel Test Variables. Limitations on certain variables.
+@end menu
+
+@node Parallel Test Basics
+@subsubsection Parallel Test Basics
+
@anchor{Basics of test metadata}
@vindex TESTS
@cindex @file{.log} files
@cindex @file{.trs} files
@cindex test metadata
+@cindex metadata, for tests
The parallel test harness operates by defining a set of @command{make}
rules that run the test scripts listed in @code{TESTS}, and, for each
such script, save its output in a corresponding @file{.log} file and
@vindex TEST_EXTENSIONS
@vindex TEST_LOGS
-Each couple of @file{.log} and @file{.trs} files is created when the
+Each pair of @file{.log} and @file{.trs} files is created when the
corresponding test has completed. The set of log files is listed in
the read-only variable @code{TEST_LOGS}, and defaults to @code{TESTS},
with the executable extension if any (@pxref{EXEEXT}), as well as any
only work if they will expand to a list of tests that have a suffix listed
in @code{TEST_EXTENSIONS}.
+@node Parallel Test Runners
+@subsubsection Parallel Test Runners
+
+@cindex test runners
+@cindex runners, custom for tests
@vindex _LOG_COMPILE
@vindex _LOG_COMPILER
@vindex _LOG_FLAGS
AM_LOG_FLAGS = -Mstrict -w
@end example
+@node Parallel Test Selection
+@subsubsection Parallel Test Selection
+
+@cindex selecting tests to run
+@cindex limiting tests that are run
+@cindex test selection
By default, the test suite harness will run all tests, but there are
several ways to limit the set of tests that are run:
@end example
@c https://bugs.gnu.org/42883
+@vindex SUBDIRS@r{, setting when selecting tests}
If you're using a recursive make setup, you'll probably also need to
override @code{SUBDIRS}:
before invoking the main test harness.
@end itemize
+@node Parallel Test Dependencies
+@subsubsection Parallel Test Dependencies
+
+@cindex serialization of tests
+@cindex tests, serialization of
+@cindex parallel tests, serializing
@noindent
In order to guarantee an ordering between tests even with @code{make
-j@var{N}}, dependencies between the corresponding @file{.log} files
Please note that this ordering ignores the @emph{results} of required
tests, thus the test @file{foo-execute.test} is run even if the test
@file{foo-compile.test} failed or was skipped beforehand. Further,
-please note that specifying such dependencies currently works only for
-tests that end in one of the suffixes listed in @code{TEST_EXTENSIONS}.
+specifying such dependencies currently works only for tests that end
+in one of the suffixes listed in @code{TEST_EXTENSIONS}.
Tests without such specified dependencies may be run concurrently with
parallel @command{make -j@var{N}}, so be sure they are prepared for
concurrent execution.
-@cindex Unit tests
+@cindex unit tests
@c Keep in sync with 'parallel-tests-extra-programs.sh'.
-The combination of lazy test execution and correct dependencies between
-tests and their sources may be exploited for efficient unit testing
-during development. To further speed up the edit-compile-test cycle, it
-may even be useful to specify compiled programs in @code{EXTRA_PROGRAMS}
-instead of with @code{check_PROGRAMS}, as the former allows intertwined
-compilation and test execution (but note that @code{EXTRA_PROGRAMS} are
-not cleaned automatically; @pxref{Uniform}).
+The combination of lazy test execution and correct dependencies
+between tests and their sources may be exploited for efficient unit
+testing during development. To further speed up the edit-compile-test
+cycle, it may even be useful to specify compiled programs in
+@code{EXTRA_PROGRAMS} instead of with @code{check_PROGRAMS}, as the
+former allows intertwined compilation and test execution (but
+@code{EXTRA_PROGRAMS} are not cleaned automatically; @pxref{Uniform}).
+
+@node Parallel Test Variables
+@subsubsection Parallel Test Variables
The variables @code{TESTS} and @code{XFAIL_TESTS} may contain
conditional parts as well as configure substitutions. In the latter
literal test names, @command{automake} can generate per-target rules
to avoid this limitation.
-Please note that it is currently not possible to use @code{$(srcdir)/}
-or @code{$(top_srcdir)/} in the @code{TESTS} variable. This technical
-limitation is necessary to avoid generating test logs in the source tree
-and has the unfortunate consequence that it is not possible to specify
-distributed tests that are themselves generated by means of explicit
-rules, in a way that is portable to all @command{make} implementations
-(@pxref{Make Target Lookup,,, autoconf, The Autoconf Manual}, the
-semantics of FreeBSD and OpenBSD @command{make} conflict with this).
-In case of doubt you may want to require to use GNU @command{make},
-or work around the issue with inference rules to generate the tests.
+It is currently not possible to use @code{$(srcdir)/} or
+@code{$(top_srcdir)/} in the @code{TESTS} variable. This technical
+limitation is necessary to avoid generating test logs in the source
+tree and has the unfortunate consequence that it is not possible to
+specify distributed tests that are themselves generated by means of
+explicit rules, in a way that is portable to all @command{make}
+implementations (@pxref{Make Target Lookup,,, autoconf, The Autoconf
+Manual}; the semantics of FreeBSD and OpenBSD @command{make} conflict
+with this). In case of doubt you may want to require to use GNU
+@command{make}, or work around the issue with inference rules to
+generate the tests.
@node Custom Test Drivers
@section Custom Test Drivers
projects / thirdparty / automake.git / commitdiff
