From: Stefano Lattarini Date: Tue, 10 May 2011 09:30:05 +0000 (+0200) Subject: parallel-tests: add auxiliary script 'test-driver', refactor X-Git-Tag: ng-0.5a~89^2~155^2~29 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=a27c9c4956fd1643cc3a0b3a553ce40039867680;p=thirdparty%2Fautomake.git parallel-tests: add auxiliary script 'test-driver', refactor This refactoring should cause no API of functionality change, and is meant only to simplify the future implementation of TAP and SubUnit testsuite drivers. More precisely, our roadmap is to move most of the "testsuite driving" features out of the Automake-generated Makefiles, and into external scripts with well-defined interfaces. This will allow the user to define its own personalized testsuite drivers, and will also offer us a framework upon which to implement our new TAP and SubUnit drivers, all in a very unobtrusive way and retaining an high degree of code reuse and backward-compatibility. * lib/test-driver: New auxiliary script. * lib/Makefile.am (dist_SCRIPT_DATA): Add it. * automake.in (handle_tests): Require the new auxiliary script `test-driver', and define a new internal makefile variable `$(am__test_driver)', used to call it. Perform new substitution on `DRIVER' when processing the `check2.am' file. * lib/check.am (am__tty_colors): Define new shell variable `$am__color_tests'. (am__rst_section): Removed, its role taken over by the new `test-driver' script. (am__test_driver_flags): New variable, contains the command line options passed to `test-driver'. (am__check_pre): Do not deal with temporary files and exit traps anymore, as the `test-driver' script takes care of that now. Define shell variable `$am__enable_hard_errors', used by `$(am__test_driver_flags)'. Reorder so that we don't need to save and restore the value of the `TERM' environment variable anymore. Other related adjustments. (am__check_post): Remove, as its role has been completely taken over by the `test-driver' script. * am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log, ?!GENERIC?%OBJ%): Call the test script through the Automake substituted `%DRIVER%', and honor the command-line options in `$(am__test_driver_flags)'. Do not call the obsoleted `$(am__check_post)' anymore. * doc/automake.texi (Auxiliary Programs): Mention the new `test-driver' script. (Optional): Mention `test-driver' in AC_CONFIG_AUX_DIR. Since we are at it, break the list of auxiliary scripts by placing one per line, to simplify potential future additions of new scripts. * tests/check.test: Adjust. * tests/check2.test : Likewise. * tests/check3.test : Likewise. * tests/check4.test : Likewise. * tests/check10.test: Likewise. * tests/color.test: Likewise. * tests/color2.test: Likewise. * tests/comment9.test: Likewise. * tests/dejagnu.test: Likewise. * tests/exeext4.test: Likewise. * tests/maken3.test: Likewise. * tests/maken4.test: Likewise. * tests/parallel-tests-interrupt.test: Likewise. * tests/posixsubst-tests.test: Likewise. * tests/repeated-options.test: Likewise. * tests/check-no-test-driver.test: New test. * tests/parallel-test-driver-install.test: Likewise. * tests/Makefile.am (TESTS): Update. * NEWS: Update. --- diff --git a/ChangeLog b/ChangeLog index 8cb078ae9..81a54d285 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,68 @@ +2011-06-21 Stefano Lattarini + + parallel-tests: add auxiliary script 'test-driver', refactor + This refactoring should cause no API of functionality change, + and is meant only to simplify the future implementation of TAP + and SubUnit testsuite drivers. More precisely, our roadmap is + to move most of the "testsuite driving" features out of the + Automake-generated Makefiles, and into external scripts with + well-defined interfaces. This will allow the user to define + its own personalized testsuite drivers, and will also offer us + a framework upon which to implement our new TAP and SubUnit + drivers, all in a very unobtrusive way and retaining an high + degree of code reuse and backward-compatibility. + * lib/test-driver: New auxiliary script. + * lib/Makefile.am (dist_SCRIPT_DATA): Add it. + * automake.in (handle_tests): Require the new auxiliary script + `test-driver', and define a new internal makefile variable + `$(am__test_driver)', used to call it. Perform new substitution + on `DRIVER' when processing the `check2.am' file. + * lib/check.am (am__tty_colors): Define new shell variable + `$am__color_tests'. + (am__rst_section): Removed, its role taken over by the new + `test-driver' script. + (am__test_driver_flags): New variable, contains the command + line options passed to `test-driver'. + (am__check_pre): Do not deal with temporary files and exit + traps anymore, as the `test-driver' script takes care of that + now. Define shell variable `$am__enable_hard_errors', used by + `$(am__test_driver_flags)'. Reorder so that we don't need to + save and restore the value of the `TERM' environment variable + anymore. + Other related adjustments. + (am__check_post): Remove, as its role has been completely taken + over by the `test-driver' script. + * am/check2.am (?GENERIC?%EXT%$(EXEEXT).log, ?GENERIC?%EXT%.log, + ?!GENERIC?%OBJ%): Call the test script through the Automake + substituted `%DRIVER%', and honor the command-line options + in `$(am__test_driver_flags)'. Do not call the obsoleted + `$(am__check_post)' anymore. + * doc/automake.texi (Auxiliary Programs): Mention the new + `test-driver' script. + (Optional): Mention `test-driver' in AC_CONFIG_AUX_DIR. + Since we are at it, break the list of auxiliary scripts by + placing one per line, to simplify potential future additions + of new scripts. + * tests/check.test: Adjust. + * tests/check2.test : Likewise. + * tests/check3.test : Likewise. + * tests/check4.test : Likewise. + * tests/check10.test: Likewise. + * tests/color.test: Likewise. + * tests/color2.test: Likewise. + * tests/comment9.test: Likewise. + * tests/dejagnu.test: Likewise. + * tests/exeext4.test: Likewise. + * tests/maken3.test: Likewise. + * tests/maken4.test: Likewise. + * tests/parallel-tests-interrupt.test: Likewise. + * tests/posixsubst-tests.test: Likewise. + * tests/repeated-options.test: Likewise. + * tests/check-no-test-driver.test: New test. + * tests/parallel-test-driver-install.test: Likewise. + * tests/Makefile.am (TESTS): Update. + * NEWS: Update. + 2011-06-21 Stefano Lattarini maintcheck: extend 'sc_tests_plain_*' checks diff --git a/NEWS b/NEWS index 35b223aab..41d0dfe58 100644 --- a/NEWS +++ b/NEWS @@ -11,6 +11,47 @@ New in 1.11a: - New `cscope' target to build a cscope database for the source tree. +* Changes to Automake-generated testsuite harnesses: + + - The parallel-tests driver is now implemented (partly at least) with + the help of automake-provided auxiliary scripts (e.g., `test-driver'), + instead of relying entirely on code in the generated Makefile.in. + This has two noteworthy implications. The first one is that projects + using the `parallel-tests' option should now either run automake with + the `--add-missing' option, or manually copy the `test-driver' script + into their tree. The second, and more important, implication is that + now, when the `parallel-tests' option is in use, TESTS_ENVIRONMENT can + not be used anymore to define a test runner, and the command specified + in LOG_COMPILER (and _LOG_COMPILER) must be a *real* executable + program or script. For example, this is still a valid usage (albeit + a little contorted): + + TESTS_ENVIRONMENT = \ + if test -n '$(STRICT_TESTS)'; then \ + maybe_errexit='-e'; \ + else \ + maybe_errexit=''; \ + fi; + LOG_COMPILER = $(SHELL) $$maybe_errexit + + while this is not anymore: + + TESTS_ENVIRONMENT = \ + $(SHELL) `test -n '$(STRICT_TESTS_CHECKING)' && echo ' -e'` + + neither is this: + + TESTS_ENVIRONMENT = \ + run_with_perl_or_shell () \ + { \ + if grep -q '^#!.*perl' $$1; then + $(PERL) $$1; \ + else \ + $(SHELL) $$1; \ + fi; \ + } + LOG_COMPILER = run_with_per_or_shell + * WARNING: Future backward-incompatibilities! - The Automake support for automatic de-ANSI-fication will be removed in diff --git a/automake.in b/automake.in index 782768022..e1c7a9231 100644 --- a/automake.in +++ b/automake.in @@ -4989,8 +4989,13 @@ sub handle_tests append_exeext { exists $known_programs{$_[0]} } 'XFAIL_TESTS' if (var ('XFAIL_TESTS')); - if (option 'parallel-tests') + if (my $parallel_tests = option 'parallel-tests') { + require_conf_file ($parallel_tests->{position}, FOREIGN, + 'test-driver'); + define_variable ('am__test_driver', + "\$(SHELL) $am_config_aux_dir/test-driver", + INTERNAL); define_variable ('TEST_SUITE_LOG', 'test-suite.log', INTERNAL); define_variable ('TEST_SUITE_HTML', '$(TEST_SUITE_LOG:.log=.html)', INTERNAL); my $suff = '.test'; @@ -5041,6 +5046,7 @@ sub handle_tests GENERIC => 0, OBJ => $obj, SOURCE => $val, + DRIVER => '$(am__test_driver)', COMPILE =>'$(' . $compile . ')', EXT => '', am__EXEEXT => 'FALSE'); @@ -5080,6 +5086,7 @@ sub handle_tests GENERIC => 1, OBJ => '', SOURCE => '$<', + DRIVER => '$(am__test_driver)', COMPILE => '$(' . $compile . ')', EXT => $test_suffix, am__EXEEXT => $am_exeext); diff --git a/doc/automake.texi b/doc/automake.texi index 59ddcfd12..dd3a34005 100644 --- a/doc/automake.texi +++ b/doc/automake.texi @@ -2308,6 +2308,10 @@ instead of copying files. Such an operation is performed when building multilibs (@pxref{Multilibs}). This file is maintained in the GCC tree at @url{http://gcc.gnu.org/svn.html}. +@item test-driver +This implements the default testsuite driver offered by the +@option{parallel-tests} testsuite harness. + @item texinfo.tex Not a program, this file is required for @samp{make dvi}, @samp{make ps} and @samp{make pdf} to work when Texinfo sources are in the @@ -2901,12 +2905,24 @@ The Autoconf Manual}. @item AC_CONFIG_AUX_DIR Automake will look for various helper scripts, such as @file{install-sh}, in the directory named in this macro invocation. -@c This list is accurate relative to version 1.8 -(The full list of scripts is: @file{ar-lib}, @file{config.guess}, -@file{config.sub}, @file{depcomp}, @file{elisp-comp}, @file{compile}, -@file{install-sh}, @file{ltmain.sh}, @file{mdate-sh}, @file{missing}, -@file{mkinstalldirs}, @file{py-compile}, @file{texinfo.tex}, and -@file{ylwrap}.) Not all scripts are always searched for; some scripts +@c This list is accurate relative to version 1.11 +(The full list of scripts is: +@file{ar-lib}, +@file{config.guess}, +@file{config.sub}, +@file{depcomp}, +@file{elisp-comp}, +@file{compile}, +@file{install-sh}, +@file{ltmain.sh}, +@file{mdate-sh}, +@file{missing}, +@file{mkinstalldirs}, +@file{py-compile}, +@file{test-driver}, +@file{texinfo.tex}, +@file{ylwrap}.) +Not all scripts are always searched for; some scripts will only be sought if the generated @file{Makefile.in} requires them. If @code{AC_CONFIG_AUX_DIR} is not given, the scripts are looked for in diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in index 67a074775..6ae57f18d 100644 --- a/lib/Automake/tests/Makefile.in +++ b/lib/Automake/tests/Makefile.in @@ -75,11 +75,13 @@ DIST_SOURCES = # then this fails; a conservative approach. Of course do not redirect # stdout here, just stderr. am__tty_colors = \ +am__color_tests=no; \ red=; grn=; lgn=; blu=; std=; \ test "X$(AM_COLOR_TESTS)" != Xno \ && test "X$$TERM" != Xdumb \ && { test "X$(AM_COLOR_TESTS)" = Xalways || test -t 1 2>/dev/null; } \ && { \ + am__color_tests=yes; \ red=''; \ grn=''; \ lgn=''; \ @@ -107,9 +109,8 @@ am__nobase_list = $(am__nobase_strip_setup); \ am__base_list = \ sed '$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;s/\n/ /g' | \ sed '$$!N;$$!N;$$!N;$$!N;s/\n/ /g' -# Restructured Text title and section. +# Restructured Text title. am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//' -am__rst_section = sed 'p;s/./=/g;p;g' # Put stdin (possibly several lines separated by ". ") in a box. # Prefix each line by 'col' and terminate each with 'std', for coloring. # Multi line coloring is problematic with "less -R", so we really need @@ -131,60 +132,46 @@ am__text_box = $(AWK) '{ \ # pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it # by disabling -e (using the XSI extension "set +e") if it's set. am__sh_e_setup = case $$- in *e*) set +e;; esac +# Default flags passed to all log compiler wrappers. +am__test_driver_flags = \ + --test-name "$$f" \ + --log-file '$@' \ + --color-tests "$$am__color_tests" \ + --enable-hard-errors "$$am__enable_hard_errors" \ + --expect-failure "$$am__expect_failure" # To be inserted before the command running the test. Creates the # directory for the log if needed. Stores in $dir the directory # containing $f, in $tst the test, in $log the log. Executes the # developer- defined test setup AM_TESTS_ENVIRONMENT (if any), and -# passes TESTS_ENVIRONMENT. Saves and restores TERM around uses of -# TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT, in case any of them -# unsets it. +# passes TESTS_ENVIRONMENT. Set up options for the wrapper that +# will run the test scripts (or their associated LOG_COMPILER, if +# thy have one). am__check_pre = \ $(am__sh_e_setup); \ $(am__vpath_adj_setup) $(am__vpath_adj) \ +$(am__tty_colors); \ srcdir=$(srcdir); export srcdir; \ -rm -f $@-t; \ -am__trap='rm -f '\''$(abs_builddir)/$@-t'\''; (exit $$st); exit $$st'; \ -trap "st=129; $$am__trap" 1; trap "st=130; $$am__trap" 2; \ -trap "st=141; $$am__trap" 13; trap "st=143; $$am__trap" 15; \ am__odir=`echo "./$@" | sed 's|/[^/]*$$||'`; \ test "x$$am__odir" = x. || $(MKDIR_P) "$$am__odir" || exit $$?; \ if test -f "./$$f"; then dir=./; \ elif test -f "$$f"; then dir=; \ else dir="$(srcdir)/"; fi; \ -tst=$$dir$$f; log='$@'; __SAVED_TERM=$$TERM; \ -$(AM_TESTS_ENVIRONMENT) \ -$(TESTS_ENVIRONMENT) -# To be appended to the command running the test. Handle the stdout -# and stderr redirection, and catch the exit status. -am__check_post = \ ->$@-t 2>&1; \ -estatus=$$?; \ -if test -n '$(DISABLE_HARD_ERRORS)' \ - && test $$estatus -eq 99; then \ - estatus=1; \ -fi; \ -TERM=$$__SAVED_TERM; export TERM; \ -$(am__tty_colors); \ -xfailed=PASS; \ +tst=$$dir$$f; log='$@'; \ +if test -n '$(DISABLE_HARD_ERRORS)'; then \ + am__enable_hard_errors=no; \ +else \ + am__enable_hard_errors=yes; \ +fi; \ case " $(XFAIL_TESTS) " in \ *[\ \ ]$$f[\ \ ]* | *[\ \ ]$$dir$$f[\ \ ]*) \ - xfailed=XFAIL;; \ -esac; \ -case $$estatus.$$xfailed in \ - 0.XFAIL) col=$$red; res=XPASS;; \ - 0.*) col=$$grn; res=PASS ;; \ - 77.*) col=$$blu; res=SKIP ;; \ - 99.*) col=$$red; res=FAIL ;; \ - *.XFAIL) col=$$lgn; res=XFAIL;; \ - *.*) col=$$red; res=FAIL ;; \ -esac; \ -echo "$${col}$$res$${std}: $$f"; \ -echo "$$res: $$f (exit: $$estatus)" | \ - $(am__rst_section) >$@; \ -cat $@-t >>$@; \ -rm -f $@-t + am__expect_failure=yes;; \ + *) \ + am__expect_failure=no;; \ +esac; \ +$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT) RECHECK_LOGS = $(TEST_LOGS) AM_RECURSIVE_TARGETS = check check-html recheck recheck-html +am__test_driver = $(SHELL) $(top_srcdir)/lib/test-driver TEST_SUITE_LOG = test-suite.log TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html) am__test_logs1 = $(TESTS:=.log) @@ -479,7 +466,9 @@ recheck recheck-html: list=`echo "$$list" | sed 's/ *$$//'`; \ $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"' .pl.log: - @p='$<'; $(am__check_pre) $(PL_LOG_COMPILE) "$$tst" $(am__check_post) + @p='$<'; $(am__check_pre) \ + $(am__test_driver) $(am__test_driver_flags) -- \ + $(PL_LOG_COMPILE) "$$tst" distdir: $(DISTFILES) @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ diff --git a/lib/Makefile.am b/lib/Makefile.am index 5bdc02e13..a26d6633e 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -29,7 +29,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c ansi2knr.1 \ scriptdir = $(pkgvdatadir) dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \ mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \ - symlink-tree ar-lib + symlink-tree ar-lib test-driver EXTRA_DIST = gnupload diff --git a/lib/Makefile.in b/lib/Makefile.in index 91e9baef9..5f8aa913b 100644 --- a/lib/Makefile.in +++ b/lib/Makefile.in @@ -242,7 +242,7 @@ dist_pkgvdata_DATA = COPYING INSTALL texinfo.tex ansi2knr.c ansi2knr.1 \ scriptdir = $(pkgvdatadir) dist_script_DATA = config.guess config.sub install-sh mdate-sh missing \ mkinstalldirs elisp-comp ylwrap acinstall depcomp compile py-compile \ - symlink-tree ar-lib + symlink-tree ar-lib test-driver EXTRA_DIST = gnupload all: all-recursive diff --git a/lib/am/check.am b/lib/am/check.am index 82ba7ce8c..6ddf61f4c 100644 --- a/lib/am/check.am +++ b/lib/am/check.am @@ -20,11 +20,13 @@ if %?COLOR% # then this fails; a conservative approach. Of course do not redirect # stdout here, just stderr. am__tty_colors = \ +am__color_tests=no; \ red=; grn=; lgn=; blu=; std=; \ test "X$(AM_COLOR_TESTS)" != Xno \ && test "X$$TERM" != Xdumb \ && { test "X$(AM_COLOR_TESTS)" = Xalways || test -t 1 2>/dev/null; } \ && { \ + am__color_tests=yes; \ red=''; \ grn=''; \ lgn=''; \ @@ -32,8 +34,7 @@ test "X$(AM_COLOR_TESTS)" != Xno \ std=''; \ } else !%?COLOR% -am__tty_colors = \ -red=; grn=; lgn=; blu=; std= +am__tty_colors = red= grn= lgn= blu= std= am__color_tests=no endif !%?COLOR% .PHONY: check-TESTS @@ -70,9 +71,8 @@ include inst-vars.am ## test is XFAIL or not. You can disable this feature by setting the ## variable DISABLE_HARD_ERRORS to a nonempty value. -# Restructured Text title and section. -am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//' -am__rst_section = sed 'p;s/./=/g;p;g' +# Restructured Text title. +am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//' # Put stdin (possibly several lines separated by ". ") in a box. # Prefix each line by 'col' and terminate each with 'std', for coloring. @@ -97,59 +97,46 @@ am__text_box = $(AWK) '{ \ # by disabling -e (using the XSI extension "set +e") if it's set. am__sh_e_setup = case $$- in *e*) set +e;; esac +# Default flags passed to all log compiler wrappers. +am__test_driver_flags = \ + --test-name "$$f" \ + --log-file '$@' \ + --color-tests "$$am__color_tests" \ + --enable-hard-errors "$$am__enable_hard_errors" \ + --expect-failure "$$am__expect_failure" + # To be inserted before the command running the test. Creates the # directory for the log if needed. Stores in $dir the directory # containing $f, in $tst the test, in $log the log. Executes the # developer- defined test setup AM_TESTS_ENVIRONMENT (if any), and -# passes TESTS_ENVIRONMENT. Saves and restores TERM around uses of -# TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT, in case any of them -# unsets it. +# passes TESTS_ENVIRONMENT. Set up options for the wrapper that +# will run the test scripts (or their associated LOG_COMPILER, if +# thy have one). am__check_pre = \ $(am__sh_e_setup); \ $(am__vpath_adj_setup) $(am__vpath_adj) \ +$(am__tty_colors); \ srcdir=$(srcdir); export srcdir; \ -rm -f $@-t; \ -am__trap='rm -f '\''$(abs_builddir)/$@-t'\''; (exit $$st); exit $$st'; \ -trap "st=129; $$am__trap" 1; trap "st=130; $$am__trap" 2; \ -trap "st=141; $$am__trap" 13; trap "st=143; $$am__trap" 15; \ am__odir=`echo "./$@" | sed 's|/[^/]*$$||'`; \ test "x$$am__odir" = x. || $(MKDIR_P) "$$am__odir" || exit $$?; \ if test -f "./$$f"; then dir=./; \ elif test -f "$$f"; then dir=; \ else dir="$(srcdir)/"; fi; \ -tst=$$dir$$f; log='$@'; __SAVED_TERM=$$TERM; \ -$(AM_TESTS_ENVIRONMENT) \ -$(TESTS_ENVIRONMENT) - -# To be appended to the command running the test. Handle the stdout -# and stderr redirection, and catch the exit status. -am__check_post = \ ->$@-t 2>&1; \ -estatus=$$?; \ -if test -n '$(DISABLE_HARD_ERRORS)' \ - && test $$estatus -eq 99; then \ - estatus=1; \ -fi; \ -TERM=$$__SAVED_TERM; export TERM; \ -$(am__tty_colors); \ -xfailed=PASS; \ +tst=$$dir$$f; log='$@'; \ +if test -n '$(DISABLE_HARD_ERRORS)'; then \ + am__enable_hard_errors=no; \ +else \ + am__enable_hard_errors=yes; \ +fi; \ +## The use of $dir below is required to account for VPATH +## rewriting done by Sun make. case " $(XFAIL_TESTS) " in \ *[\ \ ]$$f[\ \ ]* | *[\ \ ]$$dir$$f[\ \ ]*) \ - xfailed=XFAIL;; \ -esac; \ -case $$estatus.$$xfailed in \ - 0.XFAIL) col=$$red; res=XPASS;; \ - 0.*) col=$$grn; res=PASS ;; \ - 77.*) col=$$blu; res=SKIP ;; \ - 99.*) col=$$red; res=FAIL ;; \ - *.XFAIL) col=$$lgn; res=XFAIL;; \ - *.*) col=$$red; res=FAIL ;; \ -esac; \ -echo "$${col}$$res$${std}: $$f"; \ -echo "$$res: $$f (exit: $$estatus)" | \ - $(am__rst_section) >$@; \ -cat $@-t >>$@; \ -rm -f $@-t + am__expect_failure=yes;; \ + *) \ + am__expect_failure=no;; \ +esac; \ +$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT) $(TEST_SUITE_LOG): $(TEST_LOGS) @$(am__sh_e_setup); \ diff --git a/lib/am/check2.am b/lib/am/check2.am index 054c62dd7..f3116c8d1 100644 --- a/lib/am/check2.am +++ b/lib/am/check2.am @@ -1,5 +1,5 @@ ## automake - create Makefile.in from Makefile.am -## Copyright (C) 2008, 2009 Free Software Foundation, Inc. +## Copyright (C) 2008, 2009, 2011 Free Software Foundation, Inc. ## This program is free software; you can redistribute it and/or modify ## it under the terms of the GNU General Public License as published by @@ -17,7 +17,9 @@ ## From a test file to a log file. ?GENERIC?%EXT%.log: ?!GENERIC?%OBJ%: %SOURCE% - @p='%SOURCE%'; $(am__check_pre) %COMPILE% "$$tst" $(am__check_post) + @p='%SOURCE%'; $(am__check_pre) \ + %DRIVER% $(am__test_driver_flags) -- \ + %COMPILE% "$$tst" ## If no programs are built in this package, then this rule is removed ## at automake time. Otherwise, %am__EXEEXT% expands to a configure time @@ -25,5 +27,7 @@ ## conflict with the previous one. if %am__EXEEXT% ?GENERIC?%EXT%$(EXEEXT).log: - @p='%SOURCE%'; $(am__check_pre) %COMPILE% "$$tst" $(am__check_post) + @p='%SOURCE%'; $(am__check_pre) \ + %DRIVER% $(am__test_driver_flags) -- \ + %COMPILE% "$$tst" endif %am__EXEEXT% diff --git a/lib/test-driver b/lib/test-driver new file mode 100755 index 000000000..87533240c --- /dev/null +++ b/lib/test-driver @@ -0,0 +1,129 @@ +#! /bin/sh +# test-driver - basic driver script for the `parallel-tests' mode. + +scriptversion=2011-06-21.19; # UTC + +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# As a special exception to the GNU General Public License, if you +# distribute this file as part of a program that contains a +# configuration script generated by Autoconf, you may include it under +# the same distribution terms that you use for the rest of that program. + +# This file is maintained in Automake, please report +# bugs to or send patches to +# . + +# Make unconditional expansion of undefined variables an error. This +# helps a lot in preventing typo-related bugs. +set -u + +fatal () +{ + echo "$0: fatal: $*" >&2 + exit 1 +} + +usage_error () +{ + echo "$0: $*" >&2 + print_usage >&2 + exit 2 +} + +print_usage () +{ + cat <$tmpfile 2>&1 +estatus=$? +if test $enable_hard_errors = no && test $estatus -eq 99; then + estatus=1 +fi + +case $estatus:$expect_failure in + 0:yes) col=$red; res=XPASS;; + 0:*) col=$grn; res=PASS ;; + 77:*) col=$blu; res=SKIP ;; + 99:*) col=$red; res=FAIL ;; + *:yes) col=$lgn; res=XFAIL;; + *:*) col=$red; res=FAIL ;; +esac +echo "${col}${res}${std}: $test_name" +echo "$res: $test_name (exit: $estatus)" | rst_section > $logfile +cat $tmpfile >> $logfile +rm -f $tmpfile + +# Local Variables: +# mode: shell-script +# sh-indentation: 2 +# eval: (add-hook 'write-file-hooks 'time-stamp) +# time-stamp-start: "scriptversion=" +# time-stamp-format: "%:y-%02m-%02d.%02H" +# time-stamp-time-zone: "UTC" +# time-stamp-end: "; # UTC" +# End: diff --git a/tests/Makefile.am b/tests/Makefile.am index 6060a6215..db43a41f9 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -235,6 +235,7 @@ check12.test \ check-exported-srcdir.test \ check-tests-in-builddir.test \ check-tests_environment.test \ +check-no-test-driver.test \ checkall.test \ clean.test \ clean2.test \ @@ -722,6 +723,7 @@ parallel-tests-unreadable-log.test \ parallel-tests-subdir.test \ parallel-tests-interrupt.test \ parallel-tests-reset-term.test \ +parallel-test-driver-install.test \ parse.test \ percent.test \ percent2.test \ diff --git a/tests/Makefile.in b/tests/Makefile.in index 3b7a63e90..24c508561 100644 --- a/tests/Makefile.in +++ b/tests/Makefile.in @@ -80,11 +80,13 @@ DIST_SOURCES = # then this fails; a conservative approach. Of course do not redirect # stdout here, just stderr. am__tty_colors = \ +am__color_tests=no; \ red=; grn=; lgn=; blu=; std=; \ test "X$(AM_COLOR_TESTS)" != Xno \ && test "X$$TERM" != Xdumb \ && { test "X$(AM_COLOR_TESTS)" = Xalways || test -t 1 2>/dev/null; } \ && { \ + am__color_tests=yes; \ red=''; \ grn=''; \ lgn=''; \ @@ -112,9 +114,8 @@ am__nobase_list = $(am__nobase_strip_setup); \ am__base_list = \ sed '$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;$$!N;s/\n/ /g' | \ sed '$$!N;$$!N;$$!N;$$!N;s/\n/ /g' -# Restructured Text title and section. +# Restructured Text title. am__rst_title = sed 's/.*/ & /;h;s/./=/g;p;x;p;g;p;s/.*//' -am__rst_section = sed 'p;s/./=/g;p;g' # Put stdin (possibly several lines separated by ". ") in a box. # Prefix each line by 'col' and terminate each with 'std', for coloring. # Multi line coloring is problematic with "less -R", so we really need @@ -136,60 +137,46 @@ am__text_box = $(AWK) '{ \ # pass "-e" to $(SHELL), and POSIX 2008 even requires this. Work around it # by disabling -e (using the XSI extension "set +e") if it's set. am__sh_e_setup = case $$- in *e*) set +e;; esac +# Default flags passed to all log compiler wrappers. +am__test_driver_flags = \ + --test-name "$$f" \ + --log-file '$@' \ + --color-tests "$$am__color_tests" \ + --enable-hard-errors "$$am__enable_hard_errors" \ + --expect-failure "$$am__expect_failure" # To be inserted before the command running the test. Creates the # directory for the log if needed. Stores in $dir the directory # containing $f, in $tst the test, in $log the log. Executes the # developer- defined test setup AM_TESTS_ENVIRONMENT (if any), and -# passes TESTS_ENVIRONMENT. Saves and restores TERM around uses of -# TESTS_ENVIRONMENT and AM_TESTS_ENVIRONMENT, in case any of them -# unsets it. +# passes TESTS_ENVIRONMENT. Set up options for the wrapper that +# will run the test scripts (or their associated LOG_COMPILER, if +# thy have one). am__check_pre = \ $(am__sh_e_setup); \ $(am__vpath_adj_setup) $(am__vpath_adj) \ +$(am__tty_colors); \ srcdir=$(srcdir); export srcdir; \ -rm -f $@-t; \ -am__trap='rm -f '\''$(abs_builddir)/$@-t'\''; (exit $$st); exit $$st'; \ -trap "st=129; $$am__trap" 1; trap "st=130; $$am__trap" 2; \ -trap "st=141; $$am__trap" 13; trap "st=143; $$am__trap" 15; \ am__odir=`echo "./$@" | sed 's|/[^/]*$$||'`; \ test "x$$am__odir" = x. || $(MKDIR_P) "$$am__odir" || exit $$?; \ if test -f "./$$f"; then dir=./; \ elif test -f "$$f"; then dir=; \ else dir="$(srcdir)/"; fi; \ -tst=$$dir$$f; log='$@'; __SAVED_TERM=$$TERM; \ -$(AM_TESTS_ENVIRONMENT) \ -$(TESTS_ENVIRONMENT) -# To be appended to the command running the test. Handle the stdout -# and stderr redirection, and catch the exit status. -am__check_post = \ ->$@-t 2>&1; \ -estatus=$$?; \ -if test -n '$(DISABLE_HARD_ERRORS)' \ - && test $$estatus -eq 99; then \ - estatus=1; \ -fi; \ -TERM=$$__SAVED_TERM; export TERM; \ -$(am__tty_colors); \ -xfailed=PASS; \ +tst=$$dir$$f; log='$@'; \ +if test -n '$(DISABLE_HARD_ERRORS)'; then \ + am__enable_hard_errors=no; \ +else \ + am__enable_hard_errors=yes; \ +fi; \ case " $(XFAIL_TESTS) " in \ *[\ \ ]$$f[\ \ ]* | *[\ \ ]$$dir$$f[\ \ ]*) \ - xfailed=XFAIL;; \ -esac; \ -case $$estatus.$$xfailed in \ - 0.XFAIL) col=$$red; res=XPASS;; \ - 0.*) col=$$grn; res=PASS ;; \ - 77.*) col=$$blu; res=SKIP ;; \ - 99.*) col=$$red; res=FAIL ;; \ - *.XFAIL) col=$$lgn; res=XFAIL;; \ - *.*) col=$$red; res=FAIL ;; \ -esac; \ -echo "$${col}$$res$${std}: $$f"; \ -echo "$$res: $$f (exit: $$estatus)" | \ - $(am__rst_section) >$@; \ -cat $@-t >>$@; \ -rm -f $@-t + am__expect_failure=yes;; \ + *) \ + am__expect_failure=no;; \ +esac; \ +$(AM_TESTS_ENVIRONMENT) $(TESTS_ENVIRONMENT) RECHECK_LOGS = $(TEST_LOGS) AM_RECURSIVE_TARGETS = check check-html recheck recheck-html +am__test_driver = $(SHELL) $(top_srcdir)/lib/test-driver TEST_SUITE_LOG = test-suite.log TEST_SUITE_HTML = $(TEST_SUITE_LOG:.log=.html) TEST_EXTENSIONS = .test @@ -508,6 +495,7 @@ check12.test \ check-exported-srcdir.test \ check-tests-in-builddir.test \ check-tests_environment.test \ +check-no-test-driver.test \ checkall.test \ clean.test \ clean2.test \ @@ -995,6 +983,7 @@ parallel-tests-unreadable-log.test \ parallel-tests-subdir.test \ parallel-tests-interrupt.test \ parallel-tests-reset-term.test \ +parallel-test-driver-install.test \ parse.test \ percent.test \ percent2.test \ @@ -1508,7 +1497,9 @@ recheck recheck-html: list=`echo "$$list" | sed 's/ *$$//'`; \ $(MAKE) $(AM_MAKEFLAGS) $$target AM_MAKEFLAGS='$(AM_MAKEFLAGS) TEST_LOGS="'"$$list"'"' .test.log: - @p='$<'; $(am__check_pre) $(TEST_LOG_COMPILE) "$$tst" $(am__check_post) + @p='$<'; $(am__check_pre) \ + $(am__test_driver) $(am__test_driver_flags) -- \ + $(TEST_LOG_COMPILE) "$$tst" distdir: $(DISTFILES) @srcdirstrip=`echo "$(srcdir)" | sed 's/[].[^$$\\*]/\\\\&/g'`; \ diff --git a/tests/check-no-test-driver.test b/tests/check-no-test-driver.test new file mode 100755 index 000000000..fc1e89b99 --- /dev/null +++ b/tests/check-no-test-driver.test @@ -0,0 +1,34 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check that auxiliary script 'test-driver' doesn't get needlessly +# installed or referenced when the 'parallel-tests' option is not +# used. + +parallel_tests=no +. ./defs || Exit 1 + +echo 'TESTS = foo.test' > Makefile.am + +$ACLOCAL + +for opts in '' '-a' '--add-missing --copy'; do + $AUTOMAKE $opts + $FGREP 'test-driver' Makefile.in && Exit 1 + find . | $FGREP 'test-driver' && Exit 1 +done + +: diff --git a/tests/check.test b/tests/check.test index 54432eca9..caacf7845 100755 --- a/tests/check.test +++ b/tests/check.test @@ -22,6 +22,8 @@ cat > Makefile.am << 'END' TESTS = frob.test END +test x"$parallel_tests" != x"yes" || : > test-driver + : > frob.test $ACLOCAL diff --git a/tests/check10.test b/tests/check10.test index 7025b9863..ef799691d 100755 --- a/tests/check10.test +++ b/tests/check10.test @@ -50,7 +50,7 @@ cp skip skip2 $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a ./configure ( diff --git a/tests/check2.test b/tests/check2.test index 9b8bfb1b5..56e2614b8 100755 --- a/tests/check2.test +++ b/tests/check2.test @@ -43,6 +43,10 @@ echo.sh: CLEANFILES = echo.sh END +if test x"$parallel_tests" = x"yes"; then + cp "$top_testsrcdir"/lib/test-driver . +fi + $ACLOCAL $AUTOCONF $AUTOMAKE diff --git a/tests/check3.test b/tests/check3.test index be40e3fae..ccfb9236d 100755 --- a/tests/check3.test +++ b/tests/check3.test @@ -54,7 +54,7 @@ END $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a ./configure --prefix "`pwd`/inst" $MAKE check >stdout cat stdout diff --git a/tests/check4.test b/tests/check4.test index 99adb23b7..7c40c0c0b 100755 --- a/tests/check4.test +++ b/tests/check4.test @@ -46,8 +46,10 @@ chmod +x ok.sh dir/fail.sh $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE --add-missing + ./configure --prefix "`pwd`/inst" + $MAKE check >stdout && { cat stdout; Exit 1; } cat stdout grep 'FAIL: fail.sh' stdout diff --git a/tests/color.test b/tests/color.test index 570b21d66..ecd4f94d0 100755 --- a/tests/color.test +++ b/tests/color.test @@ -67,8 +67,9 @@ cp pass xpass chmod +x pass fail skip xpass xfail $ACLOCAL -$AUTOMAKE $AUTOCONF +$AUTOMAKE --add-missing + ./configure test_color () @@ -95,3 +96,5 @@ test_no_color () AM_COLOR_TESTS=always $MAKE -e check >stdout && { cat stdout; Exit 1; } cat stdout test_color + +: diff --git a/tests/color2.test b/tests/color2.test index eb9c52788..4f8862e20 100755 --- a/tests/color2.test +++ b/tests/color2.test @@ -80,7 +80,7 @@ cp pass xpass chmod +x pass fail skip xpass xfail $ACLOCAL -$AUTOMAKE +$AUTOMAKE -a $AUTOCONF ./configure @@ -120,3 +120,5 @@ AM_COLOR_TESTS=no MAKE=$MAKE expect -f expect-make >stdout \ || { cat stdout; Exit 1; } cat stdout test_no_color + +: diff --git a/tests/comment9.test b/tests/comment9.test index e95d99be0..e20b59f9e 100755 --- a/tests/comment9.test +++ b/tests/comment9.test @@ -30,6 +30,8 @@ TESTS = \ 7.test EOF +: > test-driver + $ACLOCAL $AUTOMAKE diff --git a/tests/dejagnu.test b/tests/dejagnu.test index df385b033..fe994b5d9 100755 --- a/tests/dejagnu.test +++ b/tests/dejagnu.test @@ -25,6 +25,8 @@ AUTOMAKE_OPTIONS = dejagnu TESTS = frob.test END +test x"$parallel_tests" != x"yes" || : > test-driver + $ACLOCAL $AUTOMAKE diff --git a/tests/exeext4.test b/tests/exeext4.test index c3f968c42..2ec3bca61 100755 --- a/tests/exeext4.test +++ b/tests/exeext4.test @@ -50,7 +50,7 @@ END $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE --add-missing --copy ./configure $MAKE print-bin > output cat output diff --git a/tests/maken3.test b/tests/maken3.test index 5a77f1a50..eb54d35e2 100755 --- a/tests/maken3.test +++ b/tests/maken3.test @@ -149,7 +149,7 @@ check_targets () done } -$AUTOMAKE -Wno-override +$AUTOMAKE -a -Wno-override ./configure check_targets || Exit 1 diff --git a/tests/maken4.test b/tests/maken4.test index 5c2194371..4a1191094 100755 --- a/tests/maken4.test +++ b/tests/maken4.test @@ -154,7 +154,7 @@ check_targets () done } -$AUTOMAKE -Wno-override +$AUTOMAKE -a -Wno-override ./configure check_targets || Exit 1 diff --git a/tests/parallel-test-driver-install.test b/tests/parallel-test-driver-install.test new file mode 100755 index 000000000..10455fb61 --- /dev/null +++ b/tests/parallel-test-driver-install.test @@ -0,0 +1,93 @@ +#! /bin/sh +# Copyright (C) 2011 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +# Check that auxiliary script 'test-driver' gets automatically installed +# in the correct directory by 'parallel-tests' option. + +parallel_tests=yes +. ./defs || Exit 1 + +: Try first with parallel-tests defined in AM_INIT_AUTOMAKE. + +mkdir am-init-automake +cd am-init-automake + +cat > configure.in < Makefile.am < sub/Makefile.am <stderr || { cat stderr >&2; Exit 1; } +cat stderr >&2 + +ls -l . sub my_aux_dir # For debugging. +test -f my_aux_dir/test-driver +test ! -r test-driver +test ! -r sub/test-driver + +grep '^configure\.in:3:.*installing.*my_aux_dir/test-driver' stderr + +cd .. + +: Now try with parallel-tests defined in AUTOMAKE_OPTIONS. + +mkdir automake-options +cd automake-options + +cat > configure.in < dir/GNUmakefile.am <stderr \ + || { cat stderr >&2; Exit 1; } +cat stderr >&2 + +ls -l . dir build-aux # For debugging. +test -f build-aux/test-driver +test ! -r test-driver +test ! -r dir/test-driver + +grep '^dir/GNUmakefile\.am:2:.*installing.*build-aux/test-driver' stderr + +: diff --git a/tests/parallel-tests-interrupt.test b/tests/parallel-tests-interrupt.test index 45e55d4ae..1c5d9ce05 100755 --- a/tests/parallel-tests-interrupt.test +++ b/tests/parallel-tests-interrupt.test @@ -28,12 +28,20 @@ END cat > Makefile.am << 'END' TESTS = foo.test -## Ugly, but required by foo.test. See below. -TEST_LOG_COMPILER = echo $$$$ > pid && exec 9>&2 && $(SHELL) -x +## Provide more debugging info. +TEST_LOG_COMPILER = $(SHELL) -x +## Rut required by foo.test; see below. +AM_TESTS_ENVIRONMENT = 9>&2 END # This is hacky and ugly, but has the great advantage of avoiding us a lot # of pain with background processes and related synchronization issues. + +cat - "$top_testsrcdir"/lib/test-driver > test-driver <<'END' +#!/bin/sh +echo $$ > pid +END + cat > foo.test << 'END' #!/bin/sh exec 2>&9 @@ -48,7 +56,7 @@ chmod a+x foo.test $ACLOCAL $AUTOCONF -$AUTOMAKE -a +$AUTOMAKE ./configure @@ -60,6 +68,8 @@ for signum in $trapped_signals; do rm -f pid fail *.log *.log-t env signum=$signum $MAKE check && { ls -l; Exit 1; } ls -l + # These files shouldn't exist, but in case they do, their content might + # provide helpful information about the causes of the failure(s). cat foo.log-t || : cat foo.log || : cat test-suite.log || : diff --git a/tests/posixsubst-tests.test b/tests/posixsubst-tests.test index 26ae69fa5..3f45c72ce 100755 --- a/tests/posixsubst-tests.test +++ b/tests/posixsubst-tests.test @@ -50,7 +50,7 @@ chmod +x foo1.test bary $ACLOCAL $AUTOCONF -$AUTOMAKE +$AUTOMAKE -a ./configure $MAKE check >out 2>&1 && { cat out; Exit 1; } diff --git a/tests/repeated-options.test b/tests/repeated-options.test index c3c2e29a9..13999d9aa 100755 --- a/tests/repeated-options.test +++ b/tests/repeated-options.test @@ -58,7 +58,7 @@ int main (void) } END -cp $testsrcdir/../lib/compile . +cp "$top_testsrcdir"/lib/compile "$top_testsrcdir"/lib/test-driver . $ACLOCAL $AUTOMAKE --foreign --foreign -Wall 2>stderr || { cat stderr >&2; Exit 1; }