test-lib: make '--stress' more bisect-friendly

Let's suppose that a test somehow becomes flaky between 'master' and
'pu', and tends to fail within the first 50 repetitions when run with
'--stress'.  In such a case we could use 'git bisect' to find the
culprit: if the test script fails with '--stress', then the commit is
definitely bad, but if it survives, say, 300 repetitions, then we could
consider it good with reasonable confidence.

Unfortunately, all this could only be done manually, because
'--stress' would run the test script repeatedly for all eternity on a
good commit, and it would exit with success even when it found a
failure on a bad commit.

So let's make '--stress' usable with 'git bisect run':

  - Make it exit with failure if a failure is found.

  - Add the '--stress-limit=<N>' option to repeat the test script
    at most N times in each of the parallel jobs, and exit with
    success when the limit is reached.

And then we could simply run something like:

  $ git bisect start origin/pu master
  $ git bisect run sh -c 'make && cd t &&
                          ./t1234-foo.sh --stress --stress-limit=300'

Sure, as a brand new feature it won't be any useful right now, but in
a release or three most cooking topics will already contain this, so
we could automatically bisect at least newly introduced flakiness.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff --git a/t/README b/t/README
index 11ce7675e32e7042f0ae5ddee1b65a0f6516f175..3aed32124819fe9b5ce7bfec3b330f2ef7a15d3b 100644 (file)
--- a/t/README
+++ b/t/README
@@ -202,6 +202,11 @@ appropriately before running "make".
        '.stress-<nr>' suffix, and the trash directory of the failed
        test job is renamed to end with a '.stress-failed' suffix.
 
+--stress-limit=<N>::
+       When combined with --stress run the test script repeatedly
+       this many times in each of the parallel jobs or until one of
+       them fails, whichever comes first.
+
 You can also set the GIT_TEST_INSTALLED environment variable to
 the bindir of an existing git installation to test that installation.
 You still need to have built this git sandbox, from which various

diff --git a/t/test-lib.sh b/t/test-lib.sh
index a1abb1177a15c85ef7c75f0b9b056a59ff9843e9..77eff04c92ab183c992e871aa34720f2d78014b1 100644 (file)
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -152,6 +152,17 @@ do
                        ;;
                esac
                ;;
+       --stress-limit=*)
+               stress_limit=${opt#--*=}
+               case "$stress_limit" in
+               *[^0-9]*|0*|"")
+                       echo "error: --stress-limit=<N> requires the number of repetitions" >&2
+                       exit 1
+                       ;;
+               *)      # Good.
+                       ;;
+               esac
+               ;;
        *)
                echo "error: unknown test option '$opt'" >&2; exit 1 ;;
        esac
@@ -237,8 +248,10 @@ then
                                exit 1
                        ' TERM INT
 
-                       cnt=0
-                       while ! test -e "$stressfail"
+                       cnt=1
+                       while ! test -e "$stressfail" &&
+                             { test -z "$stress_limit" ||
+                               test $cnt -le $stress_limit ; }
                        do
                                $TEST_SHELL_PATH "$0" "$@" >"$TEST_RESULTS_BASE.stress-$job_nr.out" 2>&1 &
                                test_pid=$!
@@ -261,6 +274,7 @@ then
 
        if test -f "$stressfail"
        then
+               stress_exit=1
                echo "Log(s) of failed test run(s):"
                for failed_job_nr in $(sort -n "$stressfail")
                do