Detect missing system test results

At the end of each system test suite run, the system test framework
collects all existing test.output files from system test subdirectories
and produces bin/tests/system/systests.output from those files.
However, it does not check whether a test.output file was found for
every executed test.  Thus, if the test.output file is accidentally
deleted by the system test itself (e.g. due to an overly broad file
removal wildcard present in clean.sh), its output will not be included
in bin/tests/system/systests.output.  Since the result of each system
test suite run is determined by bin/tests/system/testsummary.sh, which
only operates on the contents of bin/tests/system/systests.output, this
can lead to test failures being ignored.  Fix by ensuring the number of
test results found in bin/tests/system/systests.output is equal to the
number of tests run and triggering a system test suite failure in case
of a discrepancy between these two values.

diff --git a/bin/tests/system/testsummary.sh b/bin/tests/system/testsummary.sh
index 06c356671a64836d27139fe940b08795f76d2df0..1c008cdbad0a8605624d3c96d70d7c29b045fa4d 100644 (file)
--- a/bin/tests/system/testsummary.sh
+++ b/bin/tests/system/testsummary.sh
@@ -56,4 +56,11 @@ if [ -n "${FAILED_TESTS}" ]; then
        status=1
 fi
 
+RESULTS_FOUND=`grep -c 'R:[a-z0-9_-][a-z0-9_-]*:[A-Z][A-Z]*' systests.output`
+TESTS_RUN=`echo "${SUBDIRS}" | wc -w`
+if [ "${RESULTS_FOUND}" -ne "${TESTS_RUN}" ]; then
+       echofail "I:Found ${RESULTS_FOUND} test results, but ${TESTS_RUN} tests were run"
+       status=1
+fi
+
 exit $status