From: Tom Krizek <tkrizek@isc.org>
Date: Wed, 15 Jun 2022 13:00:27 +0000 (+0200)
Subject: Report reasons for skipped/xfailed system pytests
X-Git-Tag: v9.19.3~38^2
X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=f6d368167a4a12609e1b6082c445babed8449a18;p=thirdparty%2Fbind9.git

Report reasons for skipped/xfailed system pytests

If skip/xfail is used in pytest, it can have a reason string associated
with it. When evaluating these tests, it can be useful to be able to
differentiate the reason why the test was skipped/xfailed/xpassed,
because there might be multiple possible reasons for that.

The extra options passed to pytest ensure that the string with the
reason appears in the test summary and thus we're able to find the
string with the reason in the log output.

See https://docs.pytest.org/en/7.1.x/how-to/skipping.html for more info
---

diff --git a/bin/tests/system/run.sh.in b/bin/tests/system/run.sh.in
index 87a98bc6718..0341a33f126 100644
--- a/bin/tests/system/run.sh.in
+++ b/bin/tests/system/run.sh.in
@@ -227,7 +227,7 @@ if [ $status -eq 0 ]; then
             if start_servers; then
                 run=$((run+1))
                 test_status=0
-                (cd "$systest" && "$PYTEST" -v "$test" "$@" || echo "$?" > "$test.status") | SYSTESTDIR="$systest" cat_d
+                (cd "$systest" && "$PYTEST" -rsxX -v "$test" "$@" || echo "$?" > "$test.status") | SYSTESTDIR="$systest" cat_d
                 if [ -f "$systest/$test.status" ]; then
                     if [ "$(cat "$systest/$test.status")" = "5" ]; then
                         echowarn "R:$systest:SKIPPED"