Support both pytest and legacy system test runner

The legacy system test framework uses pytest to execute some tests.
Since it'd be quite difficult to convince pytest to decide whether to
include conftest.py (or which ones to include when launching from
subdir), it makes more sense to have a shared conftest.py which is used
by both the legacy test runner invocations of pytest and the new pytest
system test runner. It is ugly, but once we drop support for the legacy
runner, we'll get rid of it.

Properly scope the *port fixtures in order to ensure they'll work as
expected with the new pytest runner. Instead of using "session" (which
means the fixture is only evaluated once for the entire execution of
pytest), use "module" scope, which is evaluated separately for each
module. The legacy runner invoked pytest for each system test
separately, while the new pytest runner is invoked once for all system
tests -- therefore it requires the more fine-grained "module" scope to
for the fixtures to work properly.

Remove python shebang, as conftest.py isn't supposed to be an executable
script.

diff --git a/bin/tests/system/conftest.py b/bin/tests/system/conftest.py
index 96de101c3c7f8c1e3baccf53187b8faf0a8edc9e..009d989ceb7b3ab3730fba85cc202dd4bdc359fc 100644 (file)
--- a/bin/tests/system/conftest.py
+++ b/bin/tests/system/conftest.py
@@ -1,5 +1,3 @@
-#!/usr/bin/python3
-
 # Copyright (C) Internet Systems Consortium, Inc. ("ISC")
 #
 # SPDX-License-Identifier: MPL-2.0
@@ -12,25 +10,43 @@
 # information regarding copyright ownership.
 
 import os
-
 import pytest
 
 
-@pytest.fixture(scope="session")
+# ======================= LEGACY=COMPATIBLE FIXTURES =========================
+# The following fixtures are designed to work with both pytest system test
+# runner and the legacy system test framework.
+
+
+@pytest.fixture(scope="module")
 def named_port():
     return int(os.environ.get("PORT", default=5300))
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="module")
 def named_tlsport():
     return int(os.environ.get("TLSPORT", default=8853))
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="module")
 def named_httpsport():
     return int(os.environ.get("HTTPSPORT", default=4443))
 
 
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="module")
 def control_port():
     return int(os.environ.get("CONTROLPORT", default=9953))
+
+
+# ======================= PYTEST SYSTEM TEST RUNNER ==========================
+# From this point onward, any setting, fixtures or functions only apply to the
+# new pytest runner. Ideally, these would be in a separate file. However, due
+# to how pytest works and how it's used by the legacy runner, the best approach
+# is to have everything in this file to avoid duplication and set the
+# LEGACY_TEST_RUNNER if pytest is executed from the legacy framework.
+#
+# FUTURE: Once legacy runner is no longer supported, remove the env var and
+# don't branch the code.
+
+if os.getenv("LEGACY_TEST_RUNNER", "0") == "0":
+    pass  # will be implemented in followup commits

diff --git a/bin/tests/system/run.sh.in b/bin/tests/system/run.sh.in
index 81038f1eb1d4ae3021697667e67573766c74f267..a0abb1d8dda20f778cc65bbc8fdc78780dd7ee40 100644 (file)
--- a/bin/tests/system/run.sh.in
+++ b/bin/tests/system/run.sh.in
@@ -230,7 +230,7 @@ if [ $status -eq 0 ]; then
             if start_servers; then
                 run=$((run+1))
                 test_status=0
-                (cd "$systest" && "$PYTEST" -rsxX -v "$test" "$@" || echo "$?" > "$test.status") | SYSTESTDIR="$systest" cat_d
+                (cd "$systest" && LEGACY_TEST_RUNNER=1 "$PYTEST" -rsxX -v "$test" "$@" || echo "$?" > "$test.status") | SYSTESTDIR="$systest" cat_d
                 if [ -f "$systest/$test.status" ]; then
                     if [ "$(cat "$systest/$test.status")" = "5" ]; then
                         echowarn "R:$systest:SKIPPED"