Report hung system tests

At times, a problem might occur where a test is not responding,
especially in the CI, determining the specific test responsible can be
difficult. Fortunately, when running tests with the pytest runner,
pytest sets the PYTEST_CURRENT_TEST environment variable to the current
test nodeid and stage. Afterward, the variable can be examined to
identify the test that has stopped responding.

The monitoring script needs to be started in the background. Still, the
shell executor used for BSD and FIPS testing can't handle the background
process cleanly, and the script step will wait for the background
process for the entire duration of the background process (currently
3000 seconds). Therefore, run the monitoring script only when the Docker
executor is used where this is not a problem.

(cherry picked from commit 35792b1700b404b2eac7f0c60f6dfbf9a6bb3e31)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index adab5116a5a3338faa4e93474cafae46cdce9a12..5fc7be6e3a1e0212d44d7d7a70e6da3f3aa834e1 100644 (file)
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -404,6 +404,8 @@ cross-version-config-tests:
     - *setup_interfaces
   script:
     - *find_pytest
+    - *find_python
+    - ( if [ "${CI_DISPOSABLE_ENVIRONMENT}" = "true" ]; then sleep 3000; "$PYTHON" "${CI_PROJECT_DIR}/util/get-running-system-tests.py"; fi ) &
     - cd bin/tests/system
     - >
       "$PYTEST" --junit-xml="$CI_PROJECT_DIR"/junit.xml -n "$TEST_PARALLEL_JOBS" | tee pytest.out.txt

diff --git a/util/get-running-system-tests.py b/util/get-running-system-tests.py
new file mode 100755 (executable)
index 0000000..1653e0a
--- /dev/null
+++ b/util/get-running-system-tests.py
@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+
+# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
+#
+# SPDX-License-Identifier: MPL-2.0
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0.  If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# See the COPYRIGHT file distributed with this work for additional
+# information regarding copyright ownership.
+
+import psutil
+
+for pid in psutil.pids():
+    try:
+        environ = psutil.Process(pid).environ()
+        if "PYTEST_CURRENT_TEST" in environ:
+            name = psutil.Process(pid).name()
+            print(
+                f'pytest process {name}/{pid} running: {environ["PYTEST_CURRENT_TEST"]}'
+            )
+    except (psutil.AccessDenied, psutil.NoSuchProcess):
+        pass