Report hung system tests

At times, a problem might occur where a test is not responding,
especially in the CI, determining the specific test responsible can be
difficult. Fortunately, when running tests with the pytest runner,
pytest sets the PYTEST_CURRENT_TEST environment variable to the current
test nodeid and stage. Afterward, the variable can be examined to
identify the test that has stopped responding.

The monitoring script needs to be started in the background. Still, the
shell executor used for BSD and FIPS testing can't handle the background
process cleanly, and the script step will wait for the background
process for the entire duration of the background process (currently
3000 seconds). Therefore, run the monitoring script only when the Docker
executor is used where this is not a problem.

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index cfd11105b97a9ec864203374703abc85827eec02..5665123a99608e4732cce79ace1c5a3230009e3c 100644 (file)
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -402,6 +402,8 @@ cross-version-config-tests:
     - *setup_interfaces
   script:
     - *find_pytest
+    - *find_python
+    - ( if [ "${CI_DISPOSABLE_ENVIRONMENT}" = "true" ]; then sleep 3000; "$PYTHON" "${CI_PROJECT_DIR}/util/get-running-system-tests.py"; fi ) &
     - cd bin/tests/system
     - >
       "$PYTEST" --junit-xml="$CI_PROJECT_DIR"/junit.xml -n "$TEST_PARALLEL_JOBS" | tee pytest.out.txt

diff --git a/util/get-running-system-tests.py b/util/get-running-system-tests.py
new file mode 100755 (executable)
index 0000000..1653e0a
--- /dev/null
+++ b/util/get-running-system-tests.py
@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+
+# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
+#
+# SPDX-License-Identifier: MPL-2.0
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0.  If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# See the COPYRIGHT file distributed with this work for additional
+# information regarding copyright ownership.
+
+import psutil
+
+for pid in psutil.pids():
+    try:
+        environ = psutil.Process(pid).environ()
+        if "PYTEST_CURRENT_TEST" in environ:
+            name = psutil.Process(pid).name()
+            print(
+                f'pytest process {name}/{pid} running: {environ["PYTEST_CURRENT_TEST"]}'
+            )
+    except (psutil.AccessDenied, psutil.NoSuchProcess):
+        pass