test: wait until the nspawn process is completely dead

Before calling io.systemd.MachineImage.List.

The systemd-nspawn process takes a lock in the run() function in
nspawn.c and holds it for the entire runtime of that function. If we
call `machinectl terminate` the machine gets unregistered _before_ we
release the lock, so the original `machinectl status` check would return
early, allowing for a race where we call io.systemd.MachineImage.List
over Varlink when systemd-nspawn still holds the lock because the
process is still running.:

[   41.691826] TEST-13-NSPAWN.sh[1102]: + machinectl terminate long-running
[   41.695009] systemd-nspawn[2171]: Trying to halt container by sending TERM to container PID 1. Send SIGTERM again to trigger immediate termination.
[   41.698235] systemd-machined[1192]: Machine long-running terminated.
[   41.709520] TEST-13-NSPAWN.sh[1102]: + systemctl kill --signal=KILL systemd-nspawn@long-running.service
[   41.709169] systemd-nspawn[2171]: Failed to unregister machine: No machine 'long-running' known
[   41.720869] TEST-13-NSPAWN.sh[2346]: + varlinkctl --more call /run/systemd/machine/io.systemd.MachineImage io.systemd.MachineImage.List '{}'
[   41.723359] TEST-13-NSPAWN.sh[2347]: + grep long-running
...
[   41.735453] TEST-13-NSPAWN.sh[2352]: + varlinkctl call /run/systemd/machine/io.systemd.MachineImage io.systemd.MachineImage.List '{"name":"long-running", "acquireMetadata": "yes"}'
[   41.736222] TEST-13-NSPAWN.sh[2353]: + grep OSRelease
[   41.739500] TEST-13-NSPAWN.sh[2352]: Method call io.systemd.MachineImage.List() failed: Device or resource busy
[   41.740641] systemd[1]: Received SIGCHLD.
[   41.740670] systemd[1]: Child 2171 (systemd-nspawn) died (code=killed, status=9/KILL)
[   41.740725] systemd[1]: systemd-nspawn@long-running.service: Child 2171 belongs to systemd-nspawn@long-running.service.
[   41.740748] systemd[1]: systemd-nspawn@long-running.service: Main process exited, code=killed, status=9/KILL
[   41.740755] systemd[1]: systemd-nspawn@long-running.service: Will spawn child (service_enter_stop_post): systemd-nspawn
[   41.740872] systemd[1]: systemd-nspawn@long-running.service: About to execute: systemd-nspawn --cleanup --machine=long-running
...

Let's mitigate this by waiting until the corresponding
systemd-nspawn@.service instance enters the 'inactive' state where the
lock should be properly released.

Resolves: #39547

diff --git a/test/units/TEST-13-NSPAWN.machined.sh b/test/units/TEST-13-NSPAWN.machined.sh
index 8c7087c55d9620d75bfb755b57c883c3a3840bad..a9e3817688527c7d8497d750a38d3956f21f9d31 100755 (executable)
--- a/test/units/TEST-13-NSPAWN.machined.sh
+++ b/test/units/TEST-13-NSPAWN.machined.sh
@@ -441,9 +441,14 @@ varlinkctl call /run/systemd/machine/io.systemd.Machine io.systemd.Machine.OpenR
 
 # Terminating machine, otherwise acquiring image metadata by io.systemd.MachineImage.List may fail in the below.
 machinectl terminate long-running
-# wait for the container being stopped, otherwise acquiring image metadata by io.systemd.MachineImage.List may fail in the below.
-timeout 30 bash -c "while machinectl status long-running &>/dev/null; do sleep .5; done"
-systemctl kill --signal=KILL systemd-nspawn@long-running.service || :
+# Wait for the container to stop, otherwise acquiring image metadata by io.systemd.MachineImage.List below
+# may fail.
+#
+# We need to wait until the systemd-nspawn process is completely stopped, as the lock is held for almost the
+# entire life of the process (see the run() function in nspawn.c). This means that the machine gets
+# unregistered _before_ this lock is lifted which makes `machinectl status` return non-zero EC earlier than
+# we need.
+timeout 30 bash -xec 'until [[ "$(systemctl show -P ActiveState systemd-nspawn@long-running.service)" == inactive ]]; do sleep .5; done'
 
 # test io.systemd.MachineImage.List
 varlinkctl --more call /run/systemd/machine/io.systemd.MachineImage io.systemd.MachineImage.List '{}' | grep 'long-running'