This is a required first step before testing and it's not specifically
mentioned in the doc. This tripped me up for a while, so let's save the
trouble for the next person.
Daan De Meyer [Thu, 27 Mar 2025 20:38:00 +0000 (21:38 +0100)]
test: Make it possible to run the integration tests standalone (#36868)
Currently, to run the integration tests, it's still necessary to
install various other build tools besides meson: A compiler, gperf,
libcap, ... which we want to avoid in CI systems where we receive
prebuilt systemd packages and only want to test them. Examples are
Debian's autopkgtest CI and Fedora CI. Let's make it possible for
these systems to run the integration tests without having to install
any other build dependency besides meson by extracting the logic
required to run the integration tests with meson into a separate
subdirectory and adding a standalone top-level meson.build file which
can be used to configure a meson tree with as its only purpose running
the integration tests.
Practically, we do the following:
- all the integration test directories and integration-test-wrapper.py
are moved from test/ to test/integration-tests/.
- All the installation logic is kept out of test/integration-tests/ or
any of its subdirectories and moved into test/meson.build instead.
- We add test/integration-tests/standalone/meson.build to run the
integration tests standalone. This meson file includes
test/integration-tests via a cute symlink hack to trick meson into
including a parent directory with subdir().
- Documentation is included on how to use the new standalone mode.
Daan De Meyer [Wed, 26 Mar 2025 13:30:20 +0000 (14:30 +0100)]
test: Make it possible to run the integration tests standalone
Currently, to run the integration tests, it's still necessary to
install various other build tools besides meson: A compiler, gperf,
libcap, ... which we want to avoid in CI systems where we receive
prebuilt systemd packages and only want to test them. Examples are
Debian's autopkgtest CI and Fedora CI. Let's make it possible for
these systems to run the integration tests without having to install
any other build dependency besides meson by extracting the logic
required to run the integration tests with meson into a separate
subdirectory and adding a standalone top-level meson.build file which
can be used to configure a meson tree with as its only purpose running
the integration tests.
Practically, we do the following:
- all the integration test directories and integration-test-wrapper.py
are moved from test/ to test/integration-test/.
- All the installation logic is kept out of test/integration-test/ or
any of its subdirectories and moved into test/meson.build instead.
- We add test/integration-test/standalone/meson.build to run the
integration tests standalone. This meson file includes
test/integration-test via a cute symlink hack to trick meson into
including a parent directory with subdir().
- Documentation is included on how to use the new standalone mode.
- TEST-64-UDEV-STORAGE and TEST-85-NETWORK are changed to generate separate
units for each testcase to make them behave more like the other integration
tests.
Yu Watanabe [Wed, 26 Mar 2025 21:10:53 +0000 (06:10 +0900)]
core: drop manager_get_unit_by_pid() and friends (#36872)
Except for one place, they are only used by test-watch-pid. Let's also
use manager_get_unit_by_pidref() and friends in the test, and drop the
_pid() variants.
Yu Watanabe [Wed, 26 Mar 2025 19:27:01 +0000 (04:27 +0900)]
coredump: make sure pid1/journal coredumps are not lost when Storage=journal is selected (#36870)
It confused the hell of me, that if pid1 crashes in an mkosi system run
from the build tree there's no coredump kept. Because mkosi configures
journal storage for coredumps, but that's not going to work for pid1 or
journald. Hence use external storage for these two even if everything
else is stored in the journal.
coredump: do not remove PID1/journal coredumps if Storage=journal is used
We always redirect PID1/journal coredumps directly onto disk instead of
the journal even if that's configured because that might cause a
deadlock because we are still pinning the old journal process while
processing the coredump. However, so far we then immediately deleted the
coredumps because of Storage=journal, which is very annoying, since
there's hence no copy kept whatsoever.
Let's hence exclude PID1+journal from the removal.
This in particulary brings the code in line with the log messages which
claim we kept the file around but we actually did not.
Daan De Meyer [Wed, 26 Mar 2025 12:18:40 +0000 (13:18 +0100)]
Revert "test: dynamically generate list of test cases"
We want to decouple the integration tests in meson from the
rest of the source files so the integration tests can be run
without the source files available. Let's revert the change to
dynamically figure out the test cases from the networkd tests for
now so that the tests can be generated without the test source file
being available.
Daan De Meyer [Wed, 26 Mar 2025 09:26:36 +0000 (10:26 +0100)]
test: Make sure serial is always set explicitly for scsi-hd qemu devices
Now that mkosi uses -blockdev instead -drive, the device_id property
of scsi-hd devices is not populated automatically anymore so we have to
make sure to always specify serial= to make sure /dev/disk/by-id is populated
as expected in the test.
Nick Rosbrook [Mon, 24 Mar 2025 13:06:33 +0000 (09:06 -0400)]
polkit: use -EBUSY to indicate shortcut after error or denial
Commit 536c18e5c3 ("bus-polkit: shortcut auth. after first denial")
added logic to async_polkit_query_check_action() that returns
-EALREADY when a failure or denial decision was made for a previous
action.
Tweak this to return -EBUSY instead of -EALREADY. This hopefully makes
the intent of the error more clear. EALREADY suggests that the request
is OK, but polkit is processing something else, and we should come back
later. EBUSY suggests that polkit is busy or unusable, hence the
request cannot be processed, and we should go away.
Nick Rosbrook [Fri, 21 Mar 2025 19:14:20 +0000 (15:14 -0400)]
login: add polkit example rules for allowing root to ignore inhibitors
The semantics of strong inhibitors require that POLKIT_ALWAYS_QUERY
always be set when checking if we can allow blocking inhibitors to be
ignored on shutdown, reboot, etc. With the default polkit rules and
policy, users may experience a situation where users in the sudo group
are authorized to run:
systemctl reboot --check-inhibitors=no
but the root user is not authorized. Instead, the following error is
given:
Call to Reboot failed: Interactive authentication required.
While this is correct according to the semantics of strong inhibitors,
it is confusing. To help the situation, provide example polkit rules
that allow root to perform these actions.
Finally, when root receives SD_BUS_ERROR_INTERACTIVE_AUTHORIZATION_REQUIRED
when calling e.g. systemctl reboot, print a message explaining that this
is due to the current polkit policy, and point to the new example rule.
Nick Rosbrook [Fri, 21 Mar 2025 18:38:20 +0000 (14:38 -0400)]
login: handle -EALREADY from bus_verify_polkit_async_full()
Commit 536c18e5c3 ("bus-polkit: shortcut auth. after first denial")
added logic to async_polkit_query_check_action() that returns
-EALREADY when a failure or denial decision was made for a previous
action.
This has the consequence that root is able to ignore inhibitors and
shutdown etc. even when polkit explicitly denies it. This is because
when systemctl's verb_start_special() calls logind_reboot(), unless
the call succeeds or returns one of -EACCES, -EOPNOTSUPP, or
-EINPROGRESS, a fallback path is taken to attempt the action without
going through logind. Hence, since logind_reboot() started returning
-EALREADY in some cases, the fallback path was taken, and the shutdown
was performed anyways.
For example:
root@ubuntu:/# cat /etc/polkit-1/rules.d/10-systemd-logind-no-skip-inhibitors.rules
// Never allow strong inhibitors to be ignored.
polkit.addRule(function(action, subject) {
if ((action.id == "org.freedesktop.login1.power-off-ignore-inhibit" ||
action.id == "org.freedesktop.login1.reboot-ignore-inhibit" ||
action.id == "org.freedesktop.login1.halt-ignore-inhibit" ||
action.id == "org.freedesktop.login1.suspend-ignore-inhibit" ||
action.id == "org.freedesktop.login1.hibernate-ignore-inhibit")) {
return polkit.Result.NO;
}
});
root@ubuntu:/# systemctl reboot -i
Call to Reboot failed: Operation already in progress
..but the reboot continues anyways due to the fallback.
To fix this, add logic in systemd-logind's verify_shutdown_creds() to
handle -EALREADY from bus_verify_polkit_async_full(): if we receive
-EALREADY when checking authorization for <action>-multiple-sessions,
and we are blocked on inhibitors, continue on to get the decision for
<action>-ignore-inhibit directly.
While here, add similar logic to method_inhibit(), which may need to
verify multiple polkit actions in a single call.
Luca Boccassi [Tue, 25 Mar 2025 19:34:44 +0000 (19:34 +0000)]
tools/check-version-history: avoid DeprecationWarning with newer lxml (#36860)
We get the same warning thousands of times:
/work/src/tools/check-version-history.py:28: FutureWarning: This search
incorrectly ignores the root element, and will be fixed in a future
version. If you rely on the current behaviour, change it to
tools/check-version-history: avoid DeprecationWarning with newer lxml
We get the same warning thousands of times:
/work/src/tools/check-version-history.py:28: FutureWarning: This search incorrectly
ignores the root element, and will be fixed in a future version. If you rely on the
current behaviour, change it to
"./refsynopsisdiv/funcsynopsis/funcprototype/funcdef/function[.='udev_device_get_properties_list_entry']"
We also need to update the ignorelist to the new form.
Daan De Meyer [Tue, 25 Mar 2025 12:36:53 +0000 (13:36 +0100)]
mkosi: update fedora commit reference
* 13d523f84d Relax dependencies from noarch packages on archful packages for OBS builds
* 59378485be Remove purge-nobody-user script
* d1380dc114 Add more services to %post for udev and networkd
* 6f0d03443d Fix paths for /usr/sbin/nologin and related progs
* df9a74d530 Make the source tarball glob in the test script more generic
Daan De Meyer [Tue, 25 Mar 2025 09:37:32 +0000 (10:37 +0100)]
test: Disable pager in integration test units
Integration test units are now connected to the tty when running
interactively, so let's make sure we disable the pager to avoid tests
hanging in the pager.
Daan De Meyer [Sun, 23 Mar 2025 21:52:10 +0000 (22:52 +0100)]
user-record: Allow/strip status for mask/extract privileged helpers
If we're using these helpers, we want to split a user record into two,
one with the privileged section, and one without. This should work even
when the user record has a "status" section, so adapt the helpers to
account for that.
core/manager: do not exclude watchdog logic from busy-loop protection
As reported in https://github.com/systemd/systemd/issues/35405, if the watchdog
ping failed, we effectively started a busy loop here. The previous commits
should fix this, but in general, the protection here is intended as a safety
net in case the logic is broken somewhere else. We shouldn't exclude the
watchdog stuff from this.
Closes https://github.com/systemd/systemd/issues/35405. Apparently some
watchdog devices can be opened, but then the pings start failing after some
time. Since the timestamp of the last successful ping is not updated, we try to
ping again immediately, causing a busy loop and excessive logging.
After trying a few different approaches to fit this into the existing framework
without changing the logic too much, I settled on an approach with a second
timestamp. In particular, the timestamp of the last successful ping is public,
exposed as WatchdogLastPingTimestamp over dbus. It'd be wrong to redefine this
to mean the last ping *attempt*. So we need a second timestamp in some form.
Also, if we give up on pinging, we probably should attempt to disarm the
watchdog. It's possible that the pinging fails, but the watchdog would still
fire. I don't think we want that, since it seems that our internal loop is
working, it's just the watchdog that is broken.
Structured message with SD_MESSAGE_WATCHDOG_PING_FAILED is logged if we fail
to ping.
I tested this by attaching gdb to pid 1 and calling close(watchdog_fd).
We get a bunch of warning messages and then an attempt to close the watchdog:
Mar 21 15:46:17 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:20 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:23 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:26 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:29 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:32 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:35 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:37 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:40 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:43 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:46 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:49 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:52 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:55 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0: Bad file descriptor
Mar 21 15:46:58 fedora systemd[1]: Failed to ping hardware watchdog /dev/watchdog0, closing watchdog after 15 attempts: Bad file descriptor
Mar 21 15:46:58 fedora systemd[1]: Failed to disable hardware watchdog, ignoring: Bad file descriptor
Mar 21 15:46:58 fedora systemd[1]: Failed to disarm watchdog timer, ignoring: Bad file descriptor
Daan De Meyer [Fri, 21 Mar 2025 14:13:22 +0000 (15:13 +0100)]
packit: Load fmf metadata from rpm spec repository
Maintaining the fmf metadata and script upstream makes it painful
to reuse downstream so let's move the metadata and testing script
downstream and load it upstream instead.
Daan De Meyer [Fri, 21 Mar 2025 15:30:12 +0000 (16:30 +0100)]
mkosi: update fedora commit reference
* 2ecfbec1a4 Support specifying extra mkosi repositories to the test script
* f5b47b1302 Use old setup sysusers files on Fedora < 43
* 2da5793357 Merge #196 `Migrate fmf metadata and test script from the upstream repository`
* e346d9f33e Limit sdubby dependency to Fedora
shared/watchdog: add MESSAGE_IDs to logs about watchdog opening
One ID for the success case, with WATCHDOG_DEVICE= showing the device,
and one ID for the failure case, with WATCHDOG_DEVICE= if configured,
and ERRNO= set automatically.
shared/watchdog: ratelimit the number of attempts to open watchdog
We need to retry the open attempts for the watchdog, because the device becomes
available asynchronously.
The watchdog is opened in two places:
- in pid1 in the main loop. The loop has a ratelimit, but during a boot we
iterate in it fairly quickly. On my test VM with 'iTCO_wdt', version 2:
$ journalctl -b --grep 'Failed to open any watchdog' | wc -l
3398
After the device has been processed by udev, it is initialized successfully.
- in shutdown. In that case, we most likely don't need to try more than once,
because we mostly care about the case where the watchdog device was present
and configured previously. But in principle it is possible that we might
attempt shutdown while the machine was initializing, so we don't want to
disable retries. Nevertheless, watchdog_ping() is called from a loop that
might be fairly tight, so we could end up trying to reopen the device fairly
often. This probably doesn't matter *too* much, but it's still ugly to try to
open the device without any ratelimit.
Usually the watchdog timeout would be set to something like 30 s or a few
minutes. OTOH, on my VM, the device becomes avaiable at 4.35 s after boot. So
let's use 5 s or half the watchdog timeout, whatever is smaller, as the
interval.
Yu Watanabe [Fri, 21 Mar 2025 00:54:45 +0000 (09:54 +0900)]
udev: make udevadm and friends not warn about unknown settings
Without this change, when e.g. event_timeout= is specified in udev.conf,
udevadm and friends which loads udev.conf warn about unknown key:
===
$ udevadm info /sys/class/net/lo
/run/udev/udev.conf.d/test-17.conf:1: Unknown key 'event_timeout', ignoring.
/run/udev/udev.conf.d/test-17.conf:2: Unknown key 'timeout_signal', ignoring.
===
I think we need to log at some point if the user configured a watchdog device,
but no devices were found. We can't log ENOENT immediately, because the device
may likely appear during boot. So wait until the end of the initial transaction
and log then.
shared/watchdog: raise log levels for watchdog errors
If we failed to open the watchdog device for any reason, we'd only log at debug
level. This seems iffy: if the user configured a timeout for the watchdog, we
should report when we can't set it up. ENOENT is still logged at debug level
only, since it's somewhat expected to have a watchdog timeout set up, even for
systems which don't have a watchdog, or the device might appear later.
If the device doesn't support WDIOC_GETSUPPORT, still log that we opened a
device.
No change in behaviour, except for the log level threshold. As a side effect,
the reason why we failed to open the device is now stored in watchdog_fd
(previously it was -1 always), but this isn't used for anything.
core: drop duplicated check in manager_{set,override}_watchdog
Those functions call watchdog_setup() and watchdog_setup_pretimeout(), which
internally do a similar check against the static variables watchdog_timeout and
watchdog_pretimeout. The second check is not useful.
update-done: create /etc and /var if they didn't exist
Previously, we would fail. But this doesn't seem useful: we may want to
mark the update as done even if /etc/ or /var/ no updates were necessary
and there was no need to create /etc/ or /var/ yet.
The idea is to use this when building an image to mark the image as not
needing updates after the reboot. In general it is impossible to say if
any of the early boot update services can be safely skipped, except when
the creator of the image knows all the contents there and has made sure
that all the updates have been processed. (This is in fact what happens
in a typical package-based installation: the packages have scriptlets which
implement the changes during or after the installation process.)
With this patch, the image build process can do 'systemd-update-done --root=…'
at the appropriate point to avoid triggering of ldconfig.service,
systemd-hwdb-update.service, etc.
I didn't write --image=, because it doesn't seem immediately useful. The
approach with --root is most useful when we're building the image "offline",
which means that we have a directory we're working on.
The man page was right, but the comment in the generated file was wrong. The
timestamp is *not* the timestamp when the update is being done. While at it,
say to what directory the message applies. This makes it easier for a casual
reader to figure out what is happening.
Also rename the function to better reflect what it does.
Inspired by https://github.com/systemd/systemd/issues/36045.
vmspawn: allow TPM state to be persistent + rework runtime dir logic
When using vmspawn on particleos image we really want that the TPM state
is retained between invocation, since the encryption key is locked to
the TPM after all. Hence let's support that.
This adds --tpm-state= which can be used to configure a path to store
the TPM state in. It can also be used to force tpm state to be transient
or to let vmpsawn pick the path automatically.
While we are at it, let's also revamp the runtime dir handling in
vmspawn: let's no longer place the sockets the auxiliary services listen
on within their own runtime directories. Instead, just drop the runtime
directories for them entirely (since neither virtiofsd, nor swtpm
actually use them). Also, let systemd clean up the sockets
automatically.
Currently this is picked up from the main branch of the fork which is
suboptimal. The packit folks implemented this new option for us which
should fix the problem.
Daan De Meyer [Wed, 19 Mar 2025 13:08:49 +0000 (14:08 +0100)]
fmf: Use mkosi -f together with ToolsTreePackageDirectories=
There's no need to build various systemd tools from source again to
build the mkosi image when we can just install the packages that were
already built from source into the tools tree so let's do that to avoid
unnecessary compiling.
projects / thirdparty / systemd.git / log
