Until now, we read the tools tree configuration both inside and
outside the sandbox. This is problematic for a few reasons:
- For a number of reasons, the tools tree configuration inside and
outside the sandbox may be different. This is especially true when
the history is used, as when invoking mkosi sandbox and mkosi is
invoked again within the sandbox, we don't know when invoking mkosi
sandbox whether the nested invocation will reuse the history or not.
This can lead to situations where mkosi sandbox thinks the tools tree
is up-to-date whereas mkosi inside the sandbox thinks the tools tree
is out of date, which is a situation that is very hard to resolve for
users.
- Even if we read the tools tree configuration inside the sandbox, there's
no real point to it except maybe for showing the summary. Since the
tools tree has already been built and mounted at that point, we can't
build or use a new tools tree, or clean up an old one.
Let's fix these issues by *not* reading the tools tree configuration
inside the sandbox, neither from configuration not from history. The
one piece of tools tree configuration we do need to pass into the sandbox
is the default tools tree path as that gets included in the history of
the other images so we pass it in via an environment variable
$MKOSI_DEFAULT_TOOLS_TREE_PATH.
As a consequence of this approach, we cannot write the tools tree history
from within the sandbox anymore, so we write it to a separate file from
outside the sandbox instead.
Additionally, we make more commands use the history. Up until now summary and
other verbs did not make use of the history. This doesn't really make sense,
so let's make sure they make use of the history as well. summary in particular
is important as it may be used to query information about the built image after
building it which needs to take the history into account.
We still allow summary to ignore the history by passing --force. Also we
still ignore the history for clean as there are use cases both for having it
use the history and not yet the --force flag already has meaning there.
action: Use inputs to work around github actions empty variables
Even using env doesn't seem to work, so let's try the other workaround
from the issue, using inputs default values to capture the right values
before they become empty.
Unshallowing the repository only works when the action is used as a
local action within the mkosi repository. If the action is used in any
other github repository, the git history is not available so we cannot
unshallow the repository.
Instead, detect if the git history is available and clone the repository
ourselves if the git history is not available.
In systemd we very often require newer mkosi git commits which means
that setting MinimumVersion=25~devel doesn't work anymore. Let's allow
the MinimumVersion= to be a git sha so that we can enforce the mkosi
version to be new enough.
We skip git commit checks if we're in the sandbox since mkosi will be
a zipapp there and we won't have access to the git repository to do the
check.
Let's pass in a queue to the target function of AsyncioThread() to which
it can write messages. Let's also keep track of all sent messages in
AsyncioThread for easy access later and add some utility functions to
process new messages.
We also split out notify() from vsock_notify_handler() and make it non-vsock
specific by passing in the socket as an argument.
Allow --rerun-build-scripts in combination with none output
To make the logic work without being able to check if the output
exists or not, we simply never write the history if --rerun-build-scripts
if specified with the none output format.
It wants a date in the format Y-m-d H:M:S rather than unix seconds.
Also if SOURCE_DATE_EPOCH is not defined, use actual epoch, as the
timestamps must match when creating the signature and when attaching
it
DEBUG is very generic and can picked up by anything, but sometimes
with drastically different meanings to what is supposed to be in the
environment variable value.
For example the xfstests configure script expects DEBUG to contain
compiler options, which makes it break in extremely hard to debug
ways if we set DEBUG ourselves to a boolean value.
Changing the working directory to mkosi/ is not intuitive. We want
to parse configuration from there, but the working directory should
not change (e.g. you don't want mkosi sandbox to execute inside the
mkosi/ subdirectory when invoked in the top level directory).
Use mkosi/ subdirectory if no configuration exists in cwd
In systemd we've ended up with a large amount of mkosi specific
files and directories in the top level of the repository. Let's allow
moving these to a mkosi/ directory by transparently picking this up.
Don't allow configuring distribution, release and mirror in tools tree config
config_default_repository_key_check() depends on ToolsTreeDistribution=
to select the right default. If the tools tree distribution is configured
in mkosi.tools.conf, we're unable to select the right default for
RepositoryKeyCheck= for the main image. Until we figure out a decent solution
for this, let's insist that the tools tree distribution is configured via
ToolsTreeDistribution=.
Similarly, selecting the default tools tree mirror depends on the selected
target distribution and mirror, so we insist on configuring the tools tree
mirror and release via ToolsTreeMirror= and ToolsTreeRelease= for now.
Currently we assume fetching is not required if the host distribution
is not ubuntu, but this is wrong. If the host distribution is not ubuntu
but a ubuntu tools tree is used to build rpm based or Arch images, then
we still need key fetching.
Additionally, make RepositoryKeyFetch= a universal setting so that the
tools tree can have it enabled without enabling it for the main and subimages.
We then also drop the condition from config_default_repository_key_fetch()
that checks whether we need key fetching to build the tools tree distribution
because this will be determined automatically now when parsing the tools tree
config.
initrd: Copy Arch Linux keyring from host when running as root
When running as root, we run with --cache-only=metadata, which means
we won't try to sync the keyring, so let's reuse the one from the host
to make sure that a keyring is available.
Daan De Meyer [Mon, 31 Mar 2025 12:37:27 +0000 (14:37 +0200)]
Allow adding extra tools tree configuration via mkosi.tools.conf
Instead of adding ever more ToolsTreeXXX= settings, let's allow
adding extra tools tree configuration via mkosi.tools.conf which can
be either a file or a directory containing extra configuration for the
default tools tree.
Daan De Meyer [Mon, 31 Mar 2025 08:46:41 +0000 (10:46 +0200)]
Switch tools tree parsing to ParseContext
Default tools tree configuration currently depends on manually plumbing
through settings from the main configuration to the tools tree configuration.
This has already bitten us a few times where we forgot to add the necessary
plumbing when adding a new setting.
Instead, let's improve by encoding in the settings themselves whether they
apply to the tools tree or not. We call settings that apply to the tools tree
and all subimages "multiversal" settings (one step bigger than "universal"
settings).
For the tools tree specific settings (e.g. ToolsTreeDistribution= and friends),
we simply copy the fields from the main image configuration namespace to the tools
tree configuration namespace.
Additionally, we restrict which settings can be applied to the tools tree by
a new "tools" field for each setting which has to be set to True to make a setting
configurable in the default tools tree configuration.
Finally, we stop storing the tools tree specific settings in the main image config,
and instead store the tools tree configuration separately in the history JSON and
summary. This will allow us to add more ways to configure the default tools tree in
the future as well.
Daan De Meyer [Sun, 30 Mar 2025 19:42:03 +0000 (21:42 +0200)]
config: Use dicts everywhere in favor of argparse.Namespace()
These behave the same, except that argparse.Namespace() allows dot
based accessed to its keys and dict does not. Dot based access suggests
to the reader that a typed property is being accessed that definitely
exists, whereas that's anything but guaranteed when using argparse.Namespace().
Let's switch to using dicts everywhere so that whenever we access a
property, we do so in a way that suggests to the reader that we're doing
an untyped access into a dictionary.
Daan De Meyer [Sun, 30 Mar 2025 15:10:00 +0000 (17:10 +0200)]
Disable selinux relabeling by default for directory images
For directory images it's more likely to not want these selinux
relabeled, so let's default to that. Also drop the explicitly selinux
relabeling disablement from mkosi-tools.
Daan De Meyer [Sun, 30 Mar 2025 15:02:00 +0000 (17:02 +0200)]
mkosi-tools: Stop removing BuildSources=
Tools tree prepare scripts are now a thing so it's possible to require
build sources for the default tools tree so let's stop removing them
unconditionally.
Daan De Meyer [Sat, 29 Mar 2025 12:06:29 +0000 (13:06 +0100)]
Introduce 'main' setting scope
There's a bunch of settings that aren't universal but also don't make
sense to be configured in subimages, so let's add a new 'main' setting
scope that disallows configuring these settings in subimages without
passing them to subimages.
Daan De Meyer [Fri, 28 Mar 2025 09:19:29 +0000 (10:19 +0100)]
Allow combining --force and --rerun-build-scripts
Until now we didn't allow this, but it turns out there's actually
a use case for this, build the image if it doesn't exist yet, reuse
the existing image otherwise.
mkosi-initrd: fix misleading "mkosi not found" error message
There is a special handling when a script under /work fails with 127, which is
also reached when mkosi is called within mkosi-initrd.
```
‣ Running finalize script /etc/mkosi-initrd/mkosi.finalize…
/work/finalize: line 5: strip: command not found
‣ /work/finalize failed with non-zero exit code 127
‣ (Maybe a program was not found or the script interpreter (e.g. bash) is not installed?)
‣ mkosi not found.
```
Daan De Meyer [Thu, 27 Mar 2025 16:27:02 +0000 (17:27 +0100)]
pacman: Always bind mount /var/lib/pacman/local from sandbox
We want any writes to /var/lib/pacman/local to go to any configured
overlayfs on /buildroot, instead of going directly to the
/var/lib/pacman/local directory which might be a lowerdir in the
overlayfs if one is used. Let's implement this by simply specifying
a path relative to the sandbox instead of specifying the path on the
host.
Daan De Meyer [Thu, 27 Mar 2025 10:43:23 +0000 (11:43 +0100)]
config: Cache lookups of fields
We were calling inspect.signature() > 4000 times because it was evaluated
over and over again in the list comprehensions. Move the fields lookup into
a cached classmethod where possible to make sure we only do it once.
This reduces the time needed to parse the systemd config from history from
1.18s => 188ms on my machine.
Daan De Meyer [Tue, 25 Mar 2025 21:13:42 +0000 (22:13 +0100)]
Don't cache the package manager used in the tools tree cache manifest
Doing this causes issues when using mkosi sandbox as have_cache(tools)
will be different inside and outside of the sandbox when building Fedora
and only dnf is available outside the sandbox but dnf5 is available inside
the sandbox. Since we don't need to cache the package manager used anyway
when building the default tools tree because we don't cache the package
manager metadata either, don't cache it to avoid this problem.
Daan De Meyer [Tue, 25 Mar 2025 19:28:51 +0000 (20:28 +0100)]
Implement ToolsTreeSyncScripts=
In systemd, we have prepare scripts for the build image to install
all the build dependencies of the corresponding distribution's systemd
packaging spec. The distribution packaging spec is checked out by a sync
script.
We also need the systemd build dependencies installed in the default tools
tree in systemd, so that we can build all the systemd tools required to build
the image in the tools tree as well to later be used to build the image. Currently
we list the the required dependencies manually as packages since we can't reuse
the prepare script from the build image as the sync scripts run after the default
tools tree has been built which means we can't depend on the packaging specs
being available when building the default tools tree.
Let's fix the issue by introducing ToolsTreeSyncScripts= to define sync scripts
to run before building the default tools tree.
Daan De Meyer [Tue, 25 Mar 2025 15:42:38 +0000 (16:42 +0100)]
Drop logic for using /root/.cache
Let's just have root use the cache in /var/cache/mkosi to avoid accidentally
leaking files into /root/.cache accidentally where it's much less likely that
they'll ever get cleaned up.
Daan De Meyer [Tue, 25 Mar 2025 13:41:26 +0000 (14:41 +0100)]
Make sure tools tree recorded in history doesn't change when in sandbox
Currently, when in the sandbox, the tools tree will always be recorded
as "null" in the history. This causes problems if an image build is done
inside of mkosi sandbox and mkosi is later invoked outside of the sandbox
as the history will say no tools tree was used and so the tools tree won't
be used, even if we're running outside of the sandbox.
To fix the issue, let's make sure we always record the proper tools tree,
but let's not make use of it in the sandbox by moving the sandbox check to
the Config object's tools() method.
We also simplify all the tools tree related checks in run_verb() and add a
check to make sure we don't try to rebuild the default tools tree when in
"mkosi sandbox" as that can't ever work.
projects / thirdparty / mkosi.git / log
