Daan De Meyer [Tue, 27 May 2025 09:00:07 +0000 (11:00 +0200)]
Enforce C.UTF-8 locale for all commands we run
So this is a shitty situation either way. We generally have no idea
which locales will be available in the tools tree. Most likely, it'll
just be the C.UTF-8 one. In that case, using the locale environment
variables from the host is pointless and leads to annoying errors from
lots of tools.
We could make sure all locales are installed but glibc-all-langpacks is
> 200M installed size on Fedora which feels like too much. So let's
instead always use C.UTF-8, it's not like mkosi itself is locale aware
anyway.
Daan De Meyer [Mon, 26 May 2025 13:44:42 +0000 (15:44 +0200)]
Fix version bump check if image version was passed on CLI
We only want to generate a new version if no version was specified on
the CLI. To do that we need to check that the version on the CLI is None,
not the opposite.
Alberto Planas [Thu, 24 Apr 2025 13:57:56 +0000 (15:57 +0200)]
repart: use --append-fstab=auto if available
When generating images there is a chance that a pre-existent /etc/fstab
is present in the mkosi buildroot. systemd-repart can merge the already
existing fstab with the one generated, separating the sections in a way
that the systemd-repart can be reused several times.
Clayton Craft [Wed, 21 May 2025 22:59:36 +0000 (15:59 -0700)]
sandbox: handle case where dev node for tty doesn't exist
This fixes this crash, which happens even when isatty(2) is True:
Traceback (most recent call last):
File "/sandbox.py", line 1095, in <module>
main()
File "/sandbox.py", line 891, in main
ttyname = os.ttyname(2) if os.isatty(2) else ""
^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory
IIUC this is because isatty(2) is a valid file descriptor (inherited
from the parent process?), but ttyname() is unable find a device node
for it because it's not mounted in the sandbox.
This doesn't quite work as on the second run, mkosi.tools will be
picked up as the default value of ToolsTree= instead of "default"
so we lose cache invalidation.
Let's revert for now until we find a better solution.
Don't use default value if optional settings are set to none
Just like it's possible to disable usage of default values for collection
based settings, we should also support it for optional settings.
This can be used to disable usage of the tools tree even if mkosi.tools.conf
exists by setting ToolsTree= in mkosi.local.conf.
These symlinks to a different kernel directory make mkosi to fail:
```
‣ Applying kernel module filters
‣ Running modinfo to fetch kernel module dependencies
modinfo: ERROR: could not get modinfo from 'nvidia': No such file or directory
libkmod: ERROR: kmod_module_parse_depline: ctx=0x55cda95602a0 path=/usr/lib/modules/6.14.1-1-default/weak-updates/updates/thinkpad_ec.ko error=No such file or directory
modinfo: ERROR: could not get modinfo from 'hdaps': No such file or directory
libkmod: ERROR: kmod_module_parse_depline: ctx=0x55cda95602a0 path=/usr/lib/modules/6.14.1-1-default/weak-updates/updates/nvidia-open-driver-G06-signed-570.133.07/nvidia-modeset.ko.zst error=No such file or directory
modinfo: ERROR: could not get modinfo from 'nvidia_drm': No such file or directory
libkmod: ERROR: kmod_module_parse_depline: ctx=0x55cda95602a0 path=/usr/lib/modules/6.14.1-1-default/weak-updates/updates/nvidia-open-driver-G06-signed-570.133.07/nvidia.ko.zst error=No such file or directory
modinfo: ERROR: could not get modinfo from 'nvidia_uvm': No such file or directory
modinfo: ERROR: could not get modinfo from 'thinkpad_ec': No such file or directory
libkmod: ERROR: kmod_module_parse_depline: ctx=0x55cda95602a0 path=/usr/lib/modules/6.14.1-1-default/weak-updates/updates/nvidia-open-driver-G06-signed-570.133.07/nvidia.ko.zst error=No such file or directory
modinfo: ERROR: could not get modinfo from 'nvidia_modeset': No such file or directory
libkmod: ERROR: kmod_module_parse_depline: ctx=0x55cda95602a0 path=/usr/lib/modules/6.14.1-1-default/weak-updates/updates/thinkpad_ec.ko error=No such file or directory
modinfo: ERROR: could not get modinfo from 'tp_smapi': No such file or directory
```
Also, do not run `depmod` for this old kernel directory that contains only the
`update` directory.
These KMPs install their kernel modules under the updates folder of a certain
kernel directory, even with that kernel directory without any other content:
```
$ mkosi-initrd -k 6.14.0-1-default
‣ Validating certificates and keys
‣ Copying repository metadata
‣ Building main image
‣ Copying in sandbox trees…
‣ Installing openSUSE
...
‣ Applying kernel module filters
‣ Running modinfo to fetch kernel module dependencies
modinfo: ERROR: Module nvidia not found.
modinfo: ERROR: Module thinkpad-ec not found.
modinfo: ERROR: Module nvidia-modeset not found.
modinfo: ERROR: Module tp-smapi not found.
modinfo: ERROR: Module hdaps not found.
modinfo: ERROR: Module nvidia-uvm not found.
modinfo: ERROR: Module nvidia-drm not found.
‣ "modinfo --set-version 6.14.0-1-default --null nvidia thinkpad-ec nvidia-modeset tp-smapi hdaps nvidia-uvm nvidia-drm" returned non-zero exit code 1.
```
But, there is no need fail so far (after calling the package manager within
mkosi), some basic checks can be performed before spawning other processes.
Ryan Wilson [Mon, 21 Apr 2025 17:00:53 +0000 (10:00 -0700)]
mkosi: Override misconfigured gitconfig HTTP/HTTPS proxy with ProxyUrl
When running mkosi on a server with an outgoing HTTP/HTTPS proxy,
Build.ProxyUrl must be set. However, some repositories (e.g. systemd)
call git in their mkosi scripts (e.g. cloning package repositories for
distros) and .gitconfig can also contain different http.proxy and https.proxy
settings. Thus, if .gitconfig is misconfigured with the wrong proxy,
the user will get confusing errors related to git fetching repositories.
To ensure we use a consistent proxy across git and mkosi, we override
git http proxy configuration via GIT_CONFIG_COUNT, GIT_CONFIG_KEY_{n},
GIT_CONFIG_VALUE{n} environment variables. Most of the code complexity
is dealing with the case when these variables are set by the user via
Environment / EnvironmentFiles.
We disabled plugins by default a long time ago because the
versionlock plugin's configuration from the host was interfering
with image builds. Now that we have completely ownership over /etc,
there shouldn't be a need to disable plugins anymore if we assume
they won't do anything nasty by default without configuration.
Dan Streetman [Tue, 15 Apr 2025 23:28:49 +0000 (19:28 -0400)]
mkosi-obs: simplify generation of signed UEFI auth files
We can use systemd-keyutil to generate the PKCS#7 signature from the
OBS-provided PKCS#1 signature and OBS-provided x509 certificate, instead of the
'kernel-sign-file' script.
Also we can use the ESL files directly, instead of the perl statement that
removes the auth header, as mkosi.postoutput leaves them for us to use.
Currently, --auto-bump= bumps the version after finishing a build.
Effectively this means the version in mkosi.version doesn't reflect
the actual version of the image which seems weird. Instead, let's
generate the new version upfront so that it's used by the build but
only write it to mkosi.version when we do a successful build.
Additionally, look for a mkosi.bump script and use it if it exists
to generate the new version so we can generate a timestamp as the
version in particleos.
Only write CLI arguments to history instead of full config
The huge downside of writing the full config to the history is that
any modifications to config files are ignored until the image is
rebuilt.
If we make the argument that changes to config files are usually
supposed to be permanent rather than ephemeral, it doesn't make sense
to not take them into account when reusing the history. Especially since
the primary purpose of the history is to avoid specifying the same CLI
arguments over and over again without encoding those CLI arguments into
a config file.
So instead of saving the full config to the history and not reparsing
the configuration when reusing the history, let's only save the CLI
arguments to the history and always reparse the configuration, even when
we end up reusing the history.
This keeps the crucial benefit of the history intact (not having to repeat
yourself endlessly) while still making sure any new settings added to the
configuration files are taken into account immediately.
It also simplifies the implementation quite a bit, as we can get away
again with maintaining a single history file instead of two.
Additionally, we opt to completely ignore the history when using the sandbox
verb. The idea here is that the history only applies to the final image (and
subimages). This was already the case as any --tools-tree-xxx CLI options aren't
saved in the history as they are not fields of the Config object.
Disallow using --rerun-build-scripts with --force again
In the next commit, we're going to move history handling purely into
parse_config(), which means we can't check anymore if we built the
image or not, so let's disallow using --force with --rerun-build-scripts
again as the only way that works is if we can check whether we built the
image or not before writing the history.
installer: Make sure package manager state is preserved in the image
Package manager state should be preserved in the image as various
local features of the package manager may depend on this data to be
available. For pacman and zypper we already store local state inside
the image. For apt, we achieve this by storing all state in the image
by default except lists which is the repository metadata which we still
don't include in the image. For dnf/dnf5, we make sure the persistdir
points to inside the image.
Note that this state is still subject to removal by CleanPackageMetadata=
if the package manager is not installed or the option is explicitly enabled.
Only copy repository metadata from specific subdirs from /var
/var usually contains state specific to the local system so it's more
prudent to specify an allowlist of what should be copied rather than a
denylist of what shouldn't be copied.
This doesn't matter so much when using mkosi's own package cache directory
since that is only used when syncing repository metadata and not when actually
installing stuff, but it does matter when PackageCacheDirectory=/var is used since
then we're copying from a state directory in /var that is also used when installing
packages and as such will contain a lot of stuff that we don't want.
mkosi-obs: append certs from mkosi.uefi.db/ to 'db'
Allow appending an arbitrary number of certs to the autoenrolled UEFI
db. Expects list of PEM formatted files. Only implemented for OBS
builds for now.
Fail if --rerun-build-scripts is used and tools is out of date
If --rerun-build-scripts is used, we shouldn't rebuild the tools
tree unless --force is specified as well, so make sure we check for
that specific case.
Until now, we read the tools tree configuration both inside and
outside the sandbox. This is problematic for a few reasons:
- For a number of reasons, the tools tree configuration inside and
outside the sandbox may be different. This is especially true when
the history is used, as when invoking mkosi sandbox and mkosi is
invoked again within the sandbox, we don't know when invoking mkosi
sandbox whether the nested invocation will reuse the history or not.
This can lead to situations where mkosi sandbox thinks the tools tree
is up-to-date whereas mkosi inside the sandbox thinks the tools tree
is out of date, which is a situation that is very hard to resolve for
users.
- Even if we read the tools tree configuration inside the sandbox, there's
no real point to it except maybe for showing the summary. Since the
tools tree has already been built and mounted at that point, we can't
build or use a new tools tree, or clean up an old one.
Let's fix these issues by *not* reading the tools tree configuration
inside the sandbox, neither from configuration not from history. The
one piece of tools tree configuration we do need to pass into the sandbox
is the default tools tree path as that gets included in the history of
the other images so we pass it in via an environment variable
$MKOSI_DEFAULT_TOOLS_TREE_PATH.
As a consequence of this approach, we cannot write the tools tree history
from within the sandbox anymore, so we write it to a separate file from
outside the sandbox instead.
Additionally, we make more commands use the history. Up until now summary and
other verbs did not make use of the history. This doesn't really make sense,
so let's make sure they make use of the history as well. summary in particular
is important as it may be used to query information about the built image after
building it which needs to take the history into account.
We still allow summary to ignore the history by passing --force. Also we
still ignore the history for clean as there are use cases both for having it
use the history and not yet the --force flag already has meaning there.
Allow PCR signing settings to be overridden in sub-images
A sub-image is usually built from the same base image. That base image may be
used for a rescue image and a primary workstation image. If the rescue image
has PCR signing enabled then it could allow the disk to be automatically
unlocked. This change allows specifying separate PCR signing keys which will
support only the workstation sub-image performing an automatic unlock.
zypper: do not fail if a package configured to be removed is not found
`dnf` does not fail in this case:
```
‣ Removing 1 packages…
No matches were found for the following plugin name patterns while enabling libdnf plugins: builddep, reflink, versionlock
No matches were found for the following plugin name patterns while disabling libdnf plugins: *
No packages to remove for argument: shadow-utils
Nothing to do.
‣ Removing files…
```
But `zypper` fails by default, breaking the build. Found with mkosi-initrd in
openSUSE, the `shadow` package is not installed when using `WithRecommends=yes`.
```
‣ Removing 2 packages…
Warning: Enforced setting: $releasever=20241001
Reading installed packages...
'shadow' not found in package names. Trying capabilities.
No provider of 'shadow' found.
Resolving package dependencies...
The following package is going to be REMOVED:
sysuser-shadow
1 package to remove.
Package install size change:
| 0 B required by packages that will be installed
-3.0 KiB | - 3.0 KiB released by packages that will be removed
Backend: classic_rpmtrans
Continue? [y/n/v/...? shows all options] (y): y
(1/1) Removing: sysuser-shadow-3.3-8.2.noarch ..........................................................................................[done]
Installation has completed with error.
‣ "zypper --installroot=/buildroot --cache-dir=/var/cache/zypp --non-interactive --no-refresh --releasever=20241001 remove --clean-deps shadow sysuser-shadow" returned non-zero exit code 104.
```
This behavior can be changed using the `--ignore-unknown` global option (which
goes before the command).
action: Use inputs to work around github actions empty variables
Even using env doesn't seem to work, so let's try the other workaround
from the issue, using inputs default values to capture the right values
before they become empty.
Unshallowing the repository only works when the action is used as a
local action within the mkosi repository. If the action is used in any
other github repository, the git history is not available so we cannot
unshallow the repository.
Instead, detect if the git history is available and clone the repository
ourselves if the git history is not available.
In systemd we very often require newer mkosi git commits which means
that setting MinimumVersion=25~devel doesn't work anymore. Let's allow
the MinimumVersion= to be a git sha so that we can enforce the mkosi
version to be new enough.
We skip git commit checks if we're in the sandbox since mkosi will be
a zipapp there and we won't have access to the git repository to do the
check.
Let's pass in a queue to the target function of AsyncioThread() to which
it can write messages. Let's also keep track of all sent messages in
AsyncioThread for easy access later and add some utility functions to
process new messages.
We also split out notify() from vsock_notify_handler() and make it non-vsock
specific by passing in the socket as an argument.
Allow --rerun-build-scripts in combination with none output
To make the logic work without being able to check if the output
exists or not, we simply never write the history if --rerun-build-scripts
if specified with the none output format.
It wants a date in the format Y-m-d H:M:S rather than unix seconds.
Also if SOURCE_DATE_EPOCH is not defined, use actual epoch, as the
timestamps must match when creating the signature and when attaching
it
DEBUG is very generic and can picked up by anything, but sometimes
with drastically different meanings to what is supposed to be in the
environment variable value.
For example the xfstests configure script expects DEBUG to contain
compiler options, which makes it break in extremely hard to debug
ways if we set DEBUG ourselves to a boolean value.
Changing the working directory to mkosi/ is not intuitive. We want
to parse configuration from there, but the working directory should
not change (e.g. you don't want mkosi sandbox to execute inside the
mkosi/ subdirectory when invoked in the top level directory).
Use mkosi/ subdirectory if no configuration exists in cwd
In systemd we've ended up with a large amount of mkosi specific
files and directories in the top level of the repository. Let's allow
moving these to a mkosi/ directory by transparently picking this up.
Don't allow configuring distribution, release and mirror in tools tree config
config_default_repository_key_check() depends on ToolsTreeDistribution=
to select the right default. If the tools tree distribution is configured
in mkosi.tools.conf, we're unable to select the right default for
RepositoryKeyCheck= for the main image. Until we figure out a decent solution
for this, let's insist that the tools tree distribution is configured via
ToolsTreeDistribution=.
Similarly, selecting the default tools tree mirror depends on the selected
target distribution and mirror, so we insist on configuring the tools tree
mirror and release via ToolsTreeMirror= and ToolsTreeRelease= for now.
Currently we assume fetching is not required if the host distribution
is not ubuntu, but this is wrong. If the host distribution is not ubuntu
but a ubuntu tools tree is used to build rpm based or Arch images, then
we still need key fetching.
Additionally, make RepositoryKeyFetch= a universal setting so that the
tools tree can have it enabled without enabling it for the main and subimages.
We then also drop the condition from config_default_repository_key_fetch()
that checks whether we need key fetching to build the tools tree distribution
because this will be determined automatically now when parsing the tools tree
config.
projects / thirdparty / mkosi.git / log
