Currently, mkosi image builds can differ depending on the host they
were built from. This can happen because we execute all kinds of
binaries to build the image and depending on the host these binaries
can differ. Usually, it's different versions of tools causing issues,
but it can also be due to different build configurations, such as rpm
writing its database in a different format depending on whether it's
executed from CentOS, Fedora, or Opensuse.
To allow for more reproducibility in image builds regardless of the
host system, this commit adds a new option --tools-tree= that allows
specifying a tree in which we look up most of the programs that we
execute during an image build.
Of course, that still leaves the question of what tree should be passed
to --tools-tree=. To solve that problem, --tools-tree= can be used
together with presets, so that as the first preset, a "bootstrap" image
can be built which can then be used with --tools-tree= in later presets.
Note that we only use /usr from the given tree. If tools end up using
config files from /etc or such, we expect those tools to expose a knob
to specify a different configuration file (instead of us overmounting
/etc).
Note that in a few cases, we don't yet execute tools in the given tree:
- systemd-analyze in GenericVersion() can't be executed in the tree
because it could be executed during config parsing when we don't
know the tree to use yet.
- newuidmap/newgidmap have to be executed before we can run
bubblewrap so we can't run them in bubblewrap itself
- Figuring out the credentials is inherently tied to the host system
so we execute all scripts and tools to figure out credentials on
the host system as well
- mount because bubblewrap does not propagate mounts to the real root
so any mounts we do within bubblewrap don't survive the bubblewrap
process
- systemd-dissect for the same reason
Use stat tool to check if we're on a btrfs subsystem
Currently, we let the btrfs tool log errors when we're not on a
btrfs filesystem. Let's avoid unnecessary errors by checking if we're
on a btrfs filesystem before invoking the btrfs tool.
Drop shell workaround in bwrap() and run_workspace_command()
Let's use --perms and --chmod to fix the permissions of /tmp, /var/tmp
and /dev/shm instead of our hacky shell workaround. Let's also drop
all usage of shlex.join() in run() since it doesn't really by us anything.
Daan De Meyer [Tue, 27 Jun 2023 11:40:43 +0000 (13:40 +0200)]
Gentoo fixes
- Use boot use flag for systemd instead of gnuefi
- Add --deep to emerge invocation to fix dep resolution failures
- Use curl to download stage3 tarball so we get a progress bar
- Do not exclude dev, proc and sys directories when extracting tarball
(only exclude their contents)
- Copy pkgmngr/ directory into stage3/ directory wholesale instead of
individual files
- Various coding style fixes
- Stop using Repositories= to specify binary package repositories as it
is not its intended purpose. Instead, pass configured environment
variables to emerge so users can set PORTAGE_BINHOST instead.
Daan De Meyer [Tue, 27 Jun 2023 14:04:58 +0000 (16:04 +0200)]
Use systemd-repart's new --offline argument
When building images, we never want to use loop devices, so use
--offline=yes in that case. When booting images, we know that
systemd-nspawn requires loop devices, so require them for
systemd-repart as well using --offline=no.
Paymon MARANDI [Tue, 20 Jun 2023 15:41:05 +0000 (11:41 -0400)]
gentoo: hardcode stage3 path
2 more things:
- bring back cache_clean so we extract stage3 once. that also means
configure pkgmngr once
- add ./proc to exclusion list during extraction
Paymon MARANDI [Thu, 25 May 2023 13:19:34 +0000 (09:19 -0400)]
gentoo: default to btrfs
Given that mkosi is *bespoken* and since it leverages features from
btrfs in some configurations we switch to btrfs by default.
Gentoo doesn't actually care one way or the other what the undelying fs
is and ext4, previous default, was in fact an arbitrarily choice (a
copy-pasted from some other distro's module).
Marius Schiffer [Fri, 16 Jun 2023 10:00:53 +0000 (10:00 +0000)]
Add support for ukify config at /etc/kernel/uki.conf.
This allows specifying further options, e.g. a splash image for the
generated UKI file, given to ukify.
Ukify is run from the same working directory as mkosi itself,
so given paths in the ukify config can be relative to this.
config: reword help message for --root-{password,shell}
"system root" doesn't seem right, because it sounds like we're talking about
the file system. We would often say just "root password", but that's nor
gramatically correct. "root's password" would be correct, but seems strange.
So let's rephase this to avoid the awkwardness.
@@ -85,6 +85,6 @@
- --bootable [FEATURE] Generate ESP partition with systemd-boot and UKIs for
- installed kernels
--kernel-command-line OPTIONS
Set the kernel command line (only bootable images)
+ --bootable [FEATURE] Generate ESP partition with systemd-boot and UKIs for
+ installed kernels
Daan De Meyer [Tue, 20 Jun 2023 15:40:49 +0000 (17:40 +0200)]
Add "none" output format
This is a re-implementation of the --skip-final-phase option, but
instead of doing it via an option, we do it via a new output format,
which feels much more natural. In combination with mounting the
staging directory into the build script, this allows using mkosi to
produce arbitrary artifacts using the build script.
Daan De Meyer [Tue, 20 Jun 2023 15:24:12 +0000 (17:24 +0200)]
Mount staging directory into build script
The build script might produce additional outputs, so let's allow
the user to funnel those out of the container by mounting the staging
directory and setting the OUTPUTDIR environment variable.
Daan De Meyer [Tue, 20 Jun 2023 14:07:49 +0000 (16:07 +0200)]
Remove --install-directory= option
We don't benefit from the caching anymore since we started emptying
the directory completely on reuse as otherwise old leftover files
might get installed. Without the caching, the option does not have
a ton of use anymore, so let's remove it.
Daan De Meyer [Wed, 14 Jun 2023 15:41:32 +0000 (17:41 +0200)]
Run systemd-repart before booting image with systemd-nspawn
To match the behavior when running in qemu, let's run systemd-repart
on the image before running it in systemd-nspawn to make sure that
all the necessary partitions are added if the image has repart
definition files included.
Daan De Meyer [Tue, 13 Jun 2023 13:45:11 +0000 (15:45 +0200)]
qemu: Use SOCK_STREAM for notify socket
The CentOS 8 Stream kernel does not support SOCK_SEQPACKET for
AF_VSOCK so let's use SOCK_STREAM instead and explicitly instruct
systemd running in the VM to use SOCK_STREAM as well.
Daan De Meyer [Mon, 12 Jun 2023 13:12:01 +0000 (15:12 +0200)]
Ensure we return the same exit code in debug mode
When running in debug mode, we shouldn't return a different exit
code compared to when we run outside of debug mode.
A trivial example is when running the boot or qemu verbs where we
exit with the exit code of the container/VM by raising an instance
of CalledProcessError. In --debug mode, this exception is handled
as an unhandled exception by the python runtime which always exits
with 1 when an unhandled exception is encountered, whereas outside
of debug mode we exit with the value contained in
CalledProcessError.returncode.
With this commit, we'll always exit with the return value in
CalledProcessError.returncode.
Daan De Meyer [Sun, 11 Jun 2023 19:55:06 +0000 (21:55 +0200)]
apt: Set "Dir" instead of "Dir::Etc" and set it via APT_CONFIG
We need to use APT_CONFIG to set "Dir" as otherwise the value of "Dir"
won't be taken into account by apt when looking for configuration files
as options set via the CLI are applied last after parsing the apt
configuration files.
Daan De Meyer [Sun, 11 Jun 2023 18:24:10 +0000 (20:24 +0200)]
dnf: Write repos to /etc/yum.repos.d/<distribution>.repo
Let's allow providing a config file for dnf without also overriding
all the default repositories by writing the default repositories to
a file in /etc/yum.repos.d/.
Daan De Meyer [Sun, 11 Jun 2023 18:10:53 +0000 (20:10 +0200)]
Do not overwrite package manager config files if they exist
Let's allow users to override mkosi's default package manager
settings by providing their own package manager config file. This
is especially important for distributions built using dnf as dnf
does not yet have support for dropins.
Daan De Meyer [Sun, 11 Jun 2023 18:03:30 +0000 (20:03 +0200)]
arch: Drop useless options from config
CheckSpace isn't really useful as we'll just fail a little later
while unpacking or installing packages. HoldPkg isn't useful either
as there's no chance of breaking the system by uninstalling packages
when we're just building an image.
Daan De Meyer [Thu, 8 Jun 2023 15:42:35 +0000 (17:42 +0200)]
Prefer configuration via CLI over configuration via config file
This makes it easier to debug things with --debug as all the options
used will be logged instead of having to go and figure out what's in
the config file.
projects / thirdparty / mkosi.git / log
