We shouldn't special case updates Arch if we don't do so on other
distros either. We don't guarantee availability of a package
manager in generated images, and the only reason Arch images ship
with pacman is because it's part of the base group. This might
change in the future and as such, there's no guarantee that Arch
images will always have pacman. Hence, let's remove the pacman hooks
we install for Arch to bring it more in line with the other distros.
opensuse: do not attempt to overwrite pam.d/common-auth
Tumbleweed builds started to fail:
File /usr/local/lib/python3.10/dist-packages/mkosi/__init__.py, line 3225, in install_opensuse
shutil.copy2(root / usr/etc/pam.d/common-auth, root / etc/pam.d/common-auth)
File /usr/lib/python3.10/shutil.py, line 434, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File /usr/lib/python3.10/shutil.py, line 254, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/var/tmp/mkosi-vb08b_gb/root/usr/etc/pam.d/common-auth'
/etc/pam.d/common-auth is already installed now. Do not attempt to
overwrite it if it already exists.
Due to lack of a maintainer, and deviating from established practices
almost universally (custom bootloader, custom package manager, ...),
it's not productive to keep support for Clear Linux in mkosi. We have
no idea if it works at all, no idea if something we do is going to
break it, and haven't received any feedback from Clear Linux users
for multiple years. Hence let's drop support for Clear Linux from
mkosi to reduce maintenance costs.
Building initrds is slow, let's add an option that allows building
the initrd as part of a cached image so that's it's only built once
and reused in any final images.
Daan De Meyer [Fri, 24 Jun 2022 10:04:35 +0000 (12:04 +0200)]
action: Pin systemd-container version
A faulty metadata update in Github Actions azure mirror for Ubuntu
is causing issues with unmet systemd-container versions. Let's pin
the systemd-container version as a workaround until the issue is
fixed.
Daan De Meyer [Mon, 7 Mar 2022 16:10:41 +0000 (16:10 +0000)]
Add support for qemu's direct Linux boot
We add a "linux" boot protocol that, when enabled, instructs mkosi
to extract the kernel image, initrd and kernel cmdline out of the
image when building it.
On top, we add a --qemu-boot option that takes one of the possible
values from --boot-protocols and instructs qemu to use that boot
protocol. In case of the new "linux" boot protocol, we use qemu's
-kernel, -initrd and -append options to do a direct Linux kernel
boot.
Being able to do direct Linux boots with qemu is a prerequisite for
booting images of architectures that don't support UEFI in qemu. It's
also faster than booting in UEFI mode which is useful when iterating
and not working on the bootloader.
mkosi: link /var/lib/rpm to /usr/lib/sysimage/rpm for compat with old rpm
This (partially) fixes #993. There are two aspects of compatiblity: rpm changed
the db backend from bdb to sqlite, and the location was changed. This patch
resolves the second issue. It does using the same logic that e.g. Fedora uses
after the move. But it doesn't do anything for the first part. But luckily, rpm
in Rocky/Centos 8 is linked with support for sqlite, so things work.
We need to unlink /var/lib/rpm because something creates it during
installation. I wanted to just create the symlink if there's nothing there,
but that doesn't work.
The same logic was used for dnf/tdnf/rpm/dpkg/apt, so let's split
it out to a helper.
I opted to use full paths with "/". Such paths are easier to read
and we avoid the risk of removing something from the host if somebody
uses an absolute path by mistake.
Daan De Meyer [Mon, 20 Jun 2022 12:51:10 +0000 (14:51 +0200)]
ci: Show output by default
Let's have pytest show output even for successful tests. This helps
with debugging issues that aren't related to a failing test (e.g.
why is gentoo slow to build).
Daan De Meyer [Wed, 16 Feb 2022 10:46:42 +0000 (10:46 +0000)]
Default to -cpu max when running VMs using QEMU
Currently, we don't specify the -cpu option when running under the TCG
accelerator. This leads QEMU to choose a very conservative default that
doesn't emulate all the instructions that modern distros are compiled with.
To avoid such issues, let's default to having QEMU emulate as many CPU
instructions as possible to avoid illegal opcode errors when running
virtual machines.
Daan De Meyer [Wed, 16 Feb 2022 10:44:41 +0000 (10:44 +0000)]
Change --qemu-smp default from 2 to 1
When trying to a boot a centos epel VM using QEMU, it will hang during
boot if the qemu smp option is set to a number higher than 1. To avoid
this and similar issues, let's default to 1 core per VM. If users need
more they can always configure the option explicitly.
Daan De Meyer [Mon, 21 Feb 2022 14:54:31 +0000 (14:54 +0000)]
Don't override stat of destination root when using copy_path()
Fixes unexpected scenarios where we modify the permissions of / when
using mkosi.extra/. See https://github.com/systemd/systemd/pull/22569#issuecomment-1045992142
for more information.
Daan De Meyer [Mon, 16 May 2022 14:57:02 +0000 (16:57 +0200)]
ci: Update to Ubuntu 22.04 LTS
Note: The LTS image is still in beta (https://github.com/actions/virtual-environments/issues/5490).
In Jammy, we have recent versions of zypper and dnf packaged so we
don't have to build them from source anymore. Also, sq is packaged
so we don't have to build sq from source anymore either.
The setup-github-actions.sh script is renamed to setup-pacman.sh
and it's reduced to only install the dependencies necessary to
build pacman and archlinux-keyring.
All other dependencies are moved to action.yaml and the action
mkosi.default script.
Daan De Meyer [Thu, 16 Jun 2022 21:43:45 +0000 (17:43 -0400)]
Communicate the associated dir in the ESP via /etc/kernel/entry-token
/etc/kernel/entry-token is the new way introduced in systemd v251 to
identify the directory that kernel-install and bootctl should install
their stuff to. If it exists, bootctl and kernel-install will read it
and use the directory inside to install things to.
Currently, in mkosi, we generate a random machine ID during the build
and use that as the directory under the ESP to install things to. Until
all distros we support get support for /etc/kernel/entry-token, we're
limited to using the machine ID as the directory to install stuff under
in the ESP.
Since the machine ID used during the build is scrubbed from the image,
users don't a way to figure out the directory in the ESP associated
with the rootfs after the build is finished. To fix this, let's write
the machine ID to /etc/kernel/entry-token before it is scrubbed so
that users can read the file to figure out which directory in the ESP
that they should look under to find stuff associated with the
corresponding rootfs.
Daan De Meyer [Mon, 13 Jun 2022 15:09:00 +0000 (11:09 -0400)]
Rework initramfs generation (again)
Currently, when building without unified kernel images, the generated
initramfs is generated as part of running the package manager for the
first time. Because of this, none of the changes made to the rootfs
after running the package manager (build script, extra-trees, postinst
script, ...) are taken into account when generating the initramfs. Also,
when building with unified kernel images, the generated initramfs does
take all changes made to the image into account.
To solve this inconsistency, let's generate the initrd manually instead
of relying on the package manager to do it for us. With the addition
of KERNEL_INSTALL_BYPASS, we can skip initramfs generation when
kernel-install is called by a post-installation script to generate the
initramfs. Similarly, we can use the INITRD environment variable to
do the same on Debian/Ubuntu systems.
To generate the initramfs manually, we simply call kernel-install at
a later point in the image build process. Because kernel-install doesn't
actually regenerate the initramfs on Debian/Ubuntu, we call
"dpkg-reconfigure dracut" there to make sure the initramfs is regenerated
before calling kernel-install.
Finally, because we now make sure the initramfs always includes all changes
made to the image, we modify install_unified_kernel() to call objcopy
again instead of dracut as we used to do but had to revert because calling
objcopy meant the initramfs wasn't regenerated.
Daan De Meyer [Fri, 17 Jun 2022 14:16:18 +0000 (10:16 -0400)]
machine: Add retries for ssh
We've been seeing quite a bit of "connection refused" errors in CI.
These are likely happening because sshd hasn't finished starting
yet.
The proper fix for this is to add notify socket support for systemd
running qemu VMs via virtio sockets, but even if that's added, it
will be a very long time before we can rely on it.
For now, let's add a retry mechanism for SSH connections to make
our CI setup more reliable.
Daan De Meyer [Fri, 17 Jun 2022 14:13:49 +0000 (10:13 -0400)]
Refactor command running in integration tests
Let's move run_command_image() into Machine.run(), introduce
run_systemd_cmdline() to get the systemd-run command line, and
remove all arguments from run_ssh() that aren't required anymore
now.
mkosi: optimize/fix patching of root part-type uuid
The bug was that the part-type write we did would get overwritten when the
partition table was subsequently rewritten when we were adding the verity and
verity-sig paritions. We don't need to write out the part-type manually, it's
enough to store the right value in our partition list. This makes things a bit
faster too.
We know that if we calculated the verity info, we'll insert a partition soon
after and then it'll get written correctly.
Daan De Meyer [Wed, 15 Jun 2022 20:24:41 +0000 (16:24 -0400)]
arch: Use gpgdir from host system
Instead of setting up the keyring in the image, let's reuse the
keyring from the host. If users want to use pacman in the image,
they just have to run pacman-key themselves in a postinst script
or such.
This speeds up building of images and hopefully also gets rid of
our CI issues with Arch where there's something keeping files open
in the root mount (which I expect is gpg-agent).
Daan De Meyer [Fri, 20 May 2022 13:05:09 +0000 (15:05 +0200)]
Fix losetup race condition with initializing partition devices
This fixes the same issue we've seen in the systemd repo where
using PARTSCAN introduces a race condition with trying to use
the partition device since the kernel initializes the partition
devices asynchronously. To avoid the issue, we initialize partition
devices manually using the BLKPG ioctl(). We also avoid the same
problem on detaching loop devices by removing partition devices
explicitly using the BLKPG ioctl().
See https://github.com/systemd/systemd/pull/22992,
https://github.com/systemd/systemd/pull/23427,
https://github.com/systemd/systemd/issues/23174 and
https://github.com/systemd/systemd/issues/17469 for more context.
The default formatter would wrap all text into a single paragraph,
which is rather hard to read in case when we have a list of options
and an explanation for each of the values. Let's add a custom formatter.
Also, never split option names or other words.
Part of output wrapped to the default 80 columns:
--source-file-transfer-final METHOD
How to copy build sources to the final image:
'copy-all': normal file copy
'copy-git-cached': use git ls-files --cached, ignoring
any file that git itself ignores
'copy-git-others': use git ls-files --others, ignoring
any file that git itself ignores
'copy-git-more': use git ls-files --cached, ignoring
any file that git itself ignores, but include the
.git/ directory
(default: None)
--source-resolve-symlinks [BOOL]
If true, symbolic links in the build sources are
followed and the file contents copied to the build
image. If false, they are left as symbolic links. Only
applies if --source-file-transfer-final is set to
'copy-all'.
(default: false)
--source-resolve-symlinks-final [BOOL]
If true, symbolic links in the build sources are
followed and the file contents copied to the final
image. If false, they are left as symbolic links in
the final image. Only applies if
--source-file-transfer-final is set to 'copy-all'.
(default: false)
--with-network [WITH_NETWORK]
Run build and postinst scripts with network access
(instead of private network)
--settings PATH Add in .nspawn settings file
--help would print something like "--source-file-transfer-final SOURCE_FILE_TRANSFER_FINAL"
which takes a lot of space but is not very helpful. In particular it
might not be clear whether this expects some custom string or just a
yes/no boolean. Let's use "BOOL" instead to tell the user the type of
the argument, which immediately implies what values can be specified.
Similarly, say "--source-file-transfer METHOD", "--source-file-transfer-final METHOD".
Also drop metavar= when it matches the default value anyway.
Creation of bmap files needs to take place before any compression
happens, since bmaptool has to know where the "holes" of the image lie.
Compression removes the holes, preventing bmap from recreating the
original raw image. Move the bmap calculation step before the
compression.
Joerg Behrmann [Wed, 1 Jun 2022 17:35:39 +0000 (19:35 +0200)]
ssh: make parse_ssh_agent only handle strings
pyright complains (wrongly I think) about value being None when passed to Path
to create the socket variable. Let's work around this by eliminating Nones as
values.
Daan De Meyer [Mon, 16 May 2022 13:57:53 +0000 (15:57 +0200)]
mkosi: Always use the embedded default version when no release is specified
Let's not have the host system determine the image distribution release.
Instead, let's always default to the default release embedded within mkosi.
This gives more consistent results when building images for a single distro
regardless of the host distribution.
Daan De Meyer [Tue, 17 May 2022 09:45:08 +0000 (11:45 +0200)]
machine: Translate \r\n to \n in logfile
Output lines from pexpect sent to the logfile will always end with
"\r\n" (side-effect of working with pseudo-TTYs) . On Github Actions,
this results in blank lines in the test output. Let's add a simple
adapter that translates "\r\n" back to "\n" before actually writing
to the logfile.
Daan De Meyer [Wed, 11 May 2022 11:54:24 +0000 (13:54 +0200)]
Install util-linux explicitly on Fedora
In Fedora 36, by default only util-linux-core is pulled in which
is missing /bin/login which is required by /sbin/agetty to function
properly. Let's pull it in explicitly until the bug is resolved.
projects / thirdparty / mkosi.git / log
