It looks nice if the command line is tidy and the same on repeated
invocations. Plain sorted() would put '(' first, but conceptually it's
nice to have it last, so one can think "this will be installed if
those other things are listed first". The installer does not care of
course.
mkosi: optionally insert less packages automatically
systemd, systemd-udev and some other packages were always
automatically inserted since the first version of mkosi. But this
approach is too heavy in some cases. To allow more flexibility while
maintaining backwards compat, this patch does two interleaved changes:
1. BasePackages=yes|no|conditional is added.
With BasePackages=yes, we get status quo ante.
With BasePackages=no, only packages specified by the user are
installed.
With BasePackages=conditional, on rpm systems, packages are
installed if the matching base package would be installed. On
deb/arch/… systems, equivalent to BasePackages=yes. (In other words,
only rpm supports this natively. I'm not aware of similar
functionality in other installers. install_arch() tries to achieve
something similar by comparing package sets.)
The default is "yes", to preseve unchanged behaviour by default.
2. Various packages that were installed unconditionally are converted
to be conditional.
This way we make more cases work automatically out of the box
(e.g. the user only needs to specify 'systemd', not the
subpackages), but allow deps to be cut down.
Looking at some concrete examples in Fedora:
- dracut: dracut is only necessary if using an initrd (and of
course only if the initrd image is generated using dracut, and only
if it is generated locally). But even for bootable images, a kernel
with no initramfs might be enough. Or when using kvm, a direct
kernel boot with an externally-provided kernel+initrd, etc.
So pulling in dracut and dependencies is often not necessary.
- kernel-modules: despite the name, actually basic modules are in
kernel-core. And kernel-modules is non-core modules. (And yes,
kernel-modules-extra is extra-non-core-modules, and
kernel-modules-internal is something else yet again).
- kernel-core: as with dracut, having a kernel image is not necessary
in many scenarios.
- systemd: on big example where it's not useful is systemd itself.
When using mkosi to build a systemd test image from sources,
installing the system package forces all deps to be installed, which
might not be desired. Dropping systemd from the installation list
makes mkosi useful for non-systemd stuff (some container types or
non-systemd inits).
When installing systemd from sources, we do *not* want subpackages like
systemd-udev to be installed. Having a mix of systemd-from-sources and
a different compilation of systemd-udev-from-distro is likely to cause
linking issues and other incompatibilities.
- grub* / systemd-boot: people might want to use grub even if sd-boot
is available, etc.
For conditionalization, why do it using rpm syntax, and not interally
in code? For some cases, the effect is almost the same. But in other
cases, the installer has more information. For example, glibc usually
is not specified explicitly, but is pulled in through deps. We want
glibc-minimal-langpack only if something else pulls in glibc, and we
cannot resolve this from inside mkosi. But even for other packages,
the user might use a meta-package or a package group or a file path or
a virtual provide, and we cannot know in general what package will be
used to resolve the dependency. Using boolean dependencies makes this
much more reliable.
For example, it's super useful to comment on a list of packages:
Packages=
glibc-minimal-langpack # avoid pulling in other glibc langpacks
less # this makes 'systemctl' much nicer to use ;)
;-comments are not allowed, because the colon doesn't stand out
enough. I was considering also adding comment_prefixes=('#',) to
disallow ;-comments everywhere, but that'd be a compatibility break
which I don't see enough justification for.
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/zbyszek/src/mkosi/mkosi/__main__.py", line 31, in <module>
main()
File "/home/zbyszek/src/mkosi/mkosi/__main__.py", line 11, in main
args = parse_args()
File "/home/zbyszek/src/mkosi/mkosi/__init__.py", line 5283, in parse_args
args = parse_args_file_group(argv, default_path)
File "/home/zbyszek/src/mkosi/mkosi/__init__.py", line 5320, in parse_args_file_group
return parser.parse_args(argv_post_parsed)
File "/usr/lib64/python3.9/argparse.py", line 1818, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/usr/lib64/python3.9/argparse.py", line 1851, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib64/python3.9/argparse.py", line 2060, in _parse_known_args
start_index = consume_optional(start_index)
File "/usr/lib64/python3.9/argparse.py", line 2000, in consume_optional
take_action(action, args, option_string)
File "/usr/lib64/python3.9/argparse.py", line 1912, in take_action
argument_values = self._get_values(action, argument_strings)
File "/usr/lib64/python3.9/argparse.py", line 2444, in _get_values
self._check_value(action, value)
File "/usr/lib64/python3.9/argparse.py", line 2496, in _check_value
if action.choices is not None and value not in action.choices:
File "/usr/lib64/python3.9/enum.py", line 373, in __contains__
raise TypeError(
TypeError: unsupported operand type(s) for 'in': 'str' and 'EnumMeta'
With the patch:
usage: mkosi [-h] [--version]
...
{build,clean,help,summary,genkey,bump,serve,build,shell,boot,qemu,ssh} ...
mkosi: error: argument -t/--format: unknown Format: 'asfd'
It would be nice to also get rid of the list of options in the help
message: it's not helpful to the user at all.
mkosi: trim file system after finishing all writes
This is not really about actually generating sparse images, but about the
side effect that unused blocks will appear as zero, and thus the images
will be able to compress better.
mkosi: make workspace dir configurable and discover mkosi.workspace/
So far we determined the workspace directory from $TMPDIR and /var/tmp
if that wasn't set. In some cases this is less than ideal though, in
particular if one wants to build a btrfs image but /var/tmp is not
btrfs. In that case it makes sense to always use a local workspace dir.
This adds support for this:
WorkspaceDirectory= + --workspace-dir= may not be used to configure it
explicitly.
if "mkosi.workspace/" exists, we use it automatically.
i kept hitting this with OutputFormat.gpt_btrfs.
maybe the logic of including usr_only in minimize cases requires
revisiting? maybe "/usr" itself can be a subvol?
mkosi: in UsrOnly=yes mode, generate a mount.usr= kernel cmdline switch
So far, we relied on gpt-auto's logic to automatically find the root fs
on the image. This is not supported for finding the /usr partition
though, hence generate a mount.usr= kernel cmdline option to address
this.
(And it's unlikely we'll add auto-discovery for /usr, since we cannot
realistically decide whether its worth waiting for /usr. After all we
*know* we can't proceed without a rootfs, hence it makes sense to
generically wait for one if we never got one configured one. But a /usr
partition is an optional thing, hence it doesn't make sense to wait for
it if we didn#t get told about it)
mkosi: rework how we pass roothash/usrhash/other metadata to kernel-install
Previously, we'd replace the main kernel-install script with one
generated by mkosi and pass the roothash/usrhash as final argument on
the command line. That's less than ideal though, since it collides with
the documented way kernel-install dropins are called (those arguments
are initrds, not root hashes according to the docs). Moreover, it leaves
no nice way to pass usrhash vs. roothash or pass other information.
Let's fix this all, by changing the script to look for the roothash in
$ROOTHASH and the usrhash in $USRHASH. Then pass these env vars when
invoking the script, and drop any additional arguments.
Also, add the $IMAGE_ID and $IMAGE_VERSION in a similar fashion and then
use those preferably to name the generated kernel images.
With all this in place we are fully "standards-compliant" again,
matching the documented kernel-install drop-in API, the kernel images
are named a lot more nicely, and the usrhash logic starts working too.
mkosi: make sure to mount /root/ from outside if UsrOnly=
When using mkosi in build script mode we copy stuff into /root/, as
that's where we build things in. In UsrOnly= mode that directory is not
persistent though, which makes the excercise pointless. Hence, in
UsrOnly= mode let's mount /root/ from a persistent directory from our
workspace.
This is a recent addition in systemd, required to make cases work where
the initrd creates a root fs on first boot and images are initially
shipped without one.
(See https://github.com/systemd/systemd/pull/19234)
While we are at it, let's order the list of extra initrd files
alphabetically.
mkosi: if we have an image id (any maybe image version) use it to label partitions
This makes a lot of conceptual sense (since labels probably should
better declare what you find *in* a partition instead of just a
description of the type of it, because we already can generate that
automatically from the GPT UUID).
But it's also useful in conjunction with systemd's logic of always
picking the newest version of a root partition to mount in
systemd-nspawn and related commands, i.e.
https://github.com/systemd/systemd/pull/18958
On top of that this is useful in combination with SplitArtifacts=as then
the partitions and the split files will carry the same (except for the
suffix).
mkosi: add "image-id" concept in addition to "image-version"
A previous commit added an image version concept. Let' build on that and
add "image-id" too to match this systemd PR:
https://github.com/systemd/systemd/pull/19093
The image ID is used for naming the enerating artifacts if that's not
explicitly set (instead of the generic "image" as before) and is passed
to th build scripts via an IMAGE_ID env var matching the IMAGE_VERSION
the earlier commit added.
The idea is that the image version and ID passed here is ultimately
written to /etc/os-release by the build script (possibly mangled) and
the mkosi concepts for this hence ultimately propagate into the image
itself in some form even if not 1:1 (depending on the project of
course).
mkosi: add new --auto-bump switch for automatically bumping version after each successful build
Often it make sense to combine "build" and "bump" into one, so that the
a series of builds comes with a linearly increasing series of version
numbers. Add a new switch --auto-bump for that (or -B in short).
mkosi: add "mkosi bump" verb for bumping the image version
This is a simple verb that reads the image version, and increases the
last dot-separated component of it by one and writes it to
mkosi.version. It's supposed to be a very simple call only for linear
version bumps. For anything more complex people should manually edit the
file or override it temporarily via --image-version= or so.
This adds a "image version" concept to mkos. Via the
ImageVersion=/--image-version= settings or the `mkosi.version` file a
version string may be defined. This is used for two things:
1. It is included in the (default) output filename. i.e. instead of
"image.raw" the files will be called "image-0.1.raw" for a version of
"0.1" and so on.
2. It is passed as IMAGE_VERSION to the build script (the script could
use it and patch it into /etc/os-release or so for example)
(In case you wonder why this is called ImageVersion=/--image-version=
instead of just Version=/--version, that's because there's already a
command line switch for the latter that prints mkosi's own version
number)
This fixes two issues with UsrOnly=yes builds that are bootable and have
verity:
1. use the right partition type uuid for the verity partition
2. fix the umount path when we invoke dracut in a /usr-only build, since
"umount --recursive" insists on having a mount point at the specified
path, which we previously didn't necessarily. Let's add a bind mount
on itself there to address this.
Frantisek Sumsal [Tue, 23 Mar 2021 15:55:48 +0000 (16:55 +0100)]
Attempt to retry the SSH connection if the veth/SSH server is not up yet
This commit introduces a new option `--ssh-timeout=` to allow setting
a timeout for which `mkosi` will attempt to retry the SSH connection in
case the link not properly set up yet (e.g. when running `mkosi qemu` and
`mkosi ssh` in a script, the veth may not be configured yet, causing the
ssh connection to fail).
Frantisek Sumsal [Tue, 16 Mar 2021 18:01:46 +0000 (19:01 +0100)]
Allow overriding the SSH key pair
Allow using a pre-defined SSH key pair instead of generating a new one.
This should be useful, for example, in CI environments, where the images
are built and used at different times, and using the same SSH key pair
across multiple images is more convenient.
mkosi: add UsrOnly= setting for generating images with only a /usr/ partition
This adds a new boolean option that changes our image layout slightly:
instead of include the whole root fs in the image, we just pack up the
/usr/ subtree.
This has two usecases:
1. Truly stateless systems that come up pristine on every single boot,
where /etc and /var are populated by tmpfiles/sysusers and related
calls
2. Systems that are shipped without root fs, but where systemd-repart
adds one in, with a locally generated encryption key and so on. (this
doesn't work fully yet, because of some initrd issues, will look at
this later.)
A companion PR on the systemd side is: https://github.com/systemd/systemd/pull/18958
Fixes: #634
(This incorporates many fixes by @behrmann. Thank you!)
Nicolas Trangez [Thu, 11 Mar 2021 13:37:49 +0000 (14:37 +0100)]
Include `systemd.volatile` infrastructure in initrd
For the `systemd.volatile=...` feature to work, the
`systemd-volatile-root` generator and `systemd-volatile-root.service`
service are required to be present. Booting a read-only image generated
by `mkosi` (e.g., `gpt_ext4` built with the `--read-only` flag, or a
`gpt_squashfs` image) with a volatile but mutable overlay is useful, so
we need to provide these files in the `initrd` Dracut generates.
Hence, adding both files to `DRACUT_SYSTEMD_EXTRAS`.
Liam McBirnie [Sun, 28 Feb 2021 20:43:27 +0000 (21:43 +0100)]
Fix generated root partition being too small for encrypted images.
Building encrypted minimized or squashfs images was failing
at the 'dd' command in 'insert_partition' with the
error 'No space left on device'.
Increasing the luks overhead from 2MB to 16MB allows enough space.
16MB was found to be the smallest overhead which wouldn't fail.
Liam McBirnie [Sun, 28 Feb 2021 20:31:00 +0000 (21:31 +0100)]
Fix incorrect arguments given to 'luks_format_root'.
Commit 99453d9c added a 'cached' parameter to the 'luks_format_root'
command but didn't add the paramter to where the command is called
inside 'insert_partition'.
This patch sets 'cached' to False and fixes 'inserting_generated_root'
which should be True.
Liam McBirnie [Sun, 28 Feb 2021 20:24:40 +0000 (21:24 +0100)]
Reread partition table after inserting partition.
Fixes issue where the sfdisk command fails to re-read the partition
table when creating a squashfs image with encrypted data.
This is because the encrypted data partition is mounted so sfdisk gives
the warning 'Device or resource busy'.
This causes the 'dd' command failing because it can't find the newly
created partition.
mkosi: add new simple "serve" verb for serving built images via HTTP
If found this quite useful for testing things with "machinectl pull-tar"
and "machinectl pull-raw": add a tiny HTTP server that can serve the
generated output once it#s done. Love the simplicity of this:
Apparently we have to flush things explicitly after writing it,
otherwise things remain buffered by python IO and the SHA256SUMS file
will remain empty on disk.
Frantisek Sumsal [Mon, 22 Feb 2021 21:42:59 +0000 (22:42 +0100)]
Check if btrfs is available before trying to unlink a subvolume
otherwise the output is full of pointless errors on certain systems
(like CentOS 8, which doesn't support btrfs):
# mkosi --force --debug run --qemu-headless=true build
‣ Removing output files...
+ btrfs subvol show /home/vagrant/mkosi/mkosi.output/fedora.raw
‣ Error: btrfs not found in PATH.
...
‣ Configuring serial tty (ttyS0)...
‣ Cleaning dnf metadata......
+ btrfs subvol show /var/tmp/mkosi-polqexiq/root/var/log/dnf.log
‣ Error: btrfs not found in PATH.
+ btrfs subvol show /var/tmp/mkosi-polqexiq/root/var/cache/dnf
‣ Error: btrfs not found in PATH.
+ btrfs subvol show /var/tmp/mkosi-polqexiq/root/var/log/dnf.librepo.log
‣ Error: btrfs not found in PATH.
+ btrfs subvol show /var/tmp/mkosi-polqexiq/root/var/log/hawkey.log
‣ Error: btrfs not found in PATH.
+ btrfs subvol show /var/tmp/mkosi-polqexiq/root/var/lib/dnf
‣ Error: btrfs not found in PATH.
+ btrfs subvol show /var/tmp/mkosi-polqexiq/root/var/log/dnf.rpm.log
‣ Error: btrfs not found in PATH.
‣ Cleaning rpm metadata......
+ btrfs subvol show /var/tmp/mkosi-polqexiq/root/var/lib/rpm
‣ Error: btrfs not found in PATH.
‣ Resetting machine ID...
...
Frantisek Sumsal [Thu, 25 Feb 2021 15:33:48 +0000 (16:33 +0100)]
Allow overriding # of CPUs and amount of RAM for qemu guests
This commit introduces two options - `--qemu-smp` and `--qemu-mem` - which
can be used to override the default number of CPUs and amount of RAM for
guests started via the `qemu` verb.
This commit extends CommandLineArguments with all necessary fields
from the argument parser, removes the inheritance from argparse.Namespace
and fixes all resulting typing errors.
We try to stick to typing only changes as much as possible to reduce the
chance of breaking something (although there are a few non-typing changes
where doing so made things easier).
Because we use a dataclass now for the CommandLineArguments class, we
up the required python version to 3.7.
After:
sudo mkosi -d centos -r 7 --bootable --force
‣ Error: Sorry, CentOS 7 does not support unified kernel images. You must use --without-unified-kernel-images.
Michal Koutný [Fri, 12 Feb 2021 17:01:45 +0000 (18:01 +0100)]
openSUSE: Fix autologin setup
The distro PAM config resides under /usr/etc and customizations are
supposed to be under /etc. Use the distro file as a template for the
autologin customization.
Daan De Meyer [Wed, 3 Feb 2021 23:12:54 +0000 (23:12 +0000)]
qemu: Add secure boot support
Adds support for booting OVMF firmware blobs with secure boot
support.
We have to point qemu to an OVMF vars file to get this working.
Currently, we point directly at the vars file in /usr/share but
add the readonly flag so we make sure we don't modify it. I'm
hoping this means the UEFI variable changes in the VM will be
ephmeral instead of writes to these variables failing when the
readonly flag is set. If readonly means we can't enroll secure-boot,
we'll need to make a copy of the VARS file and store it somewhere so
we can remove the readonly flag.
Daan De Meyer [Wed, 3 Feb 2021 22:03:15 +0000 (22:03 +0000)]
qemu: Use q35 machine
This is necessary for booting with secure boot enabled in QEMU.
Switching to q35 somehow changed the boot order causing us to drop
into the EFI shell at boot. To fix this, I switched from virtio-blk
to virtio-scsci-pci (which is supposed to be the future anyway)
which allows us to set bootindex to override the boot order.
Joerg Behrmann [Sat, 23 Jan 2021 17:50:36 +0000 (18:50 +0100)]
mkosi: switch to RawConfigParser instead of ConfigParser
To allow for expanding systemd.unit like specifier like %u for user, we have to
disable the basic interpolation that configparser allows, because it too uses %
as its specifier.
Daan De Meyer [Sat, 23 Jan 2021 20:36:40 +0000 (20:36 +0000)]
Add support for passing arguments to the build command
Aside from environment variables, it's also useful to be able to
pass arguments to the build script. For example, in systemd we can
use this to pass the target to build to the build script. Build
scripts determine how the passed arguments are interpreted, mkosi
just passes them on.
Michal Koutný [Thu, 21 Jan 2021 17:52:43 +0000 (18:52 +0100)]
UEFI boot: make sure efivarfs loaded in initrd
The efivarfs is needed in order to GPT root partition discovery work.
Without efivarfs initrd won't be able to switch to the real root.
Add the module regardless of hostonly or distro configuration because
dracut takes into account info from the build host and not the target
distro. Adding a module that's already included in dracut's list makes
no harm.
Daan De Meyer [Thu, 21 Jan 2021 19:23:28 +0000 (19:23 +0000)]
Drop networkd not running to a warning instead of a fatal error
Without networkd, the veth link won't come up properly on the host
but that doesn't prevent mkosi boot or mkosi qemu from working so
let's drop those messages to warnings instead of fatal errors.
Daan De Meyer [Tue, 12 Jan 2021 20:36:16 +0000 (21:36 +0100)]
Remove "complete" steps from output
The next step start message implies the end of the previous step so
there's no real benefit to printing a complete message for every step.
Let's remove the message and only print a "complete" message if the
caller of complete_step() has provided a custom "complete" message.
projects / thirdparty / mkosi.git / log
