We said that all files are optional *before* starting the list of files,
then for each individual file, and then again *after* the list. Let's
simplify this by describing files without mentioning that is is optional.
Similarly, we don't need to emphasize the fact that scripts are executable
for each script… readers generally know that.
Let's also make the terms bold, so it's easier to find the interesting
paragraph in this wall of text.
Also break into paragraphs more; remove underlining from "OS" — it's a
normal abbreviation and the markup was confusing; use "/" at the end
of directories; use more markup for paths; move the description of
mkosi.skeleton above mkosi.extra.
Luca Boccassi [Fri, 18 Jun 2021 12:14:42 +0000 (13:14 +0100)]
qemu: revert back to virtio from virtio-scsi-pci on Debian
Debian images on qemu fails to boot with virtio-scsi-pci. Reverting back to
virtio does not show issues with the EFI shell, everything boots correctly,
so do that when running Debian images.
mkosi: if image version logic is enabled, make sure to generate root= entry in kernel command line
When the --image-version= logic is enabled this indicates that multiple
versions of an OS might be used in parallel within the same partition
table eventually. That makes it essential to boot the right root file
system from each unified kernel, so that kernel and root fs always match
up correctly.
If Verity is on this already worked, since we encode roothash= on the
kernel cmdline in that case, and that implies the root fs to use. Now,
ensure for verity-less cases that this works too.
We use PARTLABEL= match, using the image-id/image-version combination,
i.e. matching how we pick the label for the partition.
If the image version logic is not used, let's continue to not use the
a root= for simplicity reasons. In that case automatically finding the
root fs via fully automatic discovery should be simple and robust.
Michal Koutný [Fri, 11 Jun 2021 16:54:13 +0000 (18:54 +0200)]
Github Action: Install pip module matching the git revision
When the action is referenced with a particular git revision (e.g.
'uses: foo/mkosi@refspec') the given revision should be deployed.
Achieve that by installing pip directly from the action directory
(instead of a released/packaged version).
Alex Wicks [Thu, 1 Jul 2021 01:58:07 +0000 (02:58 +0100)]
Correct pacman hook name
Fixes systemd/mkosi#750
Pacman hooks must end with .hook as per Arch wiki
However mkosi's hook to update bootctl when the kernel updates ends in -hook instead of .hook
This means the hook doesn't get fired and kernel upgrades fail
Luca Boccassi [Fri, 18 Jun 2021 18:14:38 +0000 (19:14 +0100)]
nspawn: fallback to host network mode if networkd is not running
On Debian systemd-networkd is not the default network manager, so
there's nothing configuring the veth bridge.
If networkd is not running, simply fallback to using the host
network. This will not provide isolation, but at least the
connectivity is restored.
Luca Boccassi [Fri, 18 Jun 2021 16:52:49 +0000 (17:52 +0100)]
qemu: fallback to usermode network if networkd is not running
On Debian systemd-networkd is not the default network manager, Network-Manager is.
Fallback from TAP + veth to usermode networking. It is slower and NATted, but it
still works.
Blair Bonnett [Fri, 25 Jun 2021 07:40:57 +0000 (09:40 +0200)]
mkosi: add option to resolve symlinks when copying build sources.
This allows common build scripts or resources to be symlinked into the
build source directory, but then be copied into the build/final image as
actual files instead of symlinks.
mkosi: when using cached images, randomize fs and partition uuids explicitly
This addresses an annoying issue when building cached images containing
btrfs file systems: the kernel btrfs driver refuses mounting or handling
multiple different file systems with the same uuid. This means using an
image and building the next at the same time fails — as long as the
UUIDs of the newly build image aren't refreshed.
This patches makes sure when using a cached image we'll refresh disk,
partition and file system uuids. We generate them randomly, exactly like
we would have them when using non-cached builds.
This also ensures that the partition labels are rewritten when images
versions are bumped.
(Eventually we should probably start hashing the uuids from the
configuration state in some form, to provide a certain level of
reproducibility, but for now let's just randomize them.)
When building images with image IDs and image versions it's a good idea
to reuse the pre-pared cached images from the earlier versions (I mean,
that's why cacheing/incremental operation exists). So far the cached
image names was derived 1:1 from the output file name, but if that
includs the image version it will be different for each build and hence
never suitable as cache.
Hence, if the image ID/version logic is used, let's change the naming
regime for the cache file, and let's derive it direcly from image name,
not taking the version into account.
rpmlint warns that this is dangerous. We don't modify the argument,
so actually this is safe here, but it's still nice to avoid this
potential pitfall.
× Run python3 -m mypy mkosi
mkosi/__init__.py:12:1: error: Library stubs not installed for "dataclasses"
(or incompatible with Python 3.6) [import]
import dataclasses
^
mkosi/__init__.py:12:1: note: Hint: "python3 -m pip install types-dataclasses"
mkosi/__init__.py:12:1: note: (or run "mypy --install-types" to install all missing stub packages)
mkosi/__init__.py:12:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
mkosi/__init__.py:6247:12: error: Too many arguments for "CommandLineArguments"
[call-arg]
return CommandLineArguments(**vars(args))
^
Found 2 errors in 1 file (checked 3 source files)
Error: Process completed with exit code 1.
mkosi: rework compression config, add support for zstd
We had --compress and --xz. This is changed to --compress-fs (which is like
the old --compress), --compress-output (which is like --xz, except configurable),
and --compress (which tries to do the right thing depending on the output type).
With zstd, the difference in decompression time is massive:
$ time zstdcat image.cpio.zstd >/dev/null
0.221 s total
$ time xzcat image.cpio.xz >/dev/null
3.592 s total
$ ls -lg image.cpio.* image.cpio
-rw-rw-r-- 1 zbyszek 149408256 May 21 13:56 image.cpio
-rw-r--r-- 1 zbyszek 53032056 May 21 13:57 image.cpio.xz
-rw-r--r-- 1 zbyszek 59085907 May 21 13:49 image.cpio.zstd
v2:
- rename options to --compress-fs and --compres-output, unify handling
of cpio and tar.
v3:
- use tar --use-compress-program= instead of subprocess.Popen()
This may be useful to create initramfs images, see
https://github.com/keszybz/mkosi-initrd.
I also tried an implementation using libarchive. The code was a little
bit simpler, but unfortunately the generated archive wasn't to the
kernel's liking. I think both the compressio options *and* the
underlying cpio format were a bit different. I think we should just accept
that the kernel is only tested with the specific output produced by cpio
with some specific settings.
mkosi: add variant of subprocess.Popen() that does not wait
We had run() which was a wrapper around subprocess.run().
This addes spawn(), which starts the process and does not wait for it.
v2:
- run() is changed to be call spawn() and wait and throw an exception
if check is True.
v3:
- run() is a completely separate function again.
It turns out that run() does a bunch of handling of input and output
that would need to be recreated. In the end, the duplication of
logging is a smaller issue than fiddling with input and output
encodings.
--- /tmp/out1 2021-06-05 17:01:24.327828067 +0200
+++ /tmp/out2 2021-06-05 17:01:32.367897483 +0200
@@ -59,21 +59,19 @@
--version show program's version number and exit
Distribution:
- -d {fedora,debian,ubuntu,arch,opensuse,mageia,centos,centos_epel,clear,photon,openmandriva}, --distribution {fedora,debian,ubuntu,arch,opensuse,mageia,centos,centos_epel,clear,photon,openmandriva}
+ -d, --distribution {fedora,debian,ubuntu,arch,opensuse,mageia,centos,centos_epel,clear,photon,openmandriva}
Distribution to install
- -r RELEASE, --release RELEASE
+ -r, --release RELEASE
Distribution release to install
- -m MIRROR, --mirror MIRROR
- Distribution mirror to use
+ -m, --mirror MIRROR Distribution mirror to use
--repositories REPOS Repositories to use
--architecture ARCHITECTURE
Override the architecture of installation
Joerg Behrmann [Fri, 4 Jun 2021 15:24:43 +0000 (17:24 +0200)]
factor out leaf functions that will be needed for separating out distributions
This reuses knowledge gained from #715. The functions included here are needed
to when distributiosn are moved to their own subpackages. This change has no
functional changes.
Instead of saying that it only works for [str, str], say it can create
a mapping of any two types. I verified that if I add a generator of a
different type, everything works as expected and mypy can detect type
mismatches.
I still don't know how to say that the wrapper takes the same
arguments as the decorated function. It would require a type
metavariable that would mean "some set of args and kwargs", and I
don't think there is syntax for anything like this.
It looks nice if the command line is tidy and the same on repeated
invocations. Plain sorted() would put '(' first, but conceptually it's
nice to have it last, so one can think "this will be installed if
those other things are listed first". The installer does not care of
course.
mkosi: optionally insert less packages automatically
systemd, systemd-udev and some other packages were always
automatically inserted since the first version of mkosi. But this
approach is too heavy in some cases. To allow more flexibility while
maintaining backwards compat, this patch does two interleaved changes:
1. BasePackages=yes|no|conditional is added.
With BasePackages=yes, we get status quo ante.
With BasePackages=no, only packages specified by the user are
installed.
With BasePackages=conditional, on rpm systems, packages are
installed if the matching base package would be installed. On
deb/arch/… systems, equivalent to BasePackages=yes. (In other words,
only rpm supports this natively. I'm not aware of similar
functionality in other installers. install_arch() tries to achieve
something similar by comparing package sets.)
The default is "yes", to preseve unchanged behaviour by default.
2. Various packages that were installed unconditionally are converted
to be conditional.
This way we make more cases work automatically out of the box
(e.g. the user only needs to specify 'systemd', not the
subpackages), but allow deps to be cut down.
Looking at some concrete examples in Fedora:
- dracut: dracut is only necessary if using an initrd (and of
course only if the initrd image is generated using dracut, and only
if it is generated locally). But even for bootable images, a kernel
with no initramfs might be enough. Or when using kvm, a direct
kernel boot with an externally-provided kernel+initrd, etc.
So pulling in dracut and dependencies is often not necessary.
- kernel-modules: despite the name, actually basic modules are in
kernel-core. And kernel-modules is non-core modules. (And yes,
kernel-modules-extra is extra-non-core-modules, and
kernel-modules-internal is something else yet again).
- kernel-core: as with dracut, having a kernel image is not necessary
in many scenarios.
- systemd: on big example where it's not useful is systemd itself.
When using mkosi to build a systemd test image from sources,
installing the system package forces all deps to be installed, which
might not be desired. Dropping systemd from the installation list
makes mkosi useful for non-systemd stuff (some container types or
non-systemd inits).
When installing systemd from sources, we do *not* want subpackages like
systemd-udev to be installed. Having a mix of systemd-from-sources and
a different compilation of systemd-udev-from-distro is likely to cause
linking issues and other incompatibilities.
- grub* / systemd-boot: people might want to use grub even if sd-boot
is available, etc.
For conditionalization, why do it using rpm syntax, and not interally
in code? For some cases, the effect is almost the same. But in other
cases, the installer has more information. For example, glibc usually
is not specified explicitly, but is pulled in through deps. We want
glibc-minimal-langpack only if something else pulls in glibc, and we
cannot resolve this from inside mkosi. But even for other packages,
the user might use a meta-package or a package group or a file path or
a virtual provide, and we cannot know in general what package will be
used to resolve the dependency. Using boolean dependencies makes this
much more reliable.
For example, it's super useful to comment on a list of packages:
Packages=
glibc-minimal-langpack # avoid pulling in other glibc langpacks
less # this makes 'systemctl' much nicer to use ;)
;-comments are not allowed, because the colon doesn't stand out
enough. I was considering also adding comment_prefixes=('#',) to
disallow ;-comments everywhere, but that'd be a compatibility break
which I don't see enough justification for.
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/zbyszek/src/mkosi/mkosi/__main__.py", line 31, in <module>
main()
File "/home/zbyszek/src/mkosi/mkosi/__main__.py", line 11, in main
args = parse_args()
File "/home/zbyszek/src/mkosi/mkosi/__init__.py", line 5283, in parse_args
args = parse_args_file_group(argv, default_path)
File "/home/zbyszek/src/mkosi/mkosi/__init__.py", line 5320, in parse_args_file_group
return parser.parse_args(argv_post_parsed)
File "/usr/lib64/python3.9/argparse.py", line 1818, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "/usr/lib64/python3.9/argparse.py", line 1851, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "/usr/lib64/python3.9/argparse.py", line 2060, in _parse_known_args
start_index = consume_optional(start_index)
File "/usr/lib64/python3.9/argparse.py", line 2000, in consume_optional
take_action(action, args, option_string)
File "/usr/lib64/python3.9/argparse.py", line 1912, in take_action
argument_values = self._get_values(action, argument_strings)
File "/usr/lib64/python3.9/argparse.py", line 2444, in _get_values
self._check_value(action, value)
File "/usr/lib64/python3.9/argparse.py", line 2496, in _check_value
if action.choices is not None and value not in action.choices:
File "/usr/lib64/python3.9/enum.py", line 373, in __contains__
raise TypeError(
TypeError: unsupported operand type(s) for 'in': 'str' and 'EnumMeta'
With the patch:
usage: mkosi [-h] [--version]
...
{build,clean,help,summary,genkey,bump,serve,build,shell,boot,qemu,ssh} ...
mkosi: error: argument -t/--format: unknown Format: 'asfd'
It would be nice to also get rid of the list of options in the help
message: it's not helpful to the user at all.
mkosi: trim file system after finishing all writes
This is not really about actually generating sparse images, but about the
side effect that unused blocks will appear as zero, and thus the images
will be able to compress better.
mkosi: make workspace dir configurable and discover mkosi.workspace/
So far we determined the workspace directory from $TMPDIR and /var/tmp
if that wasn't set. In some cases this is less than ideal though, in
particular if one wants to build a btrfs image but /var/tmp is not
btrfs. In that case it makes sense to always use a local workspace dir.
This adds support for this:
WorkspaceDirectory= + --workspace-dir= may not be used to configure it
explicitly.
if "mkosi.workspace/" exists, we use it automatically.
i kept hitting this with OutputFormat.gpt_btrfs.
maybe the logic of including usr_only in minimize cases requires
revisiting? maybe "/usr" itself can be a subvol?
mkosi: in UsrOnly=yes mode, generate a mount.usr= kernel cmdline switch
So far, we relied on gpt-auto's logic to automatically find the root fs
on the image. This is not supported for finding the /usr partition
though, hence generate a mount.usr= kernel cmdline option to address
this.
(And it's unlikely we'll add auto-discovery for /usr, since we cannot
realistically decide whether its worth waiting for /usr. After all we
*know* we can't proceed without a rootfs, hence it makes sense to
generically wait for one if we never got one configured one. But a /usr
partition is an optional thing, hence it doesn't make sense to wait for
it if we didn#t get told about it)
mkosi: rework how we pass roothash/usrhash/other metadata to kernel-install
Previously, we'd replace the main kernel-install script with one
generated by mkosi and pass the roothash/usrhash as final argument on
the command line. That's less than ideal though, since it collides with
the documented way kernel-install dropins are called (those arguments
are initrds, not root hashes according to the docs). Moreover, it leaves
no nice way to pass usrhash vs. roothash or pass other information.
Let's fix this all, by changing the script to look for the roothash in
$ROOTHASH and the usrhash in $USRHASH. Then pass these env vars when
invoking the script, and drop any additional arguments.
Also, add the $IMAGE_ID and $IMAGE_VERSION in a similar fashion and then
use those preferably to name the generated kernel images.
With all this in place we are fully "standards-compliant" again,
matching the documented kernel-install drop-in API, the kernel images
are named a lot more nicely, and the usrhash logic starts working too.
mkosi: make sure to mount /root/ from outside if UsrOnly=
When using mkosi in build script mode we copy stuff into /root/, as
that's where we build things in. In UsrOnly= mode that directory is not
persistent though, which makes the excercise pointless. Hence, in
UsrOnly= mode let's mount /root/ from a persistent directory from our
workspace.
This is a recent addition in systemd, required to make cases work where
the initrd creates a root fs on first boot and images are initially
shipped without one.
(See https://github.com/systemd/systemd/pull/19234)
While we are at it, let's order the list of extra initrd files
alphabetically.
mkosi: if we have an image id (any maybe image version) use it to label partitions
This makes a lot of conceptual sense (since labels probably should
better declare what you find *in* a partition instead of just a
description of the type of it, because we already can generate that
automatically from the GPT UUID).
But it's also useful in conjunction with systemd's logic of always
picking the newest version of a root partition to mount in
systemd-nspawn and related commands, i.e.
https://github.com/systemd/systemd/pull/18958
On top of that this is useful in combination with SplitArtifacts=as then
the partitions and the split files will carry the same (except for the
suffix).
mkosi: add "image-id" concept in addition to "image-version"
A previous commit added an image version concept. Let' build on that and
add "image-id" too to match this systemd PR:
https://github.com/systemd/systemd/pull/19093
The image ID is used for naming the enerating artifacts if that's not
explicitly set (instead of the generic "image" as before) and is passed
to th build scripts via an IMAGE_ID env var matching the IMAGE_VERSION
the earlier commit added.
The idea is that the image version and ID passed here is ultimately
written to /etc/os-release by the build script (possibly mangled) and
the mkosi concepts for this hence ultimately propagate into the image
itself in some form even if not 1:1 (depending on the project of
course).
projects / thirdparty / mkosi.git / log
