man: refer to settings primarily by their config file name
This implements the first part of #769: the structure of the man page
is changed from:
Command Line Parameters
--foo
...
Command Line Parameters and their Settings File Counterparts
--foo Output Foo=
...
to:
Settings
[Output] section
Foo=
--foo
...
I think the new order is both easier to read _and_ easier to maintain.
E.g. the table with option name mappings was missing a few entries.
I dropped "(see above)" and "(see below)" in various places. The user
knows that they can search for a given option name, and we don't need
to tell them. And since the target of this "anchor" usually isn't right
below or right above, they will need to use search _anyway_.
An exception is when we wanted to refer to a description of a file
in the Files section. In those cases I changed "(see below)" to
"(see "Files" section below)" so the user actually knows what to look
for.
I inserted FIXMEs for various issues. Trying to fix everything at once
would lead to an unreviable commit…
The regeneration of man/ is not included in the commits, because it's
annoying to constantly fix this up when rebasing patches. I'll do it
one go at the end.
mkosi: print paths relative to the current working directory
We would print a bunch of paths, and they tend to be quite long,
especially if 'mkosi.output' is used. The output on the console is
supposed to be pretty and easy to read. The user is most likely to
use the path to interact with the file in the same terminal, e.g.
'less mkosi.output/something.manifest', so let's just print the
paths that are underneath the current directory as relative to it.
This makes the output more readable while the paths can still be
used directly in subsequent commands most of the time.
Also, drop "Successfully" from messages: it is a long word, and
superfluous, since we either print an error message or this.
* Tue Mar 16 2021 Kamil Dudka <kdudka@redhat.com> - 2.3.1-1
- new upstream release
...
"""
The idea is that it's easy to diff such files. When the changes are
relatively small, for example between the same distro on different days,
we can easily see what changed, and why. For example, between F34 and F35
I get the following diff:
So when the package changed, we know *why* it changed, and can often
jump directly to the related bugs and such. For example, we now get
cracklib-dicts (yes, very useful), but the package itself didn't change.
Something else pulls it in [1]. Such a diff is useful when trying to
see debug regressions between related builds or minimizing the package
set.
I would like to set environment variables for other scripts. But I
don't think adding separate options make sense: it should be fine to
just set the same environment variable for all scripts that are
invoked… After all, variables are best for "global" settings. Scripts
already get positional arguments that allow them to distinguish build
phases, so they don't need to use variables for this.
mkosi: pull in envvar value from the environment if no value specified
This makes --build-env=FOO behave like --build-env=FOO=$FOO. This is a
common case, and it makes some invocations more convenient. In
particular, we avoid exposing the variable on the commandline.
mkosi: define NamedTuple to hold luks_setup_all() output
I'm using a NamedTuple here, instead of a dataclass, because the
dataclass does not support indexing and iteration. It certainly would
be possible to define those, but with typing it becomes quite a lot of
code (and typing of __iter__() that supports both indexing and slicing
is tricky and verbose).
BuildOutput has no inherent ordering — it's a bunch of completely
different objects. But here we have a list of partitions, so it's
reasonable to use a tuple for this.
mkosi: create cache copy also for non-raw output formats
I don't quite grok the logic to skip the creation of the cache copy.
I'm using Format=directory, and I expected the cache copy to be there.
Since we can't overwrite one directory with another, unlink the output
when copying. (For raw output, which is a file, we don't need to unlink
because we can replace the file atomically.)
With this change, it is created and everything seems to work fine.
mkosi: do not suppress non-Exception exceptions in unlink_try_hard()
In 01eac0078a06334ed938b68a09db8799b490dfe0 annotations to suppress
warnings about this were added, sadly without any explanation, so we
can only guess what the author had in mind. But it doesn't seem
appropriate… If I press ^C while removing some files, it seems quite
wrong to continue execution. Also, if those operations fail on
permission errors, we also don't want to continue.
("except:" catches BaseException, which includes KeyboardInterrupt and
other things which should only be caught in very special
circumstances, like interactive shells.)
mkosi: include the ellipsis+dot in complete_step() in the format string
complete_step() already added an ellipsis and dot on its own (to the
opening message and to the cloing message, respectively), but in some
places we had "..." or "." in the format string anyway, so we eneded
up with six dots or four dots in output. But instead of fixing those
places, I think it's better to go in the opposite direction, and include
the terminator in the format string.
This is nicer because:
- It's just easier to see the complete format string including the
terminator. The "saving" of a few characters by including it
automatically isn't very important.
- If the line ends in a path, we should skip the terminator to make
the path easier to select-and-paste. This is now trivial: just skip
the terminator in the format string.
- If the step is "immediate" and we expect it to finish quickly and
without any further output (like calling os.unlink() or such), we
can skip the ellipsis.
While at it, let's change "..." to "…", it looks nicer and saves some
screen real estate.
mkosi: de-emphasize completion of steps after an exception has occured
I had the following failure:
...
‣ Unmounting API VFS...
‣ Unmounting Package Cache...
‣ Removing random seed...
‣ Unmounting image...
‣ Running second (final) stage...
‣ Mounting image...
‣ Setting up basic OS tree...
‣ Unmounting image...
Traceback (most recent call last):
...
FileExistsError: [Errno 17] File exists: '/home/zbyszek/src/mkosi-initrd/.mkosi-5jv7jtu2/root/etc'
The error is generated in the block "Setting up basic OS tree...", but
the exception traceback is not printed immediately, so it seems like
it was generated in the "Unmounting image..." block. There could be
multiple lines of such cleanup, and it's hard to tell where the error
occurred without knowledge of mkosi internals. Let's de-bold and
parenthesize the steps that happen when we are in exception handling
blocks, so the reader can discern this more easily.
‣ Unmounting API VFS...
‣ Unmounting Package Cache...
‣ Removing random seed...
‣ Unmounting image...
‣ Running second (final) stage...
‣ Mounting image...
‣ Setting up basic OS tree...
‣ (Unmounting image...)
Traceback (most recent call last):
...
FileExistsError: [Errno 17] File exists: '/home/zbyszek/src/mkosi-initrd/.mkosi-3legjys3/root/etc'
We said that all files are optional *before* starting the list of files,
then for each individual file, and then again *after* the list. Let's
simplify this by describing files without mentioning that is is optional.
Similarly, we don't need to emphasize the fact that scripts are executable
for each script… readers generally know that.
Let's also make the terms bold, so it's easier to find the interesting
paragraph in this wall of text.
Also break into paragraphs more; remove underlining from "OS" — it's a
normal abbreviation and the markup was confusing; use "/" at the end
of directories; use more markup for paths; move the description of
mkosi.skeleton above mkosi.extra.
Luca Boccassi [Fri, 18 Jun 2021 12:14:42 +0000 (13:14 +0100)]
qemu: revert back to virtio from virtio-scsi-pci on Debian
Debian images on qemu fails to boot with virtio-scsi-pci. Reverting back to
virtio does not show issues with the EFI shell, everything boots correctly,
so do that when running Debian images.
mkosi: if image version logic is enabled, make sure to generate root= entry in kernel command line
When the --image-version= logic is enabled this indicates that multiple
versions of an OS might be used in parallel within the same partition
table eventually. That makes it essential to boot the right root file
system from each unified kernel, so that kernel and root fs always match
up correctly.
If Verity is on this already worked, since we encode roothash= on the
kernel cmdline in that case, and that implies the root fs to use. Now,
ensure for verity-less cases that this works too.
We use PARTLABEL= match, using the image-id/image-version combination,
i.e. matching how we pick the label for the partition.
If the image version logic is not used, let's continue to not use the
a root= for simplicity reasons. In that case automatically finding the
root fs via fully automatic discovery should be simple and robust.
Michal Koutný [Fri, 11 Jun 2021 16:54:13 +0000 (18:54 +0200)]
Github Action: Install pip module matching the git revision
When the action is referenced with a particular git revision (e.g.
'uses: foo/mkosi@refspec') the given revision should be deployed.
Achieve that by installing pip directly from the action directory
(instead of a released/packaged version).
Alex Wicks [Thu, 1 Jul 2021 01:58:07 +0000 (02:58 +0100)]
Correct pacman hook name
Fixes systemd/mkosi#750
Pacman hooks must end with .hook as per Arch wiki
However mkosi's hook to update bootctl when the kernel updates ends in -hook instead of .hook
This means the hook doesn't get fired and kernel upgrades fail
Luca Boccassi [Fri, 18 Jun 2021 18:14:38 +0000 (19:14 +0100)]
nspawn: fallback to host network mode if networkd is not running
On Debian systemd-networkd is not the default network manager, so
there's nothing configuring the veth bridge.
If networkd is not running, simply fallback to using the host
network. This will not provide isolation, but at least the
connectivity is restored.
Luca Boccassi [Fri, 18 Jun 2021 16:52:49 +0000 (17:52 +0100)]
qemu: fallback to usermode network if networkd is not running
On Debian systemd-networkd is not the default network manager, Network-Manager is.
Fallback from TAP + veth to usermode networking. It is slower and NATted, but it
still works.
Blair Bonnett [Fri, 25 Jun 2021 07:40:57 +0000 (09:40 +0200)]
mkosi: add option to resolve symlinks when copying build sources.
This allows common build scripts or resources to be symlinked into the
build source directory, but then be copied into the build/final image as
actual files instead of symlinks.
mkosi: when using cached images, randomize fs and partition uuids explicitly
This addresses an annoying issue when building cached images containing
btrfs file systems: the kernel btrfs driver refuses mounting or handling
multiple different file systems with the same uuid. This means using an
image and building the next at the same time fails — as long as the
UUIDs of the newly build image aren't refreshed.
This patches makes sure when using a cached image we'll refresh disk,
partition and file system uuids. We generate them randomly, exactly like
we would have them when using non-cached builds.
This also ensures that the partition labels are rewritten when images
versions are bumped.
(Eventually we should probably start hashing the uuids from the
configuration state in some form, to provide a certain level of
reproducibility, but for now let's just randomize them.)
When building images with image IDs and image versions it's a good idea
to reuse the pre-pared cached images from the earlier versions (I mean,
that's why cacheing/incremental operation exists). So far the cached
image names was derived 1:1 from the output file name, but if that
includs the image version it will be different for each build and hence
never suitable as cache.
Hence, if the image ID/version logic is used, let's change the naming
regime for the cache file, and let's derive it direcly from image name,
not taking the version into account.
rpmlint warns that this is dangerous. We don't modify the argument,
so actually this is safe here, but it's still nice to avoid this
potential pitfall.
× Run python3 -m mypy mkosi
mkosi/__init__.py:12:1: error: Library stubs not installed for "dataclasses"
(or incompatible with Python 3.6) [import]
import dataclasses
^
mkosi/__init__.py:12:1: note: Hint: "python3 -m pip install types-dataclasses"
mkosi/__init__.py:12:1: note: (or run "mypy --install-types" to install all missing stub packages)
mkosi/__init__.py:12:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
mkosi/__init__.py:6247:12: error: Too many arguments for "CommandLineArguments"
[call-arg]
return CommandLineArguments(**vars(args))
^
Found 2 errors in 1 file (checked 3 source files)
Error: Process completed with exit code 1.
mkosi: rework compression config, add support for zstd
We had --compress and --xz. This is changed to --compress-fs (which is like
the old --compress), --compress-output (which is like --xz, except configurable),
and --compress (which tries to do the right thing depending on the output type).
With zstd, the difference in decompression time is massive:
$ time zstdcat image.cpio.zstd >/dev/null
0.221 s total
$ time xzcat image.cpio.xz >/dev/null
3.592 s total
$ ls -lg image.cpio.* image.cpio
-rw-rw-r-- 1 zbyszek 149408256 May 21 13:56 image.cpio
-rw-r--r-- 1 zbyszek 53032056 May 21 13:57 image.cpio.xz
-rw-r--r-- 1 zbyszek 59085907 May 21 13:49 image.cpio.zstd
v2:
- rename options to --compress-fs and --compres-output, unify handling
of cpio and tar.
v3:
- use tar --use-compress-program= instead of subprocess.Popen()
This may be useful to create initramfs images, see
https://github.com/keszybz/mkosi-initrd.
I also tried an implementation using libarchive. The code was a little
bit simpler, but unfortunately the generated archive wasn't to the
kernel's liking. I think both the compressio options *and* the
underlying cpio format were a bit different. I think we should just accept
that the kernel is only tested with the specific output produced by cpio
with some specific settings.
mkosi: add variant of subprocess.Popen() that does not wait
We had run() which was a wrapper around subprocess.run().
This addes spawn(), which starts the process and does not wait for it.
v2:
- run() is changed to be call spawn() and wait and throw an exception
if check is True.
v3:
- run() is a completely separate function again.
It turns out that run() does a bunch of handling of input and output
that would need to be recreated. In the end, the duplication of
logging is a smaller issue than fiddling with input and output
encodings.
--- /tmp/out1 2021-06-05 17:01:24.327828067 +0200
+++ /tmp/out2 2021-06-05 17:01:32.367897483 +0200
@@ -59,21 +59,19 @@
--version show program's version number and exit
Distribution:
- -d {fedora,debian,ubuntu,arch,opensuse,mageia,centos,centos_epel,clear,photon,openmandriva}, --distribution {fedora,debian,ubuntu,arch,opensuse,mageia,centos,centos_epel,clear,photon,openmandriva}
+ -d, --distribution {fedora,debian,ubuntu,arch,opensuse,mageia,centos,centos_epel,clear,photon,openmandriva}
Distribution to install
- -r RELEASE, --release RELEASE
+ -r, --release RELEASE
Distribution release to install
- -m MIRROR, --mirror MIRROR
- Distribution mirror to use
+ -m, --mirror MIRROR Distribution mirror to use
--repositories REPOS Repositories to use
--architecture ARCHITECTURE
Override the architecture of installation
projects / thirdparty / mkosi.git / log
