Ray Sit [Sat, 11 Sep 2021 05:09:52 +0000 (15:09 +1000)]
Renamed argument UseSystemRepositories to UseHostRepositories
The use of the term system can be misleading as system can refer
to many different things. The term host is more accurate for what the
option is. Also updated the command help to identify the option is
for dnf-based distros.
The option is there, we should have some documentation for it.
But I think the current split is not very useful (e.g. why
"workspace-command", what does this even mean?), and I expect that
we'll want to review the list and hide some options before documenting
this. So the choices are not described in the man page yet.
Add --debug=disk and show sectors and raw sfdisk configs if selected
The low-level sfdisk configs are quite useful when trying to figure
out what sfdisk doesn't like. But we shouldn't show this by default,
so this adds a new --debug selector and hides the detailed output
otherwise.
mkosi: reserve much less area for GPT header and footer
We would reserve 1MB on both ends. This is what sfdisk (and other
tools) do by default. It probably makes sense for real disks, which
are large enough that 1–2 MB don't matter, and one might want to add
partitions later. For our images this isn't very useful. In fact, we
could probably go lower than 128 partitions, since we generally know
exactly how many we will create. But I'm leaving that for later,
because the savings are not large, and there might be compatiblity
issues involved. If it turns out that the changes done here don't
cause problems, we could consider making the max partition count
smaller, to save another 12 kb or so.
The "grain" (partition alignment) is set to 4096 bytes, even on
devices with 512 byte sectors. 4k devices are becoming more popular,
and we could trigger bad performance if the partitions was misaligned.
The code is reworked to take the specified first-lba into account.
(In the initial version of this patch that was posted, 'grain:4096'
was passed to sfdisk, and this seemed enough. But with various other
combinations of image sizes, sfdisk sometimes refuses to create the
expected layout when first-lba is not specified. So this version of
the patch specifies both values.)
Before: ‣ Resulting image size is 1.8M, consumes 848.0K.
After: ‣ Resulting image size is 808.0K, consumes 808.0K.
mkosi: use blockdev --rereadpt instead of partx --update
With smaller devices, we trigger a bug in partx. It was fixed in
util-linux-2.37, but at least Fedora 34 still has 2.36. With
blockdev --rereadpt, it's the kernel which parses the table, which
should avoid the issue.
manifest: do not gather changelogs if they will not be used
I wanted to keep the data collection and serialization parts completely
separate, and I thought that gathering of the changelogs is insignificant.
But with a lot of packages, it can add up. So let's do it only if it'll
be useful later on.
A direct motivation is https://bugzilla.redhat.com/show_bug.cgi?id=2002038:
rpm gets confused by some package names and cannot show the changelog.
Ray Sit [Thu, 9 Sep 2021 09:42:03 +0000 (19:42 +1000)]
Added option to skip use of hardcoded default dnf-distro repositories
This UseSystemRepositories/--use-system-repositories option allows
mkosi to use the dnf repo files that exist on the system, instead of
locking the script into using the hardcoded repo file that only
contains the default repos for each distro. It is only applicable
to dnf based distros.
action: Extract package manager build logic into separate script
To make the action easier to debug, let's extract the logic for
building dnf and pacman into a script that we can run outside of
github actions. We also include an accompanying mkosi config that
can be used to test the script locally. Building the mkosi image
builds and installs all the package managers into the final image.
The final image can then be booted in QEMU, mkosi can be cloned and
tested using the installed package managers.
This commit also includes a bunch of general improvements to the
build logic:
- Install everything to /usr/lib/x86_64-linux-gnu so Ubuntu finds
libraries
- Make the script idempotent so it can take advantage of caching
(both in mkosi now using mkosi.builddir/ and in the future with GA)
- Use versioned build directories
- Update all package versions to the latest and make the necessary
modifications to the build script to keep everything building.
Let's turn off "DES" encryption of X509 private keys we generate. It's
not really a useful feature, we aren't really an interactive tool (and
thus asking for a password is weird). And it's not secure at all.
mkosi+tests: fix handling and tests for Path arguments
This should have been done in 5a24a9fd9e9eea4c1a9cf2e9c1980697fd9b922b.
The code is getting more and more complex, because previously we
relied on having a string which could be both multiple items with
a separator, or just a single thing… With Path() this doesn't work
nearly so nicely.
mkosi: do not expand globs in manager metadata removal twice
We would do globbing internally in clean_*_metadata(), and then again
in remove_glob(). We want to do globbing early to only emit the
message when there's anything to remove. So let's stop calling
remove_glob().
While at it, make the internal variable names a bit shorter.
The idea is to allow specifing "--finalize-script=" to override the finalize
script to not be invoked, even in "mkosi.finalize" is present.
Right now, if there's a mkosi.build/mkosi.finalize/… file present, the
only way to prevent the script from running is to specify something
like --build-script=/bin/true (or move the script out of the way).
Sometimes it's useful to be able to skip the script. The code is changed
to convert '' to a placeholder value that is taken to mean that no script
should be executed. (None is already used to mean "no config", and implies
autodetection, so we can't use it for this purpose.)
Allow overriding autodetection of package manager metadata cleanup
There are cases where the package metadata should be removed, even
though the manager is present. For example, when the manager is pulled
in through dependencies or needs to be present in the image for
whatever other reasons. Similarly, there are cases where the metadata
should remain in the image, even though the package manager is
provided externally.
In particular, it is useful when building an image which will later on
be used as a "sysext base". Without the metadata, we will try to
install all packages again. So we want to keep the dnf/rpm metadata.
Later, when building a "sysext", dnf/rpm will be visible from the base
layer, even though it will not be present in the sysext layer being built.
We want to remove the dnf/rpm metadata.
The Path object generally works better for the kind of operations that
we need to do. In some cases it can be a bit awkward — for example just
appending a suffix requires some annoying gyrations. But overall, the
amount of code text is reduced and I think it's easier to read.
Some bugs were fixed on the way: in particular dedent() was missed in
when writing some configuration files (90-mkosi-vmlinuz-add.hook,
60-mkosi-vmlinuz-remove.hook).
The intent is to have no functional change, though with this amount of
churn some regression is likely… I tested this by building a bunch
of different projects with diverse configurations, and at least mkosi
doesn't crash, though I didn't check the produced images in detail.
An additional advantage is that we have stronger typing: before many
functions took a bunch of 'str' arguments, and now the paths at least
have a distinct type.
The code was modifing the passed argv array internally. And in fact,
parse_args_distribution_group() only worked correctly if
parse_args_file_group() was previously used to modify the argv array. Let's
drop all this, and never modify the input arguments. In the few cases where
it's appropriate, a copy is made internally.
Also shorten names of internal variables. defaults_file vs. default_file vs.
all_defaults_files was only making the code harder to read.
I also added older keys: we might want to build images with
older Fedora, and it's better to have the key hashes than
not to have them. The list now includes all keys listed by
fedora-gpg-keys-34-2.noarch.
Surprisingly, none of the standard tools complain about tabs in the
code, at least in some locations. I expected that python3 would always
complain, or at least tabnanny, or pyflakes… Let's roll a simple custom
check.
Use '-P' to request perl-compatible regular expressions even the user
has a different default configures so that '\t' works as expected.
Replace contextlib.ExitStack by direct cm invocations
I think it's nicer to keep things simple. Unfortunately black
turns the first chunk into a complete mess, so it needs to be disabled.
With python 3.9+ we will be able to drop the line continuations.
With this change, ARG_DEBUG is always the same (mutable) object.
Before it was first a tuple, and then a list, so the object (and type)
changed. When we did 'from backend import ARG_DEBUG', we would get a
reference to the tuple, and then when ARG_DEBUG was adjusted in backend,
the import wasn't properly updated. ('backend.ARG_DEBUG' would refer
to one object, and '__init__.ARG_DEBUG' to a different one.)
man: refer to settings primarily by their config file name
This implements the first part of #769: the structure of the man page
is changed from:
Command Line Parameters
--foo
...
Command Line Parameters and their Settings File Counterparts
--foo Output Foo=
...
to:
Settings
[Output] section
Foo=
--foo
...
I think the new order is both easier to read _and_ easier to maintain.
E.g. the table with option name mappings was missing a few entries.
I dropped "(see above)" and "(see below)" in various places. The user
knows that they can search for a given option name, and we don't need
to tell them. And since the target of this "anchor" usually isn't right
below or right above, they will need to use search _anyway_.
An exception is when we wanted to refer to a description of a file
in the Files section. In those cases I changed "(see below)" to
"(see "Files" section below)" so the user actually knows what to look
for.
I inserted FIXMEs for various issues. Trying to fix everything at once
would lead to an unreviable commit…
The regeneration of man/ is not included in the commits, because it's
annoying to constantly fix this up when rebasing patches. I'll do it
one go at the end.
dnf: when runnning with --network=never, use package metadata cache
This implements the ability to rebuild packages from cache that was requested
in #700. I don't think it can be said to resolve the issue, since this is
only a partial solution, but useful nonetheless.
Daan De Meyer [Fri, 20 Aug 2021 13:33:13 +0000 (14:33 +0100)]
Don't let dracut generate implicit devices in hostonly mode
If hostonly_default_device is enabled, details from the host system devices
leak into the initrd which we want to avoid. For example, this caused the
seclabel mount option to leak into the mount options of my mkosi generated
BTRFS mount options rendering the image unbootable in QEMU.
Daan De Meyer [Sun, 8 Aug 2021 12:58:25 +0000 (13:58 +0100)]
Fix pyright typing errors
pyright is an alternative type checker for Python. It's main advantage
over mypy is that it's a lot faster than mypy which makes it a lot more
pleasant to use in VSCode for in-editor type checking.
This commit fixes the typing errors flagged by pyright.
mkosi: only print the summary when explicitly requested
Originally, the summary was printed whenever a build was done.
With 1f9a758fad3debbdddd2947a19978de3909aabf7, it is only printed
for cached builds. But that is surprising: when I turn on caching,
I suddently get extra output. Let's just print the summary when
requested.
mkosi: print paths relative to the current working directory
We would print a bunch of paths, and they tend to be quite long,
especially if 'mkosi.output' is used. The output on the console is
supposed to be pretty and easy to read. The user is most likely to
use the path to interact with the file in the same terminal, e.g.
'less mkosi.output/something.manifest', so let's just print the
paths that are underneath the current directory as relative to it.
This makes the output more readable while the paths can still be
used directly in subsequent commands most of the time.
Also, drop "Successfully" from messages: it is a long word, and
superfluous, since we either print an error message or this.
* Tue Mar 16 2021 Kamil Dudka <kdudka@redhat.com> - 2.3.1-1
- new upstream release
...
"""
The idea is that it's easy to diff such files. When the changes are
relatively small, for example between the same distro on different days,
we can easily see what changed, and why. For example, between F34 and F35
I get the following diff:
So when the package changed, we know *why* it changed, and can often
jump directly to the related bugs and such. For example, we now get
cracklib-dicts (yes, very useful), but the package itself didn't change.
Something else pulls it in [1]. Such a diff is useful when trying to
see debug regressions between related builds or minimizing the package
set.
projects / thirdparty / mkosi.git / log
