We would create a file with 0 padding to force it to a certain size.
This seems fairly ugly, in particular when we later want to use the
blob in other contexts. Also, since 9e0b115e0c, this shouldn't be necessary.
And if we're just copying bytes from a file we already have open, with no
padding, we might just as well do this without forking.
mkosi: add support for verity also for generated roots
In a sysext, I have a squashfs partition that I want to do verity for.
Before this change, we'd fail with an assertion that the root device partition
is not set.
The conditional is bit busy, but I couldn't find a shorter form that
would make mypy happy.
The return value currently isn't used. By returning the Partition
object we can get access to things like the blockdev path, but also
size, so this is more flexible.
Overlay uses char(0,0) files to mark items that are present in the lower
layers, but were removed in the upper layer. At least for the case
of sysexts, we want to get rid of those. Maybe if other uses of this
feature appear, we might want to make this removal optional.
Hat tip to @brau_ner for explaining what those files are.
Add support for creating "sysexts" with the BaseImage option
The general approach is to first create a base image with the base
set of packages, and then install the new layer on top. To make this
work nicely, the base layer should have --clean-package-metadata=false:
This produces a sysext, except that the release files are missing.
I currently create them through a finalize script. At some point we might
move this functionality into mkosi, but I think it's better to get some
experience with sysexts first.
This will have very bad results unless one controls *all* places where
a file can be created. This is possible in systemd and C code, but is
unlikely to work well here, where we have lots of high-level code and
call helpers which can also create files.
So let's use a reasonable value, i.e. 0o022 as the umask during runtime.
This will let us create files with the expected permissions in the image.
After we produce our outputs, we already chown and chmod using the
original umask, so the output have the expected permissions anyway.
indentation: keep each condition on a separate line
We would have two conditions joined by 'and', and we would have
the first operand and half of the second on the first line, and
the remainder of the second operand on multiple lines.
The general idea is that those closes collect attributes and information
about the abstract partition table, but are independent of any underlying
block device. The definitions exists indendependently of a block device
and can be applied later on.
Partitions are referred to by enum PartitionIdentifier.
The calculation of the space necessary for those partitions is centralized
in the class, so we don't have multiplace places where we arrive at slightly
different formulas for the expected disk size.
This adds support for creating signed GPT disk images. If Verity=signed
is set this will not only generate and insert Verity data into the
image, but then use the resulting root hash, sign it and include it in
an additional partition. It will also write the resulting PKCS7
signature out into a new .roothash.p7s file.
This scheme is compatible with kernel 5.4's PKCS7 signature logic for
dm-verity: the resulting .p7s file can be passed as-is to the kernel (or
systemd's RootHashSignature= setting).
The partition this embedds contains a simple JSON object containing
three fields. The verity root hash, the PKCS7 data (i.e. the same data
as in the .p7s file, but in base64), and SHA256 fingerprint of the
signing key. This partition is supposed to be read by the image
dissection logic of systemd, to implement signed single-file images.
(The corresponding PR for systemd I am still working on).
This opens up two avenues for image verification:
1. Everything in one file: the single, "unified" GPT disk image contains
three partitions, for payload data, verity data and verity signature.
2. Split out: root hash and its signature are stored in two "sidecar"
files.
(Of course I'd personally always go the "unified" way, but given the
RootHashSignature= logic exists already, and it's easy to support, let's
support it.)
This uses the key already used for doing secureboot image signing.
Conceptually this makes a ton of sense: we sign the same stuff here
after all: the contents of the image, supporting two different
entrypoints to the image: one via UEFI booting the image, and once for
attaching directly to an image from a running system. Admittedly, the
"mkosi.secure-boot.key" and "mkosi.secure-boot.certificate" monikers for
this key pair might be a bit suprising though.
Joerg Behrmann [Mon, 4 Oct 2021 15:17:06 +0000 (17:17 +0200)]
typing: appease pyright and disable mypy unusued import warning
Since pyright reuses the same comments to ignore types, one can get into
unsatisfiable situations, where one type checker accepts something but the other
doesn't. At this point we only have three instances of type ignore hints anyway,
sol et's relax our mypy settings somewhat, so that we can shut up pyright, when
necessary.
dracut: make sure images with IMAGE_VERSION but without IMAGE_ID can boot
If an image version is set but no image ID mkosi's table currently
places generic labels in the partition labels, instead of the image ids
(because we have none...). This means we cannot refrence the root
partition via root=PARTLABEL=… on the kernel cmdline. Hence do not do
that.
This change ensures we don't try to use IMAGE_ID-based root=PARTLABEL=
kernel cmdline swtches without IMAGE_ID being set.
(Note that the main reason IMAGE_ID/IMAGE_VERSION exists is to allow
versioned setups, i.e. where multiple versions of the same thing exist.
In such a case it's important to reference the right rootfs version that
matches the whole setup we are building here. But if IMAGE_ID isn't set
then this multi-version logic is not desired and we can assume that only
a single version of the OS is in the partition table, and thus rely on
gpt-auto-generators automatic root file system discovery)
Ray Sit [Sat, 11 Sep 2021 05:09:52 +0000 (15:09 +1000)]
Renamed argument UseSystemRepositories to UseHostRepositories
The use of the term system can be misleading as system can refer
to many different things. The term host is more accurate for what the
option is. Also updated the command help to identify the option is
for dnf-based distros.
The option is there, we should have some documentation for it.
But I think the current split is not very useful (e.g. why
"workspace-command", what does this even mean?), and I expect that
we'll want to review the list and hide some options before documenting
this. So the choices are not described in the man page yet.
Add --debug=disk and show sectors and raw sfdisk configs if selected
The low-level sfdisk configs are quite useful when trying to figure
out what sfdisk doesn't like. But we shouldn't show this by default,
so this adds a new --debug selector and hides the detailed output
otherwise.
mkosi: reserve much less area for GPT header and footer
We would reserve 1MB on both ends. This is what sfdisk (and other
tools) do by default. It probably makes sense for real disks, which
are large enough that 1–2 MB don't matter, and one might want to add
partitions later. For our images this isn't very useful. In fact, we
could probably go lower than 128 partitions, since we generally know
exactly how many we will create. But I'm leaving that for later,
because the savings are not large, and there might be compatiblity
issues involved. If it turns out that the changes done here don't
cause problems, we could consider making the max partition count
smaller, to save another 12 kb or so.
The "grain" (partition alignment) is set to 4096 bytes, even on
devices with 512 byte sectors. 4k devices are becoming more popular,
and we could trigger bad performance if the partitions was misaligned.
The code is reworked to take the specified first-lba into account.
(In the initial version of this patch that was posted, 'grain:4096'
was passed to sfdisk, and this seemed enough. But with various other
combinations of image sizes, sfdisk sometimes refuses to create the
expected layout when first-lba is not specified. So this version of
the patch specifies both values.)
Before: ‣ Resulting image size is 1.8M, consumes 848.0K.
After: ‣ Resulting image size is 808.0K, consumes 808.0K.
mkosi: use blockdev --rereadpt instead of partx --update
With smaller devices, we trigger a bug in partx. It was fixed in
util-linux-2.37, but at least Fedora 34 still has 2.36. With
blockdev --rereadpt, it's the kernel which parses the table, which
should avoid the issue.
manifest: do not gather changelogs if they will not be used
I wanted to keep the data collection and serialization parts completely
separate, and I thought that gathering of the changelogs is insignificant.
But with a lot of packages, it can add up. So let's do it only if it'll
be useful later on.
A direct motivation is https://bugzilla.redhat.com/show_bug.cgi?id=2002038:
rpm gets confused by some package names and cannot show the changelog.
Ray Sit [Thu, 9 Sep 2021 09:42:03 +0000 (19:42 +1000)]
Added option to skip use of hardcoded default dnf-distro repositories
This UseSystemRepositories/--use-system-repositories option allows
mkosi to use the dnf repo files that exist on the system, instead of
locking the script into using the hardcoded repo file that only
contains the default repos for each distro. It is only applicable
to dnf based distros.
action: Extract package manager build logic into separate script
To make the action easier to debug, let's extract the logic for
building dnf and pacman into a script that we can run outside of
github actions. We also include an accompanying mkosi config that
can be used to test the script locally. Building the mkosi image
builds and installs all the package managers into the final image.
The final image can then be booted in QEMU, mkosi can be cloned and
tested using the installed package managers.
This commit also includes a bunch of general improvements to the
build logic:
- Install everything to /usr/lib/x86_64-linux-gnu so Ubuntu finds
libraries
- Make the script idempotent so it can take advantage of caching
(both in mkosi now using mkosi.builddir/ and in the future with GA)
- Use versioned build directories
- Update all package versions to the latest and make the necessary
modifications to the build script to keep everything building.
Let's turn off "DES" encryption of X509 private keys we generate. It's
not really a useful feature, we aren't really an interactive tool (and
thus asking for a password is weird). And it's not secure at all.
mkosi+tests: fix handling and tests for Path arguments
This should have been done in 5a24a9fd9e9eea4c1a9cf2e9c1980697fd9b922b.
The code is getting more and more complex, because previously we
relied on having a string which could be both multiple items with
a separator, or just a single thing… With Path() this doesn't work
nearly so nicely.
mkosi: do not expand globs in manager metadata removal twice
We would do globbing internally in clean_*_metadata(), and then again
in remove_glob(). We want to do globbing early to only emit the
message when there's anything to remove. So let's stop calling
remove_glob().
While at it, make the internal variable names a bit shorter.
The idea is to allow specifing "--finalize-script=" to override the finalize
script to not be invoked, even in "mkosi.finalize" is present.
Right now, if there's a mkosi.build/mkosi.finalize/… file present, the
only way to prevent the script from running is to specify something
like --build-script=/bin/true (or move the script out of the way).
Sometimes it's useful to be able to skip the script. The code is changed
to convert '' to a placeholder value that is taken to mean that no script
should be executed. (None is already used to mean "no config", and implies
autodetection, so we can't use it for this purpose.)
Allow overriding autodetection of package manager metadata cleanup
There are cases where the package metadata should be removed, even
though the manager is present. For example, when the manager is pulled
in through dependencies or needs to be present in the image for
whatever other reasons. Similarly, there are cases where the metadata
should remain in the image, even though the package manager is
provided externally.
In particular, it is useful when building an image which will later on
be used as a "sysext base". Without the metadata, we will try to
install all packages again. So we want to keep the dnf/rpm metadata.
Later, when building a "sysext", dnf/rpm will be visible from the base
layer, even though it will not be present in the sysext layer being built.
We want to remove the dnf/rpm metadata.
projects / thirdparty / mkosi.git / log
