Stop mounting build sources when running package managers
We now have PackageDirectories= to make local packages available
for installation so let's stop mounting build sources when running
package managers and tell users to use PackageDirectories= instead.
Recommend PackageDirectories= to install local packages
apt-get requires absolute paths to be specified and doesn't work
with relative paths. Let's instead recommend PackageDirectories=
which we know works the same for all distributions.
Choose default tools tree distribution based on host distribution
Let's choose the default tools tree distribution based on the host
distribution instead of the target distribution. Why? When building
a Fedora image from Ubuntu, It's much more likely that apt-get will
be installed to build a Debian tools tree rather than requiring dnf
to be installed to build a Fedora tools tree.
Python does not have block scoped variables which messes up typing
if you use the same variable name in 2 unrelated blocks so let's
rename this one to make mypy happy.
Let's simplify the config parsing implementation by making ParseContext
a regular class instead of a singleton. Additionally, we make ConfigAction
a global class and slightly change the behavior of --include= on the command
line by parsing all given includes after parsing all the other command line
arguments.
Make sure we always update the local repository metadata
The latest release of dnf5 introduced a change in behavior causing us
to not always sync the local repository metadata. To mitigate this, let's
always specify --refresh or similar for all package managers when we sync
the local repository to make sure its metadata is updated.
Let's leave this up to the individual subimage build scripts instead,
so that it's also possible for different subimages to share a build
directory if they want to do so.
Let's make BuildSources=, BuildSourcesEphemeral=, WithTests= and
WithNetwork= universal so that packages can easily be built in a
subimage instead of in the main image build.
We can't mark Environment= as a universal setting as only some environment
variables should be universal and others shouldn't be. So let's introduce
PassEnvironment= to mark specific environment variables as universal so that
they are passed to subimage builds.
Let's allow scripts to access the build directory so that outputs produced
by previous (sub)image builds can be accessed. Let's mount the build directory
read-only so that these scripts can't actually write to it.
Downgrade SELinuxRelabel=yes to SELinuxRelabel=auto for the initrd
Otherwise we fail if there's no policy installed in the initramfs.
Putting a policy in the initramfs isn't really a thing so let's not
fail if there is one available but still relabel if there is one
installed.
Let's track the modification timestamps of the packages from configured
package directories so we can rebuild the image if new packages appear
or are removed or changed.
For packages that should not be available when building the cached
image but should be available when installing volatile packages.
Additionally, we have build scripts write their packages to a separate
directory instead of the repository directory and later on pass these
packages as a separate volatile package directory to the default initrd
build.
This will allow us to introduce proper caching for package directories
in the next commit.
Only install systemd-ukify on Fedora on UEFI architectures
- systemd-ukify is only available on Fedora on UEFI architectures,
so let's not try to install it on other architectures.
- systemd-ukify is not in CentOS Stream 10 at the moment, so let's
not try to install it on CentOS just yet.
In preparation for the next commit, let's decouple the location of
the local repository from the location of the package directory where
scripts can look for packages.
Now that we have a Repositories= match, we can conditionally enable
EPEL for CentOS Stream 9 only and override the repositories when we
call mkosi dependencies. This means that the CentOS Stream 9 default
tools tree will have all the EPEL packages but we won't list them in
the output of mkosi dependencies.
We also add various missing packages to the CentOS Stream default tools
tree.
As explained in #2846, there are multiple issues with the current
implementation of mkosi.images. Let's take what we learned from the
default initrd and the default tools tree and apply it mkosi.images.
Specifically, all issues arise from the fact that we apply every option
from the global configuration (including CLI arguments) to the images
from mkosi.images/. To avoid the issues that arise from this (e.g --package
abc installing abc in all images), we made configuration values override
CLI arguments again so that we could override faulty CLI arguments again
in subimages so that they would only apply to the main image (e.g. set
Format= explicitly for each subimage so that --format on the command line
only applies to the main image).
Because we still wanted to allow configurable settings that can be modified
via the command line, we introduced the default specifier '@' which can be
prefixed to a setting to set a default value instead of overriding the value.
The '@' specifier is generally used in the global image independent
configuration to specify default values that can be overridden from the command
line. This specifier has led to a lot of confusion, along with the behavior that
the CLI does not override the configuration.
From the default tools tree and default initrd, we learned that what works
very well is to only have specific settings from the main image configuration
apply to the default tools tree and default initrd. For example, the distribution,
release, mirror and architecture should be the same for the main image and the
initrd, but the packages from the main image should not all be installed in the
initrd.
We can apply this idea to the images from mkosi.images/ as well, if we
introduce the assumption that all images defined in mkosi.images are
subimages intended to be included in some way or form in the main image.
This assumption allows us to divide all settings into either image specific
settings or "universal" settings that should apply to the main image and all
its subimages. The universal settings are passed on to each subimage. The image
specific settings are not.
This idea also allows us to define the "main" image outside of mkosi.images
again. Since only "universal" settings are passed on, we can safely define
an output format and such again in the global configuration, as we know this
won't be passed on to subimages.
It also allows us to make CLI arguments override configuration again. Since
there is no need anymore for subimages to override the CLI configuration as
inappropriate CLI configuration such as extra packages will only apply to the
main image and not any subimages from mkosi.images/. Because CLI configuration
overrides file configuration again, we also don't need the '@' specifier anymore,
as default values can simply be set without '@', since the CLI will override
the configuration file values by default.
We also lose the need for --append, because the sole use for --append was again
to override file based configuration.
Note that configuration from mkosi.local.conf is special in that it
should override settings from other configuration files, but not settings
that are specified on the CLI.
This commit implements all of what's mentioned above, specifically:
- CLI configuration now always trumps file based configuration.
- The '@' specifier is dropped automatically during parsing
- The main image is now always added from global configuration, even
if there are images in mkosi.images/. The main image is always built
last, and cannot be used as a dependency in the Dependencies= setting
for images defined in mkosi.images/.
- The Dependencies= setting for the main image now is used to specify
which subimages from mkosi.images/ to build. By default all subimages
are built.
- A universal tag is introduced for settings and appropriate settings
are marked as universal. Universal settings are passed on from the
main image configuration to subimage configuration.
- The Images= setting is removed, as it's role is replaced by
Dependencies=.
- The old name mkosi.presets and the Preset section name are removed
as they have been deprecated for a long time now.
- The config parsing tests are extended to cover more cases.
- All builtin configuration is adapted to stop using the '@' specifier.
- The documentation is updated in accordance with the changes.
Parse functions should return None to pick the default value. Also,
we don't get any values at all if unescape=True and the empty string
is passed so make sure we handle that case as well.
Instead of doing a complicated scheme with cache overlays which
aren't invalidated when the base tree changes, let's not do caching
when there are base trees. We assume that if there's a base tree
only a minimal amount of extra packages is installed that is sufficiently
fast without caching.
Use subdirectory of build directory for each subimage
Instead of sharing the build directory between all images, let's
use a subdirectory of the build directory for subimages.
This requires us to unshare the user namespace in run_build() before
we create the directories so that we always have permissions to create
any nested build directories.
We don't need to keep track of the current amount of includes since
those includes are already tracked in parsed_includes and will be
ignored. Slightly less efficient but this shouldn't matter here.
We also store the inode in parsed_includes before we parse the config
to make sure we don't try to parse it more than once.
Various fixes for grub packages in default tools trees
- Install grub2-tools on all architectures on CentOS/Fedora instead
of just x86-64.
- Only install grub-pc-bin on x86-64 on Debian/Ubuntu as it is not
available on other architectures
- Install grub-common instead of grub2 as grub-common contains all
the binaries we need and grub2 configures the system to use grub
as the bootloader which we don't want when these packages might be
installed on user's systems via mkosi dependencies.
- Do not install grub2 on OpenSUSE due to the same problem, it
configures the system to use grub as the bootloader which we don't
want. Unfortunately there is no subpackage with only the grub binaries
on OpenSUSE. Reported as https://bugzilla.opensuse.org/show_bug.cgi?id=1227464.
The tests are changed to not try to build or boot opensuse images
with bios.
Debian has a bespoke setup where if only grub-pc-bin is installed,
grub-bios-setup is installed in /usr/lib/i386-pc instead of in
/usr/bin. Let's take that into account and look for binaries in
/usr/lib/grub/i386-pc as well.
This requires another rework for the repositories() method for
OpenSUSE. Whereas before we picked up all the repositories from
https://download.opensuse.org/tumbleweed/repo/, it turns out that
the debug and source repositories can also be found at
https://download.opensuse.org/debug/tumbleweed/repo/ and
https://download.opensuse.org/source/tumbleweed/repo/ respectively.
Furthermore, the latter locations are the only ones that are available
for ports under https://download.opensuse.org/ports/ so we switch to
those instead.
Additionally, openSUSE-current is not available for ports so we disallow
using current, stable and release with architectures other than x86-64.
While there are more ports than just aarch64, for now let's just add
aarch64 and wait for user demand before we add any others.
run: Don't try to communicate inner pid for debug shell
The pipe write end has already been closed by then so bash will
fail with "bad file descriptor". Since there's no reason in having
the inner pid in the debug shell let's make sure we don't try to
communicate it there.
Always execute ukify with the configured python interpreter
We extend python_binary() with a binary argument so we can pass in
the binary we want to execute with python so python_binary() can
figure out whether we're going to use the python binary from the
tools tree or not.
Only use cache.direct if image size is aligned to page size
Trying to use cache.direct with an image whose size is not a multiple
of the page size fails so let's make sure we only use cache.direct if
that's the case and make sure the image size is a multiple of the page
size if we can.
Let's reduce duplication by using mkosi dependencies to install
all required dependencies to build the default image. To make this
work we add support for tools trees on jammy/bookworm. We also drop
support for a focal default image as it's too much of a burden to
maintain.
We also remove the EPEL only packages from the CentOS tools tree so
that mkosi dependencies works on a CentOS Stream system without EPEL
enabled.
Finally, curl-minimal is replaced by curl because trying to install
curl-minimal on a system with curl leads to a dnf failure so we opt
for the larger curl in order to make mkosi dependencies useful in
more situations.
Add %F to access the default filesystem of a distribution
One annoyance about using mkosi.repart has always been that to keep
using the default filesystem per distribution you have to write a
lot of matches. Now that systemd-repart supports
$SYSTEMD_REPART_OVERRIDE_FSTYPE_ROOT, let's add a specifier to access
the default filesystem so that it can be combined with the environment
variable to get the same result.
edk2 now has a virtio serial driver so let's switch to virtconsole
for the serial console as it's significantly faster compared to the
old ISA serial console.
All the latest releases of distributions now have a systemd stub
that knows how to read extra kernel command line arguments from
SMBIOS (It was backported to CentOS Stream 9), so let's drop the
default kernel command line and rely completely on passing the
console to use via SMBIOS.
This makes sure that users trying to deploy our images on bare metal
have a working console. Users that want to access the system via the
serial console will have to add the required console= argument
themselves.
Call rmtree() on context.root after finalize_staging()
After we've finalized the staging directory, there is no more need
for the root directory in the workspace. It used to get cleaned by
the setup_workspace() context manager but this is rather slow (can
take more than a second). By calling rmtree() explicitly, if we're
on a btrfs filesystem, we'll call btrfs subvolume delete which is
much faster than rm.
projects / thirdparty / mkosi.git / log
