- To split up our dependencies more, we need to make run.py
independent of MkosiState, so let's do that.
- To get rid of the shell hacks in both functions to chmod /tmp,
/var/tmp and /dev/shm, let's just mount the relevant files from
the host which have the right permissions.
- Fixing the above exposed a bug in the logic to set up rpm based
systems, which all ship a filesystem package that includes
directories such as /tmp, /proc, /sys, ... which we overmount
with apivfs or tmpfs filesystems when running rpm, causing errors
when the filesystem package tries to set up these directories. To
ensure these directories are created with the permissions from the
filesystem package, the run_with_apivfs() function is renamed to
bwrap() and gains an apivfs argument, which takes a path to set
up apivfs directories in. If not provided, no apivfs is set up.
This is then used to install the filesystem package without apivfs
so that the directories can be created with the right permissions.
- The various rpm distributions now install the filesystem package
instead of the setup package by default, so we can disable apivfs
properly while filesystem is being installed. system-user-root
was removed for opensuse because the filesystem package depends on
it.
On CentOS, we were creating the wrong symlink. Let's use a proper
relative symlink that always works regardless of whether we're chrooted
or not.
On Debian/Ubuntu, we weren't creating a symlink at all. So let's make
sure we also create a symlink there and use a relative symlink there as
well.
We use os.path.relpath() because Path().relative_to() can only do relative
paths to subpaths which isn't the case here. In the future we can set
strict=False and use relative_to().
Georges Discry [Thu, 20 Apr 2023 20:10:06 +0000 (22:10 +0200)]
Use an entry point instead of a script
The use of a custom script to launch mkosi has several shortcomings:
1. It translates `$PKEXEC_UID` into various `$SUDO_`* variables. That
translation is lost if mkosi is run as a zipapp or as a module with
`pkexec python3 -m mkosi`.
2. It has to muck around with `$PYTHONPATH` to launch mkosi directly
from its source code.
3. It is tied to setuptools and has no equivalent in the standard
project metadata (as in pyproject.toml).
4. And it also has to find the correct interpreter and maybe muck around
with `$PYTHONPATH` to launch mkosi from its installation, which may
be on the system, in the user's sitedir or in a virtualenv, as a
standard or an editable installation.
Point 1 is not needed anymore now that `mkosi.backend.current_user()` is
used everywhere.
Point 2 can be done by directly running mkosi as a module from its
sources with:
- `[sudo|pkexec] python3 -m mkosi` inside the project directory
- `[sudo] PYTHONPATH=<PROJECT_PATH> python3 -m mkosi` from anywhere
- `[sudo|pkexec] <PROJECT_PATH>/bin/mkosi` from anywhere. The script is
now a simple shim that is only used for this use case and not
installed anymore.
Point 3 and 4 are better supported by using a `"console_scripts"` entry
point when installed. However, it is not possible anymore to run a
user-installed mkosi with sudo/pkexec.
Georges Discry [Sun, 23 Apr 2023 12:51:08 +0000 (14:51 +0200)]
Disallow `<SOURCE>:<TARGET>` with an empty target
The configurations of the form `<SOURCE>[:<TARGET>]` require an absolute
target if given. However, `<SOURCE>:` would be ambiguously interpreted
as `<SOURCE>` so it is now disallowed.
For reference, `systemd-nspawn --bind` uses a similar form and refuses
to take an empty target.
Georges Discry [Sun, 23 Apr 2023 12:47:44 +0000 (14:47 +0200)]
Allow setting empty environment variables
Trying to set an empty environment variable with `FOO=` would instead
pass the `FOO` variable from the host, which is the expected behavior
when `=` is missing.
debian: Add TODO to drop /var/lib/dpkg/status logic when possible
It's created automatically by apt since apt 2.5.4 so add a note that
we can drop it once that version is widely available. Also disable
dpkg locking since it causes a apt failure due to not being able to
access the lock file in /var/lib/dpkg. Since we're building in a
chroot, we don't care about locking anyway.
Georges Discry [Fri, 21 Apr 2023 23:24:24 +0000 (01:24 +0200)]
Fix installation of essential packages on Debian stretch
There is a bug in Debian stretch where `libuuid1` (which is essential)
unnecessarily depends on `passwd`. When configuring the essential
packages and their dependencies, `passwd` is configured before
`base-passwd` and the installation fails.
debootstrap did not have this issue because it explicitly orders the
installation of some essential packages, including `base-passwd`.
By explicitly installing `base-passwd` first, the rest of the essential
packages can then be configured without failure.
Georges Discry [Fri, 21 Apr 2023 19:56:10 +0000 (21:56 +0200)]
Use shlex escaping in Environment
The Environment configuration takes lists of space-separated values and
combine them in a single list, splitting the values on spaces and
newlines. However, environment variables might sometimes need a space in
their value, which is currently impossible to configure.
Instead of blindly splitting the values on spaces and newlines, they are
now split with shlex. Spaces, newlines and backslashes can be escaped
with backslashes, single quotes and double quotes.
This behavior also more closely matches the Environment directive of systemd.
Instead of automatically making cpio's initrds, let's put this behind
a MakeInitrd= option. This allows booting directly into cpios with systemd
when they're not configured as an initramfs.
Georges Discry [Thu, 20 Apr 2023 20:09:58 +0000 (22:09 +0200)]
Apply path expansion when parsing
When the `extra_search_paths` configuration was introduced, it would
perform path expansions with the environment variables (using $) and the
home directories (starting with ~). Home directories would not be
expanded when mkosi was run with sudo or pkexec. Instead, `$SUDO_HOME`
could be used but only when run with sudo or pkexec.
However, those expansions were not applied to other paths in the
configuration and were broken with the rewrite of the configuration
loading, as the parser would check if the paths exist before expanding.
Those expansions are now handled directly in the parser and are
performed before validation. Environment variables expansion is enabled
for all paths and home directory expansion is enabled for all paths on
the host.
Furthermore, home directory expansion is always available, with `~`
expanding to the user's home directory when sudo or pkexec are used (and
not `/root`), replacing the need for `$SUDO_HOME`.
Georges Discry [Thu, 20 Apr 2023 20:09:54 +0000 (22:09 +0200)]
Unify loading the current user info
There are a few places where mkosi wants to know who is the current user
invoking the script with sudo or pkexec, instead of root. Some
combinations of `$SUDO_UID`, `$PKEXEC_UID` and `$SUDO_USER` are used,
but without consistency. Furthermore, when pkexec is used, several
places depend on `bin/mkosi` setting the correct `$SUDO_`* environment
variables.
Those places now use the `mkosi.backend.current_user()` function.
Furthermore, that function returns an `InvokingUser` wrapping the
complete passwd entry from the `pwd` module instead of just the uid/gid.
Split up --base-image into --base-tree and --overlay
--base-tree indicates a base image. It differs from --skeleton-tree
in that use of --base-tree indicates that we've already installed a
distribution and should only install extra packages, not install the
distribution from scratch.
If --overlay is not specified, we just copy all the specified base trees
to the root directory before doing anything else (even before copying the
skeleton trees) and after that we operate as usual.
If --overlay is specified, instead of copying all the base trees to
the root directory, we set up an overlayfs mount with all the base
trees as lowerdirs. After that we operate as usual. The effect is that
all our usual steps will operate on a full view of the image, but the
output will only contain the additions we made on top of the specified
base trees.
Empty install directory before running build script
The install directory is shared between distros, and depending on
the distro, different files might be installed. Let's make sure we
empty the install directory before running the build so that files
from different distros don't end up mixed in the install directory.
Fix stdin being redirect to /dev/null when running ssh, shell or qemu
30b8996a22022289bbf9820b6a04c5bfaf771d5e changed the default of
stdin to /dev/null, which is great, but for the ssh, shell, boot and
qemu commands, we actually need it connected to stdin of the calling
terminal, so let's fix that.
Georges Discry [Thu, 20 Apr 2023 20:09:34 +0000 (22:09 +0200)]
Check if already root in become_root
Instead of requiring the check for `os.getuid() != 0` before calling
`mkosi.run.become_root()`, that check is now performed at the beginning
with an early return if the user is already root.
Georges Discry [Thu, 20 Apr 2023 20:31:45 +0000 (22:31 +0200)]
Override sys.excepthook when calling main
Overriding `sys.excepthook` with `mkosi.run.excepthook` has to happen
when calling `mkosi.__main__.main()`, otherwise it is not applied when
`main` is simply imported and then called (by using a zipapp or a
console script entry point).
The -smbios args are very long. I was checking out secureboot config,
and I thought that the second -drive argument is missing. Let's make
them adjacent again.
Reformat run commands to have "--option", "value" pairs on the same line
We were already doing this for the majority of invocations, but there
were some exceptions. Since this makes it much easier to mentally
split the command into logical parts, let's do this everywhere.
To make the command stand out a bit, add one space of seperation between
"[" and the first argument and between the last argument and "]".
Set stdin=/dev/null for all commands invoked by run()
In ed93e8c49fc2afb9255280d1dec9b7d81d07ff94 I added this for two
commands that were invoked directly from summary. But the same issue
occurs for other commands that we invoke. Instead of handling them
one-by-one, let's set stdin for all commands. We don't want anything that
we invoke to ever read input from the console, so this is a suitable
default for us.
Do not print hint about exception for RuntimeError
If debug is already on, the hint is not useful. Also, if we already printed an
error, we don't need to tell the user that we internally used an exception,
this is not relevant to the user.
Also, change str(e) to e.__class__.__name__. 'str(e)' is the same as 'e' here,
and is not guaranteed to contain any message. E.g. for 'raise ValueError',
str(e) is just ''. Let's say "internal exception CLASS" instead.
I still don't think RuntimeError is a great choice. It'd be totally unsuitable
if mkosi was used as a library. We don't do this right now, so it's not so bad,
but we could still get confused by RuntimeError generated in some sloppy code
in one of the packages that we call. This is hopefully unlikely, but our own
custom exception would be clearer and free of this risk. Alas, RuntimeError was
added purposefully in e25d746f9d7668edc5134cde4ec381c8b2da0136, so I'm not
changing it.
Reformat hint and don't print as part of the error
The error about SecureBoot keys was line-broken in the source code and looked
strange on the tty: the line break was in the middle of the terminal. Let's
rework things a bit to print the hint in gray and print the two filenames at
the end so it is easy to select-and-paste into an actual command. Also
use hint= in other cases where part of the error is advisory.
Let's add back some form of direct linux boot by allowing to boot
cpio images. To make this work, either the user has to pass -kernel
or a kernel has to be installed inside the cpio, which we'll copy
out and use as the -kernel argument.
Let's introduce an enum and a helper to refer to the compressed output
path.
We also go back to always reading the password in find_password() if
set in a mkosi.rootpw file. This was changed to only be done during
a build so that the file could be owned by root, but this doesn't apply
anymore because mkosi doesn't need to be root anymore to do a build.
The current compression level takes a really long time to complete
when the output is large. Let's use the default compression level
instead so compression is a bit faster while still giving good results.
The build overlay is not modified during the final build, it's only
used to run the build script which cannot touch the build image. Let's
enforce this even more by mounting the build overlay read-only when
running the script.
With this assurance, we can stop copying the cached build overlay every
time and just use it directly since we're certain it cannot be modified
anyway.
Let's just add an argument log to run that's true by default which
allows us to disable logging if the command failed. We also disable
logging in run_ssh(), run_qemu() and run_shell() since we expect
those to fail and shouldn't log if they do (like we did before).
While this achieved the desired effect of dpkg being looked up in
the PATH, it also modified the PATH variable for everything executed
by dpkg, which we don't want. Let's just lookup dpkg in PATH before
running apt and put the result in Dir::Bin::dpkg which looks up
dpkg in the PATH without modifying the PATH used by dpkg itself.
Let's always use regular exceptions and stop outputting stack traces
on exceptions by default. To get a stacktrace, users can run mkosi with
--debug run and we will output the usual stacktrace.
This also makes all outut that was conditional on some other settings
unconditional, since it was a bit mixed what we always showed and for which we
defaulted to some fallback string anyway.
with_docs was conditioned on which distributions supported it, but distribution
support for other features has been mixed as well and accompanying settings have
been shown regardless.
There are valid scenarios where one might want to install a kernel
without wanting a bootable image, so let's add back the --bootable=
option as a feature option. It defaults to "auto" which is the current
behavior, but can also be explicitly enabled or disabled. If enabled,
we'll fail if we can't produce a bootable image. If disabled, we won't
generate a bootable image even if all the necessary components are
available.
With the new Fedora website the location of the gpg keys has changed.
Also, all the gpg keys are now in one file instead of having a separate
file for each release.
Make sure we resolve relative paths in config files correctly
Relative paths in config files should always be interpreted relative
to their config file. We could pass the directory to each parse method,
but that's rather verbose, so let's use chdir() so that relative paths
are resolved correctly instead.
Make sure the /efi mountpoint exists for every distro
https://0pointer.net/blog/linux-boot-partitions.html recommends
using the /efi mountpoint universally so let's accomodate that
and do it for every distro.
Do not fail when writing root password and /etc/shadow is missing
I had this failure when testing https://github.com/systemd/mkosi/pull/1442. But
it would happen whenever the installed system has no /etc/shadow and we want to
set the password, so it's worth fixing regardless.
Let's get rid of our dependency on debootstrap by replicating its
core functionality ourselves. To make sure the necessary tools for
maintainer scripts to run are available in the chroot, we have to
extract the essential debs manually first before installing all the
essential debs.
This also allows us to get rid of our skeleton tree hack we added for
debootstrap.
Only write repositories to image on Debian/Ubuntu if apt was installed
For other distributions, we don't install the repositories at all but
because Debian and Ubuntu do not have an easy way to install the
repositories via a package we give them some leeway and install the
repositories if apt was installed.
We also rework repository handling to accomodate this. When running
apt ourselves on the host, we now use the apt directory in the
workspace as the config directory for apt instead of using etc/apt
in the image.
Additionally, we also use the keyring from the host instead of the
one from the image when running apt.
Finally, we switch to the http mirror for debian security updates
to avoid having to install ca-certificates in the image for apt
to work properly.
projects / thirdparty / mkosi.git / log
