Something went wrong on my machine and /dev/loop8p2 was not present,
even though loop8 and loop8p1 were. (I think the loopback device was
mounted somewhere and the kernel wouldn't reread the partition table.)
Since we are running as root, we can easily create a new file in /dev.
Let's avoid this.
Lénaïc Huard [Mon, 18 Mar 2019 22:02:16 +0000 (23:02 +0100)]
Disable COW (for btrfs) on qcow2 files
According to qemu-img(1):
Btrfs has low performance when hosting a VM image file,
even more when the guest on the VM also using btrfs as file
system.
Turning off COW is a way to mitigate this bad performance.
This is the reason of the disable_cow() function, but this function
applies only to the raw file. So, let’s disable COW also on the qcow2
one.
Adrian Freihofer [Sun, 24 Feb 2019 15:49:24 +0000 (16:49 +0100)]
parse_args: major refactoring and improvements
The man page describes the powerful configuration approach
based on a mkosi.default file and several file in a
mkosi.default.d folder as well as command line arguments.
This does not work for many parameters. The problem is that
argparse is called before config files are loaded.
For some values the implementation basically looks like:
a = False # argparse lines
...
if a is None: # config file loading, lower priority
a = True
This new implementation delegates loading of mkosi.default files
to python's argparse.ArgumentParser. According to docummentation
ArgumentParser supports loading of settings from files by
overloading the convert_arg_line_to_args function. The function
gets one line from the config file and returns the corresponding
command line counterpart. For example WithTest=yes found in a
defaults file would be converted to ['--with-tests', 'yes'].
Unfortunately this line by line conversion is not really compatible
with python's configparser module which works file wise. To stay
with ArgumentParser and ConfigParser the new ArgumentParserMkosi
overloads the undocumented _read_args_from_files function of the
ArgumentParser. This function is called for each @ prefixed
command line argument.
The new implementation supports unlimited number of mkosi.default
files. The files are loaded according their alphabetical priority.
Command line arguments are processed with highest priority.
The second problem solved by this patch is data type conversion
for settings loaded from mkosi.default files. Since ArgumentParser
handles all settings, data type conversion is always done by
ArgumentParser.
Signed-off-by: Adrian Freihofer <adrian.freihofer@gmail.com>
Adrian Freihofer [Sun, 13 Jan 2019 11:55:21 +0000 (12:55 +0100)]
parse_args: Improve handling of boolean arguments
First step to handle boolean configuration parameters from
mkosi.default files as well as from command line arguments. The
implementation supports multiple definitions of one argument. The
last value gets priority. Examples:
--foo=yes --foo=no --> args.foo=False
--foo --> args.foo=True
--without-foo --> args.foo=False
--foo --without-foo --> args.foo=False
Purpose:
Let argparse import mkosi.default files in a future commit.
Make parse_boolean function case-insensitive.
Signed-off-by: Adrian Freihofer <adrian.freihofer@gmail.com>
Adrian Freihofer [Thu, 24 Jan 2019 23:26:16 +0000 (00:26 +0100)]
parse_args: Support removing list entries with ! prefix
Entries added handled by the ListAction class may be prefixed with
! to remove an entry if it is already in the list.
!* removes all entries from the list.
Examples:
- mkosi -p httpd -p !httpd
has no effect
- mkosi -p vi -p emacs -p nano -p "!*,httpd"
Only the httpd package will be added to the image.
Note: This function becomes even more useful if the configuration
files are also processed by argparse. This will be the case
with a subsequent commit.
Signed-off-by: Adrian Freihofer <adrian.freihofer@gmail.com>
The package was renamed some time ago (around March 2016)[1]. The
package btrfs-progs is available from Stretch[2] but it's also in Jessie
backports[3]; in Ubuntu it was introduced in Bionic[4], so it's probably
more appropriate to use the new name.
Adrian Freihofer [Sun, 13 Jan 2019 15:35:31 +0000 (16:35 +0100)]
Do not override run_build_script function
Does not change anything. It's a renaming of a function parameter
from run_build_script to do_run_build_script to resolve the name
clashing with function run_build_script.
Signed-off-by: Adrian Freihofer <adrian.freihofer@gmail.com>
Lucas De Marchi [Tue, 19 Feb 2019 21:54:56 +0000 (13:54 -0800)]
arch: kill dirmngr only
Using fuser to identify processes using a path doesn't work as I wanted:
it relies on the filesystem mounted there. The outcome is that when
using "-t directory" it will try to kill all processes using that
filesystem rather than processes using files under that path.
Even by bind-mounting root so it's always a mountpoint, it doesn't work.
So instead of killing everything, pinpoint what process pacman/pacstrap
left running and kill only that one.
Lucas De Marchi [Tue, 19 Feb 2019 21:53:06 +0000 (13:53 -0800)]
arch: trust new signatures while downloading base packages
Otherwise it aborts with:
error: libxml2: signature from "Levente Polyak (anthraxx) <levente@leventepolyak.net>" is unknown trust
:: File /var/tmp/mkosi-__fdh4oy/root/var/cache/pacman/pkg/libxml2-2.9.9-1-x86_64.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n]
docs: Replace README.md by a pointer to the man page
The man page contains most relevant stuff now, and is a lot more
comprehensive, hence let's drop these explanations from README.md and
point people directly to the man page if they want to know more.
This adds a pretty comprehensive man page as a MarkDown document.
This does not contain any build system hookup, simply because I couldn't
figure out how to teach Python's setuptools to invoke external programs
as part of the build process.
Currently a manual step is necessary, to convert the man page into an
actual man page:
# pandoc mkosi.md -s -t man > mkosi.1
The man page is based on the old README.md but substantially extended.
Let's tell the build script whether it is running with networking or
not. I don#t have an immediate usecase for this, but it's more
systematic given how --with-docs and --without-tests is currently
handled and result in environment variables, bus --with-network is not.
A usecase might be to generate the `-nonet` option of `xsltproc` from
this option, but that's purely theoretical.
mkosi: always mention XFS before btrfs, as it is more like ext4
ext4 and xfs images are pretty much put together the same way, except
for the file system used. btrfs otoh is different as instead of a GPT
partition table any subdivisions are are done as btrfs subvolumes.
Hence, let's reorder the enums a bit, and always handle ext4/xfs
together, and btrfs after that.
mkosi: add "-a" switch for building everything in mkosi.files/
Let's add a one-stop solution for building a set of files all in one go.
For that, simply place "mkosi.default"-style files in mkosi.files/ and
use "mkosi -a" to build them all.
mkosi: when a .nspawn file is used, actually use it when running nspawn
Previously, we'd move it in place, but when invoking nspawn on "mkosi
shell" or "mkosi boot" it would be ignored, since that's what nspawn
does unless the file is in /etc or /run.
Let's change this and mark it trusted, after all it's coming from the
same source that just built the image.
This is very useful for cases where you want to default to
"--private-network" when "mkosi boot" is invoked.
Michael Weghorn [Wed, 5 Dec 2018 14:23:41 +0000 (15:23 +0100)]
Update help for '--with-docs'
The restriction "only Fedora, CentOS and Mageia"
does not (no longer?) apply, since the option is also handled for
other distributions (Debian, Ubuntu, openSUSE).
Luke Shumaker [Sun, 2 Dec 2018 19:01:00 +0000 (14:01 -0500)]
mypy: ListAction: assert that 'values' is a string
We must declare 'values' as Union[str, Sequence[Any], None], to keep the
__call__ definition compatible with the parent class. However, we know
(and rely on) that how we call .add_argument() means that it will always
be a str. Add an 'assert' to let mypy know what we know.
Luke Shumaker [Sun, 2 Dec 2018 18:38:46 +0000 (13:38 -0500)]
mypy: Make arguments Optional[], assert is not None as appropriate
There are several places where we unconditionally call a function, then
that function checks `args` to decide if it immediately returns, or if it
actually gets to do work. In the cases where it immediately returns, some
of the arguments can be None, so they should be hinted as Optional[].
However, in the case were it does get to do work, they cannot be None.
So, hint them as Optional[], and the add `assert ARGNAME is not None`
after the check that implies it's not None, to let mypy know what we know.
For instance: args.output_format.is_disk() implies that raw is not None.
So it's safe to make the change:
if not args.output_format.is_disk():
return None
+ assert raw is not None
Luke Shumaker [Sun, 2 Dec 2018 18:38:03 +0000 (13:38 -0500)]
mypy: Fix wrong type annotations
Common cases:
- An `if foo is None` check clearly indicates that foo should be
Optional[]:
* prepare_root(): dev
* prepare_home(): dev
* prepare_srv(): dev
* mount_image(): loopdev, home_dev, srv_dev
* save_cache(): raw, cache_path
* link_output_nspawn_settings(): path
* link_output_checksum(): checksum
* link_output_root_hash(): root_hash_file
* link_output_signature(): signature
* link_output_bmap(): bmap
* process_settings(): key
* run_build_script(): raw
- In general, replace IO[str] with TextIO, except that in many cases
BinaryIO is correct:
* create_image() returns binary, not text
* reuse_cache_image() returns binary, not text
* attach_image_loopback() takes binary, not text
* make_verity() returns binary, not text
* calculate_sha256sum():
. argument: raw is binary, not text
. argument: tar is binary, not text
* make_build_dir():
. return: (raw|squashfs) is binary, not text
. return: tar is binary, not text
* run_build_script(): raw is binary, not text
* remove_artifacts():
. argument: raw is binary, not text
. argument: tar is binary, not text
- In general, replace IO[bytes] with BinaryIO, except that in many cases
TextIO is correct:
* calculate_sha256sum() returns text, not binary
* calculate_bmap() returns text, not binary
- Avoid ambiguous IO[Any]:
* hash_file() writes to a TextIO
* calculate_bmap() reads a BinaryIO
Special cases:
- run(): Either returns a CompletedProcess, or doesn't return. Drop
the Optional[].
- mount_image(): ???
- calculate_sha256sum():
. argument: root_hash_file is a BinaryIO, not a str
. argument: nspawn_settings is a BinaryIO, not a str
Wait, BinaryIO, not TextIO!? Yes, even though the .nspawn file is
plaintext. The IO object we have here is a NamedTemporaryFile copy
of the user's .nspawn file. We made that copy in binary mode, to
avoid pointlessly decoding then encoding the text.
- remove_glob(): When providing a type hint for '*args', you provide the
hint for the individual 'arg's, not the 'args' list.
Luke Shumaker [Sun, 2 Dec 2018 16:19:37 +0000 (11:19 -0500)]
mypy: Figure out generators and decorators
- A @contextlib.contextmanager should return a
typing.Generator[yield, send, return]
- Because of <https://github.com/python/mypy/issues/1317>, mypy 0.641
doesn't think complete_step() has the correct type to be a decorator
that works the way it does. So, set
completestep = complete_step
But do typecasting to make mypy think it's the correct type. And use
`@completestep` instead of `@complete_step`. That's an easy
search/replace.