Adrian Freihofer [Sun, 13 Jan 2019 15:35:31 +0000 (16:35 +0100)]
Do not override run_build_script function
Does not change anything. It's a renaming of a function parameter
from run_build_script to do_run_build_script to resolve the name
clashing with function run_build_script.
Signed-off-by: Adrian Freihofer <adrian.freihofer@gmail.com>
Lucas De Marchi [Tue, 19 Feb 2019 21:54:56 +0000 (13:54 -0800)]
arch: kill dirmngr only
Using fuser to identify processes using a path doesn't work as I wanted:
it relies on the filesystem mounted there. The outcome is that when
using "-t directory" it will try to kill all processes using that
filesystem rather than processes using files under that path.
Even by bind-mounting root so it's always a mountpoint, it doesn't work.
So instead of killing everything, pinpoint what process pacman/pacstrap
left running and kill only that one.
Lucas De Marchi [Tue, 19 Feb 2019 21:53:06 +0000 (13:53 -0800)]
arch: trust new signatures while downloading base packages
Otherwise it aborts with:
error: libxml2: signature from "Levente Polyak (anthraxx) <levente@leventepolyak.net>" is unknown trust
:: File /var/tmp/mkosi-__fdh4oy/root/var/cache/pacman/pkg/libxml2-2.9.9-1-x86_64.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n]
docs: Replace README.md by a pointer to the man page
The man page contains most relevant stuff now, and is a lot more
comprehensive, hence let's drop these explanations from README.md and
point people directly to the man page if they want to know more.
This adds a pretty comprehensive man page as a MarkDown document.
This does not contain any build system hookup, simply because I couldn't
figure out how to teach Python's setuptools to invoke external programs
as part of the build process.
Currently a manual step is necessary, to convert the man page into an
actual man page:
# pandoc mkosi.md -s -t man > mkosi.1
The man page is based on the old README.md but substantially extended.
Let's tell the build script whether it is running with networking or
not. I don#t have an immediate usecase for this, but it's more
systematic given how --with-docs and --without-tests is currently
handled and result in environment variables, bus --with-network is not.
A usecase might be to generate the `-nonet` option of `xsltproc` from
this option, but that's purely theoretical.
mkosi: always mention XFS before btrfs, as it is more like ext4
ext4 and xfs images are pretty much put together the same way, except
for the file system used. btrfs otoh is different as instead of a GPT
partition table any subdivisions are are done as btrfs subvolumes.
Hence, let's reorder the enums a bit, and always handle ext4/xfs
together, and btrfs after that.
mkosi: add "-a" switch for building everything in mkosi.files/
Let's add a one-stop solution for building a set of files all in one go.
For that, simply place "mkosi.default"-style files in mkosi.files/ and
use "mkosi -a" to build them all.
mkosi: when a .nspawn file is used, actually use it when running nspawn
Previously, we'd move it in place, but when invoking nspawn on "mkosi
shell" or "mkosi boot" it would be ignored, since that's what nspawn
does unless the file is in /etc or /run.
Let's change this and mark it trusted, after all it's coming from the
same source that just built the image.
This is very useful for cases where you want to default to
"--private-network" when "mkosi boot" is invoked.
Michael Weghorn [Wed, 5 Dec 2018 14:23:41 +0000 (15:23 +0100)]
Update help for '--with-docs'
The restriction "only Fedora, CentOS and Mageia"
does not (no longer?) apply, since the option is also handled for
other distributions (Debian, Ubuntu, openSUSE).
Luke Shumaker [Sun, 2 Dec 2018 19:01:00 +0000 (14:01 -0500)]
mypy: ListAction: assert that 'values' is a string
We must declare 'values' as Union[str, Sequence[Any], None], to keep the
__call__ definition compatible with the parent class. However, we know
(and rely on) that how we call .add_argument() means that it will always
be a str. Add an 'assert' to let mypy know what we know.
Luke Shumaker [Sun, 2 Dec 2018 18:38:46 +0000 (13:38 -0500)]
mypy: Make arguments Optional[], assert is not None as appropriate
There are several places where we unconditionally call a function, then
that function checks `args` to decide if it immediately returns, or if it
actually gets to do work. In the cases where it immediately returns, some
of the arguments can be None, so they should be hinted as Optional[].
However, in the case were it does get to do work, they cannot be None.
So, hint them as Optional[], and the add `assert ARGNAME is not None`
after the check that implies it's not None, to let mypy know what we know.
For instance: args.output_format.is_disk() implies that raw is not None.
So it's safe to make the change:
if not args.output_format.is_disk():
return None
+ assert raw is not None
Luke Shumaker [Sun, 2 Dec 2018 18:38:03 +0000 (13:38 -0500)]
mypy: Fix wrong type annotations
Common cases:
- An `if foo is None` check clearly indicates that foo should be
Optional[]:
* prepare_root(): dev
* prepare_home(): dev
* prepare_srv(): dev
* mount_image(): loopdev, home_dev, srv_dev
* save_cache(): raw, cache_path
* link_output_nspawn_settings(): path
* link_output_checksum(): checksum
* link_output_root_hash(): root_hash_file
* link_output_signature(): signature
* link_output_bmap(): bmap
* process_settings(): key
* run_build_script(): raw
- In general, replace IO[str] with TextIO, except that in many cases
BinaryIO is correct:
* create_image() returns binary, not text
* reuse_cache_image() returns binary, not text
* attach_image_loopback() takes binary, not text
* make_verity() returns binary, not text
* calculate_sha256sum():
. argument: raw is binary, not text
. argument: tar is binary, not text
* make_build_dir():
. return: (raw|squashfs) is binary, not text
. return: tar is binary, not text
* run_build_script(): raw is binary, not text
* remove_artifacts():
. argument: raw is binary, not text
. argument: tar is binary, not text
- In general, replace IO[bytes] with BinaryIO, except that in many cases
TextIO is correct:
* calculate_sha256sum() returns text, not binary
* calculate_bmap() returns text, not binary
- Avoid ambiguous IO[Any]:
* hash_file() writes to a TextIO
* calculate_bmap() reads a BinaryIO
Special cases:
- run(): Either returns a CompletedProcess, or doesn't return. Drop
the Optional[].
- mount_image(): ???
- calculate_sha256sum():
. argument: root_hash_file is a BinaryIO, not a str
. argument: nspawn_settings is a BinaryIO, not a str
Wait, BinaryIO, not TextIO!? Yes, even though the .nspawn file is
plaintext. The IO object we have here is a NamedTemporaryFile copy
of the user's .nspawn file. We made that copy in binary mode, to
avoid pointlessly decoding then encoding the text.
- remove_glob(): When providing a type hint for '*args', you provide the
hint for the individual 'arg's, not the 'args' list.
Luke Shumaker [Sun, 2 Dec 2018 16:19:37 +0000 (11:19 -0500)]
mypy: Figure out generators and decorators
- A @contextlib.contextmanager should return a
typing.Generator[yield, send, return]
- Because of <https://github.com/python/mypy/issues/1317>, mypy 0.641
doesn't think complete_step() has the correct type to be a decorator
that works the way it does. So, set
completestep = complete_step
But do typecasting to make mypy think it's the correct type. And use
`@completestep` instead of `@complete_step`. That's an easy
search/replace.
Luke Shumaker [Sun, 2 Dec 2018 06:41:38 +0000 (01:41 -0500)]
mypy: Introduce manual cast()s around tempfile.NamedTemporaryFile
Unfortunately, mypy 0.641 isn't yet smart enough to figure out the
appropriate subtype of IO to return from tempfile.NamedTemporaryFile()
based on whether the mode= argument contains "b", so we'll have to cast()
its return value for now.
If this makes a line wider than 119 characters, move the dir= argument to
the next line.
Luke Shumaker [Wed, 7 Nov 2018 15:14:38 +0000 (10:14 -0500)]
mypy: Convert type-hint comments to PEP 526 type-hints
Commit a415718e27b673032a415d35af41f9d3aa1b9a99 bumped the required
Python version to 3.6; which means that we can now use PEP 526
type-hints, instead of doing it in magical `# type:` comments.
Luke Shumaker [Sun, 2 Dec 2018 05:37:03 +0000 (00:37 -0500)]
flake8: E741: Rename ambiguous variable
PEP 8 says to avoid `l` as a variable name, because it can be hard to
distinguish from `1` and `I`. This means that flake8 complains about
the variable inside of ListAction.
However, flake8 doesn't trigger E741 for *argument* names. Even
though flake8 doesn't complain about it, also rename the argument of
line_join_list() from `l` to `ary`.
Luke Shumaker [Sun, 2 Dec 2018 05:35:32 +0000 (00:35 -0500)]
flake8: E201,E202,E203,E221,E222,E251: Remove individual spaces around things
- E201: Don't put a space after the `[` in a list:
ary = [ item,
^
- E201: Don't put a space after the `{` in a dict:
d = { key: val,
^
- E202: Don't put a space before the `]` in a list:
ary = [item1, item2 ]
^
- E202: Don't put a space before the `}` in a dict:
d = [key: val }
^
- E203: Don't put a space before the `:` in a dict:
d = {key : val}
^
- E221: Write `# NOQA: E221` if we're lining up the `=` in a list of
assignments
- E222: Write `# NOQA: E222` if we're lining numbers on the right
side of `=` in a list of assignments
- E251: When defining an argument a default value, and the argument
isn't type-hinted, don't put spaces around the `=`:
def fn(argname = default) -> type:
^ ^
This is in contrast with type-hinted arguments with defaults, which
DO get spaces around the `=` (E252).
- E251: Don't put spaces around the `=` when passing a keyword
argument to a function:
fn(kw = val)
^ ^
Luke Shumaker [Sun, 2 Dec 2018 05:04:12 +0000 (00:04 -0500)]
flake8: E225,E227,E231,E252,E261: Add individual spaces around things
- E225: Put a space on each side of the `+` operator
- E227: Put a space on each side of the `|` operator
- E227: Put a space on each side of the `<<` operator
- E231: Put a space after `,` in a list/set/tuple.
- E231: Put a space after the `:` when defining a lambda
- E231: Put a space after the `:` in dict literals
- E231: When type-hinting an argument, put a space between the `:`
and the type:
def fn(argname: type) -> type:
^
- E252: When type-hinting an argument with a default value, put
spaces around the `=`:
def fn(argname: type = default) -> type:
^ ^
This is in contrast with non-type-hinted arguments with defaults,
which do NOT get spaces around the `=` (E251).
- E261: Put two spaces before the `#` of an inline comment
Luke Shumaker [Sun, 2 Dec 2018 04:27:16 +0000 (23:27 -0500)]
flake8: E501: Address lines that are 120 characters or longer
- Set the maximum line length to 119, same as systemd.git
- If the line is a function definition that is long because of many args,
put one arg per line
- Don't split user-facing strings over multiple lines; use NOQA to silence
those complaints
- In a few cases, re-structure the code to be easier to follow
This doesn't blindly modify the code to flake8's complaints, it looks at
long lines, and sees if we can make them easier to read.
Lucas De Marchi [Thu, 29 Nov 2018 22:45:12 +0000 (14:45 -0800)]
Fix package manager cleanup
As the cache is still mounted when we try to cleanup the metadata, it
was removing the downloaded packages from mkosi.cache (or whatever path
the user specified).
Lucas De Marchi [Wed, 21 Nov 2018 09:16:13 +0000 (01:16 -0800)]
Allow to mount build sources
This allows us to just mount the build sources into the build image in
order to run the script. By doing this the user may avoid problems with
recursive submodules and at the same time speed up execution for large
trees. This means the user will need to take care of possibly cleaning
up the tree if he wants to make sure he has a clean build.
Lucas De Marchi [Tue, 20 Nov 2018 16:01:25 +0000 (08:01 -0800)]
Fold --use-git-files and --git-files into one option
Besides being nicer to have a single option, it will allow us to easily
extend with new copy methods. While at it, also add an entry in the
config file.
Lucas De Marchi [Sat, 17 Nov 2018 00:24:18 +0000 (16:24 -0800)]
Add bios support to Archlinux
In order to add support to boot on bios mode for Archlinux, the
following changes were made:
- kernel packages are now installed as a second step: this allows us
to patch the initrd config file so to disable the autodetection
hook
- patching the initrd config is now part of the distro install
rather than bootloader install
uefi-only is the default and we can generate uefi-only, bios-only and
hybrid images. In the latter case grub is used for bios and systemd-boot
for UEFI.
This implements boot on systems with legacy bios rather
than UEFI. It works by installing grub-pc and systemd-boot
side by side. The former is responsible for bios boot while
the latter continues to be the bootloader for UEFI.
The resulting image, that is currently working only on Fedora,
is capable of booting on both systems. Generate the image with: