Felipe Sateler [Fri, 21 Jul 2017 08:33:57 +0000 (04:33 -0400)]
Ensure machine name is unique (#127)
Even though we don't register the machine, systemd-nspawn creates a
scope based on the name. Therefore, make sure the name is unique to
prevent collisions.
See https://github.com/systemd/systemd/issues/6347
Lucas De Marchi [Tue, 18 Jul 2017 20:25:22 +0000 (13:25 -0700)]
rework copy_git_files() to use a git clone
Using a git clone allows to solve some problems:
- It allows to copy git submodules without having to fallback to
--use-git-files=no since git-ls-files doesn't recurse submodules
- It allows build systems that rely on git describe or similar to
tag the build: this is more important when projects use
git modules and they want to tag the git revision of each
submodule.
- It uses less disk space when using output as directory since
when passing a local directory git will use hard links
Some additional handling for files that were modified (or added with
git -A) was put in place to allow tests to be performed before
committing.
In addition the call to git ls-files was fixed in the following cases:
- When mkosi is called with -C option: it was calling ls-files inside
the caller directory, not inside the src dir.
- When mkosi is called with --git-files=others and .gitignore
doesn't contain an entry for '.mkosi-*': it would try to copy the
temporary directory we created and fail
mkosi: support reusing builddirs between subsequent mkosi runs (#114)
With this change, mkosi will look for "mkosi.builddir" when invoked.
When this directory exists it is mounted into the build container and
may be used as location for out-of-tree builds, which can be shared
between multiple invocations to speed up the building process.
This is particularly useful when used in conjunction with incremental
building (-i), as with this in place building OS images using mkosi is
only a little slower than just building the project natively on the host
now.
"mkosi clean -f" will empty the builddir if it exists now too, in
addition to the incremental image trees.
Lucas De Marchi [Sat, 15 Jul 2017 11:51:23 +0000 (04:51 -0700)]
Ignore temporary directories while copying (#115)
If we call mkosi passing --build-sources with a directory that contains
the temporary .mkosi-* directories, it will fall in a inifinite
recursion.
This is the case in which we are have the mkosi.* files in a
subdirectory of the git repository and pass the git root dir as argument
to --build-sources.
Optionally ignore all untracked files in git ls-files (#94)
It is very easy for a large file to end up in the built image. Either
the package cache, or some other already built image, or the new
*.cache-* files are large enough to overflow the available space in
the destination image.
Some people (me ;)) find it convenient to just use the list of files
that git knows about ('git ls-files --cached') and ignore anything
else (git can be taught about the existence of new not-yet-staged
files with 'git add -N'). Then build products or images or cache files
can be safely present in the build directory, and there's no need to
craft --exclude patterns for them.
--git-files=others is the default and preserves previous behaviour.
--git-files=cached uses all committed or staged files as found in
the working directory.
It would be an interesting option to use an actual committed tree
instead of files in the working directory, but it would require a
significant changes, because files would have to be extracted directly
from the git cache. It's an interesting future direction.
I made 'git-files' a new option instead of adding a new value to
'use-git-files' because it seems better to keep the detection of
whether git ls-files should be used orthogonal to what files are
included.
Also drop the .mkosi-* excludes, because those particular files are
most likely mkosi configuration, and there's no need to exclude them.
The package cache is called mkosi.cache by default, and mkosi build
products and image caches don't match this pattern either.
We really shouldn't follow symlinks when we copy stuff. Otherwise we
can't build casync, because it contains a number of dangling symlinks in
its test-files/ directory.
Incremental builds permit caching the build and final images right after
OS package installation — before the source or build tree is copied in.
This is useful to drastically reduce the runtime of mkosi image building.
If you have a source tree, you may now type in:
# mkosi -if -t raw_gpt -o output.raw
This will build an image possibly making use of a pre-existing
"output.raw.cache-pre-dev" and "output.raw.cache-pre-inst" image files
to speed up the build. If the files exist, they are used under the
assumption they contain a pre-generated version of the disk images in
the state immediately before copying in the source tree or build tree.
(-i is short for --incremental). If the files don't exist, they are
generated, and thus available for speeding up subsequent runs. The
".cache-pre-dev" file contains the image for the development build of
the image, the ".cache-pre-inst" file contains the image for the final
build of the image.
If "mkosi -iff" is run (i.e. with two --force parameters) any
pre-existing cache image is removed before the operation is run. In this
mode the cache images are hence generated from scratch, and never used.
If --incremental is not specified behaviour is identical to the status
quo ante.
Note that there currently is no logic in place to only rebuild the image
automatically in full if the mkosi.defaults file is newer than the cache
files, this may be added in a later commit.
To remove the cache files without rebuilding, use "mkosi clean -f".
Note that this kind of "incremental" caching is orthogonal to the
already existing "package" caching on the package manager level. The
former caches the result of a the initial package manager run, while the
latter only caches individual packages before the package manager is
run. The latter is particular useful as it permits optimizing the build
time of the usual double image generation of mkosi (i.e. when a
development build is done followed by a final build).
Note that the caching works very differently for the output modes for
raw disk images and those for directory trees. In the former case we'll
cache the actual raw disk image after all partition setup and basic
directory tree setup. In the latter case we'll store the image directory
tree as directory. Note that the "raw_squashfs" is treated like a
directory mode in this regard, since squashfs compression is applied
much later than the point in time we create the cache version of the
image.
Chris Morin [Wed, 21 Jun 2017 10:20:20 +0000 (06:20 -0400)]
fix breakage when making directory format in current directory (#89)
Making an osi of type "directory" when using the default path (the
current working directory) causes the build step to fail. This occurs
because mkosi attempts to copy the .mkosi-* directory into itself and
causes an infinite recursion.
This also resolve an issue where directories that don't contain any
tracked files are copied to the build directory (e.g. the .git
directory).
Lénaïc Huard [Thu, 15 Jun 2017 08:01:57 +0000 (10:01 +0200)]
Do not duplicate btrfs metadata (#91)
By default `mkfs.btrfs` uses the `dup` profile for metadata of non-SSD
single device btrfs filesystem.
This duplication is not desired for VM drives. So, let’s use the `single`
profile instead.
David Herrmann [Thu, 11 May 2017 15:00:54 +0000 (17:00 +0200)]
mkosi: strip man-db from arch-linux package list (#81)
The man-db hooks might trigger on boot and regenerate the database, taking up to several minutes. There is no need to trigger this in a minimal image, so strip `man-db` from the arch-linux package list. While at it, let's remove the man-pages as well, since there is little need for those either.
The default file should only be used if an option is not specified on
the command line, but instead, it was only used if the option *was*
specified (overriding the specified value).
Felipe Sateler [Sun, 26 Feb 2017 13:59:21 +0000 (10:59 -0300)]
Reset machine-id right before making root read-only
kernel-install, which is used to install the bootloader, requires that
the machine-id is set. Since the build or postinst scripts might want
it too, move it to the end.
mkosi: properly remove root tree and /var/tmp between first and second build
Let's make sure to remove both the old OS tree and its /var/tmp file
after the first build finished and we ran the build script inside of it,
but before we start the next iteration.
This was an unfortunate typo I unfortunately didn't notice earlier. The
file was really supposed to be created as SHA256SUMS rather than
SHA256SUM, in order to follow Ubuntu's naming of the checksums of OS
images. Moreover systemd's importd tool will actually look for
SHA256SUMS rather than SHA256SUM.
Hence, let's rename this now, even if this is to some level a compat
break. But as nobody noticed this earlier I figure people didn't care so
far.
mkosi: also bind mount /var/tmp for the build script
This is a follow-up for efa2f8593f0adf1009a9b221b51fb2b5203ac81c, and
also ensures /var/tmp is writable during the main build script
invocation (in addition to the already patched "workspace" invocations).
mkosi: optionally sign resulting kernel images with "sbsign" for UEFI SecureBoot
Maybe one day we can also support using "pesign" for this, but for now
"sbsign" is much simpler to use, and fits neatly into mkosi's design as
we can simply invoke it with key material loaded from mkosi.* key
material in the working directory.
This also moves the "Bootable" information away from the "Partitions"
section, and into the "Output" section, as "Bootable" is hardly a
specific type of partition, but more general information about the
output.
mkosi: generate a unified UEFI kernel+initrd binary as last step of the build
On verity-enabled builds we want to embedd the root hash of the root
file system in the kernel command line, and want to package the kernel,
the initrd, a kernel command line as a single EFI executable that later
may be signed.
This adds an aditional step to the build process: after the verity data
has been generated, invoke "dracut --uefi" in the image to generate the
unified image, and place it in /EFI/Linux in the ESP. sd-boot already
knows how to deal with these single-file boot items, hence this is
sufficient to make a boot image appear in its menu.
This also disables ESP kernel installation during the usual RPM process,
as we aren't interested in the images generated that way.
This patch makes the necessary changes for all of this only for Fedora.
Other distributions using dracut should be able to make the similar
logic work, but that's left for a later commit.
(the roothash= related bits in this PR will only be honoured by systemd
at boot-time with https://github.com/systemd/systemd/pull/4879 in place)
mkosi: tweak the default fedora package list in some conditions
When we generate a bootable ext4 image, include the e2fsck in the image. Similar for btrfs. This way we can properly fsck all devices during boot.
The Arch Linux code already had a similar tweak, let's copy that.
Also, install "cryptsetup" if encryption is requested. Strictly speaking
nothing will call the tool, however Dracut uses it has condition when to
pack DM rules into the initrd. Hence let's add it for now, even though
we can eventually drop this, when Dracut gets fixed.
This makes sure the --read-only setting has an effect on the GPT
read-only flags of the root partition.
Note that we generate the sfdisk lines in a format that works around an
sfdisk bug with older version: we place the partition name last in each
line, as we need to enclose it in "", but the sfdisk parser is confused
by it otherwise. (See: https://github.com/karelzak/util-linux/issues/380)
This adds an additional scripting hook to the build script. We look for
"mkosi.postinst". If it exists the script will be run as last step of the
each build. The script will be passed one parameter: either "build" or
"final", depending if it is invoked on the first iteration of the image
creation (the "build phase") or the second iteration (the "final
phase").
There's no point in running it for any of the other commands, and as it
conflicts with using nspawn on the directory as is let's just mount them
for the shortest possible time.
mkosi: pass the same uuid to all nspawn invocations
This is useful so that tools that need one have always the same. This is
particularly useful for dracut which uses the machine ID for creating
directories in the ESP.
mkosi: let's remove if checks from build_image() a bit
Let's move most remaining if conditions into the functions we are about
to call. This way the flow of control of build_image() is easier to
follow as the exceptions are hidden and the common flow of control
emphasised.
There's really nothing to hide here, and installation might fail
if the cache is not readable:
W: Can't drop privileges for downloading as file '/var/cache/apt/archives/partial/libexpat1_2.2.0-1_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
We need to do some buffering here, because the files we are hashing can be gigabytes
in size, but Python loops are slow, so it's better to use a larger buffer.
mkosi: extend complete_step with customizable end message
A context manager cannot see any values from the block directly, so
a mutable object (list) is used to pass the state to format the final message.
This isn't entirely elegant, but I think it is worth it, because it allows
the opening and closing messages to be specified next to one another and makes
things a bit more concise and consistent.
This adds a new switch --verity. When passed the root partition will be
complemented by a dm-verity partition, and the root hash is written to a
<image>.roothash file, as well as printed to the status output.
If this mode is enabled the partition UUID of the root partition is
changed to the first 128bit of the top-level hash, and the partition
UUID of the verity partition is changed to the second (and final) 128bit
of the top-level hash (assuming sha256 is used as hash algorithm, which
is the default). This means the two partitions can be easily determined
from the specification of the roothash alone. Since both the root
partition and the verity partition in this case are read-only and the
roothash describes both it's a good candidate for using as partition
UUID.
The partition type UUID of the verity partition is initialized to a new
set of UUIDs that match the existing per-architecture root partition
type UUIDs, that I generated randomly.
projects / thirdparty / mkosi.git / log
