Lucas De Marchi [Thu, 4 Jan 2018 22:45:34 +0000 (14:45 -0800)]
Add tar archive support for extra and skeleton trees
Like documentation here says, it allows one to preseve file ownership
when adding files to the final image. This may be particularly useful if
you are adding configuration files for a daemon that is not supposed to
run as root, but as a pre-defined user.
Lucas De Marchi [Thu, 4 Jan 2018 22:23:40 +0000 (14:23 -0800)]
Simplify assignments by using empty lists as default
Make argparse use empty lists as default rather than None
for arguments that receive lists. This allows us to simplify all the
conditional assign or extend throught the code.
This is done for packages, build_packages, extra_trees and
skeleton_trees, removing the conditional assignments and leaving only
a call to extend() that can handle the case the list is empty. Also
checks for `if X is None` need to be converted to `if not X`.
Lucas De Marchi [Wed, 8 Nov 2017 16:09:32 +0000 (08:09 -0800)]
Add support to skeleton trees
It's already possible to pass extra trees to copy over the final tree
after installing all packages. However in some cases it is desirable to
copy the tree before installing packages or running anything in the
image. One of the cases is if we want to enable additional repositories
for the package manager or want to configure it in a particular way.
Now mkosi can use a mkosi.skeleton directory (or by passing a
--skeleton-tree argument) in the same way it currently supports extra
trees, however copying them before running the "install_distro" phase.
To allow changing the package manager configuration distros may need
more tweaks to point the package manager to the right files. Right now
this has been tested with Fedora 27 by adding
`mkosi.skeleton/etc/yum.repos.d/rpmfusion-free.repo` and installing a
package from that repo.
Sebastian [Mon, 27 Nov 2017 07:04:49 +0000 (08:04 +0100)]
Change opensuse zypper pattern to a form that works everywhere (#109)
[zj: The pattern "lost visibility" at some point. This is fixed in tumbleweed, but
not in some other versions, so let's use the form that works everywhere for now.
See https://bugzilla.opensuse.org/show_bug.cgi?id=1049997.]
... instead of using subprocess.run etc. We use those in some many places it makes
sense to economize a bit. It makes the long lines much shorter and more readable.
Georges Discry [Fri, 17 Nov 2017 15:52:59 +0000 (16:52 +0100)]
mkosi: Recursively delete btrfs subvolumes (#177)
`btrfs subvol delete` can only delete a subvolume if it is not read-only
and if it does not contain another subvolume.
`btrfs subvol list -o` prints the subvolumes below a certain path, but
only gives the paths relative to the filesystem mount point for those
subvolumes. To compute the paths of the children, we need the relative
path of the parent subvolume given by `btrfs subvol show`.
Read-only subvolumes also have to be writeable again before being
deleted. We unconditionally unset the readonly property on a subvolume
before deletion.
This adds a new "mkosi qemu" verb that is like "mkosi boot" but invokes
things in a QEMU environment.
This tries to find the native qemu implementation, with a couple of
fallbacks if none is found, always opting for KVM acceleration.
This also tries to automatically discover the UEFI firmware blobs. For
now only the location where the files are on Fedora is encoded in the
search path, but this can be updated for other distributions easily
later on.
mkosi: rework "mkosi shell"/"mkosi boot" to use os.execvp()
When invoked this way we are just preparators for the final nspawn
command, hence it makes a lot of sense to simply replace our process
with nspawn, hence don't fork off nspawn, but simply invoke os.execvp().
Dracut really should imply that the "qemu" module is used when
"nohostonly" mode is selected (which we select), but it currently does
not, so deal with that.
mkosi: make sure when copying files we unlink existing matching files first (#198)
Previously, if a file already existed before, we'd open it for write and
write the new file into the same file. If the old file was larger than
the new file we'd not truncate it, so that in that case the resulting
file was a combination of the new small file plus the old suffix.
Florent Thiery [Tue, 14 Nov 2017 09:56:56 +0000 (10:56 +0100)]
rework image cache logic so that it works without the presence of a build script (#155)
In that case, pre-dev and dev images are not generated at all anymore, so that cache images are
not generated every time; note that this means that pre-dev and pre-inst images currently need to
be manually removed to be re-generated.
Fixes #143.
I my build-less case i used the mkosi.postinst to perform the actual customizations, and in that case
the image caching was not used and it took way longer. With these patches the postinst is applied to
the mediacoder-image.raw.cache-pre-inst cache image, speeding up the process.
I use the postinst script to create users, enable/disable services, setup locale, install custom prebuilt
packages. Rebuilding the base image is pretty slow, so i think it is still useful to support -i without
caching (ex: for producing distro spins).
[zj: add commit message based on comments in the PR]
There are various circumstances in which it is OK to have permissions
mask which goes above 0o700:
- when the user has own group
- when the file is shared with others using a special group
- when doing QA and the password should be set but is not secret
- when the access is limited through other means
So let's not be too strict here.
Also, let's use plain octal mask instead of the stat.S_* macros. This
is way more legible! This is inspired by a recent comment from Linus
Torvalds:
https://github.com/torvalds/linux/commit/277642dcca765a1955d4c753a5a315ff7f2eb09d.
In Python we have the advantage that we have an unambigous notation
for octal numbers, so let's just use them.
Lucas De Marchi [Tue, 14 Nov 2017 08:36:59 +0000 (00:36 -0800)]
Fix breakage when generating image without bmap (#195)
When generating an image without bmap it was failing with:
‣ Creating BMAP file...
Traceback (most recent call last):
File "/usr/bin/mkosi", line 3304, in <module>
main()
File "/usr/bin/mkosi", line 3297, in main
build_stuff(args)
File "/usr/bin/mkosi", line 3231, in build_stuff
bmap = calculate_bmap(args, raw)
File "/usr/bin/mkosi", line 1995, in calculate_bmap
dir=os.path.dirname(args.output_bmap))
AttributeError: 'Namespace' object has no attribute 'output_bmap'
Fix it by doing the same as we do in other functions, checking if
bmap is actually enabled.
Lucas De Marchi [Sun, 12 Nov 2017 21:47:59 +0000 (13:47 -0800)]
Add support to bmap-tools (#190)
bmap-tools (https://github.com/intel/bmap-tools) allows us to write
images to physical disks like USB drives without having to write the
entire image. It rather writes only the used blocks. With this we
reduce the time to write to a disk when we manually set the size of the
partitions.
The general idea is to copy everything, preserving access mode, times,
xattrs on files. An attempt to create the new object is first made,
followed by an unlink and a repeated attempt. This optimizes for the
case where the output tree is empty, which seems to be the common
thing in our case.
Reflink is attempted for normal files.
Symlinks are not adjusted: if they pointed outside of the tree being
copied, they might be dangling now.
Block and character device nodes and any other special files are not
copied.
mkosi: create builddir if configured but missing (#181)
If a builddir is explicitly configured but missing, automatically create
it.
This follows the logic already implemented for mkosi.output/ and
mkosi.cache/.
This is particularly useful when storing mkosi settings in a git repo,
as git normally doesn't allow us to store empty directories, and hence
we cannot make use of auto-discovery of mkosi.output/ and mkosi.cache/.
By configuring these paths explicitly in mkosi.default however, we can
work around this, as in that case we'll create the directories if
needed.
mkosi: exclude all cache and output directories from the copied build sources
We should exclude mkosi.builddir/, mkosi.output/, mkosi.cache/ when
preparing the build image, as they aren't really part of the sources,
but contain artifacts of previous mkosi invocations.
Franck Bui [Thu, 12 Oct 2017 15:08:11 +0000 (17:08 +0200)]
openSUSE: don't use https to access mirrors for the time being
Some openSUSE mirrors redirect https to http and this seems to make
curl(1) unhappy now.
There's a request to fix that on mirror side:
https://github.com/openSUSE/mirrorbrain/issues/3 but for the time
being switch to http to access all mirrors.
There seems to be no additional values in https:// as the rpms are
signed by gpg keys already, so is the repodata anyways.
Let's add a way to store the root pw in a separate file, outside of
mkosi.default. That way it's easy for people to build images locally of
an upstream project without having to modify the mkosi.default file,
simply by dropping their own file.
mkosi: unbreak the squashfs mode, when used together with build trees (#160)
We need to create the mount point for the build tree early on, so that
it is included in the read-only squashfs image, and we can mount the
host's builddir into it.
Without this patch squashfs builds with builddir usage fail.
mkosi: add new switch --without-tests/-T for running the build script without running tests
This new option simply controls the $WITH_TESTS environment variable
(which defaults to 1), that is supposed to be honoured by the build
script the same way as $WITH_DOCS is honoured. If set to 0 the build script
should avoid unit running tests during the build process.
This is useful for speeding up the build process, in particular when
combining this with incremental mode, as it brings build-times for mkosi
much closer to build-times on the host, when developing software.
Note that this new switch changes exactly nothing in the way mkosi puts
together image, it only controls the value of $WITH_TESTS, and it is
exclusively up to the build script to honour this or not.
My new favourite command is now:
# mkosi -ifT shell
This command will build a new image, avoid running tests and quickly
give me a shell in it.
mkosi: also clear out the package cache on "mkosi -ff clean"
We already clear out the build dir and the incremental images, hence
let's clean out the package cache too, in order to avoid surprises and
confusion, but do so only if the use specifies at least two "--force"
parameters on "mkosi clean" or three on "mkosi build"
Both verbs are very similar: they build the image similar to "mkosi
build" (except if it exists already) and then invoke "systemd-nspawn"
interactively on the output image. "boot" will pass "--boot" to nspawn,
while "shell" doesn't.
The command may be combined with "-f" as usual, to force a rebuild
before invoking nspawn on the result.
With this in place getting to a test shell from a current git checkout
of a project like systemd is as easy as typing "mkosi shell".
In a later PR we should really add a "mkosi vm" command or so, which
permits booting up the image in qemu properly finding UEFI and kvm
support for it, and so on.
Felipe Sateler [Fri, 21 Jul 2017 08:33:57 +0000 (04:33 -0400)]
Ensure machine name is unique (#127)
Even though we don't register the machine, systemd-nspawn creates a
scope based on the name. Therefore, make sure the name is unique to
prevent collisions.
See https://github.com/systemd/systemd/issues/6347
Lucas De Marchi [Tue, 18 Jul 2017 20:25:22 +0000 (13:25 -0700)]
rework copy_git_files() to use a git clone
Using a git clone allows to solve some problems:
- It allows to copy git submodules without having to fallback to
--use-git-files=no since git-ls-files doesn't recurse submodules
- It allows build systems that rely on git describe or similar to
tag the build: this is more important when projects use
git modules and they want to tag the git revision of each
submodule.
- It uses less disk space when using output as directory since
when passing a local directory git will use hard links
Some additional handling for files that were modified (or added with
git -A) was put in place to allow tests to be performed before
committing.
In addition the call to git ls-files was fixed in the following cases:
- When mkosi is called with -C option: it was calling ls-files inside
the caller directory, not inside the src dir.
- When mkosi is called with --git-files=others and .gitignore
doesn't contain an entry for '.mkosi-*': it would try to copy the
temporary directory we created and fail
mkosi: support reusing builddirs between subsequent mkosi runs (#114)
With this change, mkosi will look for "mkosi.builddir" when invoked.
When this directory exists it is mounted into the build container and
may be used as location for out-of-tree builds, which can be shared
between multiple invocations to speed up the building process.
This is particularly useful when used in conjunction with incremental
building (-i), as with this in place building OS images using mkosi is
only a little slower than just building the project natively on the host
now.
"mkosi clean -f" will empty the builddir if it exists now too, in
addition to the incremental image trees.
Lucas De Marchi [Sat, 15 Jul 2017 11:51:23 +0000 (04:51 -0700)]
Ignore temporary directories while copying (#115)
If we call mkosi passing --build-sources with a directory that contains
the temporary .mkosi-* directories, it will fall in a inifinite
recursion.
This is the case in which we are have the mkosi.* files in a
subdirectory of the git repository and pass the git root dir as argument
to --build-sources.
Optionally ignore all untracked files in git ls-files (#94)
It is very easy for a large file to end up in the built image. Either
the package cache, or some other already built image, or the new
*.cache-* files are large enough to overflow the available space in
the destination image.
Some people (me ;)) find it convenient to just use the list of files
that git knows about ('git ls-files --cached') and ignore anything
else (git can be taught about the existence of new not-yet-staged
files with 'git add -N'). Then build products or images or cache files
can be safely present in the build directory, and there's no need to
craft --exclude patterns for them.
--git-files=others is the default and preserves previous behaviour.
--git-files=cached uses all committed or staged files as found in
the working directory.
It would be an interesting option to use an actual committed tree
instead of files in the working directory, but it would require a
significant changes, because files would have to be extracted directly
from the git cache. It's an interesting future direction.
I made 'git-files' a new option instead of adding a new value to
'use-git-files' because it seems better to keep the detection of
whether git ls-files should be used orthogonal to what files are
included.
Also drop the .mkosi-* excludes, because those particular files are
most likely mkosi configuration, and there's no need to exclude them.
The package cache is called mkosi.cache by default, and mkosi build
products and image caches don't match this pattern either.
We really shouldn't follow symlinks when we copy stuff. Otherwise we
can't build casync, because it contains a number of dangling symlinks in
its test-files/ directory.
Incremental builds permit caching the build and final images right after
OS package installation — before the source or build tree is copied in.
This is useful to drastically reduce the runtime of mkosi image building.
If you have a source tree, you may now type in:
# mkosi -if -t raw_gpt -o output.raw
This will build an image possibly making use of a pre-existing
"output.raw.cache-pre-dev" and "output.raw.cache-pre-inst" image files
to speed up the build. If the files exist, they are used under the
assumption they contain a pre-generated version of the disk images in
the state immediately before copying in the source tree or build tree.
(-i is short for --incremental). If the files don't exist, they are
generated, and thus available for speeding up subsequent runs. The
".cache-pre-dev" file contains the image for the development build of
the image, the ".cache-pre-inst" file contains the image for the final
build of the image.
If "mkosi -iff" is run (i.e. with two --force parameters) any
pre-existing cache image is removed before the operation is run. In this
mode the cache images are hence generated from scratch, and never used.
If --incremental is not specified behaviour is identical to the status
quo ante.
Note that there currently is no logic in place to only rebuild the image
automatically in full if the mkosi.defaults file is newer than the cache
files, this may be added in a later commit.
To remove the cache files without rebuilding, use "mkosi clean -f".
Note that this kind of "incremental" caching is orthogonal to the
already existing "package" caching on the package manager level. The
former caches the result of a the initial package manager run, while the
latter only caches individual packages before the package manager is
run. The latter is particular useful as it permits optimizing the build
time of the usual double image generation of mkosi (i.e. when a
development build is done followed by a final build).
Note that the caching works very differently for the output modes for
raw disk images and those for directory trees. In the former case we'll
cache the actual raw disk image after all partition setup and basic
directory tree setup. In the latter case we'll store the image directory
tree as directory. Note that the "raw_squashfs" is treated like a
directory mode in this regard, since squashfs compression is applied
much later than the point in time we create the cache version of the
image.
projects / thirdparty / mkosi.git / log
