From: Daan De Meyer Date: Thu, 20 Apr 2023 10:46:04 +0000 (+0200) Subject: Split up --base-image into --base-tree and --overlay X-Git-Tag: v15~222^2 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=refs%2Fpull%2F1468%2Fhead;p=thirdparty%2Fmkosi.git Split up --base-image into --base-tree and --overlay --base-tree indicates a base image. It differs from --skeleton-tree in that use of --base-tree indicates that we've already installed a distribution and should only install extra packages, not install the distribution from scratch. If --overlay is not specified, we just copy all the specified base trees to the root directory before doing anything else (even before copying the skeleton trees) and after that we operate as usual. If --overlay is specified, instead of copying all the base trees to the root directory, we set up an overlayfs mount with all the base trees as lowerdirs. After that we operate as usual. The effect is that all our usual steps will operate on a full view of the image, but the output will only contain the additions we made on top of the specified base trees. --- diff --git a/NEWS.md b/NEWS.md index a48d49de5..b332d28af 100644 --- a/NEWS.md +++ b/NEWS.md @@ -79,6 +79,7 @@ - Apt now uses the keyring from the host instead of the keyring from the image. This means `debian-archive-keyring` or `ubuntu-archive-keyring` are now required to be installed to build Debian or Ubuntu images respectively. +- `--base-image` is split into `--base-tree` and `--overlay`. ## v14 diff --git a/action.yaml b/action.yaml index 6c0c23d76..b0f29eb58 100644 --- a/action.yaml +++ b/action.yaml @@ -41,21 +41,26 @@ runs: sudo apt-get update sudo apt-get build-dep systemd sudo apt-get install libfdisk-dev + git clone https://github.com/systemd/systemd --depth=1 meson systemd/build systemd -Drepart=true -Defi=true -Dbootloader=true - ninja -C systemd/build systemd-nspawn systemd-repart bootctl ukify systemd-analyze systemd-nspawn systemctl - sudo ln -svf $PWD/systemd/build/systemd-repart /usr/bin/systemd-repart - sudo ln -svf $PWD/systemd/build/bootctl /usr/bin/bootctl - sudo ln -svf $PWD/systemd/build/ukify /usr/bin/ukify - sudo ln -svf $PWD/systemd/build/systemd-analyze /usr/bin/systemd-analyze - sudo ln -svf $PWD/systemd/build/systemd-nspawn /usr/bin/systemd-nspawn - sudo ln -svf $PWD/systemd/build/systemctl /usr/bin/systemctl - systemd-repart --version - bootctl --version - ukify --version - systemd-analyze --version - systemd-nspawn --version - systemctl --version + + BINARIES=( + bootctl + systemctl + systemd-analyze + systemd-dissect + systemd-nspawn + systemd-nspawn + systemd-repart + ukify + ) + + ninja -C systemd/build ${BINARIES[@]} + + for BINARY in "${BINARIES[@]}"; do + sudo ln -svf $PWD/systemd/build/$BINARY /usr/bin/$BINARY + done - name: Install shell: bash diff --git a/mkosi.md b/mkosi.md index 2d8b4d9d9..39230ebc3 100644 --- a/mkosi.md +++ b/mkosi.md @@ -500,6 +500,19 @@ a boolean argument: either "1", "yes", or "true" to enable, or "0", image root, so any `CopyFiles=` source paths in partition definition files will be relative to the image root directory. +`Overlay=`, `--overlay` + +: When used together with `BaseTrees=`, the output will consist only out of + changes to the specified base trees. Each base tree is attached as a lower + layer in an overlayfs structure, and the output becomes the upper layer, + initially empty. Thus files that are not modified compared to the base trees + will not be present in the final output. + +: This option may be used to create systemd "system extensions" or + portable services. See + https://uapi-group.org/specifications/specs/extension_image for more + information. + `TarStripSELinuxContext=`, `--tar-strip-selinux-context` : If running on a SELinux-enabled system (Fedora Linux, CentOS, Rocky Linux, @@ -570,6 +583,22 @@ a boolean argument: either "1", "yes", or "true" to enable, or "0", way is mounted into both the development and the final image while the package manager is running. +`BaseTrees=`, `--base-tree=` + +: Takes a colon separated pair of directories to use as base images. When + used, these base images are each copied into the OS tree and form the + base distribution instead of installing the distribution from scratch. + Only extra packages are installed on top of the ones already installed + in the base images. Note that for this to work properly, the base image + still needs to contain the package manager metadata (see + `CleanPackageMetadata=`). + +: Instead of a directory, a tar file or a disk image may be provided. In + this case it is unpacked into the OS tree. This mode of operation allows + setting permissions and file ownership explicitly, in particular for projects + stored in a version control system such as `git` which retain full file + ownership and access mode metadata for committed files. + `SkeletonTrees=`, `--skeleton-tree=` : Takes a colon separated pair of paths. The first path refers to a @@ -583,14 +612,9 @@ a boolean argument: either "1", "yes", or "true" to enable, or "0", for this purpose with the root directory as target (also see the "Files" section below). -: Instead of a directory, a tar file may be provided. In this case - it is unpacked into the OS tree before the package manager is - invoked. This mode of operation allows setting permissions and file - ownership explicitly, in particular for projects stored in a version - control system such as `git` which retain full file ownership and - access mode metadata for committed files. If the tar file - `mkosi.skeleton.tar` is found in the local directory it will be - automatically used for this purpose. +: As with the base tree logic above, instead of a directory, a tar + file may be provided too. `mkosi.skeleton.tar` will be automatically + used if found in the local directory. `ExtraTrees=`, `--extra-tree=` @@ -604,8 +628,8 @@ a boolean argument: either "1", "yes", or "true" to enable, or "0", automatically used for this purpose with the root directory as target. (also see the "Files" section below). -: As with the skeleton tree logic above, instead of a directory, a tar - file may be provided too. `mkosi.skeleton.tar` will be automatically +: As with the base tree logic above, instead of a directory, a tar + file may be provided too. `mkosi.extra.tar` will be automatically used if found in the local directory. `CleanPackageMetadata=`, `--clean-package-metadata=` @@ -774,20 +798,6 @@ a boolean argument: either "1", "yes", or "true" to enable, or "0", files. This option may be used multiple times in which case the initrd lists are combined. -`BaseImage=`, `--base-image=` - -: Use the specified directory or file system image as the base image, - and create the output image that consists only of changes from this - base. The base image is attached as the lower file system in an - overlayfs structure, and the output filesystem becomes the upper - layer, initially empty. Thus files that are not modified compared to - the base image are not present in the output image. - -: This option may be used to create systemd "system extensions" or - portable services. See - https://systemd.io/PORTABLE_SERVICES/#extension-images for more - information. - ### [Validation] Section `Checksum=`, `--checksum` diff --git a/mkosi/__init__.py b/mkosi/__init__.py index b48f4c319..2a6558162 100644 --- a/mkosi/__init__.py +++ b/mkosi/__init__.py @@ -106,13 +106,26 @@ def btrfs_subvol_create(path: Path, mode: int = 0o755) -> None: def mount_image(state: MkosiState) -> Iterator[None]: with complete_step("Mounting image…", "Unmounting image…"), contextlib.ExitStack() as stack: - if state.config.base_image is not None: - if state.config.base_image.is_dir(): - base = state.config.base_image - else: - base = stack.enter_context(dissect_and_mount(state.config.base_image, state.workspace / "base")) + if state.config.base_trees and state.config.overlay: + bases = [] + state.workspace.joinpath("bases").mkdir(exist_ok=True) + + for path in state.config.base_trees: + d = Path(stack.enter_context(tempfile.TemporaryDirectory(dir=state.workspace / "base", prefix=path.name))) + d.rmdir() # We need the random name, but we want to create the directory ourselves + + if path.is_dir(): + bases += [path] + elif path.suffix == ".tar": + shutil.unpack_archive(path, d) + bases += [d] + elif path.suffix == ".raw": + stack.enter_context(dissect_and_mount(path, d)) + bases += [d] + else: + die(f"Unsupported base tree source {path}") - stack.enter_context(mount_overlay(base, state.root, state.workdir, state.root)) + stack.enter_context(mount_overlay(bases, state.root, state.workdir, state.root, read_only=False)) yield @@ -263,7 +276,7 @@ def install_distribution(state: MkosiState, cached: bool) -> None: if cached: return - if state.config.base_image: + if state.config.base_trees: if not state.config.packages: return @@ -389,7 +402,7 @@ def configure_autologin(state: MkosiState) -> None: def mount_build_overlay(state: MkosiState, read_only: bool = False) -> ContextManager[Path]: - return mount_overlay(state.root, state.build_overlay, state.workdir, state.root, read_only) + return mount_overlay([state.root], state.build_overlay, state.workdir, state.root, read_only) def run_prepare_script(state: MkosiState, cached: bool, build: bool) -> None: @@ -552,6 +565,28 @@ def install_boot_loader(state: MkosiState) -> None: ) +def install_base_trees(state: MkosiState, cached: bool) -> None: + if not state.config.base_trees or cached or state.config.overlay: + return + + with complete_step("Copying in base trees…"): + for path in state.config.base_trees: + + if path.is_dir(): + copy_path(path, state.root) + elif path.suffix == ".tar": + shutil.unpack_archive(path, state.root) + elif path.suffix == ".raw": + run(["systemd-dissect", "--copy-from", path, "/", state.root]) + else: + die(f"Unsupported base tree source {path}") + + if path.is_dir(): + copy_path(path, state.root) + else: + shutil.unpack_archive(path, state.root) + + def install_skeleton_trees(state: MkosiState, cached: bool) -> None: if not state.config.skeleton_trees or cached: return @@ -1267,9 +1302,12 @@ def load_args(args: argparse.Namespace) -> MkosiConfig: if not p.is_file(): die(f"Initrd {p} is not a file") + if args.overlay and not args.base_trees: + die("--overlay can only be used with --base-tree") + # For unprivileged builds we need the userxattr OverlayFS mount option, which is only available in Linux v5.11 and later. with prepend_to_environ_path(args.extra_search_paths): - if (args.build_script is not None or args.base_image is not None) and GenericVersion(platform.release()) < GenericVersion("5.11") and os.geteuid() != 0: + if (args.build_script is not None or args.base_trees) and GenericVersion(platform.release()) < GenericVersion("5.11") and os.geteuid() != 0: die("This unprivileged build configuration requires at least Linux v5.11") return MkosiConfig(**vars(args)) @@ -1319,7 +1357,8 @@ def check_script_input(path: Optional[Path]) -> None: def check_inputs(config: MkosiConfig) -> None: try: - check_tree_input(config.base_image) + for base in config.base_trees: + check_tree_input(base) for tree in (config.skeleton_trees, config.extra_trees): @@ -1777,11 +1816,9 @@ def invoke_repart(state: MkosiState, skip: Sequence[str] = [], split: bool = Fal def build_image(state: MkosiState, *, manifest: Optional[Manifest] = None) -> None: - cached = reuse_cache_tree(state) - if state.for_cache and cached: - return - with mount_image(state): + cached = reuse_cache_tree(state) + install_base_trees(state, cached) install_skeleton_trees(state, cached) install_distribution(state, cached) run_prepare_script(state, cached, build=False) diff --git a/mkosi/backend.py b/mkosi/backend.py index 5a70728e4..27fedd50a 100644 --- a/mkosi/backend.py +++ b/mkosi/backend.py @@ -220,6 +220,7 @@ class MkosiConfig: repositories: list[str] repo_dirs: list[Path] repart_dirs: list[Path] + overlay: bool architecture: str output_format: OutputFormat manifest_format: list[ManifestFormat] @@ -243,6 +244,7 @@ class MkosiConfig: with_docs: bool with_tests: bool cache_dir: Optional[Path] + base_trees: list[Path] extra_trees: list[tuple[Path, Optional[Path]]] skeleton_trees: list[tuple[Path, Optional[Path]]] clean_package_metadata: Optional[bool] @@ -259,7 +261,6 @@ class MkosiConfig: with_network: bool cache_only: bool nspawn_settings: Optional[Path] - base_image: Optional[Path] checksum: bool split_artifacts: bool sign: bool diff --git a/mkosi/config.py b/mkosi/config.py index 16c404f55..e67bd8fc5 100644 --- a/mkosi/config.py +++ b/mkosi/config.py @@ -479,6 +479,11 @@ class MkosiConfigParser: parse=config_make_list_parser(delimiter=",", parse=make_path_parser(required=True)), paths=("mkosi.repart",), ), + MkosiConfigSetting( + dest="overlay", + section="Output", + parse=config_parse_boolean, + ), MkosiConfigSetting( dest="packages", section="Content", @@ -526,6 +531,11 @@ class MkosiConfigParser: parse=config_make_path_parser(required=False), paths=("mkosi.cache",), ), + MkosiConfigSetting( + dest="base_trees", + section="Content", + parse=config_make_list_parser(delimiter=",", parse=make_path_parser(required=True)), + ), MkosiConfigSetting( dest="extra_trees", section="Content", @@ -620,11 +630,6 @@ class MkosiConfigParser: parse=config_make_path_parser(required=True), paths=("mkosi.nspawn",), ), - MkosiConfigSetting( - dest="base_image", - section="Content", - parse=config_make_path_parser(required=True), - ), MkosiConfigSetting( dest="initrds", section="Content", @@ -1040,6 +1045,13 @@ class MkosiConfigParser: dest="repart_dirs", action=action, ) + group.add_argument( + "--overlay", + metavar="BOOL", + help="Only output the additions on top of the given base trees", + nargs="?", + action=action, + ) group = parser.add_argument_group("Content options") group.add_argument( @@ -1098,6 +1110,13 @@ class MkosiConfigParser: help="Package cache path", action=action, ) + group.add_argument( + '--base-tree', + metavar='PATH', + help='Use the given tree as base tree (e.g. lower sysext layer)', + dest="base_trees", + action=action, + ) group.add_argument( "--extra-tree", metavar="PATH", @@ -1199,12 +1218,6 @@ class MkosiConfigParser: dest="nspawn_settings", action=action, ) - group.add_argument( - '--base-image', - metavar='IMAGE', - help='Use the given image as base (e.g. lower sysext layer)', - action=action, - ) group.add_argument( "--initrd", help="Add a user-provided initrd to image", diff --git a/mkosi/manifest.py b/mkosi/manifest.py index cd8fb937d..eec4b85ad 100644 --- a/mkosi/manifest.py +++ b/mkosi/manifest.py @@ -134,7 +134,7 @@ class Manifest: # If we are creating a layer based on a BaseImage=, e.g. a sysext, filter by # packages that were installed in this execution of mkosi. We assume that the # upper layer is put together in one go, which currently is always true. - if self.config.base_image and installtime < self._init_timestamp: + if self.config.base_trees and installtime < self._init_timestamp: continue package = PackageManifest("rpm", name, evr, arch, size) @@ -178,7 +178,7 @@ class Manifest: # If we are creating a layer based on a BaseImage=, e.g. a sysext, filter by # packages that were installed in this execution of mkosi. We assume that the # upper layer is put together in one go, which currently is always true. - if self.config.base_image and installtime < self._init_timestamp: + if self.config.base_trees and installtime < self._init_timestamp: continue package = PackageManifest("deb", name, version, arch, size) diff --git a/mkosi/mounts.py b/mkosi/mounts.py index 66df4ea3a..6eef34d41 100644 --- a/mkosi/mounts.py +++ b/mkosi/mounts.py @@ -86,13 +86,13 @@ def mount( @contextlib.contextmanager def mount_overlay( - lower: Path, - upper: Path, + lowerdirs: Sequence[Path], + upperdir: Path, workdir: Path, where: Path, read_only: bool = True, ) -> Iterator[Path]: - options = [f"lowerdir={lower}", f"upperdir={upper}", f"workdir={workdir}"] + options = [f"lowerdir={lower}" for lower in lowerdirs] + [f"upperdir={upperdir}", f"workdir={workdir}"] # userxattr is only supported on overlayfs since kernel 5.11 if GenericVersion(platform.release()) >= GenericVersion("5.11"): @@ -103,7 +103,7 @@ def mount_overlay( yield where finally: with complete_step("Cleaning up overlayfs"): - delete_whiteout_files(upper) + delete_whiteout_files(upperdir) @contextlib.contextmanager @@ -112,4 +112,4 @@ def dissect_and_mount(image: Path, where: Path) -> Iterator[Path]: try: yield where finally: - run(["umount", "--recursive", where]) + run(["systemd-dissect", "-U", where])