Put build history into the output directory

Running e.g. `test_initrd` and `test_initrd_luks` in parallel fails one of
them with "Image 'main' has not been built yet". The integration tests build
into a per-test `--output-directory`, but `vm()`/`boot()` did not pass it, so
those verbs recovered the build configuration from the *shared* global history
in `<configdir>/.mkosi-private/history/latest.json`. With concurrent builds
that file holds whatever the last build wrote, so a verb reads back another
build's config (e.g. the wrong `Format=`).

Tie the build history to the output directory: when an output directory is
given on the CLI, store and read the history under it instead of in the config
directory. Each build's history is then isolated, and a verb pointed at a
given `--output-directory` reads back exactly that build's configuration. In
the tests, pass `--output-directory` to `vm()` and `boot()` as well.

As a consequence, `mkosi vm` (and the other verbs that consume a previous
build) now require `-O`/`--output-directory` when the build used one. This is
a behaviour change, but unbreaks having more than one output dir.

Note: If a config file sets `OutputDirectory=`, the history continues to
be in the config dir, as before. The computation of the history
directory (necessarily) happens before parsing the config
files/includes. This *only* applies to the CLI option.

Rejected alternatives:
 * This cannot be worked around with `--history=no` in the tests': that
   only disables *writing* history, not *reading* it, so vm/boot still
   pick up a stale (in our setup, empty) `latest.json` and fall back to
   the wrong config.
 * A dedicated `--history-dir` option would just be redundant with
   `--output-dir`.

diff --git a/mkosi/config.py b/mkosi/config.py
index 1008495071ec2e3356a154428dbaeb4960a03352..9513b488c390a49dc8f37bba7e64516f01db1ef9 100644 (file)
--- a/mkosi/config.py
+++ b/mkosi/config.py
@@ -5445,7 +5445,15 @@ def want_default_initrd(config: Config) -> bool:
     return Path("default") in config.initrds
 
 
-def finalize_historydir(args: Args) -> Path:
+def finalize_historydir(args: Args, output_dir: Optional[Path] = None) -> Path:
+    # When an output dir is given on the CLI, store the build history there so that concurrent builds with
+    # different output dirs don't clobber a shared history. Don't check the finalized OutputDirectory=
+    # config, only the CLI value: the former isn't known yet here (config files and includes are
+    # parsed later) and vm/boot can't see it anyway since they recover the config from the history instead of
+    # parsing it. An output dir set only in config files keeps the history in the config dir.
+    if output_dir is not None:
+        return output_dir / ".mkosi-private/history"
+
     configdir = finalize_configdir(args.directory)
     return (configdir or Path.cwd()) / ".mkosi-private/history"
 
@@ -5502,7 +5510,7 @@ def parse_config(
         return args, None, ()
 
     configdir = finalize_configdir(args.directory)
-    historydir = finalize_historydir(args)
+    historydir = finalize_historydir(args, context.cli.get("output_dir"))
 
     if have_history(args, historydir):
         history = Config.from_partial_json((historydir / "latest.json").read_text())

diff --git a/tests/__init__.py b/tests/__init__.py
index 3e107a62e653c95251673e8d3d288f5970b87537..440d90a7c3a7fb23fde6596f2b89763e0b4dea77 100644 (file)
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -126,12 +126,13 @@ class Image:
                 "--register=no",
                 "--machine",
                 self.machine,
+                "--output-directory", self.output_dir,
                 *options,
             ],
             args,
             stdin=sys.stdin if sys.stdin.isatty() else None,
             check=False,
-        )
+        )  # fmt: skip
 
         if result.returncode != 123:
             raise subprocess.CalledProcessError(result.returncode, result.args, result.stdout, result.stderr)
@@ -158,12 +159,13 @@ class Image:
                 "--register=no",
                 "--machine",
                 self.machine,
+                "--output-directory", self.output_dir,
                 *options,
             ],
             args,
             stdin=sys.stdin if sys.stdin.isatty() else None,
             check=False,
-        )
+        )  # fmt: skip
 
         if result.returncode != 123:
             raise subprocess.CalledProcessError(result.returncode, result.args, result.stdout, result.stderr)