kmod: eliminate duplication in calls to modinfo

The original guard when building the new todo is:

```
todo += [m for m in depinfo.modules if m not in mods and m in nametofile]
```

`mods` only accumulates modules from previous iterations of the outer while
loop. Within a single iteration, modules in the current batch are added to
mods one at a time as `moddep.items()` is iterated. This means two distinct
failure modes:

1. Cross-dependency within a batch: if modules A and B are both in the current
todo and both depend on X, X passes the m not in mods check when processing
A's deps and again when processing B's (since X hasn't been added to mods
yet). X ends up in todo twice.

2. Intra-batch back-edge: if A depends on B and both are already in the current
todo/moddep, B still passes m not in mods when processing A's dep list,
queuing B for a redundant second modinfo call in the next iteration.

Both boil down to the same root cause: the check doesn't account for modules
currently being processed.

The fix addresses both issues:

1. `m not in moddep.keys()` prevents modules from the current batch being
re-queued into the next iteration's `todo`.

2. Turning`todo` into a set prevents a module from appearing *multiple times*
in the same next `todo`, when two modules in the current batch share a
dependency

Because these duplicates frequently occur in practice, eliminating them
provides an incremental speedup on top of the original optimization
made in GH4092/GH4017.

Co-Authored-By: Jörg Behrmann <behrmann@physik.fu-berlin.de>

diff --git a/mkosi/kmod.py b/mkosi/kmod.py
index 1e83001e213a57dfa9d77c69900a7c5d2abb6e6c..5c2dbfd0356fb56e65b596a24bbf6741080d4e95 100644 (file)
--- a/mkosi/kmod.py
+++ b/mkosi/kmod.py
@@ -292,8 +292,8 @@ def resolve_module_dependencies(
     with chdir(context.root):
         nametofile = {module_path_to_name(m): m for m in all_modules(modulesd)}
 
     with chdir(context.root):
         nametofile = {module_path_to_name(m): m for m in all_modules(modulesd)}
 

-    todo = [*builtin, *modules]
-    mods = set()
+    todo = {*builtin, *modules}
+    mods: set[str] = set()

     firmware = set()
 
     while todo:
     firmware = set()
 
     while todo:


@@ -304,20 +304,20 @@ def resolve_module_dependencies(
         # build a map that maps the module name to both its module dependencies and its firmware
         # dependencies. Because there's more kernel modules than the max number of accepted CLI
         # arguments, we split the modules list up into chunks if needed.
         # build a map that maps the module name to both its module dependencies and its firmware
         # dependencies. Because there's more kernel modules than the max number of accepted CLI
         # arguments, we split the modules list up into chunks if needed.

-        for i in range(0, len(todo), 8500):
-            chunk = todo[i : i + 8500]
+        ittodo = iter(todo)
+        while chunk := tuple(itertools.islice(ittodo, 8500)):

             moddep |= modinfo(context, kver, chunk)
 
             moddep |= modinfo(context, kver, chunk)
 

-        todo = []
+        todo = set()
+        mods |= moddep.keys()

 
         for name, depinfo in moddep.items():
             for d in depinfo.modules:
                 if d not in nametofile and d not in builtin:
                     logging.warning(f"{d} is a dependency of {name} but is not installed, ignoring ")
 
 
         for name, depinfo in moddep.items():
             for d in depinfo.modules:
                 if d not in nametofile and d not in builtin:
                     logging.warning(f"{d} is a dependency of {name} but is not installed, ignoring ")
 

-            mods.add(name)

             firmware.update(depinfo.firmware)
             firmware.update(depinfo.firmware)

-            todo += [m for m in depinfo.modules if m not in mods and m in nametofile]
+            todo.update(m for m in depinfo.modules if m not in mods and m in nametofile)

 
     return set(nametofile[m] for m in mods if m in nametofile), set(firmware)
 
 
     return set(nametofile[m] for m in mods if m in nametofile), set(firmware)