hook: add jobs option

Allow the API callers to specify the number of jobs across which
hook execution can be parallelized. It defaults to 1 and no hook
currently changes it, so all hooks run sequentially as before.

This allows us to both pave the way for parallel hook execution
(that will be a follow-up patch series building upon this) and to
finish the API conversion of builtin/receive-pack.c, keeping the
output async sideband thread ("muxer") design as Peff suggested.

When .jobs==1 nothing changes, the "copy_to_sideband" async thread
still outputs directly via sideband channel 2, keeping the current
(mostly) real-time output characteristics, avoids unnecessary poll
delays or deadlock risks.

When .jobs > 1, a more complex muxer is needed to buffer the hook
output and avoid interleaving. After working on this mux I quickly
realized I was re-implementing run-command with ungroup=0 so that
idea was dropped in favor of run-command which outputs to stderr.

In other words, run-command itself already can buffer/deinterleave
pp child outputs (ungroup=0), so we can just connect its stderr to
the sideband async task when jobs > 1.

Maybe it helps to illustrate how it works with ascii graphics:

 [ Sequential (jobs = 1) ]             [ Parallel (jobs > 1) ]

 +--------------+                      +--------+   +--------+
 | Hook Process |                      | Hook 1 |   | Hook 2 |
 +--------------+                      +--------+   +--------+
        |                                  |             |
        | stderr (inherited)               | stderr pipe |
        |                                  | (captured)  |
        v                                  v             v
 +-------------------------------------------------------------+
 |                      Parent Process                         |
 |                                                             |
 |      (direct write)              [run-command (buffered)]   |
 |             |                                 |             |
 |             |                                 | writes      |
 |             v                                 v             |
 |      +-------------------------------------------+          |
 |      |             stderr (FD 2)                 |          |
 |      +-------------------------------------------+          |
 |                           |                                 |
 |                           | (dup2'd to pipe)                |
 |                           v                                 |
 |               +-----------------------+                     |
 |               | sideband async thread |                     |
 |               +-----------------------+                     |
 +-------------------------------------------------------------+

When use_sideband == 0, the sideband async thread is missing, so
this same architecture just outputs via the parent stderr stream.

See the following commits for the hook API conversions doing this,
using pre-existing sideband thread logic from `copy_to_sideband`.

Suggested-by: Jeff King <peff@peff.net>
Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff --git a/hook.c b/hook.c
index fde1f88ce87b0a6aa4a943ca19c1300b501178d7..cde719841250d16c8fc798ab9f3d2d7619bc0c7b 100644 (file)
--- a/hook.c
+++ b/hook.c
@@ -152,8 +152,8 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
                .tr2_category = "hook",
                .tr2_label = hook_name,
 
-               .processes = 1,
-               .ungroup = 1,
+               .processes = options->jobs,
+               .ungroup = options->jobs == 1,
 
                .get_next_task = pick_next_hook,
                .start_failure = notify_start_failure,
@@ -169,6 +169,9 @@ int run_hooks_opt(struct repository *r, const char *hook_name,
        if (options->path_to_stdin && options->feed_pipe)
                BUG("options path_to_stdin and feed_pipe are mutually exclusive");
 
+       if (!options->jobs)
+               BUG("run_hooks_opt must be called with options.jobs >= 1");
+
        if (options->invoked_hook)
                *options->invoked_hook = 0;
 

diff --git a/hook.h b/hook.h
index 2c8a23a569ae66db69e26219a6e44488e4d56a90..20eb56fd634c8ddf6036c1be7c3e0d30ce1c6f1f 100644 (file)
--- a/hook.h
+++ b/hook.h
@@ -16,6 +16,14 @@ struct run_hooks_opt
        /* Emit an error if the hook is missing */
        unsigned int error_if_missing:1;
 
+       /**
+        *  Number of processes to parallelize across.
+        *
+        * If > 1, output will be buffered and de-interleaved (ungroup=0).
+        * If == 1, output will be real-time (ungroup=1).
+        */
+       unsigned int jobs;
+
        /**
         * An optional initial working directory for the hook,
         * translates to "struct child_process"'s "dir" member.
@@ -90,6 +98,7 @@ struct run_hooks_opt
        .env = STRVEC_INIT, \
        .args = STRVEC_INIT, \
        .stdout_to_stderr = 1, \
+       .jobs = 1, \
 }
 
 struct hook_cb_data {