pack-bitmap: reuse stored selected bitmaps

When `fill_bitmap_commit()` reaches an ancestor that was selected for
its own bitmap and processed earlier, its object closure is already
stored in `writer->bitmaps` as an EWAH bitmap. As a result, walking
through that commit's tree and parents again is redundant.

Teach `fill_bitmap_commit()` to notice that case. For non-root commits in
the walk, look for a stored selected bitmap and OR it into the bitmap
being built. If one exists, skip the commit, its tree, and its parents.

Building bitmaps from scratch on the same test repository from the
previous commits yields a significant speed-up:

    +------------------+-------------+-------------+---------------------+
    |                  | HEAD^       | HEAD        | Delta               |
    +------------------+-------------+-------------+---------------------+
    | elapsed          |   562.8 s   |   324.8 s   |   -237.9 s (-42.3%) |
    | cycles           | 2,621.3 B   | 1,508.6 B   | -1,112.7 B (-42.4%) |
    | instructions     | 2,348.9 B   | 1,436.6 B   |   -912.3 B (-38.8%) |
    | CPI              |     1.116   |     1.050   |   -0.066    (-5.9%) |
    +------------------+-------------+-------------+---------------------+

In our testing repository, there are 1,261 commits selected for bitmap
coverage, and 1,382 maximal commits induced as a result of that. Of the
1,382 calls made to `fill_bitmap_commit()` (one per maximal commit), 131
of them can be short-circuited at some point during their traversal as a
consequence of this change.

In large repositories where the cost of filling the bitmap for any
individual commit is large, being able to short-circuit even ~9.5% of
the calls to `fill_bitmap_commit()` results in a significant savings.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 726103970206643e64c9853835ecbf38d0f3aefa..651ad467469f4413bce4a522fc63dee7b3b4e188 100644 (file)
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -509,6 +509,9 @@ static int fill_bitmap_tree(struct bitmap_writer *writer,
 static int reused_bitmaps_nr;
 static int reused_pseudo_merge_bitmaps_nr;
 
+static int fill_bitmap_commit_calls_nr;
+static int fill_bitmap_commit_found_ancestor_nr;
+
 static int fill_bitmap_commit(struct bitmap_writer *writer,
                              struct bb_commit *ent,
                              struct commit *commit,
@@ -519,6 +522,9 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,
 {
        int found;
        uint32_t pos;
+
+       fill_bitmap_commit_calls_nr++;
+
        if (!ent->bitmap)
                ent->bitmap = bitmap_new();
 
@@ -553,6 +559,28 @@ static int fill_bitmap_commit(struct bitmap_writer *writer,
                        bitmap_free(remapped);
                }
 
+               /*
+                * If we encounter an ancestor for which we have already
+                * computed a bitmap during this build (i.e. a regular
+                * selected commit processed earlier in topo order), we can
+                * short-circuit the walk: its stored bitmap already covers
+                * the commit itself, its tree, and all of its ancestors.
+                */
+               if (c != commit) {
+                       khiter_t hash_pos = kh_get_oid_map(writer->bitmaps,
+                                                          c->object.oid);
+                       if (hash_pos != kh_end(writer->bitmaps)) {
+                               struct bitmapped_commit *stored =
+                                       kh_value(writer->bitmaps, hash_pos);
+                               if (stored && stored->bitmap) {
+                                       fill_bitmap_commit_found_ancestor_nr++;
+                                       bitmap_or_ewah(ent->bitmap,
+                                                      stored->bitmap);
+                                       continue;
+                               }
+                       }
+               }
+
                /*
                 * Mark ourselves and queue our tree. The commit
                 * walk ensures we cover all parents.
@@ -692,6 +720,12 @@ int bitmap_writer_build(struct bitmap_writer *writer)
        trace2_data_intmax("pack-bitmap-write", writer->repo,
                           "building_bitmaps_pseudo_merge_reused",
                           reused_pseudo_merge_bitmaps_nr);
+       trace2_data_intmax("pack-bitmap-write", writer->repo,
+                          "fill_bitmap_commit_calls_nr",
+                          fill_bitmap_commit_calls_nr);
+       trace2_data_intmax("pack-bitmap-write", writer->repo,
+                          "fill_bitmap_commit_found_ancestor_nr",
+                          fill_bitmap_commit_found_ancestor_nr);
 
        stop_progress(&writer->progress);