]> git.ipfire.org Git - thirdparty/git.git/commit - pack-bitmap-write.c
pack-bitmap-write: relax unique revwalk condition
authorDerrick Stolee <dstolee@microsoft.com>
Tue, 8 Dec 2020 22:05:26 +0000 (17:05 -0500)
committerJunio C Hamano <gitster@pobox.com>
Tue, 8 Dec 2020 22:49:07 +0000 (14:49 -0800)
commit45f4eeb2919e2c802a6c1f64a2b1c299f7434938
tree22b0301626576d50f487bedf98ab1fbd0fd8bc64
parent341fa34887630a7adf6b3a771ae866ce66e7967e
pack-bitmap-write: relax unique revwalk condition

The previous commits improved the bitmap computation process for very
long, linear histories with many refs by removing quadratic growth in
how many objects were walked. The strategy of computing "intermediate
commits" using bitmasks for which refs can reach those commits
partitioned the poset of reachable objects so each part could be walked
exactly once. This was effective for linear histories.

However, there was a (significant) drawback: wide histories with many
refs had an explosion of memory costs to compute the commit bitmasks
during the exploration that discovers these intermediate commits. Since
these wide histories are unlikely to repeat walking objects, the benefit
of walking objects multiple times was not expensive before. But now, the
commit walk *before computing bitmaps* is incredibly expensive.

In an effort to discover a happy medium, this change reduces the walk
for intermediate commits to only the first-parent history. This focuses
the walk on how the histories converge, which still has significant
reduction in repeat object walks. It is still possible to create
quadratic behavior in this version, but it is probably less likely in
realistic data shapes.

Here is some data taken on a fresh clone of the kernel:

             |   runtime (sec)    |   peak heap (GB)   |
             |                    |                    |
             |   from  |   with   |   from  |   with   |
             | scratch | existing | scratch | existing |
  -----------+---------+----------+---------+-----------
    original |  64.044 |   83.241 |   2.088 |    2.194 |
  last patch |  45.049 |   37.624 |   2.267 |    2.334 |
  this patch |  88.478 |   53.218 |   2.157 |    2.224 |

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Helped-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
pack-bitmap-write.c
t/t5310-pack-bitmaps.sh