multi-pack-index: repack batches below --batch-size

The --batch-size=<size> option of 'git multi-pack-index repack' is
intended to limit the amount of work done by the repack. In the case of
a large repository, this command should repack a number of small
pack-files but leave the large pack-files alone. Most often, the
repository has one large pack-file from a 'git clone' operation and
number of smaller pack-files from incremental 'git fetch' operations.

The issue with '--batch-size' is that it also _prevents_ the repack from
happening if the expected size of the resulting pack-file is too small.
This was intended as a way to avoid frequent churn of small pack-files,
but it has mostly caused confusion when a repository is of "medium"
size. That is, not enormous like the Windows OS repository, but also not
so small that this incremental repack isn't valuable.

The solution presented here is to collect pack-files for repack if their
expected size is smaller than the batch-size parameter until either the
total expected size exceeds the batch-size or all pack-files are
considered. If there are at least two pack-files, then these are
combined to a new pack-file whose size should not be too much larger
than the batch-size.

This new strategy should succeed in keeping the number of pack-files
small in these "medium" size repositories. The concern about churn is
likely not interesting, as the real control over that is the frequency
in which the repack command is run.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Reviewed-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt
index 0c6619493c1eddd92323c620e4ae374a7c6a4b91..eb0caa04393a7136e125c95038b6b740bc215086 100644 (file)
--- a/Documentation/git-multi-pack-index.txt
+++ b/Documentation/git-multi-pack-index.txt
@@ -51,11 +51,12 @@ repack::
        multi-pack-index, then divide by the total number of objects in
        the pack and multiply by the pack size. We select packs with
        expected size below the batch size until the set of packs have
-       total expected size at least the batch size. If the total size
-       does not reach the batch size, then do nothing. If a new pack-
-       file is created, rewrite the multi-pack-index to reference the
-       new pack-file. A later run of 'git multi-pack-index expire' will
-       delete the pack-files that were part of this batch.
+       total expected size at least the batch size, or all pack-files
+       are considered. If only one pack-file is selected, then do
+       nothing. If a new pack-file is created, rewrite the
+       multi-pack-index to reference the new pack-file. A later run of
+       'git multi-pack-index expire' will delete the pack-files that
+       were part of this batch.
 +
 If `repack.packKeptObjects` is `false`, then any pack-files with an
 associated `.keep` file will not be selected for the batch to repack.

diff --git a/midx.c b/midx.c
index a5fb797edeeae46b76f8f926d4b9d40b2236878d..55afc8d72c3bc3f3971d7f055611376125bfbf29 100644 (file)
--- a/midx.c
+++ b/midx.c
@@ -1371,7 +1371,7 @@ static int fill_included_packs_batch(struct repository *r,
 
        free(pack_info);
 
-       if (total_size < batch_size || packs_to_repack < 2)
+       if (packs_to_repack < 2)
                return 1;
 
        return 0;

diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index 7214cab36c0f74ef4c731a189b2490c14a2fa6e4..b05190f500a8fca06517a35d9e26f3867a38daf1 100755 (executable)
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -643,6 +643,7 @@ test_expect_success 'expire respects .keep files' '
 '
 
 test_expect_success 'repack --batch-size=0 repacks everything' '
+       cp -r dup dup2 &&
        (
                cd dup &&
                rm .git/objects/pack/*.keep &&
@@ -662,4 +663,21 @@ test_expect_success 'repack --batch-size=0 repacks everything' '
        )
 '
 
+test_expect_success 'repack --batch-size=<large> repacks everything' '
+       (
+               cd dup2 &&
+               rm .git/objects/pack/*.keep &&
+               ls .git/objects/pack/*idx >idx-list &&
+               test_line_count = 2 idx-list &&
+               git multi-pack-index repack --batch-size=2000000 &&
+               ls .git/objects/pack/*idx >idx-list &&
+               test_line_count = 3 idx-list &&
+               test-tool read-midx .git/objects | grep idx >midx-list &&
+               test_line_count = 3 midx-list &&
+               git multi-pack-index expire &&
+               ls -al .git/objects/pack/*idx >idx-list &&
+               test_line_count = 1 idx-list
+       )
+'
+
 test_done