fetch-pack: exclude blobs when lazy-fetching trees

A partial clone with missing trees can be obtained using "git clone
--filter=tree:none <repo>". In such a repository, when a tree needs to
be lazily fetched, any tree or blob it directly or indirectly references
is fetched as well, regardless of whether the original command required
those objects, or if the local repository already had some of them.

This is because the fetch protocol, which the lazy fetch uses, does not
allow clients to request that only the wanted objects be sent, which
would be the ideal solution. This patch implements a partial solution:
specify the "blob:none" filter, somewhat reducing the fetch payload.

This change has no effect when lazily fetching blobs (due to how filters
work). And if lazily fetching a commit (such repositories are difficult
to construct and is not a use case we support very well, but it is
possible), referenced commits and trees are still fetched - only the
blobs are not fetched.

The necessary code change is done in fetch_pack() instead of somewhere
closer to where the "filter" instruction is written to the wire so that
only one part of the code needs to be changed in order for users of all
protocol versions to benefit from this optimization.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff --git a/fetch-pack.c b/fetch-pack.c
index 973d72f367c9f39c7f4a1e5470d5ac4c33d9f060..79007f996c899272b6ae9044a9029776181f3438 100644 (file)
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -1615,6 +1615,20 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
        if (nr_sought)
                nr_sought = remove_duplicates_in_refs(sought, nr_sought);
 
+       if (args->no_dependents && !args->filter_options.choice) {
+               /*
+                * The protocol does not support requesting that only the
+                * wanted objects be sent, so approximate this by setting a
+                * "blob:none" filter if no filter is already set. This works
+                * for all object types: note that wanted blobs will still be
+                * sent because they are directly specified as a "want".
+                *
+                * NEEDSWORK: Add an option in the protocol to request that
+                * only the wanted objects be sent, and implement it.
+                */
+               parse_list_objects_filter(&args->filter_options, "blob:none");
+       }
+
        if (!ref) {
                packet_flush(fd[1]);
                die(_("no matching remote head"));

diff --git a/fetch-pack.h b/fetch-pack.h
index 5b6e868802b53ca5fa59eb864199274ac538c242..43ec344d95b40c701b2fc703b4b2b1e3ff72a2d2 100644 (file)
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -43,6 +43,13 @@ struct fetch_pack_args {
        unsigned from_promisor:1;
 
        /*
+        * Attempt to fetch only the wanted objects, and not any objects
+        * referred to by them. Due to protocol limitations, extraneous
+        * objects may still be included. (When fetching non-blob
+        * objects, only blobs are excluded; when fetching a blob, the
+        * blob itself will still be sent. The client does not need to
+        * know whether a wanted object is a blob or not.)
+        *
         * If 1, fetch_pack() will also not modify any object flags.
         * This allows fetch_pack() to safely be called by any function,
         * regardless of which object flags it uses (if any).

diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh
index 128130066499feb5bdad705b6fb0ef03bf446fe9..08a0c3651c7977d3c9abd1ab883ec00cb0af9dc3 100755 (executable)
--- a/t/t0410-partial-clone.sh
+++ b/t/t0410-partial-clone.sh
@@ -170,6 +170,47 @@ test_expect_success 'fetching of missing objects' '
        git verify-pack --verbose "$IDX" | grep "$HASH"
 '
 
+test_expect_success 'fetching of missing blobs works' '
+       rm -rf server repo &&
+       test_create_repo server &&
+       test_commit -C server foo &&
+       git -C server repack -a -d --write-bitmap-index &&
+
+       git clone "file://$(pwd)/server" repo &&
+       git hash-object repo/foo.t >blobhash &&
+       rm -rf repo/.git/objects/* &&
+
+       git -C server config uploadpack.allowanysha1inwant 1 &&
+       git -C server config uploadpack.allowfilter 1 &&
+       git -C repo config core.repositoryformatversion 1 &&
+       git -C repo config extensions.partialclone "origin" &&
+
+       git -C repo cat-file -p $(cat blobhash)
+'
+
+test_expect_success 'fetching of missing trees does not fetch blobs' '
+       rm -rf server repo &&
+       test_create_repo server &&
+       test_commit -C server foo &&
+       git -C server repack -a -d --write-bitmap-index &&
+
+       git clone "file://$(pwd)/server" repo &&
+       git -C repo rev-parse foo^{tree} >treehash &&
+       git hash-object repo/foo.t >blobhash &&
+       rm -rf repo/.git/objects/* &&
+
+       git -C server config uploadpack.allowanysha1inwant 1 &&
+       git -C server config uploadpack.allowfilter 1 &&
+       git -C repo config core.repositoryformatversion 1 &&
+       git -C repo config extensions.partialclone "origin" &&
+       git -C repo cat-file -p $(cat treehash) &&
+
+       # Ensure that the tree, but not the blob, is fetched
+       git -C repo rev-list --objects --missing=print $(cat treehash) >objects &&
+       grep "^$(cat treehash)" objects &&
+       grep "^[?]$(cat blobhash)" objects
+'
+
 test_expect_success 'rev-list stops traversal at missing and promised commit' '
        rm -rf repo &&
        test_create_repo repo &&