fetch, upload-pack: --deepen=N extends shallow boundary by N commits

In git-fetch, --depth argument is always relative with the latest
remote refs. This makes it a bit difficult to cover this use case,
where the user wants to make the shallow history, say 3 levels
deeper. It would work if remote refs have not moved yet, but nobody
can guarantee that, especially when that use case is performed a
couple months after the last clone or "git fetch --depth". Also,
modifying shallow boundary using --depth does not work well with
clones created by --since or --not.

This patch fixes that. A new argument --deepen=<N> will add <N> more (*)
parent commits to the current history regardless of where remote refs
are.

Have/Want negotiation is still respected. So if remote refs move, the
server will send two chunks: one between "have" and "want" and another
to extend shallow history. In theory, the client could send no "want"s
in order to get the second chunk only. But the protocol does not allow
that. Either you send no want lines, which means ls-remote; or you
have to send at least one want line that carries deep-relative to the
server..

The main work was done by Dongcan Jiang. I fixed it up here and there.
And of course all the bugs belong to me.

(*) We could even support --deepen=<N> where <N> is negative. In that
case we can cut some history from the shallow clone. This operation
(and --depth=<shorter depth>) does not require interaction with remote
side (and more complicated to implement as a result).

Helped-by: Duy Nguyen <pclouds@gmail.com>
Helped-by: Eric Sunshine <sunshine@sunshineco.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Dongcan Jiang <dongcan.jiang@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 7aa1285fc92ea7a0a39d708614ceb712d48776be..3b91f1528e01c91f2b278dcae87db152418ccc75 100644 (file)
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -14,6 +14,11 @@
        linkgit:git-clone[1]), deepen or shorten the history to the specified
        number of commits. Tags for the deepened commits are not fetched.
 
+--deepen=<depth>::
+       Similar to --depth, except it specifies the number of commits
+       from the current shallow boundary instead of from the tip of
+       each remote branch history.
+
 --shallow-since=<date>::
        Deepen or shorten the history of a shallow repository to
        include all reachable commits after <date>.

diff --git a/Documentation/git-fetch-pack.txt b/Documentation/git-fetch-pack.txt
index 4d15b0469c6ef6dfe391e6f22dc6355e9e7e7f22..c20958f58fac0da0b9b0be5317941d66ab59505d 100644 (file)
--- a/Documentation/git-fetch-pack.txt
+++ b/Documentation/git-fetch-pack.txt
@@ -96,6 +96,11 @@ be in a separate packet, and the list must end with a flush packet.
        exclude commits reachable from a specified remote branch or tag.
        This option can be specified multiple times.
 
+--deepen-relative::
+       Argument --depth specifies the number of commits from the
+       current shallow boundary instead of from the tip of each
+       remote branch history.
+
 --no-progress::
        Do not show the progress.
 

diff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt
index 75bb6381f674003220e08f58530bdda1b971836e..6fca2687422be78106c6bc3ec534a1c046c02a14 100644 (file)
--- a/Documentation/gitremote-helpers.txt
+++ b/Documentation/gitremote-helpers.txt
@@ -422,6 +422,10 @@ set by Git if the remote helper has the 'option' capability.
        Deepens the history of a shallow repository excluding ref.
        Multiple options add up.
 
+'option deepen-relative {'true'|'false'}::
+       Deepens the history of a shallow repository relative to
+       current boundary. Only valid when used with "option depth".
+
 'option followtags' {'true'|'false'}::
        If enabled the helper should automatically fetch annotated
        tag objects if the object the tag points at was transferred

diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt
index 0e6b57df75b839aba5aa2a2acedf42477eb4e434..4fd6dccdb52770c1ec0e57229ae87faa33e37eae 100644 (file)
--- a/Documentation/technical/protocol-capabilities.txt
+++ b/Documentation/technical/protocol-capabilities.txt
@@ -197,6 +197,13 @@ specific revision, instead of depth. Internally it's equivalent of
 doing "rev-list --not <rev>" on the server side. "deepen-not"
 cannot be used with "deepen", but can be used with "deepen-since".
 
+deepen-relative
+---------------
+
+If this capability is requested by the client, the semantics of
+"deepen" command is changed. The "depth" argument is the depth from
+the current shallow boundary, instead of the depth from remote refs.
+
 no-progress
 -----------
 

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 07570bee2d367937cba1023156f64b04b9095bdd..82653487f8ac5ff1e4f44532f2c0bb278f65b554 100644 (file)
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -113,6 +113,10 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)
                        string_list_append(&deepen_not, arg);
                        continue;
                }
+               if (!strcmp(arg, "--deepen-relative")) {
+                       args.deepen_relative = 1;
+                       continue;
+               }
                if (!strcmp("--no-progress", arg)) {
                        args.no_progress = 1;
                        continue;

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 147504dd51974cf7eb3881948a3b46c40219027e..7b0ea1c86ea8c3528d109faa4b9210c0d9cab875 100644 (file)
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -34,7 +34,7 @@ static int fetch_prune_config = -1; /* unspecified */
 static int prune = -1; /* unspecified */
 #define PRUNE_BY_DEFAULT 0 /* do we prune by default? */
 
-static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity;
+static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity, deepen_relative;
 static int progress = -1, recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
 static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen;
 static int max_children = 1;
@@ -121,6 +121,8 @@ static struct option builtin_fetch_options[] = {
                   N_("deepen history of shallow repository based on time")),
        OPT_STRING_LIST(0, "shallow-exclude", &deepen_not, N_("revision"),
                        N_("deepen history of shallow clone by excluding rev")),
+       OPT_INTEGER(0, "deepen", &deepen_relative,
+                   N_("deepen history of shallow clone")),
        { OPTION_SET_INT, 0, "unshallow", &unshallow, NULL,
                   N_("convert to a complete repository"),
                   PARSE_OPT_NONEG | PARSE_OPT_NOARG, NULL, 1 },
@@ -881,6 +883,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)
        if (deepen && deepen_not.nr)
                set_option(transport, TRANS_OPT_DEEPEN_NOT,
                           (const char *)&deepen_not);
+       if (deepen_relative)
+               set_option(transport, TRANS_OPT_DEEPEN_RELATIVE, "yes");
        if (update_shallow)
                set_option(transport, TRANS_OPT_UPDATE_SHALLOW, "yes");
        return transport;
@@ -906,6 +910,7 @@ static void backfill_tags(struct transport *transport, struct ref *ref_map)
 
        transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, NULL);
        transport_set_option(transport, TRANS_OPT_DEPTH, "0");
+       transport_set_option(transport, TRANS_OPT_DEEPEN_RELATIVE, NULL);
        fetch_refs(transport, ref_map);
 
        if (gsecondary) {
@@ -1177,6 +1182,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
        argc = parse_options(argc, argv, prefix,
                             builtin_fetch_options, builtin_fetch_usage, 0);
 
+       if (deepen_relative) {
+               if (deepen_relative < 0)
+                       die(_("Negative depth in --deepen is not supported"));
+               if (depth)
+                       die(_("--deepen and --depth are mutually exclusive"));
+               depth = xstrfmt("%d", deepen_relative);
+       }
        if (unshallow) {
                if (depth)
                        die(_("--depth and --unshallow cannot be used together"));

diff --git a/fetch-pack.c b/fetch-pack.c
index ad7d00f09c694248c4e1db6637ff772ef8c15f14..e2a235f536a2448feb9a396019ec6da5e3b9d609 100644 (file)
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -324,6 +324,7 @@ static int find_common(struct fetch_pack_args *args,
                        if (no_done)            strbuf_addstr(&c, " no-done");
                        if (use_sideband == 2)  strbuf_addstr(&c, " side-band-64k");
                        if (use_sideband == 1)  strbuf_addstr(&c, " side-band");
+                       if (args->deepen_relative) strbuf_addstr(&c, " deepen-relative");
                        if (args->use_thin_pack) strbuf_addstr(&c, " thin-pack");
                        if (args->no_progress)   strbuf_addstr(&c, " no-progress");
                        if (args->include_tag)   strbuf_addstr(&c, " include-tag");
@@ -883,6 +884,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,
                deepen_not_ok = 1;
        else if (args->deepen_not)
                die(_("Server does not support --shallow-exclude"));
+       if (!server_supports("deepen-relative") && args->deepen_relative)
+               die(_("Server does not support --deepen"));
 
        if (everything_local(args, &ref, sought, nr_sought)) {
                packet_flush(fd[1]);

diff --git a/fetch-pack.h b/fetch-pack.h
index 144301fbcd2f43263927856041d3dd6e4d7d3e95..c912e3d321c6a80952dbcd266786d1042a49e579 100644 (file)
--- a/fetch-pack.h
+++ b/fetch-pack.h
@@ -12,6 +12,7 @@ struct fetch_pack_args {
        int depth;
        const char *deepen_since;
        const struct string_list *deepen_not;
+       unsigned deepen_relative:1;
        unsigned quiet:1;
        unsigned keep_pack:1;
        unsigned lock_pack:1;

diff --git a/remote-curl.c b/remote-curl.c
index 1406e6a4eb25c55da02822658af3cfc5b0c1c197..d56412deb14237fcb4c5dca1f11cda1ae8f4fcdf 100644 (file)
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -30,7 +30,8 @@ struct options {
                dry_run : 1,
                thin : 1,
                /* One of the SEND_PACK_PUSH_CERT_* constants. */
-               push_cert : 2;
+               push_cert : 2,
+               deepen_relative : 1;
 };
 static struct options options;
 static struct string_list cas_options = STRING_LIST_INIT_DUP;
@@ -70,6 +71,15 @@ static int set_option(const char *name, const char *value)
                string_list_append(&options.deepen_not, value);
                return 0;
        }
+       else if (!strcmp(name, "deepen-relative")) {
+               if (!strcmp(value, "true"))
+                       options.deepen_relative = 1;
+               else if (!strcmp(value, "false"))
+                       options.deepen_relative = 0;
+               else
+                       return -1;
+               return 0;
+       }
        else if (!strcmp(name, "followtags")) {
                if (!strcmp(value, "true"))
                        options.followtags = 1;
@@ -761,6 +771,8 @@ static int fetch_git(struct discovery *heads,
        for (i = 0; i < options.deepen_not.nr; i++)
                argv_array_pushf(&args, "--shallow-exclude=%s",
                                 options.deepen_not.items[i].string);
+       if (options.deepen_relative && options.depth)
+               argv_array_push(&args, "--deepen-relative");
        argv_array_push(&args, url.buf);
 
        for (i = 0; i < nr_heads; i++) {

diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index 145b3707685c1751d0a161c73c077dfe6dad01fc..a908036be72808c4bc2f17f1b43dc0b6e96cfc73 100755 (executable)
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -682,4 +682,27 @@ test_expect_success 'fetch exclude tag one' '
        test_cmp expected actual
 '
 
+test_expect_success 'fetching deepen' '
+       test_create_repo shallow-deepen &&
+       (
+       cd shallow-deepen &&
+       test_commit one &&
+       test_commit two &&
+       test_commit three &&
+       git clone --depth 1 "file://$(pwd)/." deepen &&
+       test_commit four &&
+       git -C deepen log --pretty=tformat:%s master >actual &&
+       echo three >expected &&
+       test_cmp expected actual &&
+       git -C deepen fetch --deepen=1 &&
+       git -C deepen log --pretty=tformat:%s origin/master >actual &&
+       cat >expected <<-\EOF &&
+       four
+       three
+       two
+       EOF
+       test_cmp expected actual
+       )
+'
+
 test_done

diff --git a/t/t5539-fetch-http-shallow.sh b/t/t5539-fetch-http-shallow.sh
index 8e38c1be659fd7d19ca758042ab5683df91d983e..5fbf67c4468897184f46ef9ea151f1dd6ad1d8d0 100755 (executable)
--- a/t/t5539-fetch-http-shallow.sh
+++ b/t/t5539-fetch-http-shallow.sh
@@ -120,5 +120,31 @@ test_expect_success 'fetch exclude tag one' '
        test_cmp expected actual
 '
 
+test_expect_success 'fetching deepen' '
+       test_create_repo shallow-deepen &&
+       (
+       cd shallow-deepen &&
+       test_commit one &&
+       test_commit two &&
+       test_commit three &&
+       mv .git "$HTTPD_DOCUMENT_ROOT_PATH/shallow-deepen.git" &&
+       git clone --depth 1 $HTTPD_URL/smart/shallow-deepen.git deepen &&
+       mv "$HTTPD_DOCUMENT_ROOT_PATH/shallow-deepen.git" .git &&
+       test_commit four &&
+       git -C deepen log --pretty=tformat:%s master >actual &&
+       echo three >expected &&
+       test_cmp expected actual &&
+       mv .git "$HTTPD_DOCUMENT_ROOT_PATH/shallow-deepen.git" &&
+       git -C deepen fetch --deepen=1 &&
+       git -C deepen log --pretty=tformat:%s origin/master >actual &&
+       cat >expected <<-\EOF &&
+       four
+       three
+       two
+       EOF
+       test_cmp expected actual
+       )
+'
+
 stop_httpd
 test_done

diff --git a/transport-helper.c b/transport-helper.c
index cc1a396c25e53e75f5914e4ae753a21a18dce304..a5cdd7730ceeb763bb9f08096f3b2d8e657e204a 100644 (file)
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -258,6 +258,7 @@ static const char *boolean_options[] = {
        TRANS_OPT_THIN,
        TRANS_OPT_KEEP,
        TRANS_OPT_FOLLOWTAGS,
+       TRANS_OPT_DEEPEN_RELATIVE
        };
 
 static int strbuf_set_helper_option(struct helper_data *data,

diff --git a/transport.c b/transport.c
index 3e6f3aa1175f81a9cde32488c6d4be9b27dacbd3..3e76a9af4d58fba352bcb89f0ea158f0b24b270a 100644 (file)
--- a/transport.c
+++ b/transport.c
@@ -157,6 +157,9 @@ static int set_git_option(struct git_transport_options *opts,
        } else if (!strcmp(name, TRANS_OPT_DEEPEN_NOT)) {
                opts->deepen_not = (const struct string_list *)value;
                return 0;
+       } else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {
+               opts->deepen_relative = !!value;
+               return 0;
        }
        return 1;
 }
@@ -213,6 +216,7 @@ static int fetch_refs_via_pack(struct transport *transport,
        args.depth = data->options.depth;
        args.deepen_since = data->options.deepen_since;
        args.deepen_not = data->options.deepen_not;
+       args.deepen_relative = data->options.deepen_relative;
        args.check_self_contained_and_connected =
                data->options.check_self_contained_and_connected;
        args.cloning = transport->cloning;

diff --git a/transport.h b/transport.h
index ab61932b5b3d0632a56d8d348bc27c2ac8a0b903..bdc35181c303a7d85083583a6f967e87089cd653 100644 (file)
--- a/transport.h
+++ b/transport.h
@@ -14,6 +14,7 @@ struct git_transport_options {
        unsigned check_self_contained_and_connected : 1;
        unsigned self_contained_and_connected : 1;
        unsigned update_shallow : 1;
+       unsigned deepen_relative : 1;
        int depth;
        const char *deepen_since;
        const struct string_list *deepen_not;
@@ -181,6 +182,9 @@ int transport_restrict_protocols(void);
 /* Limit the depth of the fetch based on revs if not null */
 #define TRANS_OPT_DEEPEN_NOT "deepen-not"
 
+/* Limit the deepen of the fetch if not null */
+#define TRANS_OPT_DEEPEN_RELATIVE "deepen-relative"
+
 /* Aggressively fetch annotated tags if possible */
 #define TRANS_OPT_FOLLOWTAGS "followtags"
 

diff --git a/upload-pack.c b/upload-pack.c
index 3227df872c71f819f52fec216f46f1235eae471d..e40d15adaff6d164d72156ebddb9b4f152d92d96 100644 (file)
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -32,6 +32,7 @@ static const char upload_pack_usage[] = "git upload-pack [--strict] [--timeout=<
 
 static unsigned long oldest_have;
 
+static int deepen_relative;
 static int multi_ack;
 static int no_done;
 static int use_thin_pack, use_ofs_delta, use_include_tag;
@@ -674,7 +675,8 @@ static void send_unshallow(const struct object_array *shallows)
        }
 }
 
-static void deepen(int depth, const struct object_array *shallows)
+static void deepen(int depth, int deepen_relative,
+                  struct object_array *shallows)
 {
        if (depth == INFINITE_DEPTH && !is_repository_shallow()) {
                int i;
@@ -683,6 +685,17 @@ static void deepen(int depth, const struct object_array *shallows)
                        struct object *object = shallows->objects[i].item;
                        object->flags |= NOT_SHALLOW;
                }
+       } else if (deepen_relative) {
+               struct object_array reachable_shallows = OBJECT_ARRAY_INIT;
+               struct commit_list *result;
+
+               get_reachable_list(shallows, &reachable_shallows);
+               result = get_shallow_commits(&reachable_shallows,
+                                            depth + 1,
+                                            SHALLOW, NOT_SHALLOW);
+               send_shallow(result);
+               free_commit_list(result);
+               object_array_clear(&reachable_shallows);
        } else {
                struct commit_list *result;
 
@@ -779,6 +792,8 @@ static void receive_needs(void)
 
                features = arg + 40;
 
+               if (parse_feature_request(features, "deepen-relative"))
+                       deepen_relative = 1;
                if (parse_feature_request(features, "multi_ack_detailed"))
                        multi_ack = 2;
                else if (parse_feature_request(features, "multi_ack"))
@@ -828,7 +843,7 @@ static void receive_needs(void)
        if (depth > 0 && deepen_rev_list)
                die("git upload-pack: deepen and deepen-since (or deepen-not) cannot be used together");
        if (depth > 0)
-               deepen(depth, &shallows);
+               deepen(depth, deepen_relative, &shallows);
        else if (deepen_rev_list) {
                struct argv_array av = ARGV_ARRAY_INIT;
                int i;
@@ -899,8 +914,8 @@ static int send_ref(const char *refname, const struct object_id *oid,
                    int flag, void *cb_data)
 {
        static const char *capabilities = "multi_ack thin-pack side-band"
-               " side-band-64k ofs-delta shallow deepen-since deepen-not no-progress"
-               " include-tag multi_ack_detailed";
+               " side-band-64k ofs-delta shallow deepen-since deepen-not"
+               " deepen-relative no-progress include-tag multi_ack_detailed";
        const char *refname_nons = strip_namespace(refname);
        struct object_id peeled;