Documentation/technical/protocol-v2.txt

   1 Git Wire Protocol, Version 2
   2 ============================
   3
   4 This document presents a specification for a version 2 of Git's wire
   5 protocol.  Protocol v2 will improve upon v1 in the following ways:
   6
   7   * Instead of multiple service names, multiple commands will be
   8     supported by a single service
   9   * Easily extendable as capabilities are moved into their own section
  10     of the protocol, no longer being hidden behind a NUL byte and
  11     limited by the size of a pkt-line
  12   * Separate out other information hidden behind NUL bytes (e.g. agent
  13     string as a capability and symrefs can be requested using 'ls-refs')
  14   * Reference advertisement will be omitted unless explicitly requested
  15   * ls-refs command to explicitly request some refs
  16   * Designed with http and stateless-rpc in mind.  With clear flush
  17     semantics the http remote helper can simply act as a proxy
  18
  19 In protocol v2 communication is command oriented.  When first contacting a
  20 server a list of capabilities will advertised.  Some of these capabilities
  21 will be commands which a client can request be executed.  Once a command
  22 has completed, a client can reuse the connection and request that other
  23 commands be executed.
  24
  25 Packet-Line Framing
  26 -------------------
  27
  28 All communication is done using packet-line framing, just as in v1.  See
  29 `Documentation/technical/pack-protocol.txt` and
  30 `Documentation/technical/protocol-common.txt` for more information.
  31
  32 In protocol v2 these special packets will have the following semantics:
  33
  34   * '0000' Flush Packet (flush-pkt) - indicates the end of a message
  35   * '0001' Delimiter Packet (delim-pkt) - separates sections of a message
  36   * '0002' Response End Packet (response-end-pkt) - indicates the end of a
  37     response for stateless connections
  38
  39 Initial Client Request
  40 ----------------------
  41
  42 In general a client can request to speak protocol v2 by sending
  43 `version=2` through the respective side-channel for the transport being
  44 used which inevitably sets `GIT_PROTOCOL`.  More information can be
  45 found in `pack-protocol.txt` and `http-protocol.txt`, as well as the
  46 `GIT_PROTOCOL` definition in `git.txt`. In all cases the
  47 response from the server is the capability advertisement.
  48
  49 Git Transport
  50 ~~~~~~~~~~~~~
  51
  52 When using the git:// transport, you can request to use protocol v2 by
  53 sending "version=2" as an extra parameter:
  54
  55    003egit-upload-pack /project.git\0host=myserver.com\0\0version=2\0
  56
  57 SSH and File Transport
  58 ~~~~~~~~~~~~~~~~~~~~~~
  59
  60 When using either the ssh:// or file:// transport, the GIT_PROTOCOL
  61 environment variable must be set explicitly to include "version=2".
  62 The server may need to be configured to allow this environment variable
  63 to pass.
  64
  65 HTTP Transport
  66 ~~~~~~~~~~~~~~
  67
  68 When using the http:// or https:// transport a client makes a "smart"
  69 info/refs request as described in `http-protocol.txt` and requests that
  70 v2 be used by supplying "version=2" in the `Git-Protocol` header.
  71
  72    C: GET $GIT_URL/info/refs?service=git-upload-pack HTTP/1.0
  73    C: Git-Protocol: version=2
  74
  75 A v2 server would reply:
  76
  77    S: 200 OK
  78    S: <Some headers>
  79    S: ...
  80    S:
  81    S: 000eversion 2\n
  82    S: <capability-advertisement>
  83
  84 Subsequent requests are then made directly to the service
  85 `$GIT_URL/git-upload-pack`. (This works the same for git-receive-pack).
  86
  87 Uses the `--http-backend-info-refs` option to
  88 linkgit:git-upload-pack[1].
  89
  90 The server may need to be configured to pass this header's contents via
  91 the `GIT_PROTOCOL` variable. See the discussion in `git-http-backend.txt`.
  92
  93 Capability Advertisement
  94 ------------------------
  95
  96 A server which decides to communicate (based on a request from a client)
  97 using protocol version 2, notifies the client by sending a version string
  98 in its initial response followed by an advertisement of its capabilities.
  99 Each capability is a key with an optional value.  Clients must ignore all
 100 unknown keys.  Semantics of unknown values are left to the definition of
 101 each key.  Some capabilities will describe commands which can be requested
 102 to be executed by the client.
 103
 104     capability-advertisement = protocol-version
 105                                capability-list
 106                                flush-pkt
 107
 108     protocol-version = PKT-LINE("version 2" LF)
 109     capability-list = *capability
 110     capability = PKT-LINE(key[=value] LF)
 111
 112     key = 1*(ALPHA | DIGIT | "-_")
 113     value = 1*(ALPHA | DIGIT | " -_.,?\/{}[]()<>!@#$%^&*+=:;")
 114
 115 Command Request
 116 ---------------
 117
 118 After receiving the capability advertisement, a client can then issue a
 119 request to select the command it wants with any particular capabilities
 120 or arguments.  There is then an optional section where the client can
 121 provide any command specific parameters or queries.  Only a single
 122 command can be requested at a time.
 123
 124     request = empty-request | command-request
 125     empty-request = flush-pkt
 126     command-request = command
 127                       capability-list
 128                       delim-pkt
 129                       command-args
 130                       flush-pkt
 131     command = PKT-LINE("command=" key LF)
 132     command-args = *command-specific-arg
 133
 134     command-specific-args are packet line framed arguments defined by
 135     each individual command.
 136
 137 The server will then check to ensure that the client's request is
 138 comprised of a valid command as well as valid capabilities which were
 139 advertised.  If the request is valid the server will then execute the
 140 command.  A server MUST wait till it has received the client's entire
 141 request before issuing a response.  The format of the response is
 142 determined by the command being executed, but in all cases a flush-pkt
 143 indicates the end of the response.
 144
 145 When a command has finished, and the client has received the entire
 146 response from the server, a client can either request that another
 147 command be executed or can terminate the connection.  A client may
 148 optionally send an empty request consisting of just a flush-pkt to
 149 indicate that no more requests will be made.
 150
 151 Capabilities
 152 ------------
 153
 154 There are two different types of capabilities: normal capabilities,
 155 which can be used to convey information or alter the behavior of a
 156 request, and commands, which are the core actions that a client wants to
 157 perform (fetch, push, etc).
 158
 159 Protocol version 2 is stateless by default.  This means that all commands
 160 must only last a single round and be stateless from the perspective of the
 161 server side, unless the client has requested a capability indicating that
 162 state should be maintained by the server.  Clients MUST NOT require state
 163 management on the server side in order to function correctly.  This
 164 permits simple round-robin load-balancing on the server side, without
 165 needing to worry about state management.
 166
 167 agent
 168 ~~~~~
 169
 170 The server can advertise the `agent` capability with a value `X` (in the
 171 form `agent=X`) to notify the client that the server is running version
 172 `X`.  The client may optionally send its own agent string by including
 173 the `agent` capability with a value `Y` (in the form `agent=Y`) in its
 174 request to the server (but it MUST NOT do so if the server did not
 175 advertise the agent capability). The `X` and `Y` strings may contain any
 176 printable ASCII characters except space (i.e., the byte range 32 < x <
 177 127), and are typically of the form "package/version" (e.g.,
 178 "git/1.8.3.1"). The agent strings are purely informative for statistics
 179 and debugging purposes, and MUST NOT be used to programmatically assume
 180 the presence or absence of particular features.
 181
 182 ls-refs
 183 ~~~~~~~
 184
 185 `ls-refs` is the command used to request a reference advertisement in v2.
 186 Unlike the current reference advertisement, ls-refs takes in arguments
 187 which can be used to limit the refs sent from the server.
 188
 189 Additional features not supported in the base command will be advertised
 190 as the value of the command in the capability advertisement in the form
 191 of a space separated list of features: "<command>=<feature 1> <feature 2>"
 192
 193 ls-refs takes in the following arguments:
 194
 195     symrefs
 196         In addition to the object pointed by it, show the underlying ref
 197         pointed by it when showing a symbolic ref.
 198     peel
 199         Show peeled tags.
 200     ref-prefix <prefix>
 201         When specified, only references having a prefix matching one of
 202         the provided prefixes are displayed. Multiple instances may be
 203         given, in which case references matching any prefix will be
 204         shown. Note that this is purely for optimization; a server MAY
 205         show refs not matching the prefix if it chooses, and clients
 206         should filter the result themselves.
 207
 208 If the 'unborn' feature is advertised the following argument can be
 209 included in the client's request.
 210
 211     unborn
 212         The server will send information about HEAD even if it is a symref
 213         pointing to an unborn branch in the form "unborn HEAD
 214         symref-target:<target>".
 215
 216 The output of ls-refs is as follows:
 217
 218     output = *ref
 219              flush-pkt
 220     obj-id-or-unborn = (obj-id | "unborn")
 221     ref = PKT-LINE(obj-id-or-unborn SP refname *(SP ref-attribute) LF)
 222     ref-attribute = (symref | peeled)
 223     symref = "symref-target:" symref-target
 224     peeled = "peeled:" obj-id
 225
 226 fetch
 227 ~~~~~
 228
 229 `fetch` is the command used to fetch a packfile in v2.  It can be looked
 230 at as a modified version of the v1 fetch where the ref-advertisement is
 231 stripped out (since the `ls-refs` command fills that role) and the
 232 message format is tweaked to eliminate redundancies and permit easy
 233 addition of future extensions.
 234
 235 Additional features not supported in the base command will be advertised
 236 as the value of the command in the capability advertisement in the form
 237 of a space separated list of features: "<command>=<feature 1> <feature 2>"
 238
 239 A `fetch` request can take the following arguments:
 240
 241     want <oid>
 242         Indicates to the server an object which the client wants to
 243         retrieve.  Wants can be anything and are not limited to
 244         advertised objects.
 245
 246     have <oid>
 247         Indicates to the server an object which the client has locally.
 248         This allows the server to make a packfile which only contains
 249         the objects that the client needs. Multiple 'have' lines can be
 250         supplied.
 251
 252     done
 253         Indicates to the server that negotiation should terminate (or
 254         not even begin if performing a clone) and that the server should
 255         use the information supplied in the request to construct the
 256         packfile.
 257
 258     thin-pack
 259         Request that a thin pack be sent, which is a pack with deltas
 260         which reference base objects not contained within the pack (but
 261         are known to exist at the receiving end). This can reduce the
 262         network traffic significantly, but it requires the receiving end
 263         to know how to "thicken" these packs by adding the missing bases
 264         to the pack.
 265
 266     no-progress
 267         Request that progress information that would normally be sent on
 268         side-band channel 2, during the packfile transfer, should not be
 269         sent.  However, the side-band channel 3 is still used for error
 270         responses.
 271
 272     include-tag
 273         Request that annotated tags should be sent if the objects they
 274         point to are being sent.
 275
 276     ofs-delta
 277         Indicate that the client understands PACKv2 with delta referring
 278         to its base by position in pack rather than by an oid.  That is,
 279         they can read OBJ_OFS_DELTA (aka type 6) in a packfile.
 280
 281 If the 'shallow' feature is advertised the following arguments can be
 282 included in the clients request as well as the potential addition of the
 283 'shallow-info' section in the server's response as explained below.
 284
 285     shallow <oid>
 286         A client must notify the server of all commits for which it only
 287         has shallow copies (meaning that it doesn't have the parents of
 288         a commit) by supplying a 'shallow <oid>' line for each such
 289         object so that the server is aware of the limitations of the
 290         client's history.  This is so that the server is aware that the
 291         client may not have all objects reachable from such commits.
 292
 293     deepen <depth>
 294         Requests that the fetch/clone should be shallow having a commit
 295         depth of <depth> relative to the remote side.
 296
 297     deepen-relative
 298         Requests that the semantics of the "deepen" command be changed
 299         to indicate that the depth requested is relative to the client's
 300         current shallow boundary, instead of relative to the requested
 301         commits.
 302
 303     deepen-since <timestamp>
 304         Requests that the shallow clone/fetch should be cut at a
 305         specific time, instead of depth.  Internally it's equivalent to
 306         doing "git rev-list --max-age=<timestamp>". Cannot be used with
 307         "deepen".
 308
 309     deepen-not <rev>
 310         Requests that the shallow clone/fetch should be cut at a
 311         specific revision specified by '<rev>', instead of a depth.
 312         Internally it's equivalent of doing "git rev-list --not <rev>".
 313         Cannot be used with "deepen", but can be used with
 314         "deepen-since".
 315
 316 If the 'filter' feature is advertised, the following argument can be
 317 included in the client's request:
 318
 319     filter <filter-spec>
 320         Request that various objects from the packfile be omitted
 321         using one of several filtering techniques. These are intended
 322         for use with partial clone and partial fetch operations. See
 323         `rev-list` for possible "filter-spec" values. When communicating
 324         with other processes, senders SHOULD translate scaled integers
 325         (e.g. "1k") into a fully-expanded form (e.g. "1024") to aid
 326         interoperability with older receivers that may not understand
 327         newly-invented scaling suffixes. However, receivers SHOULD
 328         accept the following suffixes: 'k', 'm', and 'g' for 1024,
 329         1048576, and 1073741824, respectively.
 330
 331 If the 'ref-in-want' feature is advertised, the following argument can
 332 be included in the client's request as well as the potential addition of
 333 the 'wanted-refs' section in the server's response as explained below.
 334
 335     want-ref <ref>
 336         Indicates to the server that the client wants to retrieve a
 337         particular ref, where <ref> is the full name of a ref on the
 338         server.
 339
 340 If the 'sideband-all' feature is advertised, the following argument can be
 341 included in the client's request:
 342
 343     sideband-all
 344         Instruct the server to send the whole response multiplexed, not just
 345         the packfile section. All non-flush and non-delim PKT-LINE in the
 346         response (not only in the packfile section) will then start with a byte
 347         indicating its sideband (1, 2, or 3), and the server may send "0005\2"
 348         (a PKT-LINE of sideband 2 with no payload) as a keepalive packet.
 349
 350 If the 'packfile-uris' feature is advertised, the following argument
 351 can be included in the client's request as well as the potential
 352 addition of the 'packfile-uris' section in the server's response as
 353 explained below.
 354
 355     packfile-uris <comma-separated list of protocols>
 356         Indicates to the server that the client is willing to receive
 357         URIs of any of the given protocols in place of objects in the
 358         sent packfile. Before performing the connectivity check, the
 359         client should download from all given URIs. Currently, the
 360         protocols supported are "http" and "https".
 361
 362 If the 'wait-for-done' feature is advertised, the following argument
 363 can be included in the client's request.
 364
 365     wait-for-done
 366         Indicates to the server that it should never send "ready", but
 367         should wait for the client to say "done" before sending the
 368         packfile.
 369
 370 The response of `fetch` is broken into a number of sections separated by
 371 delimiter packets (0001), with each section beginning with its section
 372 header. Most sections are sent only when the packfile is sent.
 373
 374     output = acknowledgements flush-pkt |
 375              [acknowledgments delim-pkt] [shallow-info delim-pkt]
 376              [wanted-refs delim-pkt] [packfile-uris delim-pkt]
 377              packfile flush-pkt
 378
 379     acknowledgments = PKT-LINE("acknowledgments" LF)
 380                       (nak | *ack)
 381                       (ready)
 382     ready = PKT-LINE("ready" LF)
 383     nak = PKT-LINE("NAK" LF)
 384     ack = PKT-LINE("ACK" SP obj-id LF)
 385
 386     shallow-info = PKT-LINE("shallow-info" LF)
 387                    *PKT-LINE((shallow | unshallow) LF)
 388     shallow = "shallow" SP obj-id
 389     unshallow = "unshallow" SP obj-id
 390
 391     wanted-refs = PKT-LINE("wanted-refs" LF)
 392                   *PKT-LINE(wanted-ref LF)
 393     wanted-ref = obj-id SP refname
 394
 395     packfile-uris = PKT-LINE("packfile-uris" LF) *packfile-uri
 396     packfile-uri = PKT-LINE(40*(HEXDIGIT) SP *%x20-ff LF)
 397
 398     packfile = PKT-LINE("packfile" LF)
 399                *PKT-LINE(%x01-03 *%x00-ff)
 400
 401     acknowledgments section
 402         * If the client determines that it is finished with negotiations by
 403           sending a "done" line (thus requiring the server to send a packfile),
 404           the acknowledgments sections MUST be omitted from the server's
 405           response.
 406
 407         * Always begins with the section header "acknowledgments"
 408
 409         * The server will respond with "NAK" if none of the object ids sent
 410           as have lines were common.
 411
 412         * The server will respond with "ACK obj-id" for all of the
 413           object ids sent as have lines which are common.
 414
 415         * A response cannot have both "ACK" lines as well as a "NAK"
 416           line.
 417
 418         * The server will respond with a "ready" line indicating that
 419           the server has found an acceptable common base and is ready to
 420           make and send a packfile (which will be found in the packfile
 421           section of the same response)
 422
 423         * If the server has found a suitable cut point and has decided
 424           to send a "ready" line, then the server can decide to (as an
 425           optimization) omit any "ACK" lines it would have sent during
 426           its response.  This is because the server will have already
 427           determined the objects it plans to send to the client and no
 428           further negotiation is needed.
 429
 430     shallow-info section
 431         * If the client has requested a shallow fetch/clone, a shallow
 432           client requests a fetch or the server is shallow then the
 433           server's response may include a shallow-info section.  The
 434           shallow-info section will be included if (due to one of the
 435           above conditions) the server needs to inform the client of any
 436           shallow boundaries or adjustments to the clients already
 437           existing shallow boundaries.
 438
 439         * Always begins with the section header "shallow-info"
 440
 441         * If a positive depth is requested, the server will compute the
 442           set of commits which are no deeper than the desired depth.
 443
 444         * The server sends a "shallow obj-id" line for each commit whose
 445           parents will not be sent in the following packfile.
 446
 447         * The server sends an "unshallow obj-id" line for each commit
 448           which the client has indicated is shallow, but is no longer
 449           shallow as a result of the fetch (due to its parents being
 450           sent in the following packfile).
 451
 452         * The server MUST NOT send any "unshallow" lines for anything
 453           which the client has not indicated was shallow as a part of
 454           its request.
 455
 456     wanted-refs section
 457         * This section is only included if the client has requested a
 458           ref using a 'want-ref' line and if a packfile section is also
 459           included in the response.
 460
 461         * Always begins with the section header "wanted-refs".
 462
 463         * The server will send a ref listing ("<oid> <refname>") for
 464           each reference requested using 'want-ref' lines.
 465
 466         * The server MUST NOT send any refs which were not requested
 467           using 'want-ref' lines.
 468
 469     packfile-uris section
 470         * This section is only included if the client sent
 471           'packfile-uris' and the server has at least one such URI to
 472           send.
 473
 474         * Always begins with the section header "packfile-uris".
 475
 476         * For each URI the server sends, it sends a hash of the pack's
 477           contents (as output by git index-pack) followed by the URI.
 478
 479         * The hashes are 40 hex characters long. When Git upgrades to a new
 480           hash algorithm, this might need to be updated. (It should match
 481           whatever index-pack outputs after "pack\t" or "keep\t".
 482
 483     packfile section
 484         * This section is only included if the client has sent 'want'
 485           lines in its request and either requested that no more
 486           negotiation be done by sending 'done' or if the server has
 487           decided it has found a sufficient cut point to produce a
 488           packfile.
 489
 490         * Always begins with the section header "packfile"
 491
 492         * The transmission of the packfile begins immediately after the
 493           section header
 494
 495         * The data transfer of the packfile is always multiplexed, using
 496           the same semantics of the 'side-band-64k' capability from
 497           protocol version 1.  This means that each packet, during the
 498           packfile data stream, is made up of a leading 4-byte pkt-line
 499           length (typical of the pkt-line format), followed by a 1-byte
 500           stream code, followed by the actual data.
 501
 502           The stream code can be one of:
 503                 1 - pack data
 504                 2 - progress messages
 505                 3 - fatal error message just before stream aborts
 506
 507 server-option
 508 ~~~~~~~~~~~~~
 509
 510 If advertised, indicates that any number of server specific options can be
 511 included in a request.  This is done by sending each option as a
 512 "server-option=<option>" capability line in the capability-list section of
 513 a request.
 514
 515 The provided options must not contain a NUL or LF character.
 516
 517  object-format
 518 ~~~~~~~~~~~~~~~
 519
 520 The server can advertise the `object-format` capability with a value `X` (in the
 521 form `object-format=X`) to notify the client that the server is able to deal
 522 with objects using hash algorithm X.  If not specified, the server is assumed to
 523 only handle SHA-1.  If the client would like to use a hash algorithm other than
 524 SHA-1, it should specify its object-format string.
 525
 526 session-id=<session id>
 527 ~~~~~~~~~~~~~~~~~~~~~~~
 528
 529 The server may advertise a session ID that can be used to identify this process
 530 across multiple requests. The client may advertise its own session ID back to
 531 the server as well.
 532
 533 Session IDs should be unique to a given process. They must fit within a
 534 packet-line, and must not contain non-printable or whitespace characters. The
 535 current implementation uses trace2 session IDs (see
 536 link:api-trace2.html[api-trace2] for details), but this may change and users of
 537 the session ID should not rely on this fact.
 538
 539 object-info
 540 ~~~~~~~~~~~
 541
 542 `object-info` is the command to retrieve information about one or more objects.
 543 Its main purpose is to allow a client to make decisions based on this
 544 information without having to fully fetch objects. Object size is the only
 545 information that is currently supported.
 546
 547 An `object-info` request takes the following arguments:
 548
 549         size
 550         Requests size information to be returned for each listed object id.
 551
 552         oid <oid>
 553         Indicates to the server an object which the client wants to obtain
 554         information for.
 555
 556 The response of `object-info` is a list of the requested object ids
 557 and associated requested information, each separated by a single space.
 558
 559         output = info flush-pkt
 560
 561         info = PKT-LINE(attrs) LF)
 562                 *PKT-LINE(obj-info LF)
 563
 564         attrs = attr | attrs SP attrs
 565
 566         attr = "size"
 567
 568         obj-info = obj-id SP obj-size