Documentation/technical/commit-graph-format.txt

   1 Git commit graph format
   2 =======================
   3
   4 The Git commit graph stores a list of commit OIDs and some associated
   5 metadata, including:
   6
   7 - The generation number of the commit. Commits with no parents have
   8   generation number 1; commits with parents have generation number
   9   one more than the maximum generation number of its parents. We
  10   reserve zero as special, and can be used to mark a generation
  11   number invalid or as "not computed".
  12
  13 - The root tree OID.
  14
  15 - The commit date.
  16
  17 - The parents of the commit, stored using positional references within
  18   the graph file.
  19
  20 - The Bloom filter of the commit carrying the paths that were changed between
  21   the commit and its first parent, if requested.
  22
  23 These positional references are stored as unsigned 32-bit integers
  24 corresponding to the array position within the list of commit OIDs. Due
  25 to some special constants we use to track parents, we can store at most
  26 (1 << 30) + (1 << 29) + (1 << 28) - 1 (around 1.8 billion) commits.
  27
  28 == Commit graph files have the following format:
  29
  30 In order to allow extensions that add extra data to the graph, we organize
  31 the body into "chunks" and provide a binary lookup table at the beginning
  32 of the body. The header includes certain values, such as number of chunks
  33 and hash type.
  34
  35 All 4-byte numbers are in network order.
  36
  37 HEADER:
  38
  39   4-byte signature:
  40       The signature is: {'C', 'G', 'P', 'H'}
  41
  42   1-byte version number:
  43       Currently, the only valid version is 1.
  44
  45   1-byte Hash Version (1 = SHA-1)
  46       We infer the hash length (H) from this value.
  47
  48   1-byte number (C) of "chunks"
  49
  50   1-byte number (B) of base commit-graphs
  51       We infer the length (H*B) of the Base Graphs chunk
  52       from this value.
  53
  54 CHUNK LOOKUP:
  55
  56   (C + 1) * 12 bytes listing the table of contents for the chunks:
  57       First 4 bytes describe the chunk id. Value 0 is a terminating label.
  58       Other 8 bytes provide the byte-offset in current file for chunk to
  59       start. (Chunks are ordered contiguously in the file, so you can infer
  60       the length using the next chunk position if necessary.) Each chunk
  61       ID appears at most once.
  62
  63   The remaining data in the body is described one chunk at a time, and
  64   these chunks may be given in any order. Chunks are required unless
  65   otherwise specified.
  66
  67 CHUNK DATA:
  68
  69   OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)
  70       The ith entry, F[i], stores the number of OIDs with first
  71       byte at most i. Thus F[255] stores the total
  72       number of commits (N).
  73
  74   OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)
  75       The OIDs for all commits in the graph, sorted in ascending order.
  76
  77   Commit Data (ID: {'C', 'D', 'A', 'T' }) (N * (H + 16) bytes)
  78     * The first H bytes are for the OID of the root tree.
  79     * The next 8 bytes are for the positions of the first two parents
  80       of the ith commit. Stores value 0x7000000 if no parent in that
  81       position. If there are more than two parents, the second value
  82       has its most-significant bit on and the other bits store an array
  83       position into the Extra Edge List chunk.
  84     * The next 8 bytes store the generation number of the commit and
  85       the commit time in seconds since EPOCH. The generation number
  86       uses the higher 30 bits of the first 4 bytes, while the commit
  87       time uses the 32 bits of the second 4 bytes, along with the lowest
  88       2 bits of the lowest byte, storing the 33rd and 34th bit of the
  89       commit time.
  90
  91   Extra Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]
  92       This list of 4-byte values store the second through nth parents for
  93       all octopus merges. The second parent value in the commit data stores
  94       an array position within this list along with the most-significant bit
  95       on. Starting at that array position, iterate through this list of commit
  96       positions for the parents until reaching a value with the most-significant
  97       bit on. The other bits correspond to the position of the last parent.
  98
  99   Bloom Filter Index (ID: {'B', 'I', 'D', 'X'}) (N * 4 bytes) [Optional]
 100     * The ith entry, BIDX[i], stores the number of bytes in all Bloom filters
 101       from commit 0 to commit i (inclusive) in lexicographic order. The Bloom
 102       filter for the i-th commit spans from BIDX[i-1] to BIDX[i] (plus header
 103       length), where BIDX[-1] is 0.
 104     * The BIDX chunk is ignored if the BDAT chunk is not present.
 105
 106   Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]
 107     * It starts with header consisting of three unsigned 32-bit integers:
 108       - Version of the hash algorithm being used. We currently only support
 109         value 1 which corresponds to the 32-bit version of the murmur3 hash
 110         implemented exactly as described in
 111         https://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double
 112         hashing technique using seed values 0x293ae76f and 0x7e646e2 as
 113         described in https://doi.org/10.1007/978-3-540-30494-4_26 "Bloom Filters
 114         in Probabilistic Verification"
 115       - The number of times a path is hashed and hence the number of bit positions
 116               that cumulatively determine whether a file is present in the commit.
 117       - The minimum number of bits 'b' per entry in the Bloom filter. If the filter
 118               contains 'n' entries, then the filter size is the minimum number of 64-bit
 119               words that contain n*b bits.
 120     * The rest of the chunk is the concatenation of all the computed Bloom
 121       filters for the commits in lexicographic order.
 122     * Note: Commits with no changes or more than 512 changes have Bloom filters
 123       of length zero.
 124     * The BDAT chunk is present if and only if BIDX is present.
 125
 126   Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]
 127       This list of H-byte hashes describe a set of B commit-graph files that
 128       form a commit-graph chain. The graph position for the ith commit in this
 129       file's OID Lookup chunk is equal to i plus the number of commits in all
 130       base graphs.  If B is non-zero, this chunk must exist.
 131
 132 TRAILER:
 133
 134         H-byte HASH-checksum of all of the above.