]> git.ipfire.org Git - thirdparty/git.git/blob - Documentation/technical/index-format.txt
clone: allow "--bare" with "-o"
[thirdparty/git.git] / Documentation / technical / index-format.txt
1 Git index format
2 ================
3
4 == The Git index file has the following format
5
6 All binary numbers are in network byte order.
7 In a repository using the traditional SHA-1, checksums and object IDs
8 (object names) mentioned below are all computed using SHA-1. Similarly,
9 in SHA-256 repositories, these values are computed using SHA-256.
10 Version 2 is described here unless stated otherwise.
11
12 - A 12-byte header consisting of
13
14 4-byte signature:
15 The signature is { 'D', 'I', 'R', 'C' } (stands for "dircache")
16
17 4-byte version number:
18 The current supported versions are 2, 3 and 4.
19
20 32-bit number of index entries.
21
22 - A number of sorted index entries (see below).
23
24 - Extensions
25
26 Extensions are identified by signature. Optional extensions can
27 be ignored if Git does not understand them.
28
29 4-byte extension signature. If the first byte is 'A'..'Z' the
30 extension is optional and can be ignored.
31
32 32-bit size of the extension
33
34 Extension data
35
36 - Hash checksum over the content of the index file before this checksum.
37
38 == Index entry
39
40 Index entries are sorted in ascending order on the name field,
41 interpreted as a string of unsigned bytes (i.e. memcmp() order, no
42 localization, no special casing of directory separator '/'). Entries
43 with the same name are sorted by their stage field.
44
45 An index entry typically represents a file. However, if sparse-checkout
46 is enabled in cone mode (`core.sparseCheckoutCone` is enabled) and the
47 `extensions.sparseIndex` extension is enabled, then the index may
48 contain entries for directories outside of the sparse-checkout definition.
49 These entries have mode `040000`, include the `SKIP_WORKTREE` bit, and
50 the path ends in a directory separator.
51
52 32-bit ctime seconds, the last time a file's metadata changed
53 this is stat(2) data
54
55 32-bit ctime nanosecond fractions
56 this is stat(2) data
57
58 32-bit mtime seconds, the last time a file's data changed
59 this is stat(2) data
60
61 32-bit mtime nanosecond fractions
62 this is stat(2) data
63
64 32-bit dev
65 this is stat(2) data
66
67 32-bit ino
68 this is stat(2) data
69
70 32-bit mode, split into (high to low bits)
71
72 4-bit object type
73 valid values in binary are 1000 (regular file), 1010 (symbolic link)
74 and 1110 (gitlink)
75
76 3-bit unused
77
78 9-bit unix permission. Only 0755 and 0644 are valid for regular files.
79 Symbolic links and gitlinks have value 0 in this field.
80
81 32-bit uid
82 this is stat(2) data
83
84 32-bit gid
85 this is stat(2) data
86
87 32-bit file size
88 This is the on-disk size from stat(2), truncated to 32-bit.
89
90 Object name for the represented object
91
92 A 16-bit 'flags' field split into (high to low bits)
93
94 1-bit assume-valid flag
95
96 1-bit extended flag (must be zero in version 2)
97
98 2-bit stage (during merge)
99
100 12-bit name length if the length is less than 0xFFF; otherwise 0xFFF
101 is stored in this field.
102
103 (Version 3 or later) A 16-bit field, only applicable if the
104 "extended flag" above is 1, split into (high to low bits).
105
106 1-bit reserved for future
107
108 1-bit skip-worktree flag (used by sparse checkout)
109
110 1-bit intent-to-add flag (used by "git add -N")
111
112 13-bit unused, must be zero
113
114 Entry path name (variable length) relative to top level directory
115 (without leading slash). '/' is used as path separator. The special
116 path components ".", ".." and ".git" (without quotes) are disallowed.
117 Trailing slash is also disallowed.
118
119 The exact encoding is undefined, but the '.' and '/' characters
120 are encoded in 7-bit ASCII and the encoding cannot contain a NUL
121 byte (iow, this is a UNIX pathname).
122
123 (Version 4) In version 4, the entry path name is prefix-compressed
124 relative to the path name for the previous entry (the very first
125 entry is encoded as if the path name for the previous entry is an
126 empty string). At the beginning of an entry, an integer N in the
127 variable width encoding (the same encoding as the offset is encoded
128 for OFS_DELTA pack entries; see pack-format.txt) is stored, followed
129 by a NUL-terminated string S. Removing N bytes from the end of the
130 path name for the previous entry, and replacing it with the string S
131 yields the path name for this entry.
132
133 1-8 nul bytes as necessary to pad the entry to a multiple of eight bytes
134 while keeping the name NUL-terminated.
135
136 (Version 4) In version 4, the padding after the pathname does not
137 exist.
138
139 Interpretation of index entries in split index mode is completely
140 different. See below for details.
141
142 == Extensions
143
144 === Cache tree
145
146 Since the index does not record entries for directories, the cache
147 entries cannot describe tree objects that already exist in the object
148 database for regions of the index that are unchanged from an existing
149 commit. The cache tree extension stores a recursive tree structure that
150 describes the trees that already exist and completely match sections of
151 the cache entries. This speeds up tree object generation from the index
152 for a new commit by only computing the trees that are "new" to that
153 commit. It also assists when comparing the index to another tree, such
154 as `HEAD^{tree}`, since sections of the index can be skipped when a tree
155 comparison demonstrates equality.
156
157 The recursive tree structure uses nodes that store a number of cache
158 entries, a list of subnodes, and an object ID (OID). The OID references
159 the existing tree for that node, if it is known to exist. The subnodes
160 correspond to subdirectories that themselves have cache tree nodes. The
161 number of cache entries corresponds to the number of cache entries in
162 the index that describe paths within that tree's directory.
163
164 The extension tracks the full directory structure in the cache tree
165 extension, but this is generally smaller than the full cache entry list.
166
167 When a path is updated in index, Git invalidates all nodes of the
168 recursive cache tree corresponding to the parent directories of that
169 path. We store these tree nodes as being "invalid" by using "-1" as the
170 number of cache entries. Invalid nodes still store a span of index
171 entries, allowing Git to focus its efforts when reconstructing a full
172 cache tree.
173
174 The signature for this extension is { 'T', 'R', 'E', 'E' }.
175
176 A series of entries fill the entire extension; each of which
177 consists of:
178
179 - NUL-terminated path component (relative to its parent directory);
180
181 - ASCII decimal number of entries in the index that is covered by the
182 tree this entry represents (entry_count);
183
184 - A space (ASCII 32);
185
186 - ASCII decimal number that represents the number of subtrees this
187 tree has;
188
189 - A newline (ASCII 10); and
190
191 - Object name for the object that would result from writing this span
192 of index as a tree.
193
194 An entry can be in an invalidated state and is represented by having
195 a negative number in the entry_count field. In this case, there is no
196 object name and the next entry starts immediately after the newline.
197 When writing an invalid entry, -1 should always be used as entry_count.
198
199 The entries are written out in the top-down, depth-first order. The
200 first entry represents the root level of the repository, followed by the
201 first subtree--let's call this A--of the root level (with its name
202 relative to the root level), followed by the first subtree of A (with
203 its name relative to A), and so on. The specified number of subtrees
204 indicates when the current level of the recursive stack is complete.
205
206 === Resolve undo
207
208 A conflict is represented in the index as a set of higher stage entries.
209 When a conflict is resolved (e.g. with "git add path"), these higher
210 stage entries will be removed and a stage-0 entry with proper resolution
211 is added.
212
213 When these higher stage entries are removed, they are saved in the
214 resolve undo extension, so that conflicts can be recreated (e.g. with
215 "git checkout -m"), in case users want to redo a conflict resolution
216 from scratch.
217
218 The signature for this extension is { 'R', 'E', 'U', 'C' }.
219
220 A series of entries fill the entire extension; each of which
221 consists of:
222
223 - NUL-terminated pathname the entry describes (relative to the root of
224 the repository, i.e. full pathname);
225
226 - Three NUL-terminated ASCII octal numbers, entry mode of entries in
227 stage 1 to 3 (a missing stage is represented by "0" in this field);
228 and
229
230 - At most three object names of the entry in stages from 1 to 3
231 (nothing is written for a missing stage).
232
233 === Split index
234
235 In split index mode, the majority of index entries could be stored
236 in a separate file. This extension records the changes to be made on
237 top of that to produce the final index.
238
239 The signature for this extension is { 'l', 'i', 'n', 'k' }.
240
241 The extension consists of:
242
243 - Hash of the shared index file. The shared index file path
244 is $GIT_DIR/sharedindex.<hash>. If all bits are zero, the
245 index does not require a shared index file.
246
247 - An ewah-encoded delete bitmap, each bit represents an entry in the
248 shared index. If a bit is set, its corresponding entry in the
249 shared index will be removed from the final index. Note, because
250 a delete operation changes index entry positions, but we do need
251 original positions in replace phase, it's best to just mark
252 entries for removal, then do a mass deletion after replacement.
253
254 - An ewah-encoded replace bitmap, each bit represents an entry in
255 the shared index. If a bit is set, its corresponding entry in the
256 shared index will be replaced with an entry in this index
257 file. All replaced entries are stored in sorted order in this
258 index. The first "1" bit in the replace bitmap corresponds to the
259 first index entry, the second "1" bit to the second entry and so
260 on. Replaced entries may have empty path names to save space.
261
262 The remaining index entries after replaced ones will be added to the
263 final index. These added entries are also sorted by entry name then
264 stage.
265
266 == Untracked cache
267
268 Untracked cache saves the untracked file list and necessary data to
269 verify the cache. The signature for this extension is { 'U', 'N',
270 'T', 'R' }.
271
272 The extension starts with
273
274 - A sequence of NUL-terminated strings, preceded by the size of the
275 sequence in variable width encoding. Each string describes the
276 environment where the cache can be used.
277
278 - Stat data of $GIT_DIR/info/exclude. See "Index entry" section from
279 ctime field until "file size".
280
281 - Stat data of core.excludesFile
282
283 - 32-bit dir_flags (see struct dir_struct)
284
285 - Hash of $GIT_DIR/info/exclude. A null hash means the file
286 does not exist.
287
288 - Hash of core.excludesFile. A null hash means the file does
289 not exist.
290
291 - NUL-terminated string of per-dir exclude file name. This usually
292 is ".gitignore".
293
294 - The number of following directory blocks, variable width
295 encoding. If this number is zero, the extension ends here with a
296 following NUL.
297
298 - A number of directory blocks in depth-first-search order, each
299 consists of
300
301 - The number of untracked entries, variable width encoding.
302
303 - The number of sub-directory blocks, variable width encoding.
304
305 - The directory name terminated by NUL.
306
307 - A number of untracked file/dir names terminated by NUL.
308
309 The remaining data of each directory block is grouped by type:
310
311 - An ewah bitmap, the n-th bit marks whether the n-th directory has
312 valid untracked cache entries.
313
314 - An ewah bitmap, the n-th bit records "check-only" bit of
315 read_directory_recursive() for the n-th directory.
316
317 - An ewah bitmap, the n-th bit indicates whether hash and stat data
318 is valid for the n-th directory and exists in the next data.
319
320 - An array of stat data. The n-th data corresponds with the n-th
321 "one" bit in the previous ewah bitmap.
322
323 - An array of hashes. The n-th hash corresponds with the n-th "one" bit
324 in the previous ewah bitmap.
325
326 - One NUL.
327
328 == File System Monitor cache
329
330 The file system monitor cache tracks files for which the core.fsmonitor
331 hook has told us about changes. The signature for this extension is
332 { 'F', 'S', 'M', 'N' }.
333
334 The extension starts with
335
336 - 32-bit version number: the current supported versions are 1 and 2.
337
338 - (Version 1)
339 64-bit time: the extension data reflects all changes through the given
340 time which is stored as the nanoseconds elapsed since midnight,
341 January 1, 1970.
342
343 - (Version 2)
344 A null terminated string: an opaque token defined by the file system
345 monitor application. The extension data reflects all changes relative
346 to that token.
347
348 - 32-bit bitmap size: the size of the CE_FSMONITOR_VALID bitmap.
349
350 - An ewah bitmap, the n-th bit indicates whether the n-th index entry
351 is not CE_FSMONITOR_VALID.
352
353 == End of Index Entry
354
355 The End of Index Entry (EOIE) is used to locate the end of the variable
356 length index entries and the beginning of the extensions. Code can take
357 advantage of this to quickly locate the index extensions without having
358 to parse through all of the index entries.
359
360 Because it must be able to be loaded before the variable length cache
361 entries and other index extensions, this extension must be written last.
362 The signature for this extension is { 'E', 'O', 'I', 'E' }.
363
364 The extension consists of:
365
366 - 32-bit offset to the end of the index entries
367
368 - Hash over the extension types and their sizes (but not
369 their contents). E.g. if we have "TREE" extension that is N-bytes
370 long, "REUC" extension that is M-bytes long, followed by "EOIE",
371 then the hash would be:
372
373 Hash("TREE" + <binary representation of N> +
374 "REUC" + <binary representation of M>)
375
376 == Index Entry Offset Table
377
378 The Index Entry Offset Table (IEOT) is used to help address the CPU
379 cost of loading the index by enabling multi-threading the process of
380 converting cache entries from the on-disk format to the in-memory format.
381 The signature for this extension is { 'I', 'E', 'O', 'T' }.
382
383 The extension consists of:
384
385 - 32-bit version (currently 1)
386
387 - A number of index offset entries each consisting of:
388
389 - 32-bit offset from the beginning of the file to the first cache entry
390 in this block of entries.
391
392 - 32-bit count of cache entries in this block
393
394 == Sparse Directory Entries
395
396 When using sparse-checkout in cone mode, some entire directories within
397 the index can be summarized by pointing to a tree object instead of the
398 entire expanded list of paths within that tree. An index containing such
399 entries is a "sparse index". Index format versions 4 and less were not
400 implemented with such entries in mind. Thus, for these versions, an
401 index containing sparse directory entries will include this extension
402 with signature { 's', 'd', 'i', 'r' }. Like the split-index extension,
403 tools should avoid interacting with a sparse index unless they understand
404 this extension.