]>
Commit | Line | Data |
---|---|---|
5f40520f JH |
1 | git-pack-objects(1) |
2 | =================== | |
5f40520f JH |
3 | |
4 | NAME | |
5 | ---- | |
7bd7f280 | 6 | git-pack-objects - Create a packed archive of objects |
5f40520f JH |
7 | |
8 | ||
9 | SYNOPSIS | |
10 | -------- | |
ca5381d4 | 11 | [verse] |
4f366275 NP |
12 | 'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied] |
13 | [--no-reuse-delta] [--delta-base-offset] [--non-empty] | |
62b4698e | 14 | [--local] [--incremental] [--window=<n>] [--depth=<n>] |
ed7e5fc3 | 15 | [--revs [--unpacked | --all]] [--keep-pack=<pack-name>] |
9535ce73 | 16 | [--stdout [--filter=<filter-spec>] | base-name] |
de3a8641 | 17 | [--shallow] [--keep-true-parents] [--[no-]sparse] < object-list |
5f40520f JH |
18 | |
19 | ||
20 | DESCRIPTION | |
21 | ----------- | |
4a4becfb JT |
22 | Reads list of objects from the standard input, and writes either one or |
23 | more packed archives with the specified base-name to disk, or a packed | |
24 | archive to the standard output. | |
5f40520f | 25 | |
738820a9 SB |
26 | A packed archive is an efficient way to transfer a set of objects |
27 | between two repositories as well as an access efficient archival | |
28 | format. In a packed archive, an object is either stored as a | |
29 | compressed whole or as a difference from some other object. | |
30 | The latter is often called a delta. | |
31 | ||
32 | The packed archive format (.pack) is designed to be self-contained | |
33 | so that it can be unpacked without any further information. Therefore, | |
34 | each object that a delta depends upon must be present within the pack. | |
35 | ||
36 | A pack index file (.idx) is generated for fast, random access to the | |
37 | objects in the pack. Placing both the index file (.idx) and the packed | |
38 | archive (.pack) in the pack/ subdirectory of $GIT_OBJECT_DIRECTORY (or | |
1e6ab5de | 39 | any of the directories on $GIT_ALTERNATE_OBJECT_DIRECTORIES) |
2de9b711 | 40 | enables Git to read from the pack archive. |
1e6ab5de | 41 | |
0b444cdb | 42 | The 'git unpack-objects' command can read the packed archive and |
5f40520f JH |
43 | expand the objects contained in the pack into "one-file |
44 | one-object" format; this is typically done by the smart-pull | |
45 | commands when a pack is created on-the-fly for efficient network | |
46 | transport by their peers. | |
47 | ||
5f40520f JH |
48 | |
49 | OPTIONS | |
50 | ------- | |
51 | base-name:: | |
4a4becfb | 52 | Write into pairs of files (.pack and .idx), using |
5f40520f | 53 | <base-name> to determine the name of the created file. |
4a4becfb | 54 | When this option is used, the two files in a pair are written in |
d5fa1f1a | 55 | <base-name>-<SHA-1>.{pack,idx} files. <SHA-1> is a hash |
40a4f5a7 | 56 | based on the pack content and is written to the standard |
5f40520f JH |
57 | output of the command. |
58 | ||
59 | --stdout:: | |
89438677 | 60 | Write the pack contents (what would have been written to |
5f40520f JH |
61 | .pack file) out to the standard output. |
62 | ||
4321134c JH |
63 | --revs:: |
64 | Read the revision arguments from the standard input, instead of | |
65 | individual object names. The revision arguments are processed | |
0b444cdb | 66 | the same way as 'git rev-list' with the `--objects` flag |
4321134c JH |
67 | uses its `commit` arguments to build the list of objects it |
68 | outputs. The objects on the resulting list are packed. | |
b790e0f6 NTND |
69 | Besides revisions, `--not` or `--shallow <SHA-1>` lines are |
70 | also accepted. | |
4321134c JH |
71 | |
72 | --unpacked:: | |
73 | This implies `--revs`. When processing the list of | |
74 | revision arguments read from the standard input, limit | |
75 | the objects packed to those that are not already packed. | |
76 | ||
77 | --all:: | |
78 | This implies `--revs`. In addition to the list of | |
79 | revision arguments read from the standard input, pretend | |
cc1b8d8b | 80 | as if all refs under `refs/` are specified to be |
4321134c JH |
81 | included. |
82 | ||
f0a24aa5 SP |
83 | --include-tag:: |
84 | Include unasked-for annotated tags if the object they | |
85 | reference was included in the resulting packfile. This | |
2de9b711 | 86 | can be useful to send new tags to native Git clients. |
f0a24aa5 | 87 | |
339bce27 TB |
88 | --stdin-packs:: |
89 | Read the basenames of packfiles (e.g., `pack-1234abcd.pack`) | |
90 | from the standard input, instead of object names or revision | |
91 | arguments. The resulting pack contains all objects listed in the | |
92 | included packs (those not beginning with `^`), excluding any | |
93 | objects listed in the excluded packs (beginning with `^`). | |
94 | + | |
95 | Incompatible with `--revs`, or options that imply `--revs` (such as | |
96 | `--all`), with the exception of `--unpacked`, which is compatible. | |
97 | ||
62b4698e ŠN |
98 | --window=<n>:: |
99 | --depth=<n>:: | |
3df19671 | 100 | These two options affect how the objects contained in |
5f40520f JH |
101 | the pack are stored using delta compression. The |
102 | objects are first internally sorted by type, size and | |
103 | optionally names and compared against the other objects | |
104 | within --window to see if using delta compression saves | |
105 | space. --depth limits the maximum delta depth; making | |
106 | it too deep affects the performance on the unpacker | |
107 | side, because delta data needs to be applied that many | |
108 | times to get to the necessary object. | |
b5c0cbd8 NTND |
109 | + |
110 | The default value for --window is 10 and --depth is 50. The maximum | |
111 | depth is 4095. | |
5f40520f | 112 | |
62b4698e | 113 | --window-memory=<n>:: |
e93b15cd BD |
114 | This option provides an additional limit on top of `--window`; |
115 | the window size will dynamically scale down so as to not take | |
62b4698e | 116 | up more than '<n>' bytes in memory. This is useful in |
e93b15cd BD |
117 | repositories with a mix of large and small objects to not run |
118 | out of memory with a large window, but still be able to take | |
119 | advantage of the large window for the smaller objects. The | |
120 | size can be suffixed with "k", "m", or "g". | |
954176c1 MS |
121 | `--window-memory=0` makes memory usage unlimited. The default |
122 | is taken from the `pack.windowMemory` configuration variable. | |
e93b15cd | 123 | |
62b4698e | 124 | --max-pack-size=<n>:: |
4a4becfb JT |
125 | In unusual scenarios, you may not be able to create files |
126 | larger than a certain size on your filesystem, and this option | |
127 | can be used to tell the command to split the output packfile | |
128 | into multiple independent packfiles, each not larger than the | |
129 | given size. The size can be suffixed with | |
07cf0f24 | 130 | "k", "m", or "g". The minimum size allowed is limited to 1 MiB. |
2b84b5a8 | 131 | The default is unlimited, unless the config variable |
6fb9195f JK |
132 | `pack.packSizeLimit` is set. Note that this option may result in |
133 | a larger and slower repository; see the discussion in | |
134 | `pack.packSizeLimit`. | |
6b94b1a0 | 135 | |
e96fb9b8 BC |
136 | --honor-pack-keep:: |
137 | This flag causes an object already in a local pack that | |
0353a0c4 | 138 | has a .keep file to be ignored, even if it would have |
18879bc5 | 139 | otherwise been packed. |
e96fb9b8 | 140 | |
ed7e5fc3 NTND |
141 | --keep-pack=<pack-name>:: |
142 | This flag causes an object already in the given pack to be | |
143 | ignored, even if it would have otherwise been | |
24966cd9 | 144 | packed. `<pack-name>` is the pack file name without |
ed7e5fc3 NTND |
145 | leading directory (e.g. `pack-123.pack`). The option could be |
146 | specified multiple times to keep multiple packs. | |
147 | ||
5f40520f | 148 | --incremental:: |
21da4262 | 149 | This flag causes an object already in a pack to be ignored |
3909f14f | 150 | even if it would have otherwise been packed. |
5f40520f | 151 | |
12ea5bea | 152 | --local:: |
21da4262 | 153 | This flag causes an object that is borrowed from an alternate |
3909f14f JH |
154 | object store to be ignored even if it would have otherwise been |
155 | packed. | |
5f40520f | 156 | |
63ae26f8 NW |
157 | --non-empty:: |
158 | Only create a packed archive if it would contain at | |
159 | least one object. | |
160 | ||
231f240b NP |
161 | --progress:: |
162 | Progress status is reported on the standard error stream | |
163 | by default when it is attached to a terminal, unless -q | |
164 | is specified. This flag forces progress status even if | |
165 | the standard error stream is not directed to a terminal. | |
166 | ||
167 | --all-progress:: | |
168 | When --stdout is specified then progress report is | |
4f366275 | 169 | displayed during the object count and compression phases |
231f240b NP |
170 | but inhibited during the write-out phase. The reason is |
171 | that in some cases the output stream is directly linked | |
172 | to another command which may wish to display progress | |
173 | status of its own as it processes incoming pack data. | |
174 | This flag is like --progress except that it forces progress | |
175 | report for the write-out phase as well even if --stdout is | |
176 | used. | |
177 | ||
4f366275 NP |
178 | --all-progress-implied:: |
179 | This is used to imply --all-progress whenever progress display | |
180 | is activated. Unlike --all-progress this flag doesn't actually | |
181 | force any progress display by itself. | |
182 | ||
ca5381d4 JH |
183 | -q:: |
184 | This flag makes the command not to report its progress | |
185 | on the standard error stream. | |
186 | ||
187 | --no-reuse-delta:: | |
188 | When creating a packed archive in a repository that | |
189 | has existing packs, the command reuses existing deltas. | |
190 | This sometimes results in a slightly suboptimal pack. | |
191 | This flag tells the command not to reuse existing deltas | |
192 | but compute them from scratch. | |
193 | ||
fa736f72 NP |
194 | --no-reuse-object:: |
195 | This flag tells the command not to reuse existing object data at all, | |
196 | including non deltified object, forcing recompression of everything. | |
960ccca6 | 197 | This implies --no-reuse-delta. Useful only in the obscure case where |
fa736f72 NP |
198 | wholesale enforcement of a different compression level on the |
199 | packed data is desired. | |
200 | ||
62b4698e | 201 | --compression=<n>:: |
960ccca6 DH |
202 | Specifies compression level for newly-compressed data in the |
203 | generated pack. If not specified, pack compression level is | |
204 | determined first by pack.compression, then by core.compression, | |
205 | and defaults to -1, the zlib default, if neither is set. | |
483bc4f0 | 206 | Add --no-reuse-object if you want to force a uniform compression |
05cc2ffc | 207 | level on all data no matter the source. |
960ccca6 | 208 | |
de3a8641 DS |
209 | --[no-]sparse:: |
210 | Toggle the "sparse" algorithm to determine which objects to include in | |
4f6d26b1 DS |
211 | the pack, when combined with the "--revs" option. This algorithm |
212 | only walks trees that appear in paths that introduce new objects. | |
213 | This can have significant performance benefits when computing | |
214 | a pack to send a small change. However, it is possible that extra | |
215 | objects are added to the pack-file if the included commits contain | |
de3a8641 DS |
216 | certain types of direct renames. If this option is not included, |
217 | it defaults to the value of `pack.useSparse`, which is true unless | |
218 | otherwise specified. | |
4f6d26b1 | 219 | |
738820a9 SB |
220 | --thin:: |
221 | Create a "thin" pack by omitting the common objects between a | |
222 | sender and a receiver in order to reduce network transfer. This | |
223 | option only makes sense in conjunction with --stdout. | |
224 | + | |
225 | Note: A thin pack violates the packed archive format by omitting | |
2de9b711 | 226 | required objects and is thus unusable by Git without making it |
738820a9 SB |
227 | self-contained. Use `git index-pack --fix-thin` |
228 | (see linkgit:git-index-pack[1]) to restore the self-contained property. | |
229 | ||
2dacf26d | 230 | --shallow:: |
231 | Optimize a pack that will be provided to a client with a shallow | |
1c262bb7 | 232 | repository. This option, combined with --thin, can result in a |
2dacf26d | 233 | smaller pack at the cost of speed. |
234 | ||
63fba759 | 235 | --delta-base-offset:: |
2f8ee02c SB |
236 | A packed archive can express the base object of a delta as |
237 | either a 20-byte object name or as an offset in the | |
2de9b711 | 238 | stream, but ancient versions of Git don't understand the |
0b444cdb | 239 | latter. By default, 'git pack-objects' only uses the |
63fba759 JH |
240 | former format for better compatibility. This option |
241 | allows the command to use the latter format for | |
242 | compactness. Depending on the average delta chain | |
243 | length, this option typically shrinks the resulting | |
244 | packfile by 3-5 per-cent. | |
c14f3727 JH |
245 | + |
246 | Note: Porcelain commands such as `git gc` (see linkgit:git-gc[1]), | |
247 | `git repack` (see linkgit:git-repack[1]) pass this option by default | |
2de9b711 | 248 | in modern Git when they put objects in your repository into pack files. |
c14f3727 | 249 | So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle. |
63fba759 | 250 | |
367f4a43 NP |
251 | --threads=<n>:: |
252 | Specifies the number of threads to spawn when searching for best | |
253 | delta matches. This requires that pack-objects be compiled with | |
254 | pthreads otherwise this option is ignored with a warning. | |
255 | This is meant to reduce packing time on multiprocessor machines. | |
256 | The required amount of memory for the delta search window is | |
257 | however multiplied by the number of threads. | |
2de9b711 | 258 | Specifying 0 will cause Git to auto-detect the number of CPU's |
833e3df1 | 259 | and set the number of threads accordingly. |
367f4a43 | 260 | |
be18c1fe NP |
261 | --index-version=<version>[,<offset>]:: |
262 | This is intended to be used by the test suite only. It allows | |
263 | to force the version for the generated pack index, and to force | |
264 | 64-bit index entries on objects located above the given offset. | |
265 | ||
7f3140cd JS |
266 | --keep-true-parents:: |
267 | With this option, parents that are hidden by grafts are packed | |
268 | nevertheless. | |
269 | ||
9535ce73 JH |
270 | --filter=<filter-spec>:: |
271 | Requires `--stdout`. Omits certain objects (usually blobs) from | |
272 | the resulting packfile. See linkgit:git-rev-list[1] for valid | |
273 | `<filter-spec>` forms. | |
274 | ||
4875c979 JH |
275 | --no-filter:: |
276 | Turns off any previous `--filter=` argument. | |
277 | ||
9535ce73 JH |
278 | --missing=<missing-action>:: |
279 | A debug option to help with future "partial clone" development. | |
280 | This option specifies how missing objects are handled. | |
281 | + | |
282 | The form '--missing=error' requests that pack-objects stop with an error if | |
ee47243d JT |
283 | a missing object is encountered. If the repository is a partial clone, an |
284 | attempt to fetch missing objects will be made before declaring them missing. | |
285 | This is the default action. | |
9535ce73 JH |
286 | + |
287 | The form '--missing=allow-any' will allow object traversal to continue | |
ee47243d JT |
288 | if a missing object is encountered. No fetch of a missing object will occur. |
289 | Missing objects will silently be omitted from the results. | |
0c16cd49 JT |
290 | + |
291 | The form '--missing=allow-promisor' is like 'allow-any', but will only | |
292 | allow object traversal to continue for EXPECTED promisor missing objects. | |
ee47243d JT |
293 | No fetch of a missing object will occur. An unexpected missing object will |
294 | raise an error. | |
0c16cd49 JT |
295 | |
296 | --exclude-promisor-objects:: | |
297 | Omit objects that are known to be in the promisor remote. (This | |
298 | option has the purpose of operating only on locally created objects, | |
299 | so that when we repack, we still maintain a distinction between | |
300 | locally created objects [without .promisor] and objects from the | |
301 | promisor remote [with .promisor].) This is used with partial clone. | |
9535ce73 | 302 | |
58bd77b6 NTND |
303 | --keep-unreachable:: |
304 | Objects unreachable from the refs in packs named with | |
305 | --unpacked= option are added to the resulting pack, in | |
306 | addition to the reachable objects that are not in packs marked | |
307 | with *.keep files. This implies `--revs`. | |
308 | ||
309 | --pack-loose-unreachable:: | |
310 | Pack unreachable loose objects (and their loose counterparts | |
311 | removed). This implies `--revs`. | |
312 | ||
313 | --unpack-unreachable:: | |
314 | Keep unreachable objects in loose form. This implies `--revs`. | |
315 | ||
28b8a730 JK |
316 | --delta-islands:: |
317 | Restrict delta matches based on "islands". See DELTA ISLANDS | |
318 | below. | |
319 | ||
320 | ||
321 | DELTA ISLANDS | |
322 | ------------- | |
323 | ||
324 | When possible, `pack-objects` tries to reuse existing on-disk deltas to | |
325 | avoid having to search for new ones on the fly. This is an important | |
326 | optimization for serving fetches, because it means the server can avoid | |
327 | inflating most objects at all and just send the bytes directly from | |
328 | disk. This optimization can't work when an object is stored as a delta | |
329 | against a base which the receiver does not have (and which we are not | |
330 | already sending). In that case the server "breaks" the delta and has to | |
331 | find a new one, which has a high CPU cost. Therefore it's important for | |
332 | performance that the set of objects in on-disk delta relationships match | |
333 | what a client would fetch. | |
334 | ||
335 | In a normal repository, this tends to work automatically. The objects | |
336 | are mostly reachable from the branches and tags, and that's what clients | |
337 | fetch. Any deltas we find on the server are likely to be between objects | |
338 | the client has or will have. | |
339 | ||
340 | But in some repository setups, you may have several related but separate | |
341 | groups of ref tips, with clients tending to fetch those groups | |
342 | independently. For example, imagine that you are hosting several "forks" | |
343 | of a repository in a single shared object store, and letting clients | |
344 | view them as separate repositories through `GIT_NAMESPACE` or separate | |
345 | repos using the alternates mechanism. A naive repack may find that the | |
346 | optimal delta for an object is against a base that is only found in | |
347 | another fork. But when a client fetches, they will not have the base | |
348 | object, and we'll have to find a new delta on the fly. | |
349 | ||
350 | A similar situation may exist if you have many refs outside of | |
351 | `refs/heads/` and `refs/tags/` that point to related objects (e.g., | |
352 | `refs/pull` or `refs/changes` used by some hosting providers). By | |
353 | default, clients fetch only heads and tags, and deltas against objects | |
354 | found only in those other groups cannot be sent as-is. | |
355 | ||
356 | Delta islands solve this problem by allowing you to group your refs into | |
357 | distinct "islands". Pack-objects computes which objects are reachable | |
358 | from which islands, and refuses to make a delta from an object `A` | |
359 | against a base which is not present in all of `A`'s islands. This | |
360 | results in slightly larger packs (because we miss some delta | |
361 | opportunities), but guarantees that a fetch of one island will not have | |
362 | to recompute deltas on the fly due to crossing island boundaries. | |
363 | ||
364 | When repacking with delta islands the delta window tends to get | |
365 | clogged with candidates that are forbidden by the config. Repacking | |
366 | with a big --window helps (and doesn't take as long as it otherwise | |
367 | might because we can reject some object pairs based on islands before | |
368 | doing any computation on the content). | |
369 | ||
370 | Islands are configured via the `pack.island` option, which can be | |
371 | specified multiple times. Each value is a left-anchored regular | |
372 | expressions matching refnames. For example: | |
373 | ||
374 | ------------------------------------------- | |
375 | [pack] | |
376 | island = refs/heads/ | |
377 | island = refs/tags/ | |
378 | ------------------------------------------- | |
379 | ||
380 | puts heads and tags into an island (whose name is the empty string; see | |
381 | below for more on naming). Any refs which do not match those regular | |
382 | expressions (e.g., `refs/pull/123`) is not in any island. Any object | |
383 | which is reachable only from `refs/pull/` (but not heads or tags) is | |
384 | therefore not a candidate to be used as a base for `refs/heads/`. | |
385 | ||
386 | Refs are grouped into islands based on their "names", and two regexes | |
387 | that produce the same name are considered to be in the same | |
388 | island. The names are computed from the regexes by concatenating any | |
389 | capture groups from the regex, with a '-' dash in between. (And if | |
390 | there are no capture groups, then the name is the empty string, as in | |
391 | the above example.) This allows you to create arbitrary numbers of | |
392 | islands. Only up to 14 such capture groups are supported though. | |
393 | ||
394 | For example, imagine you store the refs for each fork in | |
395 | `refs/virtual/ID`, where `ID` is a numeric identifier. You might then | |
396 | configure: | |
397 | ||
398 | ------------------------------------------- | |
399 | [pack] | |
400 | island = refs/virtual/([0-9]+)/heads/ | |
401 | island = refs/virtual/([0-9]+)/tags/ | |
402 | island = refs/virtual/([0-9]+)/(pull)/ | |
403 | ------------------------------------------- | |
404 | ||
405 | That puts the heads and tags for each fork in their own island (named | |
406 | "1234" or similar), and the pull refs for each go into their own | |
407 | "1234-pull". | |
408 | ||
409 | Note that we pick a single island for each regex to go into, using "last | |
410 | one wins" ordering (which allows repo-specific config to take precedence | |
411 | over user-wide config, and so forth). | |
412 | ||
3a837b58 CW |
413 | |
414 | CONFIGURATION | |
415 | ------------- | |
416 | ||
417 | Various configuration variables affect packing, see | |
418 | linkgit:git-config[1] (search for "pack" and "delta"). | |
419 | ||
420 | Notably, delta compression is not used on objects larger than the | |
421 | `core.bigFileThreshold` configuration variable and on files with the | |
422 | attribute `delta` set to false. | |
423 | ||
56ae8df5 | 424 | SEE ALSO |
e31bb3bb | 425 | -------- |
5162e697 DM |
426 | linkgit:git-rev-list[1] |
427 | linkgit:git-repack[1] | |
428 | linkgit:git-prune-packed[1] | |
e31bb3bb | 429 | |
5f40520f JH |
430 | GIT |
431 | --- | |
9e1f0a85 | 432 | Part of the linkgit:git[1] suite |