]>
Commit | Line | Data |
---|---|---|
c401b33c JS |
1 | git-filter-branch(1) |
2 | ==================== | |
3 | ||
4 | NAME | |
5 | ---- | |
6 | git-filter-branch - Rewrite branches | |
7 | ||
8 | SYNOPSIS | |
9 | -------- | |
10 | [verse] | |
07c49845 DG |
11 | 'git filter-branch' [--setup <command>] [--subdirectory-filter <directory>] |
12 | [--env-filter <command>] [--tree-filter <command>] | |
13 | [--index-filter <command>] [--parent-filter <command>] | |
14 | [--msg-filter <command>] [--commit-filter <command>] | |
15 | [--tag-name-filter <command>] [--prune-empty] | |
5433235d | 16 | [--original <namespace>] [-d <directory>] [-f | --force] |
bd2c79fb | 17 | [--state-branch <branch>] [--] [<rev-list options>...] |
c401b33c | 18 | |
9df53c5d EN |
19 | WARNING |
20 | ------- | |
21 | 'git filter-branch' has a plethora of pitfalls that can produce non-obvious | |
22 | manglings of the intended history rewrite (and can leave you with little | |
23 | time to investigate such problems since it has such abysmal performance). | |
24 | These safety and performance issues cannot be backward compatibly fixed and | |
25 | as such, its use is not recommended. Please use an alternative history | |
26 | filtering tool such as https://github.com/newren/git-filter-repo/[git | |
27 | filter-repo]. If you still need to use 'git filter-branch', please | |
28 | carefully read <<SAFETY>> (and <<PERFORMANCE>>) to learn about the land | |
29 | mines of filter-branch, and then vigilantly avoid as many of the hazards | |
30 | listed there as reasonably possible. | |
31 | ||
c401b33c JS |
32 | DESCRIPTION |
33 | ----------- | |
2de9b711 | 34 | Lets you rewrite Git revision history by rewriting the branches mentioned |
08203668 | 35 | in the <rev-list options>, applying custom filters on each revision. |
c401b33c JS |
36 | Those filters can modify each tree (e.g. removing a file or running |
37 | a perl rewrite on all files) or information about each commit. | |
38 | Otherwise, all information (including original commit times or merge | |
39 | information) will be preserved. | |
40 | ||
08203668 | 41 | The command will only rewrite the _positive_ refs mentioned in the |
bf7c9021 | 42 | command line (e.g. if you pass 'a..b', only 'b' will be rewritten). |
08203668 JS |
43 | If you specify no filters, the commits will be recommitted without any |
44 | changes, which would normally have no effect. Nevertheless, this may be | |
2de9b711 | 45 | useful in the future for compensating for some Git bugs or such, |
08203668 | 46 | therefore such a usage is permitted. |
c401b33c | 47 | |
831e61f8 JH |
48 | *NOTE*: This command honors `.git/info/grafts` file and refs in |
49 | the `refs/replace/` namespace. | |
0dc310e8 PC |
50 | If you have any grafts or replacement refs defined, running this command |
51 | will make them permanent. | |
c6d8f763 | 52 | |
73616fd3 | 53 | *WARNING*! The rewritten history will have different object names for all |
c401b33c JS |
54 | the objects and will not converge with the original branch. You will not |
55 | be able to easily push and distribute the rewritten branch on top of the | |
56 | original branch. Please do not use this command if you do not know the | |
57 | full implications, and avoid using it anyway, if a simple single commit | |
97c33c65 TR |
58 | would suffice to fix your problem. (See the "RECOVERING FROM UPSTREAM |
59 | REBASE" section in linkgit:git-rebase[1] for further information about | |
60 | rewriting published history.) | |
c401b33c | 61 | |
dfd05e38 JS |
62 | Always verify that the rewritten version is correct: The original refs, |
63 | if different from the rewritten ones, will be stored in the namespace | |
64 | 'refs/original/'. | |
c401b33c | 65 | |
bf7c9021 | 66 | Note that since this operation is very I/O expensive, it might |
08203668 | 67 | be a good idea to redirect the temporary directory off-disk with the |
23f8239b | 68 | `-d` option, e.g. on tmpfs. Reportedly the speedup is very noticeable. |
c401b33c JS |
69 | |
70 | ||
71 | Filters | |
72 | ~~~~~~~ | |
73 | ||
74 | The filters are applied in the order as listed below. The <command> | |
bf7c9021 RW |
75 | argument is always evaluated in the shell context using the 'eval' command |
76 | (with the notable exception of the commit filter, for technical reasons). | |
47d81b5c | 77 | Prior to that, the `$GIT_COMMIT` environment variable will be set to contain |
c401b33c JS |
78 | the id of the commit being rewritten. Also, GIT_AUTHOR_NAME, |
79 | GIT_AUTHOR_EMAIL, GIT_AUTHOR_DATE, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, | |
bee3eb07 TAK |
80 | and GIT_COMMITTER_DATE are taken from the current commit and exported to |
81 | the environment, in order to affect the author and committer identities of | |
82 | the replacement commit created by linkgit:git-commit-tree[1] after the | |
83 | filters have run. | |
84 | ||
bf7c9021 RW |
85 | If any evaluation of <command> returns a non-zero exit status, the whole |
86 | operation will be aborted. | |
c401b33c JS |
87 | |
88 | A 'map' function is available that takes an "original sha1 id" argument | |
89 | and outputs a "rewritten sha1 id" if the commit has been already | |
32c37c12 JS |
90 | rewritten, and "original sha1 id" otherwise; the 'map' function can |
91 | return several ids on separate lines if your commit filter emitted | |
92 | multiple commits. | |
c401b33c JS |
93 | |
94 | ||
95 | OPTIONS | |
96 | ------- | |
97 | ||
3b117f73 AH |
98 | --setup <command>:: |
99 | This is not a real filter executed for each commit but a one | |
100 | time setup just before the loop. Therefore no commit-specific | |
101 | variables are defined yet. Functions or variables defined here | |
102 | can be used or modified in the following filter steps except | |
103 | the commit filter, for technical reasons. | |
104 | ||
07c49845 DG |
105 | --subdirectory-filter <directory>:: |
106 | Only look at the history which touches the given subdirectory. | |
107 | The result will contain that directory (and only that) as its | |
108 | project root. Implies <<Remap_to_ancestor>>. | |
109 | ||
c401b33c | 110 | --env-filter <command>:: |
bf7c9021 RW |
111 | This filter may be used if you only need to modify the environment |
112 | in which the commit will be performed. Specifically, you might | |
113 | want to rewrite the author/committer name/email/time environment | |
ba746ff9 | 114 | variables (see linkgit:git-commit-tree[1] for details). |
c401b33c JS |
115 | |
116 | --tree-filter <command>:: | |
117 | This is the filter for rewriting the tree and its contents. | |
118 | The argument is evaluated in shell with the working | |
119 | directory set to the root of the checked out tree. The new tree | |
120 | is then used as-is (new files are auto-added, disappeared files | |
121 | are auto-removed - neither .gitignore files nor any other ignore | |
73616fd3 | 122 | rules *HAVE ANY EFFECT*!). |
c401b33c JS |
123 | |
124 | --index-filter <command>:: | |
125 | This is the filter for rewriting the index. It is similar to the | |
126 | tree filter but does not check out the tree, which makes it much | |
6cf378f0 JK |
127 | faster. Frequently used with `git rm --cached |
128 | --ignore-unmatch ...`, see EXAMPLES below. For hairy | |
3bc427e0 | 129 | cases, see linkgit:git-update-index[1]. |
c401b33c JS |
130 | |
131 | --parent-filter <command>:: | |
132 | This is the filter for rewriting the commit's parent list. | |
133 | It will receive the parent string on stdin and shall output | |
134 | the new parent string on stdout. The parent string is in | |
483bc4f0 | 135 | the format described in linkgit:git-commit-tree[1]: empty for |
c401b33c JS |
136 | the initial commit, "-p parent" for a normal commit and |
137 | "-p parent1 -p parent2 -p parent3 ..." for a merge commit. | |
138 | ||
139 | --msg-filter <command>:: | |
140 | This is the filter for rewriting the commit messages. | |
141 | The argument is evaluated in the shell with the original | |
142 | commit message on standard input; its standard output is | |
143 | used as the new commit message. | |
144 | ||
145 | --commit-filter <command>:: | |
146 | This is the filter for performing the commit. | |
147 | If this filter is specified, it will be called instead of the | |
0b444cdb | 148 | 'git commit-tree' command, with arguments of the form |
0adda936 | 149 | "<TREE_ID> [(-p <PARENT_COMMIT_ID>)...]" and the log message on |
c401b33c JS |
150 | stdin. The commit id is expected on stdout. |
151 | + | |
152 | As a special extension, the commit filter may emit multiple | |
c5833f6e | 153 | commit ids; in that case, the rewritten children of the original commit will |
c401b33c | 154 | have all of them as parents. |
f95eef15 JS |
155 | + |
156 | You can use the 'map' convenience function in this filter, and other | |
157 | convenience functions, too. For example, calling 'skip_commit "$@"' | |
158 | will leave out the current commit (but not its changes! If you want | |
0b444cdb | 159 | that, use 'git rebase' instead). |
d3240d93 | 160 | + |
ca768288 TR |
161 | You can also use the `git_commit_non_empty_tree "$@"` instead of |
162 | `git commit-tree "$@"` if you don't wish to keep commits with a single parent | |
d3240d93 | 163 | and that makes no change to the tree. |
c401b33c JS |
164 | |
165 | --tag-name-filter <command>:: | |
166 | This is the filter for rewriting tag names. When passed, | |
167 | it will be called for every tag ref that points to a rewritten | |
168 | object (or to a tag object which points to a rewritten object). | |
169 | The original tag name is passed via standard input, and the new | |
170 | tag name is expected on standard output. | |
171 | + | |
172 | The original tags are not deleted, but can be overwritten; | |
5876b8ee | 173 | use "--tag-name-filter cat" to simply update the tags. In this |
c401b33c JS |
174 | case, be very careful and make sure you have the old tags |
175 | backed up in case the conversion has run afoul. | |
176 | + | |
1bf6551e BC |
177 | Nearly proper rewriting of tag objects is supported. If the tag has |
178 | a message attached, a new tag object will be created with the same message, | |
179 | author, and timestamp. If the tag has a signature attached, the | |
180 | signature will be stripped. It is by definition impossible to preserve | |
181 | signatures. The reason this is "nearly" proper, is because ideally if | |
182 | the tag did not change (points to the same object, has the same name, etc.) | |
183 | it should retain any signature. That is not the case, signatures will always | |
184 | be removed, buyer beware. There is also no support for changing the | |
185 | author or timestamp (or the tag message for that matter). Tags which point | |
186 | to other tags will be rewritten to point to the underlying commit. | |
c401b33c | 187 | |
d3240d93 | 188 | --prune-empty:: |
a582a82d DP |
189 | Some filters will generate empty commits that leave the tree untouched. |
190 | This option instructs git-filter-branch to remove such commits if they | |
191 | have exactly one or zero non-pruned parents; merge commits will | |
192 | therefore remain intact. This option cannot be used together with | |
193 | `--commit-filter`, though the same effect can be achieved by using the | |
194 | provided `git_commit_non_empty_tree` function in a commit filter. | |
d3240d93 | 195 | |
5433235d GB |
196 | --original <namespace>:: |
197 | Use this option to set the namespace where the original commits | |
198 | will be stored. The default value is 'refs/original'. | |
199 | ||
c401b33c JS |
200 | -d <directory>:: |
201 | Use this option to set the path to the temporary directory used for | |
202 | rewriting. When applying a tree filter, the command needs to | |
bf7c9021 | 203 | temporarily check out the tree to some directory, which may consume |
c401b33c | 204 | considerable space in case of large projects. By default it |
68ed71b5 | 205 | does this in the `.git-rewrite/` directory but you can override |
c401b33c JS |
206 | that choice by this parameter. |
207 | ||
3240240f SB |
208 | -f:: |
209 | --force:: | |
0b444cdb | 210 | 'git filter-branch' refuses to start with an existing temporary |
dfd05e38 JS |
211 | directory or when there are already refs starting with |
212 | 'refs/original/', unless forced. | |
213 | ||
bd2c79fb IC |
214 | --state-branch <branch>:: |
215 | This option will cause the mapping from old to new objects to | |
216 | be loaded from named branch upon startup and saved as a new | |
217 | commit to that branch upon exit, enabling incremental of large | |
218 | trees. If '<branch>' does not exist it will be created. | |
219 | ||
f448e24e | 220 | <rev-list options>...:: |
0b444cdb | 221 | Arguments for 'git rev-list'. All positive refs included by |
8afa4210 | 222 | these options are rewritten. You may also specify options |
04b125de | 223 | such as `--all`, but you must use `--` to separate them from |
7ec344d8 CH |
224 | the 'git filter-branch' options. Implies <<Remap_to_ancestor>>. |
225 | ||
226 | ||
227 | [[Remap_to_ancestor]] | |
228 | Remap to ancestor | |
229 | ~~~~~~~~~~~~~~~~~ | |
230 | ||
1cca17df | 231 | By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit the |
7ec344d8 CH |
232 | set of revisions which get rewritten. However, positive refs on the command |
233 | line are distinguished: we don't let them be excluded by such limiters. For | |
234 | this purpose, they are instead rewritten to point at the nearest ancestor that | |
235 | was not excluded. | |
c401b33c JS |
236 | |
237 | ||
0a0eb2e5 ML |
238 | EXIT STATUS |
239 | ----------- | |
240 | ||
241 | On success, the exit status is `0`. If the filter can't find any commits to | |
242 | rewrite, the exit status is `2`. On any other error, the exit status may be | |
243 | any other non-zero value. | |
244 | ||
245 | ||
76a8788c | 246 | EXAMPLES |
c401b33c JS |
247 | -------- |
248 | ||
249 | Suppose you want to remove a file (containing confidential information | |
250 | or copyright violation) from all commits: | |
251 | ||
252 | ------------------------------------------------------- | |
dfd05e38 | 253 | git filter-branch --tree-filter 'rm filename' HEAD |
c401b33c JS |
254 | ------------------------------------------------------- |
255 | ||
e4d594c6 JL |
256 | However, if the file is absent from the tree of some commit, |
257 | a simple `rm filename` will fail for that tree and commit. | |
258 | Thus you may instead want to use `rm -f filename` as the script. | |
259 | ||
6cf378f0 | 260 | Using `--index-filter` with 'git rm' yields a significantly faster |
3bc427e0 TR |
261 | version. Like with using `rm filename`, `git rm --cached filename` |
262 | will fail if the file is absent from the tree of a commit. If you | |
263 | want to "completely forget" a file, it does not matter when it entered | |
6cf378f0 | 264 | history, so we also add `--ignore-unmatch`: |
c401b33c | 265 | |
dfd05e38 | 266 | -------------------------------------------------------------------------- |
3bc427e0 | 267 | git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD |
dfd05e38 | 268 | -------------------------------------------------------------------------- |
c401b33c | 269 | |
8ef44519 | 270 | Now, you will get the rewritten history saved in HEAD. |
c401b33c | 271 | |
8afa4210 TR |
272 | To rewrite the repository to look as if `foodir/` had been its project |
273 | root, and discard all other history: | |
274 | ||
275 | ------------------------------------------------------- | |
276 | git filter-branch --subdirectory-filter foodir -- --all | |
277 | ------------------------------------------------------- | |
278 | ||
279 | Thus you can, e.g., turn a library subdirectory into a repository of | |
6cf378f0 JK |
280 | its own. Note the `--` that separates 'filter-branch' options from |
281 | revision options, and the `--all` to rewrite all branches and tags. | |
8afa4210 | 282 | |
32c37c12 JS |
283 | To set a commit (which typically is at the tip of another |
284 | history) to be the parent of the current initial commit, in | |
285 | order to paste the other history behind the current history: | |
c401b33c | 286 | |
dfd05e38 JS |
287 | ------------------------------------------------------------------- |
288 | git filter-branch --parent-filter 'sed "s/^\$/-p <graft-id>/"' HEAD | |
289 | ------------------------------------------------------------------- | |
c401b33c | 290 | |
08203668 JS |
291 | (if the parent string is empty - which happens when we are dealing with |
292 | the initial commit - add graftcommit as a parent). Note that this assumes | |
c401b33c JS |
293 | history with a single root (that is, no merge without common ancestors |
294 | happened). If this is not the case, use: | |
295 | ||
dfd05e38 | 296 | -------------------------------------------------------------------------- |
c401b33c | 297 | git filter-branch --parent-filter \ |
41e86a37 | 298 | 'test $GIT_COMMIT = <commit-id> && echo "-p <graft-id>" || cat' HEAD |
dfd05e38 | 299 | -------------------------------------------------------------------------- |
c401b33c | 300 | |
32c37c12 JS |
301 | or even simpler: |
302 | ||
303 | ----------------------------------------------- | |
e2d65c1e | 304 | git replace --graft $commit-id $graft-id |
dfd05e38 | 305 | git filter-branch $graft-id..HEAD |
32c37c12 JS |
306 | ----------------------------------------------- |
307 | ||
c401b33c JS |
308 | To remove commits authored by "Darl McBribe" from the history: |
309 | ||
310 | ------------------------------------------------------------------------------ | |
311 | git filter-branch --commit-filter ' | |
312 | if [ "$GIT_AUTHOR_NAME" = "Darl McBribe" ]; | |
313 | then | |
f95eef15 | 314 | skip_commit "$@"; |
c401b33c JS |
315 | else |
316 | git commit-tree "$@"; | |
dfd05e38 | 317 | fi' HEAD |
c401b33c JS |
318 | ------------------------------------------------------------------------------ |
319 | ||
8451c565 | 320 | The function 'skip_commit' is defined as follows: |
f95eef15 JS |
321 | |
322 | -------------------------- | |
323 | skip_commit() | |
324 | { | |
325 | shift; | |
326 | while [ -n "$1" ]; | |
327 | do | |
328 | shift; | |
329 | map "$1"; | |
330 | shift; | |
331 | done; | |
332 | } | |
333 | -------------------------- | |
334 | ||
c401b33c JS |
335 | The shift magic first throws away the tree id and then the -p |
336 | parameters. Note that this handles merges properly! In case Darl | |
337 | committed a merge between P1 and P2, it will be propagated properly | |
338 | and all children of the merge will become merge commits with P1,P2 | |
339 | as their parents instead of the merge commit. | |
340 | ||
8093ae88 AS |
341 | *NOTE* the changes introduced by the commits, and which are not reverted |
342 | by subsequent commits, will still be in the rewritten branch. If you want | |
343 | to throw out _changes_ together with the commits, you should use the | |
344 | interactive mode of 'git rebase'. | |
345 | ||
a1748890 | 346 | You can rewrite the commit log messages using `--msg-filter`. For |
0b444cdb | 347 | example, 'git svn-id' strings in a repository created by 'git svn' can |
ed10d9aa MV |
348 | be removed this way: |
349 | ||
350 | ------------------------------------------------------- | |
a1748890 | 351 | git filter-branch --msg-filter ' |
ed10d9aa MV |
352 | sed -e "/^git-svn-id:/d" |
353 | ' | |
354 | ------------------------------------------------------- | |
f95eef15 | 355 | |
b8f42332 JS |
356 | If you need to add 'Acked-by' lines to, say, the last 10 commits (none |
357 | of which is a merge), use this command: | |
358 | ||
359 | -------------------------------------------------------- | |
360 | git filter-branch --msg-filter ' | |
361 | cat && | |
362 | echo "Acked-by: Bugs Bunny <bunny@bugzilla.org>" | |
363 | ' HEAD~10..HEAD | |
364 | -------------------------------------------------------- | |
365 | ||
21b6e4f2 TAK |
366 | The `--env-filter` option can be used to modify committer and/or author |
367 | identity. For example, if you found out that your commits have the wrong | |
368 | identity due to a misconfigured user.email, you can make a correction, | |
369 | before publishing the project, like this: | |
370 | ||
371 | -------------------------------------------------------- | |
372 | git filter-branch --env-filter ' | |
373 | if test "$GIT_AUTHOR_EMAIL" = "root@localhost" | |
374 | then | |
375 | GIT_AUTHOR_EMAIL=john@example.com | |
21b6e4f2 TAK |
376 | fi |
377 | if test "$GIT_COMMITTER_EMAIL" = "root@localhost" | |
378 | then | |
379 | GIT_COMMITTER_EMAIL=john@example.com | |
21b6e4f2 TAK |
380 | fi |
381 | ' -- --all | |
382 | -------------------------------------------------------- | |
383 | ||
8093ae88 AS |
384 | To restrict rewriting to only part of the history, specify a revision |
385 | range in addition to the new branch name. The new branch name will | |
386 | point to the top-most revision that a 'git rev-list' of this range | |
387 | will print. | |
08203668 | 388 | |
c401b33c JS |
389 | Consider this history: |
390 | ||
391 | ------------------ | |
392 | D--E--F--G--H | |
393 | / / | |
394 | A--B-----C | |
395 | ------------------ | |
396 | ||
397 | To rewrite only commits D,E,F,G,H, but leave A, B and C alone, use: | |
398 | ||
399 | -------------------------------- | |
dfd05e38 | 400 | git filter-branch ... C..H |
c401b33c JS |
401 | -------------------------------- |
402 | ||
403 | To rewrite commits E,F,G,H, use one of these: | |
404 | ||
405 | ---------------------------------------- | |
dfd05e38 JS |
406 | git filter-branch ... C..H --not D |
407 | git filter-branch ... D..H --not C | |
c401b33c JS |
408 | ---------------------------------------- |
409 | ||
410 | To move the whole tree into a subdirectory, or remove it from there: | |
411 | ||
412 | --------------------------------------------------------------- | |
413 | git filter-branch --index-filter \ | |
d2d66f15 | 414 | 'git ls-files -s | sed "s-\t\"*-&newsubdir/-" | |
c401b33c JS |
415 | GIT_INDEX_FILE=$GIT_INDEX_FILE.new \ |
416 | git update-index --index-info && | |
6cb0186a | 417 | mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD |
c401b33c JS |
418 | --------------------------------------------------------------- |
419 | ||
420 | ||
d0268de6 | 421 | |
76a8788c | 422 | CHECKLIST FOR SHRINKING A REPOSITORY |
d0268de6 TR |
423 | ------------------------------------ |
424 | ||
615b8f1a | 425 | git-filter-branch can be used to get rid of a subset of files, |
6cf378f0 JK |
426 | usually with some combination of `--index-filter` and |
427 | `--subdirectory-filter`. People expect the resulting repository to | |
d0268de6 | 428 | be smaller than the original, but you need a few more steps to |
2de9b711 | 429 | actually make it smaller, because Git tries hard not to lose your |
d0268de6 TR |
430 | objects until you tell it to. First make sure that: |
431 | ||
432 | * You really removed all variants of a filename, if a blob was moved | |
6cf378f0 JK |
433 | over its lifetime. `git log --name-only --follow --all -- filename` |
434 | can help you find renames. | |
d0268de6 | 435 | |
6cf378f0 JK |
436 | * You really filtered all refs: use `--tag-name-filter cat -- --all` |
437 | when calling git-filter-branch. | |
d0268de6 TR |
438 | |
439 | Then there are two ways to get a smaller repository. A safer way is | |
440 | to clone, that keeps your original intact. | |
441 | ||
6cf378f0 | 442 | * Clone it with `git clone file:///path/to/repo`. The clone |
d0268de6 TR |
443 | will not have the removed objects. See linkgit:git-clone[1]. (Note |
444 | that cloning with a plain path just hardlinks everything!) | |
445 | ||
446 | If you really don't want to clone it, for whatever reasons, check the | |
447 | following points instead (in this order). This is a very destructive | |
448 | approach, so *make a backup* or go back to cloning it. You have been | |
449 | warned. | |
450 | ||
451 | * Remove the original refs backed up by git-filter-branch: say `git | |
6cf378f0 | 452 | for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git |
d0268de6 TR |
453 | update-ref -d`. |
454 | ||
6cf378f0 | 455 | * Expire all reflogs with `git reflog expire --expire=now --all`. |
d0268de6 | 456 | |
6cf378f0 | 457 | * Garbage collect all unreferenced objects with `git gc --prune=now` |
d0268de6 | 458 | (or if your git-gc is not new enough to support arguments to |
6cf378f0 | 459 | `--prune`, use `git repack -ad; git prune` instead). |
d0268de6 | 460 | |
9df53c5d EN |
461 | [[PERFORMANCE]] |
462 | PERFORMANCE | |
463 | ----------- | |
464 | ||
465 | The performance of git-filter-branch is glacially slow; its design makes it | |
466 | impossible for a backward-compatible implementation to ever be fast: | |
467 | ||
468 | * In editing files, git-filter-branch by design checks out each and | |
df5be016 | 469 | every commit as it existed in the original repo. If your repo has |
e469afe1 MÅ |
470 | `10^5` files and `10^5` commits, but each commit only modifies five |
471 | files, then git-filter-branch will make you do `10^10` modifications, | |
472 | despite only having (at most) `5*10^5` unique blobs. | |
9df53c5d EN |
473 | |
474 | * If you try and cheat and try to make git-filter-branch only work on | |
df5be016 | 475 | files modified in a commit, then two things happen |
9df53c5d EN |
476 | |
477 | ** you run into problems with deletions whenever the user is simply | |
478 | trying to rename files (because attempting to delete files that | |
479 | don't exist looks like a no-op; it takes some chicanery to remap | |
480 | deletes across file renames when the renames happen via arbitrary | |
481 | user-provided shell) | |
482 | ||
483 | ** even if you succeed at the map-deletes-for-renames chicanery, you | |
df5be016 JNA |
484 | still technically violate backward compatibility because users |
485 | are allowed to filter files in ways that depend upon topology of | |
486 | commits instead of filtering solely based on file contents or | |
487 | names (though this has not been observed in the wild). | |
9df53c5d EN |
488 | |
489 | * Even if you don't need to edit files but only want to e.g. rename or | |
df5be016 JNA |
490 | remove some and thus can avoid checking out each file (i.e. you can |
491 | use --index-filter), you still are passing shell snippets for your | |
492 | filters. This means that for every commit, you have to have a | |
493 | prepared git repo where those filters can be run. That's a | |
494 | significant setup. | |
495 | ||
496 | * Further, several additional files are created or updated per commit | |
497 | by git-filter-branch. Some of these are for supporting the | |
498 | convenience functions provided by git-filter-branch (such as map()), | |
499 | while others are for keeping track of internal state (but could have | |
500 | also been accessed by user filters; one of git-filter-branch's | |
501 | regression tests does so). This essentially amounts to using the | |
502 | filesystem as an IPC mechanism between git-filter-branch and the | |
503 | user-provided filters. Disks tend to be a slow IPC mechanism, and | |
504 | writing these files also effectively represents a forced | |
505 | synchronization point between separate processes that we hit with | |
506 | every commit. | |
9df53c5d EN |
507 | |
508 | * The user-provided shell commands will likely involve a pipeline of | |
df5be016 JNA |
509 | commands, resulting in the creation of many processes per commit. |
510 | Creating and running another process takes a widely varying amount | |
511 | of time between operating systems, but on any platform it is very | |
512 | slow relative to invoking a function. | |
9df53c5d EN |
513 | |
514 | * git-filter-branch itself is written in shell, which is kind of slow. | |
df5be016 JNA |
515 | This is the one performance issue that could be backward-compatibly |
516 | fixed, but compared to the above problems that are intrinsic to the | |
517 | design of git-filter-branch, the language of the tool itself is a | |
518 | relatively minor issue. | |
9df53c5d EN |
519 | |
520 | ** Side note: Unfortunately, people tend to fixate on the | |
521 | written-in-shell aspect and periodically ask if git-filter-branch | |
522 | could be rewritten in another language to fix the performance | |
523 | issues. Not only does that ignore the bigger intrinsic problems | |
524 | with the design, it'd help less than you'd expect: if | |
525 | git-filter-branch itself were not shell, then the convenience | |
526 | functions (map(), skip_commit(), etc) and the `--setup` argument | |
527 | could no longer be executed once at the beginning of the program | |
528 | but would instead need to be prepended to every user filter (and | |
529 | thus re-executed with every commit). | |
530 | ||
531 | The https://github.com/newren/git-filter-repo/[git filter-repo] tool is | |
532 | an alternative to git-filter-branch which does not suffer from these | |
533 | performance problems or the safety problems (mentioned below). For those | |
534 | with existing tooling which relies upon git-filter-branch, 'git | |
535 | repo-filter' also provides | |
536 | https://github.com/newren/git-filter-repo/blob/master/contrib/filter-repo-demos/filter-lamely[filter-lamely], | |
537 | a drop-in git-filter-branch replacement (with a few caveats). While | |
538 | filter-lamely suffers from all the same safety issues as | |
031fd4b9 | 539 | git-filter-branch, it at least ameliorates the performance issues a |
9df53c5d EN |
540 | little. |
541 | ||
542 | [[SAFETY]] | |
543 | SAFETY | |
544 | ------ | |
545 | ||
546 | git-filter-branch is riddled with gotchas resulting in various ways to | |
547 | easily corrupt repos or end up with a mess worse than what you started | |
548 | with: | |
549 | ||
550 | * Someone can have a set of "working and tested filters" which they | |
df5be016 JNA |
551 | document or provide to a coworker, who then runs them on a different |
552 | OS where the same commands are not working/tested (some examples in | |
553 | the git-filter-branch manpage are also affected by this). | |
554 | BSD vs. GNU userland differences can really bite. If lucky, error | |
555 | messages are spewed. But just as likely, the commands either don't | |
556 | do the filtering requested, or silently corrupt by making some | |
557 | unwanted change. The unwanted change may only affect a few commits, | |
558 | so it's not necessarily obvious either. (The fact that problems | |
559 | won't necessarily be obvious means they are likely to go unnoticed | |
560 | until the rewritten history is in use for quite a while, at which | |
561 | point it's really hard to justify another flag-day for another | |
562 | rewrite.) | |
9df53c5d EN |
563 | |
564 | * Filenames with spaces are often mishandled by shell snippets since | |
df5be016 JNA |
565 | they cause problems for shell pipelines. Not everyone is familiar |
566 | with find -print0, xargs -0, git-ls-files -z, etc. Even people who | |
567 | are familiar with these may assume such flags are not relevant | |
568 | because someone else renamed any such files in their repo back | |
569 | before the person doing the filtering joined the project. And | |
570 | often, even those familiar with handling arguments with spaces may | |
571 | not do so just because they aren't in the mindset of thinking about | |
572 | everything that could possibly go wrong. | |
573 | ||
574 | * Non-ascii filenames can be silently removed despite being in a | |
575 | desired directory. Keeping only wanted paths is often done using | |
576 | pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. | |
577 | ls-files will only quote filenames if needed, so folks may not | |
578 | notice that one of the files didn't match the regex (at least not | |
579 | until it's much too late). Yes, someone who knows about | |
580 | core.quotePath can avoid this (unless they have other special | |
581 | characters like \t, \n, or "), and people who use ls-files -z with | |
582 | something other than grep can avoid this, but that doesn't mean they | |
583 | will. | |
584 | ||
585 | * Similarly, when moving files around, one can find that filenames | |
586 | with non-ascii or special characters end up in a different | |
587 | directory, one that includes a double quote character. (This is | |
588 | technically the same issue as above with quoting, but perhaps an | |
589 | interesting different way that it can and has manifested as a | |
590 | problem.) | |
9df53c5d EN |
591 | |
592 | * It's far too easy to accidentally mix up old and new history. It's | |
df5be016 JNA |
593 | still possible with any tool, but git-filter-branch almost |
594 | invites it. If lucky, the only downside is users getting frustrated | |
595 | that they don't know how to shrink their repo and remove the old | |
596 | stuff. If unlucky, they merge old and new history and end up with | |
597 | multiple "copies" of each commit, some of which have unwanted or | |
598 | sensitive files and others which don't. This comes about in | |
599 | multiple different ways: | |
9df53c5d EN |
600 | |
601 | ** the default to only doing a partial history rewrite ('--all' is not | |
602 | the default and few examples show it) | |
603 | ||
604 | ** the fact that there's no automatic post-run cleanup | |
605 | ||
606 | ** the fact that --tag-name-filter (when used to rename tags) doesn't | |
607 | remove the old tags but just adds new ones with the new name | |
608 | ||
609 | ** the fact that little educational information is provided to inform | |
610 | users of the ramifications of a rewrite and how to avoid mixing old | |
611 | and new history. For example, this man page discusses how users | |
612 | need to understand that they need to rebase their changes for all | |
613 | their branches on top of new history (or delete and reclone), but | |
614 | that's only one of multiple concerns to consider. See the | |
615 | "DISCUSSION" section of the git filter-repo manual page for more | |
616 | details. | |
617 | ||
df5be016 JNA |
618 | * Annotated tags can be accidentally converted to lightweight tags, |
619 | due to either of two issues: | |
9df53c5d EN |
620 | |
621 | ** Someone can do a history rewrite, realize they messed up, restore | |
622 | from the backups in refs/original/, and then redo their | |
623 | git-filter-branch command. (The backup in refs/original/ is not a | |
624 | real backup; it dereferences tags first.) | |
625 | ||
626 | ** Running git-filter-branch with either --tags or --all in your | |
627 | <rev-list options>. In order to retain annotated tags as | |
628 | annotated, you must use --tag-name-filter (and must not have | |
629 | restored from refs/original/ in a previously botched rewrite). | |
630 | ||
631 | * Any commit messages that specify an encoding will become corrupted | |
df5be016 JNA |
632 | by the rewrite; git-filter-branch ignores the encoding, takes the |
633 | original bytes, and feeds it to commit-tree without telling it the | |
634 | proper encoding. (This happens whether or not --msg-filter is | |
635 | used.) | |
9df53c5d EN |
636 | |
637 | * Commit messages (even if they are all UTF-8) by default become | |
df5be016 JNA |
638 | corrupted due to not being updated -- any references to other commit |
639 | hashes in commit messages will now refer to no-longer-extant | |
640 | commits. | |
641 | ||
642 | * There are no facilities for helping users find what unwanted crud | |
643 | they should delete, which means they are much more likely to have | |
644 | incomplete or partial cleanups that sometimes result in confusion | |
645 | and people wasting time trying to understand. (For example, folks | |
646 | tend to just look for big files to delete instead of big directories | |
647 | or extensions, and once they do so, then sometime later folks using | |
648 | the new repository who are going through history will notice a build | |
649 | artifact directory that has some files but not others, or a cache of | |
650 | dependencies (node_modules or similar) which couldn't have ever been | |
651 | functional since it's missing some files.) | |
9df53c5d EN |
652 | |
653 | * If --prune-empty isn't specified, then the filtering process can | |
df5be016 | 654 | create hoards of confusing empty commits |
9df53c5d EN |
655 | |
656 | * If --prune-empty is specified, then intentionally placed empty | |
df5be016 JNA |
657 | commits from before the filtering operation are also pruned instead |
658 | of just pruning commits that became empty due to filtering rules. | |
9df53c5d | 659 | |
a6d39f2e | 660 | * If --prune-empty is specified, sometimes empty commits are missed |
df5be016 | 661 | and left around anyway (a somewhat rare bug, but it happens...) |
9df53c5d EN |
662 | |
663 | * A minor issue, but users who have a goal to update all names and | |
df5be016 JNA |
664 | emails in a repository may be led to --env-filter which will only |
665 | update authors and committers, missing taggers. | |
9df53c5d EN |
666 | |
667 | * If the user provides a --tag-name-filter that maps multiple tags to | |
df5be016 JNA |
668 | the same name, no warning or error is provided; git-filter-branch |
669 | simply overwrites each tag in some undocumented pre-defined order | |
670 | resulting in only one tag at the end. (A git-filter-branch | |
671 | regression test requires this surprising behavior.) | |
9df53c5d EN |
672 | |
673 | Also, the poor performance of git-filter-branch often leads to safety | |
674 | issues: | |
675 | ||
df5be016 JNA |
676 | * Coming up with the correct shell snippet to do the filtering you |
677 | want is sometimes difficult unless you're just doing a trivial | |
678 | modification such as deleting a couple files. Unfortunately, people | |
679 | often learn if the snippet is right or wrong by trying it out, but | |
680 | the rightness or wrongness can vary depending on special | |
681 | circumstances (spaces in filenames, non-ascii filenames, funny | |
682 | author names or emails, invalid timezones, presence of grafts or | |
683 | replace objects, etc.), meaning they may have to wait a long time, | |
684 | hit an error, then restart. The performance of git-filter-branch is | |
685 | so bad that this cycle is painful, reducing the time available to | |
686 | carefully re-check (to say nothing about what it does to the | |
687 | patience of the person doing the rewrite even if they do technically | |
688 | have more time available). This problem is extra compounded because | |
689 | errors from broken filters may not be shown for a long time and/or | |
690 | get lost in a sea of output. Even worse, broken filters often just | |
691 | result in silent incorrect rewrites. | |
692 | ||
693 | * To top it all off, even when users finally find working commands, | |
694 | they naturally want to share them. But they may be unaware that | |
695 | their repo didn't have some special cases that someone else's does. | |
696 | So, when someone else with a different repository runs the same | |
697 | commands, they get hit by the problems above. Or, the user just | |
698 | runs commands that really were vetted for special cases, but they | |
699 | run it on a different OS where it doesn't work, as noted above. | |
615b8f1a | 700 | |
c401b33c JS |
701 | GIT |
702 | --- | |
9e1f0a85 | 703 | Part of the linkgit:git[1] suite |