]>
Commit | Line | Data |
---|---|---|
1 | git-filter-branch(1) | |
2 | ==================== | |
3 | ||
4 | NAME | |
5 | ---- | |
6 | git-filter-branch - Rewrite branches | |
7 | ||
8 | SYNOPSIS | |
9 | -------- | |
10 | [verse] | |
11 | 'git filter-branch' [--setup <command>] [--subdirectory-filter <directory>] | |
12 | [--env-filter <command>] [--tree-filter <command>] | |
13 | [--index-filter <command>] [--parent-filter <command>] | |
14 | [--msg-filter <command>] [--commit-filter <command>] | |
15 | [--tag-name-filter <command>] [--prune-empty] | |
16 | [--original <namespace>] [-d <directory>] [-f | --force] | |
17 | [--state-branch <branch>] [--] [<rev-list options>...] | |
18 | ||
19 | DESCRIPTION | |
20 | ----------- | |
21 | Lets you rewrite Git revision history by rewriting the branches mentioned | |
22 | in the <rev-list options>, applying custom filters on each revision. | |
23 | Those filters can modify each tree (e.g. removing a file or running | |
24 | a perl rewrite on all files) or information about each commit. | |
25 | Otherwise, all information (including original commit times or merge | |
26 | information) will be preserved. | |
27 | ||
28 | The command will only rewrite the _positive_ refs mentioned in the | |
29 | command line (e.g. if you pass 'a..b', only 'b' will be rewritten). | |
30 | If you specify no filters, the commits will be recommitted without any | |
31 | changes, which would normally have no effect. Nevertheless, this may be | |
32 | useful in the future for compensating for some Git bugs or such, | |
33 | therefore such a usage is permitted. | |
34 | ||
35 | *NOTE*: This command honors `.git/info/grafts` file and refs in | |
36 | the `refs/replace/` namespace. | |
37 | If you have any grafts or replacement refs defined, running this command | |
38 | will make them permanent. | |
39 | ||
40 | *WARNING*! The rewritten history will have different object names for all | |
41 | the objects and will not converge with the original branch. You will not | |
42 | be able to easily push and distribute the rewritten branch on top of the | |
43 | original branch. Please do not use this command if you do not know the | |
44 | full implications, and avoid using it anyway, if a simple single commit | |
45 | would suffice to fix your problem. (See the "RECOVERING FROM UPSTREAM | |
46 | REBASE" section in linkgit:git-rebase[1] for further information about | |
47 | rewriting published history.) | |
48 | ||
49 | Always verify that the rewritten version is correct: The original refs, | |
50 | if different from the rewritten ones, will be stored in the namespace | |
51 | 'refs/original/'. | |
52 | ||
53 | Note that since this operation is very I/O expensive, it might | |
54 | be a good idea to redirect the temporary directory off-disk with the | |
55 | `-d` option, e.g. on tmpfs. Reportedly the speedup is very noticeable. | |
56 | ||
57 | ||
58 | Filters | |
59 | ~~~~~~~ | |
60 | ||
61 | The filters are applied in the order as listed below. The <command> | |
62 | argument is always evaluated in the shell context using the 'eval' command | |
63 | (with the notable exception of the commit filter, for technical reasons). | |
64 | Prior to that, the `$GIT_COMMIT` environment variable will be set to contain | |
65 | the id of the commit being rewritten. Also, GIT_AUTHOR_NAME, | |
66 | GIT_AUTHOR_EMAIL, GIT_AUTHOR_DATE, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, | |
67 | and GIT_COMMITTER_DATE are taken from the current commit and exported to | |
68 | the environment, in order to affect the author and committer identities of | |
69 | the replacement commit created by linkgit:git-commit-tree[1] after the | |
70 | filters have run. | |
71 | ||
72 | If any evaluation of <command> returns a non-zero exit status, the whole | |
73 | operation will be aborted. | |
74 | ||
75 | A 'map' function is available that takes an "original sha1 id" argument | |
76 | and outputs a "rewritten sha1 id" if the commit has been already | |
77 | rewritten, and "original sha1 id" otherwise; the 'map' function can | |
78 | return several ids on separate lines if your commit filter emitted | |
79 | multiple commits. | |
80 | ||
81 | ||
82 | OPTIONS | |
83 | ------- | |
84 | ||
85 | --setup <command>:: | |
86 | This is not a real filter executed for each commit but a one | |
87 | time setup just before the loop. Therefore no commit-specific | |
88 | variables are defined yet. Functions or variables defined here | |
89 | can be used or modified in the following filter steps except | |
90 | the commit filter, for technical reasons. | |
91 | ||
92 | --subdirectory-filter <directory>:: | |
93 | Only look at the history which touches the given subdirectory. | |
94 | The result will contain that directory (and only that) as its | |
95 | project root. Implies <<Remap_to_ancestor>>. | |
96 | ||
97 | --env-filter <command>:: | |
98 | This filter may be used if you only need to modify the environment | |
99 | in which the commit will be performed. Specifically, you might | |
100 | want to rewrite the author/committer name/email/time environment | |
101 | variables (see linkgit:git-commit-tree[1] for details). | |
102 | ||
103 | --tree-filter <command>:: | |
104 | This is the filter for rewriting the tree and its contents. | |
105 | The argument is evaluated in shell with the working | |
106 | directory set to the root of the checked out tree. The new tree | |
107 | is then used as-is (new files are auto-added, disappeared files | |
108 | are auto-removed - neither .gitignore files nor any other ignore | |
109 | rules *HAVE ANY EFFECT*!). | |
110 | ||
111 | --index-filter <command>:: | |
112 | This is the filter for rewriting the index. It is similar to the | |
113 | tree filter but does not check out the tree, which makes it much | |
114 | faster. Frequently used with `git rm --cached | |
115 | --ignore-unmatch ...`, see EXAMPLES below. For hairy | |
116 | cases, see linkgit:git-update-index[1]. | |
117 | ||
118 | --parent-filter <command>:: | |
119 | This is the filter for rewriting the commit's parent list. | |
120 | It will receive the parent string on stdin and shall output | |
121 | the new parent string on stdout. The parent string is in | |
122 | the format described in linkgit:git-commit-tree[1]: empty for | |
123 | the initial commit, "-p parent" for a normal commit and | |
124 | "-p parent1 -p parent2 -p parent3 ..." for a merge commit. | |
125 | ||
126 | --msg-filter <command>:: | |
127 | This is the filter for rewriting the commit messages. | |
128 | The argument is evaluated in the shell with the original | |
129 | commit message on standard input; its standard output is | |
130 | used as the new commit message. | |
131 | ||
132 | --commit-filter <command>:: | |
133 | This is the filter for performing the commit. | |
134 | If this filter is specified, it will be called instead of the | |
135 | 'git commit-tree' command, with arguments of the form | |
136 | "<TREE_ID> [(-p <PARENT_COMMIT_ID>)...]" and the log message on | |
137 | stdin. The commit id is expected on stdout. | |
138 | + | |
139 | As a special extension, the commit filter may emit multiple | |
140 | commit ids; in that case, the rewritten children of the original commit will | |
141 | have all of them as parents. | |
142 | + | |
143 | You can use the 'map' convenience function in this filter, and other | |
144 | convenience functions, too. For example, calling 'skip_commit "$@"' | |
145 | will leave out the current commit (but not its changes! If you want | |
146 | that, use 'git rebase' instead). | |
147 | + | |
148 | You can also use the `git_commit_non_empty_tree "$@"` instead of | |
149 | `git commit-tree "$@"` if you don't wish to keep commits with a single parent | |
150 | and that makes no change to the tree. | |
151 | ||
152 | --tag-name-filter <command>:: | |
153 | This is the filter for rewriting tag names. When passed, | |
154 | it will be called for every tag ref that points to a rewritten | |
155 | object (or to a tag object which points to a rewritten object). | |
156 | The original tag name is passed via standard input, and the new | |
157 | tag name is expected on standard output. | |
158 | + | |
159 | The original tags are not deleted, but can be overwritten; | |
160 | use "--tag-name-filter cat" to simply update the tags. In this | |
161 | case, be very careful and make sure you have the old tags | |
162 | backed up in case the conversion has run afoul. | |
163 | + | |
164 | Nearly proper rewriting of tag objects is supported. If the tag has | |
165 | a message attached, a new tag object will be created with the same message, | |
166 | author, and timestamp. If the tag has a signature attached, the | |
167 | signature will be stripped. It is by definition impossible to preserve | |
168 | signatures. The reason this is "nearly" proper, is because ideally if | |
169 | the tag did not change (points to the same object, has the same name, etc.) | |
170 | it should retain any signature. That is not the case, signatures will always | |
171 | be removed, buyer beware. There is also no support for changing the | |
172 | author or timestamp (or the tag message for that matter). Tags which point | |
173 | to other tags will be rewritten to point to the underlying commit. | |
174 | ||
175 | --prune-empty:: | |
176 | Some filters will generate empty commits that leave the tree untouched. | |
177 | This option instructs git-filter-branch to remove such commits if they | |
178 | have exactly one or zero non-pruned parents; merge commits will | |
179 | therefore remain intact. This option cannot be used together with | |
180 | `--commit-filter`, though the same effect can be achieved by using the | |
181 | provided `git_commit_non_empty_tree` function in a commit filter. | |
182 | ||
183 | --original <namespace>:: | |
184 | Use this option to set the namespace where the original commits | |
185 | will be stored. The default value is 'refs/original'. | |
186 | ||
187 | -d <directory>:: | |
188 | Use this option to set the path to the temporary directory used for | |
189 | rewriting. When applying a tree filter, the command needs to | |
190 | temporarily check out the tree to some directory, which may consume | |
191 | considerable space in case of large projects. By default it | |
192 | does this in the `.git-rewrite/` directory but you can override | |
193 | that choice by this parameter. | |
194 | ||
195 | -f:: | |
196 | --force:: | |
197 | 'git filter-branch' refuses to start with an existing temporary | |
198 | directory or when there are already refs starting with | |
199 | 'refs/original/', unless forced. | |
200 | ||
201 | --state-branch <branch>:: | |
202 | This option will cause the mapping from old to new objects to | |
203 | be loaded from named branch upon startup and saved as a new | |
204 | commit to that branch upon exit, enabling incremental of large | |
205 | trees. If '<branch>' does not exist it will be created. | |
206 | ||
207 | <rev-list options>...:: | |
208 | Arguments for 'git rev-list'. All positive refs included by | |
209 | these options are rewritten. You may also specify options | |
210 | such as `--all`, but you must use `--` to separate them from | |
211 | the 'git filter-branch' options. Implies <<Remap_to_ancestor>>. | |
212 | ||
213 | ||
214 | [[Remap_to_ancestor]] | |
215 | Remap to ancestor | |
216 | ~~~~~~~~~~~~~~~~~ | |
217 | ||
218 | By using linkgit:git-rev-list[1] arguments, e.g., path limiters, you can limit the | |
219 | set of revisions which get rewritten. However, positive refs on the command | |
220 | line are distinguished: we don't let them be excluded by such limiters. For | |
221 | this purpose, they are instead rewritten to point at the nearest ancestor that | |
222 | was not excluded. | |
223 | ||
224 | ||
225 | EXIT STATUS | |
226 | ----------- | |
227 | ||
228 | On success, the exit status is `0`. If the filter can't find any commits to | |
229 | rewrite, the exit status is `2`. On any other error, the exit status may be | |
230 | any other non-zero value. | |
231 | ||
232 | ||
233 | EXAMPLES | |
234 | -------- | |
235 | ||
236 | Suppose you want to remove a file (containing confidential information | |
237 | or copyright violation) from all commits: | |
238 | ||
239 | ------------------------------------------------------- | |
240 | git filter-branch --tree-filter 'rm filename' HEAD | |
241 | ------------------------------------------------------- | |
242 | ||
243 | However, if the file is absent from the tree of some commit, | |
244 | a simple `rm filename` will fail for that tree and commit. | |
245 | Thus you may instead want to use `rm -f filename` as the script. | |
246 | ||
247 | Using `--index-filter` with 'git rm' yields a significantly faster | |
248 | version. Like with using `rm filename`, `git rm --cached filename` | |
249 | will fail if the file is absent from the tree of a commit. If you | |
250 | want to "completely forget" a file, it does not matter when it entered | |
251 | history, so we also add `--ignore-unmatch`: | |
252 | ||
253 | -------------------------------------------------------------------------- | |
254 | git filter-branch --index-filter 'git rm --cached --ignore-unmatch filename' HEAD | |
255 | -------------------------------------------------------------------------- | |
256 | ||
257 | Now, you will get the rewritten history saved in HEAD. | |
258 | ||
259 | To rewrite the repository to look as if `foodir/` had been its project | |
260 | root, and discard all other history: | |
261 | ||
262 | ------------------------------------------------------- | |
263 | git filter-branch --subdirectory-filter foodir -- --all | |
264 | ------------------------------------------------------- | |
265 | ||
266 | Thus you can, e.g., turn a library subdirectory into a repository of | |
267 | its own. Note the `--` that separates 'filter-branch' options from | |
268 | revision options, and the `--all` to rewrite all branches and tags. | |
269 | ||
270 | To set a commit (which typically is at the tip of another | |
271 | history) to be the parent of the current initial commit, in | |
272 | order to paste the other history behind the current history: | |
273 | ||
274 | ------------------------------------------------------------------- | |
275 | git filter-branch --parent-filter 'sed "s/^\$/-p <graft-id>/"' HEAD | |
276 | ------------------------------------------------------------------- | |
277 | ||
278 | (if the parent string is empty - which happens when we are dealing with | |
279 | the initial commit - add graftcommit as a parent). Note that this assumes | |
280 | history with a single root (that is, no merge without common ancestors | |
281 | happened). If this is not the case, use: | |
282 | ||
283 | -------------------------------------------------------------------------- | |
284 | git filter-branch --parent-filter \ | |
285 | 'test $GIT_COMMIT = <commit-id> && echo "-p <graft-id>" || cat' HEAD | |
286 | -------------------------------------------------------------------------- | |
287 | ||
288 | or even simpler: | |
289 | ||
290 | ----------------------------------------------- | |
291 | git replace --graft $commit-id $graft-id | |
292 | git filter-branch $graft-id..HEAD | |
293 | ----------------------------------------------- | |
294 | ||
295 | To remove commits authored by "Darl McBribe" from the history: | |
296 | ||
297 | ------------------------------------------------------------------------------ | |
298 | git filter-branch --commit-filter ' | |
299 | if [ "$GIT_AUTHOR_NAME" = "Darl McBribe" ]; | |
300 | then | |
301 | skip_commit "$@"; | |
302 | else | |
303 | git commit-tree "$@"; | |
304 | fi' HEAD | |
305 | ------------------------------------------------------------------------------ | |
306 | ||
307 | The function 'skip_commit' is defined as follows: | |
308 | ||
309 | -------------------------- | |
310 | skip_commit() | |
311 | { | |
312 | shift; | |
313 | while [ -n "$1" ]; | |
314 | do | |
315 | shift; | |
316 | map "$1"; | |
317 | shift; | |
318 | done; | |
319 | } | |
320 | -------------------------- | |
321 | ||
322 | The shift magic first throws away the tree id and then the -p | |
323 | parameters. Note that this handles merges properly! In case Darl | |
324 | committed a merge between P1 and P2, it will be propagated properly | |
325 | and all children of the merge will become merge commits with P1,P2 | |
326 | as their parents instead of the merge commit. | |
327 | ||
328 | *NOTE* the changes introduced by the commits, and which are not reverted | |
329 | by subsequent commits, will still be in the rewritten branch. If you want | |
330 | to throw out _changes_ together with the commits, you should use the | |
331 | interactive mode of 'git rebase'. | |
332 | ||
333 | You can rewrite the commit log messages using `--msg-filter`. For | |
334 | example, 'git svn-id' strings in a repository created by 'git svn' can | |
335 | be removed this way: | |
336 | ||
337 | ------------------------------------------------------- | |
338 | git filter-branch --msg-filter ' | |
339 | sed -e "/^git-svn-id:/d" | |
340 | ' | |
341 | ------------------------------------------------------- | |
342 | ||
343 | If you need to add 'Acked-by' lines to, say, the last 10 commits (none | |
344 | of which is a merge), use this command: | |
345 | ||
346 | -------------------------------------------------------- | |
347 | git filter-branch --msg-filter ' | |
348 | cat && | |
349 | echo "Acked-by: Bugs Bunny <bunny@bugzilla.org>" | |
350 | ' HEAD~10..HEAD | |
351 | -------------------------------------------------------- | |
352 | ||
353 | The `--env-filter` option can be used to modify committer and/or author | |
354 | identity. For example, if you found out that your commits have the wrong | |
355 | identity due to a misconfigured user.email, you can make a correction, | |
356 | before publishing the project, like this: | |
357 | ||
358 | -------------------------------------------------------- | |
359 | git filter-branch --env-filter ' | |
360 | if test "$GIT_AUTHOR_EMAIL" = "root@localhost" | |
361 | then | |
362 | GIT_AUTHOR_EMAIL=john@example.com | |
363 | fi | |
364 | if test "$GIT_COMMITTER_EMAIL" = "root@localhost" | |
365 | then | |
366 | GIT_COMMITTER_EMAIL=john@example.com | |
367 | fi | |
368 | ' -- --all | |
369 | -------------------------------------------------------- | |
370 | ||
371 | To restrict rewriting to only part of the history, specify a revision | |
372 | range in addition to the new branch name. The new branch name will | |
373 | point to the top-most revision that a 'git rev-list' of this range | |
374 | will print. | |
375 | ||
376 | Consider this history: | |
377 | ||
378 | ------------------ | |
379 | D--E--F--G--H | |
380 | / / | |
381 | A--B-----C | |
382 | ------------------ | |
383 | ||
384 | To rewrite only commits D,E,F,G,H, but leave A, B and C alone, use: | |
385 | ||
386 | -------------------------------- | |
387 | git filter-branch ... C..H | |
388 | -------------------------------- | |
389 | ||
390 | To rewrite commits E,F,G,H, use one of these: | |
391 | ||
392 | ---------------------------------------- | |
393 | git filter-branch ... C..H --not D | |
394 | git filter-branch ... D..H --not C | |
395 | ---------------------------------------- | |
396 | ||
397 | To move the whole tree into a subdirectory, or remove it from there: | |
398 | ||
399 | --------------------------------------------------------------- | |
400 | git filter-branch --index-filter \ | |
401 | 'git ls-files -s | sed "s-\t\"*-&newsubdir/-" | | |
402 | GIT_INDEX_FILE=$GIT_INDEX_FILE.new \ | |
403 | git update-index --index-info && | |
404 | mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD | |
405 | --------------------------------------------------------------- | |
406 | ||
407 | ||
408 | ||
409 | CHECKLIST FOR SHRINKING A REPOSITORY | |
410 | ------------------------------------ | |
411 | ||
412 | git-filter-branch can be used to get rid of a subset of files, | |
413 | usually with some combination of `--index-filter` and | |
414 | `--subdirectory-filter`. People expect the resulting repository to | |
415 | be smaller than the original, but you need a few more steps to | |
416 | actually make it smaller, because Git tries hard not to lose your | |
417 | objects until you tell it to. First make sure that: | |
418 | ||
419 | * You really removed all variants of a filename, if a blob was moved | |
420 | over its lifetime. `git log --name-only --follow --all -- filename` | |
421 | can help you find renames. | |
422 | ||
423 | * You really filtered all refs: use `--tag-name-filter cat -- --all` | |
424 | when calling git-filter-branch. | |
425 | ||
426 | Then there are two ways to get a smaller repository. A safer way is | |
427 | to clone, that keeps your original intact. | |
428 | ||
429 | * Clone it with `git clone file:///path/to/repo`. The clone | |
430 | will not have the removed objects. See linkgit:git-clone[1]. (Note | |
431 | that cloning with a plain path just hardlinks everything!) | |
432 | ||
433 | If you really don't want to clone it, for whatever reasons, check the | |
434 | following points instead (in this order). This is a very destructive | |
435 | approach, so *make a backup* or go back to cloning it. You have been | |
436 | warned. | |
437 | ||
438 | * Remove the original refs backed up by git-filter-branch: say `git | |
439 | for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git | |
440 | update-ref -d`. | |
441 | ||
442 | * Expire all reflogs with `git reflog expire --expire=now --all`. | |
443 | ||
444 | * Garbage collect all unreferenced objects with `git gc --prune=now` | |
445 | (or if your git-gc is not new enough to support arguments to | |
446 | `--prune`, use `git repack -ad; git prune` instead). | |
447 | ||
448 | NOTES | |
449 | ----- | |
450 | ||
451 | git-filter-branch allows you to make complex shell-scripted rewrites | |
452 | of your Git history, but you probably don't need this flexibility if | |
453 | you're simply _removing unwanted data_ like large files or passwords. | |
454 | For those operations you may want to consider | |
455 | http://rtyley.github.io/bfg-repo-cleaner/[The BFG Repo-Cleaner], | |
456 | a JVM-based alternative to git-filter-branch, typically at least | |
457 | 10-50x faster for those use-cases, and with quite different | |
458 | characteristics: | |
459 | ||
460 | * Any particular version of a file is cleaned exactly _once_. The BFG, | |
461 | unlike git-filter-branch, does not give you the opportunity to | |
462 | handle a file differently based on where or when it was committed | |
463 | within your history. This constraint gives the core performance | |
464 | benefit of The BFG, and is well-suited to the task of cleansing bad | |
465 | data - you don't care _where_ the bad data is, you just want it | |
466 | _gone_. | |
467 | ||
468 | * By default The BFG takes full advantage of multi-core machines, | |
469 | cleansing commit file-trees in parallel. git-filter-branch cleans | |
470 | commits sequentially (i.e. in a single-threaded manner), though it | |
471 | _is_ possible to write filters that include their own parallelism, | |
472 | in the scripts executed against each commit. | |
473 | ||
474 | * The http://rtyley.github.io/bfg-repo-cleaner/#examples[command options] | |
475 | are much more restrictive than git-filter branch, and dedicated just | |
476 | to the tasks of removing unwanted data- e.g: | |
477 | `--strip-blobs-bigger-than 1M`. | |
478 | ||
479 | GIT | |
480 | --- | |
481 | Part of the linkgit:git[1] suite |