Outline:
- 0. Assumptions
+ 1. Assumptions
- 1. How rebasing and cherry-picking work
+ 2. How rebasing and cherry-picking work
- 2. Why the renames on MERGE_SIDE1 in any given pick are *always* a
+ 3. Why the renames on MERGE_SIDE1 in any given pick are *always* a
superset of the renames on MERGE_SIDE1 for the next pick.
- 3. Why any rename on MERGE_SIDE1 in any given pick is _almost_ always also
+ 4. Why any rename on MERGE_SIDE1 in any given pick is _almost_ always also
a rename on MERGE_SIDE1 for the next pick
- 4. A detailed description of the counter-examples to #3.
+ 5. A detailed description of the counter-examples to #4.
- 5. Why the special cases in #4 are still fully reasonable to use to pair
+ 6. Why the special cases in #5 are still fully reasonable to use to pair
up files for three-way content merging in the merge machinery, and why
they do not affect the correctness of the merge.
- 6. Interaction with skipping of "irrelevant" renames
+ 7. Interaction with skipping of "irrelevant" renames
- 7. Additional items that need to be cached
+ 8. Additional items that need to be cached
- 8. How directory rename detection interacts with the above and why this
+ 9. How directory rename detection interacts with the above and why this
optimization is still safe even if merge.directoryRenames is set to
"true".
-=== 0. Assumptions ===
+== 1. Assumptions ==
There are two assumptions that will hold throughout this document:
* All merges are fully automatic
-and a third that will hold in sections 2-5 for simplicity, that I'll later
-address in section 8:
+and a third that will hold in sections 3-6 for simplicity, that I'll later
+address in section 9:
* No directory renames occur
stored on disk, and thus is thrown away as soon as the rebase or cherry
pick stops for the user to resolve the operation.
-The third assumption makes sections 2-5 simpler, and allows people to
+The third assumption makes sections 3-6 simpler, and allows people to
understand the basics of why this optimization is safe and effective, and
-then I can go back and address the specifics in section 8. It is probably
+then I can go back and address the specifics in section 9. It is probably
also worth noting that if directory renames do occur, then the default of
merge.directoryRenames being set to "conflict" means that the operation
will stop for users to resolve the conflicts and the cache will be thrown
users will have set merge.directoryRenames to "true" to allow the merges to
continue to proceed automatically. The optimization is still safe with
this config setting, but we have to discuss a few more cases to show why;
-this discussion is deferred until section 8.
+this discussion is deferred until section 9.
-=== 1. How rebasing and cherry-picking work ===
+== 2. How rebasing and cherry-picking work ==
Consider the following setup (from the git-rebase manpage):
+------------
A---B---C topic
/
D---E---F---G main
+------------
After rebasing or cherry-picking topic onto main, this will appear as:
+------------
A'--B'--C' topic
/
D---E---F---G main
+------------
The way the commits A', B', and C' are created is through a series of
merges, where rebase or cherry-pick sequentially uses each of the three
in the merge operation as MERGE_BASE, MERGE_SIDE1, and MERGE_SIDE2. For
this picture, the three commits for each of the three merges would be:
+....
To create A':
MERGE_BASE: E
MERGE_SIDE1: G
MERGE_BASE: B
MERGE_SIDE1: B'
MERGE_SIDE2: C
+....
Sometimes, folks are surprised that these three-way merges are done. It
can be useful in understanding these three-way merges to view them in a
B, B', and C, at least the parts before you decide to record a commit.
-=== 2. Why the renames on MERGE_SIDE1 in any given pick are always a ===
-=== superset of the renames on MERGE_SIDE1 for the next pick. ===
+== 3. Why the renames on MERGE_SIDE1 in any given pick are always a superset of the renames on MERGE_SIDE1 for the next pick. ==
The merge machinery uses the filenames it is fed from MERGE_BASE,
MERGE_SIDE1, and MERGE_SIDE2. It will only move content to a different
First, let's remember what commits are involved in the first and second
picks of the cherry-pick or rebase sequence:
+....
To create A':
MERGE_BASE: E
MERGE_SIDE1: G
MERGE_BASE: A
MERGE_SIDE1: A'
MERGE_SIDE2: B
+....
So, in particular, we need to show that the renames between E and G are a
superset of those between A and A'.
and G are a superset of those between A and A'.
-=== 3. Why any rename on MERGE_SIDE1 in any given pick is _almost_ ===
-=== always also a rename on MERGE_SIDE1 for the next pick. ===
+== 4. Why any rename on MERGE_SIDE1 in any given pick is _almost_ always also a rename on MERGE_SIDE1 for the next pick. ==
Let's again look at the first two picks:
+....
To create A':
MERGE_BASE: E
MERGE_SIDE1: G
MERGE_BASE: A
MERGE_SIDE1: A'
MERGE_SIDE2: B
+....
Now let's look at any given rename from MERGE_SIDE1 of the first pick, i.e.
any given rename from E to G. Let's use the filenames 'oldfile' and
'newfile' for demonstration purposes. That first pick will function as
follows; when the rename is detected, the merge machinery will do a
three-way content merge of the following:
+
+....
E:oldfile
G:newfile
A:oldfile
+....
+
and produce a new result:
+
+....
A':newfile
+....
Note above that I've assumed that E->A did not rename oldfile. If that
side did rename, then we most likely have a rename/rename(1to2) conflict
detectable as renames almost always.
-=== 4. A detailed description of the counter-examples to #3. ===
+== 5. A detailed description of the counter-examples to #4. ==
-We already noted in section 3 that rename/rename(1to1) (i.e. both sides
+We already noted in section 4 that rename/rename(1to1) (i.e. both sides
renaming a file the same way) was one counter-example. The more
interesting bit, though, is why did we need to use the "almost" qualifier
when stating that A:oldfile and A':newfile are "almost" always detectable
as renames?
-Let's repeat an earlier point that section 3 made:
+Let's repeat an earlier point that section 4 made:
+....
A':newfile was created by applying the changes between E:oldfile and
G:newfile to A:oldfile. The changes between E:oldfile and G:newfile were
<50% of the size of E:oldfile.
+....
If those changes that were <50% of the size of E:oldfile are also <50% of
the size of A:oldfile, then A:oldfile and A':newfile will be detectable as
detect A:oldfile and A':newfile as renames.
Here's an example where that can happen:
+
* E:oldfile had 20 lines
* G:newfile added 10 new lines at the beginning of the file
* A:oldfile kept the first 3 lines of the file, and deleted all the rest
+
then
+
+....
=> A':newfile would have 13 lines, 3 of which matches those in A:oldfile.
-E:oldfile -> G:newfile would be detected as a rename, but A:oldfile and
-A':newfile would not be.
+ E:oldfile -> G:newfile would be detected as a rename, but A:oldfile and
+ A':newfile would not be.
+....
-=== 5. Why the special cases in #4 are still fully reasonable to use to ===
-=== pair up files for three-way content merging in the merge machinery, ===
-=== and why they do not affect the correctness of the merge. ===
+== 6. Why the special cases in #5 are still fully reasonable to use to pair up files for three-way content merging in the merge machinery, and why they do not affect the correctness of the merge. ==
In the rename/rename(1to1) case, A:newfile and A':newfile are not renames
since they use the *same* filename. However, files with the same filename
machinery has never employed break detection). The interesting
counter-example case is thus not the rename/rename(1to1) case, but the case
where A did not rename oldfile. That was the case that we spent most of
-the time discussing in sections 3 and 4. The remainder of this section
+the time discussing in sections 4 and 5. The remainder of this section
will be devoted to that case as well.
So, even if A:oldfile and A':newfile aren't detectable as renames, why is
it still reasonable to pair them up for three-way content merging in the
merge machinery? There are multiple reasons:
- * As noted in sections 3 and 4, the diff between A:oldfile and A':newfile
+ * As noted in sections 4 and 5, the diff between A:oldfile and A':newfile
is *exactly* the same as the diff between E:oldfile and G:newfile. The
latter pair were detected as renames, so it seems unlikely to surprise
users for us to treat A:oldfile and A':newfile as renames.
optimization than without.
-=== 6. Interaction with skipping of "irrelevant" renames ===
+== 7. Interaction with skipping of "irrelevant" renames ==
Previous optimizations involved skipping rename detection for paths
considered to be "irrelevant". See for example the following commits:
already detected renames.
-=== 7. Additional items that need to be cached ===
+== 8. Additional items that need to be cached ==
It turns out we have to cache more than just renames; we also cache:
+....
A) non-renames (i.e. unpaired deletes)
B) counts of renames within directories
C) sources that were marked as RELEVANT_LOCATION, but which were
downgraded to RELEVANT_NO_MORE
D) the toplevel trees involved in the merge
+....
These are all stored in struct rename_info, and respectively appear in
+
* cached_pairs (along side actual renames, just with a value of NULL)
* dir_rename_counts
* cached_irrelevant
* merge_trees
-The reason for (A) comes from the irrelevant renames skipping
-optimization discussed in section 6. The fact that irrelevant renames
+The reason for `(A)` comes from the irrelevant renames skipping
+optimization discussed in section 7. The fact that irrelevant renames
are skipped means we only get a subset of the potential renames
detected and subsequent commits may need to run rename detection on
the upstream side on a subset of the remaining renames (to get the
repeatedly check that those paths remain unpaired on the upstream side
with every commit we are transplanting.
-The reason for (B) is that diffcore_rename_extended() is what
+The reason for `(B)` is that diffcore_rename_extended() is what
generates the counts of renames by directory which is needed in
directory rename detection, and if we don't run
diffcore_rename_extended() again then we need to have the output from
it, including dir_rename_counts, from the previous run.
-The reason for (C) is that merge-ort's tree traversal will again think
+The reason for `(C)` is that merge-ort's tree traversal will again think
those paths are relevant (marking them as RELEVANT_LOCATION), but the
fact that they were downgraded to RELEVANT_NO_MORE means that
dir_rename_counts already has the information we need for directory
rename detection. (A path which becomes RELEVANT_CONTENT in a
subsequent commit will be removed from cached_irrelevant.)
-The reason for (D) is that is how we determine whether the remember
+The reason for `(D)` is that is how we determine whether the remember
renames optimization can be used. In particular, remembering that our
sequence of merges looks like:
+....
Merge 1:
MERGE_BASE: E
MERGE_SIDE1: G
MERGE_SIDE1: A'
MERGE_SIDE2: B
=> Creates B'
+....
It is the fact that the trees A and A' appear both in Merge 1 and in
Merge 2, with A as a parent of A' that allows this optimization. So
time.
-=== 8. How directory rename detection interacts with the above and ===
-=== why this optimization is still safe even if ===
-=== merge.directoryRenames is set to "true". ===
+== 9. How directory rename detection interacts with the above and why this optimization is still safe even if merge.directoryRenames is set to "true". ==
As noted in the assumptions section:
+....
"""
...if directory renames do occur, then the default of
merge.directoryRenames being set to "conflict" means that the operation
is that some users will have set merge.directoryRenames to "true" to
allow the merges to continue to proceed automatically.
"""
+....
Let's remember that we need to look at how any given pick affects the next
one. So let's again use the first two picks from the diagram in section
one:
+....
First pick does this three-way merge:
MERGE_BASE: E
MERGE_SIDE1: G
MERGE_SIDE1: A'
MERGE_SIDE2: B
=> creates B'
+....
Now, directory rename detection exists so that if one side of history
renames a directory, and the other side adds a new file to the old
concerned; see the assumptions section). Two interesting sub-notes
about these counts:
- * If we need to perform rename-detection again on the given side (e.g.
+ ** If we need to perform rename-detection again on the given side (e.g.
some paths are relevant for rename detection that weren't before),
then we clear dir_rename_counts and recompute it, making use of
cached_pairs. The reason it is important to do this is optimizations
easiest way to "fix up" dir_rename_counts in such cases is to just
recompute it.
- * If we prune rename/rename(1to1) entries from the cache, then we also
+ ** If we prune rename/rename(1to1) entries from the cache, then we also
need to update dir_rename_counts to decrement the counts for the
involved directory and any relevant parent directories (to undo what
update_dir_rename_counts() in diffcore-rename.c incremented when the
Case 1: MERGE_SIDE1 renames old dir, MERGE_SIDE2 adds new file to old dir
+....
This case looks like this:
MERGE_BASE: E, Has olddir/
* MERGE_SIDE1 has cached olddir/newfile -> newdir/newfile
Given the cached rename noted above, the second merge can proceed as
expected without needing to perform rename detection from A -> A'.
+....
Case 2: MERGE_SIDE1 renames old dir, MERGE_SIDE2 renames file into old dir
+....
This case looks like this:
+
MERGE_BASE: E oldfile, olddir/
MERGE_SIDE1: G oldfile, olddir/ -> newdir/
MERGE_SIDE2: A oldfile -> olddir/newfile
Given the cached rename noted above, the second merge can proceed as
expected without needing to perform rename detection from A -> A'.
+....
Case 3: MERGE_SIDE1 adds new file to old dir, MERGE_SIDE2 renames old dir
+....
This case looks like this:
MERGE_BASE: E, Has olddir/
In this case, with the optimization, note that after the first commit there
were no renames on MERGE_SIDE1, and any renames on MERGE_SIDE2 are tossed.
But the second merge didn't need any renames so this is fine.
+....
Case 4: MERGE_SIDE1 renames file into old dir, MERGE_SIDE2 renames old dir
+....
This case looks like this:
MERGE_BASE: E, Has olddir/
Given the cached rename noted above, the second merge can proceed as
expected without needing to perform rename detection from A -> A'.
+....
Finally, I'll just note here that interactions with the
skip-irrelevant-renames optimization means we sometimes don't detect
projects / thirdparty / git.git / commitdiff
