commit-graph verify: detect inability to read the graph
Change "commit-graph verify" to error on open() failures other than
ENOENT. As noted in the third paragraph of 283e68c72f ("commit-graph:
add 'verify' subcommand", 2018-06-27) and the test it added it's
intentional that "commit-graph verify" doesn't error out when the file
doesn't exist.
But let's not be overly promiscuous in what we accept. If we can't
read the file for other reasons, e.g. permission errors, bad file
descriptor etc. we'd like to report an error to the user.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
commit-graph: don't pass filename to load_commit_graph_one_fd_st()
An earlier change implemented load_commit_graph_one_fd_st() in a way
that was bug-compatible with earlier code in terms of the "graph file
%s is too small" error message printing out the path to the
commit-graph (".git/objects/info/commit-graph").
But change that, because:
* A function that takes an already-open file descriptor also needing
the filename isn't very intuitive.
* The vast majority of errors we might emit when loading the graph
come from parse_commit_graph(), which doesn't report the
filename. Let's not do that either in this case for consistency.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
commit-graph: don't early exit(1) on e.g. "git status"
Make the commit-graph loading code work as a library that returns an
error code instead of calling exit(1) when the commit-graph is
corrupt. This means that e.g. "status" will now report commit-graph
corruption as an "error: [...]" at the top of its output, but then
proceed to work normally.
This required splitting up the load_commit_graph_one() function so
that the code that deals with open()-ing and stat()-ing the graph can
now be called independently as open_commit_graph().
This is needed because "commit-graph verify" where the graph doesn't
exist isn't an error. See the third paragraph in 283e68c72f ("commit-graph: add 'verify' subcommand",
2018-06-27). There's a bug in that logic where we conflate the
intended ENOENT with other errno values (e.g. EACCES), but this change
doesn't address that. That'll be addressed in a follow-up change.
I'm then splitting most of the logic out of load_commit_graph_one()
into load_commit_graph_one_fd_st(), which allows for providing an
existing file descriptor and stat information to the loading
code. This isn't strictly needed, but it would be redundant and
confusing to open() and stat() the file twice for some of the
codepaths, this allows for calling open_commit_graph() followed by
load_commit_graph_one_fd_st(). The "graph_file" still needs to be
passed to that function for the the "graph file %s is too small" error
message.
This leaves load_commit_graph_one() unused by everything except the
internal prepare_commit_graph_one() function, so let's mark it as
"static". If someone needs it in the future we can remove the "static"
attribute. I could also rewrite its sole remaining
user ("prepare_commit_graph_one()") to use
load_commit_graph_one_fd_st() instead, but let's leave it at this.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
When core.commitGraph=true is set, various common commands now consult
the commit graph. Because the commit-graph code is very trusting of
its input data, it's possibly to construct a graph that'll cause an
immediate segfault on e.g. "status" (and e.g. "log", "blame", ...). In
some other cases where git immediately exits with a cryptic error
about the graph being broken.
The root cause of this is that while the "commit-graph verify"
sub-command exhaustively verifies the graph, other users of the graph
simply trust the graph, and will e.g. deference data found at certain
offsets as pointers, causing segfaults.
This change does the bare minimum to ensure that we don't segfault in
the common fill_commit_in_graph() codepath called by
e.g. setup_revisions(), to do this instrument the "commit-graph
verify" tests to always check if "status" would subsequently
segfault. This fixes the following tests which would previously
segfault:
not ok 50 - detect low chunk count
not ok 51 - detect missing OID fanout chunk
not ok 52 - detect missing OID lookup chunk
not ok 53 - detect missing commit data chunk
Those happened because with the commit-graph enabled setup_revisions()
would eventually call fill_commit_in_graph(), where e.g.
g->chunk_commit_data is used early as an offset (and will be
0x0). With this change we get far enough to detect that the graph is
broken, and show an error instead. E.g.:
$ git status; echo $?
error: commit-graph is missing the Commit Data chunk
1
That also sucks, we should *warn* and not hard-fail "status" just
because the commit-graph is corrupt, but fixing is left to a follow-up
change.
A side-effect of changing the reporting from graph_report() to error()
is that we now have an "error: " prefix for these even for
"commit-graph verify". Pseudo-diff before/after:
$ git commit-graph verify
-commit-graph is missing the Commit Data chunk
+error: commit-graph is missing the Commit Data chunk
Changing that is OK. Various errors it emits now early on are prefixed
with "error: ", moving these over and changing the output doesn't
break anything.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Elijah Newren [Wed, 20 Feb 2019 22:58:46 +0000 (14:58 -0800)]
fast-import: fix erroneous handling of get-mark with empty orphan commits
When get-mark was introduced in commit 28c7b1f7b7b7 ("fast-import: add a
get-mark command", 2015-07-01), it followed the precedent of the
cat-blob command to be allowed on any line other than in the middle of a
data directive; see commit 777f80d7429b ("fast-import: Allow cat-blob
requests at arbitrary points in stream", 2010-11-28). It was useful to
allow cat-blob directives in the middle of a commit to get more data
that would be used in writing the current commit object. get-mark is
not similarly useful since fast-import can already use either object id
or mark. Further, trying to allow this command anywhere caused parsing
bugs. Fix the parsing problems by only allowing get-mark commands to
appear when other commands have completed.
Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Elijah Newren [Wed, 20 Feb 2019 22:58:45 +0000 (14:58 -0800)]
fast-import: only allow cat-blob requests where it makes sense
In commit 777f80d7429b ("fast-import: Allow cat-blob requests at
arbitrary points in stream", 2010-11-28), fast-import started allowing
cat-blob commands to appear on the start of any line except in the
middle of a "data" command. It could be in the middle of various
directives that were part of a tag command, or in the middle of
checkpoints or progresses (each of which allow an optional second empty
newline), or even immediately after the mark command of a blob before
the data directive appeared (raising the question of what if it used the
mark for the blob that just barely appeared in the stream that we do not
yet have the data for). None of these locations make any sense as
places to put cat-blob requests.
The purpose of this change as stated in that commit message was to
[save] frontends from having to loop over everything they want to
commit in the next commit and cat-ing the necessary objects in
advance.
However, that can be achieved by simply allowing cat-blob requests to
appear whenever a filemodify directive is allowed. Further, it avoids
setting a bad precedent for other commands to follow (e.g. get-mark); a
precedent which caused parsing problems in corner cases.
Technically, inline filemodify directives add a slight wrinkle in that
frontends might want to have cat-blob directives appear after the start
of the filemodify and before the data directive contained within it. I
think it would have been better to disallow such a case (it would be
trivial to use cat-blob before the filemodify instead), but since there
is evidence this was used, for backwards compatibility let's support
that case too.
Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Elijah Newren [Wed, 20 Feb 2019 22:58:44 +0000 (14:58 -0800)]
fast-import: check most prominent commands first
This is not a very important change, and one that I expect to have no
performance impact whatsoever, but reading the code bothered me. The
parsing of command types in cmd_main() mostly runs in order of most
common to least common commands; sure, it's hard to say for sure what
the most common are without some type of study, but it seems fairly
clear to mark the original four ("blob", "commit", "tag", "reset") as
the most prominent. Indeed, the parsing for most other commands were
added to later in the list. However, when "ls" was added, it was stuck
near the top of the list, with no rationale for that particular
location. Move it down to later to appease my Tourette's-like internal
twitching that its former location was causing.
Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:26 +0000 (00:05 +0000)]
gitweb: make hash size independent
Gitweb has several hard-coded 40 values throughout it to check for
values that are passed in or acquired from Git. To simplify the code,
introduce a regex variable that matches either exactly 40 or exactly 64
hex characters, and use this variable anywhere we would have previously
hard-coded a 40 in a regex.
Add some helper functions which allow us to write tighter regexes that
match exactly the number of hex characters we're expecting.
Similarly, switch the code that looks for deleted diffinfo information
to look for either 40 or 64 zeros, and update one piece of code to use
this function. Finally, when formatting a log line, allow an
abbreviated describe output to contain up to 64 characters.
Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:25 +0000 (00:05 +0000)]
Git.pm: make hash size independent
The cat_blob function was matching on exactly 40 hex characters. This
won't work with SHA-256, which uses 64-character hex object IDs. While
it should be fine to simply match any number of hex characters since the
output is space delimited, be extra safe by matching either exactly 40
or exactly 64 hex characters.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:24 +0000 (00:05 +0000)]
read-cache: read data in a hash-independent way
Index entries are structured with a variety of fields up front, followed
by a hash and one or two flags fields. Because the hash field is stored
in the middle of the structure, it's difficult to use one fixed-size
structure that easily allows access to the hash and flags fields.
Adjust the structure to hold the maximum amount of data that may be
needed using a member called "data" and read and write this field
independently in the various places that need to read and write the
structure.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:23 +0000 (00:05 +0000)]
dir: make untracked cache extension hash size independent
Instead of using a struct with a flex array member to read and write the
untracked cache extension, use a shorter, fixed-length struct and add
the name and hash data explicitly.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:22 +0000 (00:05 +0000)]
builtin/difftool: use parse_oid_hex
Instead of using get_oid_hex and adding constants to the result, use
parse_oid_hex to make this code independent of the hash size.
Additionally, correct a typo that would cause us to print one too few
characters on error, since we will already have incremented the pointer
to point to the beginning of the object ID before we get to printing the
error message.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:20 +0000 (00:05 +0000)]
archive: convert struct archiver_args to object_id
Change the commit_sha1 member to be called "commit_oid" and change it to
be a pointer to struct object_id. Additionally, update some uses of
GIT_SHA1_HEXSZ and hard-coded values to use the_hash_algo instead.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Rene Scharfe [Tue, 19 Feb 2019 00:05:18 +0000 (00:05 +0000)]
get-tar-commit-id: parse comment record
Parse pax comment records properly and get rid of magic numbers for
acceptable comment length. This simplifies a later change to handle
longer hashes.
Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:17 +0000 (00:05 +0000)]
hash: add a function to lookup hash algorithm by length
There are some cases, such as the dumb HTTP transport and bundles, where
we can only determine the hash algorithm in use by the length of the
object IDs. Provide a function that looks up the algorithm by length.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:16 +0000 (00:05 +0000)]
remote-curl: make hash size independent
Change one hard-coded use of the constant 40 to a reference to
the_hash_algo. In addition, switch a use of get_oid_hex to
parse_oid_hex to avoid the need to use a constant.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:12 +0000 (00:05 +0000)]
http-walker: replace sha1_to_hex
Since sha1_to_hex is limited to SHA-1, replace the uses of it in this
file with hash_to_hex. Rename several variables accordingly to reflect
that they are no longer limited to SHA-1.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:11 +0000 (00:05 +0000)]
http-push: remove remaining uses of sha1_to_hex
Since sha1_to_hex is limited to SHA-1, switch all remaining uses of it
in this file to hash_to_hex or oid_to_hex. Modify update_remote to take
a pointer to struct object_id, and since we don't modify that parameter
in the function, set it to be const as well.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:10 +0000 (00:05 +0000)]
http-backend: allow 64-character hex names
In an SHA-256-backed repository using the http-backend handler for dumb
protocol clients, it may be necessary to access the raw packs using
their full SHA-256-specified names. Allow packs and loose objects to be
accessed using their full SHA-256-specified 64-character hex names.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:09 +0000 (00:05 +0000)]
http-push: convert to use the_hash_algo
Switch the lock token code to use the_hash_algo and increase its buffers
to be allocated using GIT_MAX_* constants. Update the parsing of object
paths to use the_hash_algo as well.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:06 +0000 (00:05 +0000)]
fast-import: replace sha1_to_hex
Replace the uses of sha1_to_hex in this function with hash_to_hex to
allow the use of SHA-256 as well. Rename a variable since it is no
longer limited to SHA-1.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:05 +0000 (00:05 +0000)]
fast-import: make hash-size independent
Replace several uses of GIT_SHA1_HEXSZ and 40-based constants with
references to the_hash_algo. Update the note handling code here to
compute path sizes based on GIT_MAX_RAWSZ as well.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:04 +0000 (00:05 +0000)]
builtin/name-rev: make hash-size independent
Use the_hash_algo when parsing instead of GIT_SHA1_HEXSZ so that this
function works with any size hash. Rename the variable forty to
counter, as this is a better name and is independent of the hash size.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:03 +0000 (00:05 +0000)]
object-store: rename and expand packed_git's sha1 member
This member is used to represent the pack checksum of the pack in
question. Expand this member to be GIT_MAX_RAWSZ bytes in length so it
works with longer hashes and rename it to be "hash" instead of "sha1".
This transformation was made with a change to the definition and the
following semantic patch:
brian m. carlson [Tue, 19 Feb 2019 00:05:02 +0000 (00:05 +0000)]
notes: replace sha1_to_hex
Replace the uses of sha1_to_hex in this function with hash_to_hex to
allow the use of SHA-256 as well. Rename some variables since this code
is no longer limited to SHA-1.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:05:01 +0000 (00:05 +0000)]
notes: make hash size independent
Switch out various uses of the GIT_SHA1_* constants with GIT_MAX_*
constants for allocations and the_hash_algo for general parsing. Update
a comment to no longer be SHA-1 specific.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:04:58 +0000 (00:04 +0000)]
pack-bitmap: switch hash tables to use struct object_id
Instead of storing unsigned char pointers in the hash tables, switch to
storing instances of struct object_id. Update several internal functions
and one external function to take pointers to struct object_id.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:04:56 +0000 (00:04 +0000)]
pack-bitmap: replace sha1_to_hex
Replace the uses of sha1_to_hex in the pack bitmap code with hash_to_hex
to allow the use of SHA-256 as well. Rename a few variables since they
are no longer limited to SHA-1.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:04:54 +0000 (00:04 +0000)]
pack-bitmap: make bitmap header handling hash agnostic
Increase the checksum field in struct bitmap_disk_header to be
GIT_MAX_RAWSZ bytes in length and ensure that we hash the proper number
of bytes out when computing the bitmap checksum.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
brian m. carlson [Tue, 19 Feb 2019 00:04:53 +0000 (00:04 +0000)]
khash: move oid hash table definition
Move the oid khash table definition to khash.h and define a typedef for
it, similar to the one we have for unsigned char pointers. Define
variants that are maps as well.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
diff_opt_parse() is a heavy hammer to just set diff filter. But it's
the only way because of the diff_status_letters[] mapping. Add a new
API to set diff filter and use it in git-am. diff_opt_parse()'s only
remaining call site in revision.c will be gone soon and having it here
just because of git-am does not make sense.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
range-diff: use parse_options() instead of diff_opt_parse()
Diff's internal option parsing is now done with 'struct option', which
makes it possible to combine all diff options to range-diff and parse
everything all at once. Parsing code becomes simpler, and we get a
looong 'git range-diff -h'
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This option is added in commit b73bcbac4a (diff: allow
--no-color-moved-ws - 2018-11-23) in pw/diff-color-moved-ws-fix. To ease
merge conflict resolution, re-implement the option handling here so that
the conflict could be resolved by taking this side of change.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
OPT__ABBREV() has the same behavior as the deleted code with one
difference: it does check for valid number and error out if not. And the
'40' change is self explanatory.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
SZEDER Gábor [Fri, 22 Mar 2019 14:27:25 +0000 (15:27 +0100)]
t5318-commit-graph: remove unused variable
This is a remnant from early versions of the commit-graph patch series
[1], when 'git commit-graph --write' printed the hash of the created
commit-graph file, and tests did look at the command's output, because
the commit-graph file's name included that hash as well.
checkout: prevent losing staged changes with --merge
When --merge is specified, we may need to do a real merge (instead of
three-way tree unpacking), the steps are best seen in git-checkout.sh
version before it's removed:
# Match the index to the working tree, and do a three-way.
git diff-files --name-only | git update-index --remove --stdin &&
work=`git write-tree` &&
git read-tree $v --reset -u $new || exit
git merge-recursive $old -- $new $work
# Do not register the cleanly merged paths in the index yet.
# this is not a real merge before committing, but just carrying
# the working tree changes along.
unmerged=`git ls-files -u`
git read-tree $v --reset $new
case "$unmerged" in
'') ;;
*)
(
z40=0000000000000000000000000000000000000000
echo "$unmerged" |
sed -e 's/^[0-7]* [0-9a-f]* /'"0 $z40 /"
echo "$unmerged"
) | git update-index --index-info
;;
esac
Notice the last 'read-tree --reset' step. We restore worktree back to
'new' tree after worktree's messed up by merge-recursive. If there are
staged changes before this whole command sequence is executed, they
are lost because they are unlikely part of the 'new' tree to be
restored.
There is no easy way to fix this. Elijah may have something up his
sleeves [1], but until then, check if there are staged changes and
refuse to run and lose them. The user would need to do "git reset" to
continue in this case.
A note about the test update. 'checkout -m' in that test will fail
because a deletion is staged. This 'checkout -m' was previously needed
to verify quietness behavior of unpack-trees. But a different check
has been put in place in the last patch. We can safely drop
'checkout -m' now.
read-tree is basically the front end of unpack-trees code and shoud
expose all of its functionality (unless it's designed for internal
use). This "opts.quiet" (formerly "opts.gently") was added for
builtin/checkout.c but there is no reason why other read-tree users
won't find this useful.
The test that is updated to run 'read-tree --quiet' was added because
unpack-trees was accidentally not being quiet [1] in 6a143aa2b2
(checkout -m: attempt merge when deletion of path was staged -
2014-08-12). Because checkout is the only "opts.quiet" user, there was
no other way to test quiet behavior. But we can now test it directly.
6a143aa2b2 was manually reverted to verify that read-tree --quiet
works correctly (i.e. test_must_be_empty fails).
[1] the commit message there say "errors out instead of performing a
merge" but I'm pretty sure the "performing a merge" happens anyway
even before that commit. That line should say "errors out
_in addition to_ performing a merge"
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The gently flag was added in 17e4642667 (Add flag to make unpack_trees()
not print errors. - 2008-02-07) to suppress error messages. The name
"gently" does not quite express that. Granted, being quiet is gentle but
it could mean not performing some other actions. Rename the flag to
"quiet" to be more on point.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This basically follows the footsteps of 6a143aa2b2 (checkout -m:
attempt merge when deletion of path was staged - 2014-08-12) where
there gently check is moved inside reject_merge() so that callers do
not accidentally forget it.
add_rejected_path() has the same usage pattern. All call sites check
gently first, then decide to call add_rejected_path() if needed. Move
the check inside.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Clean up the grammar in the documentation for
"gc.bigPackThreshold". This documentation was added in 9806f5a7bf ("gc
--auto: exclude base pack if not enough mem to "repack -ad"",
2018-04-15).
Saying "the amount of memory estimated for" flows more smoothly than
the previous "the amount of memory is estimated not enough".
Suggested-by: Jeff King <peff@peff.net> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Remove the mention of specific flags from the "gc" documentation, and
leave it at describing what we'll do instead. As seen in builtin/gc.c
we'll use various repack flags depending on what we detect we need to
do, so this isn't always accurate.
More importantly, a subsequent change is about to remove all this
documentation and replace it with an include of the gc.* docs in
git-config(1). By first changing this it's easier to reason about that
subsequent change.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
gc docs: modernize the advice for manually running "gc"
The docs have been recommending that users need to run this manually,
but that hasn't been needed in practice for a long time except in
exceptional circumstances.
Let's instead have this reflect reality and say that most users don't
need to run this manually at all, while briefly describing the sorts
sort of cases where "gc" does need to be run manually.
Since we're recommending that users run this most of the and usually
don't need to tweak it, let's tone down the very prominent example of
the gc.auto=0 command. It's sufficient to point to the gc.auto
documentation below.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jonathan Tan [Fri, 22 Mar 2019 19:01:39 +0000 (12:01 -0700)]
t5551: mark half-auth no-op fetch test as v0-only
When using protocol v0, upload-pack over HTTP permits a "half-auth"
configuration in which, at the web server layer, the info/refs path is
not protected by authentication but the git-upload-pack path is, so that
a user can perform fetches that do not download any objects without
authentication, but still needs authentication to download objects.
But protocol v2 does not support this, because both ref and pack are
obtained from the git-upload-pack path.
Mark the test verifying this behavior as protocol v0-only, with a
description of what needs to be done to make v2 support this.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Sun, 24 Mar 2019 12:13:16 +0000 (08:13 -0400)]
http: use normalize_curl_result() instead of manual conversion
When we switched off CURLOPT_FAILONERROR in 17966c0a63 (http: avoid
disconnecting on 404s for loose objects, 2016-07-11), the fetch_object()
function started manually handling 404's. Since we now have
normalize_curl_result() for use elsewhere, we can use it here as well,
shortening the code.
Note that we lose the check for http/https in the URL here. None of the
other result-normalizing code paths bother with this. Looking at
missing_target(), which checks specifically for an FTP-specific CURLcode
and "http" code 550, it seems likely that git-over-ftp has been subtly
broken since 17966c0a63. This patch does nothing to fix that, but nor
should it make anything worse (in fact, it may be slightly better
because we'll actually recognize an error as such, rather than assuming
CURLE_OK means we actually got some data).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Sun, 24 Mar 2019 12:09:46 +0000 (08:09 -0400)]
http: normalize curl results for dumb loose and alternates fetches
If the dumb-http walker encounters a 404 when fetching a loose object,
it then looks at any http-alternates for the object. The 404 check is
implemented by missing_target(), which checks not only the http code,
but also that we got an http error from the CURLcode.
That broke when we stopped using CURLOPT_FAILONERROR in 17966c0a63
(http: avoid disconnecting on 404s for loose objects, 2016-07-11), since
our CURLcode will now be CURLE_OK. As a result, fetching over dumb-http
from a repository with alternates could result in Git printing "Unable
to find abcd1234..." and aborting.
We could probably fix this just by loosening missing_target(). However,
there's other code which looks at the curl result, and it would have to
be tweaked as well. Instead, let's just normalize the result the same
way the smart-http code does.
There's a similar case in processing the alternates (where we failover
from "info/http-alternates" to "info/alternates"). We'll give it the
same treatment.
After this patch, we should be hitting all code paths that need this
normalization (notably absent here is the http_pack_request path, but it
does not use FAILONERROR, nor missing_target()).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Sun, 24 Mar 2019 12:08:38 +0000 (08:08 -0400)]
http: factor out curl result code normalization
We make some requests with CURLOPT_FAILONERROR and some without, and
then handle_curl_result() normalizes any failures to a uniform CURLcode.
There are some other code paths in the dumb-http walker which don't use
handle_curl_result(); let's pull the normalization into its own function
so it can be reused.
Arguably those code paths would benefit from the rest of
handle_curl_result(), notably the auth handling. But retro-fitting it
now would be a lot of work, and in practice it doesn't matter too much
(whatever authentication we needed to make the initial contact with the
server is generally sufficient for the rest of the dumb-http requests).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Thu, 21 Mar 2019 09:28:54 +0000 (05:28 -0400)]
refs/files-backend: don't look at an aborted transaction
When deleting refs, we hold packed-refs.lock and prepare a packed
transaction to drop the refs from the packed-refs file. If it turns out
that we don't need to rewrite the packed refs (e.g., because none of the
deletions were present in the file), then we abort the transaction.
If that abort succeeds, then the transaction struct will have been
freed, and we set our local pointer to NULL so we don't look at it
again.
However, if it fails, then the struct will _still_ have been freed
(because ref_transaction_abort() always frees). But we don't clean up
the pointer, and will jump to our cleanup code, which will try to abort
it again, causing a use-after-free.
It's actually impossible for this to trigger in practice, since
packed_transaction_abort() will never return anything but success. But
let's fix it anyway, since that's more than we should assume about the
packed-refs code (after all, we are already bothering to check for an
error result which cannot be triggered).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
In files_transaction_prepare(), if we have to delete some refs, we use a
subordinate packed_transaction to do so. It's rare for that
sub-transaction's prepare step to fail, since we hold the packed-refs
lock. But if it does, we trigger a BUG() due to these steps:
- we've attached the packed transaction to the files transaction as
backend_data->packed_transaction
- when the prepare step fails, the packed transaction cleans itself
up, putting itself into the CLOSED state
- the error value from preparing the packed transaction lets us know
in files_transaction_prepare() that we should also clean up and
return an error. We call files_transaction_cleanup(), which tries to
abort backend_data->packed_transaction. Since it's already CLOSED,
that triggers an assertion in ref_transaction_abort().
We can fix that by disconnecting the packed transaction from the outer
files transaction, and then free-ing (not aborting!) it ourselves.
A few other options/alternatives I considered:
- we could just make it a noop to abort a CLOSED transaction. But that
seems less safe, since clearly this code expects (and enforces) a
particular set of state transitions.
- we could have files_transaction_cleanup() selectively call abort()
vs free() based on the state of the on the packed transaction.
That's basically a more restricted version of the above, but also
potentially unsafe.
- instead of disconnecting backend_data->packed_transaction on error,
we could wait to install it until we successfully prepare. That
might make the flow a little simpler, but it introduces a hassle.
Earlier parts of files_transaction_prepare() that encounter an error
will jump to the cleanup label, and expect that cleaning up the
outer transaction will clean up the packed transaction, too. We'd
have to adjust those sites to clean up the packed transaction.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff Hostetler [Thu, 21 Mar 2019 19:36:15 +0000 (12:36 -0700)]
midx: during verify group objects by packfile to speed verification
Teach `multi-pack-index verify` to sort the set of object by
packfile so that only one packfile needs to be open at a time.
This is a performance improvement. Previously, objects were
verified in OID order. This essentially requires all packfiles
to be open at the same time. If the number of packfiles exceeds
the open file limit, packfiles would be LRU-closed and re-opened
many times.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff Hostetler [Thu, 21 Mar 2019 19:36:11 +0000 (12:36 -0700)]
progress: add sparse mode to force 100% complete message
Add new start_sparse_progress() and start_delayed_sparse_progress()
constructors and "sparse" flag to struct progress.
Teach stop_progress() to force a 100% complete progress message before
printing the final "done" message when "sparse" is set.
Calling display_progress() for every item in a large set can
be expensive. If callers try to filter this for performance
reasons, such as emitting every k-th item, progress would
not reach 100% unless they made a final call to display_progress()
with the item count before calling stop_progress().
Now this is automatic when "sparse" is set.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Josh Steadmon [Thu, 21 Mar 2019 21:09:51 +0000 (14:09 -0700)]
trace2: write to directory targets
When the value of a trace2 environment variable is an absolute path
referring to an existing directory, write output to files (one per
process) underneath the given directory. Files will be named according
to the final component of the trace2 SID, followed by a counter to avoid
potential collisions.
This makes it more convenient to collect traces for every git invocation
by unconditionally setting the relevant trace2 envvar to a constant
directory name.
Signed-off-by: Josh Steadmon <steadmon@google.com> Reviewed-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
checkout.txt: note about losing staged changes with --merge
If you have staged changes in path A and perform 'checkout
--merge' (which could result in conflicts in a totally unrelated path
B), changes in A will be gone. Which is unexpected. We are supposed
to keep all changes, or kick and scream otherwise.
This is the result of how --merge is implemented, from the very first
day in 1be0659efc (checkout: merge local modifications while switching
branches., 2006-01-12):
1. a merge is done, unmerged entries are collected
2. a hard switch to a new branch is done, then unmerged entries added
back
There is no trivial fix for this. Going with 3-way merge one file at a
time loses rename detection. Going with 3-way merge by trees requires
teaching the algorithm to pick up staged changes. And even if we detect
staged changes with --merge and abort for safety, an option to continue
--merge is very weird. Such an option would keep worktree changes, but
drop staged changes.
Because the problem has been with us since the introduction of --merge
and everybody has been pretty happy (except Phillip, who found this
problem), I'll just take a note here to acknowledge it and wait for
merge wizards to come in and work their magic. There may be a way
forward [1].
unpack-trees: fix oneway_merge accidentally carry over stage index
Phillip found out that 'git checkout -f <branch>' does not restore
conflict/unmerged files correctly. All tracked files should be taken
from <branch> and all non-zero stages removed. Most of this is true,
except that the final file could be in stage one instead of zero.
"checkout -f" (among other commands) does this with one-way merge, which
is supposed to take stat info from the index and everything else from
the given tree. The add_entry(.., old, ...) call in oneway_merge()
though will keep stage index from the index.
This is normally not a problem if the entry from the index is
normal (stage #0). But if there is a conflict, stage #0 does not exist
and we'll get stage #1 entry as "old" variable, which gets recorded in
the final index. Fix it by clearing stage mask.
This bug probably comes from b5b425074e (git-read-tree: make one-way
merge also honor the "update" flag, 2005-06-07). Before this commit, we
may create the final ("dst") index entry from the one in index, but we
do clear CE_STAGEMASK.
I briefly checked two- and three-way merge functions. I think we don't
have the same problem in those.
Reported-by: Phillip Wood <phillip.wood@dunelm.org.uk> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Wed, 20 Mar 2019 20:22:15 +0000 (16:22 -0400)]
parse_opt_ref_sorting: always use with NONEG flag
The "--sort" parameter of for-each-ref, etc, does not handle negation,
and instead returns an error to the parse-options code. But neither
piece of code prints anything for the user, which may leave them
confused:
$ git for-each-ref --no-sort
$ echo $?
129
As the comment in the callback function notes, this probably should
clear the list, which would make it consistent with other list-like
options (i.e., anything that uses OPT_STRING_LIST currently).
Unfortunately that's a bit tricky due to the way the ref-filter code
works. But in the meantime, let's at least make the error a little less
confusing:
- switch to using PARSE_OPT_NONEG in the option definition, which will
cause the options code to produce a useful message
- since this was cut-and-pasted to four different spots, let's define
a single OPT_REF_SORT() macro that we can use everywhere
- the callback can use BUG_ON_OPT_NEG() to make sure the correct flags
are used (incidentally, this also satisfies -Wunused-parameters,
since we're now looking at "unset")
- expand the comment into a NEEDSWORK to make it clear that the
direction is right, but the details need to be worked out
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Todd Zullinger [Wed, 20 Mar 2019 18:03:29 +0000 (14:03 -0400)]
completion: use __git when calling --list-cmds
As we made --list-cmds read the local configuration file in an
earlier step, the completion.commands variable respects repo-level
configuration. Use __git which ensures that the proper repo config
is consulted if the command line contains 'git -C /some/other/repo'.
Suggested-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Wed, 20 Mar 2019 18:03:28 +0000 (14:03 -0400)]
completion: fix multiple command removals
Commit 6532f3740b ("completion: allow to customize the completable
command list", 2018-05-20) tried to allow multiple space-separated
entries in completion.commands. To do this, it copies each parsed token
into a strbuf so that the result is NUL-terminated.
However, for tokens starting with "-", it accidentally passes the
original non-terminated string, meaning that only the final one worked.
Switch to using the strbuf.
Reported-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Todd Zullinger [Wed, 20 Mar 2019 18:03:27 +0000 (14:03 -0400)]
t9902: test multiple removals via completion.commands
6532f3740b ("completion: allow to customize the completable command
list", 2018-05-20) added the completion.commands config variable.
Multiple commands may be added or removed, separated by a space.
Demonstrate the failure of multiple removals.
Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Wed, 20 Mar 2019 18:03:26 +0000 (14:03 -0400)]
git: read local config in --list-cmds
Normally code that is checking config before we've decided to do
setup_git_directory() would use read_early_config(), which uses
discover_git_directory() to tentatively see if we're in a repo,
and if so to add it to the config sequence.
But list_cmds() uses the caching configset mechanism which
rightly does not use read_early_config(), because it has no
idea if it's being called early.
Call setup_git_directory_gently() so we can pick up repo-level
config (like completion.commands).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Thomas Gummerer [Wed, 20 Mar 2019 22:49:55 +0000 (22:49 +0000)]
stash: setup default diff output format if necessary
In the scripted 'git stash show' when no arguments are passed, we just
pass '--stat' to 'git diff'. When any argument is passed to 'stash
show', we no longer pass '--stat' to 'git diff', and pass whatever
flags are passed directly through to 'git diff'.
By default 'git diff' shows the patch output. So when a user uses
'git stash show --patience', they would be shown the diff as expected,
using the patience algorithm. '--patience' in this case only changes
the diff algorithm, but does not cause 'git diff' to show the diff by
itself. The diff is shown because that's the default behaviour of
'git diff'.
In the C version of 'git stash show', we try to emulate that behaviour
using the internal diff API. However we forgot to set up the default
output format, in case it wasn't set by any of the flags that were
passed through. So 'git stash show --patience' in the builtin version
of stash would be completely silent, while it would show the diff in
the scripted version.
The same thing would happen for other flags that only affect the way a
patch is displayed, rather than switching to a different output format
than the default one.
Fix this by setting up the default output format for 'git diff'.
Reported-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Wed, 20 Mar 2019 08:16:42 +0000 (04:16 -0400)]
pretty: drop unused strbuf from parse_padding_placeholder()
Unlike other parts of the --pretty user-format expansion,
this function is not actually writing to the output, but
instead just storing the padding values into a context
struct. We don't need to be passed a strbuf at all.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Wed, 20 Mar 2019 08:16:36 +0000 (04:16 -0400)]
pretty: drop unused "type" parameter in needs_rfc2047_encoding()
The "should we encode" check was split off from add_rfc2047() into its
own function in 41dd00bad3 (format-patch: fix rfc2047 address encoding
with respect to rfc822 specials, 2012-10-18). But only the "add" half
needs to know the rfc2047_type, since it only affects _how_ we encode,
not whether we do.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Wed, 20 Mar 2019 08:16:27 +0000 (04:16 -0400)]
parse-options: drop unused ctx parameter from show_gitcomp()
The completion display doesn't actually care about where we are in the
parsing. It's generated completely from the set of available options. So
we don't need to see the parse-options context struct at all.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Wed, 20 Mar 2019 08:16:14 +0000 (04:16 -0400)]
fetch_pack(): drop unused parameters
We don't need the caller of fetch_pack() to pass in "dest", which is the
remote URL. Since ba227857d2 (Reduce the number of connects when
fetching, 2008-02-04), the caller is responsible for calling
git_connect() itself, and our "dest" parameter is unused.
That commit also started passing us the resulting "conn" child_process
from git_connect(). But likewise, we do not need do anything with it.
The descriptors in "fd" are enough for us, and the caller is responsible
for cleaning up "conn".
We can just drop both parameters.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Wed, 20 Mar 2019 08:15:48 +0000 (04:15 -0400)]
report_path_error(): drop unused prefix parameter
This hasn't been used since 17ddc66e70 (convert report_path_error to
take struct pathspec, 2013-07-14), as the names in the struct will have
already been prefixed when they were parsed.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff King [Wed, 20 Mar 2019 08:15:27 +0000 (04:15 -0400)]
unpack-trees: drop unused error_type parameters
The verify_clean_subdirectory() helper takes an error_type parameter
from the caller, but doesn't actually use it. Instead, when it calls
add_rejected_path() it passes NOT_UPTODATE_DIR, its own custom error
type which is more specific than what the caller provides. Likewise for
verify_clean_submodule(), which always passes WOULD_LOSE_SUBMODULE.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
projects / thirdparty / git.git / log
