Don't redundantly run "git reflog expire --all" when gc.reflogExpire
and gc.reflogExpireUnreachable are set to "never", and die immediately
if those configuration valuer are bad.
As an earlier "assert lack of early exit" change to the tests for "git
reflog expire" shows, an early check of gc.reflogExpire{Unreachable,}
isn't wanted in general for "git reflog expire", but it makes sense
for "gc" because:
1) Similarly to 8ab5aa4bd8 ("parseopt: handle malformed --expire
arguments more nicely", 2018-04-21) we'll now die early if the
config variables are set to invalid values.
We run "pack-refs" before "reflog expire", which can take a while,
only to then die on an invalid gc.reflogExpire{Unreachable,}
configuration.
2) Not invoking the command at all means it won't show up in trace
output, which makes what's going on more obvious when the two are
set to "never".
3) As a later change documents we lock the refs when looping over the
refs to expire, even in cases where we end up doing nothing due to
this config.
For the reasons noted in the earlier "assert lack of early exit"
change I don't think it's worth it to bend over backwards in "git
reflog expire" itself to carefully detect if we'll really do
nothing given the combination of all its possible options and skip
that locking, but that's easy to detect here in "gc" where we'll
only run "reflog expire" in a relatively simple mode.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
reflog tests: assert lack of early exit with expiry="never"
When gc.reflogExpire and gc.reflogExpireUnreachable are set to "never"
and --stale-fix isn't in effect we *could* exit early without
pointlessly looping over all the reflogs.
However, as an earlier change to add a test for the "points nowhere"
warning shows even in such a mode we might want to print out a
warning.
So while it's conceivable to implement this, I don't think it's worth
it. It's going to be too easy to inadvertently add some flag that'll
make the expiry happen anyway, and even with "never" we'd like to see
all the lines we're going to keep.
So let's assert that we're going to loop over all the references even
when this configuration is in effect.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
reflog tests: test for the "points nowhere" warning
The "git reflog expire" command when given an unknown reference has
since 4264dc15e1 ("git reflog expire", 2006-12-19) when this command
was implemented emit an error, but this has never been tested for.
Let's test for it, also under gc.reflogExpire{Unreachable,}=never in
case a future change is tempted to take shortcuts in the presence of
such config.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Change an idiom we're using to ensure that gc_before_repack() only
does work once (see 62aad1849f ("gc --auto: do not lock refs in the
background", 2014-05-25)) to be more obvious.
Nothing except this function cares about the "pack_refs" and
"prune_reflogs" variables, so let's not leave the reader wondering if
they're being zero'd out for later use somewhere else.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
There's been a lot of changing of the hardcoded "40" values to
the_hash_algo->hexsz, but we've so far missed this one where we
hardcoded 38 for the loose object file length.
This is because a SHA-1 like abcde[...] gets turned into
objects/ab/cde[...]. There's no reason to suppose the same won't be
the case for SHA-256, and reading between the lines in
hash-function-transition.txt the format is planned to be the same.
In the future we may want to further modify this code for the hash
function transition. There's a potential pathological case here where
we'll only consider the loose objects for the currently active hash,
but objects for that hash will share a directory storage with the
other hash.
Thus we could theoretically have e.g. 1k SHA-1 loose objects, and 1
million SHA-256 objects. Then not notice that we need to pack them
because we're currently using SHA-1, even though our FS may be
straining under the stress of such humongous directories.
So assuming that "gc" eventually learns to pack up both SHA-1 and
SHA-256 objects regardless of what the current the_hash_algo is,
perhaps this check should be changed to consider all files in
objects/17/ matching [0-9a-f] 38 or 62 characters in length (i.e. both
SHA-1 and SHA-256).
But none of that is something we need to worry about now, and
supporting both 38 and 62 characters depending on "the_hash_algo"
removes another case of SHA-1 hardcoding.
As noted in [1] I'm making no effort to somehow remove the hardcoding
for "2" as in "use the first two hexdigits for the directory
name". There's no indication that that'll ever change, and somehow
generalizing it here would be a drop in the ocean, so there's no point
in doing that. It also couldn't be done without coming up with some
generalized version of the magical "objects/17" directory. See [2] for
a discussion of that directory.
Checking gc_auto_threshold in too_many_loose_objects() was added in 17815501a8 ("git-gc --auto: run "repack -A -d -l" as necessary.",
2007-09-17) when need_to_gc() itself was also reliant on
gc_auto_pack_limit before its early return:
gc_auto_threshold <= 0 && gc_auto_pack_limit <= 0
When that check was simplified to just checking "gc_auto_threshold <=
0" in b14d255ba8 ("builtin-gc.c: allow disabling all auto-gc'ing by
assigning 0 to gc.auto", 2008-03-19) this unreachable code should have
been removed. We only call too_many_loose_objects() from within
need_to_gc() itself, which will return if this condition holds, and in
cmd_gc() which will return before ever getting to "auto_gc &&
too_many_loose_objects()" if "auto_gc && !need_to_gc()" is true
earlier in the function.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The Makefile uses 'find' utility to enumerate all the *.h header
files, which is expensive on platforms with slow filesystems; it
now optionally uses "ls-files" if working within a repository,
which is a trick similar to how all sources are enumerated to run
ETAGS on.
* js/find-lib-h-with-ls-files-when-possible:
Makefile: use `git ls-files` to list header files, if possible
Junio C Hamano [Mon, 11 Mar 2019 07:16:25 +0000 (16:16 +0900)]
Merge branch 'rj/hdr-check-gcrypt-fix'
The set of header files used by "make hdr-check" unconditionally
included sha256/gcrypt.h, even when it is not used, causing the
make target to fail. We now skip it when GCRYPT_SHA256 is not in
use.
* rj/hdr-check-gcrypt-fix:
Makefile: fix 'hdr-check' when GCRYPT not installed
Recently the Git for Windows project started the upgrade process to
a MSYS2 runtime version based on Cygwin v3.x.
This has the very notable consequence that `$(uname -r)` no longer
reports a version starting with "2", but a version with "3".
That breaks our build, as df5218b4c30b (config.mak.uname: support MSys2,
2016-01-13) simply did not expect the version reported by `uname -r` to
depend on the underlying Cygwin version: it expected the reported
version to match the "2" in "MSYS2".
So let's invert that test case to test for *anything else* than a
version starting with "1" (for MSys). That should safeguard us for the
future, even if Cygwin ends up releasing versionsl like 314.272.65536.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
In 20d2a30f8ffe (Makefile: replace perl/Makefile.PL with simple make
rules, 2017-12-10), Git stopped using MakeMaker. Therefore, that
definition in the MINGW-specific section became useless.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Junio C Hamano [Thu, 7 Mar 2019 00:59:59 +0000 (09:59 +0900)]
Merge branch 'jt/http-auth-proto-v2-fix'
Unify RPC code for smart http in protocol v0/v1 and v2, which fixes
a bug in the latter (lack of authentication retry) and generally
improves the code base.
* jt/http-auth-proto-v2-fix:
remote-curl: use post_rpc() for protocol v2 also
remote-curl: refactor reading into rpc_state's buf
remote-curl: reduce scope of rpc_state.result
remote-curl: reduce scope of rpc_state.stdin_preamble
remote-curl: reduce scope of rpc_state.argv
Junio C Hamano [Thu, 7 Mar 2019 00:59:59 +0000 (09:59 +0900)]
Merge branch 'jk/diff-no-index-initialize'
"git diff --no-index" may still want to access Git goodies like
--ext-diff and --textconv, but so far these have been ignored,
which has been corrected.
* jk/diff-no-index-initialize:
diff: reuse diff setup for --no-index case
Junio C Hamano [Thu, 7 Mar 2019 00:59:56 +0000 (09:59 +0900)]
Merge branch 'jk/unused-params'
Code clean-up.
* jk/unused-params:
ref-filter: drop unused "sz" parameters
ref-filter: drop unused "obj" parameters
ref-filter: drop unused buf/sz pairs
files-backend: drop refs parameter from split_symref_update()
pack-objects: drop unused parameter from oe_map_new_pack()
merge-recursive: drop several unused parameters
diff: drop complete_rewrite parameter from run_external_diff()
diff: drop unused emit data parameter from sane_truncate_line()
diff: drop unused color reset parameters
diff: drop options parameter from diffcore_fix_diff_index()
Junio C Hamano [Thu, 7 Mar 2019 00:59:56 +0000 (09:59 +0900)]
Merge branch 'jk/prune-optim'
"git prune" has been taught to take advantage of reachability
bitmap when able.
* jk/prune-optim:
t5304: rename "sha1" variables to "oid"
prune: check SEEN flag for reachability
prune: use bitmaps for reachability traversal
prune: lazily perform reachability traversal
Junio C Hamano [Thu, 7 Mar 2019 00:59:54 +0000 (09:59 +0900)]
Merge branch 'en/combined-all-paths'
Output from "diff --cc" did not show the original paths when the
merge involved renames. A new option adds the paths in the
original trees to the output.
Junio C Hamano [Thu, 7 Mar 2019 00:59:54 +0000 (09:59 +0900)]
Merge branch 'sc/pack-redundant'
Update the implementation of pack-redundant for performance in a
repository with many packfiles.
* sc/pack-redundant:
pack-redundant: consistent sort method
pack-redundant: rename pack_list.all_objects
pack-redundant: new algorithm to find min packs
pack-redundant: delete redundant code
pack-redundant: delay creation of unique_objects
t5323: test cases for git-pack-redundant
Junio C Hamano [Thu, 7 Mar 2019 00:59:52 +0000 (09:59 +0900)]
Merge branch 'aw/pretty-trailers'
The %(trailers) formatter in "git log --format=..." now allows to
optionally pick trailers selectively by keyword, show only values,
etc.
* aw/pretty-trailers:
pretty: add support for separator option in %(trailers)
strbuf: separate callback for strbuf_expand:ing literals
pretty: add support for "valueonly" option in %(trailers)
pretty: allow showing specific trailers
pretty: single return path in %(trailers) handling
pretty: allow %(trailers) options with explicit value
doc: group pretty-format.txt placeholders descriptions
Junio C Hamano [Thu, 7 Mar 2019 00:59:52 +0000 (09:59 +0900)]
Merge branch 'nd/diff-parseopt'
The diff machinery, one of the oldest parts of the system, which
long predates the parse-options API, uses fairly long and complex
handcrafted option parser. This is being rewritten to use the
parse-options API.
* nd/diff-parseopt:
diff.c: convert --raw
diff.c: convert -W|--[no-]function-context
diff.c: convert -U|--unified
diff.c: convert -u|-p|--patch
diff.c: prepare to use parse_options() for parsing
diff.h: avoid bit fields in struct diff_flags
diff.h: keep forward struct declarations sorted
parse-options: allow ll_callback with OPTION_CALLBACK
parse-options: avoid magic return codes
parse-options: stop abusing 'callback' for lowlevel callbacks
parse-options: add OPT_BITOP()
parse-options: disable option abbreviation with PARSE_OPT_KEEP_UNKNOWN
parse-options: add one-shot mode
parse-options.h: remove extern on function prototypes
Junio C Hamano [Thu, 7 Mar 2019 00:59:51 +0000 (09:59 +0900)]
Merge branch 'tg/checkout-no-overlay'
"git checkout --no-overlay" can be used to trigger a new mode of
checking out paths out of the tree-ish, that allows paths that
match the pathspec that are in the current index and working tree
and are not in the tree-ish.
* tg/checkout-no-overlay:
revert "checkout: introduce checkout.overlayMode config"
checkout: introduce checkout.overlayMode config
checkout: introduce --{,no-}overlay option
checkout: factor out mark_cache_entry_for_checkout function
checkout: clarify comment
read-cache: add invalidate parameter to remove_marked_cache_entries
entry: support CE_WT_REMOVE flag in checkout_entry
entry: factor out unlink_entry function
move worktree tests to t24*
Jeff King [Wed, 6 Mar 2019 19:05:10 +0000 (14:05 -0500)]
compat/bswap: add include header guards
Our compat/bswap.h lacks the usual preprocessor guards against multiple
inclusion. This usually isn't an issue since it only gets included from
git-compat-util.h, which has its own guards. But it would produce
redeclaration errors if any file included it separately.
Our hdr-check target would complain about this, except that it currently
skips items in compat/ entirely.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Ramsay Jones [Wed, 6 Mar 2019 00:11:13 +0000 (00:11 +0000)]
Makefile: fix 'hdr-check' when GCRYPT not installed
If the GCRYPT_SHA256 build variable is not set, then the 'hdr-check'
target complains about the missing <gcrypt.h> header file. Add the
'sha256/gcrypt.h' header file to the exception list, if the build
variable is not defined. While here, replace the 'xdiff%' filter
pattern with 'xdiff/%' (and similarly for the compat pattern) since
the original pattern inadvertently excluded the 'xdiff-interface.h'
header.
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Makefile: use `git ls-files` to list header files, if possible
In d85b0dff72 (Makefile: use `find` to determine static header
dependencies, 2014-08-25), we switched from a static list of header
files to a dynamically-generated one, asking `find` to enumerate them.
Back in those days, we did not use `$(LIB_H)` by default, and many a
`make` implementation seems smart enough not to run that `find` command
in that case, so it was deemed okay to run `find` for special targets
requiring this macro.
However, as of ebb7baf02f (Makefile: add a hdr-check target,
2018-09-19), $(LIB_H) is part of a global rule and therefore must be
expanded. Meaning: this `find` command has to be run upon every
`make` invocation. In the presence of many a worktree, this can tax the
developers' patience quite a bit.
Even in the absence of worktrees or other untracked files and
directories, the cost of I/O to generate that list of header files is
simply a lot larger than a simple `git ls-files` call.
Therefore, just like in 335339758c (Makefile: ask "ls-files" to list
source files if available, 2011-10-18), we now prefer to use `git
ls-files` to enumerate the header files to enumerating them via `find`,
falling back to the latter if the former failed (which would be the case
e.g. in a worktree that was extracted from a source .tar file rather
than from a clone of Git's sources).
This has one notable consequence: we no longer include `command-list.h`
in `LIB_H`, as it is a generated file, not a tracked one, but that is
easily worked around. Of the three sites that use `LIB_H`, two
(`LOCALIZED_C` and `CHK_HDRS`) already handle generated headers
separately. In the third, the computed-dependency fallback, we can just
add in a reference to $(GENERATED_H).
Likewise, we no longer include not-yet-tracked header files in `LIB_H`.
Given the speed improvements, these consequences seem a comparably small
price to pay.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Acked-by: Ramsay Jones <ramsay@ramsayjones.plus.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jonathan Tan [Thu, 21 Feb 2019 20:24:41 +0000 (12:24 -0800)]
remote-curl: use post_rpc() for protocol v2 also
When transmitting and receiving POSTs for protocol v0 and v1,
remote-curl uses post_rpc() (and associated functions), but when doing
the same for protocol v2, it uses a separate set of functions
(proxy_rpc() and others). Besides duplication of code, this has caused
at least one bug: the auth retry mechanism that was implemented in v0/v1
was not implemented in v2.
To fix this issue and avoid it in the future, make remote-curl also use
post_rpc() when handling protocol v2. Because line lengths are written
to the HTTP request in protocol v2 (unlike in protocol v0/v1), this
necessitates changes in post_rpc() and some of the functions it uses;
perform these changes too.
A test has been included to ensure that the code for both the unchunked
and chunked variants of the HTTP request is exercised.
Note: stateless_connect() has been updated to use the lower-level packet
reading functions instead of struct packet_reader. The low-level control
is necessary here because we cannot change the destination buffer of
struct packet_reader while it is being used; struct packet_buffer has a
peeking mechanism which relies on the destination buffer being present
in between a peek and a read.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
travis: remove the hack to build the Windows job on Azure Pipelines
Since Travis did not support Windows (and now only supports very limited
Windows jobs, too limited for our use, the test suite would time out
*all* the time), we added a hack where a Travis job would trigger an
Azure Pipeline (which back then was still called VSTS Build), wait for
it to finish (or time out), and download the log (if available).
Needless to say that it was a horrible hack, necessitated by a bad
situation.
Nowadays, however, we have Azure Pipelines support, and do not need that
hack anymore. So let's retire it.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jean-Noël Avila [Mon, 25 Feb 2019 21:11:15 +0000 (22:11 +0100)]
l10n: fr.po remove obsolete entries
On NetBSD, the version of msgfmt is still 0.14.4. There's no hope for
an upgrade due to some GPLv3 allergy of NetBSD's. This version chokes
on heavily decorated commented entries in po files. It's safer to get
rid of all these obsolete entries.
Jeff King [Sat, 16 Feb 2019 06:57:56 +0000 (01:57 -0500)]
diff: reuse diff setup for --no-index case
When "--no-index" is in effect (or implied by the arguments), git-diff
jumps early to a special code path to perform that diff. This means we
miss out on some settings like enabling --ext-diff and --textconv by
default.
Let's jump to the no-index path _after_ we've done more setup on
rev.diffopt. Since some of the options don't affect us (e.g., items
related to the index), let's re-order the setup into two blocks (see the
in-code comments).
Note that we also need to stop re-initializing the diffopt struct in
diff_no_index(). This should not be necessary, as it will already have
been initialized by cmd_diff() (and there are no other callers). That in
turn lets us drop the "repository" argument from diff_no_index (which
never made much sense, since the whole point is that you don't need a
repository).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
During the six months of development of the Azure Pipelines support, the
patches went through quite a few iterations of changes, and to test
those iterations, a temporary build definition was used.
In the meantime, Azure Pipelines support made it to `master`, and we now
have a regular Azure Pipeline, installed via the common GitHub App
workflow. This new pipeline has a different name (git.git instead of
test-git.git), and a new ID (11 instead of 2).
Let's adjust the badge in our README to reflect that final shape of the
Azure Pipeline.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Derrick Stolee [Fri, 22 Feb 2019 22:25:07 +0000 (14:25 -0800)]
trace2:data: pack-objects: add trace2 regions
When studying the performance of 'git push' we would like to know
how much time is spent at various parts of the command. One area
that could cause performance trouble is 'git pack-objects'.
Add trace2 regions around the three main actions taken in this
command:
Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff Hostetler [Fri, 22 Feb 2019 22:25:03 +0000 (14:25 -0800)]
trace2:data: add trace2 regions to wt-status
Add trace2_region_enter() and trace2_region_leave() calls around the
various phases of a status scan. This gives elapsed time for each
phase in the GIT_TR2_PERF and GIT_TR2_EVENT trace target.
Also, these Trace2 calls now use s->repo rather than the_repository.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jeff Hostetler [Fri, 22 Feb 2019 22:25:01 +0000 (14:25 -0800)]
trace2: create new combined trace facility
Create a new unified tracing facility for git. The eventual intent is to
replace the current trace_printf* and trace_performance* routines with a
unified set of git_trace2* routines.
In addition to the usual printf-style API, trace2 provides higer-level
event verbs with fixed-fields allowing structured data to be written.
This makes post-processing and analysis easier for external tools.
Trace2 defines 3 output targets. These are set using the environment
variables "GIT_TR2", "GIT_TR2_PERF", and "GIT_TR2_EVENT". These may be
set to "1" or to an absolute pathname (just like the current GIT_TRACE).
* GIT_TR2 is intended to be a replacement for GIT_TRACE and logs command
summary data.
* GIT_TR2_PERF is intended as a replacement for GIT_TRACE_PERFORMANCE.
It extends the output with columns for the command process, thread,
repo, absolute and relative elapsed times. It reports events for
child process start/stop, thread start/stop, and per-thread function
nesting.
* GIT_TR2_EVENT is a new structured format. It writes event data as a
series of JSON records.
Calls to trace2 functions log to any of the 3 output targets enabled
without the need to call different trace_printf* or trace_performance*
routines.
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is git-checy-racy command, added a long time ago [1] and was never
part of the default build. Naturally after some makefile changes [2],
git-check-racy was no longer recognized as a build target. Even if it
compiles to day, it will not link after the introduction of
common-main.c [3].
Racy index has not been a problem for a long time [4]. It's time to let
this code go. I briefly consider if check-racy should be part of
test-tool. But I don't think it's worth the effort.
[1] 42f774063d (Add check program "git-check-racy" - 2006-08-15)
[2] c373991375 (Makefile: list generated object files in OBJECTS -
2010-01-26)
[3] 3f2e2297b9 (add an extra level of indirection to main() -
2016-07-01)
[4] I pretend I don't remember anything about the recent split-index's
racy problem
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Jonathan Tan [Thu, 21 Feb 2019 20:24:40 +0000 (12:24 -0800)]
remote-curl: refactor reading into rpc_state's buf
Currently, whenever remote-curl reads pkt-lines from its response file
descriptor, only the payload is written to its buf, not the 4 characters
denoting the length. A future patch will require the ability to also
write those 4 characters, so in preparation for that, refactor this read
into its own function.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Change an unportable invocation of "dd" with count=0, that wanted to
truncate the commit-graph file. In POSIX it is unspecified what
happens when count=0 is provided[1]. The NetBSD "dd" behavior
differs from GNU (and seemingly other BSDs), which has left this test
broken since d2b86fbaa1 ("commit-graph: fix buffer read-overflow",
2019-01-15).
Copying from /dev/null would seek/truncate to seek=$zero_pos and
stop immediately after that (without being able to copy anything),
which is the right way to truncate the file.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Helped-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Fix widely supported but non-POSIX basic regex syntax introduced in
[1] and [2]. On GNU, NetBSD and FreeBSD the following works:
$ echo xy >f
$ grep 'xy\?' f; echo $?
xy
0
The same goes for "\+". The "?" and "+" syntax is not in the BRE
syntax, just in ERE, but on some implementations it can be invoked by
prefixing the meta-operator with "\", but not on OpenBSD:
projects / thirdparty / git.git / log
