regexec: work around macOS TRE leak on invalid UTF-8

regexec: work around macOS TRE leak on invalid UTF-8

On macOS, the system regex engine leaks an internal buffer when
regexec() encounters an invalid multibyte sequence in a UTF-8 locale.
The line-by-line path can call regexec_buf() for each pattern on every
line, so "git grep" can leak repeatedly on a file containing invalid
UTF-8.  The total leak grows with the number of calls, and the per-call
allocation grows with the pattern's automaton.  In one case, grepping a
repository containing PDFs exhausted memory and caused the machine to
restart.

ce025ae4f61e (grep: disable lookahead on error, 2024-10-20) made "git
grep" fall back to line-by-line matching when regexec() reports an error
on invalid UTF-8.  That fallback cannot prevent this leak: the allocation
has already leaked when regexec() returns REG_ILLSEQ.

Avoid the leaking path by providing a Darwin-specific regexec_buf().
Walk the input with mbrtowc(), split it at bytes that cannot form a
complete multibyte character, and search each valid segment separately.
This preserves matches in valid text on either side of an invalid byte.

Search each segment with REG_STARTEND so match offsets remain relative to
the original buffer.  Set REG_NOTBOL and REG_NOTEOL for internal segment
boundaries so "^" and "$" do not match there.  Keep the flags clear at
the true beginning and end of the buffer.

Use the normal regexec_buf() path in single-byte locales, where no byte
can form an invalid multibyte sequence.  Use the bundled regex
implementation unchanged when NO_REGEX is enabled.

Declare the Darwin override in compat/darwin.h and map regexec_buf() to
darwin_regexec_buf().  This follows the platform override pattern used
by the other compatibility headers and leaves the common inline
implementation as the default.

There is no reliable way to detect a future macOS version in which the
system regex implementation has been fixed.  Even after a fix, Git will
need the workaround while it supports affected macOS releases, so treat
it as an indefinite compatibility workaround.

Add tests for matches before, after, and between invalid bytes, including
an offset check after an invalid byte.  Also check incomplete trailing
input and anchors at true and internal line boundaries.

Signed-off-by: Chungmin Lee <chungmin@chungminlee.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>