intl: Handle translation output codesets with suffixes [BZ #26383]
Commit 91927b7c7643 (Rewrite iconv option parsing [BZ #19519]) did not
handle cases where the output codeset for translations (via the `gettext'
family of functions) might have a caller specified encoding suffix such as
TRANSLIT or IGNORE. This led to a regression where translations did not
work when the codeset had a suffix.
This commit fixes the above issue by parsing any suffixes passed to
__dcigettext and adds two new test-cases to intl/tst-codeset.c to
verify correct behaviour. The iconv-internal function __gconv_create_spec
and the static iconv-internal function gconv_destroy_spec are now visible
internally within glibc and used in intl/dcigettext.c.
This commit replaces string manipulation during `iconv_open' and iconv_prog
option parsing with a structured, flag based conversion specification. In
doing so, it alters the internal `__gconv_open' interface and accordingly
adjusts its uses.
This change fixes several hangs in the iconv program and therefore includes
a new test to exercise iconv_prog options that originally led to these hangs.
It also includes a new regression test for option handling in the iconv
function.
Arjun Shankar [Wed, 4 Nov 2020 11:19:38 +0000 (12:19 +0100)]
iconv: Accept redundant shift sequences in IBM1364 [BZ #26224]
The IBM1364, IBM1371, IBM1388, IBM1390 and IBM1399 character sets
share converter logic (iconvdata/ibm1364.c) which would reject
redundant shift sequences when processing input in these character
sets. This led to a hang in the iconv program (CVE-2020-27618).
This commit adjusts the converter to ignore redundant shift sequences
and adds test cases for iconv_prog hangs that would be triggered upon
their rejection. This brings the implementation in line with other
converters that also ignore redundant shift sequences (e.g. IBM930
etc., fixed in commit 692de4b3960d).
Previously, in UCS4 conversion routines we limit the number of
characters we examine to the minimum of the number of characters in the
input and the number of characters in the output. This is not the
correct behavior when __GCONV_IGNORE_ERRORS is set, as we do not consume
an output character when we skip a code unit. Instead, track the input
and output pointers and terminate the loop when either reaches its
limit.
This resolves assertion failures when resetting the input buffer in a step of
iconv, which assumes that the input will be fully consumed given sufficient
output space.
Joseph Myers [Wed, 12 Feb 2020 23:31:56 +0000 (23:31 +0000)]
Avoid ldbl-96 stack corruption from range reduction of pseudo-zero (bug 25487).
Bug 25487 reports stack corruption in ldbl-96 sinl on a pseudo-zero
argument (an representation where all the significand bits, including
the explicit high bit, are zero, but the exponent is not zero, which
is not a valid representation for the long double type).
Although this is not a valid long double representation, existing
practice in this area (see bug 4586, originally marked invalid but
subsequently fixed) is that we still seek to avoid invalid memory
accesses as a result, in case of programs that treat arbitrary binary
data as long double representations, although the invalid
representations of the ldbl-96 format do not need to be consistently
handled the same as any particular valid representation.
This patch makes the range reduction detect pseudo-zero and unnormal
representations that would otherwise go to __kernel_rem_pio2, and
returns a NaN for them instead of continuing with the range reduction
process. (Pseudo-zero and unnormal representations whose unbiased
exponent is less than -1 have already been safely returned from the
function before this point without going through the rest of range
reduction.) Pseudo-zero representations would previously result in
the value passed to __kernel_rem_pio2 being all-zero, which is
definitely unsafe; unnormal representations would previously result in
a value passed whose high bit is zero, which might well be unsafe
since that is not a form of input expected by __kernel_rem_pio2.
This patch syncs the regex implementation with gnulib (commit 0ee5212).
Only two changes in GLIBC regex testing are required:
1. posix/bug-regex28.c: as previously discussed [1] the change of
expected results on the pattern should be safe.
2. posix/PCRE.tests: the ERE (a)|\1 is malformed (in the sense that
the \1 doesn't mean anything) and although current GLIBC accepts
it has undefined behavior. This patch removes the specific test.
This sync contains some patches from thread 'Regex: Make libc regex
more usable outside GLIBC.' [2] which have been pushed upstream in
gnulib. This patches also fixes some regex issues (BZ #23233,
BZ #21163, BZ #18986, BZ #13762) and I did not add testcases for
both #23233 and #13762 because I couldn't think a simple way to
trigger the expected failure path to trigger them.
Andreas Schwab [Mon, 21 Dec 2020 03:26:43 +0000 (08:56 +0530)]
Fix buffer overrun in EUC-KR conversion module (bz #24973)
The byte 0xfe as input to the EUC-KR conversion denotes a user-defined
area and is not allowed. The from_euc_kr function used to skip two bytes
when told to skip over the unknown designation, potentially running over
the buffer end.
Florian Weimer [Wed, 27 Jan 2021 12:36:12 +0000 (13:36 +0100)]
gconv: Fix assertion failure in ISO-2022-JP-3 module (bug 27256)
The conversion loop to the internal encoding does not follow
the interface contract that __GCONV_FULL_OUTPUT is only returned
after the internal wchar_t buffer has been filled completely. This
is enforced by the first of the two asserts in iconv/skeleton.c:
/* We must run out of output buffer space in this
rerun. */
assert (outbuf == outerr);
assert (nstatus == __GCONV_FULL_OUTPUT);
This commit solves this issue by queuing a second wide character
which cannot be written immediately in the state variable, like
other converters already do (e.g., BIG5-HKSCS or TSCII).
Shu-Chun Weng [Mon, 3 May 2021 23:47:10 +0000 (16:47 -0700)]
Don't crash if /var/tmp doesn't exist
`xstat` is checked `stat64` crashing the program if the latter returns
failure. In this loop, we are trying to find one folder that satisfies
the condition, no reason to crash the program if one folder doesn't.
Shu-Chun Weng [Mon, 3 May 2021 23:12:44 +0000 (16:12 -0700)]
More aggressively prevent a buffer from being optimized out
The volatile global variable was first introduced in e86f9654c. I have
noticed the compiler still optimizing the buffer out on AArch64
presumably because the assignment is after all other observable
behaviors so it's still valid to eliminate it.
Fangrui Song [Thu, 8 Jul 2021 21:26:22 +0000 (14:26 -0700)]
x86_64: Remove unneeded static PIE check for undefined weak diagnostic
https://sourceware.org/bugzilla/show_bug.cgi?id=21782 dropped an ld
diagnostic for R_X86_64_PC32 referencing an undefined weak symbol in
-pie links. Arguably keeping the diagnostic like other ports is more
correct, since statically resolving movl foo(%rip), %eax to the
link-time zero address produces a corrupted output.
It turns out that --enable-static-pie builds do not depend on the ld
behavior. GCC generates GOT indirection for weak declarations for
-fPIE/-fPIC, so what ld does with the PC-relative relocation doesn't
really matter.
Refactor the sincos implementation - rather than rely on odd partial inlining
of preprocessed portions from sin and cos, explicitly write out the cases.
This makes sincos much easier to maintain and provides an additional 16-20%
speedup between 0 and 2^27. The overall speedup of sincos is 48% over this range.
Between 0 and PI it is 66% faster.
* sysdeps/ieee754/dbl-64/s_sin.c (__sin): Cleanup ifdefs.
(__cos): Likewise.
* sysdeps/ieee754/dbl-64/s_sin.c (__sincos): Refactor using the same
logic as sin and cos.
[PATCH 6/7] sin/cos slow paths: refactor duplicated code into dosin
Refactor duplicated code into do_sin. Since all calls to do_sin use copysign to
set the sign of the result, move it inside do_sin. Small inputs use a separate
polynomial, so move this into do_sin as well (the check is based on the more
conservative case when doing large range reduction, but could be relaxed).
* sysdeps/ieee754/dbl-64/s_sin.c (do_sin): Use TAYLOR_SIN for small
inputs. Return correct sign.
(do_sincos): Remove small input check before do_sin, let do_sin set
the sign.
(__sin): Likewise.
(__cos): Likewise.
[PATCH 4/7] sin/cos slow paths: remove slow paths from huge range reduction
For huge inputs use the improved do_sincos function as well. Now no cases use
the correction factor returned by do_sin, do_cos and TAYLOR_SIN, so remove it.
[PATCH 3/7] sin/cos slow paths: remove slow paths from small range reduction
This patch improves the accuracy of the range reduction. When the input is
large (2^27) and very close to a multiple of PI/2, using 110 bits of PI is not
enough. Improve range reduction accuracy to 136 bits. As a result the special
checks for results close to zero can be removed. The ULP of the polynomials is
at worst 0.55ULP, so there is no reason for the slow functions, and they can be
removed.
* sysdeps/ieee754/dbl-64/s_sin.c (reduce_sincos_1): Rename to
reduce_sincos, improve accuracy to 136 bits.
(do_sincos_1): Rename to do_sincos, remove fallbacks to slow functions.
(__sin): Use improved reduction and simplified do_sincos calculation.
(__cos): Likewise.
* sysdeps/ieee754/dbl-64/s_sincos.c (__sincos): Likewise.
[PATCH 2/7] sin/cos slow paths: remove large range reduction
This patch removes the large range reduction code and defers to the huge range
reduction code. The first level range reducer supports inputs up to 2^27,
which is way too large given that inputs for sin/cos are typically small
(< 10), and optimizing for a smaller range would give a significant speedup.
Input values above 2^27 are practically never used, so there is no reason for
supporting range reduction between 2^27 and 2^48. Removing it significantly
simplifies code and enables further speedups. There is about a 2.3x slowdown
in this range due to __branred being extremely slow (a better algorithm could
easily more than double performance).
[PATCH 1/7] sin/cos slow paths: avoid slow paths for small inputs
This series of patches removes the slow patchs from sin, cos and sincos.
Besides greatly simplifying the implementation, the new version is also much
faster for inputs up to PI (41% faster) and for large inputs needing range
reduction (27% faster).
ULP is ~0.55 with no errors found after testing 1.6 billion inputs across most
of the range with mpsin and mpcos. The number of incorrectly rounded results
(ie. ULP >0.5) is at most ~2750 per million inputs between 0.125 and 0.5,
the average is ~850 per million between 0 and PI.
Tested on AArch64 and x86_64 with no regressions.
The first patch removes the slow paths for the cases where the input is small
and doesn't require range reduction. Update ULP tables for sin, cos and sincos
on AArch64 and x86_64.
* sysdeps/aarch64/libm-test-ulps: Update ULP for sin, cos, sincos.
* sysdeps/ieee754/dbl-64/s_sin.c (__sin): Remove slow paths for small
inputs.
(__cos): Likewise.
* sysdeps/x86_64/fpu/libm-test-ulps: Update ULP for sin, cos, sincos.
Josh Kunz [Fri, 24 Jan 2020 01:37:14 +0000 (17:37 -0800)]
Additional fixes for llvm-as
Unlike GCC, llvm always uses an integrated assembler, which attempts to
recognized all `asm` statements written in the C code. glibc uses some
syntactically invalid asm statements to emit constants into assembly that
are later extracted with a sed or AWK script.
This change fixes two such invalid `asm` statements by wrapping the
output in a `.ascii` directive.. This does not break the sed/AWK (the same
special sequence is output) but it makes the statement syntactically valid.
See cf8e3f8757 for a previous fix for the same issue.
Joseph Myers [Fri, 18 May 2018 11:57:15 +0000 (11:57 +0000)]
Fix year 2039 bug for localtime with 64-bit time_t (bug 22639).
Bug 22639 reports localtime failing to handle time offset transitions
correctly in 2039 and later on platforms with 64-bit time_t.
The problem is the use of SECSPERDAY (constant 86400) in calculations
such as
t = ((year - 1970) * 365
+ /* Compute the number of leapdays between 1970 and YEAR
(exclusive). There is a leapday every 4th year ... */
+ ((year - 1) / 4 - 1970 / 4)
/* ... except every 100th year ... */
- ((year - 1) / 100 - 1970 / 100)
/* ... but still every 400th year. */
+ ((year - 1) / 400 - 1970 / 400)) * SECSPERDAY;
where t is of type time_t and year is of type int. Before my commit 92bd70fb85bce57ac47ba5d8af008736832c955a (an update from tzcode,
included in 2.26 and later releases), SECSPERDAY was obtained from a
file imported from tzcode, where the value included a cast to
int_fast32_t. On 64-bit platforms, glibc defines int_fast32_t to be
long int, so 64-bit, but my patch resulted in it changing to int.
(The bug would probably have existed even before my patch for x32,
which has 64-bit time_t but 32-bit int_fast32_t, but I haven't
verified that.)
This patch fixes the problem by including a cast to time_t in the
definition of SECSPERDAY. (64-bit time support for 32-bit systems
should move such code that isn't a public interface to using the
internal 64-bit version of time_t throughout.)
Tested for x86_64 and x86.
[BZ #22639]
* time/tzset.c (SECSPERDAY): Cast to time_t.
* time/tst-y2039.c: New file.
* time/Makefile (tests): Add tst-y2039.
These comments should make it easier to see the (small) diff introduced
in cf8e3f8757. Without these comments, the diff may get list on a future
upstream merge.
Josh Kunz [Wed, 7 Aug 2019 18:40:50 +0000 (11:40 -0700)]
Make gen-XX-const scripts work with llvm-as
The gen-as-const and gen-py-const scripts are used to generate integer constant
definitions from a list of constant C-expressions. This is achieved by
generating a C program with inline `asm` statements, that depend on
these constant expressions. During compilation, the constant expressions
are evaluated, and included in the inline asm. The build process
generates only the assembly, and then used `sed` to extract the values
from the assembly text.
This is clever. It allows the build process to extract the value of C
statements built under the target architecture. The implementation is a
bit fragile, but it is not immediately obvious to me how it could be
improved.
This change slightly modifies `gen-as-const` and `gen-py-const` to emit
valid assembly directives instead of invalid directives that were
previously emitted. Since the values are extracted via string parsing,
this has no effect on the values extracted. This is needed because the
LLVM assembler validates all statements before emitting them, whereas it
appears GCC will literally emit any `asm` directives without validation
or recognition.
According to ARMv8 architecture reference manual section C7.2.188, SIMD MOV (to
general) instruction format is
MOV <Xd>, <Vn>.D[<index>]
gas appears to accept "<Vn>.2D[<index>]" as well, but clang's assembler does
not. C.f. https://community.arm.com/developer/ip-products/processors/f/cortex-a-forum/5214/aarch64-assembly-syntax-for-armclang
Zack Weinberg [Wed, 7 Feb 2018 20:45:58 +0000 (15:45 -0500)]
[BZ #19239] Don't include sys/sysmacros.h from sys/types.h.
This completes the deprecation and removal of this inclusion, which
was begun in the 2.25 release.
* posix/sys/types.h: Don't include sys/sysmacros.h.
* misc/sys/sysmacros.h: Remove the conditional deprecation
warnings for the macros defined by this header.
Raman Tenneti [Fri, 27 Aug 2021 23:36:05 +0000 (16:36 -0700)]
Changes to compile glibc-2.27 on PPC (Power8) with clang.
+ Use DOT_MACHINE macro instead of ".machine" instruction.
+ Use __isinf and __isinff instead of builtin versions.
+ In s_logb, s_logbf and s_logbl functions, used float versions to
calculate "ret = x & 0x7f800000;" expression.
projects / thirdparty / glibc.git / log
