Pádraig Brady [Wed, 2 Oct 2024 15:33:42 +0000 (16:33 +0100)]
tests: fix skipping of mtab simulation tests
Where rpl_fopen() is used rather than fopen(),
wrapping fopen() is ineffective.
Note rpl_fopen() is used as of glibc-2.39 at least
(due to fflush and fclose being replaced).
* tests/df/no-mtab-status.sh: Wrap open() rather than fopen().
* tests/df/skip-duplicates.sh: Likewise.
Bernhard Voelker [Sun, 29 Sep 2024 16:27:00 +0000 (18:27 +0200)]
maint: avoid sc_preprocessor_indentation failure in factor.c
Prompted by the following 'make syntax-check' failure:
cppi: src/factor.c: line 175: not properly indented
cppi: src/factor.c: line 176: not properly indented
maint.mk: incorrect preprocessor indentation
make: *** [cfg.mk:750: sc_preprocessor_indentation] Error 1
Paul Eggert [Mon, 30 Sep 2024 05:04:02 +0000 (22:04 -0700)]
ls: use fewer xattr-related syscalls
* src/ls.c: Do not include <selinux/selinux.h> or "smack.h".
Include <linux/attr.h> if HAVE_LINUX_ATTR_H, for XATTR_NAME_CAPS.
(free_ent): Use aclinfo_scontext_free to free f->scontext.
(getfilecon_cache): Remove; no longer needed.
(file_has_aclinfo_cache): Rename from file_has_acl_cache,
and use file_has_aclinfo instead of file_has_acl. All uses changed.
(gobble_file): Use file_has_aclinfo instead of file_has_acl, so
that we get more info about the file before deciding whether to
issue further syscalls for it. Let file_has_aclinfo worry about
smack and SELinux. Call has_capability only if the xattr list
mentions XATTR_NAME_CAPS.
Paul Eggert [Fri, 27 Sep 2024 23:16:43 +0000 (16:16 -0700)]
factor: eliminate print_uuint recursion
* src/factor.c (lbuf_putint_append): New function, with
most of the old lbuf_putint body. Do the umaxtostr stuff
by hand so that we needn’t worry about the trailing NUL.a
Do the string copy by hand since the string is so short.
(lbuf_putint): Reimplement in terms of lbuf_putint_append.
Omit last arg, which is no longer needed. All callers changed.
(print_uuint): Rewrite to avoid recursion, using
lbuf_putint_append for the usual case.
Paul Eggert [Fri, 27 Sep 2024 03:20:05 +0000 (20:20 -0700)]
factor: macro refactoring
* src/factor.c (W_TYPE_SIZE): Simplify by always defining
to UINTMAX_WIDTH.
(W): Remove. All uses replaced by W_TYPE_SIZE.
We no longer need one of its static_asserts.
Paul Eggert [Fri, 27 Sep 2024 03:01:58 +0000 (20:01 -0700)]
factor: improve gmp-related buffering
Previously, the code used stdio buffers for gmp numbers,
and did its own buffering for smaller numbers. This meant
for more flushing than was needed. The code now uses its
own buffering for all standard output, which makes for
less flushing and fewer writes.
* src/factor.c (lbuf_half_flush): New function, taken from the
body of lbuf_putnl.
(lbuf_putnl): Use it.
(lbuf_putmpz): New function, to output an mpz without using stdio.
(print_factors): Output via functions instead of via stdio.
Paul Eggert [Thu, 26 Sep 2024 22:36:44 +0000 (15:36 -0700)]
factor: improve fd buffering
* src/factor.c (struct lbuf_, lbuf, lbuf_alloc): Remove.
All uses removed.
(FACTOR_PIPE_BUF): Now a constant instead of a macro.
Increase to PIPE_BUF if available.
(lbuf_buf, lbuffered): New static vars, replacing lbuf.
All uses changed.
(lbuf_flush): Avoid unlikely recursion on write failure.
(lbuf_putc): Now simply adds a byte to the buffer.
(lbuf_putnl): Do the work of the old lbuf_putc ('\n').
Use changed. Use memrchr to find the newline.
(lbuf_putint): Widths are now int, not size_t.
Paul Eggert [Thu, 26 Sep 2024 17:25:51 +0000 (10:25 -0700)]
factor: mod2 now returns uuint
* src/factor.c (uuset): New function.
(mod2): Return uuint rather than having half the returned value
stored via a pointer. This makes the code a bit easier to read
and can help the compiler avoid aliasing issues. All callers changed.
Paul Eggert [Thu, 26 Sep 2024 16:31:35 +0000 (09:31 -0700)]
factor: new type uuint
This refactors to add a new type, a uintmax_t pair, which
can simplify some code without slowing it down.
* src/factor.c (uuint): New type.
(lo, hi, hiset, make_uuint): New functions.
(struct factors.plarge): Use the new type. All uses changed.
Paul Eggert [Wed, 25 Sep 2024 23:53:52 +0000 (16:53 -0700)]
factor: switch to stdc_leading_zeros etc
* bootstrap.conf (gnulib_modules): Add stdc_trailing_zeros.
* cfg.mk (_gl_TS_unmarked_extern_vars): Remove factor_clz_tab,
as it’s no longer present.
* src/factor.c: Include stdbit.h.
(__clz_tab, factor_clz_tab): Remove.
(ASSERT, UHWtype, __GMP_DECLSPEC): Use simpler way to pacify
-Wunused-macros.
(count_leading_zeros, count_trailing_zeros):
Remove. All uses replaced by stdc_leading_zeros, stdc_trailing_zeros.
(factor_using_division, prime2_p): Add a couple of ‘assume’s
so that GCC knows the stdc_* calls are nonzero and can
optimize accordingly.
Paul Eggert [Wed, 25 Sep 2024 22:59:09 +0000 (15:59 -0700)]
factor: port to platforms
* src/factor.c (mod2): Work even if cntd <= cnta. The old version
of the code assumed that shifts by N had unspecified behavior
unless 0 <= N < wordsize. Although this assumption is portable to
all known practical platforms, the C standard says these shifts
have undefined behavior and some pedantic platforms check this.
* tests/factor/create-test.sh:
* tests/local.mk (factor_tests): New test t37.
* src/sort.c (usage): Don't mention the ambiguous "manual",
rather "full documentation", echoing the language at the
bottom of each coreutils man page.
Fixes https://bugs.gnu.org/72914
* man/env.x: Avoid confusion in the [OPTIONS] section
by renaming to [SCRIPT OPTION HANDLING], and removing info
regarding default signal handling, which is best
restricted to the full info manual.
Addresses https://bugs.gnu.org/72914
* src/echo.c (usage): Use printf(1) rather than 'printf',
which is marked up more appropriately, and can be
referenced by some man page readers.
Fixes https://bugs.gnu.org/72914
* src/printf.c (print_formatted): Add support for %i$ indexed args.
* tests/printf/printf-indexed.sh: Add a new file of test cases.
* tests/local.mk: Reference the new test file.
* doc/coreutils.texi (printf invocation): Mention how mixed
processing of indexed and sequential references are supported,
unlike the printf(2) library function.
* NEWS: Mention the new (POSIX:2024) feature.
Addresses https://bugs.gnu.org/73068
Pádraig Brady [Wed, 28 Aug 2024 11:33:17 +0000 (12:33 +0100)]
all: fix error checking in gl/lib/xdectoint.c
This issue was noticed with -flto on GCC 14.2.1
* gl/lib/xdectoint.c (__xnumtoint): Only inspect the
returned value if LONGINT_INVALID is not set,
as the returned value is uninitialized in that case.
Fixes https://bugs.gnu.org/72842
Pádraig Brady [Mon, 19 Aug 2024 11:43:09 +0000 (12:43 +0100)]
install: dereference source symlinks when comparing
* NEWS: Mention the change in behavior.
* src/install.c (need_copy): s/lstat/stat/ for the source.
* tests/install/install-C.sh: Add test cases
(and improve existing test case which wan't valid
due to the existing non standard modes on test files).
Addresses https://bugs.gnu.org/72707
Paul Eggert [Thu, 15 Aug 2024 07:22:05 +0000 (00:22 -0700)]
build: update gnulib submodule to latest
* boostrap.conf (gnulib.modules): Add xvasprintf, which
had been omitted by mistake.
* src/copy.c, src/dd.c, src/test.c: Don't include verror.h,
as Gnulib removed it.
Pádraig Brady [Sun, 11 Aug 2024 19:34:08 +0000 (20:34 +0100)]
ls: add support for explicit file name sorting
Support overriding previous sorting options
with an explicit --sort=name option.
* doc/coreutils.texi (ls invocation): Document the new option.
* src/ls.c (usage): Likewise.
(sort_args): Add the "name" entry, and sort to be consistent
with the ordering presented in --help.
* tests/ls/ls-time.sh: Add test cases.
* NEWS: Mention the new feature.
Suggested by: Tzvetelin Katchov
Pádraig Brady [Sun, 11 Aug 2024 13:36:21 +0000 (14:36 +0100)]
doc: printf %b: clarify octal processing
* src/printf.c: Remove redundant comment.
State explicitly that the leading 0 is the exception
from normal escape processing. Remove a full stop for consistency.
* doc/coreutils.texi (printf invocation): Add a reference
to C99 string escapes since these are not mentioned
in the referenced glibc printf info. Also explicitly state
the leading 0 exception. Also use NNN rather than OOO
to be consistent with the --help documentation.
Also remove and extraneous '\' and fix grammar in the info
regarding the ninth bit.
Addresses https://bugs.gnu.org/72657
Paul Eggert [Sun, 11 Aug 2024 02:30:46 +0000 (19:30 -0700)]
maint: adjust to Gnulib safe_read etc. changes
Although these patches don’t affect user-visible behavior,
they do clean up the source code a bit, and the
machine code should be a tiny bit more efficient.
* src/cat.c (simple_cat, cat):
* src/csplit.c (read_input):
* src/head.c (copy_fd, elide_tail_bytes_pipe)
(elide_tail_lines_pipe, elide_tail_lines_seekable, head_bytes)
(head_lines):
* src/install.c (have_same_content):
* src/tac-pipe.c (buf_init_from_stdin):
* src/tac.c (tac_seekable, copy_to_temp):
* src/tail.c (dump_remainder, file_lines, pipe_lines)
(pipe_bytes, start_bytes, start_lines, tail_forever_inotify):
* src/tr.c (plain_read):
Adjust to recent Gnulib changes by using new types
for safe_read, safe_write, full_read, full_write.
Paul Eggert [Sun, 11 Aug 2024 05:19:17 +0000 (22:19 -0700)]
head: fix overflows in elide_tail_bytes_pipe
Not clear that the overflows could be exploited,
but they made the code confusing.
* src/head.c (elide_tail_bytes_pipe): Don’t convert uintmax_t
to size_t first thing; wait until it’s known the value will fit,
and then use idx_t rather than size_t to prefer signed types.
Prefer idx_t in nearby code, too.
Rename locals n_elide_0 to n_elide (for consistency elsewhere)
and n_elide to in_elide.
Remove bogus (SIZE_MAX < n_elide + READ_BUFSIZE) test;
in the typical case where n_elide’s type was the same as
that of SIZE_MAX, the test never succeeded, and in the
less-common case where n_elide was wider than size_t,
the addition could silently overflow, causing the test
to fail when it should succeed. The test is not needed anyway now.
Add static asserts to document code assumptions.
Redo the ! (n_elide <= HEAD_TAIL_PIPE_BYTECOUNT_THRESHOLD) case
so that it works with enormous values of n_elide even on
32-bit platforms; for example, n_bufs is now uintmax_t not size_t.
Simplify by using xpalloc instead of by-hand code.
Remove bogus ‘if (rem)’ test, as rem is always nonzero.
Paul Eggert [Sun, 11 Aug 2024 02:02:44 +0000 (19:02 -0700)]
tail: support counts > 2**64
* src/tail.c (tail_lines): If skipping all input, use lseek if
possible.
(parse_options): Allow counts to exceed 2**64.
(main): Don’t subtract 1 from UINTMAX_MAX, since it stands
for infinity in this context.
(main): Also don’t read anything when given infinite elisions.
* tests/tail/tail.pl: Adjust to match new behavior. Rename err-5
test to big-c and expect the invocation to succeed, since ‘tail
-c99999999999999999999’ now succeeds instead of (unnecessarily)
failing.
Paul Eggert [Sun, 11 Aug 2024 01:49:54 +0000 (18:49 -0700)]
head: support counts > 2**64
* src/head.c (head): Optimize for -n-HUGE, where HUGE exceeds
2**64 - 2.
(string_to_integer): Return UINTMAX_MAX for too-large numbers,
instead of failing.
(main): Omit no-lnger-necessary test for byte count overflow.
Paul Eggert [Sat, 10 Aug 2024 20:00:33 +0000 (13:00 -0700)]
maint: distinguish EOVERFLOW vs ERANGE better
Also, prepare for allowing some arguments to overflow
without that being an error.
* gl/lib/xdectoint.c: Do not include stddef.h,
since we no longer use ‘unreachable’.
(xnumtoimax, xnumtoumax, __xnumtoint):
New arg FLAGS. All callers changed.
Stop using __xdectoint_signed. All definers removed.
* gl/lib/xdectoint.h (XTOINT_MIN_QUIET, XTOINT_MAX_QUIET)
(XTOINT_MIN_RANGE, XTOINT_MAX_RANGE): New flag constants.
* src/fmt.c (main):
* src/fold.c (main):
* src/nl.c (main):
* src/pr.c (getoptnum):
* src/split.c (main):
Use XTOINT_MIN_RANGE and XTOINT_MAX_RANGE if appropriate.
* src/pr.c (getoptnum): Return int rather than returning void
and storing through int *.
* src/stty.c (apply_settings):
Use ckd_add to check for overflow instead of doing it by hand.
(integer_arg): Accept and return uintmax_t, not unsigned long.
Pádraig Brady [Sat, 3 Aug 2024 12:37:33 +0000 (13:37 +0100)]
tests: ensure utils support writing to a closed pipe
* tests/misc/write-errors.sh: A closed pipe is a common scenario,
and should not induce an error. The general case is discussed at:
https://www.pixelbeat.org/programming/sigpipe_handling.html
Paul Eggert [Sun, 4 Aug 2024 05:59:12 +0000 (22:59 -0700)]
shuf: avoid integer overflow on huge inputs
* gl/lib/randperm.c: Include <stdckdint.h>.
(randperm_bound): Return SIZE_MAX if the multiplication overflows.
Do not overflow when converting bit count to byte count.
Paul Eggert [Sun, 4 Aug 2024 05:31:20 +0000 (22:31 -0700)]
shuf: fix randomness bug
Problem reported by Daniel Carpenter <https://bugs.gnu.org/72445>.
* gl/lib/randread.c (randread_new): Fill the ISAAC buffer
instead of storing at most BYTES_BOUND bytes into it.
build: support creating reproducible tarball contents
We already support reproducible builds since commit v8.24-99-gc1b3d6587,
and this adjusts that change to also support reproducible
tarball contents with subsequent runs of `make dist`.
* Makefile.am: Don't create a varying .timestamp file, instead ...
* man/local.mk: Rely on the timestamp of the .tarball-version file.
Fixes https://bugs.gnu.org/72232
Paul Eggert [Tue, 16 Jul 2024 01:53:46 +0000 (18:53 -0700)]
tests: port to noatime file systems
On these file systems the atime is always zero.
Problem found with ZFS on Ubuntu 24.04 LTS.
* tests/stat/stat-birthtime.sh (check_timestamps_updated):
* tests/stat/stat-nanoseconds.sh:
Work even if atimes are always zero.
* tests/stat/stat-nanoseconds.sh:
Fix typo: print_ver_ called before init.sh sourced.
Pádraig Brady [Thu, 27 Jun 2024 17:15:02 +0000 (18:15 +0100)]
ls: treat --time=mtime consistently with other time selectors
* src/ls.c: Track if --time=mtime is explicitly specified,
so that we can apply the GNU extension of sorting by the
specified time, when not displaying (-l not specified),
and not explicitly sorting (-t not specified).
* tests/ls/ls-time.sh: Add / Update test cases.
Fixes https://bugs.gnu.org/71803
Pádraig Brady [Wed, 26 Jun 2024 22:17:12 +0000 (23:17 +0100)]
doc: document the c-maybe quoting style
* doc/coreutils.texi (ls invocation): Document the "c-maybe"
--quoting-style, which was added as an option in 2008.
Reported at https://bugs.debian.org/1074334
The ERE used lacks the grouping of the extensions and therefore would
also match files where the first two patterns are not at the end of
the line:
grep -E '\.sh|\.pl|\.xpl$'
* cfg.mk (sc_tests_list_consistency): Add grouping (...) around the
sub-patterns. While at it, also remove the redundant escaping, i.e.,
\$$ -> $$ to be consistent with the rest of this file.
Pádraig Brady [Tue, 28 May 2024 12:29:15 +0000 (13:29 +0100)]
stat,tail: sync with latest Linux file systems
* src/stat.c (human_fstype): Rename "FUSEBLK" to "FUSE" to sync with
kernel adjustments. Add "bcachefs", and "pidfs". Both are local,
with the latter being similar to "proc" which is also local.
* NEWS: Mention the change in behavior, and the improvement.
Paul Eggert [Fri, 24 May 2024 15:42:14 +0000 (08:42 -0700)]
ls: improve sorting doc
Problem reported by Dan Jaobson (Bug#71171).
* doc/coreutils.texi: Clarify that directory entries are sorted,
not command-line arguments.
* src/ls.c (usage): Be less chatty about -U and
about --group-directories-first.
Pádraig Brady [Tue, 21 May 2024 12:08:45 +0000 (13:08 +0100)]
build: fix build failure in --enable-single-binary mode
* src/local.mk: Avoid overriding automake generated DEPENDENCIES,
so that it applies its adjustments to LDADD to avoid propagating
flags (like -Wl,-rpath) into make targets. This was seen on FreeBSD
where LIBINTL is set to:
/usr/local/lib/libintl.so -Wl,-rpath -Wl,/usr/local/lib
Instead let automake generate a sanitized src_coreutils_DEPENDENCIES
(based on LDADD), which we then augment with the EXTRA_... variable.
Paul Eggert [Fri, 17 May 2024 04:33:08 +0000 (21:33 -0700)]
maint: prefer stdbit.h to count-leading-zeros.h
<stdbit.h> is in C23 and should be more portable in the long run,
now that Gnulib supports it.
* bootstrap.conf (gnulib_modules): Remove count-leading-zeros.
Add stdc_leading_zeros.
* gl/lib/randperm.c, src/ioblksize.h:
Include stdbit.h instead of count-leading-zeros.h.
* gl/lib/randperm.c (floor_lg): Remove; no longer needed.
(randperm_bound): Use stdc_bit_width instead of floor_lg;
* gl/modules/randperm (Depends-on): Remove count-leading-zeros.
Add stdc_bit_width.
* src/ioblksize.h (io_blksize): Use stdc_leading_zeros_ull
instead of count_leading_zeros_ll.
Paul Eggert [Fri, 17 May 2024 05:44:23 +0000 (22:44 -0700)]
maint: port test warnings to GCC 14
* configure.ac: Disable GCC 14’s -Wmissing-variable-declarations
in the test directory, as it’s not worth the aggravation there.
Likewise for GCC's -Wsuggest-attribute=cold.
Paul Eggert [Fri, 17 May 2024 04:09:28 +0000 (21:09 -0700)]
sort: better -Wmissing-variable-declarations
* src/sort.c: Ignore -Wmissing-variable-declarations only
with GCC 14 and newer, since it didn’t exist earlier.
Ignore the warning only when including md5.h, where it
needs to be ignored, as the warning might be useful elsewhere.
Paul Eggert [Fri, 17 May 2024 04:06:58 +0000 (21:06 -0700)]
cksum: improve API consistency checking
* src/cksum.c (main) [CRCTAB]: Generate updated crctab.c (see below).
* src/crctab.c: Include cksum.h, to check consistency
between decl and defn. Include stdio.h since cksum.h needs it.
This patch is part of work done for a project from Google Summer of
Code, see the project details at
<https://summerofcode.withgoogle.com/programs/2024/projects/E9Jp7RUx>.
* src/sleep.c (usage): Directly mention the floating-point option,
which is typical for sleeping milliseconds.
Also reorganize the text to be 3 lines rather than 4.
This was seen to improve AVX performance by about 10%
on an AMD 7800X3D (Ryzen 7 (2023)) CPU,
while having neutral AVX performance,
on an Intel i7-5600U (Broadwell-U (2015)) CPU.
With avx not enabled, this gives about a 3% performance boost,
on an Intel i7-5600U.
* src/wc.c: Use the centrally configured optimum buffer size.
* src/wc_avx2.c: Likewise.
* NEWS: Mention the change in performance.
* src/wc_avx2.c (wc_lines_avx2): Change from
_mm256_sub_epi8() + _mm256_sad_epu8() to
_mm256_movemask_epi8() + __builtin_popcount().
This will allow adjusting the I/O size above 16KiB.
* configure.ac: Align check with routines used in wc_avx2.c.