Pádraig Brady [Mon, 19 Aug 2024 11:43:09 +0000 (12:43 +0100)]
install: dereference source symlinks when comparing
* NEWS: Mention the change in behavior.
* src/install.c (need_copy): s/lstat/stat/ for the source.
* tests/install/install-C.sh: Add test cases
(and improve existing test case which wan't valid
due to the existing non standard modes on test files).
Addresses https://bugs.gnu.org/72707
Paul Eggert [Thu, 15 Aug 2024 07:22:05 +0000 (00:22 -0700)]
build: update gnulib submodule to latest
* boostrap.conf (gnulib.modules): Add xvasprintf, which
had been omitted by mistake.
* src/copy.c, src/dd.c, src/test.c: Don't include verror.h,
as Gnulib removed it.
Pádraig Brady [Sun, 11 Aug 2024 19:34:08 +0000 (20:34 +0100)]
ls: add support for explicit file name sorting
Support overriding previous sorting options
with an explicit --sort=name option.
* doc/coreutils.texi (ls invocation): Document the new option.
* src/ls.c (usage): Likewise.
(sort_args): Add the "name" entry, and sort to be consistent
with the ordering presented in --help.
* tests/ls/ls-time.sh: Add test cases.
* NEWS: Mention the new feature.
Suggested by: Tzvetelin Katchov
Pádraig Brady [Sun, 11 Aug 2024 13:36:21 +0000 (14:36 +0100)]
doc: printf %b: clarify octal processing
* src/printf.c: Remove redundant comment.
State explicitly that the leading 0 is the exception
from normal escape processing. Remove a full stop for consistency.
* doc/coreutils.texi (printf invocation): Add a reference
to C99 string escapes since these are not mentioned
in the referenced glibc printf info. Also explicitly state
the leading 0 exception. Also use NNN rather than OOO
to be consistent with the --help documentation.
Also remove and extraneous '\' and fix grammar in the info
regarding the ninth bit.
Addresses https://bugs.gnu.org/72657
Paul Eggert [Sun, 11 Aug 2024 02:30:46 +0000 (19:30 -0700)]
maint: adjust to Gnulib safe_read etc. changes
Although these patches don’t affect user-visible behavior,
they do clean up the source code a bit, and the
machine code should be a tiny bit more efficient.
* src/cat.c (simple_cat, cat):
* src/csplit.c (read_input):
* src/head.c (copy_fd, elide_tail_bytes_pipe)
(elide_tail_lines_pipe, elide_tail_lines_seekable, head_bytes)
(head_lines):
* src/install.c (have_same_content):
* src/tac-pipe.c (buf_init_from_stdin):
* src/tac.c (tac_seekable, copy_to_temp):
* src/tail.c (dump_remainder, file_lines, pipe_lines)
(pipe_bytes, start_bytes, start_lines, tail_forever_inotify):
* src/tr.c (plain_read):
Adjust to recent Gnulib changes by using new types
for safe_read, safe_write, full_read, full_write.
Paul Eggert [Sun, 11 Aug 2024 05:19:17 +0000 (22:19 -0700)]
head: fix overflows in elide_tail_bytes_pipe
Not clear that the overflows could be exploited,
but they made the code confusing.
* src/head.c (elide_tail_bytes_pipe): Don’t convert uintmax_t
to size_t first thing; wait until it’s known the value will fit,
and then use idx_t rather than size_t to prefer signed types.
Prefer idx_t in nearby code, too.
Rename locals n_elide_0 to n_elide (for consistency elsewhere)
and n_elide to in_elide.
Remove bogus (SIZE_MAX < n_elide + READ_BUFSIZE) test;
in the typical case where n_elide’s type was the same as
that of SIZE_MAX, the test never succeeded, and in the
less-common case where n_elide was wider than size_t,
the addition could silently overflow, causing the test
to fail when it should succeed. The test is not needed anyway now.
Add static asserts to document code assumptions.
Redo the ! (n_elide <= HEAD_TAIL_PIPE_BYTECOUNT_THRESHOLD) case
so that it works with enormous values of n_elide even on
32-bit platforms; for example, n_bufs is now uintmax_t not size_t.
Simplify by using xpalloc instead of by-hand code.
Remove bogus ‘if (rem)’ test, as rem is always nonzero.
Paul Eggert [Sun, 11 Aug 2024 02:02:44 +0000 (19:02 -0700)]
tail: support counts > 2**64
* src/tail.c (tail_lines): If skipping all input, use lseek if
possible.
(parse_options): Allow counts to exceed 2**64.
(main): Don’t subtract 1 from UINTMAX_MAX, since it stands
for infinity in this context.
(main): Also don’t read anything when given infinite elisions.
* tests/tail/tail.pl: Adjust to match new behavior. Rename err-5
test to big-c and expect the invocation to succeed, since ‘tail
-c99999999999999999999’ now succeeds instead of (unnecessarily)
failing.
Paul Eggert [Sun, 11 Aug 2024 01:49:54 +0000 (18:49 -0700)]
head: support counts > 2**64
* src/head.c (head): Optimize for -n-HUGE, where HUGE exceeds
2**64 - 2.
(string_to_integer): Return UINTMAX_MAX for too-large numbers,
instead of failing.
(main): Omit no-lnger-necessary test for byte count overflow.
Paul Eggert [Sat, 10 Aug 2024 20:00:33 +0000 (13:00 -0700)]
maint: distinguish EOVERFLOW vs ERANGE better
Also, prepare for allowing some arguments to overflow
without that being an error.
* gl/lib/xdectoint.c: Do not include stddef.h,
since we no longer use ‘unreachable’.
(xnumtoimax, xnumtoumax, __xnumtoint):
New arg FLAGS. All callers changed.
Stop using __xdectoint_signed. All definers removed.
* gl/lib/xdectoint.h (XTOINT_MIN_QUIET, XTOINT_MAX_QUIET)
(XTOINT_MIN_RANGE, XTOINT_MAX_RANGE): New flag constants.
* src/fmt.c (main):
* src/fold.c (main):
* src/nl.c (main):
* src/pr.c (getoptnum):
* src/split.c (main):
Use XTOINT_MIN_RANGE and XTOINT_MAX_RANGE if appropriate.
* src/pr.c (getoptnum): Return int rather than returning void
and storing through int *.
* src/stty.c (apply_settings):
Use ckd_add to check for overflow instead of doing it by hand.
(integer_arg): Accept and return uintmax_t, not unsigned long.
Pádraig Brady [Sat, 3 Aug 2024 12:37:33 +0000 (13:37 +0100)]
tests: ensure utils support writing to a closed pipe
* tests/misc/write-errors.sh: A closed pipe is a common scenario,
and should not induce an error. The general case is discussed at:
https://www.pixelbeat.org/programming/sigpipe_handling.html
Paul Eggert [Sun, 4 Aug 2024 05:59:12 +0000 (22:59 -0700)]
shuf: avoid integer overflow on huge inputs
* gl/lib/randperm.c: Include <stdckdint.h>.
(randperm_bound): Return SIZE_MAX if the multiplication overflows.
Do not overflow when converting bit count to byte count.
Paul Eggert [Sun, 4 Aug 2024 05:31:20 +0000 (22:31 -0700)]
shuf: fix randomness bug
Problem reported by Daniel Carpenter <https://bugs.gnu.org/72445>.
* gl/lib/randread.c (randread_new): Fill the ISAAC buffer
instead of storing at most BYTES_BOUND bytes into it.
build: support creating reproducible tarball contents
We already support reproducible builds since commit v8.24-99-gc1b3d6587,
and this adjusts that change to also support reproducible
tarball contents with subsequent runs of `make dist`.
* Makefile.am: Don't create a varying .timestamp file, instead ...
* man/local.mk: Rely on the timestamp of the .tarball-version file.
Fixes https://bugs.gnu.org/72232
Paul Eggert [Tue, 16 Jul 2024 01:53:46 +0000 (18:53 -0700)]
tests: port to noatime file systems
On these file systems the atime is always zero.
Problem found with ZFS on Ubuntu 24.04 LTS.
* tests/stat/stat-birthtime.sh (check_timestamps_updated):
* tests/stat/stat-nanoseconds.sh:
Work even if atimes are always zero.
* tests/stat/stat-nanoseconds.sh:
Fix typo: print_ver_ called before init.sh sourced.
Pádraig Brady [Thu, 27 Jun 2024 17:15:02 +0000 (18:15 +0100)]
ls: treat --time=mtime consistently with other time selectors
* src/ls.c: Track if --time=mtime is explicitly specified,
so that we can apply the GNU extension of sorting by the
specified time, when not displaying (-l not specified),
and not explicitly sorting (-t not specified).
* tests/ls/ls-time.sh: Add / Update test cases.
Fixes https://bugs.gnu.org/71803
Pádraig Brady [Wed, 26 Jun 2024 22:17:12 +0000 (23:17 +0100)]
doc: document the c-maybe quoting style
* doc/coreutils.texi (ls invocation): Document the "c-maybe"
--quoting-style, which was added as an option in 2008.
Reported at https://bugs.debian.org/1074334
The ERE used lacks the grouping of the extensions and therefore would
also match files where the first two patterns are not at the end of
the line:
grep -E '\.sh|\.pl|\.xpl$'
* cfg.mk (sc_tests_list_consistency): Add grouping (...) around the
sub-patterns. While at it, also remove the redundant escaping, i.e.,
\$$ -> $$ to be consistent with the rest of this file.
Pádraig Brady [Tue, 28 May 2024 12:29:15 +0000 (13:29 +0100)]
stat,tail: sync with latest Linux file systems
* src/stat.c (human_fstype): Rename "FUSEBLK" to "FUSE" to sync with
kernel adjustments. Add "bcachefs", and "pidfs". Both are local,
with the latter being similar to "proc" which is also local.
* NEWS: Mention the change in behavior, and the improvement.
Paul Eggert [Fri, 24 May 2024 15:42:14 +0000 (08:42 -0700)]
ls: improve sorting doc
Problem reported by Dan Jaobson (Bug#71171).
* doc/coreutils.texi: Clarify that directory entries are sorted,
not command-line arguments.
* src/ls.c (usage): Be less chatty about -U and
about --group-directories-first.
Pádraig Brady [Tue, 21 May 2024 12:08:45 +0000 (13:08 +0100)]
build: fix build failure in --enable-single-binary mode
* src/local.mk: Avoid overriding automake generated DEPENDENCIES,
so that it applies its adjustments to LDADD to avoid propagating
flags (like -Wl,-rpath) into make targets. This was seen on FreeBSD
where LIBINTL is set to:
/usr/local/lib/libintl.so -Wl,-rpath -Wl,/usr/local/lib
Instead let automake generate a sanitized src_coreutils_DEPENDENCIES
(based on LDADD), which we then augment with the EXTRA_... variable.
Paul Eggert [Fri, 17 May 2024 04:33:08 +0000 (21:33 -0700)]
maint: prefer stdbit.h to count-leading-zeros.h
<stdbit.h> is in C23 and should be more portable in the long run,
now that Gnulib supports it.
* bootstrap.conf (gnulib_modules): Remove count-leading-zeros.
Add stdc_leading_zeros.
* gl/lib/randperm.c, src/ioblksize.h:
Include stdbit.h instead of count-leading-zeros.h.
* gl/lib/randperm.c (floor_lg): Remove; no longer needed.
(randperm_bound): Use stdc_bit_width instead of floor_lg;
* gl/modules/randperm (Depends-on): Remove count-leading-zeros.
Add stdc_bit_width.
* src/ioblksize.h (io_blksize): Use stdc_leading_zeros_ull
instead of count_leading_zeros_ll.
Paul Eggert [Fri, 17 May 2024 05:44:23 +0000 (22:44 -0700)]
maint: port test warnings to GCC 14
* configure.ac: Disable GCC 14’s -Wmissing-variable-declarations
in the test directory, as it’s not worth the aggravation there.
Likewise for GCC's -Wsuggest-attribute=cold.
Paul Eggert [Fri, 17 May 2024 04:09:28 +0000 (21:09 -0700)]
sort: better -Wmissing-variable-declarations
* src/sort.c: Ignore -Wmissing-variable-declarations only
with GCC 14 and newer, since it didn’t exist earlier.
Ignore the warning only when including md5.h, where it
needs to be ignored, as the warning might be useful elsewhere.
Paul Eggert [Fri, 17 May 2024 04:06:58 +0000 (21:06 -0700)]
cksum: improve API consistency checking
* src/cksum.c (main) [CRCTAB]: Generate updated crctab.c (see below).
* src/crctab.c: Include cksum.h, to check consistency
between decl and defn. Include stdio.h since cksum.h needs it.
This patch is part of work done for a project from Google Summer of
Code, see the project details at
<https://summerofcode.withgoogle.com/programs/2024/projects/E9Jp7RUx>.
* src/sleep.c (usage): Directly mention the floating-point option,
which is typical for sleeping milliseconds.
Also reorganize the text to be 3 lines rather than 4.
This was seen to improve AVX performance by about 10%
on an AMD 7800X3D (Ryzen 7 (2023)) CPU,
while having neutral AVX performance,
on an Intel i7-5600U (Broadwell-U (2015)) CPU.
With avx not enabled, this gives about a 3% performance boost,
on an Intel i7-5600U.
* src/wc.c: Use the centrally configured optimum buffer size.
* src/wc_avx2.c: Likewise.
* NEWS: Mention the change in performance.
* src/wc_avx2.c (wc_lines_avx2): Change from
_mm256_sub_epi8() + _mm256_sad_epu8() to
_mm256_movemask_epi8() + __builtin_popcount().
This will allow adjusting the I/O size above 16KiB.
* configure.ac: Align check with routines used in wc_avx2.c.
Nikolay Nechaev [Sun, 5 May 2024 09:06:18 +0000 (12:06 +0300)]
maint: factor out the common show_date functionality
* src/show-date.{h,c}: Declaration and definition of show_date.
* src/du.c: Wse the common show_date instead of the previous local
function.
* src/date.c: Wse the common show_date via a wrapper show_date_helper.
* src/local.mk: Corresponding adjustments.
Pádraig Brady [Fri, 3 May 2024 09:18:50 +0000 (10:18 +0100)]
cp: actually support --update=none-fail
* src/cp.c: Add the entries for the --update=none-fail option.
* tests/mv/update.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/70727
build: bootstrap with python gnulib-tool by default
* gnulib: Update to support bootstrapping with python by default.
* bootstrap: Sync with gnulib.
* cfg.mk: Don't force python implementation with `make world`,
rather rely on the auto selection of python if appropriate.
doc: sort: give example for sorting on the last field
* doc/coreutils.texi (sort invocation): Give a DSU example
for sorting names which may have a variable number of fields.
Addresses https://bugs.gnu.org/70532
build: provide way to bootstrap with python gnulib-tool
* cfg.mk: Add a new "world" default target so that one
can bootstrap (using the python implementation), configure,
and make, by using `make -f cfg.mk`.
* gnulib: Update to latest primarily to test the
bootstrap python implementation which is now in beta test.
* README-hacking: Document the `make -f cfg.mk` shortcut.
Paul Eggert [Sat, 20 Apr 2024 04:44:32 +0000 (21:44 -0700)]
tail: avoid infloop with -c on /dev/zero
Problem reported by Ionut Nicula in:
https://bugs.gnu.org/70477
* src/tail.c (tail_bytes): Do not loop forever on commands
like 'tail -c 4096 /dev/zero'.
* tests/tail/tail-c.sh: Test this fix.
* src/join.c (main): s/field/file/ in the error message
for -a and -v, introduced in TEXTUTILS-1_13-24-g6f63d53e1.
Reported at https://bugs.debian.org/1068864
Paul Eggert [Sat, 6 Apr 2024 22:13:23 +0000 (15:13 -0700)]
cat: don’t trust st_size on /proc files
* src/cat.c (main):
Improve test for when copying will exhaust the output device.
Do not rely on st_size, which is unreliable in /proc.
Use lseek instead; this is good enough here.
* tests/cat/cat-self.sh: Test the relaxation of the heuristic
for self-copying.
Bruno Haible [Wed, 27 Mar 2024 12:30:31 +0000 (12:30 +0000)]
tests: od: avoid outputting undefined floating point values
od was seen to abort() on glibc on ia64 and m68k with the error:
Fatal glibc error: printf_fp.c:501 (__printf_fp_buffer_1):
assertion failed:
cy == 1 || (p.frac[p.fracsize - 2] == 0 && p.frac[0] == 0)
* tests/od/od-multiple-t.sh: Avoid outputting long double floats
to avoid undefined behavior. 'float' and 'double' are standardized
by IEEE 754 (except on Linux/m68k) and don't have undefined values.
Pádraig Brady [Tue, 26 Mar 2024 19:17:16 +0000 (19:17 +0000)]
tests: numfmt: fix false failures on some systems
* tests/misc/numfmt.pl: Verify that printf field width specs
count characters and not bytes before enabling locale tests.
This was seen on FreeBSD 14.0 and Solaris 11 OpenIndiana.
Reported by Bruno Haible
Pádraig Brady [Tue, 26 Mar 2024 18:27:00 +0000 (18:27 +0000)]
ls: avoid ENOENT from readdir() on FreeBSD 14
* src/ls.c (print_dir): readdir() on FreeBSD 14 was
seen to pass ENOENT through. ENOENT in this context
means "Directory unlinked but still open".
Reported by Bruno Haible with tests/ls/removed-directory.sh
Pádraig Brady [Tue, 26 Mar 2024 15:02:17 +0000 (15:02 +0000)]
tests: avoid false failure due to mismatched isblank()
There is a mismatch between isblank() used by tr and c32isblank() now
used by uniq on Solaris 11 OpenIndiana. isblank() was seen to return
true for non breaking space, while c32isblank() returned false.
Interestingly on Solaris, non breaking space is considered a blank
character, and isblank() and c32isblank() honor this in all locales.
* tests/uniq/uniq.pl: Adjust the blank check to use join(1) rather than
tr(1), as join uses the same blank determination routines as uniq(1).
* src/copy.c (copy_internal): On systems that don't support ACLs,
the fallback default chmod done on directories should maintain
the set-group-ID, as that's generally auto-set by the system.
* NEWS: Mention the fix.
Reported by Bruno Haible on Alpine (with tests/cp/preserve-mode.sh)
Pádraig Brady [Mon, 25 Mar 2024 18:27:13 +0000 (18:27 +0000)]
tests: avoid false failure with partial locale info
* tests/misc/numfmt.pl: Some systems with the fr_FR.UTF-8
locale installed, do not have a thousands grouping character defined.
In this case we skip the locale tests which depend on a non empty
grouping character.
Pádraig Brady [Mon, 25 Mar 2024 12:12:19 +0000 (12:12 +0000)]
tests: fix false failure on systems that can chmod symlinks
* tests/chmod/symlinks.sh: The count of adjusted modes was
one more on systems where symlink modes can be adjusted.
Therefore only include the non symlinks in the count.
Pádraig Brady [Sun, 24 Mar 2024 20:12:53 +0000 (20:12 +0000)]
doc: fix translation issue in chown/chgrp amalgamation
* src/chown-core.h (emit_from_option_description): The conditional
string composition here caused issues for translators.
Instead move to a more general description ...
(src/chown.c (usage): ... here.
Fixes https://bugs.gnu.org/69985