]> git.ipfire.org Git - thirdparty/coreutils.git/log
thirdparty/coreutils.git
2 years agobuild: avoid false -Wmaybe-uninitialized warnings
Pádraig Brady [Thu, 18 May 2023 09:38:11 +0000 (10:38 +0100)] 
build: avoid false -Wmaybe-uninitialized warnings

Allow easily building a debug build for example with:
  make CFLAGS='-O0 -ggdb'

False -Wmaybe-uninitialized warnings hit in different
places depending on the compiler passes used.
These changes were tested with gcc 10.2.1, 12.2.1, and 13.1.1 like:
  for o in g s z fast 0 1 2 3; do
    make clean && make -j$(nproc) CFLAGS="-O$o" || break
  done

* src/digest.c: Disable -Wmaybe-uninitialized that gives
false positive here at -O0.
* src/ln.c: Avoid -Wmaybe-uninitialized that gives
false positive here at -O1.
* src/pr.c: Likewise.
* src/sort.c: Likewise.
* src/tee.c: Avoid -Wmaybe-uninitialized that gives
false positive here at -O3 on gcc 13.1.1 at least.
* src/cp.c: Avoid -Wmaybe-uninitialized that gives
false positive here at -Os on gcc 13.1.1 at least.
* src/copy.c: Avoid -Wmaybe-uninitialized that gives
false positive here at -Og on gcc 13.1.1 at least.
* src/head.c: Likewise.
* src/paste.c: Likewise.

2 years agobuild: gnulib: avoid false -Wstringop-overflow warning
Pádraig Brady [Thu, 18 May 2023 10:40:19 +0000 (11:40 +0100)] 
build: gnulib: avoid false -Wstringop-overflow warning

Tested on gcc 13.1.1 with: make CFLAGS='-O0 -ggdb'

* configure.ac: Disable -Wstringop-overflow for gnulib.
This warning is far too problematic in my experience:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88443
and triggers with gcc -O0 with versions 12,13 at least.

2 years agomaint: cleanups to NEWS
Pádraig Brady [Sat, 13 May 2023 09:33:14 +0000 (10:33 +0100)] 
maint: cleanups to NEWS

* NEWS: Use more consistent wording, ordering, and formatting
for recent entries.

2 years agosplit: advise the kernel of sequential access pattern
Pádraig Brady [Mon, 8 May 2023 20:22:48 +0000 (21:22 +0100)] 
split: advise the kernel of sequential access pattern

As split is often dealing with large files,
ensure we indicate to the kernel our sequential access pattern.
This was seen to operate 5% faster when reading from SSD,
as tested with:

dd bs=1M count=2K if=/dev/urandom of=big.in

for split in split.orig split; do
  # Ensure big file is not cached
  dd of=big.in oflag=nocache conv=notrunc,fdatasync count=0 status=none
  # Test read efficiency
  CWD=$PWD; (cd /dev/shm && time $CWD/src/$split -n2 $CWD/big.in)
done

real    0m9.039s
user    0m0.055s
sys     0m3.510s

real    0m8.568s
user    0m0.056s
sys     0m3.752s

* src/split.c (main): Use fdadvise to help the kernel
choose a more appropriate readahead buffer.
* NEWS: Mention the improvement.

2 years agodoc: adjust build instructions for disabling year 2038 support
Pádraig Brady [Mon, 8 May 2023 11:57:56 +0000 (12:57 +0100)] 
doc: adjust build instructions for disabling year 2038 support

* README-install: Adjust the instructions as per recent gnulib updates.

2 years agobuild: update gnulib submodule to latest
Bernhard Voelker [Sun, 7 May 2023 15:33:49 +0000 (17:33 +0200)] 
build: update gnulib submodule to latest

This fixes failures in "very-expensive" tests on FTS with many
directory entries:
  FAIL: tests/rm/ext3-perf
  FAIL: tests/rm/many-dir-entries-vs-OOM

The following shows the problem in the former of the above tests:
  $ mkdir d && seq 400000 | env -C d xargs touch )
  $ rm -rf d
  rm: traversal failed: d: Operation not supported

Gnulib commit 3f0950f65abb (2023-04-26) introduced this regression
which was fixed again with gnulib commit d4d8abb39eb0.

See discussion in
<https://lists.gnu.org/r/bug-gnulib/2023-05/msg00040.html>

* bootstrap.conf (gnulib_modules): Change "year2038-required" to
"year2038-recommended"; the module has been replaced.
* gnulib: Update to latest.
* tests/init.sh: Likewise.

2 years agodoc: time zone conversion example
Paul Eggert [Sun, 7 May 2023 16:16:29 +0000 (09:16 -0700)] 
doc: time zone conversion example

* doc/coreutils.texi (Examples of date):
Give time zone conversion example.

2 years agodoc: new subsection for date format specs
Paul Eggert [Sun, 7 May 2023 15:59:01 +0000 (08:59 -0700)] 
doc: new subsection for date format specs

* doc/coreutils.texi (Date format specifiers): New subsection,
which groups the date format specifiers without otherwise
changing contents.

2 years agopr: fix parsing of empty arguments
Pádraig Brady [Sat, 6 May 2023 10:22:22 +0000 (11:22 +0100)] 
pr: fix parsing of empty arguments

Before:
  $ pr --expand-tabs=
  pr: '-e' extra characters or invalid number in the argument:
   ‘SHELL=/bin/bash’: Value too large for defined data type

After:
  $ pr --expand-tabs=
  pr: '-e': Invalid argument: ‘’

* src/pr.c (getoptarg): Ensure we don't parse beyond the
end of an empty argument, thus outputting arbitrary stack
info in subsequent error messages.

Addresses https://bugs.debian.org/1035596

2 years agocp: -p --parents: minor cleanup of previous patch
Paul Eggert [Fri, 5 May 2023 18:03:25 +0000 (11:03 -0700)] 
cp: -p --parents: minor cleanup of previous patch

This doesn’t change behavior; it just clarifies the code a bit.
* src/cp.c (re_protect): New arg DST_SRC_NAME, for clarity, and so
that we need to skip '/'s only once.  Caller changed.
Rename a couple of local variables to try to make things clearer.

2 years agocp: -p --parents: fix failure to preserve permissions for absolute paths
Pádraig Brady [Wed, 3 May 2023 16:01:37 +0000 (17:01 +0100)] 
cp: -p --parents: fix failure to preserve permissions for absolute paths

* src/cp.c (re_protect): Ensure copy_acl() is passed an absolute path.
* tests/cp/cp-parents.sh: Add a test case.
* NEWS: Mention the bug.
Fixes https://bugs.gnu.org/63245

2 years agotests: provide more info on DEBUG=yes
Pádraig Brady [Tue, 2 May 2023 21:39:03 +0000 (22:39 +0100)] 
tests: provide more info on DEBUG=yes

* README: State that DEBUG=yes is particularly useful with perl tests.
* tests/split/l-chunk.sh: Use the more standard $DEBUG variable
rather than an internal $DEBUGGING variable.

2 years agodoc: provide more info on the default 32-bit cksum digest
Pádraig Brady [Sun, 30 Apr 2023 21:10:25 +0000 (22:10 +0100)] 
doc: provide more info on the default 32-bit cksum digest

* doc/coreutils.texi (cksum invocation): Say that the default
digest format is 32-bit and based on the Ethernet standard CRC.

2 years agomaint: remove redundant exit status handling
Pádraig Brady [Sun, 30 Apr 2023 12:05:12 +0000 (13:05 +0100)] 
maint: remove redundant exit status handling

* src/numfmt.c: Remove redundant / confusing
use of TIMEOUT_FAILURE.

2 years agomaint: simplify --enable-gcc-warnings='expensive'
Paul Eggert [Thu, 27 Apr 2023 00:14:54 +0000 (17:14 -0700)] 
maint: simplify --enable-gcc-warnings='expensive'

* configure.ac (WERROR_CFLAGS): Omit mention of
-Wno-analyzer-double-free, -Wno-analyzer-null-dereference, and
-Wno-analyzer-use-after-free as manywarnings no longer uses them.

2 years agomaint: suppress GCC 13 false alarms
Paul Eggert [Thu, 27 Apr 2023 00:14:54 +0000 (17:14 -0700)] 
maint: suppress GCC 13 false alarms

* src/csplit.c, src/fmt.c, src/make-prime-list.c, src/nohup.c:
Add pragmas to pacify GCC 13 when coreutils is configured
with --enable-gcc-warnings='expensive'.

2 years agochmod: pacify GCC 13
Paul Eggert [Thu, 27 Apr 2023 00:14:54 +0000 (17:14 -0700)] 
chmod: pacify GCC 13

* src/chmod.c (main): Use xpalloc instead of X2REALLOC,
and make the corresponding variables signed instead of unsigned.
When reallocating the buffer, this grows it by a factor of 1.5, not 2.
This also pacifies gcc -Wanalyzer-null-dereference.

2 years agocsplit: pacify GCC 13
Paul Eggert [Thu, 27 Apr 2023 00:14:54 +0000 (17:14 -0700)] 
csplit: pacify GCC 13

* src/csplit.c (load_buffer): Refactor for clarity.
This also xpacifies gcc -Wanalyzer-use-of-uninitialized-value.
When reallocating the buffer, grow it by a factor of 1.5, not 2.

2 years agobuild: update gnulib submodule to latest
Paul Eggert [Thu, 27 Apr 2023 00:14:54 +0000 (17:14 -0700)] 
build: update gnulib submodule to latest

2 years agotests: more cases for read input diagnostics
Pádraig Brady [Wed, 26 Apr 2023 15:20:50 +0000 (16:20 +0100)] 
tests: more cases for read input diagnostics

* tests/misc/read-errors.sh: Exercise more modes of
various utilities for better read error coverage.
* tests/split/fail.sh: Remove part refactored into the above test.

2 years agouniq: be more specific when diagnosing read errors
Pádraig Brady [Wed, 26 Apr 2023 14:20:47 +0000 (15:20 +0100)] 
uniq: be more specific when diagnosing read errors

* src/uniq.c (check_file): Use the errno when diagnosing read errors.

2 years agobuild: fix build with -mno-ssse3
Jaroslav Skarvada [Wed, 26 Apr 2023 12:48:41 +0000 (13:48 +0100)] 
build: fix build with -mno-ssse3

Avoid the following error with -mno-ssse3:
 inlining failed in call to 'always_inline' '_mm_shuffle_epi8':
 target specific option mismatch

* configure.ac: Ensure we use ssse3 specific code when
checking whether to enable the pclmul cksum implementation.

2 years agopr: fix infinite loop when double spacing
Pádraig Brady [Tue, 25 Apr 2023 13:07:03 +0000 (14:07 +0100)] 
pr: fix infinite loop when double spacing

* src/pr.c (init_parameters): Ensure we avoid a 0 lines_per_body
which was possible when adjusting for double spacing.
That caused print_page() to always return true,
causing an infinite loop.
* tests/pr/pr-tests.pl: Add a test case.
* NEWS: Mention the fix.
Fixes https://bugs.debian.org/1034808

2 years agocopy: reduce verbosity of -i and -u with --verbose
Pádraig Brady [Tue, 25 Apr 2023 10:07:36 +0000 (11:07 +0100)] 
copy: reduce verbosity of -i and -u with --verbose

Since skipping of files is central to the operation of -i and -u,
and with -u one may be updating few files out of many,
reinstate the verbosity of this functionality as it was before 9.3.

* src/copy.c (copy_internal): Only output "skipped" message
with --debug.  Also adjust so message never changes with --debug.
* tests/cp/cp-i.sh: Adjust accordingly.
* tests/mv/mv-n.sh: Likewise.
* tests/cp/debug.sh: Add explicit test case for message.
* NEWS: Mention the change in behavior.

2 years agotests: ensure all utilities that read input diagnose errors
Pádraig Brady [Sat, 22 Apr 2023 20:35:23 +0000 (21:35 +0100)] 
tests: ensure all utilities that read input diagnose errors

* tests/misc/read-errors.sh: Add a new test.
* tests/misc/date-f.sh: Remove unneeded test.
* tests/misc/dircolors.sh: Likewise.
* tests/local.mk: Reference new test, and dereference removed ones.

2 years agofactor: diagnose errors reading the input
Pádraig Brady [Sun, 23 Apr 2023 22:45:08 +0000 (23:45 +0100)] 
factor: diagnose errors reading the input

* src/factor.c (do_stdin): Exit with failure upon read errors.
* NEWS: Mention the bug fix.

2 years agonumfmt: diagnose errors reading the input
Pádraig Brady [Sat, 22 Apr 2023 20:48:51 +0000 (21:48 +0100)] 
numfmt: diagnose errors reading the input

* src/numfmt.c (main): Exit with failure upon read errors.
* NEWS: Mention the bug fix.

2 years agotsort: diagnose errors reading the input
Pádraig Brady [Sat, 22 Apr 2023 20:41:53 +0000 (21:41 +0100)] 
tsort: diagnose errors reading the input

* src/tsort.c (tsort): Check for errors after readtoken().
* NEWS: Mention the bug fix.

2 years agocksum: fix failure to diagnose read errors with crc32
Pádraig Brady [Sat, 22 Apr 2023 14:59:48 +0000 (15:59 +0100)] 
cksum: fix failure to diagnose read errors with crc32

The default crc32 mode fails to diagnose read errors.

* src/cksum.c (cksum_slice8): Fix the check for read errors.
(cksum_pclmul): Likewise.
* NEWS: Mention the bug fix.

2 years agotests: avoid failure when cp fails for proc files
Andreas Schwab [Mon, 24 Apr 2023 08:15:01 +0000 (10:15 +0200)] 
tests: avoid failure when cp fails for proc files

When run under QEmu emulation emulated /proc files have
unstable inode numbers.

* tests/cp/proc-short-read.sh: Skip if unstable inode numbers detected.

2 years agoinstall: support stripping files with a leading hyphen
Pádraig Brady [Fri, 21 Apr 2023 18:07:02 +0000 (19:07 +0100)] 
install: support stripping files with a leading hyphen

* src/install.c (strip): Prepend "./" to file names with a leading "-".
* tests/install/strip-program.sh: Add a test case.
* NEWS: Mention the bug fix.
Reported in https://bugs.debian.org/1034429

2 years agomaint: post-release administrivia
Pádraig Brady [Tue, 18 Apr 2023 14:32:53 +0000 (15:32 +0100)] 
maint: post-release administrivia

* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.

2 years agoversion 9.3 v9.3
Pádraig Brady [Tue, 18 Apr 2023 14:08:11 +0000 (15:08 +0100)] 
version 9.3

* NEWS: Record release date.

2 years agotests: avoid allocation checks on ZFS
Pádraig Brady [Sat, 15 Apr 2023 21:19:06 +0000 (22:19 +0100)] 
tests: avoid allocation checks on ZFS

* tests/du/basic.sh: Allocation of files was seen to change
asynchronously on ZFS, so avoid allocation comparisons there.

2 years agotests: tty-eof: fix various issues
Pádraig Brady [Tue, 11 Apr 2023 12:02:21 +0000 (13:02 +0100)] 
tests: tty-eof: fix various issues

* tests/misc/tty-eof.pl: Ensure we don't erroneously
skip commands with parameters.
Comment as to why cut(1) is treated differently.
Adjust expect calls to not wait needlessly for cut output.

2 years agotests: avoid dependence on file layout for cp sparse check
Pádraig Brady [Wed, 12 Apr 2023 13:37:00 +0000 (14:37 +0100)] 
tests: avoid dependence on file layout for cp sparse check

* tests/cp/sparse-2.sh: Don't depend on the copy taking
<= allocation of the source.  Instead leverage --debug
to check that zero detection is being enabled.

2 years agocopy: --debug: indicate if NUL detection is used with SEEK_HOLE
Pádraig Brady [Thu, 13 Apr 2023 12:15:31 +0000 (13:15 +0100)] 
copy: --debug: indicate if NUL detection is used with SEEK_HOLE

* src/copy.c (sparse_copy): With --sparse=always we also detect
NULs in extents we're copying, so indicate this with --debug.

2 years agodoc: update re 32-bit builds
Paul Eggert [Mon, 10 Apr 2023 18:55:54 +0000 (11:55 -0700)] 
doc: update re 32-bit builds

* README-install: Mention how to build on 32-bit-only hosts.
This builds on a previous patch by Pádraig Brady.

2 years agobuild: fix _Noreturn compilation failure
Pádraig Brady [Mon, 10 Apr 2023 18:36:23 +0000 (19:36 +0100)] 
build: fix _Noreturn compilation failure

Fix a build failure seen on gcc 3.4 on Solaris 10 at least.

* src/crctab.c: Ensure we include config.h for all compilation units.
This is now required for new _Noreturn usage in gnulib for stdint.h.
* src/cksum.c: Update generation code to ensure config.h included.
* cfg.mk: Remove crctab.c exclusion from the config.h check.

2 years agotests: avoid non portable brace expansion
Pádraig Brady [Mon, 10 Apr 2023 17:51:34 +0000 (18:51 +0100)] 
tests: avoid non portable brace expansion

* tests/cp/backup-dir.sh: Avoid non portable brace expansion
which is not supported by FreeBSD or Solaris shells at least.

2 years agomaint: require support for post-2038 timestamps
Paul Eggert [Mon, 10 Apr 2023 02:18:57 +0000 (19:18 -0700)] 
maint: require support for post-2038 timestamps

* bootstrap.conf (gnulib_modules): Replace year2038 with
year2038-required.

2 years agobuild: update gnulib submodule to latest
Paul Eggert [Mon, 10 Apr 2023 01:30:00 +0000 (18:30 -0700)] 
build: update gnulib submodule to latest

2 years agowc: ensure we update file offset
Pádraig Brady [Sun, 5 Feb 2023 19:52:31 +0000 (19:52 +0000)] 
wc: ensure we update file offset

* src/wc.c (wc): Update the offset when not reading,
and do read if we can't update the offset.
* tests/misc/wc-proc.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/61300

2 years agocp,mv: issue "skipped" messages when skipping files
Pádraig Brady [Fri, 7 Apr 2023 09:25:41 +0000 (10:25 +0100)] 
cp,mv: issue "skipped" messages when skipping files

* NEWS: Mention the change in behavior to issue a "not replaced"
error diagnostic with -n, and the "skipped" message with -v.
* src/copy.c (copy_internal): Adjust to output the "skipped" messages
depending on -i, -n, -u.
* tests/cp/cp-i.sh: Adjust accordingly.
* tests/mv/mv-n.sh: Likewise.

2 years agocp,mv: add --update=none to always skip existing files
Pádraig Brady [Sat, 1 Apr 2023 15:27:52 +0000 (16:27 +0100)] 
cp,mv: add --update=none to always skip existing files

Add --update=none which is equivalent to the --no-clobber behavior
from before coreutils 9.2.  I.e. existing files are unconditionally
skipped, and them not being replaced does not affect the exit status.

* src/copy.h [enum Update_type]: A new type to support parameters
to the --update command line option.
[enum Interactive]: Add I_ALWAYS_SKIP.
* src/copy.c: Treat I_ALWAYS_SKIP like I_ALWAYS_NO (-n),
except that we don't fail when skipping.
* src/system.h (emit_update_parameters_note): A new function
to output the description of the new --update parameters.
* src/cp.c (main): Parse --update arguments, ensuring that
-n takes precedence if specified.
(usage): Describe the new option.  Also allude that
-u is related in the -n description.
* src/mv.c: Accept the new --update parameters and
update usage() accordingly.
* doc/coreutils.texi (cp invocation): Describe the new --update
parameters.  Also reference --update from the --no-clobber description.
(mv invocation): Likewise.
* tests/mv/update.sh: Test the new parameters.
* NEWS: Mention the new feature.
Addresses https://bugs.gnu.org/62572

2 years agocp: fix --backup with subdirectories
Pádraig Brady [Mon, 3 Apr 2023 17:12:33 +0000 (18:12 +0100)] 
cp: fix --backup with subdirectories

* gnulib: Reference the latest gnulib including the
fix to the backupfile module in commit 94496522.
* tests/cp/backup-dir.sh: Add a test to ensure
we rename appropriately when backing up through subdirs.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/62607

2 years agotests: tee: avoid false failure due to fifo usage
Pádraig Brady [Thu, 30 Mar 2023 14:49:26 +0000 (15:49 +0100)] 
tests: tee: avoid false failure due to fifo usage

* tests/misc/tee.sh: Call cleanup_ in all cases to ensure
there are no overlapping interactions on the fifo that
might impact later parts of the test.  This was seen to
cause issue with dash on musl libc.
Addresses https://bugs.gnu.org/62542

2 years agotests: adjust csplit VM limit
Pádraig Brady [Thu, 30 Mar 2023 13:28:03 +0000 (14:28 +0100)] 
tests: adjust csplit VM limit

* tests/misc/csplit-heap.sh: More memory is required to avoid
a false failure on some systems.  Noticed with musl libc
with bash as the shell.  This is confirmed to still easily
trigger with the original memory leak being tested.
Addresses https://bugs.gnu.org/62542

2 years agowc: diagnose overflow of total counts
Pádraig Brady [Wed, 29 Mar 2023 14:29:52 +0000 (15:29 +0100)] 
wc: diagnose overflow of total counts

* src/wc.c (wc): Use INT_ADD_WRAPV() to detect overflow.
(main): Upon overflow, saturate the total, print a diagnostic,
and set exit status.
* tests/misc/wc-total.sh: Add a test case, which operates
on BTRFS and 64 bit systems at least.
Reported at https://bugs.debian.org/1027100

2 years agodircolors: diagnose read errors
Pádraig Brady [Tue, 28 Mar 2023 13:24:29 +0000 (14:24 +0100)] 
dircolors: diagnose read errors

* NEWS: Mention the fix.
* src/dircolors.c: Fail upon read error from getline().
* tests/misc/dircolors.sh: Add a new test.
* tests/local.mk: Reference the new test.

2 years agotests: add a test case for the previous date fix
Pádraig Brady [Tue, 28 Mar 2023 12:38:52 +0000 (13:38 +0100)] 
tests: add a test case for the previous date fix

* NEWS: Also mention this bug fix.
* tests/misc/date-f.sh: Add a new test.
* tests/local.mk: Reference the new test.

2 years agodate: diagnose -f read errors
Paul Eggert [Tue, 28 Mar 2023 08:52:43 +0000 (01:52 -0700)] 
date: diagnose -f read errors

* src/date.c (batch_convert): Diagnose read errors, fixing Bug#62497.

2 years agocp: clarify commentary
Paul Eggert [Sat, 25 Mar 2023 16:30:22 +0000 (09:30 -0700)] 
cp: clarify commentary

* src/copy.c: Make comments a bit clearer.

2 years agocopy: fix --reflink=auto to fallback in more cases
Pádraig Brady [Thu, 23 Mar 2023 13:19:04 +0000 (13:19 +0000)] 
copy: fix --reflink=auto to fallback in more cases

On restricted systems like android or some containers,
FICLONE could return EPERM, EACCES, or ENOTTY,
which would have induced the command to fail to copy
rather than falling back to a more standard copy.

* src/copy.c (is_terminal_failure): A new function refactored
from handle_clone_fail().
(is_CLONENOTSUP): Merge in the handling of EACCES, ENOTTY, EPERM
as they also pertain to determination of whether cloning is supported
if we ever use this function in that context.
(handle_clone_fail): Use is_terminal_failure() in all cases,
so that we assume a terminal failure in less errno cases.
* NEWS: Mention the bug fix.
Addresses https://bugs.gnu.org/62404

2 years agodoc: add a NEWS entry for the previous fix
Pádraig Brady [Fri, 24 Mar 2023 12:44:50 +0000 (12:44 +0000)] 
doc: add a NEWS entry for the previous fix

* NEWS: Mention the previous cksum --check fix.

2 years agocksum: fix reporting of failed checks
Pádraig Brady [Thu, 23 Mar 2023 12:31:24 +0000 (12:31 +0000)] 
cksum: fix reporting of failed checks

This applies to all checksumming utilities,
where we incorrectly report all subsequent files as checking 'OK'
once any file has passed a digest check.
The exit status was not impacted, only the printed status.

* src/digest.c (digest_check): Use the correct state variable
to determine if the _current_ file has passed or not.
* tests/misc/md5sum.pl: Add a test case.
Fixes https://bugs.gnu.org/62403

2 years agotests: skip some parts of tests/misc/tee.sh if run as root
Nick Alcock [Tue, 21 Mar 2023 12:54:02 +0000 (12:54 +0000)] 
tests: skip some parts of tests/misc/tee.sh if run as root

Similarly to the fix to tests/rmdir/ignore.sh in c0e5f8c59,
tee should not be expected to fail when run with read-only outputs
when run as root.

* tests/misc/tee.sh: Add uid_is_privileged_ guard around test for
read-only outputs.

2 years agomaint: post-release administrivia
Pádraig Brady [Mon, 20 Mar 2023 14:08:46 +0000 (14:08 +0000)] 
maint: post-release administrivia

* NEWS: Add header line for next release.
* .prev-version: Record previous version.
* cfg.mk (old_NEWS_hash): Auto-update.

2 years agoversion 9.2 v9.2
Pádraig Brady [Mon, 20 Mar 2023 13:44:36 +0000 (13:44 +0000)] 
version 9.2

* NEWS: Record release date.

2 years agotests: avoid false failure on systems without statx
Pádraig Brady [Sun, 19 Mar 2023 23:35:12 +0000 (23:35 +0000)] 
tests: avoid false failure on systems without statx

* tests/ls/stat-free-symlinks.sh: Filter out syscalls that
return ENOSYS, as that was seen with statx() on Debian 10.13
on mips64, and resulted in overcounting of stat calls.

2 years agostty: ensure arbitrary data is not displayed
Pádraig Brady [Sun, 19 Mar 2023 22:22:18 +0000 (22:22 +0000)] 
stty: ensure arbitrary data is not displayed

* src/stty.c (main): Use static structures to ensure
they're initialized (to zero), so that random data is
not displayed, or compared resulting in a inaccurate
failure reported to users.  This was seen on musl libc
where some parts of the termios c_cc array were
not initialized by tcgetattr().
Reported by Bruno Haible.

2 years agotests: df: avoid false failure on systems with multiple roots
Pádraig Brady [Sat, 18 Mar 2023 21:44:56 +0000 (21:44 +0000)] 
tests: df: avoid false failure on systems with multiple roots

* tests/df/df-symlink.sh: Skip the test on systems
with multiple file system roots.

2 years agodoc: uniq: revert previous man page adjustment
Pádraig Brady [Sat, 18 Mar 2023 14:58:20 +0000 (14:58 +0000)] 
doc: uniq: revert previous man page adjustment

* src/uniq.c (usage): -D doesn't take an argument
so should be separated in the description.

2 years agodoc: uniq: use more standard description of -D long option
Pádraig Brady [Sat, 18 Mar 2023 11:35:46 +0000 (11:35 +0000)] 
doc: uniq: use more standard description of -D long option

* src/uniq.c: Describe -D and --all-repeated together,
as with all other long options.
Fixes https://bugs.gnu.org/62249

2 years agotests: aix: avoid unsupported functionality
Pádraig Brady [Fri, 17 Mar 2023 19:57:18 +0000 (19:57 +0000)] 
tests: aix: avoid unsupported functionality

* tests/misc/tee.sh: AIX doesn't support detecting
closed outputs either with poll() or select() so avoid
testing that functionality.
* tests/tail-2/pipe-f.sh: Likewise.

2 years agomaint: avoid discarded-qualifiers warnings with SELinux 3.5
Bernhard Voelker [Thu, 16 Mar 2023 22:11:47 +0000 (23:11 +0100)] 
maint: avoid discarded-qualifiers warnings with SELinux 3.5

Since SELinux version 3.5, the return value of context_str(3) is
declared as const; see:
  https://github.com/SELinuxProject/selinux/commit/dd98fa322766
Therefore, GCC complains (here with -Werror):
  src/selinux.c: In function 'defaultcon':
  src/selinux.c:152:16: error: assignment discards 'const' qualifier \
    from pointer target type [-Werror=discarded-qualifiers]
    152 |   if (!(constr = context_str (tcontext)))
        |                ^
  src/selinux.c: In function 'restorecon_private':
  src/selinux.c:252:16: error: assignment discards 'const' qualifier \
    from pointer target type [-Werror=discarded-qualifiers]
    252 |   if (!(constr = context_str (tcontext)))
      |                ^

* src/selinux.c (defaultcon): Define CONSTR as const.
(restorecon_private): Likewise.

2 years agotest: avoid a test hang on Hurd
Pádraig Brady [Wed, 15 Mar 2023 18:11:00 +0000 (18:11 +0000)] 
test: avoid a test hang on Hurd

* tests/cp/sparse-to-pipe.sh: Protect the cp call seen to
hang on Hurd/i686 with a timeout.
Reported By: Bruno Haible

2 years agocksum: fix --raw on 64 bit big endian systems
Pádraig Brady [Wed, 15 Mar 2023 13:57:37 +0000 (13:57 +0000)] 
cksum: fix --raw on 64 bit big endian systems

* src/sum.c (output_bsd): On sparc64 for example,
a crc of 0 was output due to casting an int variable
to uint16_t and thus operating on the wrong end of the variable.
Instead use explicit assignment to the narrower type
to ensure we get the appropriate data.
(output_sysv): Likewise.
Reported by Bruno Haible.

2 years agobuild: avoid -Wsometimes-uninitialized on macOS 12
Bruno Haible [Wed, 15 Mar 2023 13:15:11 +0000 (13:15 +0000)] 
build: avoid -Wsometimes-uninitialized on macOS 12

* src/pr.c (integer_overflow): Mark as _Noreturn.
* src/test.c (beyond): Likewise.

2 years agodoc: remove older ChangeLog items
Pádraig Brady [Mon, 13 Mar 2023 21:45:50 +0000 (21:45 +0000)] 
doc: remove older ChangeLog items

* Makefile.am: Update the oldest documented version
to 8.29 which is now about 5 years old.

2 years agotee: avoid undefined behavior after fclose()
Pádraig Brady [Mon, 13 Mar 2023 21:26:21 +0000 (21:26 +0000)] 
tee: avoid undefined behavior after fclose()

* iopoll.c (fclose_wait): Rename from confusing fclose_nonblock name.
Also adjust to do no operations on the stream after fclose()
as this is undefined. Instead use fflush() to determine EAGAIN status.
(fwrite_wait): Renamed from confusing fwrite_nonblock name.

2 years agodoc: dircolors: better separate the sections in the database
Pádraig Brady [Mon, 5 Sep 2022 13:31:04 +0000 (14:31 +0100)] 
doc: dircolors: better separate the sections in the database

* src/dircolors.hin: Make the separate sections of the self
documenting dircolors database more apparent,
by adding heading comments, and appropriate separation.

2 years agols: --color: honor separate sequences for extension cases
Pádraig Brady [Sun, 4 Sep 2022 18:59:25 +0000 (19:59 +0100)] 
ls: --color: honor separate sequences for extension cases

Following on from commit v8.29-45-g24053fbd8 which unconditionally
used case insensitive extension matching, support selective
case sensitive matching when there are separate extension cases
defined with different display sequences.

* src/dircolors.hin: Document how file name suffixes are matched.
Note this is displayed with `dircolors --print-database` which
the texi info recommends to use for details.
* src/ls.c (parse_ls_color): Postprocess the list to
mark entries for case sensitive matching,
and also adjust so that unmatchable entries are more quickly ignored.
(get_color_indicator): Use exact matching rather than
case insensitive matching if so marked.
* tests/ls/color-ext.sh: Add test cases.
* NEWS: Mention the change in behavior.
Addresses https://bugs.gnu.org/33123

2 years agotests: adjust du test for recent apparent size change
Pádraig Brady [Mon, 13 Mar 2023 15:11:27 +0000 (15:11 +0000)] 
tests: adjust du test for recent apparent size change

* tests/du/threshold.sh: Directories are assumed to be
of size 0 with --apparent since commit v9.1-187-g110bcd283
so remove --apparent cases from this test.

2 years agotee: support non blocking outputs
Pádraig Brady [Sun, 5 Mar 2023 15:51:32 +0000 (15:51 +0000)] 
tee: support non blocking outputs

Non blocking outputs can be seen for example
when piping telnet through tee to a terminal.
In that case telnet sets its input to nonblocking mode,
which results in tee's output being nonblocking,
in which case in may receive an EAGAIN error upon write().
The same issue was seen with mpirun.
The following can be used to reproduce this
locally at a terminal (in most invocations):

  $ { dd iflag=nonblock count=0 status=none;
      dd bs=10K count=10 if=/dev/zero status=none; } |
    tee || echo fail >/dev/tty

* src/iopoll.c (iopoll_internal): A new function refactored from
iopoll(), to also support a mode where we check the output
descriptor is writeable.
(iopoll): Now refactored to just call iopoll_internal().
(fwait_for_nonblocking_write): A new internal function which
uses iopoll_internal() to wait for writeable output
if an EAGAIN or EWOULDBLOCK was received.
(fwrite_nonblock): An fwrite() wrapper which uses
fwait_for_nonblocking_write() to handle EAGAIN.
(fclose_nonblock): Likewise.
src/iopoll.h: Add fclose_nonblock, fwrite_nonblock.
src/tee.c: Call fclose_nonblock() and fwrite_nonblock wrappers,
instead of the standard functions.
* tests/misc/tee.sh: Add a test case.
* NEWS: Mention the improvement.

The idea was suggested by Kamil Dudka in
https://bugzilla.redhat.com/1615467

2 years agodoc: remove redundant NEWS entry
Pádraig Brady [Wed, 8 Mar 2023 18:28:06 +0000 (18:28 +0000)] 
doc: remove redundant NEWS entry

* NEWS: There was no bug on macOS,
and the split -n new feature is documented already.

2 years agosplit: support split -n on larger pipe input
Paul Eggert [Tue, 7 Mar 2023 20:58:12 +0000 (12:58 -0800)] 
split: support split -n on larger pipe input

* bootstrap.conf (gnulib_modules): Add free-posix, tmpfile.
* src/split.c (copy_to_tmpfile): New function.
(input_file_size): Use it to split larger files when sizes cannot
easily be determined via fstat or lseek.  See Bug#61386#235.
* tests/split/l-chunk.sh: Mark tests of /dev/zero as
very expensive since they exhaust /tmp.

2 years agotee: fix a crash with unwriteable files
Pádraig Brady [Tue, 7 Mar 2023 01:12:12 +0000 (01:12 +0000)] 
tee: fix a crash with unwriteable files

This was introduced recently with commit v9.1-166-g6b12e62d9

* src/tee.c (tee_files): Check the return from fopen()
before passing to fileno() etc.
* tests/misc/tee.sh: Add a test case.

2 years agomaint: pacify ‘make syntax-check’
Paul Eggert [Mon, 6 Mar 2023 23:37:45 +0000 (15:37 -0800)] 
maint: pacify ‘make syntax-check’

Problem reported by Pádraig Brady (Bug#61386#226).
* src/split.c (parse_chunk): Use die instead of error.
(main): Quote a string.
* tests/local.mk (all_root_tests): Move du/apparent.sh from here ...
(all_tests): ... to here.

2 years agotail,tee: avoid issues with many files on systems without poll
Pádraig Brady [Mon, 6 Mar 2023 13:01:26 +0000 (13:01 +0000)] 
tail,tee: avoid issues with many files on systems without poll

* src/iopoll.c (iopoll): Protect the call to select against
passing in a descriptor larger than FD_SETSIZE.

2 years agodu: --apparent counts only symlinks and regular
Paul Eggert [Sun, 5 Mar 2023 00:51:11 +0000 (16:51 -0800)] 
du: --apparent counts only symlinks and regular

Problem reported by Christoph Anton Mitterer (Bug#61884).
* src/du.c (process_file): When counting apparent sizes, count
only usable st_size members.
* tests/du/apparent.sh: New file.
* tests/local.mk (all_root_tests): Add it.

2 years agosplit: tune for when creating output files
Paul Eggert [Sat, 4 Mar 2023 21:40:28 +0000 (13:40 -0800)] 
split: tune for when creating output files

* src/split.c (create): Avoid fstat + ftruncate in the usual case
where the output file does not already exist, by trying
to create it with O_EXCL first.  This costs a failed open
in the unusual case where the output file already exists,
but that’s OK.

2 years agosplit: style fix
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: style fix

* src/split.c (ofile_open): Avoid ‘if (! (a = b))’ style.

2 years agosplit: prefer signed integers to size_t
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: prefer signed integers to size_t

This allows for better runtime checking with gcc
-fsanitize=undefined.
* src/split.c: Include idx.h.
(open_pipes_alloc, n_open_pipes, suffix_length)
(set_suffix_length, input_file_size, sufindex, outbase_length)
(outfile_length, addsuf_length, create, cwrite, bytes_split)
(lines_split, line_bytes_split, lines_chunk_split)
(bytes_chunk_extract, ofile_open, lines_rr, main):
Prefer signed integers (typically idx_t) to size_t.

2 years agosplit: handle large numbers better
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: handle large numbers better

Prefer signed types to uintmax_t, as this allows for better
runtime checking with gcc -fsanitize=undefined.
Also, when an integer overflows just use the maximal value
when the code will do the right thing anyway.
* src/split.c (set_suffix_length, bytes_split, lines_split)
(line_bytes_split, lines_chunk_split, bytes_chunk_extract)
(lines_rr, parse_chunk, main):
Prefer a signed type (typically intmax_t) to uintmax_t.
(strtoint_die): New function.
(OVERFLOW_OK): New macro.  Use it elsewhere, where we now allow
LONGINT_OVERFLOW because the code then does the right thing on all
practical platforms (they have int wide enough so that it cannot
be practically exhausted).  We can do this now that we can safely
assume intmax_t has at least 64 bits.
(parse_n_units): New function.
(parse_chunk, main): Use it.
(main): Do not worry about integer overflow when the code
will do the right thing anyway with the extreme value.
Just use the extreme value.
* tests/split/fail.sh: Adjust to match new behavior.

2 years agosplit: prefer ssize_t for read result
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: prefer ssize_t for read result

* src/split.c (bytes_split, lines_chunk_split)
(bytes_chunk_extract, main): Prefer ssize_t to size_t when
representing the return value of ‘read’.  Use a negative value
instead of SIZE_MAX to indicate a missing value.

2 years agosplit: be more careful about buffer sizes
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: be more careful about buffer sizes

* src/split.c: Include sys-limits.h, not safe-read.h.
(input_file_size, bytes_split, lines_split, line_bytes_split)
(lines_chunk_split, bytes_chunk_extract, lines_rr): Call read, not
safe_read, since safe_read no longer buys us anything.
(main): Reject outlandish buffer sizes right away,
rather than allocating huge buffers and never using them.

2 years agosplit: minor -1 / 0 refactor
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: minor -1 / 0 refactor

* src/split.c (create, bytes_split, ofile_open):
Prefer comparing to 0 to comparing to -1.

2 years agosplit: don’t worry about ECHILD
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: don’t worry about ECHILD

* src/split.c (closeout): There should be no need for a special
case for ECHILD, since we never wait for the same child twice.
Simplify with this in mind.

2 years agosplit: don’t assume pid_t fits in int
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: don’t assume pid_t fits in int

* src/split.c (filter_pid): Now pid_t, not int.
(of_t): opid member is now pid_t, not int.

2 years agosplit: simplify SIGPIPE handling
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: simplify SIGPIPE handling

Ignore and default SIGPIPE, rather than blocking and unblocking it.
* src/split.c (default_SIGPIPE):
New static var, replacing oldblocked and newblocked.
(create): Use it.
(main): Set it.

2 years agosplit: port ‘split -n N /dev/null’ better to macOS
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
split: port ‘split -n N /dev/null’ better to macOS

* src/split.c (input_file_size): Do not bother with lseek if the
initial read probe reaches EOF, since the file size is known then.
This works better on macOS, which doesn’t allow lseek on /dev/null.
Do not special-case size-zero files, as the issue can occur
with any size file (though /proc files are the most common).
If the current position is past end of file, treat this as
size zero regardless of whether the file has a usable st_size.
Pass through lseek -1 return values rather than using ‘return -1’;
this makes the code a bit easier to analyze (and a bit faster).
Avoid undefined behavior if the size calculation overflows.
(lines_chunk_split): Do not bother with lseek if it would have
no effect if successful.  This works better on macOS, which
doesn’t allow lseek on /dev/null.
* tests/split/l-chunk.sh: Adjust to match fixed behavior.

2 years agomaint: fix NEWS quoting
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)] 
maint: fix NEWS quoting

* NEWS: Use English-language quoting in recent news.

2 years agosplit: split more evenly with -n
Paul Eggert [Sat, 4 Mar 2023 19:41:03 +0000 (11:41 -0800)] 
split: split more evenly with -n

* src/split.c (bytes_split): New arg REM_BYTES.
Use this to split more evenly.  All callers changed.
(lines_chunk_split, bytes_chunk_extract):
Be consistent with new byte_split.
* tests/split/b-chunk.sh, tests/split/l-chunk.sh: Test new behavior.

2 years agosplit: small -n lines simplification
Paul Eggert [Sat, 4 Mar 2023 19:41:02 +0000 (11:41 -0800)] 
split: small -n lines simplification

* src/split.c (lines_chunk_split):
Rewrite while as if-while for clarity.

2 years agosplit: refactor lines_chunk_split
Paul Eggert [Sat, 4 Mar 2023 19:41:02 +0000 (11:41 -0800)] 
split: refactor lines_chunk_split

* src/split.c (lines_chunk_split): Simplify by having chunk_end
point to the first byte after the chunk, rather than to the last
byte of the chunk.  This will reduce confusion once we allow
chunks to be empty.

2 years agotee: tune when later -p overrides earlier
Paul Eggert [Sat, 4 Mar 2023 20:18:18 +0000 (12:18 -0800)] 
tee: tune when later -p overrides earlier

* src/tee.c (pipe_check): Make this a local var instead
of a static var.  This suppresses a -Wmaybe-uninitialized
diagnostic with gcc 12.2.1 20221121 (Red Hat 12.2.1-4).
(main): Don’t set pipe_check unnecessarily if a later
-p option overrides an earlier one that wants pipe_check.
Problem discovered when I investigated the GCC warning.

2 years agomaint: refactor tail.c to use iopoll
Pádraig Brady [Tue, 28 Feb 2023 18:36:02 +0000 (18:36 +0000)] 
maint: refactor tail.c to use iopoll

* src/tail.c (check_output_alive): Reuse iopoll()
rather than directly calling poll() or select().
* src/iopoll.c (iopoll): Refactor to support non blocking operation,
or ignoring descriptors by passing a negative value.
* src/iopoll.h (iopoll): Adjust to support a BLOCK parameter.
* src/tee.c (tee_files): Adjust iopoll() call to explicitly block.
* src/local.mk: Have tail depend on iopoll.c.

2 years agotail: avoid undefined behavior when polling outputs
Pádraig Brady [Tue, 28 Feb 2023 13:34:31 +0000 (13:34 +0000)] 
tail: avoid undefined behavior when polling outputs

* src/tail.c (check_output_alive): Only check the returned
events from poll() when it indicates there are events to check.

2 years agodoc: tee -p: clarify operation
Pádraig Brady [Mon, 27 Feb 2023 18:07:06 +0000 (18:07 +0000)] 
doc: tee -p: clarify operation

* src/tee.c (usage): Change from describing one (non pipe) aspect
to the more general point of being the option to use if working with
pipes, and referencing the more detailed info below.
* doc/coreutils.texi (tee invocation): s/standard/appropriate/ since
the standard operation with pipes is to exit immediately upon write
error.  s/early/immediately/ as it's ambiguous as to what "early"
is in relation to.