Pádraig Brady [Wed, 31 May 2023 16:21:08 +0000 (17:21 +0100)]
maint: adjust code to handle "error" syntax-check changes
* src/dd.c: Don't include no longer used error.h.
Use quoteaf() rather than quote() to quote appropriate for the shell
and to avoid the syntax-check failure,
* src/stty.c: Use quoteaf() rather than quotef()
to have more consistent quoting of the invalid arg.
Pádraig Brady [Wed, 31 May 2023 16:12:52 +0000 (17:12 +0100)]
maint: augment syntax checks to cater for more "error" functions
src/cfg.mk (sc_error_quotes, sc_error_shell_quotes,
sc_error_shell_always_quotes): Include "die" and "diagnose"
in the class of error functions to check arguments for.
Paul Eggert [Tue, 30 May 2023 21:24:46 +0000 (14:24 -0700)]
dd: fix ‘error’ name issue without macros
* src/dd.c (_GL_NO_INLINE_ERROR): Remove; no longer needed.
(diagnose): Rename from nl_error and omit first arg since it is
always zero. All uses changed.
(error): Remove macro.
Pádraig Brady [Fri, 19 May 2023 21:16:22 +0000 (22:16 +0100)]
build: modernize bootstrap prerequsite tools
Following on from commit v9.0-15-gaa31b919c
which updated README-prereq...
* bootstrap.conf: Add an explicit requirement on m4.
Add an explicit requirement on texi2pdf which is often
packaged separately to makeinfo and induces a failure
far down the distribution phase if not present.
Replace the rsync dependency with wget,
which gnulib changed to in 2018.
This reverts commit 800c86d5, as that was deemed too invasive.
We do keep the change to tee.c to allow using -O3 without warnings.
For other optimization options like -O0, -Og, -O1, -Os,
one can use WERROR_CFLAGS= to stop warnings inducing a build failure.
Pádraig Brady [Thu, 18 May 2023 09:38:11 +0000 (10:38 +0100)]
build: avoid false -Wmaybe-uninitialized warnings
Allow easily building a debug build for example with:
make CFLAGS='-O0 -ggdb'
False -Wmaybe-uninitialized warnings hit in different
places depending on the compiler passes used.
These changes were tested with gcc 10.2.1, 12.2.1, and 13.1.1 like:
for o in g s z fast 0 1 2 3; do
make clean && make -j$(nproc) CFLAGS="-O$o" || break
done
* src/digest.c: Disable -Wmaybe-uninitialized that gives
false positive here at -O0.
* src/ln.c: Avoid -Wmaybe-uninitialized that gives
false positive here at -O1.
* src/pr.c: Likewise.
* src/sort.c: Likewise.
* src/tee.c: Avoid -Wmaybe-uninitialized that gives
false positive here at -O3 on gcc 13.1.1 at least.
* src/cp.c: Avoid -Wmaybe-uninitialized that gives
false positive here at -Os on gcc 13.1.1 at least.
* src/copy.c: Avoid -Wmaybe-uninitialized that gives
false positive here at -Og on gcc 13.1.1 at least.
* src/head.c: Likewise.
* src/paste.c: Likewise.
Tested on gcc 13.1.1 with: make CFLAGS='-O0 -ggdb'
* configure.ac: Disable -Wstringop-overflow for gnulib.
This warning is far too problematic in my experience:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88443
and triggers with gcc -O0 with versions 12,13 at least.
Pádraig Brady [Mon, 8 May 2023 20:22:48 +0000 (21:22 +0100)]
split: advise the kernel of sequential access pattern
As split is often dealing with large files,
ensure we indicate to the kernel our sequential access pattern.
This was seen to operate 5% faster when reading from SSD,
as tested with:
dd bs=1M count=2K if=/dev/urandom of=big.in
for split in split.orig split; do
# Ensure big file is not cached
dd of=big.in oflag=nocache conv=notrunc,fdatasync count=0 status=none
# Test read efficiency
CWD=$PWD; (cd /dev/shm && time $CWD/src/$split -n2 $CWD/big.in)
done
real 0m9.039s
user 0m0.055s
sys 0m3.510s
real 0m8.568s
user 0m0.056s
sys 0m3.752s
* src/split.c (main): Use fdadvise to help the kernel
choose a more appropriate readahead buffer.
* NEWS: Mention the improvement.
This fixes failures in "very-expensive" tests on FTS with many
directory entries:
FAIL: tests/rm/ext3-perf
FAIL: tests/rm/many-dir-entries-vs-OOM
The following shows the problem in the former of the above tests:
$ mkdir d && seq 400000 | env -C d xargs touch )
$ rm -rf d
rm: traversal failed: d: Operation not supported
Gnulib commit 3f0950f65abb (2023-04-26) introduced this regression
which was fixed again with gnulib commit d4d8abb39eb0.
See discussion in
<https://lists.gnu.org/r/bug-gnulib/2023-05/msg00040.html>
* bootstrap.conf (gnulib_modules): Change "year2038-required" to
"year2038-recommended"; the module has been replaced.
* gnulib: Update to latest.
* tests/init.sh: Likewise.
Paul Eggert [Fri, 5 May 2023 18:03:25 +0000 (11:03 -0700)]
cp: -p --parents: minor cleanup of previous patch
This doesn’t change behavior; it just clarifies the code a bit.
* src/cp.c (re_protect): New arg DST_SRC_NAME, for clarity, and so
that we need to skip '/'s only once. Caller changed.
Rename a couple of local variables to try to make things clearer.
Pádraig Brady [Wed, 3 May 2023 16:01:37 +0000 (17:01 +0100)]
cp: -p --parents: fix failure to preserve permissions for absolute paths
* src/cp.c (re_protect): Ensure copy_acl() is passed an absolute path.
* tests/cp/cp-parents.sh: Add a test case.
* NEWS: Mention the bug.
Fixes https://bugs.gnu.org/63245
Pádraig Brady [Tue, 2 May 2023 21:39:03 +0000 (22:39 +0100)]
tests: provide more info on DEBUG=yes
* README: State that DEBUG=yes is particularly useful with perl tests.
* tests/split/l-chunk.sh: Use the more standard $DEBUG variable
rather than an internal $DEBUGGING variable.
Paul Eggert [Thu, 27 Apr 2023 00:14:54 +0000 (17:14 -0700)]
maint: simplify --enable-gcc-warnings='expensive'
* configure.ac (WERROR_CFLAGS): Omit mention of
-Wno-analyzer-double-free, -Wno-analyzer-null-dereference, and
-Wno-analyzer-use-after-free as manywarnings no longer uses them.
Paul Eggert [Thu, 27 Apr 2023 00:14:54 +0000 (17:14 -0700)]
maint: suppress GCC 13 false alarms
* src/csplit.c, src/fmt.c, src/make-prime-list.c, src/nohup.c:
Add pragmas to pacify GCC 13 when coreutils is configured
with --enable-gcc-warnings='expensive'.
Paul Eggert [Thu, 27 Apr 2023 00:14:54 +0000 (17:14 -0700)]
chmod: pacify GCC 13
* src/chmod.c (main): Use xpalloc instead of X2REALLOC,
and make the corresponding variables signed instead of unsigned.
When reallocating the buffer, this grows it by a factor of 1.5, not 2.
This also pacifies gcc -Wanalyzer-null-dereference.
Paul Eggert [Thu, 27 Apr 2023 00:14:54 +0000 (17:14 -0700)]
csplit: pacify GCC 13
* src/csplit.c (load_buffer): Refactor for clarity.
This also xpacifies gcc -Wanalyzer-use-of-uninitialized-value.
When reallocating the buffer, grow it by a factor of 1.5, not 2.
* tests/misc/read-errors.sh: Exercise more modes of
various utilities for better read error coverage.
* tests/split/fail.sh: Remove part refactored into the above test.
* src/pr.c (init_parameters): Ensure we avoid a 0 lines_per_body
which was possible when adjusting for double spacing.
That caused print_page() to always return true,
causing an infinite loop.
* tests/pr/pr-tests.pl: Add a test case.
* NEWS: Mention the fix.
Fixes https://bugs.debian.org/1034808
copy: reduce verbosity of -i and -u with --verbose
Since skipping of files is central to the operation of -i and -u,
and with -u one may be updating few files out of many,
reinstate the verbosity of this functionality as it was before 9.3.
* src/copy.c (copy_internal): Only output "skipped" message
with --debug. Also adjust so message never changes with --debug.
* tests/cp/cp-i.sh: Adjust accordingly.
* tests/mv/mv-n.sh: Likewise.
* tests/cp/debug.sh: Add explicit test case for message.
* NEWS: Mention the change in behavior.
install: support stripping files with a leading hyphen
* src/install.c (strip): Prepend "./" to file names with a leading "-".
* tests/install/strip-program.sh: Add a test case.
* NEWS: Mention the bug fix.
Reported in https://bugs.debian.org/1034429
* tests/misc/tty-eof.pl: Ensure we don't erroneously
skip commands with parameters.
Comment as to why cut(1) is treated differently.
Adjust expect calls to not wait needlessly for cut output.
tests: avoid dependence on file layout for cp sparse check
* tests/cp/sparse-2.sh: Don't depend on the copy taking
<= allocation of the source. Instead leverage --debug
to check that zero detection is being enabled.
Fix a build failure seen on gcc 3.4 on Solaris 10 at least.
* src/crctab.c: Ensure we include config.h for all compilation units.
This is now required for new _Noreturn usage in gnulib for stdint.h.
* src/cksum.c: Update generation code to ensure config.h included.
* cfg.mk: Remove crctab.c exclusion from the config.h check.
Pádraig Brady [Sun, 5 Feb 2023 19:52:31 +0000 (19:52 +0000)]
wc: ensure we update file offset
* src/wc.c (wc): Update the offset when not reading,
and do read if we can't update the offset.
* tests/misc/wc-proc.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/61300
cp,mv: issue "skipped" messages when skipping files
* NEWS: Mention the change in behavior to issue a "not replaced"
error diagnostic with -n, and the "skipped" message with -v.
* src/copy.c (copy_internal): Adjust to output the "skipped" messages
depending on -i, -n, -u.
* tests/cp/cp-i.sh: Adjust accordingly.
* tests/mv/mv-n.sh: Likewise.
cp,mv: add --update=none to always skip existing files
Add --update=none which is equivalent to the --no-clobber behavior
from before coreutils 9.2. I.e. existing files are unconditionally
skipped, and them not being replaced does not affect the exit status.
* src/copy.h [enum Update_type]: A new type to support parameters
to the --update command line option.
[enum Interactive]: Add I_ALWAYS_SKIP.
* src/copy.c: Treat I_ALWAYS_SKIP like I_ALWAYS_NO (-n),
except that we don't fail when skipping.
* src/system.h (emit_update_parameters_note): A new function
to output the description of the new --update parameters.
* src/cp.c (main): Parse --update arguments, ensuring that
-n takes precedence if specified.
(usage): Describe the new option. Also allude that
-u is related in the -n description.
* src/mv.c: Accept the new --update parameters and
update usage() accordingly.
* doc/coreutils.texi (cp invocation): Describe the new --update
parameters. Also reference --update from the --no-clobber description.
(mv invocation): Likewise.
* tests/mv/update.sh: Test the new parameters.
* NEWS: Mention the new feature.
Addresses https://bugs.gnu.org/62572
* gnulib: Reference the latest gnulib including the
fix to the backupfile module in commit 94496522.
* tests/cp/backup-dir.sh: Add a test to ensure
we rename appropriately when backing up through subdirs.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/62607
Pádraig Brady [Thu, 30 Mar 2023 14:49:26 +0000 (15:49 +0100)]
tests: tee: avoid false failure due to fifo usage
* tests/misc/tee.sh: Call cleanup_ in all cases to ensure
there are no overlapping interactions on the fifo that
might impact later parts of the test. This was seen to
cause issue with dash on musl libc.
Addresses https://bugs.gnu.org/62542
Pádraig Brady [Thu, 30 Mar 2023 13:28:03 +0000 (14:28 +0100)]
tests: adjust csplit VM limit
* tests/misc/csplit-heap.sh: More memory is required to avoid
a false failure on some systems. Noticed with musl libc
with bash as the shell. This is confirmed to still easily
trigger with the original memory leak being tested.
Addresses https://bugs.gnu.org/62542
Pádraig Brady [Wed, 29 Mar 2023 14:29:52 +0000 (15:29 +0100)]
wc: diagnose overflow of total counts
* src/wc.c (wc): Use INT_ADD_WRAPV() to detect overflow.
(main): Upon overflow, saturate the total, print a diagnostic,
and set exit status.
* tests/misc/wc-total.sh: Add a test case, which operates
on BTRFS and 64 bit systems at least.
Reported at https://bugs.debian.org/1027100
Pádraig Brady [Tue, 28 Mar 2023 13:24:29 +0000 (14:24 +0100)]
dircolors: diagnose read errors
* NEWS: Mention the fix.
* src/dircolors.c: Fail upon read error from getline().
* tests/misc/dircolors.sh: Add a new test.
* tests/local.mk: Reference the new test.
Pádraig Brady [Thu, 23 Mar 2023 13:19:04 +0000 (13:19 +0000)]
copy: fix --reflink=auto to fallback in more cases
On restricted systems like android or some containers,
FICLONE could return EPERM, EACCES, or ENOTTY,
which would have induced the command to fail to copy
rather than falling back to a more standard copy.
* src/copy.c (is_terminal_failure): A new function refactored
from handle_clone_fail().
(is_CLONENOTSUP): Merge in the handling of EACCES, ENOTTY, EPERM
as they also pertain to determination of whether cloning is supported
if we ever use this function in that context.
(handle_clone_fail): Use is_terminal_failure() in all cases,
so that we assume a terminal failure in less errno cases.
* NEWS: Mention the bug fix.
Addresses https://bugs.gnu.org/62404
Pádraig Brady [Thu, 23 Mar 2023 12:31:24 +0000 (12:31 +0000)]
cksum: fix reporting of failed checks
This applies to all checksumming utilities,
where we incorrectly report all subsequent files as checking 'OK'
once any file has passed a digest check.
The exit status was not impacted, only the printed status.
* src/digest.c (digest_check): Use the correct state variable
to determine if the _current_ file has passed or not.
* tests/misc/md5sum.pl: Add a test case.
Fixes https://bugs.gnu.org/62403
Pádraig Brady [Sun, 19 Mar 2023 23:35:12 +0000 (23:35 +0000)]
tests: avoid false failure on systems without statx
* tests/ls/stat-free-symlinks.sh: Filter out syscalls that
return ENOSYS, as that was seen with statx() on Debian 10.13
on mips64, and resulted in overcounting of stat calls.
Pádraig Brady [Sun, 19 Mar 2023 22:22:18 +0000 (22:22 +0000)]
stty: ensure arbitrary data is not displayed
* src/stty.c (main): Use static structures to ensure
they're initialized (to zero), so that random data is
not displayed, or compared resulting in a inaccurate
failure reported to users. This was seen on musl libc
where some parts of the termios c_cc array were
not initialized by tcgetattr().
Reported by Bruno Haible.
Pádraig Brady [Fri, 17 Mar 2023 19:57:18 +0000 (19:57 +0000)]
tests: aix: avoid unsupported functionality
* tests/misc/tee.sh: AIX doesn't support detecting
closed outputs either with poll() or select() so avoid
testing that functionality.
* tests/tail-2/pipe-f.sh: Likewise.
Bernhard Voelker [Thu, 16 Mar 2023 22:11:47 +0000 (23:11 +0100)]
maint: avoid discarded-qualifiers warnings with SELinux 3.5
Since SELinux version 3.5, the return value of context_str(3) is
declared as const; see:
https://github.com/SELinuxProject/selinux/commit/dd98fa322766
Therefore, GCC complains (here with -Werror):
src/selinux.c: In function 'defaultcon':
src/selinux.c:152:16: error: assignment discards 'const' qualifier \
from pointer target type [-Werror=discarded-qualifiers]
152 | if (!(constr = context_str (tcontext)))
| ^
src/selinux.c: In function 'restorecon_private':
src/selinux.c:252:16: error: assignment discards 'const' qualifier \
from pointer target type [-Werror=discarded-qualifiers]
252 | if (!(constr = context_str (tcontext)))
| ^
* src/selinux.c (defaultcon): Define CONSTR as const.
(restorecon_private): Likewise.
Pádraig Brady [Wed, 15 Mar 2023 13:57:37 +0000 (13:57 +0000)]
cksum: fix --raw on 64 bit big endian systems
* src/sum.c (output_bsd): On sparc64 for example,
a crc of 0 was output due to casting an int variable
to uint16_t and thus operating on the wrong end of the variable.
Instead use explicit assignment to the narrower type
to ensure we get the appropriate data.
(output_sysv): Likewise.
Reported by Bruno Haible.
Pádraig Brady [Mon, 13 Mar 2023 21:26:21 +0000 (21:26 +0000)]
tee: avoid undefined behavior after fclose()
* iopoll.c (fclose_wait): Rename from confusing fclose_nonblock name.
Also adjust to do no operations on the stream after fclose()
as this is undefined. Instead use fflush() to determine EAGAIN status.
(fwrite_wait): Renamed from confusing fwrite_nonblock name.
doc: dircolors: better separate the sections in the database
* src/dircolors.hin: Make the separate sections of the self
documenting dircolors database more apparent,
by adding heading comments, and appropriate separation.
ls: --color: honor separate sequences for extension cases
Following on from commit v8.29-45-g24053fbd8 which unconditionally
used case insensitive extension matching, support selective
case sensitive matching when there are separate extension cases
defined with different display sequences.
* src/dircolors.hin: Document how file name suffixes are matched.
Note this is displayed with `dircolors --print-database` which
the texi info recommends to use for details.
* src/ls.c (parse_ls_color): Postprocess the list to
mark entries for case sensitive matching,
and also adjust so that unmatchable entries are more quickly ignored.
(get_color_indicator): Use exact matching rather than
case insensitive matching if so marked.
* tests/ls/color-ext.sh: Add test cases.
* NEWS: Mention the change in behavior.
Addresses https://bugs.gnu.org/33123
Pádraig Brady [Mon, 13 Mar 2023 15:11:27 +0000 (15:11 +0000)]
tests: adjust du test for recent apparent size change
* tests/du/threshold.sh: Directories are assumed to be
of size 0 with --apparent since commit v9.1-187-g110bcd283
so remove --apparent cases from this test.
Pádraig Brady [Sun, 5 Mar 2023 15:51:32 +0000 (15:51 +0000)]
tee: support non blocking outputs
Non blocking outputs can be seen for example
when piping telnet through tee to a terminal.
In that case telnet sets its input to nonblocking mode,
which results in tee's output being nonblocking,
in which case in may receive an EAGAIN error upon write().
The same issue was seen with mpirun.
The following can be used to reproduce this
locally at a terminal (in most invocations):
* src/iopoll.c (iopoll_internal): A new function refactored from
iopoll(), to also support a mode where we check the output
descriptor is writeable.
(iopoll): Now refactored to just call iopoll_internal().
(fwait_for_nonblocking_write): A new internal function which
uses iopoll_internal() to wait for writeable output
if an EAGAIN or EWOULDBLOCK was received.
(fwrite_nonblock): An fwrite() wrapper which uses
fwait_for_nonblocking_write() to handle EAGAIN.
(fclose_nonblock): Likewise.
src/iopoll.h: Add fclose_nonblock, fwrite_nonblock.
src/tee.c: Call fclose_nonblock() and fwrite_nonblock wrappers,
instead of the standard functions.
* tests/misc/tee.sh: Add a test case.
* NEWS: Mention the improvement.
The idea was suggested by Kamil Dudka in
https://bugzilla.redhat.com/1615467
Paul Eggert [Tue, 7 Mar 2023 20:58:12 +0000 (12:58 -0800)]
split: support split -n on larger pipe input
* bootstrap.conf (gnulib_modules): Add free-posix, tmpfile.
* src/split.c (copy_to_tmpfile): New function.
(input_file_size): Use it to split larger files when sizes cannot
easily be determined via fstat or lseek. See Bug#61386#235.
* tests/split/l-chunk.sh: Mark tests of /dev/zero as
very expensive since they exhaust /tmp.
Paul Eggert [Mon, 6 Mar 2023 23:37:45 +0000 (15:37 -0800)]
maint: pacify ‘make syntax-check’
Problem reported by Pádraig Brady (Bug#61386#226).
* src/split.c (parse_chunk): Use die instead of error.
(main): Quote a string.
* tests/local.mk (all_root_tests): Move du/apparent.sh from here ...
(all_tests): ... to here.
Paul Eggert [Sun, 5 Mar 2023 00:51:11 +0000 (16:51 -0800)]
du: --apparent counts only symlinks and regular
Problem reported by Christoph Anton Mitterer (Bug#61884).
* src/du.c (process_file): When counting apparent sizes, count
only usable st_size members.
* tests/du/apparent.sh: New file.
* tests/local.mk (all_root_tests): Add it.
Paul Eggert [Sat, 4 Mar 2023 21:40:28 +0000 (13:40 -0800)]
split: tune for when creating output files
* src/split.c (create): Avoid fstat + ftruncate in the usual case
where the output file does not already exist, by trying
to create it with O_EXCL first. This costs a failed open
in the unusual case where the output file already exists,
but that’s OK.
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)]
split: handle large numbers better
Prefer signed types to uintmax_t, as this allows for better
runtime checking with gcc -fsanitize=undefined.
Also, when an integer overflows just use the maximal value
when the code will do the right thing anyway.
* src/split.c (set_suffix_length, bytes_split, lines_split)
(line_bytes_split, lines_chunk_split, bytes_chunk_extract)
(lines_rr, parse_chunk, main):
Prefer a signed type (typically intmax_t) to uintmax_t.
(strtoint_die): New function.
(OVERFLOW_OK): New macro. Use it elsewhere, where we now allow
LONGINT_OVERFLOW because the code then does the right thing on all
practical platforms (they have int wide enough so that it cannot
be practically exhausted). We can do this now that we can safely
assume intmax_t has at least 64 bits.
(parse_n_units): New function.
(parse_chunk, main): Use it.
(main): Do not worry about integer overflow when the code
will do the right thing anyway with the extreme value.
Just use the extreme value.
* tests/split/fail.sh: Adjust to match new behavior.
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)]
split: prefer ssize_t for read result
* src/split.c (bytes_split, lines_chunk_split)
(bytes_chunk_extract, main): Prefer ssize_t to size_t when
representing the return value of ‘read’. Use a negative value
instead of SIZE_MAX to indicate a missing value.
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)]
split: be more careful about buffer sizes
* src/split.c: Include sys-limits.h, not safe-read.h.
(input_file_size, bytes_split, lines_split, line_bytes_split)
(lines_chunk_split, bytes_chunk_extract, lines_rr): Call read, not
safe_read, since safe_read no longer buys us anything.
(main): Reject outlandish buffer sizes right away,
rather than allocating huge buffers and never using them.
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)]
split: don’t worry about ECHILD
* src/split.c (closeout): There should be no need for a special
case for ECHILD, since we never wait for the same child twice.
Simplify with this in mind.
Paul Eggert [Sat, 4 Mar 2023 19:42:16 +0000 (11:42 -0800)]
split: simplify SIGPIPE handling
Ignore and default SIGPIPE, rather than blocking and unblocking it.
* src/split.c (default_SIGPIPE):
New static var, replacing oldblocked and newblocked.
(create): Use it.
(main): Set it.