From 6ca5e50062601389971e8cfe23e0779e0ad118ec Mon Sep 17 00:00:00 2001 From: Chet Ramey Date: Mon, 26 Feb 2018 11:16:15 -0500 Subject: [PATCH] commit bash-20180223 snapshot --- CHANGES | 437 +++++++++++++++++++++++++++++++++++++++++++++- CWRU/CWRU.chlog | 29 +++ builtins/common.h | 2 +- doc/bashref.texi | 13 ++ doc/version.texi | 4 +- input.c | 15 +- input.h | 3 +- patchlevel.h | 2 +- subst.c | 9 +- 9 files changed, 504 insertions(+), 10 deletions(-) diff --git a/CHANGES b/CHANGES index 43cb1341a..98947eb13 100644 --- a/CHANGES +++ b/CHANGES @@ -1,3 +1,439 @@ +This document details the changes between this version, bash-5.0-alpha, and +the previous version, bash-4.4-release. + +1. Changes to Bash + +a. Fixed a bug that could cause traps in background jobs to give the terminal + to the wrong process group. + +b. Fixed a bug that caused `kill -l 0' to print an out-of-range error. + +c. Fixed a problem that could result in here-documents being displayed in + the wrong order. + +d. Fixed a number of out-of-bounds and free memory read errors found via + fuzzing. + +e. Fixed a subshell inheritance problem that could cause a subshell to wait for + the wrong process. + +f. Fixed a bug that caused SHLVL to be incremented one too many times when + creating subshells. + +g. A job dying due to SIGINT can now interrupt sourcing a file in a shell with + job control enabled. + +h. Fixed a spurious warning about unterminated ${ or $( constructs during + word completion. + +i. The shell no longer runs traps if a signal arrives while reading command + substitution output. + +j. Fixed an arithmetic expansion error that could allow ++var++ as valid + syntax. + +k. Fixed an error that allowed out-of-bounds references to the directory stack. + +l. The shell does a better job of saving multi-line history entries with + embedded comments. + +m. Fixed a bug that could cause quoted bracket expressions in regular expression + arguments to `[[' to not match correctly. + +n. Fixed a bug that could cause an IFS character in a word to result in an + extra '\001' character in the expansion. + +o. A trailing backslash in a glob pattern can match a trailing backslash in the + string. + +p. Fixed a memory leak in the process creation code path when job control is + enabled. + +q. Fixed a bug that caused `printf' to output broken surrogate pairs for + Japanese locales. + +r. Fixed a bug that caused a SIGINT generated from `kill' in a dot script to + kill an interactive shell. + +s. Fixed a bug that could cause the `read' builtin to not fully read a + multibyte character. + +t. Fixed a bug that could cause identifiers to be evaluated in conditional + arithmetic expressions even when evaluation is suppressed. + +u. Fixed a bug that could result in command substitution, when executed in a + context where word splitting is not performed, to leave a stray '\001' + character in the string. + +v. Fixed a bug that could cause history expansion to be disabled in a non- + interactive shell even if `-o histexpand' is supplied at startup. + +w. Fixed a bug that caused `read -N' to strip leading whitespace IFS characters. + +x. Fixed a bug that caused spurious tilde expansion in arithmetic expressions. + +y. If indirect expansion attempts to indirectly reference through an unset + variable, report an error. + +z. Added a guard to prevent the shell from looping while receiving an endless + stream of SIGTTIN at shell startup. + +aa. Fixed a bug with parsing here documents inside a command substitution when + looking for the closing delimiter. + +bb. Fixed a bug that caused printf to not quote all characters in the + current locale when using the `%q' format specifier. + +cc. Fixed a bug with bash's internal buffered I/O system that caused the input + pointer to not be reset when read(2) returned an EOF. + +dd. Bash now installs its SIGWINCH signal handler with SA_RESTART, so it will + not interrupt open/read/write system calls. + +ee. The ERR trap now reports line numbers more reliably. + +ff. The shell no longer tries to manipulate the terminal process group if a + command or process substitution is killed by SIGTERM when job control is + enabled. + +gg. Fixed a bug that caused extglob patterns to match filenames beginning with + a period. + +hh. File descriptors open for writing to here documents are no longer available + to subshells. + +ii. Make sure word completion doesn't perform command or process substitution. + +jj. Fixed a bug with parsing $$'...' inside a command substitution. + +kk. Fixed a bug that caused bash to remove backslash-newline pairs from the + body of a here-document with a quoted delimiter inside a command + substitution. + +ll. Fixed a bug that could cause the shell to hang when adding a pid to the + table of background process exit statuses. + +mm. Fixed a bug that could cause 0x01 characters to be doubled in the output + of process substitution. + +nn. Restricted shells now clear the hash table before making the PATH variable + read-only. + +oo. There are a number of changes to the expansion of $* and $@ in contexts + where word splitting does not occur (quoted and unquoted), with IFS set + to NULL or a non-standard value, mostly to deal with the consequences of + the behavior defined in Posix interpretation 888. + +pp. There are a number of changes to nameref variable handling to avoid + creating variables with invalid names. + +qq. A non-interactive posix mode shell no longer exits when an assignment + statement fails if the assignment is utimately being performed by the + `command' builtin. + +rr. When using character class names for globbing, don't allow case + insensitivity, even if nocaseglob is enabled. + +ss. Fixed a bug that allowed some redirections to stay in place if a later + redirection failed. + +tt. Fixed a bug in how command and process substitutions are recognized within + other parameter expansions. + +uu. Fixed a bug that caused bash to loop under certain circumstances when + performing arithmetic expansion on a variable whose value is an invalid + expression. + +vv. Fixed a bug that could cause bash to expand aliases inappropriately while + parsing compound commands like `case'. + +ww. Fixed a bug that could cause `read -N' to fail to read complete multibyte + characters, even when the sequences are incomplete or invalid, with or + without readline. + +xx. Fixed a bug that could cause `case' to fail to match patterns containing + 0x01 characters. + +yy. Fixed a bug that caused exported functions to contain stray 0x01 characters. + +zz. Fixed some inconsistencies with how the history number is handled in the + various prompt strings. + +aaa. Fixed a bug that could cause a core dump if READLINE_LINE was unset + inside a shell function bound to a key sequence with `bind -x'. + +bbb. Fixed a bug that could cause bash to not read a token terminator correctly + if a command substitution was used inside an arithmetic `for' command. + +ccc. Fixed problems that could occur with a fatal arithmetic expansion error + in a context (like prompt expansion) where you can't jump back to the + top level. + +ddd. Expression errors in arithmetic `for' commands are treated more like + shell syntax errors. + +eee. Fixed a parser synchronization error resulting from a syntax error + followed immediately by an EOF. + +fff. When executing a shell function, the first line in the function ($LINENO) + is line 1 instead of line 0, as Posix requires. + +ggg. In Posix mode, bash will canonicalize the value of PWD it inherits from + the environment and use that to set its idea of the current directory. + +hhh. If LINENO is exported, bash needs to regenerate its value each time it + constructs the environment. + +iii. Fixed a bug with restoring the SIGINT handler when using `wait -n'. + +jjj. Make sure the `coproc' command returns an appropriate status if the NAME + argument is invalid. + +kkk. Fixed a problem with arithmetic expressions containing array references + that contain arithmetic expressions with syntax errors. + +lll. The `select' command and help builtin will use $COLUMNS before the window + size returned from the kernel as the terminal width. + +mmm. `read -n 0' and `read -N 0' now try a zero-length read to attempt to + detect file descriptor errors. + +nnn. The `read' builtin now does a better job of acting on signals that don't + interrrupt read(2). + +ooo. Fixed some cases where `printf -v' did not return failure status on a + variable assignment error. + +ppp. Fixed temporary environment propagation back to the current environment + so that it doesn't happen for special builtins run by the `command' + builtin. + +qqq. Fixed a bug when searhing for the end of a here-document delimiter in a + command substitution. + +rrr. Fixed a bug that could cause `cd ${DIRSTACK[0]}' to fail. + +sss. Fixed a bug that could cause reserved words to not be recognized in a + for statement without the `in' inside a command substitution. + +ttt. Fixed a bug that could cause a double-free in a timed command with an + expansion error. + +uuu. Fixed a bug that could cause a core dump if a script switches from a UTF-8 + locale to a different locale after displaying a lone surrogate character. + +vvv. Fixed cases where bash prematurely removed FIFOs attached to process + substitutions. + +www. Fixed a problem with calculating the size of the table that stores exit + statuses from background processes when the child process resource limit + is very large. + +xxx. Fixed a memory leak with functions using `return' when using FIFOs for + standard input. + +yyy. `wait' without arguments attempts to wait for all active process + substitution processes. + +zzz. Fixed a bug where an indirect parameter was subjected to word splitting + when trying to find the indirected variable name. + +aaaa. Fixed a bug that could allow restricted shell users to add commands to + the hash table. + +bbbb. When using the `!(patlist)' extended globbing operator, a filename + beginning with a `.' that doesn't match any of the patterns is not + returned as a match if leading dots must be matched explicitly. + +cccc. Fixed a bug that could cause line number and source file information for + a function definition to be incorrect if there are multiple definitions. + +dddd. Fixed a bug that could cause builtins like `readonly' to behave + differently when applied to arrays and scalar variables within functions. + +eeee. Fixed a bug that could cause alias expansion to add an extra space to + a quoted string that begins outside the alias expansion. + +ffff. Fixed a bug that could result in unwanted alias expansion after timing + the null command. + +gggg. Fixed a bug that could cause a core dump if a timestamp in a history + file overflowed a time_t. + +hhhh. Restricted shells can no longer redirect from /dev/tcp or /dev/udp, since + the kernel calls make those file descriptors read-write. + +iiii. Fixed a problem with splitting double-quoted words for programmable + completion when the double quote immediately follows another word + delimiter. + +2. Changes to Readline + +a. Added a guard to prevent nested macros from causing an infinite expansion + loop. + +b. Instead of allocating enough history list entries to hold the maximum list + size, cap the number allocated initially. + +c. Added a strategy to avoid allocating huge amounts of memory if a block of + history entries without timestamps occurs after a block with timestamps. + +d. Added support for keyboard timeouts when an ESC character is the last + character in a macro. + +e. There are several performance improvements when in a UTF-8 locale. + +f. Readline does a better job of preserving the original set of blocked = + signals when using pselect() to wait for input. + +g. Fixed a bug that caused multibyte characters in macros to be mishandled. + +h. Fixed several bugs in the code that calculates line breaks when expanding + prompts that span several lines, contain multibyte characters, and contain + invisible character seqeuences. + +i. Fixed several bugs in cursor positioning when displaying lines with prompts + containing invisible characters and multibyte characters. + +j. When performing case-insensitive completion, Readline no longer sorts the + list of matches unless directed to do so. + +k. Fixed a problem with key sequences ending with a backslash. + +l. Fixed out-of-bounds and free memory read errors found via fuzzing. + +m. Fixed several cases where the mark was set to an invalid value. + +n. Fixed a problem with the case-changing operators in the case where the + lower and upper case versions of a character do not have the same number + of bytes. + +o. Handle incremental and non-incremental search character reads returning EOF. + +p. Handle the case where a failing readline command at the end of a multi-key + sequence could be misinterpreted. + +q. The history library now prints a meaningful error message if the history + file isn't a regular file. + +r. Fixed a problem with vi-mode redo (`.') on a command when trying to replace + a multibyte character. + +s. The key binding code now attempts to remove a keymap if a key unbinding + leaves it empty. + +t. Fixed a line-wrapping issue that caused problems for some terminal + emulators. + +u. If there is a key bound to the tty's VDISCARD special character, readline + disables VDISCARD while it is active. + +v. Fixed a problem with exiting bracketed paste mode on terminals that assume + the bracketed paste mode character sequence contains visible characters. + +3. New Features in Bash + +a. The `wait' builtin can now wait for the last process substitution created. + +b. There is an EPOCHSECONDS variable, which expands to the time in seconds + since the Unix epoch. + +c. There is an EPOCHREALTIME variable, which expands to the time in seconds + since the Unix epoch with microsecond granularity. + +d. New loadable builtins: rm, stat, fdflags. + +e. BASH_ARGV0: a new variable that expands to $0 and sets $0 on assignment. + +f. When supplied a numeric argument, the shell-expand-line bindable readline + command does not perform quote removal and suppresses command and process + substitution. + +g. `history -d' understands negative arguments: negative arguments offset from + the end of the history list. + +h. The `name' argument to the `coproc' reserved word now undergoes word + expansion, so unique coprocs can be created in loops. + +i. A nameref name resolution loop in a function now resolves to a variable by + that name in the global scope. + +j. The `wait' builtin now has a `-f' option, which signfies to wait until the + specified job or process terminates, instead of waiting until it changes + state. + +k. There is a define in config-top.h that allows the shell to use a static + value for $PATH, overriding whatever is in the environment at startup, for + use by the restricted shell. + +l. Process substitution does not inherit the `v' option, like command + substitution. + +m. If a non-interactive shell with job control enabled detects that a foreground + job died due to SIGINT, it acts as if it received the SIGINT. + +n. The SIGCHLD trap is run once for each exiting child process even if job + control is not enabled when the shell is in Posix mode. + +o. A new shopt option: localvar_inherit; if set, a local variable inherits the + value of a variable with the same name at the nearest preceding scope. + +p. `bind -r' now checks whether a key sequence is bound before binding it to + NULL, to avoid creating keymaps for a multi-key sequence. + +q. A numeric argument to the line editing `operate-and-get-next' command + specifies which history entry to use. + +r. The positional parameters are now assigned before running the shell startup + files, so startup files can use $@. + +s. There is a compile-time option that forces the shell to disable the check + for an inherited OLDPWD being a directory. + +t. The `history' builtin can now delete ranges of history entries using + `-d start-end'. + +u. The `vi-edit-and-execute-command' bindable readline command now puts readline + back in vi insertion mode after executing commands from the edited file. + +v. The command completion code now matches aliases and shell function names + case-insensitively if the readline completion-ignore-case variable is set. + +w. There is a new `assoc_expand_once' shell option that attempts to expand + associative array subscripts only once. + +x. The shell only sets up BASH_ARGV and BASH_ARGC at startup if extended + debugging mode is active. The old behavior of unconditionally setting them + is available as part of the shell compatibility options. + +4. New Features in Readline + +a. Non-incremental vi-mode search (`N', `n') can search for a shell pattern, as + Posix specifies (uses fnmatch(3) if available). + +b. There are new `next-screen-line' and `previous-screen-line' bindable + commands, which move the cursor to the same column in the next, or previous, + physical line, respectively. + +c. There are default key bindings for control-arrow-key key combinations. + +d. A negative argument (-N) to `quoted-insert' means to insert the next N + characters using quoted-insert. + +e. New public function: rl_check_signals(), which allows applications to + respond to signals that readline catches while waiting for input using + a custom read function. + +f. There is new support for conditionally testing the readline version in an + inputrc file, with a full set of arithmetic comparison operators available. + +g. There is a simple variable comparison facility available for use within an + inputrc file. Allowable operators are equality and inequality; string + variables may be compared to a value; boolean variables must be compared to + either `on' or `off'; variable names are separated from the operator by + whitespace. + +------------------------------------------------------------------------------ This document details the changes between this version, bash-4.4-release, and the previous version, bash-4.4-rc2. @@ -21,7 +457,6 @@ e. Fixed several potential buffer overflow issues in the word expansion code. 3. New Features in Bash 4. New Features in Readline - ------------------------------------------------------------------------------ This document details the changes between this version, bash-4.4-rc2, and the previous version, bash-4.4-beta2. diff --git a/CWRU/CWRU.chlog b/CWRU/CWRU.chlog index a99cbb677..fdb6ee256 100644 --- a/CWRU/CWRU.chlog +++ b/CWRU/CWRU.chlog @@ -15018,3 +15018,32 @@ builtins/shopt.def doc/{bash.1,bashref.texi} - compat44: document new shell option + + 2/21 + ---- +builtins/common.h + - ISOPTION: check s[1] before s[2] to avoid out of bound reads. Fixes + bug reported by jeremy@feusi.co + +input.h + - B_SHAREDBUF: new flag for buffered input fds, indicates that this + fd shares its b_buffer with another fd + +input.c + - duplicate_buffered_stream: when duplicating *from* the buffer + corresponding to bash input, make sure the new buffer has B_SHAREDBUF + in the flags, because copy_buffered_stream copies pointers + - duplicate_buffered_stream: if the buffer being copied to has the + B_SHAREDBUF flag set, make sure the b_buffer doesn't get freed. + Fixes use-after-free bug reported by jeremy@feusi.co + - save_bash_input,close_buffered_stream: make sure a buffer with + B_SHAREDBUF set has its b_buffer set to NULL before calling + free_buffered_stream + + 2/24 + ---- +subst.c + - parameter_brace_expand_error: add parameter saying whether or not + we are checking whether value is null, so we can have different + error messages for ${x:?} and ${x?}. Report and fix from + don fong diff --git a/builtins/common.h b/builtins/common.h index c7b3ae0c6..96a789d6b 100644 --- a/builtins/common.h +++ b/builtins/common.h @@ -23,7 +23,7 @@ #include "stdc.h" -#define ISOPTION(s, c) (s[0] == '-' && !s[2] && s[1] == c) +#define ISOPTION(s, c) (s[0] == '-' && s[1] == c && !s[2]) #define ISHELP(s) (STREQ ((s), "--help")) #define CHECK_HELPOPT(l) \ diff --git a/doc/bashref.texi b/doc/bashref.texi index 2ba845bac..d1d718d09 100644 --- a/doc/bashref.texi +++ b/doc/bashref.texi @@ -8164,6 +8164,19 @@ bash-2.04$ ./configure --help @noindent at the Bash prompt in your Bash source directory. +If you want to build Bash in a directory separate from the source +directory -- to build for multiple architectures, for example -- +just use the full path to the configure script. The following commands +will build bash in a directory under @file{/usr/local/build} from +the source code in @file{/usr/local/src/bash-4.4}: + +@example +mkdir /usr/local/build/bash-4.4 +cd /usr/local/build/bash-4.4 +bash /usr/local/src/bash-4.4/configure +make +@end example + If you need to do unusual things to compile Bash, please try to figure out how @code{configure} could check whether or not to do them, and mail diffs or instructions to diff --git a/doc/version.texi b/doc/version.texi index 5d84d6ff0..d14f7fb55 100644 --- a/doc/version.texi +++ b/doc/version.texi @@ -2,10 +2,10 @@ Copyright (C) 1988-2018 Free Software Foundation, Inc. @end ignore -@set LASTCHANGE Fri Feb 16 14:38:44 EST 2018 +@set LASTCHANGE Sun Feb 25 16:17:31 EST 2018 @set EDITION 4.4 @set VERSION 4.4 -@set UPDATED 16 February 2018 +@set UPDATED 25 February 2018 @set UPDATED-MONTH February 2018 diff --git a/input.c b/input.c index 6d0e6871c..8b3e4259b 100644 --- a/input.c +++ b/input.c @@ -1,6 +1,6 @@ /* input.c -- functions to perform buffered input with synchronization. */ -/* Copyright (C) 1992-2009 Free Software Foundation, Inc. +/* Copyright (C) 1992-2018 Free Software Foundation, Inc. This file is part of GNU Bash, the Bourne Again SHell. @@ -272,6 +272,8 @@ save_bash_input (fd, new_fd) /* What's this? A stray buffer without an associated open file descriptor? Free up the buffer and report the error. */ internal_error (_("save_bash_input: buffer already exists for new fd %d"), nfd); + if (buffers[nfd]->b_flag & B_SHAREDBUF) + buffers[nfd]->b_buffer = (char *)NULL; free_buffered_stream (buffers[nfd]); } @@ -351,6 +353,12 @@ duplicate_buffered_stream (fd1, fd2) /* If the two objects share the same b_buffer, don't free it. */ if (buffers[fd1] && buffers[fd1]->b_buffer && buffers[fd1]->b_buffer == buffers[fd2]->b_buffer) buffers[fd2] = (BUFFERED_STREAM *)NULL; + /* If this buffer is shared with another fd, don't free the buffer */ + else if (buffers[fd2]->b_flag & B_SHAREDBUF) + { + buffers[fd2]->b_buffer = (char *)NULL; + free_buffered_stream (buffers[fd2]); + } else free_buffered_stream (buffers[fd2]); } @@ -365,6 +373,9 @@ duplicate_buffered_stream (fd1, fd2) buffers[fd2]->b_flag |= B_WASBASHINPUT; } + if (fd_is_bash_input (fd1) || (buffers[fd1] && (buffers[fd1]->b_flag & B_SHAREDBUF))) + buffers[fd2]->b_flag |= B_SHAREDBUF; + return (fd2); } @@ -436,6 +447,8 @@ close_buffered_stream (bp) if (!bp) return (0); fd = bp->b_fd; + if (bp->b_flag & B_SHAREDBUF) + bp->b_buffer = (char *)NULL; free_buffered_stream (bp); return (close (fd)); } diff --git a/input.h b/input.h index 5f860962f..6aef12692 100644 --- a/input.h +++ b/input.h @@ -1,6 +1,6 @@ /* input.h -- Structures and unions used for reading input. */ -/* Copyright (C) 1993-2016 Free Software Foundation, Inc. +/* Copyright (C) 1993-2018 Free Software Foundation, Inc. This file is part of GNU Bash, the Bourne Again SHell. @@ -49,6 +49,7 @@ enum stream_type {st_none, st_stdin, st_stream, st_string, st_bstream}; #define B_UNBUFF 0x04 #define B_WASBASHINPUT 0x08 #define B_TEXT 0x10 +#define B_SHAREDBUF 0x20 /* shared input buffer */ /* A buffered stream. Like a FILE *, but with our own buffering and synchronization. Look in input.c for the implementation. */ diff --git a/patchlevel.h b/patchlevel.h index 93dbe0db3..1cd7c96c7 100644 --- a/patchlevel.h +++ b/patchlevel.h @@ -25,6 +25,6 @@ regexp `^#define[ ]*PATCHLEVEL', since that's what support/mkversion.sh looks for to find the patch level (for the sccs version string). */ -#define PATCHLEVEL 12 +#define PATCHLEVEL 0 #endif /* _PATCHLEVEL_H_ */ diff --git a/subst.c b/subst.c index ffb69f291..c98c68cb4 100644 --- a/subst.c +++ b/subst.c @@ -317,7 +317,7 @@ static WORD_DESC *parameter_brace_expand_word __P((char *, int, int, int, arrayi static char *parameter_brace_find_indir __P((char *, int, int, int)); static WORD_DESC *parameter_brace_expand_indir __P((char *, int, int, int *, int *)); static WORD_DESC *parameter_brace_expand_rhs __P((char *, char *, int, int, int, int *, int *)); -static void parameter_brace_expand_error __P((char *, char *)); +static void parameter_brace_expand_error __P((char *, char *, int)); static int valid_length_expression __P((char *)); static intmax_t parameter_brace_expand_length __P((char *)); @@ -6916,8 +6916,9 @@ parameter_brace_expand_rhs (name, value, op, quoted, pflags, qdollaratp, hasdoll used as the error message to print, otherwise a standard message is printed. */ static void -parameter_brace_expand_error (name, value) +parameter_brace_expand_error (name, value, check_null) char *name, *value; + int check_null; { WORD_LIST *l; char *temp; @@ -6931,6 +6932,8 @@ parameter_brace_expand_error (name, value) FREE (temp); dispose_words (l); } + else if (check_null == 0) + report_error (_("%s: parameter not set"), name); else report_error (_("%s: parameter null or not set"), name); @@ -8923,7 +8926,7 @@ bad_substitution: } else if (c == '?') { - parameter_brace_expand_error (name, value); + parameter_brace_expand_error (name, value, check_nullness); return (interactive_shell ? &expand_wdesc_error : &expand_wdesc_fatal); } else if (c != '+') -- 2.47.2