Eli Zaretskii [Wed, 7 Jun 2017 05:32:35 +0000 (08:32 +0300)]
Avoid crashes when SHELL=abcde is specified on the command line
* variable.c (do_variable_definition): If $SHELL was not found,
process "SHELL=foo" as any other variable definition. This avoids
segfaults when SHELL=foo is specified on the Make command line.
Reported by Orgad Shaneh <orgads@gmail.com>.
Paul Smith [Sat, 3 Jun 2017 20:20:51 +0000 (16:20 -0400)]
[SV 51159] Use a non-blocking read with pselect to avoid hangs.
* posixos.c (set_blocking): Set blocking on a file descriptor.
(jobserver_setup): Set non-blocking on the jobserver read side.
(jobserver_parse_auth): Ditto.
(jobserver_acquire_all): Set blocking to avoid a busy-wait loop.
(jobserver_acquire): If the non-blocking read() returns without
taking a token then try again.
Paul Smith [Sun, 28 May 2017 04:33:29 +0000 (00:33 -0400)]
[SV 50823] Support filenames containing '$' in MAKEFILE_LIST
* variable.h (enum variable_flavor: Add a new flavor for appended
values that shouldn't be expanded.
* variable.c (do_variable_definition): If given this new flavor,
do not expand the value before appending it.
* read.c (eval_makefile): Use this new flavor for MAKEFILE_LIST
* tests/scripts/variables/MFILE_LIST: Test filenames containing '$'.
Paul Smith [Sun, 28 May 2017 00:07:30 +0000 (20:07 -0400)]
* NEWS: Do not insert a space during '+=' if the value is empty.
* doc/make.texi (Appending): Document this behavior.
* variable.c (do_variable_definition): Only add a space if the variable
value is not empty.
* tests/scripts/variables/flavors: Test this behavior.
Paul Smith [Mon, 2 Jan 2017 20:46:30 +0000 (15:46 -0500)]
[SV 13651] Handle out-of-memory conditions slightly more gracefully.
* makeint.h: Change OUT_OF_MEM() macro to out_of_memory() function.
* output.h, job.h: Move FD_* macros from job.h to output.h.
* output.c (output_write): Write a buffer to an FD directly.
(out_of_memory): Use output_write() to avoid allocating more
memory while writing the error, and call exit() instead of die().
This does mean we can't translate the error string, though.
* misc.c (xmalloc, xcalloc, xrealloc, xstrdup, xstrndup): Call new
out_of_memory() rather than OUT_OF_MEM().
* read.c (parse_file_seq): Ditto.
Paul Smith [Mon, 17 Apr 2017 19:37:57 +0000 (15:37 -0400)]
Add test suite support to Windows
* main.c (main): Sanitize program name detection on Windows.
* makeint.h: 'program' is a const string on all platforms now.
* tests/run_make_tests.bat: Windows bat file to invoke tests
* tests/test_driver.pl: Obtain system-specific error messages.
(get_osname): Compute the $port_type here. Add more $osname checks
for different Windows Perl ports.
(_run_command): Rewrite the timeout capability to work properly
with Windows. Don't use Perl fork/exec; instead use system(1,...)
which allows a more reliable/proper kill operation.
Also, allow options to be given as a list instead of a string, to
allow more complex quoting of command-line arguments.
* tests/run_make_tests.pl (run_make_with_options): Allow options
to be provided as a list in addition to a simple string.
(set_more_defaults): Write sample makefiles and run make on them
instead of trying to run echo and invoking make with -f-, to avoid
relying on shell and echo to get basic configuration values. Also
create a $sh_name variable instead of hard-coding /bin/sh.
* tests/scripts/features/archives: Skip on Windows.
* tests/scripts/features/escape: Use list method for passing options.
* tests/scripts/features/include: Use system-specific error messages.
* tests/scripts/features/output-sync: "Command not found" errors
generate very different / odd output on Windows. This needs to be
addressed but for now disable these tests on Windows.
* tests/scripts/functions/abspath: Disable on Windows.
* tests/scripts/functions/file: Use system-specific error messages.
* tests/scripts/functions/shell: "Command not found" errors generate
very different / odd output on Windows. This needs to be addressed
but for now disable these tests on Windows.
* tests/scripts/misc/close_stdout: Disable on Windows.
* tests/scripts/options/dash-k: Use system-specific error messages.
* tests/scripts/options/dash-l: Disable on Windows.
* tests/scripts/options/eval: Use list method for passing options.
* tests/scripts/options/general: Skip some non-portable tests.
* tests/scripts/targets/ONESHELL: Skip some non-portable tests.
* tests/scripts/targets/POSIX: Skip some non-portable tests.
* tests/scripts/variables/MAKEFILES: Skip some non-portable tests.
* tests/scripts/variables/SHELL: Use a makefile not -f- for testing.
Eli Zaretskii [Wed, 11 Jan 2017 18:43:41 +0000 (20:43 +0200)]
[SV 50021] Avoid infloop on MS-Windows with short scripts
* job.c (reap_children) [WINDOWS32]: Avoid recursive call to
reap_children when the argument passed to
map_windows32_error_to_string is negative or too large.
Paul Smith [Wed, 28 Dec 2016 05:41:38 +0000 (00:41 -0500)]
* main.c (switches): Add -E as an alias for --eval.
* make.1: Document the -E and --eval options.
* doc/make.texi: Document the -E option.
* tests/scripts/options/eval: Test the -E option and MAKEFILES.
* NEWS: Add information about the new option.
Paul Smith [Wed, 28 Dec 2016 05:23:55 +0000 (00:23 -0500)]
* main.c (switches): Add --no-silent to undo -s options.
* make.1: Document the new flag.
* doc/make.texi: Document the new flag. Remove suggestions that the
.SILENT special target is deprecated or should not be used.
* tests/scripts/options/dash-s: Test the -s and --no-silent options.
* NEWS: Add information about the new option.
Paul Smith [Sun, 25 Dec 2016 22:41:50 +0000 (17:41 -0500)]
[SV 40236] Handle included file open failures properly.
* read.c (eval_makefile): Set deps->error if we discovered any
error reading makefiles, and set NONEXISTENT_MTIME so we know
it needs to be rebuilt.
* main.c (main): Clean up management of makefile_mtimes.
* tests/scripts/features/include: Add open failure testcases.
Paul Smith [Fri, 23 Dec 2016 18:33:58 +0000 (13:33 -0500)]
Portability changes for the test suite.
* tests/test_driver.pl: Save error strings for later comparison.
* tests/run_make_tests.pl: Create portable commands for later use.
* tests/*: Use these new variables.
Paul Smith [Thu, 22 Dec 2016 23:47:26 +0000 (18:47 -0500)]
[SV 20513] Un-escaped # are not comments in function invocations
* NEWS: Document the change, as a backward-incompatible change.
* main.c (main): Add 'nocomment' to the .FEATURES variable.
* read.c (remove_comments): Skip variable references during remove.
(find_char_unquote): Fix comments for new STOPMAP support.
* tests/scripts/features/escape: Test new escape syntax.
* tests/scripts/functions/guile: Ditto.
* tests/scripts/functions/shell: Ditto.
Paul Smith [Sun, 25 Sep 2016 23:06:56 +0000 (19:06 -0400)]
Resolve issues discovered by static code analysis.
* maintMakefile: Add a rule to submit code for analysis.
* configure.ac: Check for availability of the umask() function.
* output.c (output_tmpfd, output_tmpfile): Use umask on temp files.
* makeint.h (PATH_VAR): Reserve an extra character for nul bytes.
* function.c (func_error): Initialize buffer to empty string.
* job.c (child_execute_job): Verify validity of fdin.
* main.c (main): Simplify code for makefile updating algorithm.
* arscan.c (ar_scan): Verify member name length before reading.
* read.c (readline): Cast pointer arithmetic to avoid warnings.
* remake.c (update_file): Remove unreachable code.
(name_mtime): Verify symlink name length.
Marc Ullman [Sat, 12 Nov 2016 08:53:28 +0000 (10:53 +0200)]
Support more than 63 jobs on MS-Windows
* job.c (start_waiting_job, load_too_high):
* w32/w32os.c (jobserver_setup, jobserver_acquire): Abstracted out
MAXIMUM_WAIT_OBJECTS. Call process_table_full instead.
* w32/include/sub_proc.h: Update and add prototypes.
* w32/subproc/sub_proc.c (GMAKE_MAXIMUM_WAIT_OBJECTS): New macro.
(process_wait_for_multiple_objects): Drop-in replacement for Windows
API WaitForMultipleOjects.
(process_wait_for_any_private): Replaced MAXIMUM_WAIT_OBJECTS with
GMAKE_MAXIMUM_WAIT_OBJECTS.
(process_table_full): Replacement for process_used_slots.
(process_used_slots): Removed, as no longer needed.
(process_table_usable_size): Returns maximum usable size of process
table.
(process_table_actual_size): Returns actual size of process table.
(process_register): Added assertion.
(process_easy): Abstracted out MAXIMUM_WAIT_OBJECTS.
Eli Zaretskii [Thu, 6 Oct 2016 19:39:14 +0000 (22:39 +0300)]
Avoid compiler warnings with MinGW runtime 3.22.2
* main.c:
* job.c: Include strings.h, to get the prototypes of strcasecmp
and strncasecmp with latest MinGW runtime versions.
* config.h.W32.template (HAVE_STRINGS_H): Define.
Paul Smith [Tue, 31 May 2016 06:56:51 +0000 (02:56 -0400)]
[SV 47995] Ensure forced double-colon rules work with -j.
The fix for SV 44742 had a side-effect that some double-colon targets
were skipped. This happens because the "considered" facility assumed
that all targets would be visited on each walk through the dependency
graph: we used a bit for considered and toggled it on each pass; if
we didn't walk the entire graph on every pass the bit would get out
of sync. The new behavior after SV 44742 might return early without
walking the entire graph. To fix this I changed the considered value
to an integer which is monotonically increasing: it is then never
possible to incorrectly determine that a previous pass through the
graph already considered the current target.
* filedef.h (struct file): make CONSIDERED an unsigned int.
* main.c (main): No longer need to reset CONSIDERED.
* remake.c (update_goal_chain): increment CONSIDERED rather than
inverting it between 0<->1.
(update_file_1): Reset CONSIDERED to 0 so it's re-considered.
(check_dep): Ditto.
* tests/scripts/features/double_colon: Add a regression test.
Joe Crayne [Sat, 21 May 2016 21:26:00 +0000 (17:26 -0400)]
[SV 44742] Fix double-colon rules plus parallel builds.
* remake.c (update_file): Don't update double-colon target status
if we're still building targets.
(ftime_t): Don't propagate timestamps for double-colon targets that
we've not examined yet.
* tests/scripts/features/double_colon: Add parallel build tests.
Paul Smith [Sun, 1 May 2016 23:24:20 +0000 (19:24 -0400)]
Separate the GNU make load ABI from internal types.
Create an internal type "floc" and convert all users to that type.
* gnumake.h (gmk_floc): Remove the offset field from this type.
* loadapi.c (gmk_eval): Convert gmk_floc to internal floc.
Paul Eggert [Sun, 1 May 2016 19:52:58 +0000 (15:52 -0400)]
Fixes for enhanced GCC warnings.
Move function prototypes into header files and out of .c files.
Use void argument lists for functions that accept no args.
Remove unused macros. Make private functions static. Align
types with printf format characters.
Paul Smith [Sun, 1 May 2016 19:34:42 +0000 (15:34 -0400)]
Fix compile issues with Windows and VMS.
* main.c (initialize_stopchar_map): isblank() is not part of C89.
Install bits for space and tab directly.
* makeint.h: Don't define vfork; autoconf handles this for us.
* vmsjobs.c: Rename NEXT_TOKEN to V_NEXT_TOKEN to avoid conflicts.
* dir.c (print_dir_data_base): Visual Studio C doesn't have int64_t.
* w32/subproc/sub_proc.c (process_begin): Missing arg to memset().
* build_w32.bat: Don't use obsolete Visual Studio flags.
Eli Zaretskii [Sun, 24 Apr 2016 06:53:38 +0000 (09:53 +0300)]
Fix the MS-Windows MinGW build
* build_w32.bat (GccCompile): Use -std=gnu99, as some code uses
C99 features ('for' loop initial declarations).
* dir.c (print_dir_data_base) [WINDOWS32]: Cast 'mtime' to
'int64_t', and use %I64d to print it, to avoid compile-time
warning about printing a 'time_t' value, which could be either
a 32-bit or a 64 bit integral type.
Paul Smith [Sun, 10 Apr 2016 21:12:48 +0000 (17:12 -0400)]
[SV 46433] Show recipe line offsets in line number messages.
While displaying line numbers, show the relevant line number inside
the recipe not just the first line of the entire recipe.
Sample changes suggested by Brian Vandenberg <phantall@gmail.com>
* gnumake.h (gmk_floc): Add an 'offset' to track the recipe offset.
* read.c (eval, eval_makefile, eval_buffer): Initialize 'offset'.
(record_files, install_pattern_rule): Ditto.
* job.c (new_job, job_next_command): Update 'offset' based on the
line of the recipe we're expanding or invoking.
(child_error): Add 'offset' when showing the line number.
* function.c (func_shell_base): Ditto.
* output.c (error, fatal): Ditto.
* NEWS: Mention the new ability.
* tests/scripts/features/errors: Check the line number on errors.
* tests/scripts/functions/warning: Check the line number on warnings.
* tests/scripts/features/output-sync,
tests/scripts/features/parallelism, tests/scripts/functions/shell,
tests/scripts/functions/error: Update line numbers.
Paul Smith [Sat, 9 Apr 2016 23:49:27 +0000 (19:49 -0400)]
[SV 102] Don't show unnecessary include file errors.
Delay the generation of error messages for included files until we
are sure that we can't rebuild that included file.
* dep.h (struct dep): Don't reuse "changed"; make a separate field
to keep "flags". Get rid of dontcare and use the flag.
(struct goaldep): Create a new structure for goal prereqs
that tracks an errno value and the floc where the include happened.
Rework the structures to ensure they are supersets as expected.
In maintainer mode with GCC, use inline to get type checking.
* read.c (eval_makefile): Return a struct goaldep for the new
makefile. Ensure errno is set properly to denote a failure.
(read_all_makefiles): Switch to goaldep and check errno.
(eval): Don't show included file errors; instead remember them.
* remake.c (update_goal_chain): Set global variables to the current
goaldep we're building, and the entire chain.
(show_goal_error): Check if the current failure is a consequence
of building an included makefile and if so print an error.
(complain): Call show_goal_error() on rule failure.
* job.c (child_error): Call show_goal_error() on child error.
* main.c (main): Switch from struct dep to goaldep.
* misc.c (free_dep_chain): Not used; make into a macro.
* tests/scripts/features/include: Update and include new tests.
* tests/scripts/options/dash-B, tests/scripts/options/dash-W,
tests/scripts/options/print-directory,
tests/scripts/variables/MAKE_RESTARTS: Update known-good-output.
Paul Smith [Mon, 4 Apr 2016 05:23:04 +0000 (01:23 -0400)]
Preserve the real value of -jN in MAKEFLAGS using jobserver.
Previously if the jobserver was active, MAKEFLAGS would contain only
the -j option but not the number (not -j5 or whatever) so users
could not discover that value. Allow that value to be provided in
MAKEFLAGS without error but still give warnings if -jN is provided
on the command line if the jobserver is already activated.
* NEWS: Discuss the new behavior.
* os.h, posixos.c, w32/w32os.c: Return success/failure from
jobserver_setup() and jobserver_parse_auth().
* main.c (main): Separate the command line storage of job slots (now
in arg_job_slots) from the control storage (in job_slots). Make a
distinction between -jN flags read from MAKEFLAGS and those seen
on the command line: for the latter if the jobserver is enabled then
warn and disable it, as before.
* tests/scripts/features/jobserver: Add new testing.
Paul Smith [Sat, 2 Apr 2016 20:29:48 +0000 (16:29 -0400)]
Change --jobserver-fds to more generic --jobserver-auth.
* NEWS: Mention the change.
* main.c: Rename jobserver_fds variable to jobserver_auth and
--jobserver-fds option to --jobserver-auth.
* os.h, posixos.c, w32/w32os.c: Rename jobserver_parse_arg() and
jobserver_get_arg() to jobserver_parse_auth()/jobserver_get_auth().
Paul Smith [Mon, 21 Mar 2016 04:36:55 +0000 (00:36 -0400)]
[SV 46995] Strip leading/trailing space from variable names
* makeint.h: Change MAP_SPACE to MAP_NEWLINE, and add MAP_PATHSEP
and MAP_SPACE which is now MAP_BLANK|MAP_NEWLINE. Create
NEW_TOKEN(), END_OF_TOKEN(), ISBLANK(), ISSPACE() macros.
* main.c (initialize_stopchar_map): Set MAP_NEWLINE only for
newline characters.
* Convert all uses of isblank() and isspace() to macros.
* Examine all uses of isblank() (doesn't accept newlines) and
change them wherever possible to ISSPACE() (does accept newlines).
* function.c (func_foreach): Strip leading/trailing space.
* variable.c (parse_variable_definition): Clean up.
* tests/scripts/functions/foreach: Test settings and errors.
* tests/scripts/functions/call: Rewrite to new-style.
* tests/scripts/misc/bs-nl: Add many more tests for newlines.
Paul Smith [Sat, 19 Mar 2016 21:26:08 +0000 (17:26 -0400)]
* function.c (func_file): Support reading from files.
* NEWS: Add information about reading files.
* make.texi (File Function): Describe reading files.
* tests/scripts/functions/file: Test new features for $(file ...)
Paul Smith [Sun, 13 Mar 2016 22:13:00 +0000 (18:13 -0400)]
[SV 45728] Detect changes in .VARIABLES more accurately.
For performance, we only recompute .VARIABLES when (a) it's expanded
and (b) when its value will change from a previous expansion. To
determine (b) we were checking the number of entries in the hash
table which used to work until we started undefining entries: now if
you undefine and redefine the same number of entries in between
expanding .VARIABLES, it doesn't detect any change. Instead, keep
an increasing change number.
* variables.c: Add variable_changenum.
(define_variable_in_set, merge_variable_sets): Increment
variable_changenum if adding a new variable to the global set.
(undefine_variable_in_set): Increment variable_changenum if
undefining a variable from the global set.
(lookup_special_var): Test variable_changenum not the hash table.
* tests/scripts/variables/special: Test undefining variables.
Paul Smith [Sun, 13 Mar 2016 07:02:00 +0000 (03:02 -0400)]
[SV 46581] Pre-define .LOADED to avoid warnings.
* main.c (main): Pre-define .LOADED as a default-level variable.
* load.c (load_file): Set the value rather than append it. Avoid
adding an extra initial whitespace.
* tests/scripts/features/load: Run with --warn-undefined-variables.
Paul Smith [Sun, 13 Mar 2016 06:12:07 +0000 (01:12 -0500)]
[SV 44555] Use vfork() instead of fork() where available.
Testing has shown that vfork() is actually significantly
more efficient on systems where it's supported, even for
copy-on-write implementations. If make is big enough,
duplicating the page tables is significant overhead.
* configure.ac: Check for fork/vfork.
* makeint.h: Include vfork.h and set up #define for it.
* os.h, posixos.c (get_bad_stdin): For children who can't use
the normal stdin file descriptor, get a broken one.
* job.c (start_job_command): Avoid so many ifdefs and simplify
the invocation of child_execute_job()
(child_execute_job): move the fork operation here so it can
return early for the parent process. Switch to use vfork().
* function.c (func_shell_base): Use new child_execute_job() and
simplify ifdefs.
* job.h, main.c, remote-cstms.c, vmsjobs.c, w32os.c: Update
declarations and calls.
Paul Smith [Wed, 9 Mar 2016 04:07:14 +0000 (23:07 -0500)]
[SV 46261] Use pselect() for jobserver where supported.
* Makefile.am, configure.ac: Check for pselect() and sys/select.h.
* main.c (main): Block SIGCHLD if we have pselect() support.
* posixos.c (jobserver_acquire): If we support pselect() then use
it to query the jobserver pipe, while also listening for SIGCHLD.
Also pselect() supports a timeout so avoid alarm() calls.
Paul Smith [Mon, 7 Mar 2016 06:30:21 +0000 (01:30 -0500)]
Clean up some compiler warnings.
* commands.c, commands.h: Use unsigned char for flags.
* dir.c: Use time_t and size_t, and char for a boolean value.
* job.c: Use unsigned and char.
* read.c: Return a signed type since -1 is a valid return code.
Paul Smith [Sat, 5 Mar 2016 20:21:59 +0000 (15:21 -0500)]
[SV 46261] Add more EINTRLOOP wrappers.
This cannot be a perfect solution because there are always other
possible places EINTR can happen, including external libraries
such as gettext, Guile etc.
Paul Smith [Mon, 29 Feb 2016 03:07:50 +0000 (22:07 -0500)]
* strcache.c (add_hugestring): [SV 46832] Support huge strings.
The strcache was limited to strings of length 65535 or less,
because the length is kept in an unsigned short. To support
huge strings add a new simple linked list, which we don't try
to hash.
If the very first string added to the string cache is more than
half the maximum size, we failed when moving the only strcache
buffer to the full list.
Paul Smith [Mon, 29 Feb 2016 01:20:18 +0000 (20:20 -0500)]
[SV 47151] Exit with 1 when checking recursive make -q
* job.h (struct child): New bit to mark recursive command lines.
* job.c (start_job_command): Set the recursive command line bit.
(reap_children): If the child is a recursive command and it exits
with 1 during question mode, don't print an error and exit with 1.
* tests/scripts/options/dash-q: Add a regression test.
Newer versions of binutils allow ar to be compiled to generate
"deterministic archives" by default: in this mode no timestamp
information is generated in the static archive, which utterly
breaks GNU make's archive updating capability. Debian and Ubuntu
have turned this feature on by default in their distributions
which causes the regression tests to fail.
Update the regression tests to check for the availability of the
"U" option to ar which disables deterministic archives and allows
GNU make's archive support to work properly again.