Paul Smith [Mon, 19 Apr 2021 21:27:54 +0000 (17:27 -0400)]
[SV 58341] Add non-trivial options to $(MAKEFLAGS)
Previously only trivial (single-letter) options were available in
$(MAKEFLAGS) when it is examined from inside a makefile (the full
value was set before expanding recipes).
Ensure that all options (but, not command line variable overrides!)
are visible in the $(MAKEFLAGS) variable. In order to do this
reset the default values of options, particularly options which are
lists, before re-reading MAKEFLAGS after makefiles have been read.
Otherwise we'll get duplicate values for options such as -I.
Unfortunately there are complications here as well: sometimes (for
jobserver options in particular) we modify the values of these
options while running: we must not reset these modifications.
* NEWS: Announce this change
* src/main.c (main): Call new reset_switches() before re-parsing
MAKEFLAGS.
(reset_switches): Reset the value of non-special options to their
defaults.
(define_makeflags): Add non-special options into MAKEFLAGS even
if 'all' is not set.
Depending on how make was configured it may add duplicate
directories in the default include search path: avoid this.
(Tweaked by psmith@gnu.org)
* configure.ac: Set AM variable KNOWN_PREFIX if --prefix is known
* Makefile.am: Only set INCLUDEDIR if prefix is not known
* read.c [default_include_directories]: Only add INCLUDEDIR if set
Eli Zaretskii [Sat, 29 May 2021 08:37:09 +0000 (11:37 +0300)]
[SV 58656] Fix mtime for large files on MS-Windows
In MSVC builds, 'stat' fails when called on files larger than
2GB. Call '_stat64' instead to work around this.
* src/remake.c (STAT): Define to '_stat64' for MSVC builds.
(name_mtime) [WINDOWS32]: Use STAT instead of 'stat'.
Suggested by Makoto Kato <m_kato@ga2.so-net.ne.jp>.
Paul Smith [Sun, 28 Mar 2021 20:22:57 +0000 (16:22 -0400)]
Fix build and test issues on Windows
* src/dep.h (DEP): Remove extraneous semicolon.
* src/dir.c (find_directory) [W32]: Replace removed variable.
* tests/scripts/features/include: Allow the extra default makefile
searched for on Windows.
* tests/scripts/functions/file: Use $(info ...) to avoid "command
too long" errors on Windows.
Keep a count of bytes read rather than comparing pointers since the
variable_buffer might get reallocated.
Bug and patch by Ken Tossell <ken@tossell.net>
Regression tests by Dmitry Goncharov <dgoncharov@users.sf.net>
Tweaked by Paul Smith <psmith@gnu.org>
* src/function.c (func_file): Use bytes read rather than a pointer.
* tests/scripts/functions/file: Provide various tests for reading
empty files, files with/without newlines, and large files.
Dmitry Goncharov [Mon, 15 Mar 2021 06:10:49 +0000 (02:10 -0400)]
[SV 60188] Explicit prereqs cannot be intermediate files
If a prereq of a pattern is an explicit target, it should not be
considered an intermediate file.
(Minor tweaks by Paul Smith <psmith@gnu.org>)
* src/dep.h (struct nameseq): Add is_explicit flag.
* src/implicit.c (struct patdeps): Ditto.
(pattern_search): Set the is_explicit flag appropriately for each
prerequisite, based on whether it contained a pattern or not.
Update the help output to note implicit vs. explicit prereqs.
* tests/scripts/features/double_colon: Add tests.
* tests/scripts/features/grouped_targets: Ditto.
* tests/scripts/features/patternrules: Ditto.
* tests/scripts/features/se_implicit: Ditto.
* tests/scripts/features/statipattrules: Ditto.
Paul Smith [Mon, 15 Mar 2021 05:09:32 +0000 (01:09 -0400)]
Ensure variable_buffer is always set.
Initialize the global variable_buffer in main() so that it is never
a null pointer. Then invoking variable_expand("") is never needed:
simply use the variable_buffer pointer when we want to restart the
variable buffer. The main point of this simplification is not to
keep a separate pointer to the beginning of the buffer: this is
dangerous because the buffer may be re-allocated. Instead always
use the variable_buffer pointer itself.
* src/variable.h (initialize_variable_output): Publish.
* src/expand.c (initialize_variable_output): Remove static.
* src/main.c (main): Initialize variable_buffer.
* src/file.c (enter_prereqs): Don't call variable_expand("") and
don't save a separate buffer pointer than might be outdated.
(expand_deps): Ditto.
* src/read.c (record_files): Ditto.
* src/remake.c (library_search): Ditto.
Paul Smith [Sun, 14 Mar 2021 23:12:24 +0000 (19:12 -0400)]
[SV 59881] Handle vertical TAB consistently
While parsing makefiles get_next_mword() was treating VTAB as a word
character rather than a word separator. However, when using
find_next_token(), for example in patsubst_expand_pat(), we treated
VTAB as a word separator causing multiple words to appear where we
didn't expect them.
* src/makeint.h (END_OF_TOKEN): Change from a loop to a boolean check.
* src/misc.c (end_of_token): Move the loop here.
* src/read.c (get_next_mword): Skip whitespace, not just blank, to
find the start of the word and use END_OF_TOKEN() to decide when the
current word is finished.
Paul Smith [Sun, 14 Mar 2021 20:32:47 +0000 (16:32 -0400)]
[SV 59870] define/undefine prerequisites are not target-specific vars
* src/read.c (parse_var_assignment): If called in a target-specific
variable context don't allow define/undefine as variable assignments.
* test/scripts/variables/define: Add a test.
* test/scripts/variables/undefine: Add a test.
Add a new function $(let ...) which allows lexically scoped variables.
* NEWS: Add information on this feature.
* doc/make.texi (Let Function): Document the 'let' function.
* src/function.c (func_let): Create the 'let' built-in function.
* tests/scripts/functions/let: Test the 'let' built-in function.
Paul Smith [Sat, 5 Dec 2020 21:25:12 +0000 (16:25 -0500)]
[SV 59169] Add --debug=why and --debug=print options
Add debug options to print recipes even if they would otherwise be
silent, and to print the reason that a target was considered out of
date.
Modify --trace to simply be a shorthand for --debug=print,why.
* NEWS: Announce changes.
* doc/make.texi (Summary of Options): Document the new options.
* doc/make.1: Ditto.
* src/debug.h: Add new flags DB_PRINT and DB_WHY.
* src/makeint.h: Remove the trace_flag variable.
* src/job.c (start_job_command): Check debug flags not trace_flag.
(new_job): Ditto.
* src/main.c (trace_flag): Make a static variable for switches.
(decode_debug_flags): Set DB_PRINT and DB_WHY if trace_flag is set.
* tests/scripts/variables/GNUMAKEFLAGS: Update known-good messages.
* tests/scripts/variables/MAKEFLAGS: Ditto.
Paul Smith [Sat, 5 Dec 2020 20:06:25 +0000 (15:06 -0500)]
[SV 57676] Support included files being built after failing
If a -include file does not exist, then some subsequent operation
creates it, then allow it to be successfully included.
* src/read.c (eval_makefile): If our last mtime was NONEXISTENT then
reset it to 0 so we'll check it again.
* tests/scripts/features/include: Add a test for this behavior.
Paul Smith [Sun, 29 Nov 2020 19:03:30 +0000 (14:03 -0500)]
Avoid some valgrind warnings
Using sscanf() to parse archive header values (struct ar_hdr) can lead
to valgrind warnings which are probably bogus but are annoying.
To be safer, create a local method to convert the ASCII integer
strings into integers.
* src/arscan.c (parse_int): Turn integer strings into integers.
(ar_scan): Initialize struct ar_hdr memory.
Call parse_int() rather than sscanf/atol.
Paul Smith [Fri, 27 Nov 2020 22:07:53 +0000 (17:07 -0500)]
[SV 41273] Allow the directory cache to be invalidated
Each time we invoke a command it's possible that it will change the
filesystem in ways that were not described by the target. If that
happens but we have cached previous directory contents then we may
make decisions or report results based on obsolete information.
Keep a count of how many commands we've invoked, and remember the
current command count every time we load the contents of a directory.
If we request the directory and the current command count has changed
we know the cache is outdated so reload from scratch.
* NEWS: Announce the change.
* src/makeint.h (command_count): Create a global counter.
* src/main.c (command_count): Ditto.
* src/job.c (reap_children): Increment the counter on job completion.
* src/function.c (func_file): Increment if we write a file.
* src/dir.c (clear_directory_contents): Clear the current contents of
a cached directory.
(struct directory_contents): Remember the counter value.
(struct directory): Remember the counter value for non-existing dirs.
(find_directory): If we have a cached directory and the count hasn't
changed then return it. Else, clear the previous contents and re-read
from scratch.
* tests/scripts/features/dircache: Add tests of the directory cache.
Paul Smith [Sat, 28 Nov 2020 17:30:08 +0000 (12:30 -0500)]
Support "unexport" in target-specific variables.
Rewrite the environment variable algorithm to correctly inherit
export settings from parent variable sets. The new algorithm
for computing the table of environment variables is:
- Start with the most local variable set and proceed to global.
- If the variable already exists in the table and we don't know
its export status, update it with the current variable's status.
- If the variable is not in the table and it's not global, add it
regardless of its status so if it's unexported we remember that.
- If the variable is not in the table and is global, check its
export status and don't add it if we won't export it.
Then when generating the environment variables, check the export
status of each variable in case it was a target-specific variable
and we have determined it should not be exported.
Rework SHELL handling to check at the end whether we added it or
not and if we didn't, add the value from the environment.
* NEWS: Announce support for target-specific "unexport"."
* doc/make.texi (Target-specific): Document the support.
* src/variable.h (enum variable_export): Make into a global type.
* src/read.c (struct vmodifiers): Use enum variable_export rather
than individual booleans.
(parse_var_assignment): Parse the "unexport" keyword.
(eval): Remember the vmodifier value in the variable.
(record_target_var): Ditto.
* src/variable.c (should_export): Check if the variable should be
exported.
(target_environment): Implement the above algorithm.
* tests/scripts/features/export: Test export/unexport with variable
assignments on the same line.
* tests/scripts/features/targetvars: Add a comprehensive suite of
tests for different types of target-specific export / unexport.
* tests/scripts/variables/SHELL: Update the comment.
Paul Smith [Fri, 27 Nov 2020 21:56:22 +0000 (16:56 -0500)]
[SV 59230] Ensure environment variables are exportable
When checking for invalid environment variable names we searched the
entire name string instead of just the first LENGTH chars; this could
cause us to incorrectly decide the variable was not exportable.
Dmitry Goncharov <dgoncharov@users.sf.net> found this bug and
provided a test case and sample fix: I used the test but chose a
slightly different fix.
* src/variable.c (define_variable_in_set): check the variable name
not the input string.
* tests/scripts/features/targetvars: Ensure environment variable
values are exported.
Reported by Dmitry Goncharov <dgoncharov@users.sf.net>, with a patch
changing the pattern rule for building archives. I decided to
change the default value of ARFLAGS on AIX instead.
Paul Smith [Fri, 27 Nov 2020 16:37:29 +0000 (11:37 -0500)]
[SV 35711] Check for special targets earlier
GNU make must recognize some special targets as they are defined.
Because of the way targets are defined, we were not recognizing these
special targets until we were handling the NEXT statement. However
that's too late for some special targets such as .POSIX etc. which can
change the behavior of make during parsing.
Check for special targets earlier, as soon as we've finished parsing
the target introduction line (before we've even parsed the recipe).
* NEWS: Mention the change.
* src/read.c (check_specials): New function to look for special
targets. Move checks from eval() and record_files() to this new
function.
(eval): Call check_specials() after we've completed parsing the target
introduction line. Move default goal detection to check_specials().
(record_files): Move handling of .POSIX, .SECONDEXPANSION, and
.ONESHELL to check_specials().
* tests/scripts/misc/bs-nl: Remove workaround for late .POSIX issue.
* tests/scripts/targets/POSIX: Add a comment.
Paul Smith [Thu, 12 Nov 2020 22:00:39 +0000 (17:00 -0500)]
[SV 59093] Rewrite filter/filter-out to avoid large stack usage
* src/function.c (func_filter_filterout): Allocate arrays to hold
pattern and word information rather than creating linked lists on
the stack.
* tests/scripts/functions/filter-out: Test large filters.
Paul Smith [Sun, 19 Jul 2020 17:56:23 +0000 (13:56 -0400)]
[SV 58735] Define the order that makefiles are rebuilt.
Ensure that makefiles are rebuilt in the order in which make first
considered them, and document this behavior in the manual.
* NEWS: Add a note about the new behavior
* doc/make.text (How make Processes a Makefile): Document it.
* main.c (main): Inverse the list of makefile goals.
* read.c (read_all_makefiles): Add default makefiles to the list at
the front in reverse order, the same way other makefiles are added.
* tests/scripts/features/include: Add tests to verify rebuild order.
Kevin Buettner [Thu, 23 Apr 2020 21:05:34 +0000 (17:05 -0400)]
[SV 58232] Disable inheritance of jobserver FDs for recursive make
A parent make will invoke a sub-make with close-on-exec disabled for
the jobserver pipe FDs. Force close-on-exec to be to be enabled in
the sub-make so the pipe is not always passed to child jobs.
I have a test case which, when invoked with a suitable -j switch,
will hang if the recipe inherits the jobserver pipe. This test case
was inspired by a real world case in which testing GDB on Fedora
would hang due to some poorly written test GDB cases having been
passed the jobserver file descriptors.
* src/posixos.c (jobserver_parse_auth): Call fd_noinherit() for
jobserver pipe descriptors.
Paul Smith [Wed, 1 Apr 2020 05:13:02 +0000 (01:13 -0400)]
[SV 57674] Use the system default PATH if $PATH is not set
When using execvp() if $PATH is not present in the environment
it will automatically search the system default PATH string. Emulate
this by passing the system default PATH to find_in_given_path() if
we don't find PATH in the environment.
* src/job.c (child_execute_job): Use confstr(_CS_PATH) if PATH is not
found.
Paul Smith [Wed, 1 Apr 2020 05:58:33 +0000 (01:58 -0400)]
tests: Simplify customization of %ENV
Rather than having an %extraENV that is added to the default %ENV
and resetting %ENV _before_ each test, allow the test setup to
modify %ENV directly as needed then reset %ENV _after_ each test.
* tests/test_driver.pl: Remove unused %extraENV.
(resetENV): Don't add in %extraENV.
(_run_command): Reset after we run the command rather than before.
* tests/scripts/features/export: Convert %extraENV to %ENV
* tests/scripts/features/jobserver: Ditto
* tests/scripts/features/parallelism: Ditto
* tests/scripts/features/targetvars: Ditto
* tests/scripts/functions/eval: Ditto
* tests/scripts/functions/foreach: Ditto
* tests/scripts/functions/origin: Ditto
* tests/scripts/misc/general4: Ditto
* tests/scripts/options/dash-e: Ditto
* tests/scripts/targets/POSIX: Ditto
* tests/scripts/variables/GNUMAKEFLAGS: Ditto
* tests/scripts/variables/SHELL: Ditto
Paul Smith [Tue, 31 Mar 2020 02:01:09 +0000 (22:01 -0400)]
[SV 57896] Change directories before checking jobserver auth
We want to process -C options as early as possible, before we might
write informational messages, so that Entering/Leaving messages have
the correct directory.
* src/main.c (main): Move code dealing with changing directories
before parsing of the jobserver auth flag.
* tests/scripts/features/jobserver: Test the order of enter/leave.
Paul Smith [Mon, 30 Mar 2020 18:38:47 +0000 (14:38 -0400)]
[SV 57967] Only set APPEND mode for regular files
APPEND is a permanent mode shared by all users of a file. If we
set it on a tty, pipe, etc. it will stay in effect even after make
exits, which can cause problems.
Patch provided by 0xef967c36@gmail.com
* src/output.c (set_append_mode): Check for a regular file.
Paul Smith [Mon, 30 Mar 2020 18:07:10 +0000 (14:07 -0400)]
Obey order of multiple print/no-print directory options
Previously if --no-print-directory was seen anywhere even once
(environment, command line, etc.) it would always take precedence
over any --print-directory option. Change this so that the last
seen option (which will be the command line, if present there) takes
precedence.
* NEWS: Mark this change in behavior.
* src/makeint.h (print_directory): A new variable to control printing.
* src/output.c (output_dump): Use the new variable.
(output_start): Ditto.
* src/main.c: Add a new variable print_directory. Use -1 for
print_directory_flag so we know of the option was seen or not. Add a
new default_print_directory_flag set to -1 to keep options from being
added.
(switches): Use flag_off for --no-print-directory, rather than a
separate inhibit_print_directory_flag.
(main): If print_directory_flag was set by the user, use that for
print_directory. If not, compute the print_directory value based on
-s, -C, and sub-makes as before.
* tests/scripts/variables/GNUMAKEFLAGS: -w is not added automatically
* tests/scripts/options/print-directory: Add tests for overriding
print-directory options.
Add support for building with Tiny C for MS-Windows
* src/config.h.W32.template (HAVE_DIRECT_H, HAVE_STRCASECMP)
(HAVE_STRNCASECMP, HAVE_UMASK): Add __TINYC__ to MinGW condition.
(BATCH_MODE_ONLY_SHELL): Make this the default for Tiny C.
* build_w32.bat: Support building with Tiny C's tcc compiler.
Paul Smith [Thu, 16 Jan 2020 18:49:07 +0000 (13:49 -0500)]
[SV 40657] Reinstate old behavior for suffix rules with prereqs
POSIX says that suffix rules cannot have prerequisites, but after
making this change we observed a number of makefiles "in the wild"
that were relying on this behavior and failed.
For .POSIX: makefiles, obey POSIX. Otherwise preserve the old
behavior. However, generate a warning so users know this is a
problem. In a future version we will change all behavior to be
POSIX-conforming.
* NEWS: describe the change
* src/rule.c (convert_to_pattern): If posix_pedantic don't make a
pattern rule if prereqs exist. Otherwise show a warning.
* tests/scripts/features/suffixrules: Add tests for the new behavior
including .POSIX vs. non-.POSIX.
Paul Smith [Fri, 3 Jan 2020 23:19:33 +0000 (18:19 -0500)]
Enable compilation with C90 compilers
* configure.ac: Try compiling Guile headers: they don't work with C90.
* maintMakefile: Simplify config checks via target-specific variables.
* src/makeint.h: Use ATTRIBUTE rather than defining __attribute__,
as that causes compile issues with system headers.
(ENUM_BITFIELD): Don't use enum bitfields in ANSI mode.
* src/main.c: Use ATTRIBUTE instead of __attribute__.
* src/job.h: Ditto.
* src/file.c: Don't define variables inside for loops.
* src/rule.c: Ditto.
* src/dep.h (SI): Only use static inline in non-ANSI mode.
Paul Smith [Thu, 2 Jan 2020 10:08:06 +0000 (05:08 -0500)]
Support the .EXTRA_PREREQS special variable
Initial implementation by Christof Warlich <cwarlich@gmx.de>
* NEWS: Announce the new feature.
* doc/make.texi (Other Special Variables): Document .EXTRA_PREREQS.
* src/dep.h (struct dep): New flag to note extra prereq deps.
* src/filedef.h (expand_extra_prereqs): Declare a function to expand
the value of .EXTRA_PREREQS.
* src/file.c (expand_extra_prereqs): Given a struct variable lookup
of .EXTRA_PREREQS, convert it into a list of deps and for each one
make sure it has a struct file and has the new flag set.
(snap_file): A new function invoked by hash_map that will perform
per-file operations: set up second expansion, intermediate, and also
.EXTRA_PREREQS. Manage circular dependencies by ignoring them.
(snap_deps): Defer per-file operations until the end. Look up the
global .EXTRA_PREREQS and pass it along to snap_file for each file.
* src/implicit.c (struct patdeps): Remember the extra prereqs flag.
(pattern_search): Transfer extra prereqs flag settings into the
matched pattern rule.
* src/rule.h (snap_implicit_rules): Rename count_implicit_rules to
snap_implicit_rules since we now do more than count.
* src/rule.c (snap_implicit_rules): As we walk through all the pattern
rules, add in any global .EXTRA_PREREQS to the dep list. Ensure we
take them into account for the max number of prereqs and name length.
* src/main.c (main): Add extra-prereqs to .FEATURES.
Call the renamed snap_implicit_rules.
* tests/scripts/variables/EXTRA_PREREQS: Add tests.
Paul Smith [Fri, 27 Dec 2019 06:27:09 +0000 (01:27 -0500)]
[SV 57022] Avoid posix_spawn which fails asynchronously
Avoid using posix_spawn implementations that fail asynchronously when
the spawned program can't be invoked: this means instead of getting
an error such as "No such file or directory" we get just "Exit 127".
Original implementation of the configure.ac macro provided by
Martin Dorey <martin.dorey@hds.com>
Original implementation of the regression tests provided by
Dmitry Goncharov <dgoncharov@users.sf.net>
* configure.ac: Test whether posix_spawn fails asynchronously. In a
cross-compilation environment, assume that it does not. If we detect
that it does, fall back to fork/exec.
* tests/scripts/features/exec: Add regression tests for different
shebang invocation methods.
Paul Smith [Thu, 26 Dec 2019 23:39:56 +0000 (18:39 -0500)]
[SV 56655] Allow pattern expansion to contain spaces
* src/dep.h: Add a new flag PARSEFS_ONEWORD
* src/read.c (parse_file_seq): If PARSEFS_ONEWORD is given, treat the
entire incoming string as a single pattern.
* src/implicit.c (pattern_search): Pass PARSEFS_ONEWORD when parsing
patterns for wildcards.
* tests/scripts/features/patternrules: Add a new test.
Jouke Witteveen [Sat, 26 Oct 2019 10:24:15 +0000 (12:24 +0200)]
[SV 54161] Fix second expansion of $* for paths
If the stem matches a path containing a directory not just a
filename, make sure the second expansion of $* in the
prerequisites matches $* in the recipe. This requires using
$(*F) when replacing % in the first expansion to preserve the
simple filename.
* src/implicit.c (pattern_search): If lastslash is set prepend
the directory onto the stem. Then use $(*F) when expanding %.
* tests/scripts/features/se_implicit: Add a test case
Jouke Witteveen [Sat, 26 Oct 2019 10:24:29 +0000 (12:24 +0200)]
* src/implicit.c (pattern_search): Set lastslash correctly
If filename contained multiple slashes lastslash is wrongly set to 0.
* configure.ac: Check for the GNU memrchr() extension function.
* src/misc.c (memrchr): Supply memrchr() if not available.
Paul Smith [Tue, 17 Dec 2019 17:29:44 +0000 (12:29 -0500)]
* src/job.c (load_too_high): Disable Linux /proc/loadavg for now
This new feature has a problem: if you provide a load which is
larger than the number of cores then it will always run every job.
Before we can enable it we need to at the least learn how to clamp
this value to the number of cores. To experiment with it, set
PROC_FD_INIT to -2 in job.c to re-enable the feature.
Paul Smith [Mon, 16 Dec 2019 21:18:58 +0000 (16:18 -0500)]
* tests/run_make_tests.pl (subst_make_string): Force use of /
On Windows the path to the helper tool will contain '\': this will
fail if recipes are run with a POSIX shell. Convert '\' to '/'
on Windows. While here, escape any spaces in the path as well.
Paul Smith [Sat, 12 Oct 2019 20:22:01 +0000 (16:22 -0400)]
Rename jhelp.pl to thelp.pl and make it a generic test helper.
* tests/thelp.pl: Rename from tests/jhelp.pl.
(op): Use names instead of options for the operations.
(op): Add new operations for sleep, mkdir, and rm.
(op): Enhance wait to time out
* tests/run_make_tests.pl: Add a new #HELPER# replacement
(subst_make_string): Use fully-qualified path to thelp.pl
* tests/scripts/features/parallelism: Update to use thelp.pl
and the new named operations. Use thelp.pl sleep instead of
system-specific sleep commands.
* tests/scripts/features/output-sync: Update to use thelp.pl
instead of complex shell scripts.
* Makefile.am: Distribute tests/thelp.pl instead of tests/jhelp.pl