From: Paul Smith Date: Mon, 29 Aug 2022 00:15:35 +0000 (-0400) Subject: Rework output sync to lock a temp file on POSIX X-Git-Tag: 4.3.90~25 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=4da2055a10bd21b4e34f1b650484f0636d3b3ca2;p=thirdparty%2Fmake.git Rework output sync to lock a temp file on POSIX Some POSIX systems do not allow locks to be taken on non-files, such as pipes. This is a problem since very often make is invoked with its stdout redirected to a pipe. Also, if stdout is redirected to a file that already has a lock on it for some other reason (perhaps a shared file such as /dev/null) it can cause a hang. This means our previous method of locking stdout, although it had some nice advantages, is not portable enough. Instead, use a temporary file and take the lock on that. We pass the name of the file to child make processes. On Windows we continue to use a shared mutex for output sync. Remove POSIX emulation functions like fcntl from Windows; instead follow the lead of the jobserver and create an interface in os.h for output sync, and move the OS-specific content to posixos.c and w32os.c. * NEWS: Add a note. * src/makeint.h (ALL_SET): Check that all bits are set. * src/os.h: Add bits for checking the state of stdin/stdout/stderr. Add prototypes for OS-specific output sync methods. * src/posixos.c (check_io_state): Determine the status of stdin, stdout, stderr an return a suite of bits describing them. (osync_enabled): If the global variable holding the FD of the lock file (osync_handle) is valid return true. (osync_setup): Create a temporary file and remember its name in a global variable (osync_tmpfile), and set osync_handle. (osync_get_mutex): If output sync is enabled, return the filename of the lock file prefixed with "fnm:" to denote a filename. (osync_parse_mutex): If the provided filename has the wrong format disable output sync. Else open the lock file and set osync_handle. (osync_clear): Close osync_handle. If we're the parent make, then also unlink the temporary file. (osync_acquire): Take a lock on the osync_handle descriptor. (osync_release): Release the lock on the osync_handle descriptor. (fd_set_append): Add APPEND mode to a file descriptor. * src/w32/w32os.c: Perform the same actions as posixos.c, copying the details from src/w32/compat/posixfcn.c. Use a mutex rather than locking a temporary file. * src/output.h: Remove all the OS-specific content. * src/output.c: Remove all the OS-specific content. (set_append_mode): Remove and replace with fd_set_append(). (sync_init): Remove and replace with check_io_state(). (acquire_semaphore): Remove and replace with osync_acquire(). (release_semaphore): Remove and replace with osync_release(). (setup_tmpfile): If the IO state is not obtained, get it. If stdout and/or stderr are valid, set up a tempfile to capture them. (output_init): Set io_state if not set already, and check it when deciding whether to close stdout on exit. * src/main.c (main): If we're syncing, set up the mutex using the new osync_setup() / osync_parse_mutex() methods. (prepare_mutex_handl_string): Replace with osync_parse_mutex(). (die): Call osync_clear(). * src/w32/compat/posixfcn.c: Remove implementations of fcntl(), record_sync_mutex(), create_mutex(), and same_stream(). --- diff --git a/NEWS b/NEWS index 5ae1ab6e..16c67059 100644 --- a/NEWS +++ b/NEWS @@ -87,6 +87,13 @@ https://sv.gnu.org/bugs/index.php?group=make&report_id=111&fix_release_id=109&se top-level invocation of GNU make, or via MAKEFLAGS or GNUMAKEFLAGS. To detect this change search for 'jobserver-fifo' in the .FEATURES variable. +* Some POSIX systems (*BSD) do not allow locks to be taken on pipes, which + caused the output sync feature to not work properly there. Also multiple + invocations of make redirecting to the same output file (e.g., /dev/null) + would cause hangs. Instead of locking stdout (which does have some useful + performance characteristics, but is not portable) create a temporary file + and lock that. Windows continues to use a mutex as before. + * GNU make has sometimes chosen unexpected, and sub-optimal, chains of implicit rules due to the definition of "ought to exist" in the implicit rule search algorithm, which considered any prerequisite mentioned in the diff --git a/src/job.c b/src/job.c index c1e53de7..d12a9138 100644 --- a/src/job.c +++ b/src/job.c @@ -123,6 +123,10 @@ static void vmsWaitForChildren (int *); # include #endif +#if defined (HAVE_FCNTL_H) +# include +#endif + #if defined (HAVE_SYS_WAIT_H) || defined (HAVE_UNION_WAIT) # include #endif @@ -1022,12 +1026,10 @@ reap_children (int block, int err) } else { -#ifndef NO_OUTPUT_SYNC /* If we're sync'ing per line, write the previous line's output before starting the next one. */ if (output_sync == OUTPUT_SYNC_LINE) output_dump (&c->output); -#endif /* Check again whether to start remotely. Whether or not we want to changes over time. Also, start_remote_job may need state set up @@ -1058,10 +1060,8 @@ reap_children (int block, int err) /* When we get here, all the commands for c->file are finished. */ -#ifndef NO_OUTPUT_SYNC /* Synchronize any remaining parallel output. */ output_dump (&c->output); -#endif /* At this point c->file->update_status is success or failed. But c->file->command_state is still cs_running if all the commands @@ -1354,12 +1354,10 @@ start_job_command (struct child *child) OUTPUT_SET (&child->output); -#ifndef NO_OUTPUT_SYNC if (! child->output.syncout) /* We don't want to sync this command: to avoid misordered output ensure any already-synced content is written. */ output_dump (&child->output); -#endif /* Print the command if appropriate. */ if (just_print_flag || ISDB (DB_PRINT) @@ -1560,8 +1558,8 @@ start_job_command (struct child *child) { HANDLE hPID; char* arg0; - int outfd = FD_STDOUT; - int errfd = FD_STDERR; + int outfd = -1; + int errfd = -1; /* make UNC paths safe for CreateProcess -- backslash format */ arg0 = argv[0]; @@ -1573,7 +1571,6 @@ start_job_command (struct child *child) /* make sure CreateProcess() has Path it needs */ sync_Path_environment (); -#ifndef NO_OUTPUT_SYNC /* Divert child output if output_sync in use. */ if (child->output.syncout) { @@ -1582,9 +1579,7 @@ start_job_command (struct child *child) if (child->output.err >= 0) errfd = child->output.err; } -#else - outfd = errfd = -1; -#endif + hPID = process_easy (argv, child->environment, outfd, errfd); if (hPID != INVALID_HANDLE_VALUE) diff --git a/src/main.c b/src/main.c index 8cffd46b..799b2e91 100644 --- a/src/main.c +++ b/src/main.c @@ -241,8 +241,7 @@ static char *jobserver_style = NULL; static char *shuffle_mode = NULL; -/* Handle for the mutex used on Windows to synchronize output of our - children under -O. */ +/* Handle for the mutex to synchronize output of our children under -O. */ static char *sync_mutex = NULL; @@ -830,33 +829,12 @@ decode_output_sync_flags (void) } if (sync_mutex) - RECORD_SYNC_MUTEX (sync_mutex); + osync_parse_mutex (sync_mutex); #endif } #ifdef WINDOWS32 -#ifndef NO_OUTPUT_SYNC - -/* This is called from start_job_command when it detects that - output_sync option is in effect. The handle to the synchronization - mutex is passed, as a string, to sub-makes via the --sync-mutex - command-line argument. */ -void -prepare_mutex_handle_string (sync_handle_t handle) -{ - if (!sync_mutex) - { - /* Prepare the mutex handle string for our children. */ - /* 2 hex digits per byte + 2 characters for "0x" + null. */ - sync_mutex = xmalloc ((2 * sizeof (sync_handle_t)) + 2 + 1); - sprintf (sync_mutex, "0x%Ix", handle); - define_makeflags (1, 0); - } -} - -#endif /* NO_OUTPUT_SYNC */ - /* * HANDLE runtime exceptions by avoiding a requestor on the GUI. Capture * exception and print it to stderr instead. @@ -1353,9 +1331,9 @@ main (int argc, char **argv, char **envp) #endif #ifdef MAKE_JOBSERVER " jobserver" -#ifdef HAVE_MKFIFO +# ifdef HAVE_MKFIFO " jobserver-fifo" -#endif +# endif #endif #ifndef NO_OUTPUT_SYNC " output-sync" @@ -2127,6 +2105,22 @@ main (int argc, char **argv, char **envp) output_sync = OUTPUT_SYNC_NONE; } + if (syncing) + { + /* If there a mutex we're the child, else we're the origin. */ + if (!sync_mutex) + { + osync_setup (); + sync_mutex = osync_get_mutex (); + } + else if (!osync_parse_mutex (sync_mutex)) + { + osync_clear (); + free (sync_mutex); + sync_mutex = NULL; + } + } + #ifndef MAKE_SYMLINKS if (check_symlink_flag) { @@ -3687,6 +3681,8 @@ die (int status) output_close (NULL); + osync_clear (); + /* Try to move back to the original directory. This is essential on MS-DOS (where there is really only one process), and on Unix it puts core files in the original directory instead of the -C diff --git a/src/makeint.h b/src/makeint.h index 94fee7a9..dd45218d 100644 --- a/src/makeint.h +++ b/src/makeint.h @@ -393,8 +393,10 @@ extern int unixy_shell; # define WIN32_LEAN_AND_MEAN #endif /* WINDOWS32 */ +/* ALL_SET() evaluates the second argument twice. */ #define ANY_SET(_v,_m) (((_v)&(_m)) != 0) #define NONE_SET(_v,_m) (! ANY_SET ((_v),(_m))) +#define ALL_SET(_v,_m) (((_v)&(_m)) == (_m)) #define MAP_NUL 0x0001 #define MAP_BLANK 0x0002 /* space, TAB */ diff --git a/src/misc.c b/src/misc.c index 4a02ae7c..aa1fbde0 100644 --- a/src/misc.c +++ b/src/misc.c @@ -515,13 +515,9 @@ get_tmptemplate () #ifdef VMS # define DEFAULT_TMPFILE "sys$scratch:gnv$make_cmdXXXXXX.com" -#else -# define DEFAULT_TMPFILE "GmXXXXXX" -#endif - -#ifdef VMS # define DEFAULT_TMPDIR "/sys$scratch/" #else +# define DEFAULT_TMPFILE "GmXXXXXX" # ifdef P_tmpdir # define DEFAULT_TMPDIR P_tmpdir # else diff --git a/src/os.h b/src/os.h index a92d5e22..81212bc0 100644 --- a/src/os.h +++ b/src/os.h @@ -15,6 +15,30 @@ You should have received a copy of the GNU General Public License along with this program. If not, see . */ +#define IO_UNKNOWN 0x0001 +#define IO_COMBINED_OUTERR 0x0002 +#define IO_STDIN_OK 0x0004 +#define IO_STDOUT_OK 0x0008 +#define IO_STDERR_OK 0x0010 + +#if defined(VMS) || defined(_AMIGA) || defined(__MSDOS__) +# define check_io_state() (IO_STDIN_OK|IO_STDOUT_OK|IO_STDERR_OK) +# define fd_inherit(_i) (0) +# define fd_noinherit(_i) (0) +# define fd_set_append(_i) (void)(0) +#else + +/* Determine the state of stdin/stdout/stderr. */ +unsigned int check_io_state (); + +/* Set a file descriptor to close/not close in a subprocess. */ +void fd_inherit (int); +void fd_noinherit (int); + +/* If the file descriptor is for a file put it into append mode. */ +void fd_set_append (int); +#endif + /* Return a file descriptor for a new anonymous temp file, or -1. */ #if defined(WINDOWS32) int os_anontmp (); @@ -29,7 +53,7 @@ int os_anontmp (); /* Returns 1 if the jobserver is enabled, else 0. */ unsigned int jobserver_enabled (); -/* Called in the master instance to set up the jobserver initially. */ +/* Called in the parent make to set up the jobserver initially. */ unsigned int jobserver_setup (int job_slots, const char *style); /* Called in a child instance to connect to the jobserver. @@ -77,6 +101,7 @@ unsigned int jobserver_acquire (int timeout); #define jobserver_setup(_slots, _style) (0) #define jobserver_parse_auth(_auth) (0) #define jobserver_get_auth() (NULL) +#define jobserver_get_invalid_auth() (NULL) #define jobserver_clear() (void)(0) #define jobserver_release(_fatal) (void)(0) #define jobserver_acquire_all() (0) @@ -86,7 +111,45 @@ unsigned int jobserver_acquire (int timeout); #define jobserver_pre_acquire() (void)(0) #define jobserver_acquire(_tmout) (0) -#endif +#endif /* MAKE_JOBSERVER */ + +#ifndef NO_OUTPUT_SYNC + +/* Returns 1 if output sync is enabled, else 0. */ +unsigned int osync_enabled (); + +/* Called in the parent make to set up output sync initially. */ +void osync_setup (); + +/* Returns an allocated buffer containing output sync info to pass to child + instances, or NULL if not needed. */ +char *osync_get_mutex (); + +/* Called in a child instance to obtain info on the output sync mutex. + Return 1 if we got a valid mutex, else 0. */ +unsigned int osync_parse_mutex (const char *mutex); + +/* Clean up this instance's output sync facilities. */ +void osync_clear (); + +/* Acquire the output sync lock. This will wait until available. + Returns 0 if there was an error getting the semaphore. */ +unsigned int osync_acquire (); + +/* Release the output sync lock. */ +void osync_release (); + +#else + +#define osync_enabled() (0) +#define osync_setup() (void)(0) +#define osync_get_mutex() (0) +#define osync_parse_mutex(_s) (0) +#define osync_clear() (void)(0) +#define osync_acquire() (1) +#define osync_release() (void)(0) + +#endif /* NO_OUTPUT_SYNC */ /* Create a "bad" file descriptor for stdin when parallel jobs are run. */ #if defined(VMS) || defined(WINDOWS32) || defined(_AMIGA) || defined(__MSDOS__) @@ -94,12 +157,3 @@ unsigned int jobserver_acquire (int timeout); #else int get_bad_stdin (); #endif - -/* Set a file descriptor to close/not close in a subprocess. */ -#if defined(VMS) || defined(_AMIGA) || defined(__MSDOS__) -# define fd_inherit(_i) 0 -# define fd_noinherit(_i) 0 -#else -void fd_inherit (int); -void fd_noinherit (int); -#endif diff --git a/src/output.c b/src/output.c index fda783bb..91a1dc31 100644 --- a/src/output.c +++ b/src/output.c @@ -47,12 +47,6 @@ unsigned int stdio_traced = 0; #define OUTPUT_ISSET(_out) ((_out)->out >= 0 || (_out)->err >= 0) -#ifdef HAVE_FCNTL_H -# define STREAM_OK(_s) ((fcntl (fileno (_s), F_GETFD) != -1) || (errno != EBADF)) -#else -# define STREAM_OK(_s) 1 -#endif - /* Write a string to the current STDOUT or STDERR. */ static void _outputs (struct output *out, int is_err, const char *msg) @@ -143,77 +137,10 @@ log_working_directory (int entering) return 1; } - -/* Set a file descriptor referring to a regular file - to be in O_APPEND mode. If it fails, just ignore it. */ - -static void -set_append_mode (int fd) -{ -#if defined(F_GETFL) && defined(F_SETFL) && defined(O_APPEND) - struct stat stbuf; - int flags; - if (fstat (fd, &stbuf) != 0 || !S_ISREG (stbuf.st_mode)) - return; - flags = fcntl (fd, F_GETFL, 0); - if (flags >= 0) - { - int r; - EINTRLOOP(r, fcntl (fd, F_SETFL, flags | O_APPEND)); - } -#endif -} #ifndef NO_OUTPUT_SYNC -/* Semaphore for use in -j mode with output_sync. */ -static sync_handle_t sync_handle = -1; - -#define FD_NOT_EMPTY(_f) ((_f) != OUTPUT_NONE && lseek ((_f), 0, SEEK_END) > 0) - -/* Set up the sync handle. Disables output_sync on error. */ -static int -sync_init (void) -{ - int combined_output = 0; - -#ifdef WINDOWS32 - if ((!STREAM_OK (stdout) && !STREAM_OK (stderr)) - || (sync_handle = create_mutex ()) == -1) - { - perror_with_name ("output-sync suppressed: ", "stderr"); - output_sync = 0; - } - else - { - combined_output = same_stream (stdout, stderr); - prepare_mutex_handle_string (sync_handle); - } - -#else - if (STREAM_OK (stdout)) - { - struct stat stbuf_o, stbuf_e; - - sync_handle = fileno (stdout); - combined_output = (fstat (fileno (stdout), &stbuf_o) == 0 - && fstat (fileno (stderr), &stbuf_e) == 0 - && stbuf_o.st_dev == stbuf_e.st_dev - && stbuf_o.st_ino == stbuf_e.st_ino); - } - else if (STREAM_OK (stderr)) - sync_handle = fileno (stderr); - else - { - perror_with_name ("output-sync suppressed: ", "stderr"); - output_sync = 0; - } -#endif - - return combined_output; -} - /* Support routine for output_sync() */ static void pump_from_tmp (int from, FILE *to) @@ -254,39 +181,13 @@ pump_from_tmp (int from, FILE *to) #endif } -/* Obtain the lock for writing output. */ -static void * -acquire_semaphore (void) -{ - static struct flock fl; - - fl.l_type = F_WRLCK; - fl.l_whence = SEEK_SET; - fl.l_start = 0; - fl.l_len = 1; - if (fcntl (sync_handle, F_SETLKW, &fl) != -1) - return &fl; - perror ("fcntl()"); - return NULL; -} - -/* Release the lock for writing output. */ -static void -release_semaphore (void *sem) -{ - struct flock *flp = (struct flock *)sem; - flp->l_type = F_UNLCK; - if (fcntl (sync_handle, F_SETLKW, flp) == -1) - perror ("fcntl()"); -} - -/* Returns a file descriptor to a temporary file. The file is automatically - closed/deleted on exit. Don't use a FILE* stream. */ +/* Returns a file descriptor to a temporary file, that will be automatically + deleted on exit. */ int output_tmpfd (void) { int fd = get_tmpfd (NULL); - set_append_mode (fd); + fd_set_append (fd); return fd; } @@ -297,13 +198,16 @@ output_tmpfd (void) static void setup_tmpfile (struct output *out) { - /* Is make's stdout going to the same place as stderr? */ - static int combined_output = -1; + unsigned int io_state = check_io_state (); - if (combined_output < 0) - combined_output = sync_init (); + if (NONE_SET (io_state, IO_STDOUT_OK|IO_STDERR_OK)) + { + /* This is probably useless since stdout/stderr aren't working. */ + perror_with_name ("output-sync suppressed: ", "stderr"); + goto error; + } - if (STREAM_OK (stdout)) + if (ANY_SET (io_state, IO_STDOUT_OK)) { int fd = output_tmpfd (); if (fd < 0) @@ -312,9 +216,9 @@ setup_tmpfile (struct output *out) out->out = fd; } - if (STREAM_OK (stderr)) + if (ANY_SET (io_state, IO_STDERR_OK)) { - if (out->out != OUTPUT_NONE && combined_output) + if (out->out != OUTPUT_NONE && ANY_SET (io_state, IO_COMBINED_OUTERR)) out->err = out->out; else { @@ -332,6 +236,7 @@ setup_tmpfile (struct output *out) error: output_close (out); output_sync = OUTPUT_SYNC_NONE; + osync_clear (); } /* Synchronize the output of jobs in -j mode to keep the results of @@ -342,6 +247,8 @@ setup_tmpfile (struct output *out) void output_dump (struct output *out) { +#define FD_NOT_EMPTY(_f) ((_f) != OUTPUT_NONE && lseek ((_f), 0, SEEK_END) > 0) + int outfd_not_empty = FD_NOT_EMPTY (out->out); int errfd_not_empty = FD_NOT_EMPTY (out->err); @@ -352,7 +259,12 @@ output_dump (struct output *out) /* Try to acquire the semaphore. If it fails, dump the output unsynchronized; still better than silently discarding it. We want to keep this lock for as little time as possible. */ - void *sem = acquire_semaphore (); + if (!osync_acquire ()) + { + O (error, NILF, + _("warning: Cannot acquire output lock, disabling output sync.")); + osync_clear (); + } /* Log the working directory for this dump. */ if (print_directory && output_sync != OUTPUT_SYNC_RECURSE) @@ -367,8 +279,7 @@ output_dump (struct output *out) log_working_directory (0); /* Exit the critical section. */ - if (sem) - release_semaphore (sem); + osync_release (); /* Truncate and reset the output, in case we use it again. */ if (out->out != OUTPUT_NONE) @@ -455,11 +366,11 @@ output_init (struct output *out) /* Force stdout/stderr into append mode. This ensures parallel jobs won't lose output due to overlapping writes. */ - set_append_mode (fileno (stdout)); - set_append_mode (fileno (stderr)); + fd_set_append (fileno (stdout)); + fd_set_append (fileno (stderr)); #ifdef HAVE_ATEXIT - if (STREAM_OK (stdout)) + if (ANY_SET (check_io_state (), IO_STDOUT_OK)) atexit (close_stdout); #endif } diff --git a/src/output.h b/src/output.h index e0a642d4..d9b1512c 100644 --- a/src/output.h +++ b/src/output.h @@ -50,66 +50,9 @@ void output_start (void); /* Show a message on stdout or stderr. Will start the output if needed. */ void outputs (int is_err, const char *msg); -#if defined(HAVE_FCNTL_H) -# include -#elif defined(HAVE_SYS_FILE_H) -# include -#endif - -#ifdef NO_OUTPUT_SYNC -# define RECORD_SYNC_MUTEX(m) \ - O (error, NILF, \ - _("-O[TYPE] (--output-sync[=TYPE]) is not configured for this build.")); +#if defined(NO_OUTPUT_SYNC) +# define output_dump(_o) (void)(0) #else -int output_tmpfd (void); /* Dump any child output content to stdout, and reset it. */ void output_dump (struct output *out); - -# ifdef WINDOWS32 -/* For emulations in w32/compat/posixfcn.c. */ -# ifndef F_GETFD -# define F_GETFD 1 -# endif -# ifndef F_SETLKW -# define F_SETLKW 2 -# endif -/* Implementation note: None of the values of l_type below can be zero - -- they are compared with a static instance of the struct, so zero - means unknown/invalid, see w32/compat/posixfcn.c. */ -# ifndef F_WRLCK -# define F_WRLCK 1 -# endif -# ifndef F_UNLCK -# define F_UNLCK 2 -# endif -struct flock - { - short l_type; - short l_whence; - off_t l_start; - off_t l_len; - pid_t l_pid; - }; - -/* This type is actually a HANDLE, but we want to avoid including - windows.h as much as possible. */ -typedef intptr_t sync_handle_t; - -/* Public functions emulated/provided in posixfcn.c. */ -# if !defined(GNULIB_defined_rpl_fcntl) && !defined(GNULIB_defined_fcntl) -int fcntl (intptr_t fd, int cmd, ...); -# endif -intptr_t create_mutex (void); -int same_stream (FILE *f1, FILE *f2); - -# define RECORD_SYNC_MUTEX(m) record_sync_mutex(m) -void record_sync_mutex (const char *str); -void prepare_mutex_handle_string (intptr_t hdl); -# else /* !WINDOWS32 */ - -typedef int sync_handle_t; /* file descriptor */ - -# define RECORD_SYNC_MUTEX(m) (void)(m) - -# endif -#endif /* !NO_OUTPUT_SYNC */ +#endif diff --git a/src/posixos.c b/src/posixos.c index 3ab487c1..3538a6b8 100644 --- a/src/posixos.c +++ b/src/posixos.c @@ -37,7 +37,39 @@ this program. If not, see . */ #include "job.h" #include "os.h" -#ifdef MAKE_JOBSERVER +#define STREAM_OK(_s) ((fcntl (fileno (_s), F_GETFD) != -1) || (errno != EBADF)) + +unsigned int +check_io_state () +{ + static unsigned int state = IO_UNKNOWN; + + /* We only need to compute this once per process. */ + if (state != IO_UNKNOWN) + return state; + + if (STREAM_OK (stdin)) + state |= IO_STDIN_OK; + if (STREAM_OK (stdout)) + state |= IO_STDOUT_OK; + if (STREAM_OK (stderr)) + state |= IO_STDERR_OK; + + if (ALL_SET (state, IO_STDOUT_OK|IO_STDERR_OK)) + { + struct stat stbuf_o, stbuf_e; + + if (fstat (fileno (stdout), &stbuf_o) == 0 + && fstat (fileno (stderr), &stbuf_e) == 0 + && stbuf_o.st_dev == stbuf_e.st_dev + && stbuf_o.st_ino == stbuf_e.st_ino) + state |= IO_COMBINED_OUTERR; + } + + return state; +} + +#if defined(MAKE_JOBSERVER) #define FIFO_PREFIX "fifo:" @@ -454,7 +486,7 @@ jobserver_acquire (int timeout) pfatal_with_name (_("read jobs pipe")); } - /* read() should never return 0: only the master make can reap all the + /* read() should never return 0: only the parent make can reap all the tokens and close the write side...?? */ return r > 0; } @@ -577,6 +609,119 @@ jobserver_acquire (int timeout) #endif /* MAKE_JOBSERVER */ +#if !defined(NO_OUTPUT_SYNC) + +#define MUTEX_PREFIX "fnm:" + +static int osync_handle = -1; + +static char *osync_tmpfile = NULL; + +static unsigned int sync_parent = 0; + +unsigned int +osync_enabled () +{ + return osync_handle >= 0; +} + +void +osync_setup () +{ + osync_handle = get_tmpfd (&osync_tmpfile); + if (osync_handle >= 0) + sync_parent = 1; +} + +char * +osync_get_mutex () +{ + char *mutex = NULL; + + if (osync_enabled ()) + { + /* Prepare the mutex handle string for our children. */ + mutex = xmalloc (strlen (osync_tmpfile) + CSTRLEN (MUTEX_PREFIX) + 1); + sprintf (mutex, MUTEX_PREFIX "%s", osync_tmpfile); + } + + return mutex; +} + +unsigned int +osync_parse_mutex (const char *mutex) +{ + if (strncmp (mutex, MUTEX_PREFIX, CSTRLEN (MUTEX_PREFIX)) != 0) + { + OS (error, NILF, _("invalid --sync-mutex string '%s'"), mutex); + return 0; + } + + osync_tmpfile = xstrdup (mutex + CSTRLEN (MUTEX_PREFIX)); + + EINTRLOOP (osync_handle, open (osync_tmpfile, O_WRONLY)); + if (osync_handle < 0) + OSS (fatal, NILF, _("cannot open output sync mutex %s: %s"), + osync_tmpfile, strerror (errno)); + + return 1; +} + +void +osync_clear () +{ + if (osync_handle) + { + close (osync_handle); + osync_handle = -1; + } + + if (sync_parent && osync_tmpfile) + { + unlink (osync_tmpfile); + osync_tmpfile = NULL; + } +} + +unsigned int +osync_acquire () +{ + if (osync_enabled()) + { + struct flock fl; + + fl.l_type = F_WRLCK; + fl.l_whence = SEEK_SET; + fl.l_start = 0; + fl.l_len = 1; + if (fcntl (osync_handle, F_SETLKW, &fl) == -1) + { + perror ("fcntl()"); + return 0; + } + } + + return 1; +} + +void +osync_release () +{ + if (osync_enabled()) + { + struct flock fl; + + fl.l_type = F_UNLCK; + fl.l_whence = SEEK_SET; + fl.l_start = 0; + fl.l_len = 1; + if (fcntl (osync_handle, F_SETLKW, &fl) == -1) + perror ("fcntl()"); + } +} + +#endif + /* Create a "bad" file descriptor for stdin when parallel jobs are run. */ int get_bad_stdin () @@ -636,12 +781,33 @@ void fd_noinherit (int fd) { int flags; - EINTRLOOP(flags, fcntl(fd, F_GETFD)); + EINTRLOOP (flags, fcntl(fd, F_GETFD)); if (flags >= 0) { int r; flags |= FD_CLOEXEC; - EINTRLOOP(r, fcntl(fd, F_SETFD, flags)); + EINTRLOOP (r, fcntl(fd, F_SETFD, flags)); } } #endif + +/* Set a file descriptor referring to a regular file to be in O_APPEND mode. + If it fails, just ignore it. */ + +void +fd_set_append (int fd) +{ +#if defined(F_GETFL) && defined(F_SETFL) && defined(O_APPEND) + struct stat stbuf; + int flags; + if (fstat (fd, &stbuf) == 0 && S_ISREG (stbuf.st_mode)) + { + flags = fcntl (fd, F_GETFL, 0); + if (flags >= 0) + { + int r; + EINTRLOOP(r, fcntl (fd, F_SETFL, flags | O_APPEND)); + } + } +#endif +} diff --git a/src/w32/compat/posixfcn.c b/src/w32/compat/posixfcn.c index acdfe289..a7cbf4a5 100644 --- a/src/w32/compat/posixfcn.c +++ b/src/w32/compat/posixfcn.c @@ -27,246 +27,10 @@ this program. If not, see . */ #include "job.h" -#ifndef NO_OUTPUT_SYNC -/* Support for OUTPUT_SYNC and related functionality. */ - -#if !defined(GNULIB_defined_rpl_fcntl) && !defined(GNULIB_defined_fcntl) -/* Emulation of fcntl that supports only F_GETFD and F_SETLKW. */ -int -fcntl (intptr_t fd, int cmd, ...) -{ - va_list ap; - - va_start (ap, cmd); - - switch (cmd) - { - case F_GETFD: - va_end (ap); - /* Could have used GetHandleInformation, but that isn't - supported on Windows 9X. */ - if (_get_osfhandle (fd) == -1) - return -1; - return 0; - case F_SETLKW: - { - void *buf = va_arg (ap, void *); - struct flock *fl = (struct flock *)buf; - HANDLE hmutex = (HANDLE)fd; - static struct flock last_fl; - short last_type = last_fl.l_type; - - va_end (ap); - - if (hmutex == INVALID_HANDLE_VALUE || !hmutex) - return -1; - - last_fl = *fl; - - switch (fl->l_type) - { - - case F_WRLCK: - { - DWORD result; - - if (last_type == F_WRLCK) - { - /* Don't call WaitForSingleObject if we already - own the mutex, because doing so will require - us to call ReleaseMutex an equal number of - times, before the mutex is actually - released. */ - return 0; - } - - result = WaitForSingleObject (hmutex, INFINITE); - switch (result) - { - case WAIT_OBJECT_0: - /* We don't care if the mutex owner crashed or - exited. */ - case WAIT_ABANDONED: - return 0; - case WAIT_FAILED: - case WAIT_TIMEOUT: /* cannot happen, really */ - { - DWORD err = GetLastError (); - - /* Invalidate the last command. */ - memset (&last_fl, 0, sizeof (last_fl)); - - switch (err) - { - case ERROR_INVALID_HANDLE: - case ERROR_INVALID_FUNCTION: - errno = EINVAL; - return -1; - default: - errno = EDEADLOCK; - return -1; - } - } - } - } - case F_UNLCK: - { - /* FIXME: Perhaps we should call ReleaseMutex - repatedly until it errors out, to make sure the - mutext is released even if we somehow managed to - to take ownership multiple times? */ - BOOL status = ReleaseMutex (hmutex); - - if (status) - return 0; - else - { - DWORD err = GetLastError (); - - if (err == ERROR_NOT_OWNER) - errno = EPERM; - else - { - memset (&last_fl, 0, sizeof (last_fl)); - errno = EINVAL; - } - return -1; - } - } - default: - errno = ENOSYS; - return -1; - } - } - default: - errno = ENOSYS; - va_end (ap); - return -1; - } -} -#endif /* GNULIB_defined_fcntl */ - -static intptr_t mutex_handle = -1; - -/* Record in a static variable the mutex handle we were requested to - use. That nameless mutex was created by the top-level Make, and - its handle was passed to us via inheritance. The value of that - handle is passed via the command-line arguments, so that we know - which handle to use. */ -void -record_sync_mutex (const char *str) -{ - char *endp; - intptr_t hmutex = strtol (str, &endp, 16); - - if (*endp == '\0') - mutex_handle = hmutex; - else - { - mutex_handle = -1; - errno = EINVAL; - } -} - -/* Create a new mutex or reuse one created by our parent. */ -intptr_t -create_mutex (void) -{ - SECURITY_ATTRIBUTES secattr; - intptr_t hmutex = -1; - - /* If we have a mutex handle passed from the parent Make, just use - that. */ - if (mutex_handle > 0) - return mutex_handle; - - /* We are the top-level Make, and we want the handle to be inherited - by our child processes. */ - secattr.nLength = sizeof (secattr); - secattr.lpSecurityDescriptor = NULL; /* use default security descriptor */ - secattr.bInheritHandle = TRUE; - - hmutex = (intptr_t)CreateMutex (&secattr, FALSE, NULL); - if (!hmutex) - { - DWORD err = GetLastError (); - - fprintf (stderr, "CreateMutex: error %lu\n", err); - errno = ENOLCK; - hmutex = -1; - } - - mutex_handle = hmutex; - return hmutex; -} - -/* Return non-zero if F1 and F2 are 2 streams representing the same - file or pipe or device. */ -int -same_stream (FILE *f1, FILE *f2) -{ - HANDLE fh1 = (HANDLE)_get_osfhandle (fileno (f1)); - HANDLE fh2 = (HANDLE)_get_osfhandle (fileno (f2)); - - /* Invalid file descriptors get treated as different streams. */ - if (fh1 && fh1 != INVALID_HANDLE_VALUE - && fh2 && fh2 != INVALID_HANDLE_VALUE) - { - if (fh1 == fh2) - return 1; - else - { - DWORD ftyp1 = GetFileType (fh1), ftyp2 = GetFileType (fh2); - - if (ftyp1 != ftyp2 - || ftyp1 == FILE_TYPE_UNKNOWN || ftyp2 == FILE_TYPE_UNKNOWN) - return 0; - else if (ftyp1 == FILE_TYPE_CHAR) - { - /* For character devices, check if they both refer to a - console. This loses if both handles refer to the - null device (FIXME!), but in that case we don't care - in the context of Make. */ - DWORD conmode1, conmode2; - - /* Each process on Windows can have at most 1 console, - so if both handles are for the console device, they - are the same. We also compare the console mode to - distinguish between stdin and stdout/stderr. */ - if (GetConsoleMode (fh1, &conmode1) - && GetConsoleMode (fh2, &conmode2) - && conmode1 == conmode2) - return 1; - } - else - { - /* For disk files and pipes, compare their unique - attributes. */ - BY_HANDLE_FILE_INFORMATION bhfi1, bhfi2; - - /* Pipes get zero in the volume serial number, but do - appear to have meaningful information in file index - attributes. We test file attributes as well, for a - good measure. */ - if (GetFileInformationByHandle (fh1, &bhfi1) - && GetFileInformationByHandle (fh2, &bhfi2)) - return (bhfi1.dwVolumeSerialNumber == bhfi2.dwVolumeSerialNumber - && bhfi1.nFileIndexLow == bhfi2.nFileIndexLow - && bhfi1.nFileIndexHigh == bhfi2.nFileIndexHigh - && bhfi1.dwFileAttributes == bhfi2.dwFileAttributes); - } - } - } - return 0; -} - -#endif /* !NO_OUTPUT_SYNC */ - #if MAKE_LOAD /* Support for dynamic loading of objects. */ - static DWORD last_err; void * diff --git a/src/w32/w32os.c b/src/w32/w32os.c index ef8116a4..528b5b70 100644 --- a/src/w32/w32os.c +++ b/src/w32/w32os.c @@ -22,12 +22,90 @@ this program. If not, see . */ #include #include #include +#include #include "pathstuff.h" #include "sub_proc.h" #include "w32err.h" #include "os.h" #include "debug.h" +unsigned int +check_io_state () +{ + static unsigned int state = IO_UNKNOWN; + + /* We only need to compute this once per process. */ + if (state != IO_UNKNOWN) + return state; + + /* Could have used GetHandleInformation, but that isn't supported + on Windows 9X. */ + HANDLE outfd = (HANDLE)_get_osfhandle (fileno (stdout)); + HANDLE errfd = (HANDLE)_get_osfhandle (fileno (stderr)); + + if ((HANDLE)_get_osfhandle (fileno (stdin)) != INVALID_HANDLE_VALUE) + state |= IO_STDIN_OK; + if (outfd != INVALID_HANDLE_VALUE) + state |= IO_STDOUT_OK; + if (errfd != INVALID_HANDLE_VALUE) + state |= IO_STDERR_OK; + + if (ALL_SET (state, IO_STDOUT_OK|IO_STDERR_OK)) + { + unsigned int combined = 0; + + if (outfd == errfd) + combined = IO_COMBINED_OUTERR; + else + { + DWORD outtype = GetFileType (outfd), errtype = GetFileType (errfd); + + if (outtype == errtype + && outtype != FILE_TYPE_UNKNOWN && errtype != FILE_TYPE_UNKNOWN) + { + if (outtype == FILE_TYPE_CHAR) + { + /* For character devices, check if they both refer to a + console. This loses if both handles refer to the + null device (FIXME!), but in that case we don't care + in the context of Make. */ + DWORD outmode, errmode; + + /* Each process on Windows can have at most 1 console, + so if both handles are for the console device, they + are the same. We also compare the console mode to + distinguish between stdin and stdout/stderr. */ + if (GetConsoleMode (outfd, &outmode) + && GetConsoleMode (errfd, &errmode) + && outmode == errmode) + combined = IO_COMBINED_OUTERR; + } + else + { + /* For disk files and pipes, compare their unique + attributes. */ + BY_HANDLE_FILE_INFORMATION outfi, errfi; + + /* Pipes get zero in the volume serial number, but do + appear to have meaningful information in file index + attributes. We test file attributes as well, for a + good measure. */ + if (GetFileInformationByHandle (outfd, &outfi) + && GetFileInformationByHandle (errfd, &errfi) + && outfi.dwVolumeSerialNumber == errfi.dwVolumeSerialNumber + && outfi.nFileIndexLow == errfi.nFileIndexLow + && outfi.nFileIndexHigh == errfi.nFileIndexHigh + && outfi.dwFileAttributes == errfi.dwFileAttributes) + combined = IO_COMBINED_OUTERR; + } + } + } + state |= combined; + } + + return state; +} + /* A replacement for tmpfile, since the MSVCRT implementation creates the file in the root directory of the current drive, which might not be writable by our user, and also it returns a FILE* and we want a file @@ -123,6 +201,8 @@ os_anontmp () return -1; } +#if defined(MAKE_JOBSERVER) + /* This section provides OS-specific functions to support the jobserver. */ static char jobserver_semaphore_name[MAX_PATH + 1]; @@ -304,6 +384,111 @@ jobserver_acquire (int timeout) return dwEvent == WAIT_OBJECT_0; } +#endif /* MAKE_JOBSERVER */ + +#if !defined(NO_OUTPUT_SYNC) + +#define MUTEX_PREFIX "fnm:" + +/* Since we're using this with CreateMutex, NULL is invalid. */ +static HANDLE osync_handle = NULL; + +unsigned int +osync_enabled () +{ + return osync_handle != NULL; +} + +void +osync_setup () +{ + SECURITY_ATTRIBUTES secattr; + + /* We are the top-level make, and we want the handle to be inherited + by our child processes. */ + secattr.nLength = sizeof (secattr); + secattr.lpSecurityDescriptor = NULL; /* use default security descriptor */ + secattr.bInheritHandle = TRUE; + + osync_handle = CreateMutex (&secattr, FALSE, NULL); + if (!osync_handle) + { + DWORD err = GetLastError (); + fprintf (stderr, "CreateMutex: error %lu\n", err); + errno = ENOLCK; + } +} + +char * +osync_get_mutex () +{ + char *mutex = NULL; + + if (osync_enabled ()) + { + /* Prepare the mutex handle string for our children. + 2 hex digits per byte + 2 characters for "0x" + null. */ + mutex = xmalloc ((2 * sizeof (osync_handle)) + 2 + 1); + sprintf (mutex, "0x%Ix", (unsigned long long)osync_handle); + } + + return mutex; +} + +unsigned int +osync_parse_mutex (const char *mutex) +{ + char *endp; + unsigned long long i; + + errno = 0; + i = strtoull (mutex, &endp, 16); + if (errno != 0) + OSS (fatal, NILF, _("cannot parse output sync mutex %s: %s"), + mutex, strerror (errno)); + if (endp[0] != '\0') + OS (fatal, NILF, _("invalid output sync mutex: %s"), mutex); + + osync_handle = (HANDLE) i; + + return 1; +} + +void +osync_clear () +{ + if (osync_handle) + { + CloseHandle (osync_handle); + osync_handle = NULL; + } +} + +unsigned int +osync_acquire () +{ + if (osync_enabled()) + { + DWORD result = WaitForSingleObject (osync_handle, INFINITE); + if (result == WAIT_FAILED || result == WAIT_TIMEOUT) + return 0; + } + + return 1; +} + +void +osync_release () +{ + if (osync_enabled()) + /* FIXME: Perhaps we should call ReleaseMutex repatedly until it errors + out, to make sure the mutext is released even if we somehow managed to + to take ownership multiple times? */ + ReleaseMutex (osync_handle); +} + +#endif /* NO_OUTPUT_SYNC */ + void fd_inherit(int fd) { @@ -321,3 +506,7 @@ fd_noinherit(int fd) if (fh && fh != INVALID_HANDLE_VALUE) SetHandleInformation(fh, HANDLE_FLAG_INHERIT, 0); } + +void +fd_set_append (int fd) +{} diff --git a/tests/scripts/functions/shell b/tests/scripts/functions/shell index 0f33332a..1ca1e9ab 100644 --- a/tests/scripts/functions/shell +++ b/tests/scripts/functions/shell @@ -143,7 +143,8 @@ all: ; @echo '$(.SHELLSTATUS): $(out)' # If we're using pipes for jobserver, then we will close them and not # allow them to be available to sub-makes invoked via $(shell ...) - run_make_test(q! + if (exists $FEATURES{'jobserver'}) { + run_make_test(q! ifeq ($(ELT),) default:; @$(MAKE) -f #MAKEFILE# ELT=1 else ifeq ($(ELT),1) @@ -154,7 +155,8 @@ else default:;: $(ELT) endif !, - '--no-print-directory -j2 --jobserver-style=pipe', "#MAKE#[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.\n: 2\n: 1"); + '--no-print-directory -j2 --jobserver-style=pipe', "#MAKE#[2]: warning: jobserver unavailable: using -j1. Add '+' to parent make rule.\n: 2\n: 1"); + } } # If we're not using pipes for jobserver, then they are available in sub-makes