[3.12] gh-114440: Close writer pipe in multiprocessing.Queue, not concurrent.futures...

This was left out of the 3.12 backport for three related issues:
- gh-107219 (which adds `self.call_queue._writer.close()` to `_ExecutorManagerThread` in `concurrent.futures`)
- gh-109370 (which changes this to be only called on Windows)
- gh-109047 (which moves the call to `multiprocessing.Queue`'s `_terminate_broken`)

Without this change, ProcessPoolExecutor sometimes hangs on Windows
when a worker process is terminated.

Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

diff --git a/Lib/concurrent/futures/process.py b/Lib/concurrent/futures/process.py
index 33e62fe231e18779a5c046dc1bd764022b1ca87d..0e452883963c1719b7d87b334b17540306507f3c 100644 (file)
--- a/Lib/concurrent/futures/process.py
+++ b/Lib/concurrent/futures/process.py
@@ -524,11 +524,6 @@ class _ExecutorManagerThread(threading.Thread):
 
         self.call_queue._terminate_broken()
 
-        # gh-107219: Close the connection writer which can unblock
-        # Queue._feed() if it was stuck in send_bytes().
-        if sys.platform == 'win32':
-            self.call_queue._writer.close()
-
         # clean up resources
         self._join_executor_internals(broken=True)
 

diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py
index d36de75749f4edf8f9c5f5c381ef0ac8d632594d..852ae87b27686122a096dfa691c5364d87ebe8aa 100644 (file)
--- a/Lib/multiprocessing/queues.py
+++ b/Lib/multiprocessing/queues.py
@@ -164,6 +164,11 @@ class Queue(object):
         # gh-94777: Prevent queue writing to a pipe which is no longer read.
         self._reader.close()
 
+        # gh-107219: Close the connection writer which can unblock
+        # Queue._feed() if it was stuck in send_bytes().
+        if sys.platform == 'win32':
+            self._writer.close()
+
         self.close()
         self.join_thread()
 

diff --git a/Misc/NEWS.d/next/Library/2023-09-25-02-11-14.gh-issue-114440.b2TrqG.rst b/Misc/NEWS.d/next/Library/2023-09-25-02-11-14.gh-issue-114440.b2TrqG.rst
new file mode 100644 (file)
index 0000000..7231a3c
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2023-09-25-02-11-14.gh-issue-114440.b2TrqG.rst
@@ -0,0 +1,6 @@
+On Windows, closing the connection writer when cleaning up a broken
+:class:`multiprocessing.Queue` queue is now done for all queues, rather than
+only in :mod:`concurrent.futures` manager thread.
+This can prevent a deadlock when a ``multiprocessing`` worker process terminates
+without cleaning up.
+This completes the backport of patches by Victor Stinner and Serhiy Storchaka.