mpage [Sat, 16 Mar 2024 12:56:30 +0000 (05:56 -0700)]
gh-114271: Fix race in `Thread.join()` (#114839)
There is a race between when `Thread._tstate_lock` is released[^1] in `Thread._wait_for_tstate_lock()`
and when `Thread._stop()` asserts[^2] that it is unlocked. Consider the following execution
involving threads A, B, and C:
1. A starts.
2. B joins A, blocking on its `_tstate_lock`.
3. C joins A, blocking on its `_tstate_lock`.
4. A finishes and releases its `_tstate_lock`.
5. B acquires A's `_tstate_lock` in `_wait_for_tstate_lock()`, releases it, but is swapped
out before calling `_stop()`.
6. C is scheduled, acquires A's `_tstate_lock` in `_wait_for_tstate_lock()` but is swapped
out before releasing it.
7. B is scheduled, calls `_stop()`, which asserts that A's `_tstate_lock` is not held.
However, C holds it, so the assertion fails.
The race can be reproduced[^3] by inserting sleeps at the appropriate points in
the threading code. To do so, run the `repro_join_race.py` from the linked repo.
There are two main parts to this PR:
1. `_tstate_lock` is replaced with an event that is attached to `PyThreadState`.
The event is set by the runtime prior to the thread being cleared (in the same
place that `_tstate_lock` was released). `Thread.join()` blocks waiting for the
event to be set.
2. `_PyInterpreterState_WaitForThreads()` provides the ability to wait for all
non-daemon threads to exit. To do so, an `is_daemon` predicate was added to
`PyThreadState`. This field is set each time a thread is created. `threading._shutdown()`
now calls into `_PyInterpreterState_WaitForThreads()` instead of waiting on
`_tstate_lock`s.
Serhiy Storchaka [Sat, 16 Mar 2024 11:31:19 +0000 (13:31 +0200)]
gh-116484: Fix collisions between Checkbutton and ttk.Checkbutton default names (GH-116495)
Change automatically generated tkinter.Checkbutton widget names to
avoid collisions with automatically generated tkinter.ttk.Checkbutton
widget names within the same parent widget.
Zackery Spytz [Fri, 15 Mar 2024 14:38:13 +0000 (07:38 -0700)]
gh-63283: IDNA prefix should be case insensitive (GH-17726)
Any capitalization of "xn--" should be acceptable for the ACE prefix
(see https://tools.ietf.org/html/rfc3490#section-5).
Co-authored-by: Pepijn de Vos <pepijndevos@gmail.com> Co-authored-by: Erlend E. Aasland <erlend@python.org> Co-authored-by: Petr Viktorin <encukou@gmail.com>
Since 3.12, allocating a GC object cannot immediately trigger GC. This
allows us to simplify the logic for creating the canonical callback-less
weakref.
vxiiduu [Thu, 14 Mar 2024 23:09:36 +0000 (09:09 +1000)]
gh-116195: Implements a fast path for nt.getppid (GH-116205)
Use the NtQueryInformationProcess system call to efficiently retrieve the parent process ID in a single step, rather than using the process snapshots API which retrieves large amounts of unnecessary information and is more prone to failure (since it makes heap allocations).
Includes a fallback to the original win32_getppid implementation in case the unstable API appears to return strange results.
* Move Block and BlockParser classes to a new libclinic.block_parser
module.
* Move Language and PythonLanguage classes to a new
libclinic.language module.
Victor Stinner [Thu, 14 Mar 2024 15:42:41 +0000 (16:42 +0100)]
gh-88494: Use QueryPerformanceCounter() for time.monotonic() (#116781)
On Windows, time.monotonic() now uses the QueryPerformanceCounter()
clock to have a resolution better than 1 us, instead of the
gGetTickCount64() clock which has a resolution of 15.6 ms.
gh-116760: Fix pystats for trace attempts (GH-116761)
There are now at least two bytecodes that may attempt to optimize,
JUMP_BACK, and more recently, COLD_EXIT.
Only the JUMP_BACK was counting the attempt in the stats.
This moves that counter to uop_optimize itself so it should
always happen no matter where it is called from.
Sam Gross [Wed, 13 Mar 2024 18:56:28 +0000 (14:56 -0400)]
gh-116631: Fix race condition in `test_shutdown_immediate_put_join` (#116670)
The test case had a race condition: if `q.task_done()` was executed
after `shutdown(immediate=True)`, then it would raise an exception
because the immediate shutdown already emptied the queue. This happened
rarely with the GIL (due to the switching interval), but frequently in
the free-threaded build.
Sam Gross [Wed, 13 Mar 2024 13:27:36 +0000 (09:27 -0400)]
gh-116604: Fix test_gc on free-threaded build (#116662)
The free-threaded GC only does full collections, so it uses a threshold that
is a maximum of a fixed value (default 2000) and proportional to the number of
live objects. If there were many live objects after the previous collection,
then the threshold may be larger than 10,000 causing
`test_indirect_calls_with_gc_disabled` to fail.
This manually sets the threshold to `(1000, 0, 0)` for the test. The `0`
disables the proportional scaling.
Tim Peters [Wed, 13 Mar 2024 00:59:42 +0000 (19:59 -0500)]
GH-116554: Relax list.sort()'s notion of "descending" runs (#116578)
* GH-116554: Relax list.sort()'s notion of "descending" run
Rewrote `count_run()` so that sub-runs of equal elements no longer end a descending run. Both ascending and descending runs can have arbitrarily many sub-runs of arbitrarily many equal elements now. This is tricky, because we only use ``<`` comparisons, so checking for equality doesn't come "for free". Surprisingly, it turned out there's a very cheap (one comparison) way to determine whether an ascending run consisted of all-equal elements. That sealed the deal.
In addition, after a descending run is reversed in-place, we now go on to see whether it can be extended by an ascending run that just happens to be adjacent. This succeeds in finding at least one additional element to append about half the time, and so appears to more than repay its cost (the savings come from getting to skip a binary search, when a short run is artificially forced to length MIINRUN later, for each new element `count_run()` can add to the initial run).
While these have been in the back of my mind for years, a question on StackOverflow pushed it to action:
Jason R. Coombs [Tue, 12 Mar 2024 21:36:21 +0000 (17:36 -0400)]
gh-116307: Proper fix for 'mod' leaking across importlib tests (#116680)
gh-116307: Create a new import helper 'isolated modules' and use that instead of 'Clean Import' to ensure that tests from importlib_resources don't leave modules in sys.modules.
Sam Gross [Tue, 12 Mar 2024 17:12:02 +0000 (13:12 -0400)]
gh-116604: Check for `gcstate->enabled` in _Py_RunGC in free-threaded build (#116663)
This isn't strictly necessary because the implementation of `gc_should_collect`
already checks `gcstate->enabled` in the free-threaded build, but it seems
like a good idea until the common pieces of gc.c and gc_free_threading.c are
refactored out.