Eric Snow [Fri, 28 Jul 2023 23:16:12 +0000 (17:16 -0600)]
[3.12] gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974) (gh-107412)
gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974)
This fixes a crasher due to a race condition, triggered infrequently when two isolated (own GIL) subinterpreters simultaneously initialize their sys or builtins modules. The crash happened due the combination of the "detached" thread state we were using and the "last holder" logic we use for the GIL. It turns out it's tricky to use the same thread state for different threads. Who could have guessed?
We solve the problem by eliminating the one object we were still sharing between interpreters. We replace it with a low-level hashtable, using the "raw" allocator to avoid tying it to the main interpreter.
We also remove the accommodations for "detached" thread states, which were a dubious idea to start with.
Co-authored-by: Felipe A. Hernandez <ergoithz@gmail.com> Co-authored-by: Kumar Aditya <kumaraditya@python.org> Co-authored-by: Gregory P. Smith <greg@krypto.org>
[3.12] gh-104621: Check for Incompatible Extensions in import_find_extension() (gh-107184) (gh-107360)
gh-104621: Check for Incompatible Extensions in import_find_extension() (gh-107184)
This fixes a bug where incompatible modules could still be imported if attempted multiple times.
(cherry picked from commit 75c974f5353685f338344618ad7344e64c2293d0)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
[3.12] gh-104432: Use `memcpy()` to avoid misaligned loads (GH-104433) (#107355)
gh-104432: Use `memcpy()` to avoid misaligned loads (GH-104433)
Fix potential unaligned memory access on C APIs involving returned sequences
of `char *` pointers within the :mod:`grp` and :mod:`socket` modules. These
were revealed using a ``-fsaniziter=alignment`` build on ARM macOS.
(cherry picked from commit f01e4cedba1a17d321664834bb255d9d04ad16ce)
Co-authored-by: Christopher Chavez <chrischavez@gmx.us>
[3.12] gh-105699: Disable the Interpreters Stress Tests (gh-107354) (gh-107357)
gh-105699: Disable the Interpreters Stress Tests (gh-107354)
The two tests are crashing periodically in CI and on buildbots. I suspect the problem is in the _xxsubinterpreters module.
Regardless, I'm disabling the tests temporarily, to reduce the noise as we approach 3.12rc1. I'll be investigating the crashes separately.
(cherry picked from commit 4f67921ad28194155e3d4c16255fb140a6a4d89a)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
[3.12] gh-101524: Only Use Public C-API in the _xxsubinterpreters Module (gh-105258) (gh-107303)
The _xxsubinterpreters module was meant to only use public API. Some internal C-API usage snuck in over the last few years (e.g. gh-28969). This fixes that.
(cherry picked from commit e6373c0d8b59512aa7f0dea7f3fb162b6ed10fa4)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
[3.12] gh-107298: Docs: add targets for some :c:member: and :c:macro: references (GH-107316) (GH-107332)
Add targets for PyStructSequence_Desc and PyStructSequence_Field members
and macros like Py_EQ.
Fix target for Py_RETURN_RICHCOMPARE.
(cherry picked from commit abec9a1b20b70d8ced401d59fc4f02b331c6568b)
[3.12] gh-105002: [pathlib] Fix relative_to with walk_up=True using ".." (GH-107014) (#107315)
gh-105002: [pathlib] Fix relative_to with walk_up=True using ".." (GH-107014)
It makes sense to raise an Error because ".." can not
be resolved and the current working directory is unknown.
(cherry picked from commit e7e6e4b035f51ab4a962b45a957254859f264f4f)
Co-authored-by: János Kukovecz <kukoveczjanos@gmail.com>
[3.12] gh-106948: Update documentation nitpick_ignore for c:identifer domain (GH-107295) (#107297)
gh-106948: Update documentation nitpick_ignore for c:identifer domain (GH-107295)
Update the nitpick_ignore of the documentation configuration to fix
Sphinx warnings about standard C types when declaring functions with
the "c:function" markups.
Copy standard C types declared in the "c:type" domain to the
"c:identifier" domain, since "c:function" markup looks for types in
the "c:identifier" domain.
[3.12] gh-107237: Fix test_udp_reconnection() of test_logging (GH-107238) (#107242)
gh-107237: Fix test_udp_reconnection() of test_logging (GH-107238)
test_logging: Fix test_udp_reconnection() by increasing the timeout
from 100 ms to 5 minutes (LONG_TIMEOUT).
Replace also blocking wait() with wait(LONG_TIMEOUT) in
test_output() to prevent the test to hang.
(cherry picked from commit ed082383272c2c238e364e9cc83229234aee23cc)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] gh-105059: Use GCC/clang extension for PyObject union (GH-107232) (#107236)
gh-105059: Use GCC/clang extension for PyObject union (GH-107232)
Anonymous union is new in C11. To prevent compiler warning
when using -pedantic compiler option, use Clang and GCC
extension on C99 and older.
(cherry picked from commit 6261585d63a31835b65d445d99dc14cca3fe9cf5)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] gh-98608: Move PyInterpreterConfig to pylifecycle.h (GH-107191) (#107198)
gh-98608: Move PyInterpreterConfig to pylifecycle.h (GH-107191)
Move PyInterpreterConfig structure and associated macros from
initconfig.h to pylifecycle.h: it's not related to the Python
Initialization Configuration.
(cherry picked from commit e717b47ed8ae7017f0bfb835fe673aa836e8fcca)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] GH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h (GH-107188) (#107195)
GH-96803: Move PyUnstable_InterpreterFrame_GetCode() to Python.h (GH-107188)
Declare the following 3 PyUnstable functions in
Include/cpython/pyframe.h rather than Include/cpython/frameobject.h,
so they are now provided by the standard "GH-include <Python.h>".
(cherry picked from commit 837fa5c0cd92e70f625377c9ffb7ac31a23d7d63)
Co-authored-by: Victor Stinner <vstinner@python.org>
The documentation implies that PyVectorcall_Function() was available in Python 3.8.
This is half-true - it was available under a different name. I think it's clearer to set
the "version added" to 3.9.
(cherry picked from commit 0a9b339363a59be1249189c767ed6f46fd71e1c7)
[3.12] Convert `doc.yml` workflow to be reusable (GH-103914 + GH-105151) (#107042)
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
(cherry picked from commit 88d14da76f579fe014cbd7c15e42be4234135fe9)
(cherry picked from commit eaa670228066220f08c8d73f80365c50058d40b8)
[3.12] gh-106962: Detect mpicc in configure.ac (GH-106961) (#107081)
Don't let autoconf mistake MPI compilers for Intel compilers;
filter out the MPI case to prevent Intel specific options from being applied.
(cherry picked from commit 9a6b278769b9f24e0650283f6c347db8ae52b7b3)
Co-authored-by: Lukas van de Wiel <30800501+LukasvdWiel@users.noreply.github.com>
[3.12] gh-107008: Document the curses module variables LINES and COLS (GH-107011) (GH-107057)
LINES and COLS referred in curses.update_lines_cols() documentations are
the module variables, not the environment variables.
(cherry picked from commit 26e08dfdd7ac1b3d567d30cd35e4898121580390)
[3.12] gh-106714: Fix test_capi to not write a coredump (GH-107007) (#107009)
gh-106714: Fix test_capi to not write a coredump (GH-107007)
test_capi: Fix test_no_FatalError_infinite_loop() to no longer write
a coredump, by using test.support.SuppressCrashReport.
(cherry picked from commit 4a1026d7647c084b0dc80dd49163d16ba12a2e55)
Co-authored-by: Victor Stinner <vstinner@python.org>
Add test for the 'destination <name> clear' command,
and the 'destination' directive in general.
Fix two bugs in 'destination <name> clear' command:
1. The text attribute of the allocator is called 'text', not '_text'
2. Return after processing the 'clear' command,
instead of proceeding directly to the fail().
(cherry picked from commit 3372bcba9893030e4063a9264ec0b4d1b6166883)
Co-authored-by: Erlend E. Aasland <erlend@python.org>
[3.12] gh-105699: Fix an Interned Strings Crasher (gh-106930) (#106963)
gh-105699: Fix an Interned Strings Crasher (gh-106930)
A static (process-global) str object must only have its "interned" state cleared when no longer interned in any interpreters. They are the only ones that can be shared by interpreters so we don't have to worry about any other str objects.
We trigger clearing the state with the main interpreter, since no other interpreters may exist at that point and _PyUnicode_ClearInterned() is only called during interpreter finalization.
We do not address here the fact that a string will only be interned in the first interpreter that interns it. In any subsequent interpreters str.state.interned is already set so _PyUnicode_InternInPlace() will skip it. That needs to be addressed separately from fixing the crasher.
(cherry picked from commit 87e7cb09e4258c4deb01a07dc52c1021907195d7)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
[3.12] gh-105699: Fix a Crasher Related to a Deprecated Global Variable (gh-106923) (#106964)
gh-105699: Fix a Crasher Related to a Deprecated Global Variable (gh-106923)
There was a slight race in _Py_ClearFileSystemEncoding() (when called from _Py_SetFileSystemEncoding()), between freeing the value and setting the variable to NULL, which occasionally caused crashes when multiple isolated interpreters were used. (Notably, I saw at least 10 different, seemingly unrelated spooky-action-at-a-distance, ways this crashed. Yay, free threading!) We avoid the problem by only setting the global variables with the main interpreter (i.e. runtime init).
(cherry picked from commit 0ba07b2108d4763273f3fb85544dde34c5acd40a)
Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>