Sam Gross [Thu, 5 Oct 2023 15:46:33 +0000 (15:46 +0000)]
gh-109549: Add new states to PyThreadState to support PEP 703 (gh-109915)
This adds a new field 'state' to PyThreadState that can take on one of three values: _Py_THREAD_ATTACHED, _Py_THREAD_DETACHED, or _Py_THREAD_GC. The "attached" and "detached" states correspond closely to acquiring and releasing the GIL. The "gc" state is current unused, but will be used to implement stop-the-world GC for --disable-gil builds in the near future.
Victor Stinner [Thu, 5 Oct 2023 15:02:48 +0000 (17:02 +0200)]
gh-110393: Remove watchdog with hardcoded timeout (#110400)
test_builtin and test_socketserver no longer use signal.alarm() to
implement a watchdog with a hardcoded timeout (2 and 60 seconds).
Python test runner regrtest has two watchdogs: faulthandler and
timeout on running worker processes. Tests using short hardcoded
timeout can fail on slowest buildbots just because the timeout is too
short.
박문식 [Thu, 5 Oct 2023 14:49:07 +0000 (23:49 +0900)]
gh-82367: Use `FindFirstFile` Win32 API in `ntpath.realpath()` (GH-110298)
* Use `FindFirstFile` Win32 API to fix a bug where `ntpath.realpath()`
breaks out of traversing a series of paths where a (handled)
`ERROR_ACCESS_DENIED` or `ERROR_SHARING_VIOLATION` occurs.
* Update docs to reflect that `ntpath.realpath()` eliminates MS-DOS
style names.
When using worker processes (-jN) with --verbose3 option, regrtest
can now display the worker output even if a worker process does
crash. Previously, sys.stdout and sys.stderr were replaced and so
the worker output was lost on a crash.
Eric Snow [Wed, 4 Oct 2023 22:35:27 +0000 (16:35 -0600)]
gh-110310: Add a Per-Interpreter XID Registry for Heap Types (gh-110311)
We do the following:
* add a per-interpreter XID registry (PyInterpreterState.xidregistry)
* put heap types there (keep static types in _PyRuntimeState.xidregistry)
* clear the registries during interpreter/runtime finalization
* avoid duplicate entries in the registry (when _PyCrossInterpreterData_RegisterClass() is called more than once for a type)
* use Py_TYPE() instead of PyObject_Type() in _PyCrossInterpreterData_Lookup()
The per-interpreter registry helps preserve isolation between interpreters. This is important when heap types are registered, which is something we haven't been doing yet but I will likely do soon.
Victor Stinner [Wed, 4 Oct 2023 08:57:35 +0000 (10:57 +0200)]
gh-110166: Fix gdb CFunctionFullTests on ppc64le clang build (#110331)
CFunctionFullTests now also runs "bt" command before "py-bt-full",
similar to CFunctionTests which also runs "bt" command before
"py-bt". So test_gdb can skip the test if patterns like "?? ()" are
found in the gdb output.
gh-110273: dataclasses.replace() now raise TypeError for all invalid arguments (GH-110274)
dataclasses.replace() now raises TypeError instead of ValueError if
specify keyword argument for a field declared with init=False or miss keyword
argument for required InitVar field.
elfstrom [Tue, 3 Oct 2023 21:59:49 +0000 (23:59 +0200)]
gh-109917: Fix test instability in test_concurrent_futures (#110306)
The test had an instability issue due to the ordering of the dummy
queue operation and the real wakeup pipe operations. Both primitives
are thread safe but not done atomically as a single update and may
interleave arbitrarily. With the old order of operations this can lead
to an incorrect state where the dummy queue is full but the wakeup
pipe is empty. By swapping the order in clear() I think this can no
longer happen in any possible operation interleaving (famous last
words).
gh-109979: Auto-generate the target for DEOPT_IF() (#110193)
In Python/bytecodes.c, you now write
```
DEOPT_IF(condition);
```
The code generator expands this to
```
DEOPT_IF(condition, opcode);
```
where `opcode` is the name of the unspecialized instruction.
This works inside macro expansions too.
**CAVEAT:** The entire `DEOPT_IF(condition)` statement must be on a single line.
If it isn't, the substitution will fail; an error will be printed by the code generator
and the C compiler will report some errors.
Victor Stinner [Tue, 3 Oct 2023 16:53:51 +0000 (18:53 +0200)]
gh-108867: Add PyThreadState_GetUnchecked() function (#108870)
Add PyThreadState_GetUnchecked() function: similar to
PyThreadState_Get(), but don't issue a fatal error if it is NULL. The
caller is responsible to check if the result is NULL. Previously,
this function was private and known as _PyThreadState_UncheckedGet().
Eric Snow [Tue, 3 Oct 2023 15:20:48 +0000 (09:20 -0600)]
gh-109860: Use a New Thread State When Switching Interpreters, When Necessary (gh-110245)
In a few places we switch to another interpreter without knowing if it has a thread state associated with the current thread. For the main interpreter there wasn't much of a problem, but for subinterpreters we were *mostly* okay re-using the tstate created with the interpreter (located via PyInterpreterState_ThreadHead()). There was a good chance that tstate wasn't actually in use by another thread.
However, there are no guarantees of that. Furthermore, re-using an already used tstate is currently fragile. To address this, now we create a new thread state in each of those places and use it.
One consequence of this change is that PyInterpreterState_ThreadHead() may not return NULL (though that won't happen for the main interpreter).
Eric Snow [Mon, 2 Oct 2023 20:12:12 +0000 (14:12 -0600)]
gh-105716: Support Background Threads in Subinterpreters Consistently (gh-109921)
The existence of background threads running on a subinterpreter was preventing interpreters from getting properly destroyed, as well as impacting the ability to run the interpreter again. It also affected how we wait for non-daemon threads to finish.
We add PyInterpreterState.threads.main, with some internal C-API functions.
Eric Snow [Mon, 2 Oct 2023 19:59:05 +0000 (13:59 -0600)]
gh-109853: Fix sys.path[0] For Subinterpreters (gh-109994)
This change makes sure sys.path[0] is set properly for subinterpreters. Before, it wasn't getting set at all. This PR does not address the broader concerns from gh-109853.
Victor Stinner [Mon, 2 Oct 2023 17:24:08 +0000 (19:24 +0200)]
gh-107073: Make PyObject_VisitManagedDict() public (#108763)
Make PyObject_VisitManagedDict() and PyObject_ClearManagedDict()
functions public in Python 3.13 C API.
* Rename _PyObject_VisitManagedDict() to PyObject_VisitManagedDict().
* Rename _PyObject_ClearManagedDict() to PyObject_ClearManagedDict().
* Document these functions.
Victor Stinner [Mon, 2 Oct 2023 16:07:56 +0000 (18:07 +0200)]
gh-110014: Remove PY_TIMEOUT_MAX from limited C API (#110217)
If the timeout is greater than PY_TIMEOUT_MAX,
PyThread_acquire_lock_timed() uses a timeout of PY_TIMEOUT_MAX
microseconds, which is around 280.6 years. This case is unlikely and
limiting a timeout to 280.6 years sounds like a reasonable trade-off.
The constant PY_TIMEOUT_MAX is not used in PyPI top 5,000 projects.
This method supports file URIs (including variants) as described in RFC 8089, such as URIs generated by `pathlib.Path.as_uri()` and `urllib.request.pathname2url()`.
The method is added to `Path` rather than `PurePath` because it uses `os.fsdecode()`, and so its results vary from system to system. I intend to deprecate `PurePath.as_uri()` and move it to `Path` for the same reason.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Victor Stinner [Sat, 30 Sep 2023 22:37:23 +0000 (00:37 +0200)]
gh-110152: regrtest handles cross compilation and HOSTRUNNER (#110156)
* _add_python_opts() now handles cross compilation and HOSTRUNNER.
* display_header() now tells if Python is cross-compiled, display
HOSTRUNNER, and get the host platform.
* Remove Tools/scripts/run_tests.py script.
* Remove "make hostrunnertest": use "make buildbottest"
or "make test" instead.
Victor Stinner [Sat, 30 Sep 2023 22:12:51 +0000 (00:12 +0200)]
gh-109649: Add os.process_cpu_count() function (#109907)
* Refactor os_sched_getaffinity_impl(): move variable definitions to
their first assignment.
* Fix test_posix.test_sched_getaffinity(): restore the old CPU mask
when the test completes!
* Doc: Specify that os.cpu_count() counts *logicial* CPUs.
* Doc: Specify that os.sched_getaffinity(0) is related to the calling
thread.
Victor Stinner [Sat, 30 Sep 2023 20:06:45 +0000 (22:06 +0200)]
gh-110014: Include explicitly <unistd.h> header (#110155)
* Remove unused <locale.h> includes.
* Remove unused <fcntl.h> include in traceback.h.
* Remove redundant <assert.h> and <stddef.h> includes. They are already
included by "Python.h".
* Remove <object.h> include in faulthandler.c. Python.h already includes it.
* Add missing <stdbool.h> in pycore_pythread.h if HAVE_PTHREAD_STUBS
is defined.
* Fix also warnings in pthread_stubs.h: don't redefine macros if they
are already defined, like the __NEED_pthread_t macro.
Victor Stinner [Sat, 30 Sep 2023 18:23:26 +0000 (20:23 +0200)]
gh-109748: Fix again venv test_zippath_from_non_installed_posix() (#110149)
Call also copy_python_src_ignore() on listdir() names.
shutil.copytree(): replace set() with an empty tuple. An empty tuple
becomes a constant in the compiler and checking if an item is in an
empty tuple is cheap.
Victor Stinner [Sat, 30 Sep 2023 17:25:54 +0000 (19:25 +0200)]
gh-110014: Fix _POSIX_THREADS and _POSIX_SEMAPHORES usage (#110139)
* pycore_pythread.h is now the central place to make sure that
_POSIX_THREADS and _POSIX_SEMAPHORES macros are defined if
available.
* Make sure that pycore_pythread.h is included when _POSIX_THREADS
and _POSIX_SEMAPHORES macros are tested.
* PY_TIMEOUT_MAX is now defined as a constant, since its value
depends on _POSIX_THREADS, instead of being defined as a macro.
* Prevent integer overflow in the preprocessor when computing
PY_TIMEOUT_MAX_VALUE on Windows:
replace "0xFFFFFFFELL * 1000 < LLONG_MAX"
with "0xFFFFFFFELL < LLONG_MAX / 1000".
* Document the change and give hints how to fix affected code.
* Add an exception for PY_TIMEOUT_MAX name to smelly.py
* Add PY_TIMEOUT_MAX to the stable ABI
concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.
Add test_python_finalization_error() to test_concurrent_futures.
* terminate_broken() no longer calls shutdown_workers() since the
call queue is no longer working anymore (read and write ends of
the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
to prevent race conditons with ProcessPoolExecutor.submit().
multiprocessing.Queue changes:
* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
leaking "dangling threads" even if the thread was not started
yet.
projects / thirdparty / Python / cpython.git / log
