* Remove "typedef PyTime_t PyTime_t;" which was
"typedef PyTime_t _PyTime_t;" before a previous rename.
* Update comments of "Unchecked" functions.
* Remove invalid PyTime_Time() comment.
Guido van Rossum [Tue, 20 Feb 2024 20:24:35 +0000 (12:24 -0800)]
Tier 2 cleanups and tweaks (#115534)
* Rename `_testinternalcapi.get_{uop,counter}_optimizer` to `new_*_optimizer`
* Use `_PyUOpName()` instead of` _PyOpcode_uop_name[]`
* Add `target` to executor iterator items -- `list(ex)` now returns `(opcode, oparg, target, operand)` quadruples
* Add executor methods `get_opcode()` and `get_oparg()` to get `vmdata.opcode`, `vmdata.oparg`
* Define a helper for printing uops, and unify various places where they are printed
* Add a hack to summarize_stats.py to fix legacy uop names (e.g. `POP_TOP` -> `_POP_TOP`)
* Define helpers in `test_opt.py` for accessing the set or list of opnames of an executor
Sam Gross [Tue, 20 Feb 2024 18:04:37 +0000 (13:04 -0500)]
gh-115103: Implement delayed free mechanism for free-threaded builds (#115367)
This adds `_PyMem_FreeDelayed()` and supporting functions. The
`_PyMem_FreeDelayed()` function frees memory with the same allocator as
`PyMem_Free()`, but after some delay to ensure that concurrent lock-free
readers have finished.
Victor Stinner [Tue, 20 Feb 2024 16:50:43 +0000 (17:50 +0100)]
gh-110850: Cleanup pycore_time.h includes (#115724)
<pycore_time.h> include is no longer needed to get the PyTime_t type
in internal header files. This type is now provided by <Python.h>
include. Add <pycore_time.h> includes to C files instead.
Sam Gross [Tue, 20 Feb 2024 15:36:40 +0000 (10:36 -0500)]
gh-115491: Keep some fields valid across allocations (free-threading) (#115573)
This avoids filling the memory occupied by ob_tid, ob_ref_local, and
ob_ref_shared with debug bytes (e.g., 0xDD) in mimalloc in the
free-threaded build.
Brett Simmers [Tue, 20 Feb 2024 14:57:48 +0000 (06:57 -0800)]
gh-112175: Add `eval_breaker` to `PyThreadState` (#115194)
This change adds an `eval_breaker` field to `PyThreadState`. The primary
motivation is for performance in free-threaded builds: with thread-local eval
breakers, we can stop a specific thread (e.g., for an async exception) without
interrupting other threads.
The source of truth for the global instrumentation version is stored in the
`instrumentation_version` field in PyInterpreterState. Threads usually read the
version from their local `eval_breaker`, where it continues to be colocated
with the eval breaker bits.
Victor Stinner [Sun, 18 Feb 2024 20:06:39 +0000 (21:06 +0100)]
gh-115122: Add --bisect option to regrtest (#115123)
* test.bisect_cmd now exit with code 0 on success, and code 1 on
failure. Before, it was the opposite.
* test.bisect_cmd now runs the test worker process with
-X faulthandler.
* regrtest RunTests: Add create_python_cmd() and bisect_cmd()
methods.
Dmitry Marakasov [Sat, 17 Feb 2024 14:54:47 +0000 (17:54 +0300)]
Fix ProgramPriorityTests on FreeBSD with high nice value (GH-100145)
It expects priority to be capped with 19, which is the cap for Linux,
but for FreeBSD the cap is 20 and the test fails under the similar
conditions. Tweak the condition to cover FreeBSD as well.
6t8k [Sat, 17 Feb 2024 11:16:06 +0000 (11:16 +0000)]
gh-95782: Fix io.BufferedReader.tell() etc. being able to return offsets < 0 (GH-99709)
lseek() always returns 0 for character pseudo-devices like
`/dev/urandom` (for other non-regular files, e.g. `/dev/stdin`, it
always returns -1, to which CPython reacts by raising appropriate
exceptions). They are thus technically seekable despite not having seek
semantics.
When calling read() on e.g. an instance of `io.BufferedReader` that
wraps such a file, `BufferedReader` reads ahead, filling its buffer,
creating a discrepancy between the number of bytes read and the internal
`tell()` always returning 0, which previously resulted in e.g.
`BufferedReader.tell()` or `BufferedReader.seek()` being able to return
positions < 0 even though these are supposed to be always >= 0.
Invariably keep the return value non-negative by returning
max(former_return_value, 0) instead, and add some corresponding tests.
Stevoisiak [Sat, 17 Feb 2024 08:33:28 +0000 (03:33 -0500)]
gh-101699: Explain using Match.expand with \g<0> (GH-101701)
Update documentation for re library to explain that a backreference `\g<0>` is
expanded to the entire string when using Match.expand().
Note that numeric backreferences to group 0 (`\0`) are not supported.
This adds a safe memory reclamation scheme based on FreeBSD's "GUS" and
quiescent state based reclamation (QSBR). The API provides a mechanism
for callers to detect when it is safe to free memory that may be
concurrently accessed by readers.
Łukasz Langa [Fri, 16 Feb 2024 20:24:56 +0000 (21:24 +0100)]
gh-115556: Remove quotes from command-line arguments in test.bat and rt.bat (#115557)
This change essentially replaces usage of `%1` with `%~1`, which removes
quotes, if any. Without this change, the if statements fail due to
the quotes mangling the syntax.
Additionally, this change works around comma being treated as a parameter
delimiter in test.bat by escaping commas at time of parsing. Tested
combinations of rt and regrtest arguments, all seems to work as before
but now you can specify commas in arguments like "-uall,extralargefile".
mpage [Fri, 16 Feb 2024 18:29:25 +0000 (10:29 -0800)]
gh-114271: Make `thread._rlock` thread-safe in free-threaded builds (#115102)
The ID of the owning thread (`rlock_owner`) may be accessed by
multiple threads without holding the underlying lock; relaxed
atomics are used in place of the previous loads/stores.
The number of times that the lock has been acquired (`rlock_count`)
is only ever accessed by the thread that holds the lock; we do not
need to use atomics to access it.
Sam Gross [Fri, 16 Feb 2024 16:22:27 +0000 (11:22 -0500)]
gh-112529: Make the GC scheduling thread-safe (#114880)
The GC keeps track of the number of allocations (less deallocations)
since the last GC. This buffers the count in thread-local state and uses
atomic operations to modify the per-interpreter count. The thread-local
buffering avoids contention on shared state.
A consequence is that the GC scheduling is not as precise, so
"test_sneaky_frame_object" is skipped because it requires that the GC be
run exactly after allocating a frame object.
David Benjamin [Fri, 16 Feb 2024 00:24:51 +0000 (19:24 -0500)]
gh-114572: Fix locking in cert_store_stats and get_ca_certs (#114573)
* gh-114572: Fix locking in cert_store_stats and get_ca_certs
cert_store_stats and get_ca_certs query the SSLContext's X509_STORE with
X509_STORE_get0_objects, but reading the result requires a lock. See
https://github.com/openssl/openssl/pull/23224 for details.
Instead, use X509_STORE_get1_objects, newly added in that PR.
X509_STORE_get1_objects does not exist in current OpenSSLs, but we can
polyfill it with X509_STORE_lock and X509_STORE_unlock.
Dino Viehland [Thu, 15 Feb 2024 18:54:57 +0000 (10:54 -0800)]
gh-113743: Make the MRO cache thread-safe in free-threaded builds (#113930)
Makes _PyType_Lookup thread safe, including:
Thread safety of the underlying cache.
Make mutation of mro and type members thread safe
Also _PyType_GetMRO and _PyType_GetBases are currently returning borrowed references which aren't safe.
Sam Gross [Thu, 15 Feb 2024 13:37:54 +0000 (08:37 -0500)]
gh-115432: Add critical section variant that handles a NULL object (#115433)
This adds `Py_XBEGIN_CRITICAL_SECTION` and
`Py_XEND_CRITICAL_SECTION`, which accept a possibly NULL object as an
argument. If the argument is NULL, then nothing is locked or unlocked.
Otherwise, they behave like `Py_BEGIN/END_CRITICAL_SECTION`.
T. Wouters [Thu, 15 Feb 2024 13:24:13 +0000 (14:24 +0100)]
gh-115490: Work around test.support.interpreters.channels not handling unloading (#115515)
Work around test.support.interpreters.channels not handling unloading, which
regrtest does when running tests sequentially, by explicitly skipping the
unloading of test.support.interpreters and its submodules.
This can be rolled back once test.support.interpreters.channels supports
unloading, if we are keeping sequential runs in the same process around.