Petr Viktorin [Tue, 4 Mar 2025 13:10:09 +0000 (14:10 +0100)]
gh-129666: Add C11/C++11 to docs and -pedantic-errors to GCC/clang test_c[pp]ext tests (GH-130692)
Disable pedantic check for c++03 (unlimited API)
Also add a check for c++03 *limited* API, which passes in pedantic mode
after removing a comma in the `PySendResult` declaration, and allowing
`long long`.
Bénédikt Tran [Tue, 4 Mar 2025 09:47:19 +0000 (10:47 +0100)]
gh-89083: add support for UUID version 7 (RFC 9562) (#121119)
Add support for generating UUIDv7 objects according to RFC 9562, §5.7 [1].
The functionality is provided by the `uuid.uuid7()` function. The implementation
is based on a 42-bit counter as described by Method 1, §6.2 [2] and guarantees
monotonicity within the same millisecond.
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Éric <merwok@netwok.org>
gh-130599: use static constants str-to-int conversion (gh-130714)
Avoid a data race in free-threaded builds due to mutating global arrays at
runtime. Instead, compute the constants with an external Python script and
then define them as static global constant arrays. These constants are
used by `long_from_non_binary_base()`.
Barney Gale [Mon, 3 Mar 2025 17:56:57 +0000 (17:56 +0000)]
GH-128520: Merge `pathlib._abc` into `pathlib.types` (#130747)
There used to be a meaningful distinction between these modules: `pathlib`
imported `pathlib._abc` but not `pathlib.types`. This is no longer the
case (neither module is imported), so we move the ABCs as follows:
Bénédikt Tran [Mon, 3 Mar 2025 10:22:05 +0000 (11:22 +0100)]
gh-130149: refactor tests for HMAC (#130150)
Since we plan to introduce a built-in implementation for HMAC based on HACL*,
it becomes important for the HMAC tests to be flexible enough to avoid code
duplication.
In addition to the new layout based on mixin classes, we extend test coverage by
also testing the `__repr__` of HMAC objects and the HMAC one-shot functions.
We also fix the import to `_sha256` which, since gh-101924, resulted in some tests being
skipped as the module is no more available (its content was moved to the `_sha2` module).
Bénédikt Tran [Mon, 3 Mar 2025 08:20:33 +0000 (09:20 +0100)]
gh-127667: fix memory leaks in `hashlib` (#127668)
- Correctly handle `NULL` values returned by `EVP_MD_CTX_md`.
- Correctly free resources in error branches.
- Consistently suppress `_setException()` return value when needed.
- Collapse `_setException() + return NULL` into a single statement.
gh-128041: Add `terminate_workers` and `kill_workers` methods to ProcessPoolExecutor (GH-128043)
This adds two new methods to `multiprocessing`'s `ProcessPoolExecutor`:
- **`terminate_workers()`**: forcefully terminates worker processes using `Process.terminate()`
- **`kill_workers()`**: forcefully kills worker processes using `Process.kill()`
These methods provide users with a direct way to stop worker processes without `shutdown()` or relying on implementation details, addressing situations where immediate termination is needed.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com> Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> Commit-message-mostly-authored-by: Claude Sonnet 3.7 (because why not -greg)
Bénédikt Tran [Sun, 2 Mar 2025 17:16:51 +0000 (18:16 +0100)]
gh-128481: indicate that the default value for `FrameSummary.end_lineno` changed in 3.13 (#130755)
The value taken by `FrameSummary.end_lineno` when passing `end_lineno=None` changed in gh-112097.
Previously, a `end_lineno` could be specified to be `None` directly but since 939fc6d, passing None makes
the constructor use the value of `lineno` instead.
Bénédikt Tran [Sun, 2 Mar 2025 11:41:56 +0000 (12:41 +0100)]
gh-89083: add support for UUID version 6 (RFC 9562) (#120650)
Add support for generating UUIDv6 objects according to RFC 9562, §5.6 [1].
The functionality is provided by the `uuid.uuid6()` function which takes as inputs an optional 48-bit
hardware address and an optional 14-bit clock sequence. The UUIDv6 temporal fields are ordered
differently than those of UUIDv1, thereby providing improved database locality.
gh-130740: Move some `stdbool.h` includes after `Python.h` (#130738)
Move some `#include <stdbool.h>` after `#include "Python.h"` when `pyconfig.h` is not
included first and when we are in a platform-agnostic context. This is to avoid having
features defined by `stdbool.h` before those decided by `Python.h`.
Remove the *mode*, *parents* and *exist_ok* arguments from
`WritablePath.mkdir()`. These arguments imply support for POSIX permissions
and checking for preexistence of the path or its parents, but subclasses of
`WritablePath` may not have these capabilities.
The public `Path.mkdir()` method retains these arguments.
Barney Gale [Fri, 28 Feb 2025 20:33:51 +0000 (20:33 +0000)]
GH-116380: Speed up `glob.[i]glob()` by making fewer system calls. (#116392)
## Filtered recursive walk
Expanding a recursive `**` segment entails walking the entire directory
tree, and so any subsequent pattern segments (except special segments) can
be evaluated by filtering the expanded paths through a regex. For example,
`glob.glob("foo/**/*.py", recursive=True)` recursively walks `foo/` with
`os.scandir()`, and then filters paths through a regex based on "`**/*.py`,
with no further filesystem access needed.
This fixes an issue where `glob()` could return duplicate results.
## Tracking path existence
We store a flag alongside each path indicating whether the path is
guaranteed to exist. As we process the pattern:
- Certain special pattern segments (`""`, `"."` and `".."`) leave the flag
unchanged
- Literal pattern segments (e.g. `foo/bar`) set the flag to false
- Wildcard pattern segments (e.g. `*/*.py`) set the flag to true (because
children are found via `os.scandir()`)
- Recursive pattern segments (e.g. `**`) leave the flag unchanged for the
root path, and set it to true for descendants discovered via
`os.scandir()`.
If the flag is false at the end, we call `lstat()` on each path to filter
out missing paths.
## Minor speed-ups
- Exclude paths that don't match a non-terminal non-recursive wildcard
pattern _prior_ to calling `is_dir()`.
- Use a stack rather than recursion to implement recursive wildcards.
- This fixes a recursion error when globbing deep trees.
- Pre-compile regular expressions and pre-join literal pattern segments.
- Convert to/from `bytes` (a minor use-case) in `iglob()` rather than
supporting `bytes` throughout. This particularly simplifies the code
needed to handle relative bytes paths with `dir_fd`.
- Avoid calling `os.path.join()`; instead we keep paths in a normalized
form and append trailing slashes when needed.
- Avoid calling `os.path.normcase()`; instead we use case-insensitive regex
matching.
## Implementation notes
Much of this functionality is already present in pathlib's implementation
of globbing. The specific additions we make are:
1. Support for `dir_fd`
2. Support for `include_hidden`
3. Support for generating paths relative to `root_dir`
This unifies the implementations of globbing in the `glob` and `pathlib`
modules.
Sam Gross [Fri, 28 Feb 2025 14:27:51 +0000 (09:27 -0500)]
gh-124878: Add temporary TSAN suppression for free_threadstate (gh-130602)
The race condition with `free_threadstate` and daemon threads exists in
both the free threading and default builds. We were missing a
suppression in the default build.
Postpone <stdbool.h> inclusion after Python.h (#130641)
Remove inclusions prior to Python.h.
<stdbool.h> will cause <features.h> to be included before Python.h can
define some macros to enable some additional features, causing multiple
types not to be defined down the line.
gh-129200: Add locking to the iOS testbed startup sequence. (#130564)
Add a lock to ensure that only one iOS testbed per user can start at a time, so
that the simulator discovery process doesn't collide between instances.
Sam Gross [Thu, 27 Feb 2025 18:57:19 +0000 (13:57 -0500)]
gh-130091: Reorder `_PyThreadState_Attach` to avoid data race (gh-130092)
This moves `tstate_activate()` down to avoid a data race in the free
threading build on the `_PyRuntime`'s thread-local `autoTSSkey`. This
key is deleted during runtime finalization, which may happen
concurrently with a call to `_PyThreadState_Attach`.
The earlier `tstate_try/wait_attach` ensures that the thread is blocked
before it attempts to access the deleted `autoTSSkey`.
This fixes a TSAN reported data race in
`test_threading.test_import_from_another_thread`.
Sam Gross [Thu, 27 Feb 2025 13:27:54 +0000 (08:27 -0500)]
gh-130421: Fix data race on timebase initialization (gh-130592)
Windows and macOS require precomputing a "timebase" in order to convert
OS timestamps into nanoseconds. Retrieve and compute this value during
runtime initialization to avoid data races when accessing the time.
Fredrik Ahlberg [Thu, 27 Feb 2025 12:51:47 +0000 (13:51 +0100)]
gh-129288: Add optional l2_cid and l2_bdaddr_type in BTPROTO_L2CAP socket address tuple (#129293)
Add two optional, traling elements in the AF_BLUETOOTH socket address tuple:
- l2_cid, to allow e.g raw LE ATT connections
- l2_bdaddr_type. To be able to connect L2CAP sockets to Bluetooth LE devices,
the l2_bdaddr_type must be set to BDADDR_LE_PUBLIC or BDADDR_LE_RANDOM.
Sam Gross [Wed, 26 Feb 2025 21:36:53 +0000 (16:36 -0500)]
gh-130605: Temporarily disable test_concurrent_futures in TSAN CI job (gh-130606)
There are a number of data races in the default build without
suppressions that are exposed by this test. Disable the test for now
under TSAN until we have suppressions or fix the data races.
Barney Gale [Wed, 26 Feb 2025 21:07:27 +0000 (21:07 +0000)]
GH-125413: Add private `pathlib.Path` method to write metadata (#130238)
Replace `WritablePath._copy_writer` with a new `_write_info()` method. This
method allows the target of a `copy()` to preserve metadata.
Replace `pathlib._os.CopyWriter` and `LocalCopyWriter` classes with new
`copy_file()` and `copy_info()` functions. The `copy_file()` function uses
`source_path.info` wherever possible to save on `stat()`s.
Sam Gross [Wed, 26 Feb 2025 19:55:15 +0000 (14:55 -0500)]
gh-130519: Fix crash in QSBR when destructor reenters QSBR (gh-130553)
The `free_work_item()` function in QSBR may call arbitrary code via
Python object destructors, which may reenter the QSBR code. Reorder
the processing of work items to be robust to reentrancy.
Also fix the TODO for the out of memory situation.
Petr Viktorin [Wed, 26 Feb 2025 14:42:39 +0000 (15:42 +0100)]
gh-128982: Revert "#128982: Substitute regular expression in http.cookiejar.join_header_words for an efficient alternative (GH-128983)" and add tests (GH-130584)
* Revert "gh-128982: Substitute regular expression in `http.cookiejar.join_header_words` for an efficient alternative (GH-128983)"