This adds support for visiting abandoned pages in mimalloc and improves
the performance of the page visiting code. Abandoned pages contain
memory blocks from threads that have exited. At some point, they may be
later reclaimed by other threads. We still need to visit those pages in
the free-threaded GC because they contain live objects.
This also reduces the overhead of visiting mimalloc pages:
* Special cases for full, empty, and pages containing only a single
block.
* Fix free_map to use one bit instead of one byte per block.
* Use fast integer division by a constant algorithm when computing
block offset from block size and index.
Serhiy Storchaka [Mon, 22 Jan 2024 14:14:42 +0000 (16:14 +0200)]
gh-102512: Turn _DummyThread into _MainThread after os.fork() called from a foreign thread (GH-113261)
Always set a _MainThread as a main thread after os.fork() is called from
a thread started not by the threading module.
A new _MainThread was already set as a new main thread after fork if
threading.current_thread() was not called for a foreign thread before fork.
Now, if it was called before fork, the implicitly created _DummyThread will
be turned into _MainThread after fork.
It fixes, in particularly, an incompatibility of _DummyThread with
the threading shutdown logic which relies on the main thread
having tstate_lock.
Co-authored-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Gregory P. Smith [Sun, 21 Jan 2024 23:25:52 +0000 (15:25 -0800)]
gh-114328: tty cbreak mode should not alter ICRNL (#114335)
The terminal CR -> NL mapping setting should be inherited in cbreak mode as OSes do not specify altering it as part of their stty cbreak mode definition.
Serhiy Storchaka [Sun, 21 Jan 2024 20:16:45 +0000 (22:16 +0200)]
gh-114241: Fix and improve the ftplib CLI (GH-114242)
* Fix writing the retrieved binary file to stdout.
* Add a newline after writing warnings to stderr.
* Fix a TypeError if the netrc file doesn't contain a host/default entry.
* Improve the usage message.
Ronald Oussoren [Sun, 21 Jan 2024 10:25:15 +0000 (11:25 +0100)]
gh-80931: Skip some socket tests while hunting for refleaks on macOS (#114057)
Some socket tests related to sending file descriptors cause a file descriptor leak on macOS, all of them tests that send one or more descriptors than cannot be received on the read end. This appears to be a platform bug.
This PR skips those tests when doing a refleak test run to avoid hiding other problems.
Gregory P. Smith [Sat, 20 Jan 2024 20:09:50 +0000 (12:09 -0800)]
Remove the non-test Lib/test/time_hashlib.py. (#114354)
I believe I added this while chasing some performance of hash functions
when I first created hashlib. It hasn't been used since, is frankly
trivial, and not a test.
Sam Gross [Sat, 20 Jan 2024 16:14:45 +0000 (11:14 -0500)]
gh-112529: Use GC heaps for GC allocations in free-threaded builds (gh-114157)
* gh-112529: Use GC heaps for GC allocations in free-threaded builds
The free-threaded build's garbage collector implementation will need to
find GC objects by traversing mimalloc heaps. This hooks up the
allocation calls with the correct heaps by using a thread-local
"current_obj_heap" variable.
gh-114332: Fix the flags reference for ``re.compile()`` (#114334)
The GH-93000 change set inadvertently caused a sentence in re.compile()
documentation to refer to details that no longer followed. Correct this
with a link to the Flags sub-subsection.
Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>
Barney Gale [Sat, 20 Jan 2024 03:06:00 +0000 (03:06 +0000)]
GH-113225: Speed up `pathlib.Path.walk(top_down=False)` (#113693)
Use `_make_child_entry()` rather than `_make_child_relpath()` to retrieve
path objects for directories to visit. This saves the allocation of one
path object per directory in user subclasses of `PathBase`, and avoids a
second loop.
This trick does not apply when walking top-down, because users can affect
the walk by modifying *dirnames* in-place.
A side effect of this change is that, in bottom-up mode, subdirectories of
each directory are visited in reverse order, and that this order doesn't
match that of the names in *dirnames*. I suspect this is fine as the
order is arbitrary anyway.
Barney Gale [Sat, 20 Jan 2024 02:10:25 +0000 (02:10 +0000)]
GH-79634: Accept path-like objects as pathlib glob patterns. (#114017)
Allow `os.PathLike` objects to be passed as patterns to `pathlib.Path.glob()` and `rglob()`. (It's already possible to use them in `PurePath.match()`)
While we're in the area:
- Allow empty glob patterns in `PathBase` (but not `Path`)
- Speed up globbing in `PathBase` by generating paths with trailing slashes only as a final step, rather than for every intermediate directory.
- Simplify and speed up handling of rare patterns involving both `**` and `..` segments.
mpage [Fri, 19 Jan 2024 12:17:51 +0000 (04:17 -0800)]
gh-113884: Refactor `queue.SimpleQueue` to use a ring buffer to store items (#114259)
Use a ring buffer instead of a Python list in order to simplify the
process of making queue.SimpleQueue thread-safe in free-threaded
builds. The ring buffer implementation has no places where critical
sections may be released.
Petr Viktorin [Thu, 18 Jan 2024 01:15:29 +0000 (02:15 +0100)]
gh-113205: test_multiprocessing.test_terminate: Test the API on threadpools (#114186)
gh-113205: test_multiprocessing.test_terminate: Test the API works on threadpools
Threads can't be forced to terminate (without potentially corrupting too much
state), so the expected behaviour of `ThreadPool.terminate` is to wait for
the currently executing tasks to finish.
The entire test was skipped in GH-110848 (0e9c364f4ac18a2237bdbac702b96bcf8ef9cb09).
Instead of skipping it entirely, we should ensure the API eventually succeeds:
use a shorter timeout.
For the record: on my machine, when the test is un-skipped, the task manages to
start in about 1.5% cases.
Barney Gale [Tue, 16 Jan 2024 22:51:57 +0000 (22:51 +0000)]
GH-110109: pathlib docs: bring `from_uri()` and `as_uri()` together. (#110312)
This is a very soft deprecation of `PurePath.as_uri()`. We instead document
it as a `Path` method, and add a couple of sentences mentioning that it's
also available in `PurePath`.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
gh-94220: Align fnmatch docs with the implementation and amend markup (#114152)
- Align the argument spec for fnmatch functions with the actual
implementation.
- Update Sphinx markup to recent recommandations.
- Add link to 'iterable' glossary entry.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Sam Gross [Tue, 16 Jan 2024 21:42:15 +0000 (16:42 -0500)]
gh-112529: Track if debug allocator is used as underlying allocator (#113747)
* gh-112529: Track if debug allocator is used as underlying allocator
The GC implementation for free-threaded builds will need to accurately
detect if the debug allocator is used because it affects the offset of
the Python object from the beginning of the memory allocation. The
current implementation of `_PyMem_DebugEnabled` only considers if the
debug allocator is the outer-most allocator; it doesn't handle the case
of "hooks" like tracemalloc being used on top of the debug allocator.
This change enables more accurate detection of the debug allocator by
tracking when debug hooks are enabled.
Align the multiprocessing shared memory docs with Diatáxis's
recommendations for references.
- use a parameter list for the SharedMemory.__init__() argument spec
- use the imperative mode
- use versionadded, not versionchanged, for added parameters
- reflow touched lines according to SemBr