Victor Stinner [Wed, 20 Sep 2023 15:45:23 +0000 (17:45 +0200)]
[3.11] gh-103053: Skip test_freeze_simple_script() on PGO build (#109591) (#109616)
gh-103053: Skip test_freeze_simple_script() on PGO build (#109591)
Skip test_freeze_simple_script() of test_tools.test_freeze if Python
is built with "./configure --enable-optimizations", which means with
Profile Guided Optimization (PGO): it just makes the test too slow.
The freeze tool is tested by many other CIs with other (faster)
compiler flags.
test.pythoninfo now gets also get_build_info() of
test.libregrtests.utils.
SubprocessProtocol process_exited() method can be called before
pipe_data_received() and pipe_connection_lost() methods. Document it
and adapt the example in the doc.
[3.11] gh-109474: Update two Unix packaging URLs (GH-109307) (#109478)
gh-109474: Update two Unix packaging URLs (GH-109307)
update packaging URLs
fix a broken URL for fedora RPM packaging guide and fix a URL redirect for Slackware packaging guide.
(cherry picked from commit 0b38ce440bd76b3d25b6d042ee9613841fb4a947)
[3.11] gh-60283: Check for redefined test names in CI (#109161) (#109366)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
(cherry picked from commit 3cb9a8edca6e3fa0f0045b03a9a6444cf8f7affe)
[3.11] gh-109396: Fix test_socket.test_hmac_sha1() in FIPS mode (GH-109423) (#109427)
gh-109396: Fix test_socket.test_hmac_sha1() in FIPS mode (GH-109423)
Use a longer key: FIPS mode requires at least of at least 112 bits.
The previous key was only 32 bits.
(cherry picked from commit e091b9f20fa8e409003af79f3c468b8225e6dcd3)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.11] gh-104736: Fix test_gdb tests on ppc64le with clang (GH-109360) (#109362)
gh-104736: Fix test_gdb tests on ppc64le with clang (GH-109360)
Fix test_gdb on Python built with LLVM clang 16 on Linux ppc64le (ex:
Fedora 38). Search patterns in gdb "bt" command output to detect
when gdb fails to retrieve the traceback. For example, skip a test if
"Backtrace stopped: frame did not save the PC" is found.
(cherry picked from commit 44d9a71ea246e7c3fb478d9be62c16914be6c545)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.11] gh-109295: Fix test_os.test_access_denied() for TEMP=cwd (GH-109299) (#109303)
gh-109295: Fix test_os.test_access_denied() for TEMP=cwd (GH-109299)
Fix test_os.test_access_denied() when the TEMP environment variable
is equal to the current working directory. Run the test using a
different filename, since self.fname already exists in this case.
(cherry picked from commit 7dedfd36dc16d9e1e15d7d0b0a636dd401a5a543)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.11] gh-109295: Clean up multiprocessing in test_asyncio and test_compileall (GH-109298) (#109302)
gh-109295: Clean up multiprocessing in test_asyncio and test_compileall (GH-109298)
test_asyncio and test_compileall now clean up multiprocessing by
calling multiprocessing _cleanup_tests(): explicitly clean up
resources and stop background processes like the resource tracker.
(cherry picked from commit 09ea4b8706165fd9474165090a0ba86509abd6c8)
Co-authored-by: Victor Stinner <vstinner@python.org>
Fix _thread.start_new_thread() race condition. If a thread is created
during Python finalization, the newly spawned thread now exits
immediately instead of trying to access freed memory and lead to a
crash.
thread_run() calls PyEval_AcquireThread() which checks if the thread
must exit. The problem was that tstate was dereferenced earlier in
_PyThreadState_Bind() which leads to a crash most of the time.
Move _PyThreadState_CheckConsistency() from thread_run() to
_PyThreadState_Bind().
Fix a race condition in concurrent.futures. When a process in the
process pool was terminated abruptly (while the future was running or
pending), close the connection write end. If the call queue is
blocked on sending bytes to a worker process, closing the connection
write end interrupts the send, so the queue can be closed.
Changes:
* _ExecutorManagerThread.terminate_broken() now closes
call_queue._writer.
* multiprocessing PipeConnection.close() now interrupts
WaitForMultipleObjects() in _send_bytes() by cancelling the
overlapped operation.
(cherry picked from commit a9b1f84790e977fb09f75b148c4c4f5924a6ef99)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.11] gh-109230: test_pyexpat no longer depends on the current directory (GH-109233) (#109242)
gh-109230: test_pyexpat no longer depends on the current directory (GH-109233)
Fix test_pyexpat.test_exception(): it can now be run from a directory
different than Python source code directory. Before, the test failed
in this case.
Skip the test if Modules/pyexpat.c source is not available. Skip also
the test on Python implementations other than CPython.
(cherry picked from commit e55aab95786e0e9fb36a9a1122d2d0fb3d2403cd)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.11] gh-109237: Fix test_site for non-ASCII working directory (GH-109238) (#109240)
gh-109237: Fix test_site for non-ASCII working directory (GH-109238)
Fix test_site.test_underpth_basic() when the working directory
contains at least one non-ASCII character: encode the "._pth" file to
UTF-8 and enable the UTF-8 Mode to use UTF-8 for the child process
stdout.
(cherry picked from commit cbb3a6f8ada3d133c3ab9f9465b65067fce5bb42)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.11] gh-50644: Forbid pickling of codecs streams (GH-109180) (GH-109232)
Attempts to pickle or create a shallow or deep copy of codecs streams
now raise a TypeError.
Previously, copying failed with a RecursionError, while pickling
produced wrong results that eventually caused unpickling to fail with
a RecursionError.
(cherry picked from commit d6892c2b9263b39ea1c7905667942914b6a24b2c)
Victor Stinner [Fri, 8 Sep 2023 11:10:33 +0000 (13:10 +0200)]
[3.11] gh-104690: thread_run() checks for tstate dangling pointer (#109056) (#109134)
gh-104690: thread_run() checks for tstate dangling pointer (#109056)
thread_run() of _threadmodule.c now calls
_PyThreadState_CheckConsistency() to check if tstate is a dangling
pointer when Python is built in debug mode.
Rename ceval_gil.c is_tstate_valid() to
_PyThreadState_CheckConsistency() to reuse it in _threadmodule.c.
[3.11] gh-108962: Skip test_tempfile.test_flags() if not supported (GH-108964) (#108968)
gh-108962: Skip test_tempfile.test_flags() if not supported (GH-108964)
Skip test_tempfile.test_flags() if chflags() fails with "OSError:
[Errno 45] Operation not supported" (ex: on FreeBSD 13).
(cherry picked from commit cd2ef21b076b494224985e266c5f5f8b37c66618)
Co-authored-by: Victor Stinner <vstinner@python.org>
This un-skips this test now that pythontest.net implements appropriate firewall
rules for it.
(cherry picked from commit 1829a3c9a3712b6a68a3a449e4a08787c73da51d)
[3.11] GH-90915: Document that SystemExit doesn't trigger sys.excepthook (GH-31357) (#109083)
Co-authored-by: Colin Watson <cjwatson@debian.org> Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.
* "make regen-global-objects": Make sure that deepfreeze.c is
generated and up to date, and always run "make clinic".
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
"make clinic". They are now outdated, these commands are now
safe to use.
Backport notes:
* Omit Doc/using/configure.rst changes.
* no need to change "make clinic", it didn't run
generate_global_objects.py script before.
Output with two wheels:
```
❯ GITHUB_ACTIONS=true ./Tools/build/verify_ensurepip_wheels.py
::error file=/Volumes/RAMDisk/cpython/Lib/ensurepip/_bundled/pip-22.0.4-py3-none-any.whl::Found more than one wheel for package pip.
::error file=/Volumes/RAMDisk/cpython/Lib/ensurepip/_bundled/pip-23.2.1-py3-none-any.whl::Found more than one wheel for package pip.
```
Output without wheels:
```
❯ GITHUB_ACTIONS=true ./Tools/build/verify_ensurepip_wheels.py
::error file=::Could not find a pip wheel on disk.
```
(cherry picked from commit f8a047941f2e4a1848700c21d58a08c9ec6a9c68)
* Add get_recursion_available() and get_recursion_depth() functions
to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
recursion limit and compute the maximum nested array/dict depending
on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit().
* Enhance test_sys tests on sys.getrecursionlimit()
and sys.setrecursionlimit().
Backport notes:
* Set support.infinite_recursion() minimum to 4 frames.
* test_support.test_get_recursion_depth() uses limit-2, apparently
f-string counts for 2 frames in Python 3.11.
* test_sys.test_setrecursionlimit_to_depth() tests depth+2 instead of
depth+1.
gh-91960: Skip test_gdb if gdb cannot retrive Python frames (GH-108999)
Skip test_gdb if gdb is unable to retrieve Python frame objects: if a
frame is "<optimized out>". When Python is built with "clang -Og",
gdb can fail to retrive the 'frame' parameter of
_PyEval_EvalFrameDefault(). In this case, tests like py_bt() are
likely to fail. Without getting access to Python frames,
python-gdb.py is mostly clueless on retrieving the Python traceback.
Moreover, test_gdb is no longer skipped on macOS if Python is built
with Clang.
(cherry picked from commit fbce43a251488f666be9794c908a6613bf8ae260)
Co-authored-by: Victor Stinner <vstinner@python.org>
* gh-93353: regrtest checks for leaked temporary files (#93776)
When running tests with -jN, create a temporary directory per process
and mark a test as "environment changed" if a test leaks a temporary
file or directory.
* gh-93353: regrtest supports checking tmp files with -j2 (#93909)
regrtest now also implements checking for leaked temporary files and
directories when using -jN for N >= 2. Use tempfile.mkdtemp() to
create the temporary directory. Skip this check on WASI.
* gh-84461: Fix Emscripten umask and permission issues (GH-94002)
- Emscripten's default umask is too strict, see
https://github.com/emscripten-core/emscripten/issues/17269
- getuid/getgid and geteuid/getegid are stubs that always return 0
(root). Disable effective uid/gid syscalls and fix tests that use
chmod() current user.
- Cannot drop X bit from directory.
* gh-95027: Fix regrtest stdout encoding on Windows (#98492)
On Windows, when the Python test suite is run with the -jN option,
the ANSI code page is now used as the encoding for the stdout
temporary file, rather than using UTF-8 which can lead to decoding
errors.
* gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (#103927)
This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.
After porting we can see the direct impact on a multicore system.
Without this change:
Running make test is 5 min 26 seconds
With this change:
Running make test takes 3 min 39 seconds
That'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%.
The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant.
Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests.
---------
Co-authored-by: Carl Meyer <carl@oddbird.net> Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC]
(cherry picked from commit 9e011e7c77dad7d0bbb944c44891531606caeb21)
* Display the sanitizer config in the regrtest header. (#105301)
Display the sanitizers present in libregrtest.
Having this in the CI output for tests with the relevant environment
variable displayed will help make it easier to do what we need to
create an equivalent local test run.
* gh-101634: regrtest reports decoding error as failed test (#106169)
When running the Python test suite with -jN option, if a worker stdout
cannot be decoded from the locale encoding report a failed testn so the
exitcode is non-zero.
Currently, test_asyncio package is only splitted into sub-tests when
using command "./python -m test". With this change, it's also
splitted when passing it on the command line:
"./python -m test test_asyncio".
Remove the concept of "STDTESTS". Python is now mature enough to not
have to bother with that anymore. Removing STDTESTS simplify the
code.
test_netrc, test_pep646_syntax and test_xml_etree now return results
in the test_main() function.
Changes:
* Rewrite TestResult as a dataclass with a new State class.
* Add test.support.TestStats class and Regrtest.stats_dict attribute.
* libregrtest.runtest functions now modify a TestResult instance
in-place.
* libregrtest summary lists the number of run tests and skipped
tests, and denied resources.
* Add TestResult.has_meaningful_duration() method.
* Compute TestResult duration in the upper function.
* Use time.perf_counter() instead of time.monotonic().
* Regrtest: rename 'resource_denieds' attribute to 'resource_denied'.
* Rename CHILD_ERROR to MULTIPROCESSING_ERROR.
* Use match/case syntadx to have different code depending on the
test state.
* gh-108822: Add Changelog entry for regrtest statistics (#108821)
---------
Co-authored-by: Christian Heimes <christian@python.org> Co-authored-by: Zachary Ware <zach@python.org> Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Joshua Herman <zitterbewegung@gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
[3.11] Reorder some test's decorators (GH-108804) (GH-108845)
For example, do not demand the 'cpu' resource if the test cannot be run
due to non-working threads.
(cherry picked from commit 509bb61977cc8a4487efd3f9cdd63d9f7b86be62)
Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Jakub Kulík <Kulikjak@gmail.com>
(cherry picked from commit 0e6d582b3b73a88e71cae04327b31a1ee203722c)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
[3.11] Improve some C API documentation (GH-108768) (GH-108786)
* Express functions which take argument as a C string in terms of
functions which take Python object.
* Use "note" directive for PyMapping_HasKey() and
PyMapping_HasKeyString() notes.
[3.11] gh-104372: Drop the GIL around the vfork() call. (#104782) (#104958)
gh-104372: Drop the GIL around the vfork() call. (#104782)
On Linux where the `subprocess` module can use the `vfork` syscall for
faster spawning, prevent the parent process from blocking other threads
by dropping the GIL while it waits for the vfork'ed child process `exec`
outcome. This prevents spawning a binary from a slow filesystem from
blocking the rest of the application.
[3.11] `ast` docs: Fix incorrect link on `keyword` (GH-108728) (#108738)
`ast` docs: Fix incorrect link on `keyword` (GH-108728)
In two places, Sphinx was erroneously adding links to the `keyword` module instead of the `ast.keyword` class
(cherry picked from commit c1e2f3b2f70b8a72ea7e1bf792addf62a94ae65d)
[3.11] gh-108520: Fix bad fork detection in nested multiprocessing use case (GH-108568) (#108692)
gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization.
projects / thirdparty / Python / cpython.git / log
