Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.
* "make regen-global-objects": Make sure that deepfreeze.c is
generated and up to date, and always run "make clinic".
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
"make clinic". They are now outdated, these commands are now
safe to use.
Backport notes:
* Omit Doc/using/configure.rst changes.
* no need to change "make clinic", it didn't run
generate_global_objects.py script before.
Output with two wheels:
```
❯ GITHUB_ACTIONS=true ./Tools/build/verify_ensurepip_wheels.py
::error file=/Volumes/RAMDisk/cpython/Lib/ensurepip/_bundled/pip-22.0.4-py3-none-any.whl::Found more than one wheel for package pip.
::error file=/Volumes/RAMDisk/cpython/Lib/ensurepip/_bundled/pip-23.2.1-py3-none-any.whl::Found more than one wheel for package pip.
```
Output without wheels:
```
❯ GITHUB_ACTIONS=true ./Tools/build/verify_ensurepip_wheels.py
::error file=::Could not find a pip wheel on disk.
```
(cherry picked from commit f8a047941f2e4a1848700c21d58a08c9ec6a9c68)
* Add get_recursion_available() and get_recursion_depth() functions
to the test.support module.
* Change infinite_recursion() default max_depth from 75 to 100.
* Fix test_tomllib recursion tests for WASI buildbots: reduce the
recursion limit and compute the maximum nested array/dict depending
on the current available recursion limit.
* test.pythoninfo logs sys.getrecursionlimit().
* Enhance test_sys tests on sys.getrecursionlimit()
and sys.setrecursionlimit().
Backport notes:
* Set support.infinite_recursion() minimum to 4 frames.
* test_support.test_get_recursion_depth() uses limit-2, apparently
f-string counts for 2 frames in Python 3.11.
* test_sys.test_setrecursionlimit_to_depth() tests depth+2 instead of
depth+1.
gh-91960: Skip test_gdb if gdb cannot retrive Python frames (GH-108999)
Skip test_gdb if gdb is unable to retrieve Python frame objects: if a
frame is "<optimized out>". When Python is built with "clang -Og",
gdb can fail to retrive the 'frame' parameter of
_PyEval_EvalFrameDefault(). In this case, tests like py_bt() are
likely to fail. Without getting access to Python frames,
python-gdb.py is mostly clueless on retrieving the Python traceback.
Moreover, test_gdb is no longer skipped on macOS if Python is built
with Clang.
(cherry picked from commit fbce43a251488f666be9794c908a6613bf8ae260)
Co-authored-by: Victor Stinner <vstinner@python.org>
* gh-93353: regrtest checks for leaked temporary files (#93776)
When running tests with -jN, create a temporary directory per process
and mark a test as "environment changed" if a test leaks a temporary
file or directory.
* gh-93353: regrtest supports checking tmp files with -j2 (#93909)
regrtest now also implements checking for leaked temporary files and
directories when using -jN for N >= 2. Use tempfile.mkdtemp() to
create the temporary directory. Skip this check on WASI.
* gh-84461: Fix Emscripten umask and permission issues (GH-94002)
- Emscripten's default umask is too strict, see
https://github.com/emscripten-core/emscripten/issues/17269
- getuid/getgid and geteuid/getegid are stubs that always return 0
(root). Disable effective uid/gid syscalls and fix tests that use
chmod() current user.
- Cannot drop X bit from directory.
* gh-95027: Fix regrtest stdout encoding on Windows (#98492)
On Windows, when the Python test suite is run with the -jN option,
the ANSI code page is now used as the encoding for the stdout
temporary file, rather than using UTF-8 which can lead to decoding
errors.
* gh-82054: allow test runner to split test_asyncio to execute in parallel by sharding. (#103927)
This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel.
After porting we can see the direct impact on a multicore system.
Without this change:
Running make test is 5 min 26 seconds
With this change:
Running make test takes 3 min 39 seconds
That'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%.
The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant.
Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests.
---------
Co-authored-by: Carl Meyer <carl@oddbird.net> Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC]
(cherry picked from commit 9e011e7c77dad7d0bbb944c44891531606caeb21)
* Display the sanitizer config in the regrtest header. (#105301)
Display the sanitizers present in libregrtest.
Having this in the CI output for tests with the relevant environment
variable displayed will help make it easier to do what we need to
create an equivalent local test run.
* gh-101634: regrtest reports decoding error as failed test (#106169)
When running the Python test suite with -jN option, if a worker stdout
cannot be decoded from the locale encoding report a failed testn so the
exitcode is non-zero.
Currently, test_asyncio package is only splitted into sub-tests when
using command "./python -m test". With this change, it's also
splitted when passing it on the command line:
"./python -m test test_asyncio".
Remove the concept of "STDTESTS". Python is now mature enough to not
have to bother with that anymore. Removing STDTESTS simplify the
code.
test_netrc, test_pep646_syntax and test_xml_etree now return results
in the test_main() function.
Changes:
* Rewrite TestResult as a dataclass with a new State class.
* Add test.support.TestStats class and Regrtest.stats_dict attribute.
* libregrtest.runtest functions now modify a TestResult instance
in-place.
* libregrtest summary lists the number of run tests and skipped
tests, and denied resources.
* Add TestResult.has_meaningful_duration() method.
* Compute TestResult duration in the upper function.
* Use time.perf_counter() instead of time.monotonic().
* Regrtest: rename 'resource_denieds' attribute to 'resource_denied'.
* Rename CHILD_ERROR to MULTIPROCESSING_ERROR.
* Use match/case syntadx to have different code depending on the
test state.
* gh-108822: Add Changelog entry for regrtest statistics (#108821)
---------
Co-authored-by: Christian Heimes <christian@python.org> Co-authored-by: Zachary Ware <zach@python.org> Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Joshua Herman <zitterbewegung@gmail.com> Co-authored-by: Gregory P. Smith <greg@krypto.org>
[3.11] Reorder some test's decorators (GH-108804) (GH-108845)
For example, do not demand the 'cpu' resource if the test cannot be run
due to non-working threads.
(cherry picked from commit 509bb61977cc8a4487efd3f9cdd63d9f7b86be62)
Co-authored-by: Victor Stinner <vstinner@python.org> Co-authored-by: Jakub Kulík <Kulikjak@gmail.com>
(cherry picked from commit 0e6d582b3b73a88e71cae04327b31a1ee203722c)
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Nikita Sobolev <mail@sobolevn.me> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
[3.11] Improve some C API documentation (GH-108768) (GH-108786)
* Express functions which take argument as a C string in terms of
functions which take Python object.
* Use "note" directive for PyMapping_HasKey() and
PyMapping_HasKeyString() notes.
[3.11] gh-104372: Drop the GIL around the vfork() call. (#104782) (#104958)
gh-104372: Drop the GIL around the vfork() call. (#104782)
On Linux where the `subprocess` module can use the `vfork` syscall for
faster spawning, prevent the parent process from blocking other threads
by dropping the GIL while it waits for the vfork'ed child process `exec`
outcome. This prevents spawning a binary from a slow filesystem from
blocking the rest of the application.
[3.11] `ast` docs: Fix incorrect link on `keyword` (GH-108728) (#108738)
`ast` docs: Fix incorrect link on `keyword` (GH-108728)
In two places, Sphinx was erroneously adding links to the `keyword` module instead of the `ast.keyword` class
(cherry picked from commit c1e2f3b2f70b8a72ea7e1bf792addf62a94ae65d)
[3.11] gh-108520: Fix bad fork detection in nested multiprocessing use case (GH-108568) (#108692)
gh-107275 introduced a regression where a SemLock would fail being passed along nested child processes, as the `is_fork_ctx` attribute would be left missing after the first deserialization.
Serhiy Storchaka [Sun, 27 Aug 2023 12:18:58 +0000 (15:18 +0300)]
[3.11] gh-107913: Fix possible losses of OSError error codes (GH-107930) (GH-108524)
Functions like PyErr_SetFromErrno() and SetFromWindowsErr() should be
called immediately after using the C API which sets errno or the Windows
error code.
(cherry picked from commit 2b15536fa94d07e9e286826c23507402313ec7f4)
--------- Co-authored-by: Terry Jan Reedy tjreedy@udel.edu Co-authored-by: R <cherrymelon@foxmail.com>
(cherry picked from commit 7096a2be33619dc02c06a6dc30aac414a9eba462)
[3.11] gh-102211: Document `re.{Pattern,Match}`’s existence (GH-102212) (#108491)
Co-authored-by: Philipp A <flying-sheep@web.de> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
PyDict_GetItemString(), PyDict_SetItemString() and
PyDict_DelItemString() expects a UTF-8 encoding string for the key.
(cherry picked from commit 9a225d7d5b0530ee73fa00d4816897997a9eb733)
Co-authored-by: Victor Stinner <vstinner@python.org>
https://docs.python.org/3/howto/pyporting.html was written for another time. In this patch:
- material that frames Python 3 as "new" is removed
- descriptions and directions have been trimmed
(cherry picked from commit 809ea7c4b6c2b818ae510f1f58e82b6b05ed4ef9)
Łukasz Langa [Thu, 24 Aug 2023 10:08:52 +0000 (12:08 +0200)]
[3.11] gh-108342: Make ssl TestPreHandshakeClose more reliable (GH-108370) (#108405)
* In preauth tests of test_ssl, explicitly break reference cycles
invoving SingleConnectionTestServerThread to make sure that the
thread is deleted. Otherwise, the test marks the environment as
altered because the threading module sees a "dangling thread"
(SingleConnectionTestServerThread). This test leak was introduced
by the test added for the fix of issue gh-108310.
* Use support.SHORT_TIMEOUT instead of hardcoded 1.0 or 2.0 seconds
timeout.
* SingleConnectionTestServerThread.run() catchs TimeoutError
* Fix a race condition (missing synchronization) in
test_preauth_data_to_tls_client(): the server now waits until the
client connect() completed in call_after_accept().
* test_https_client_non_tls_response_ignored() calls server.join()
explicitly.
* Replace "localhost" with server.listener.getsockname()[0].
(cherry picked from commit 592bacb6fc0833336c0453e818e9b95016e9fd47)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.11] gh-108342: Break ref cycle in SSLSocket._create() exc (GH-108344) (#108349)
Explicitly break a reference cycle when SSLSocket._create() raises an
exception. Clear the variable storing the exception, since the
exception traceback contains the variables and so creates a reference
cycle.
Changed "__setattr__" to "object.__setattr__" in a section that was specifically supposed to refer to the __setattr__ method of the object class. Also suppressed the link to the data model docs for __setattr__, since we're talking about a specific __setattr__ implementation, not __setattr__ methods in general.
(cherry picked from commit 79fdacc0059a3959074d2d9d054653eae1dcfe06)
* Move test_cppext to its own directory
* Rename setup_testcppext.py to setup.py
* Rename _testcppext.cpp to extension.cpp
* The source (extension.cpp) is now also copied by the test.
[3.11] Clarify how topics.py gets created. (GH-106121) (GH-106580)
When changing docs, it was easy to find text in topics.py, and I
wondered whether I was supposed to edit it. Thankfully, the top of the
file says it's auto-generated, so I knew I didn't have to edit it. But I
didn't know what started the auto-generation process.
Łukasz Langa [Tue, 22 Aug 2023 17:53:19 +0000 (19:53 +0200)]
[3.11] gh-108310: Fix CVE-2023-40217: Check for & avoid the ssl pre-close flaw (#108317)
gh-108310: Fix CVE-2023-40217: Check for & avoid the ssl pre-close flaw
Instances of `ssl.SSLSocket` were vulnerable to a bypass of the TLS handshake
and included protections (like certificate verification) and treating sent
unencrypted data as if it were post-handshake TLS encrypted data.
The vulnerability is caused when a socket is connected, data is sent by the
malicious peer and stored in a buffer, and then the malicious peer closes the
socket within a small timing window before the other peers’ TLS handshake can
begin. After this sequence of events the closed socket will not immediately
attempt a TLS handshake due to not being connected but will also allow the
buffered data to be read as if a successful TLS handshake had occurred.
Co-authored-by: Gregory P. Smith [Google LLC] <greg@krypto.org>
* Move Python scripts related to test_module to this new directory:
good_getattr.py and bad_getattrX.py scripts.
* Move Lib/test/test_module.py to Lib/test/test_module/__init__.py.
(cherry picked from commit adfc118fdab66882599e01a84c22bd897055f3f1)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.11] Docs: align the param spec of sqlite3.Connection methods with the implementation (GH-108285) (#108288)
- no parameters of create_aggregate() are positional-only
- all parameters of create_collation() are positional-only
(cherry picked from commit 893215a4e7f59eabb8ccdf188c4b9b1de5bd8966)
Co-authored-by: Erlend E. Aasland <erlend@python.org>
[3.11] gh-107396: tarfiles: set self.exception before _init_read_gz() (GH-107485) (GH-108208)
gh-107396: tarfiles: set self.exception before _init_read_gz() (GH-107485)
In the stack call of: _init_read_gz()
```
_read, tarfile.py:548
read, tarfile.py:526
_init_read_gz, tarfile.py:491
```
a try;except exists that uses `self.exception`, so it needs to be set before
calling _init_read_gz().
(cherry picked from commit 37135d25e269ede92bc7da363bebfa574782e59a)
Serhiy Storchaka [Mon, 21 Aug 2023 10:53:36 +0000 (13:53 +0300)]
[3.11] Improve references in the tutorial (GH-108069) (GH-108204)
* Use full qualified names for references (even if they do not work now,
they will work in future).
* Silence references to examples.
(cherry picked from commit 622ddc41674c2566062af82f7b079aa01d2aae8c)
[3.11] Docs: document 'manager' and '_log' attrs of logging.Logging (GH-108145) (GH-108189)
(cherry picked from commit f904aa4e1f6943e5bd9a8a73cf762f063e6fa247) Authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Erlend E. Aasland <erlend@python.org>
[3.11] Resolve reference warnings in faq/library.rst (GH-108149) (#108183)
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com> Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
[3.11] Docs: Fix Sphinx warnings in sys.rst (GH-108106) (#108166)
Docs: Fix Sphinx warnings in sys.rst (GH-108106)
- Mark up named tuple attributes as attributes
- Remove links for external functions
- io.BufferedIOBase has no 'buffer' attribute;
remove the link and mark up using :attr:`!buffer`
- (Re)format some tables as bullet lists:
- sys._emscripten_info
- sys.hash_info
- sys.int_info
- sys.thread_info
- In the paragraphs mentioning 'f_trace_lines' and 'f_trace_opcodes',
add links to the frame objects reference.
Co-authored-by: Erlend E. Aasland <erlend@python.org> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
[3.11] gh-107659: Improve wording of the description of `ctypes.pointer` and `ctypes.POINTER` (GH-107769) (#108164)
gh-107659: Improve wording of the description of `ctypes.pointer` and `ctypes.POINTER` (GH-107769)
(cherry picked from commit beffb30dc7a07044f4198245d049ddda1f4b24db)
Co-authored-by: Tomas R <tomas.roun8@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Kumar Aditya <kumaraditya@python.org>
projects / thirdparty / Python / cpython.git / log
