[3.12] Enhance TypedDict docs around required/optional keys (GH-109547) (#109982)
Enhance TypedDict docs around required/optional keys (GH-109547)
As discussed in comments to GH-109544, the semantics of this attribute
are somewhat confusing. Add a note explaining its limitations and
steering users towards __required_keys__ and __optional_keys__ instead.
(cherry picked from commit f49958c886a2f2608f1008186d588efc2a98b445)
* Split test_pycfunction.py: add test_cfunction_full.py.
Split the function into the following 6 functions. In verbose
mode, these "pycfunction" tests now log each tested call.
* Move get_gdb_repr() to PrettyPrintTests.
* Replace DebuggerTests.get_sample_script() with SAMPLE_SCRIPT.
* Rename checkout_hook_path to CHECKOUT_HOOK_PATH.
* Rename gdb_version to GDB_VERSION_TEXT.
* Replace (gdb_major_version, gdb_minor_version) with GDB_VERSION.
* run_gdb() uses "backslashreplace" error handler instead of "replace".
* Add check_gdb() function to util.py.
* Enhance support.check_cflags_pgo(): check also for sysconfig
PGO_PROF_USE_FLAG (if available) in compiler flags.
* Move some SkipTest checks to test_gdb/__init__.py.
* Elaborate why gdb cannot be tested on Windows: gdb doesn't support
PDB debug symbol files.
* gh-104736: Fix test_gdb tests on ppc64le with clang (#109360)
Fix test_gdb on Python built with LLVM clang 16 on Linux ppc64le (ex:
Fedora 38). Search patterns in gdb "bt" command output to detect
when gdb fails to retrieve the traceback. For example, skip a test if
"Backtrace stopped: frame did not save the PC" is found.
* gh-110166: Fix gdb CFunctionFullTests on ppc64le clang build (#110331)
CFunctionFullTests now also runs "bt" command before "py-bt-full",
similar to CFunctionTests which also runs "bt" command before
"py-bt". So test_gdb can skip the test if patterns like "?? ()" are
found in the gdb output.
Fix race conditions in test_threading lock tests. Wait until a
condition is met rather than using time.sleep() with a hardcoded
number of seconds.
* Replace sleeping loops with support.sleeping_retry() which raises
an exception on timeout.
* Add wait_threads_blocked(nthread) which computes a sleep depending
on the number of threads. Remove _wait() function.
* test_set_and_clear(): use a way longer Event.wait() timeout.
* BarrierTests.test_repr(): wait until the 2 threads are waiting for
the barrier. Use a way longer timeout for Barrier.wait() timeout.
* test_thread_leak() no longer needs to count
len(threading.enumerate()): Bunch uses
threading_helper.wait_threads_exit() internally which does it in
wait_for_finished().
* Add BaseLockTests.wait_phase() which implements a timeout.
test_reacquire() and test_recursion_count() use wait_phase().
* gh-109974: Fix more threading lock_tests race conditions (#110089)
* Add context manager on Bunch class.
* Bunch now catchs exceptions on executed functions and re-raise them
at __exit__() as an ExceptionGroup.
* Rewrite BarrierProxy.test_default_timeout(). Use a single thread.
Only check that barrier.wait() blocks for at least default timeout
seconds.
* test_with(): inline _with() function.
Fix _thread.start_new_thread() race condition. If a thread is created
during Python finalization, the newly spawned thread now exits
immediately instead of trying to access freed memory and lead to a
crash.
thread_run() calls PyEval_AcquireThread() which checks if the thread
must exit. The problem was that tstate was dereferenced earlier in
_PyThreadState_Bind() which leads to a crash most of the time.
Move _PyThreadState_CheckConsistency() from thread_run() to
_PyThreadState_Bind().
test_unix_events tests using the multiprocessing module now call
multiprocessing.util._cleanup_tests().
(cherry picked from commit 1337765225d7d593169205672e004f97e15237ec)
Co-authored-by: Victor Stinner <vstinner@python.org>
Victor Stinner [Wed, 4 Oct 2023 10:27:12 +0000 (12:27 +0200)]
[3.12] gh-109615: Fix support test_copy_python_src_ignore() (#109958) (#110340)
* gh-109615: Fix support test_copy_python_src_ignore() (#109958)
Fix the test when run on an installed Python: use "abs_srcdir" of
sysconfig, and skip the test if the Python source code cannot be
found.
* Tools/patchcheck/patchcheck.py, Tools/freeze/test/freeze.py and
Lib/test/libregrtest/utils.py now first try to get "abs_srcdir"
from sysconfig, before getting "srcdir" from sysconfig.
* test.pythoninfo logs sysconfig "abs_srcdir".
* gh-109748: Fix again venv test_zippath_from_non_installed_posix() (#110149)
Call also copy_python_src_ignore() on listdir() names.
shutil.copytree(): replace set() with an empty tuple. An empty tuple
becomes a constant in the compiler and checking if an item is in an
empty tuple is cheap.
Victor Stinner [Wed, 4 Oct 2023 10:26:24 +0000 (12:26 +0200)]
[3.12] gh-109972: Split test_gdb.py into test_gdb package (#109977) (#110339)
gh-109972: Split test_gdb.py into test_gdb package (#109977)
Split test_gdb.py file into a test_gdb package made of multiple
tests, so tests can now be run in parallel.
* Create Lib/test/test_gdb/ directory.
* Split test_gdb.py into multiple files in Lib/test/test_gdb/
directory.
* Move Lib/test/gdb_sample.py to Lib/test/test_gdb/ directory.
Update get_sample_script(): use __file__ to locate gdb_sample.py.
* Move gdb_has_frame_select() and HAS_PYUP_PYDOWN to test_misc.py.
* Explicitly skip test_gdb on Windows. Previously, test_gdb was
skipped even if gdb was available because of
gdb_has_frame_select().
[3.12] gh-109917: Fix test instability in test_concurrent_futures (GH-110306) (#110315)
gh-109917: Fix test instability in test_concurrent_futures (GH-110306)
The test had an instability issue due to the ordering of the dummy
queue operation and the real wakeup pipe operations. Both primitives
are thread safe but not done atomically as a single update and may
interleave arbitrarily. With the old order of operations this can lead
to an incorrect state where the dummy queue is full but the wakeup
pipe is empty. By swapping the order in clear() I think this can no
longer happen in any possible operation interleaving (famous last
words).
(cherry picked from commit a376a72bd92cd7c9930467dd1aba40045fb75e3b)
Fix test_asyncio timeouts: don't measure the maximum duration, a test
should not measure a CI performance. Only measure the minimum
duration when a task has a timeout or delay. Add CLOCK_RES to
test_asyncio.utils.
[3.12] gh-110178: Use fewer weakrefs in test_typing.py (GH-110194) (#110224)
gh-110178: Use fewer weakrefs in test_typing.py (GH-110194)
Confirmed that without the C changes from GH-108517, this test still segfaults with only 10 weakrefs.
(cherry picked from commit 732ad44cec971be5255b1accbac6555d3615c2bf)
[3.12] gh-109590: Update shutil.which on Windows to prefer a PATHEXT extension on executable files (GH-109995) (#110202)
gh-109590: Update shutil.which on Windows to prefer a PATHEXT extension on executable files (GH-109995)
The default arguments for shutil.which() request an executable file, but extensionless files are not executable on Windows and should be ignored.
(cherry picked from commit 29b875bb93099171aeb7a60cd18d4e1f4ea3c1db)
Co-authored-by: Charles Machalow <csm10495@gmail.com>
* Doc: Specify that os.cpu_count() counts *logicial* CPUs.
* Doc: Specify that os.sched_getaffinity(0) is related to the calling
thread.
* Fix test_posix.test_sched_getaffinity(): restore the old CPU mask
when the test completes!
Don't measure the CI performance: don't test the maximum elapsed
time. The check failed on a slow CI.
(cherry picked from commit c62b49ecc8da13fa9522865ef6fe0aec194fd0d8)
Co-authored-by: Victor Stinner <vstinner@python.org>
concurrent.futures: The *executor manager thread* now catches
exceptions when adding an item to the *call queue*. During Python
finalization, creating a new thread can now raise RuntimeError. Catch
the exception and call terminate_broken() in this case.
Add test_python_finalization_error() to test_concurrent_futures.
* terminate_broken() no longer calls shutdown_workers() since the
call queue is no longer working anymore (read and write ends of
the queue pipe are closed).
* terminate_broken() now terminates child processes, not only
wait until they complete.
* _ExecutorManagerThread.terminate_broken() now holds shutdown_lock
to prevent race conditons with ProcessPoolExecutor.submit().
multiprocessing.Queue changes:
* Add _terminate_broken() method.
* _start_thread() sets _thread to None on exception to prevent
leaking "dangling threads" even if the thread was not started
yet.
On Windows, multiprocessing Popen.terminate() now catchs
PermissionError and get the process exit code. If the process is
still running, raise again the PermissionError. Otherwise, the
process terminated as expected: store its exit code.
(cherry picked from commit bd4518c60c9df356cf5e05b81305e3644ebb5e70)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] gh-109960: Remove test_pty timeout of 10 seconds (GH-110058) (#110060)
gh-109960: Remove test_pty timeout of 10 seconds (GH-110058)
In 2003, test_pty got a hardcoded timeout of 10 seconds to prevent
hanging on AIX & HPUX "if run after test_openpty":
commit 7d8145268ee282f14d6adce9305dc3c1c7ffec14. Since 2003, test_pty
was no longer reported to hang on AIX. But today, the test can fail
simply because a CI is busy running other tests in parallel.
The timeout of 10 seconds is no longer needed, just remove it.
Moreover, regrtest now has multiple built-in generic timeout
mecanisms.
(cherry picked from commit 5fdcea744024c8a19ddb57057bf5ec2889546c98)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] gh-109889: fix compiler's redundant NOP detection to look past NOPs with no lineno when looking for the next instruction's lineno (GH-109987) (#110048)
gh-109889: fix compiler's redundant NOP detection to look past NOPs with no lineno when looking for the next instruction's lineno (GH-109987)
(cherry picked from commit f580edcc6a4c528020afe46c753db713474acad6)
[3.12] gh-110033: Fix signal test_interprocess_signal() (GH-110035) (#110040)
gh-110033: Fix signal test_interprocess_signal() (GH-110035)
Fix test_interprocess_signal() of test_signal. Make sure that the
subprocess.Popen object is deleted before the test raising an
exception in a signal handler. Otherwise, Popen.__del__() can get the
exception which is logged as "Exception ignored in: ...." and the
test fails.
(cherry picked from commit 7e0fbf5175fcf21dae390ba68b7f49706d62aa49)
Co-authored-by: Victor Stinner <vstinner@python.org>
Fix test_timeout() of test_concurrent_futures.test_wait. Remove the
future which may or may not complete depending if it takes longer
than the timeout ot not. Keep the second future which does not
complete before wait(). Make also the test faster: 0.5 second instead
of 6 seconds, so remove @support.requires_resource('walltime')
decorator.
(cherry picked from commit 9be283e5e15d5d5685b78a38eb132501f7f3febb)
Co-authored-by: Victor Stinner <vstinner@python.org>
as_completed() uses a timeout of 100 ms instead of 10 ms. Windows
monotonic clock resolution is around 15.6 ms.
(cherry picked from commit b1aebf1e6576680d606068d17e2208259573e061)
Co-authored-by: Victor Stinner <vstinner@python.org>
Fix copy_source_tree() function of test_tools.test_freeze:
* Don't copy SRC_DIR/build/ anymore. This directory is modified by
other tests running in parallel.
* Add test.support.copy_python_src_ignore().
* Use sysconfig to get the source directory.
* Use sysconfig.get_config_var() to get CONFIG_ARGS variable.
[3.12] gh-109098: Fuzz re module instead of internal sre (GH-109911) (#109932)
gh-109098: Fuzz re module instead of internal sre (GH-109911)
* gh-109098: Fuzz re module instead of internal sre
* Fix c-analyzer globals test failure
* Put globals exception in ignored.tsv
(cherry picked from commit a829356f86d597e4dfe92e236a6d711c8a464f16)
[3.12] gh-109845: Make test_ftplib more stable under load (GH-109912) (#109919)
gh-109845: Make test_ftplib more stable under load (GH-109912)
recv() can return partial data cut in the middle of a multibyte
character. Test raw binary data instead of data incorrectly decoded by parts.
(cherry picked from commit 2ef2fffe3be953b91852585c75188d5475b09474)
test_error_at_task_unpickle() and
test_error_during_result_unpickle_in_result_handler() now restore
sys.stderr which is overriden by _raise_error_ignore_stderr().
(cherry picked from commit 2897142d2ec0930a8991af964c798b68fb6dcadd)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] gh-109631: Allow interruption of short repeated regex matches (GH-109867) (#109886)
gh-109631: Allow interruption of short repeated regex matches (GH-109867)
Counting for signal checking now continues in new match from the point where
it ended in the previous match instead of starting from 0.
(cherry picked from commit 8ac2085b80eca4d9b2a1093d0a7da020fd12e11a)
[3.12] gh-89363: Skip threading test_is_alive_after_fork() if ASAN (GH-109835) (#109855)
gh-89363: Skip threading test_is_alive_after_fork() if ASAN (GH-109835)
Skip test_is_alive_after_fork() of test_threading if Python is built
with Address Sanitizer (ASAN).
(cherry picked from commit bc06743533b5fea2d5ecdad6dd3caa372c67439f)
Co-authored-by: Victor Stinner <vstinner@python.org>
Expect the test to be "short" but don't measure the exact performance
of the CI. SHORT_TIMEOUT is about 30 seconds whereas the cancelled
coroutine takes around 1 hour.
(cherry picked from commit f29bc9c9a0a6794c6b8a9e84a7ba9237b427a10a)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] gh-109521: Fix obscure cases handling in PyImport_GetImporter() (GH-109522) (#109777)
gh-109521: Fix obscure cases handling in PyImport_GetImporter() (GH-109522)
PyImport_GetImporter() now sets RuntimeError if it fails to get sys.path_hooks
or sys.path_importer_cache or they are not list and dict correspondingly.
Previously it could return NULL without setting error in obscure cases,
crash or raise SystemError if these attributes have wrong type.
(cherry picked from commit 62c7015e89cbdedb5218d4fedd45f971885f67a8)
Replace harcoded sleep of 500 ms with synchronization using a pipe.
Fix also Process._feed_stdin(): catch also BrokenPipeError on
stdin.write(input), not only on stdin.drain().
(cherry picked from commit cbbdf2c1440c804adcfc32ea0470865b3b3b8eb2)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] gh-109582: test_fork_signal_handling should wait for event (GH-109605) (#109695)
gh-109582: test_fork_signal_handling should wait for event (GH-109605)
Sometimes the child_handled event was missing because either
the child quits before it gets a chance to handle the signal,
or the parent asserts before the event notification is
delivered via IPC. Synchronize explicitly to avoid this.
(cherry picked from commit 608c1f3083ea1e06d383ef1a9878a9758903de4b)
Move test_dataclasses.py and its "dataclass_*.py" modules into the
new Lib/test/test_dataclasses/ subdirectory.
(cherry picked from commit 14d6e197cc56e5256d501839a4e66e3864ab15f0)
Co-authored-by: Victor Stinner <vstinner@python.org>
* Fix Lint job: update Lib/test/.ruff.toml
---------
Co-authored-by: Victor Stinner <vstinner@python.org>
In Python 2.7, the file was used by Lib/test/test_sgmllib.py to test
Lib/sgmllib.py. The sgmllib module and its tests have been removed in
Python 3.0.
(cherry picked from commit d2879f2095abd5c8186c7f69c964a341c2053572)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] gh-109613: _pystat_fromstructstat() checks for exceptions (GH-109618) (#109641)
gh-109613: _pystat_fromstructstat() checks for exceptions (GH-109618)
Fix os.stat() and os.DirEntry.stat(): check for exceptions.
Previously, on Python built in debug mode, these functions could
trigger a fatal Python error (and abort the process) when a function
succeeded with an exception set.
_pystat_fromstructstat() now exits immediately if an exception is
raised, rather only checking for exceptions at the end. It fix
following fatal error in fill_time():
Fatal Python error: _Py_CheckSlotResult:
Slot * of type int succeeded with an exception set
(cherry picked from commit d4cea794a7b9b745817d2bd982d35412aef04710)
Co-authored-by: Victor Stinner <vstinner@python.org>
Victor Stinner [Mon, 2 Oct 2023 15:04:19 +0000 (17:04 +0200)]
[3.12] gh-103053: Skip test_freeze_simple_script() on PGO build (#109591) (#109614)
gh-103053: Skip test_freeze_simple_script() on PGO build (#109591)
Skip test_freeze_simple_script() of test_tools.test_freeze if Python
is built with "./configure --enable-optimizations", which means with
Profile Guided Optimization (PGO): it just makes the test too slow.
The freeze tool is tested by many other CIs with other (faster)
compiler flags.
test.pythoninfo now gets also get_build_info() of
test.libregrtests.utils.
SubprocessProtocol process_exited() method can be called before
pipe_data_received() and pipe_connection_lost() methods. Document it
and adapt the test for that.
Revert commit 282edd7b2a74c4dfe1bfe3c5b1d30f9c21d554d6.
_child_watcher_callback() calls immediately _process_exited(): don't
add an additional delay with call_soon(). The reverted change didn't
make _process_exited() more determistic: it can still be called
before pipe_connection_lost() for example.
[3.12] gh-109580: Skip test_perf_profiler on ASAN build (GH-109584) (#109585)
gh-109580: Skip test_perf_profiler on ASAN build (GH-109584)
Skip test_perf_profiler if Python is built with ASAN, MSAN or UBSAN
sanitizer. Python does crash randomly in this test on such build.
(cherry picked from commit 754519a9f8c2bb06d85ff9b3e9fe6f967ac46d5c)
Co-authored-by: Victor Stinner <vstinner@python.org>
Fix a race condition in concurrent.futures. When a process in the
process pool was terminated abruptly (while the future was running or
pending), close the connection write end. If the call queue is
blocked on sending bytes to a worker process, closing the connection
write end interrupts the send, so the queue can be closed.
Changes:
* _ExecutorManagerThread.terminate_broken() now closes
call_queue._writer.
* multiprocessing PipeConnection.close() now interrupts
WaitForMultipleObjects() in _send_bytes() by cancelling the
overlapped operation.
(cherry picked from commit a9b1f84790e977fb09f75b148c4c4f5924a6ef99)
Co-authored-by: Victor Stinner <vstinner@python.org>
[3.12] gh-50644: Forbid pickling of codecs streams (GH-109180) (#109231)
gh-50644: Forbid pickling of codecs streams (GH-109180)
Attempts to pickle or create a shallow or deep copy of codecs streams
now raise a TypeError.
Previously, copying failed with a RecursionError, while pickling
produced wrong results that eventually caused unpickling to fail with
a RecursionError.
(cherry picked from commit d6892c2b9263b39ea1c7905667942914b6a24b2c)
Victor Stinner [Mon, 2 Oct 2023 14:55:06 +0000 (16:55 +0200)]
[3.12] gh-104690: thread_run() checks for tstate dangling pointer (#109056) (#109133)
gh-104690: thread_run() checks for tstate dangling pointer (#109056)
thread_run() of _threadmodule.c now calls
_PyThreadState_CheckConsistency() to check if tstate is a dangling
pointer when Python is built in debug mode.
Rename ceval_gil.c is_tstate_valid() to
_PyThreadState_CheckConsistency() to reuse it in _threadmodule.c.
projects / thirdparty / Python / cpython.git / log
