From: Pablo Galindo Date: Mon, 24 Oct 2022 19:29:24 +0000 (+0100) Subject: Merge tag 'v3.11.0' into 3.11 X-Git-Tag: v3.11.1~219 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=69b6b56d857440183e227ca0b10c84bca4239985;p=thirdparty%2FPython%2Fcpython.git Merge tag 'v3.11.0' into 3.11 Python 3.11.0 --- 69b6b56d857440183e227ca0b10c84bca4239985 diff --cc Doc/library/venv.rst index adc6cd339ac1,3bed25645a92..e0f2f2babfb0 --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@@ -44,91 -36,68 +44,104 @@@ Creating virtual environment .. include:: /using/venv-create.inc +.. _venv-explanation: -.. _venv-def: +How venvs work +-------------- -.. note:: A virtual environment is a Python environment such that the Python - interpreter, libraries and scripts installed into it are isolated from those - installed in other virtual environments, and (by default) any libraries - installed in a "system" Python, i.e., one which is installed as part of your - operating system. - - A virtual environment is a directory tree which contains Python executable - files and other files which indicate that it is a virtual environment. - - Common installation tools such as setuptools_ and pip_ work as - expected with virtual environments. In other words, when a virtual - environment is active, they install Python packages into the virtual - environment without needing to be told to do so explicitly. - - When a virtual environment is active (i.e., the virtual environment's Python - interpreter is running), the attributes :attr:`sys.prefix` and - :attr:`sys.exec_prefix` point to the base directory of the virtual - environment, whereas :attr:`sys.base_prefix` and - :attr:`sys.base_exec_prefix` point to the non-virtual environment Python - installation which was used to create the virtual environment. If a virtual - environment is not active, then :attr:`sys.prefix` is the same as - :attr:`sys.base_prefix` and :attr:`sys.exec_prefix` is the same as - :attr:`sys.base_exec_prefix` (they all point to a non-virtual environment - Python installation). - - When a virtual environment is active, any options that change the - installation path will be ignored from all :mod:`distutils` configuration - files to prevent projects being inadvertently installed outside of the - virtual environment. - - When working in a command shell, users can make a virtual environment active - by running an ``activate`` script in the virtual environment's executables - directory (the precise filename and command to use the file is - shell-dependent), which prepends the virtual environment's directory for - executables to the ``PATH`` environment variable for the running shell. There - should be no need in other circumstances to activate a virtual - environment; scripts installed into virtual environments have a "shebang" - line which points to the virtual environment's Python interpreter. This means - that the script will run with that interpreter regardless of the value of - ``PATH``. On Windows, "shebang" line processing is supported if you have the - Python Launcher for Windows installed (this was added to Python in 3.3 - see - :pep:`397` for more details). Thus, double-clicking an installed script in a - Windows Explorer window should run the script with the correct interpreter - without there needing to be any reference to its virtual environment in - ``PATH``. +When a Python interpreter is running from a virtual environment, +:data:`sys.prefix` and :data:`sys.exec_prefix` +point to the directories of the virtual environment, +whereas :data:`sys.base_prefix` and :data:`sys.base_exec_prefix` +point to those of the base Python used to create the environment. +It is sufficient to check +``sys.prefix == sys.base_prefix`` to determine if the current interpreter is +running from a virtual environment. + +A virtual environment may be "activated" using a script in its binary directory +(``bin`` on POSIX; ``Scripts`` on Windows). +This will prepend that directory to your :envvar:`!PATH`, so that running +:program:`!python` will invoke the environment's Python interpreter +and you can run installed scripts without having to use their full path. +The invocation of the activation script is platform-specific +(:samp:`{}` must be replaced by the path to the directory +containing the virtual environment): + ++-------------+------------+--------------------------------------------------+ +| Platform | Shell | Command to activate virtual environment | ++=============+============+==================================================+ +| POSIX | bash/zsh | :samp:`$ source {}/bin/activate` | +| +------------+--------------------------------------------------+ +| | fish | :samp:`$ source {}/bin/activate.fish` | +| +------------+--------------------------------------------------+ +| | csh/tcsh | :samp:`$ source {}/bin/activate.csh` | +| +------------+--------------------------------------------------+ +| | PowerShell | :samp:`$ {}/bin/Activate.ps1` | ++-------------+------------+--------------------------------------------------+ +| Windows | cmd.exe | :samp:`C:\\> {}\\Scripts\\activate.bat` | +| +------------+--------------------------------------------------+ +| | PowerShell | :samp:`PS C:\\> {}\\Scripts\\Activate.ps1` | ++-------------+------------+--------------------------------------------------+ + +.. versionadded:: 3.4 + :program:`!fish` and :program:`!csh` activation scripts. + +.. versionadded:: 3.8 + PowerShell activation scripts installed under POSIX for PowerShell Core + support. + +You don't specifically *need* to activate a virtual environment, +as you can just specify the full path to that environment's +Python interpreter when invoking Python. +Furthermore, all scripts installed in the environment +should be runnable without activating it. + +In order to achieve this, scripts installed into virtual environments have +a "shebang" line which points to the environment's Python interpreter, +i.e. :samp:`#!/{}/bin/python`. +This means that the script will run with that interpreter regardless of the +value of :envvar:`!PATH`. On Windows, "shebang" line processing is supported if +you have the :ref:`launcher` installed. Thus, double-clicking an installed +script in a Windows Explorer window should run it with the correct interpreter +without the environment needing to be activated or on the :envvar:`!PATH`. + +When a virtual environment has been activated, the :envvar:`!VIRTUAL_ENV` +environment variable is set to the path of the environment. +Since explicitly activating a virtual environment is not required to use it, +:envvar:`!VIRTUAL_ENV` cannot be relied upon to determine +whether a virtual environment is being used. + +.. warning:: Because scripts installed in environments should not expect the + environment to be activated, their shebang lines contain the absolute paths + to their environment's interpreters. Because of this, environments are + inherently non-portable, in the general case. You should always have a + simple means of recreating an environment (for example, if you have a + requirements file ``requirements.txt``, you can invoke ``pip install -r + requirements.txt`` using the environment's ``pip`` to install all of the + packages needed by the environment). If for any reason you need to move the + environment to a new location, you should recreate it at the desired + location and delete the one at the old location. If you move an environment + because you moved a parent directory of it, you should recreate the + environment in its new location. Otherwise, software installed into the + environment may not work as expected. + +You can deactivate a virtual environment by typing ``deactivate`` in your shell. +The exact mechanism is platform-specific and is an internal implementation +detail (typically, a script or shell function will be used). + .. warning:: Because scripts installed in environments should not expect the + environment to be activated, their shebang lines contain the absolute paths + to their environment's interpreters. Because of this, environments are + inherently non-portable, in the general case. You should always have a + simple means of recreating an environment (for example, if you have a + requirements file ``requirements.txt``, you can invoke ``pip install -r + requirements.txt`` using the environment's ``pip`` to install all of the + packages needed by the environment). If for any reason you need to move the + environment to a new location, you should recreate it at the desired + location and delete the one at the old location. If you move an environment + because you moved a parent directory of it, you should recreate the + environment in its new location. Otherwise, software installed into the + environment may not work as expected. .. _venv-api: diff --cc Doc/whatsnew/3.11.rst index 7aa071cc97cc,8f3ef3ffc135..b42c6529303b --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@@ -1494,102 -1481,60 +1494,106 @@@ contributors are volunteers from the co CPython bytecode changes ======================== -* The bytecode now contains inline cache entries, which take the form of - :opcode:`CACHE` instructions. Many opcodes expect to be followed by an exact - number of caches, and instruct the interpreter to skip over them at runtime. - Populated caches can look like arbitrary instructions, so great care should be - taken when reading or modifying raw, adaptive bytecode containing quickened - data. +The bytecode now contains inline cache entries, +which take the form of the newly-added :opcode:`CACHE` instructions. +Many opcodes expect to be followed by an exact number of caches, +and instruct the interpreter to skip over them at runtime. +Populated caches can look like arbitrary instructions, +so great care should be taken when reading or modifying +raw, adaptive bytecode containing quickened data. -* Replaced all numeric ``BINARY_*`` and ``INPLACE_*`` instructions with a single - :opcode:`BINARY_OP` implementation. -* Replaced the three call instructions: :opcode:`CALL_FUNCTION`, - :opcode:`CALL_FUNCTION_KW` and :opcode:`CALL_METHOD` with - :opcode:`PUSH_NULL`, :opcode:`PRECALL`, :opcode:`CALL`, - and :opcode:`KW_NAMES`. - This decouples the argument shifting for methods from the handling of - keyword arguments and allows better specialization of calls. +.. _whatsnew311-added-opcodes: -* Removed ``COPY_DICT_WITHOUT_KEYS`` and ``GEN_START``. +New opcodes +----------- + +* :opcode:`ASYNC_GEN_WRAP`, :opcode:`RETURN_GENERATOR` and :opcode:`SEND`, + used in generators and co-routines. -* :opcode:`MATCH_CLASS` and :opcode:`MATCH_KEYS` no longer push an additional - boolean value indicating whether the match succeeded or failed. Instead, they - indicate failure with :const:`None` (where a tuple of extracted values would - otherwise be). +* :opcode:`COPY_FREE_VARS`, + which avoids needing special caller-side code for closures. -* Replace several stack manipulation instructions (``DUP_TOP``, ``DUP_TOP_TWO``, - ``ROT_TWO``, ``ROT_THREE``, ``ROT_FOUR``, and ``ROT_N``) with new - :opcode:`COPY` and :opcode:`SWAP` instructions. +* :opcode:`JUMP_BACKWARD_NO_INTERRUPT`, + for use in certain loops where handling interrupts is undesirable. -* Replaced :opcode:`JUMP_IF_NOT_EXC_MATCH` by :opcode:`CHECK_EXC_MATCH` which - performs the check but does not jump. +* :opcode:`MAKE_CELL`, to create :ref:`cell-objects`. -* Replaced :opcode:`JUMP_IF_NOT_EG_MATCH` by :opcode:`CHECK_EG_MATCH` which - performs the check but does not jump. +* :opcode:`CHECK_EG_MATCH` and :opcode:`PREP_RERAISE_STAR`, + to handle the :ref:`new exception groups and except* ` + added in :pep:`654`. -* Replaced :opcode:`JUMP_ABSOLUTE` by the relative :opcode:`JUMP_BACKWARD`. +* :opcode:`PUSH_EXC_INFO`, for use in exception handlers. -* Added :opcode:`JUMP_BACKWARD_NO_INTERRUPT`, which is used in certain loops where it - is undesirable to handle interrupts. +* :opcode:`RESUME`, a no-op, + for internal tracing, debugging and optimization checks. -* Replaced :opcode:`POP_JUMP_IF_TRUE` and :opcode:`POP_JUMP_IF_FALSE` by - the relative :opcode:`POP_JUMP_FORWARD_IF_TRUE`, :opcode:`POP_JUMP_BACKWARD_IF_TRUE`, - :opcode:`POP_JUMP_FORWARD_IF_FALSE` and :opcode:`POP_JUMP_BACKWARD_IF_FALSE`. -* Added :opcode:`POP_JUMP_FORWARD_IF_NOT_NONE`, :opcode:`POP_JUMP_BACKWARD_IF_NOT_NONE`, - :opcode:`POP_JUMP_FORWARD_IF_NONE` and :opcode:`POP_JUMP_BACKWARD_IF_NONE` - opcodes to speed up conditional jumps. +.. _whatsnew311-replaced-opcodes: -* :opcode:`JUMP_IF_TRUE_OR_POP` and :opcode:`JUMP_IF_FALSE_OR_POP` are now - relative rather than absolute. +Replaced opcodes +---------------- -* :opcode:`RESUME` has been added. It is a no-op. Performs internal tracing, - debugging and optimization checks. ++------------------------------------+-----------------------------------+-----------------------------------------+ +| Replaced Opcode(s) | New Opcode(s) | Notes | ++====================================+===================================+=========================================+ +| | :opcode:`!BINARY_*` | :opcode:`BINARY_OP` | Replaced all numeric binary/in-place | +| | :opcode:`!INPLACE_*` | | opcodes with a single opcode | ++------------------------------------+-----------------------------------+-----------------------------------------+ +| | :opcode:`!CALL_FUNCTION` | | :opcode:`CALL` | Decouples argument shifting for methods | +| | :opcode:`!CALL_FUNCTION_KW` | | :opcode:`KW_NAMES` | from handling of keyword arguments; | +| | :opcode:`!CALL_METHOD` | | :opcode:`PRECALL` | allows better specialization of calls | +| | | :opcode:`PUSH_NULL` | | ++------------------------------------+-----------------------------------+-----------------------------------------+ +| | :opcode:`!DUP_TOP` | | :opcode:`COPY` | Stack manipulation instructions | +| | :opcode:`!DUP_TOP_TWO` | | :opcode:`SWAP` | | +| | :opcode:`!ROT_TWO` | | | +| | :opcode:`!ROT_THREE` | | | +| | :opcode:`!ROT_FOUR` | | | +| | :opcode:`!ROT_N` | | | ++------------------------------------+-----------------------------------+-----------------------------------------+ +| | :opcode:`!JUMP_IF_NOT_EXC_MATCH` | | :opcode:`CHECK_EXC_MATCH` | Now performs check but doesn't jump | ++------------------------------------+-----------------------------------+-----------------------------------------+ +| | :opcode:`!JUMP_ABSOLUTE` | | :opcode:`JUMP_BACKWARD` | See [#bytecode-jump]_; | +| | :opcode:`!POP_JUMP_IF_FALSE` | | :opcode:`POP_JUMP_BACKWARD_IF_* | ``TRUE``, ``FALSE``, | +| | :opcode:`!POP_JUMP_IF_TRUE` | ` | ``NONE`` and ``NOT_NONE`` variants | +| | | :opcode:`POP_JUMP_FORWARD_IF_* | for each direction | +| | ` | | ++------------------------------------+-----------------------------------+-----------------------------------------+ +| | :opcode:`!SETUP_WITH` | :opcode:`BEFORE_WITH` | :keyword:`with` block setup | +| | :opcode:`!SETUP_ASYNC_WITH` | | | ++------------------------------------+-----------------------------------+-----------------------------------------+ + +.. [#bytecode-jump] All jump opcodes are now relative, including the + existing :opcode:`JUMP_IF_TRUE_OR_POP` and :opcode:`JUMP_IF_FALSE_OR_POP`. + The argument is now an offset from the current instruction + rather than an absolute location. + + +.. _whatsnew311-changed-opcodes: +.. _whatsnew311-removed-opcodes: +.. _whatsnew311-changed-removed-opcodes: + +Changed/removed opcodes +----------------------- + +* Changed :opcode:`MATCH_CLASS` and :opcode:`MATCH_KEYS` + to no longer push an additional boolean value to indicate success/failure. + Instead, ``None`` is pushed on failure + in place of the tuple of extracted values. + +* Changed opcodes that work with exceptions to reflect them + now being represented as one item on the stack instead of three + (see :gh:`89874`). + +* Removed :opcode:`!COPY_DICT_WITHOUT_KEYS`, :opcode:`!GEN_START`, + :opcode:`!POP_BLOCK`, :opcode:`!SETUP_FINALLY` and :opcode:`!YIELD_FROM`. + + ++.. _whatsnew311-deprecated: ++.. _whatsnew311-python-api-deprecated: + + .. _whatsnew311-deprecated: .. _whatsnew311-python-api-deprecated: @@@ -1741,30 -1678,12 +1745,33 @@@ Standard Librar is now deprecated. Support will be removed in Python 3.13. (Contributed by Jingchen Ye in :gh:`90224`.) -* The :func:`re.template` function and the corresponding :const:`re.TEMPLATE` - and :const:`re.T` flags are deprecated, as they were undocumented and - lacked an obvious purpose. They will be removed in Python 3.13. - (Contributed by Serhiy Storchaka and Miro Hrončok in :gh:`92728`.) +* :class:`!webbrowser.MacOSX` is deprecated and will be removed in Python 3.13. + It is untested, undocumented, and not used by :mod:`webbrowser` itself. + (Contributed by Dong-hee Na in :issue:`42255`.) + +* The behavior of returning a value from a :class:`~unittest.TestCase` and + :class:`~unittest.IsolatedAsyncioTestCase` test methods (other than the + default ``None`` value) is now deprecated. + +* Deprecated the following not-formally-documented :mod:`unittest` functions, + scheduled for removal in Python 3.13: + + * :func:`!unittest.findTestCases` + * :func:`!unittest.makeSuite` + * :func:`!unittest.getTestCaseNames` + + Use :class:`~unittest.TestLoader` methods instead: + * :meth:`unittest.TestLoader.loadTestsFromModule` + * :meth:`unittest.TestLoader.loadTestsFromTestCase` + * :meth:`unittest.TestLoader.getTestCaseNames` + + (Contributed by Erlend E. Aasland in :issue:`5846`.) + + ++.. _whatsnew311-pending-removal: ++.. _whatsnew311-python-api-pending-removal: + .. _whatsnew311-pending-removal: .. _whatsnew311-python-api-pending-removal: diff --cc Include/patchlevel.h index a54a2ce6c70d,627401409a1f..c62cdf19e165 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@@ -19,11 -19,11 +19,11 @@@ #define PY_MAJOR_VERSION 3 #define PY_MINOR_VERSION 11 #define PY_MICRO_VERSION 0 - #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_GAMMA - #define PY_RELEASE_SERIAL 2 + #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_FINAL + #define PY_RELEASE_SERIAL 0 /* Version as a string */ -#define PY_VERSION "3.11.0" +#define PY_VERSION "3.11.0rc2+" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --cc Lib/test/test_frame.py index 4b86a60d2f4c,e153bc5c7c73..2222ad6ed514 --- a/Lib/test/test_frame.py +++ b/Lib/test/test_frame.py @@@ -241,27 -239,26 +241,91 @@@ class ReprTest(unittest.TestCase) class TestIncompleteFrameAreInvisible(unittest.TestCase): def test_issue95818(self): - #See GH-95818 for details - import gc - self.addCleanup(gc.set_threshold, *gc.get_threshold()) - - gc.set_threshold(1,1,1) - class GCHello: - def __del__(self): - print("Destroyed from gc") - - def gen(): - yield - - fd = open(__file__) - l = [fd, GCHello()] - l.append(l) - del fd - del l - gen() + # See GH-95818 for details + code = textwrap.dedent(f""" + import gc + + gc.set_threshold(1,1,1) + class GCHello: + def __del__(self): + print("Destroyed from gc") + + def gen(): + yield + + fd = open({__file__!r}) + l = [fd, GCHello()] + l.append(l) + del fd + del l + gen() + """) + assert_python_ok("-c", code) + ++ @support.cpython_only ++ def test_sneaky_frame_object(self): ++ ++ def trace(frame, event, arg): ++ """ ++ Don't actually do anything, just force a frame object to be created. ++ """ ++ ++ def callback(phase, info): ++ """ ++ Yo dawg, I heard you like frames, so I'm allocating a frame while ++ you're allocating a frame, so you can have a frame while you have a ++ frame! ++ """ ++ nonlocal sneaky_frame_object ++ sneaky_frame_object = sys._getframe().f_back ++ # We're done here: ++ gc.callbacks.remove(callback) ++ ++ def f(): ++ while True: ++ yield ++ ++ old_threshold = gc.get_threshold() ++ old_callbacks = gc.callbacks[:] ++ old_enabled = gc.isenabled() ++ old_trace = sys.gettrace() ++ try: ++ # Stop the GC for a second while we set things up: ++ gc.disable() ++ # Create a paused generator: ++ g = f() ++ next(g) ++ # Move all objects to the oldest generation, and tell the GC to run ++ # on the *very next* allocation: ++ gc.collect() ++ gc.set_threshold(1, 0, 0) ++ # Okay, so here's the nightmare scenario: ++ # - We're tracing the resumption of a generator, which creates a new ++ # frame object. ++ # - The allocation of this frame object triggers a collection ++ # *before* the frame object is actually created. ++ # - During the collection, we request the exact same frame object. ++ # This test does it with a GC callback, but in real code it would ++ # likely be a trace function, weakref callback, or finalizer. ++ # - The collection finishes, and the original frame object is ++ # created. We now have two frame objects fighting over ownership ++ # of the same interpreter frame! ++ sys.settrace(trace) ++ gc.callbacks.append(callback) ++ sneaky_frame_object = None ++ gc.enable() ++ next(g) ++ # g.gi_frame should be the the frame object from the callback (the ++ # one that was *requested* second, but *created* first): ++ self.assertIs(g.gi_frame, sneaky_frame_object) ++ finally: ++ gc.set_threshold(*old_threshold) ++ gc.callbacks[:] = old_callbacks ++ sys.settrace(old_trace) ++ if old_enabled: ++ gc.enable() + + @support.cpython_only def test_sneaky_frame_object(self):