]> git.ipfire.org Git - thirdparty/Python/cpython.git/commit
[3.12] gh-113993: Make interned strings mortal (GH-120520, GH-121364, GH-121903,...
authorPetr Viktorin <encukou@gmail.com>
Fri, 27 Sep 2024 20:28:48 +0000 (22:28 +0200)
committerGitHub <noreply@github.com>
Fri, 27 Sep 2024 20:28:48 +0000 (13:28 -0700)
commit49f6beb56a64dd0b688e8c728928a38eb6d78fef
tree09101654eca8274bbf2f9a3120ae55360f692757
parent2fa9ca50704d17a3357267caed36b910f82a4bb1
[3.12] gh-113993: Make interned strings mortal (GH-120520, GH-121364, GH-121903, GH-122303) (#123065)

This backports several PRs for gh-113993, making interned strings mortal so they can be garbage-collected when no longer needed.

* Allow interned strings to be mortal, and fix related issues (GH-120520)

  * Add an InternalDocs file describing how interning should work and how to use it.

  * Add internal functions to *explicitly* request what kind of interning is done:
    - `_PyUnicode_InternMortal`
    - `_PyUnicode_InternImmortal`
    - `_PyUnicode_InternStatic`

  * Switch uses of `PyUnicode_InternInPlace` to those.

  * Disallow using `_Py_SetImmortal` on strings directly.
    You should use `_PyUnicode_InternImmortal` instead:
    - Strings should be interned before immortalization, otherwise you're possibly
      interning a immortalizing copy.
    - `_Py_SetImmortal` doesn't handle the `SSTATE_INTERNED_MORTAL` to
      `SSTATE_INTERNED_IMMORTAL` update, and those flags can't be changed in
      backports, as they are now part of public API and version-specific ABI.

  * Add private `_only_immortal` argument for `sys.getunicodeinternedsize`, used in refleak test machinery.

   Make sure the statically allocated string singletons are unique. This means these sets are now disjoint:
    - `_Py_ID`
    - `_Py_STR` (including the empty string)
    - one-character latin-1 singletons

    Now, when you intern a singleton, that exact singleton will be interned.

  * Add a `_Py_LATIN1_CHR` macro, use it instead of `_Py_ID`/`_Py_STR` for one-character latin-1 singletons everywhere (including Clinic).

  * Intern `_Py_STR` singletons at startup.

  * Beef up the tests. Cover internal details (marked with `@cpython_only`).

  * Add lots of assertions

* Don't immortalize in PyUnicode_InternInPlace; keep immortalizing in other API (GH-121364)

  * Switch PyUnicode_InternInPlace to _PyUnicode_InternMortal, clarify docs

  * Document immortality in some functions that take `const char *`

  This is PyUnicode_InternFromString;
  PyDict_SetItemString, PyObject_SetAttrString;
  PyObject_DelAttrString; PyUnicode_InternFromString;
  and the PyModule_Add convenience functions.

  Always point out a non-immortalizing alternative.

  * Don't immortalize user-provided attr names in _ctypes

* Immortalize names in code objects to avoid crash (GH-121903)

* Intern latin-1 one-byte strings at startup (GH-122303)

There are some 3.12-specific changes, mainly to allow statically allocated strings in deepfreeze. (In 3.13, deepfreeze switched to the general `_Py_ID`/`_Py_STR`.)

Co-authored-by: Eric Snow <ericsnowcurrently@gmail.com>
51 files changed:
Doc/c-api/module.rst
Doc/c-api/object.rst
Doc/c-api/unicode.rst
Doc/data/python3.12.abi
Include/internal/pycore_global_objects_fini_generated.h
Include/internal/pycore_global_strings.h
Include/internal/pycore_object.h
Include/internal/pycore_runtime_init_generated.h
Include/internal/pycore_unicodeobject.h
Include/internal/pycore_unicodeobject_generated.h
InternalDocs/string_interning.md [new file with mode: 0644]
Lib/test/clinic.test.c
Lib/test/libregrtest/refleak.py
Lib/test/test_clinic.py
Lib/test/test_scope.py
Lib/test/test_sys.py
Misc/NEWS.d/next/C API/2024-07-04-15-41-10.gh-issue-113993.cLSiWV.rst [new file with mode: 0644]
Misc/NEWS.d/next/Core and Builtins/2024-06-14-22-02-25.gh-issue-113993.MiA0vX.rst [new file with mode: 0644]
Modules/_ctypes/_ctypes.c
Modules/_io/stringio.c
Modules/_operator.c
Modules/_pickle.c
Modules/_sqlite/clinic/connection.c.h
Modules/clinic/_bisectmodule.c.h
Modules/clinic/_hashopenssl.c.h
Modules/clinic/_testclinic.c.h
Modules/clinic/_testmultiphase.c.h
Modules/clinic/cmathmodule.c.h
Modules/clinic/itertoolsmodule.c.h
Modules/clinic/mathmodule.c.h
Modules/clinic/posixmodule.c.h
Objects/codeobject.c
Objects/dictobject.c
Objects/object.c
Objects/typeobject.c
Objects/unicodeobject.c
PC/clinic/_testconsole.c.h
Parser/action_helpers.c
Parser/pegen.c
Programs/test_frozenmain.h
Python/ast_opt.c
Python/ast_unparse.c
Python/clinic/sysmodule.c.h
Python/codecs.c
Python/compile.c
Python/getargs.c
Python/marshal.c
Python/pylifecycle.c
Python/sysmodule.c
Tools/build/generate_global_objects.py
Tools/clinic/clinic.py