--- /dev/null
+.. highlight:: c
+
+.. _abi3t-migration-howto:
+
+******************************************************
+Migrating to Stable ABI for free threading (``abi3t``)
+******************************************************
+
+Starting with the 3.15 release, CPython supports a variant of the Stable ABI
+that supports :term:`free-threaded <free threading>` Python:
+Stable ABI for Free-Threaded Builds, or ``abi3t`` for short.
+This document describes how to adapt C API extensions to support free threading.
+
+Why do this
+===========
+
+The typical reason to use Stable ABI is to reduce the number of artifacts that
+you need to build and distribute for each version of your library.
+
+Without the Stable ABI, you must build a separate shared library, and typically
+a *wheel* distribution, for each feature version of CPython you wish
+to support.
+For example, each tag in the following table represents a separate
+library/wheel:
+
++-----------------+-----------------------+------------------------+
+| CPython version | Non-free-threaded | Free-threaded |
++=================+=======================+========================+
+| 3.12 | ``cpython-312`` | --- |
++-----------------+-----------------------+------------------------+
+| 3.13 | ``cpython-313`` | ``cpython-313t`` |
++-----------------+-----------------------+------------------------+
+| 3.14 | ``cpython-314`` | ``cpython-314t`` |
++-----------------+-----------------------+------------------------+
+| 3.15 | ``cpython-315`` | ``cpython-315t`` |
++-----------------+-----------------------+------------------------+
+| 3.16 | ``cpython-316`` | ``cpython-316t`` |
++-----------------+-----------------------+------------------------+
+| Later versions | :samp:`cpython-3{XX}` | :samp:`cpython-3{XX}t` |
++-----------------+-----------------------+------------------------+
+
+That's a lot of builds, especially when multiplied by the number
+of supported platforms.
+
+With the Stable ABI (``abi3``, introduced in CPython 3.2), a single extension
+(per platform) can cover all *non-free-threaded* builds of CPython:
+
++-----------------+-------------------+------------------------+
+| CPython version | Non-free-threaded | Free-threaded |
++=================+===================+========================+
+| 3.12 | ``abi3`` | --- |
++-----------------+ +------------------------+
+| 3.13 | | ``cpython-313t`` |
++-----------------+ +------------------------+
+| 3.14 | | ``cpython-314t`` |
++-----------------+ +------------------------+
+| 3.15 | | ``cpython-315t`` |
++-----------------+ +------------------------+
+| 3.16 | | ``cpython-316t`` |
++-----------------+ +------------------------+
+| Later versions | | :samp:`cpython-3{XX}t` |
++-----------------+-------------------+------------------------+
+
+The Stable ABI for free-threaded builds (``abi3t``), introduced in
+CPython 3.15, does the same for free-threaded builds.
+And it's compatible with non-free-threaded ones as well:
+
++-----------------+-------------------+------------------+
+| CPython version | Non-free-threaded | Free-threaded |
++=================+===================+==================+
+| 3.12 | ``abi3`` * | --- |
++-----------------+ +------------------+
+| 3.13 | | ``cpython-313t`` |
++-----------------+ +------------------+
+| 3.14 | | ``cpython-314t`` |
++-----------------+-------------------+------------------+
+| 3.15 | ``abi3t`` |
++-----------------+ +
+| 3.16 | |
++-----------------+ +
+| Later versions | |
++-----------------+-------------------+------------------+
+
+\* (As above, the ``abi3`` extension is compatible with all non-free-threaded
+builds; even the 3.15+ ones that this table "attributes" to ``abi3t``.)
+
+Why *not* do this
+-----------------
+
+There are two main downsides to Stable ABI.
+
+First, you extension may become slower, since Stable ABI prioritizes
+compatibility over performance.
+The difference is usually not noticeable, and often can be mitigated by
+using the same source to build both a Stable ABI build and a few
+version-specific ones for "tier 1" CPython versions.
+
+Second, not all of the C API is available.
+Extensions need to be ported to build for Stable ABI, which may be difficult
+or, in rare cases, impossible.
+
+Specifically, ``abi3t`` requires APIs added in CPython 3.15.
+If you want to build your extension for older versions of CPython from the
+same source, you have two main options:
+
+- Use preprocessor conditionals.
+
+ When following this guide, use ``#ifdef Py_TARGET_ABI3T`` blocks whenever
+ you are told to do a change that breaks the build on CPython versions you
+ care about. Keep the pre-existing code in ``#else`` blocks.
+
+ For hand-written C extensions, this approach is reasonable down to
+ CPython 3.12, due to additions introduced in :pep:`697`.
+ Keeping compatibility with 3.11 and below may be worth it for code
+ generators (for example, Cython).
+
+- Do not port to ``abi3t``, and continue building separate extensions for
+ each version of CPython, until you can drop support for the older versions.
+
+ This is a valid approach. Not all extensions need to switch to ``abi3t``
+ right now.
+
+
+Prerequisites
+=============
+
+This guide assumes that you have an extension written directly in C (or C++),
+which you want to port to ``abi3t``.
+
+If your extenstion uses a code generator (like Cython) or language binding
+(like PyO3), it's best to wait until that tool has support for ``abi3t``.
+If you maintain such a tool, you might be able to adapt the instructions
+here for your tool.
+
+Non-free-threaded Stable ABI
+----------------------------
+
+Your extension should support the Stable ABI (``abi3t``).
+If not, either port it first, or follow this guide but be prepared to fix
+issues it does not mention.
+
+Free-threading support
+----------------------
+
+While it's technically not a hard prerequisite, you will most likely want to
+prepare your extension for free threading before you port it to ``abi3t``.
+See :ref:`freethreading-extensions-howto` for instructions.
+
+.. seealso::
+
+ `Porting Extension Modules to Support Free-Threading
+ <https://py-free-threading.github.io/porting/>`__:
+ A community-maintained porting guide for extension authors.
+
+Isolating extension modules
+---------------------------
+
+Your module should use :ref:`multi-phase initialization <multi-phase-initialization>`,
+and it should either be isolated or limit itself to be loaded at most once
+per process.
+If it is not your case, follow :ref:`isolating-extensions-howto` first.
+(See the :ref:`opt-out section <isolating-extensions-optout>` for a shortcut.)
+
+Avoiding variable-sized types
+-----------------------------
+
+If your extension defines variable-sized types (using :c:macro:`Py_tp_itemsize`
+or :c:member:`PyTypeObject.tp_itemsize`), it cannot be ported to
+``abi3t`` 3.15.
+
+
+Setting up the build
+====================
+
+If you use a build tool (such as setuptools, meson-python, scikit-build-core),
+search its documentation for a way to select ``abi3t``.
+At the time of writing, not all of them have this; but if your tool does,
+use it.
+You may want to verify that it set the right flag by temporarily adding the
+following just after ``#include <Python.h>``::
+
+ #if Py_TARGET_ABI3T+0 <= 0x30f0000
+ #error "abi3t define is not set!"
+ #endif
+
+This should result in a different error than "``abt3t`` define is not set".
+
+.. note::
+
+ If your build tool doesn't support ``abi3t`` yet, set the following macro
+ before including ``Python.h``::
+
+ #define Py_TARGET_ABI3T 0x30f0000
+
+ or specify it as a compiler flag, for example::
+
+ -DPy_TARGET_ABI3T=0x30f0000
+
+ Once your extension builds with this setting, it will be compatible with
+ CPython 3.15 and above.
+
+ If you set this macro manually, you will later need to name and tag the
+ resulting extension manually as well.
+ This is covered in :ref:`abi3t-migration-tagging` below.
+
+This guide will ask you to make a series of changes.
+After each one, verify that your extension still builds in the original
+(non-``abi3t``) configuration, and ideally run tests on all Python
+versions you support.
+This will ensure that nothing breaks as you are porting.
+
+
+Module export hook
+==================
+
+Unless you've done this step already, your extension module defines a
+:ref:`module initialization function <extension-pyinit>`
+named :samp:`PyInit_{<module_name>}`.
+You will need to port it to a :ref:`module export hook <extension-export-hook>`,
+:samp:`PyModExport_{<module name>}`, a feature added in CPython 3.15 in
+:pep:`793`.
+
+Your existing init function should look like this (with your own names
+for ``<modname>`` and ``<moddef>``):
+
+.. code-block::
+ :class: bad
+
+ PyMODINIT_FUNC
+ PyInit_<modname>(void)
+ {
+ return PyModuleDef_Init(&<moddef>);
+ }
+
+If there is some code before the ``return``, move it to
+a :c:macro:`Py_mod_create` or :c:macro:`Py_mod_exec` slot function.
+See the :ref:`PyInit documentation <extension-pyinit>` for related information.
+
+The function references a ``PyModuleDef`` object (``<moddef>`` in the code
+above).
+Its definition should be similar to the following, with different values
+and perhaps some fields unnnamed or left out:
+
+.. code-block::
+ :class: bad
+
+ static PyModuleDef <moddef> = {
+ PyModuleDef_HEAD_INIT,
+ .m_name = "my_module",
+ .m_doc = "my docstring",
+ .m_size = sizeof(my_state_struct),
+ .m_methods = my_methods,
+ .m_slots = my_slots,
+ .m_traverse = my_traverse,
+ .m_clear = my_clear,
+ .m_free = my_free,
+ };
+
+Remove this definition and the ``PyInit`` function (or put them in
+an ``#ifndef Py_TARGET_ABI3T`` block, to retain backwards compatibility),
+and replace them with the following:
+
+.. code-block::
+ :class: good
+
+ PyABIInfo_VAR(abi_info);
+
+ static PySlot my_slot_array[] = {
+ PySlot_STATIC_DATA(Py_mod_abi, &abi_info),
+ PySlot_STATIC_DATA(Py_mod_name, "my_module"),
+ PySlot_STATIC_DATA(Py_mod_doc, "my docstring"),
+ PySlot_SIZE(Py_mod_state_size, sizeof(my_state_struct)),
+ PySlot_STATIC_DATA(Py_mod_methods, my_methods),
+ PySlot_STATIC_DATA(Py_mod_slots, my_slots),
+ PySlot_FUNC(Py_mod_state_traverse, my_traverse),
+ PySlot_FUNC(Py_mod_state_clear, my_clear),
+ PySlot_FUNC(Py_mod_state_free, my_free),
+ PySlot_END
+ };
+
+ PyMODEXPORT_FUNC
+ PyModExport_<modname>(void)
+ {
+ return my_slot_array;
+ }
+
+Leave out any fields that were missing (except the new :c:macro:`Py_mod_abi`),
+and substitute your own values.
+
+See the :c:type:`PySlot` and :c:ref:`export hook <extension-export-hook>`
+documentation for details on this API.
+
+Associated ``PyModuleDef``
+--------------------------
+
+Since the new API does not use a :c:type:`!PyModuleDef` structure, a definition
+will not be associated with the resulting module.
+This changes the behavior of the following functions:
+
+- :c:func:`PyModule_GetDef`
+- :c:func:`PyType_GetModuleByDef`
+
+Check your code for these.
+If you do not use them, you can skip this section.
+
+These functions are typically used for two purposes:
+
+1. To get the definition the module was created with.
+ This is no longer possible using the new API.
+ Modules no longer keep a reference to the definition, so you will need to
+ figure out a different way to pass the relevant data around.
+
+.. _abi3t-migration-module-token:
+
+2. To check if a given module object is “yours”.
+ This use case is now served by :ref:`module tokens <ext-module-token>` --
+ opaque pointers that identify a module.
+ To use a token, declare (or reuse) a unique static variable, for example:
+
+ .. code-block::
+ :class: good
+
+ static char my_token;
+
+ and add a pointer to it in a new entry to your module's ``PySlot`` array:
+
+ .. code-block::
+ :class: good
+ :emphasize-lines: 3
+
+ static PySlot my_slot_array[] = {
+ ...
+ PySlot_STATIC_DATA(Py_mod_token, &my_token),
+ PySlot_END
+ }
+
+ Then, switch from :c:func:`PyModule_GetDef` calls such as:
+
+ .. code-block::
+ :class: bad
+
+ PyModuleDef *def = PyModule_GetDef(module);
+
+ to :c:func:`PyModule_GetToken` (which uses an output argument and may fail
+ with an exception):
+
+ .. code-block::
+ :class: good
+
+ void *token;
+ if (PyModule_GetToken(module, &token) < 0) {
+ /* handle error */
+ }
+
+ and from :c:func:`PyType_GetModuleByDef` calls such as:
+
+ .. code-block::
+ :class: bad
+
+ PyObject *module = PyType_GetModuleByDef(type, my_def);
+ /* handle error; use module */
+
+ to :c:func:`PyType_GetModuleByToken` (which returns a strong reference):
+
+ .. code-block::
+ :class: good
+
+ PyObject *module = PyType_GetModuleByToken(type, my_token);
+ /* handle error; use module */
+ Py_XDECREF(module);
+
+``PyObject`` opaqueness
+=======================
+
+The :c:type:`PyObject` and :c:type:`PyVarObject` structures are opaque
+in ``abi3t``.
+
+Accessing their members is prohibited.
+If you do this, switch to getter/setter functions mentioned in
+their documentation:
+
+- :c:member:`PyObject.ob_type`
+- :c:member:`PyObject.ob_refcnt`
+- :c:member:`PyVarObject.ob_size`
+
+Also, the *size* of the :c:type:`PyObject` structures is
+unknown to the compiler.
+It can -- and *does* -- change between different CPython builds.
+
+.. note::
+
+ While the size is available at runtime (for example as
+ ``sys.getsizeof(object())`` in Python code), you should resist the
+ temptation to calculate pointer offsets from it.
+ The object memory layout is subject to change in future
+ ``abi3t`` implementations.
+
+
+Custom type definitions
+-----------------------
+
+Since :c:type:`!PyObject` is opaque, the traditional way of defining
+custom types no longer works:
+
+.. code-block::
+ :class: bad
+
+ typedef struct {
+ PyObject_HEAD // expands to `PyObject ob_base;` which has unknown size
+
+ int my_data;
+ } CustomObject;
+
+ static PyType_Spec CustomType_spec = {
+ ...
+ .basicsize = sizeof(CustomObject),
+ ...
+ };
+
+Most likely, all your class definitions, *and* all code that accesses
+your classes' data, will need to be rewritten.
+This will probably be the biggest change you need to support ``abi3t``.
+
+For each such type, instead of defining a ``struct`` for the entire instance,
+define one with only the “additional” fields -- ones specific to your class,
+not its superclasses:
+
+.. code-block::
+ :class: good
+
+ typedef struct {
+ int my_data;
+ } CustomObjectData;
+
+Change the name.
+Almost all code that uses the struct will need to change
+(notably, pointers to the new structure cannot be cast to/from ``PyObject*``),
+and changing the name will highlight the usages as compiler errors.
+(If you use ``typeof``, C++ ``auto``, or similar ways to avoid
+typing the type name, this won't work. Be extra careful, and consider running
+tools to detect undefined behavior.)
+
+Then, to create the class, use *negative* ``basicsize`` to indicate
+“extra” storage space rather than *total* instance size:
+
+.. code-block::
+ :class: good
+
+ static PyType_Spec CustomType_spec = {
+ ...
+ .basicsize = -sizeof(CustomObjectData), /* note the minus sign */
+ ...
+ };
+
+If you use :c:macro:`Py_tp_members`, set the :c:macro:`Py_RELATIVE_OFFSET`
+flag on each member and specify the :c:member:`~PyMemberDef.offset`
+relative to your new struct.
+
+
+Custom type data access
+-----------------------
+
+Then comes the hard part: in all code that needs to access this struct,
+you will need an additional :c:func:`PyObject_GetTypeData` call to
+retrieve a ``CustomObjectData *`` pointer from ``PyObject *``:
+
+.. code-block::
+ :class: good
+
+ PyObject *obj = ...;
+ CustomObjectData *data = PyObject_GetTypeData(obj, cls);
+
+Note that this call requires the *type object* for your class (``cls``).
+
+If your class is not subclassable (that is, it does not use the
+:c:macro:`Py_TPFLAGS_BASETYPE` flag), ``cls`` will be ``Py_TYPE(obj)``.
+Otherwise, **DO NOT USE** ``Py_TYPE`` with :c:func:`!PyObject_GetTypeData`:
+it might return memory reserved to an unrelated subclass!
+For example, if a user makes a subclass like this:
+
+.. code-block:: python
+
+ class Sub(YourCustomClass):
+ __slots__ = ('a', 'b')
+
+then ``Py_TYPE(obj)`` is ``YourCustomClass``, and the underlying memory may
+look like this:
+
+.. code-block:: text
+
+ ╭─ PyObject *obj
+ │ ╭─ the pointer you want
+ │ │ ╭─ PyObject_GetTypeData(obj, Py_TYPE(obj))
+ ▼ ▼ ▼
+ ┌──────────┬───┬────────────────┬───┬─────────────┬───┬─────────────┐
+ │ PyObject │...│ CustomTypeData │...│ PyObject *a │...│ PyObject *b │
+ └──────────┴───┴────────────────┴───┴─────────────┴───┴─────────────┘
+
+(Ellipses indicate possible padding.
+Note that this memory layout is not guaranteed: future versions of Python may
+add different padding or even switch the order of the structures.)
+
+There are two main ways to get the right class:
+
+- In instance methods, your implementation may use the :c:type:`PyCMethod`
+ signature (and the :c:macro:`METH_METHOD` bit in
+ :c:member:`PyMethodDef.ml_flags`),
+ and get the class as the ``defining_class`` argument.
+- Otherwise, give your class a unique static token using the
+ :c:macro:`Py_tp_token` slot, and use:
+
+ .. code-block::
+ :class: good
+
+ PyTypeObject cls;
+ if (PyType_GetBaseByToken(Py_TYPE(obj), my_tp_token, &cls) < 0) {
+ /* handle error */
+ }
+ CustomObjectData *data = PyObject_GetTypeData(obj, cls);
+
+ Type tokens work similarly to module tokens covered :ref:`earlier in this
+ guide <abi3t-migration-module-token>`.
+
+
+
+Avoid build-time conditionals
+=============================
+
+Check your code for API that identifies the version of Python used to
+*build* your extension.
+This no longer corresponds to the Python your extension runs on, so code
+that uses this information often needs changing.
+The macros to check for are:
+
+- :c:macro:`PY_VERSION_HEX`, :c:macro:`PY_MAJOR_VERSION`,
+ :c:macro:`PY_MINOR_VERSION`:
+
+ - to get the run-time version, use :c:data:`Py_Version`;
+ - to determine what C API is available, use :c:macro:`Py_TARGET_ABI3T`.
+ This macro is set to the minimum supported version.
+
+- :c:macro:`Py_GIL_DISABLED`: under ``abi3t``, this macro is always defined.
+ Code that works with free-threaded Python *should* also work with
+ the GIL enabled (since the GIL can be enabled at run time),
+ and usually *does* (unless it, for some reason, requires more than one
+ :term:`attached thread state <attached thread state>` at one time).
+
+
+Further code changes
+====================
+
+If you are still left with compiler errors or warnings, find a way to fix them.
+Alas, this guide is limited, and cannot cover all possible code
+changes extensions may need.
+
+If you find a problem that other extension authors might run into,
+consider :ref:`reporting an issue <reporting-documentation-bugs>` (or sending
+a pull request) for this guide.
+
+It is possible your issue cannot be fixed for the current version of ``abi3t``.
+In that case, reporting it may help it get prioritized for the next version
+of CPython.
+
+
+.. _abi3t-migration-tagging:
+
+Tagging and distribution
+========================
+
+If you are using a build tool with ``abi3t`` support, your extension is ready,
+but you might want to check that it was built correctly.
+
+Extensions built with ``abi3t`` should have the following extension:
+
+- On Windows: ``.pyd`` (like any other extension);
+- Linux, macOS, and other systems that use the ``.so`` suffix: ``.abi3t.so``
+ (**not** ``.cpython-315t.so`` or ``.abi3.so``).
+ Note that both free-threaded and non-free-threaded builds will
+ load ``.abi3t.so`` extensions;
+- Other systems: consult your distributor, and perhaps update this guide.
+
+If you distribute the extension as a *wheel*, use the following tags:
+
+* Python tag: :samp:`cp3{XX}`, where *XX* is the minimum Python version
+ the extension is built for.
+ (For example, ``cp315`` if you set ``Py_TARGET_ABI3T`` to ``0x30f0000``.
+ See :ref:`abi3-compiling` for more values.)
+* ABI tag: ``abi3.abi3t``. This is a *compressed tag set* that indicates
+ support for both non-free-threaded and free-threaded builds.
+
+For example, the wheel filename may look like this:
+
+.. code-block:: text
+
+ myproject-1.0-cp315-abi3.abi3t-macosx_11_0_arm64.whl
+
+.. seealso:: `Platform Compatibility Tags <https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/>`__ in the PyPA package distribution metadata.
+
+If the filename or tags are incorrect, fix them.
+
+
+Testing
+=======
+
+Note that when you build an extension compatible with multiple versions of
+CPython, you should always *test* it with each version it supports (for
+example, 3.15, 3.16, and so on).
+Stable ABI only guarantees *ABI* compatibility; there may also be behavior
+changes -- both intentional ones (covered by :pep:`387`) and bugs.
+
+Be sure to run tests on both free-threaded and non-free-threaded builds
+of CPython.
+
+If they pass, congratulations! You have an ``abi3t`` extension.
projects / thirdparty / Python / cpython.git / commitdiff
