From 0142b5d096d7cef1d9710aa54685d3cff10b5fb0 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 5 Nov 2025 15:50:21 +0100 Subject: [PATCH] gh-140550: PEP 793 reference documentation Since the PEP calls for soft-deprecation of the existing initialization function, this reorganizes the relevant docs to put the new way of doing things first, and de-emphasize the old. Some bits, like the tutorial, are left out of this patch. (See the issue for a list.) --- Doc/c-api/extension-modules.rst | 250 ++++++--- Doc/c-api/init.rst | 3 +- Doc/c-api/module.rst | 659 +++++++++++++++++------- Doc/c-api/structures.rst | 2 + Doc/c-api/type.rst | 26 +- Doc/howto/free-threading-extensions.rst | 11 +- 6 files changed, 686 insertions(+), 265 deletions(-) diff --git a/Doc/c-api/extension-modules.rst b/Doc/c-api/extension-modules.rst index 3d331e6ec12f76..10a9efe3418aea 100644 --- a/Doc/c-api/extension-modules.rst +++ b/Doc/c-api/extension-modules.rst @@ -8,7 +8,8 @@ Defining extension modules A C extension for CPython is a shared library (for example, a ``.so`` file on Linux, ``.pyd`` DLL on Windows), which is loadable into the Python process (for example, it is compiled with compatible compiler settings), and which -exports an :ref:`initialization function `. +exports an :dfn:`export hook` function (or an +old-style :ref:`initialization function `). To be importable by default (that is, by :py:class:`importlib.machinery.ExtensionFileLoader`), @@ -23,25 +24,131 @@ and must be named after the module name plus an extension listed in One suitable tool is Setuptools, whose documentation can be found at https://setuptools.pypa.io/en/latest/setuptools.html. -Normally, the initialization function returns a module definition initialized -using :c:func:`PyModuleDef_Init`. -This allows splitting the creation process into several phases: +.. _extension-export-hook: + +The export hook must be an exported function with the following signature: + +.. c:function:: PyModuleDef_Slot *PyModExport_modulename(void) + +Its name should be :samp:`PyModExport_{}`, with ```` replaced by +the name of the module. +See :ref:`extension-export-hook-name` for full details. + +.. versionadded:: next + + The :samp:`PyModExport_{}` export hook was added in Python 3.15. + The older way of defining modules is still available: consult either the + :ref:`extension-pyinit` section or earlier versions of this documentation + if you plan to support earlier Python versions. + +The export hook returns an array of :c:type:`PyModuleDef_Slot` entries, +terminated by an entry with a slot ID of ``0``. +These slots describe how the module should be created and initialized. + +This array must remain valid and constant until interpreter shutdown. +Typically, it should use ``static`` storage; for dynamic behavior you should +use the :c:macro:`Py_mod_create` and :c:macro:`Py_mod_exec` slots. + +The export hook may return ``NULL`` with an exception set to signal failure. + +It is recommended to define the export hook function using a helper macro: + +.. c:macro:: PyMODEXPORT_FUNC + + Declare an extension module export hook. + This macro: + * specifies the :c:expr:`PyModuleDef_Slot*` return type, + * adds any special linkage declarations required by the platform, and + * for C++, declares the function as ``extern "C"``. + +For example, a module called ``spam`` would be defined like this:: + + PyABIInfo_VAR(abi_info); + + static PyModuleDef_Slot spam_slots[] = { + {Py_mod_abi, &abi_info}, + {Py_mod_name, "spam"}, + {Py_mod_init, spam_init_function}, + ... + {0, NULL}, + }; + + PyMODEXPORT_FUNC + PyModExport_spam(void) + { + return spam_slots; + } + +The export hook is typically the only non-\ ``static`` +item defined in the module's C source. + +The hook should be kept short -- ideally, one line as above. +If you do need to use Python C API in this function, it is recommended to call +:c:expr:`PyABIInfo_Check(&abi_info, "modulename")` first to raise an exception, +rather than crash, in common cases of ABI mismatch. + + +.. note:: + + It is possible to export multiple modules from a single shared library by + defining multiple export hooks. + However, importing them requires a custom importer or suitably named + copies/links of the extension file, because Python's import machinery only + finds the function corresponding to the filename. + See the `Multiple modules in one library `__ + section in :pep:`489` for details. + +.. _extension-export-hook-name: + +Export hook name +................ + +For modules with ASCII-only names, the export hook must be named +:samp:`PyModExport_{}`, with ```` replaced by the module's name. + +For non-ASCII module names, the export hook must be named +:samp:`PyModExportU_{}` (note the ``U``), with ```` encoded using +Python's *punycode* encoding with hyphens replaced by underscores. In Python: + +.. code-block:: python + + def hook_name(name): + try: + suffix = b'_' + name.encode('ascii') + except UnicodeEncodeError: + suffix = b'U_' + name.encode('punycode').replace(b'-', b'_') + return b'PyModExport' + suffix + + +.. _multi-phase-initialization: + +Multi-phase initialization +.......................... + +The process of creating an extension module follows several phases: + +- Python finds and calls the export hook to get information on how to + create the module. - Before any substantial code is executed, Python can determine which capabilities the module supports, and it can adjust the environment or refuse loading an incompatible extension. -- By default, Python itself creates the module object -- that is, it does - the equivalent of :py:meth:`object.__new__` for classes. - It also sets initial attributes like :attr:`~module.__package__` and - :attr:`~module.__loader__`. -- Afterwards, the module object is initialized using extension-specific - code -- the equivalent of :py:meth:`~object.__init__` on classes. + Slots like :c:data:`Py_mod_abi`, :c:data:`Py_mod_gil` and + :c:data:`Py_mod_multiple_interpreters` influence this step. +- By default, Python itself then creates the module object -- that is, it does + the equivalent of calling :py:meth:`~object.__new__` when creating an object. + This step can be overridden using the :c:data:`Py_mod_create` slot. +- Python sets initial module attributes like :attr:`~module.__package__` and + :attr:`~module.__loader__`, and inserts the module object into + :py:attr:`sys.modules`. +- Afterwards, the module object is initialized in an extension-specific way + -- the equivalent of :py:meth:`~object.__init__` when creating an object, + or of executing top-level code in a Python-language module. + The behavior is specified using the :c:data:`Py_mod_exec` slot. This is called *multi-phase initialization* to distinguish it from the legacy -(but still supported) *single-phase initialization* scheme, -where the initialization function returns a fully constructed module. -See the :ref:`single-phase-initialization section below ` -for details. +(but still supported) :ref:`single-phase initialization `, +where an initialization function returns a fully constructed module. .. versionchanged:: 3.5 @@ -53,7 +160,7 @@ Multiple module instances By default, extension modules are not singletons. For example, if the :py:attr:`sys.modules` entry is removed and the module -is re-imported, a new module object is created, and typically populated with +is re-imported, a new module object is created and, typically, populated with fresh method and type objects. The old module is subject to normal garbage collection. This mirrors the behavior of pure-Python modules. @@ -83,36 +190,32 @@ A module may also be limited to the main interpreter using the :c:data:`Py_mod_multiple_interpreters` slot. -.. _extension-export-hook: +.. _extension-pyinit: -Initialization function -....................... +``PyInit`` function +................... -The initialization function defined by an extension module has the -following signature: +.. deprecated:: next + + This functionality is :term:`soft deprecated`. + It will not get new features, but there are no plans to remove it. + +Instead of :c:func:`PyModExport_modulename`, an extension module can define +an older-style :dfn:`initialization function` with the signature: .. c:function:: PyObject* PyInit_modulename(void) Its name should be :samp:`PyInit_{}`, with ```` replaced by the name of the module. +For non-ASCII module names, use :samp:`PyInitU_{}` instead, with +```` encoded :ref:`as for the export hook `. -For modules with ASCII-only names, the function must instead be named -:samp:`PyInit_{}`, with ```` replaced by the name of the module. -When using :ref:`multi-phase-initialization`, non-ASCII module names -are allowed. In this case, the initialization function name is -:samp:`PyInitU_{}`, with ```` encoded using Python's -*punycode* encoding with hyphens replaced by underscores. In Python: +If a module exports both :samp:`PyInit_{}` and +:samp:`PyModExport_{}`, the :samp:`PyInit_{}` function +is ignored. -.. code-block:: python - - def initfunc_name(name): - try: - suffix = b'_' + name.encode('ascii') - except UnicodeEncodeError: - suffix = b'U_' + name.encode('punycode').replace(b'-', b'_') - return b'PyInit' + suffix - -It is recommended to define the initialization function using a helper macro: +Like with :c:macro:`PyMODEXPORT_FUNC`, it is recommended to define the +initialization function using a helper macro: .. c:macro:: PyMODINIT_FUNC @@ -123,43 +226,15 @@ It is recommended to define the initialization function using a helper macro: * adds any special linkage declarations required by the platform, and * for C++, declares the function as ``extern "C"``. -For example, a module called ``spam`` would be defined like this:: - - static struct PyModuleDef spam_module = { - .m_base = PyModuleDef_HEAD_INIT, - .m_name = "spam", - ... - }; - - PyMODINIT_FUNC - PyInit_spam(void) - { - return PyModuleDef_Init(&spam_module); - } - -It is possible to export multiple modules from a single shared library by -defining multiple initialization functions. However, importing them requires -using symbolic links or a custom importer, because by default only the -function corresponding to the filename is found. -See the `Multiple modules in one library `__ -section in :pep:`489` for details. - -The initialization function is typically the only non-\ ``static`` -item defined in the module's C source. - -.. _multi-phase-initialization: +Normally, the initialization function (``PyInit_modulename``) returns +a :c:type:`PyModuleDef` instance with non-``NULL`` +:c:member:`~PyModuleDef.m_slots`. This allows Python to use +:ref:`multi-phase initialization `. -Multi-phase initialization -.......................... - -Normally, the :ref:`initialization function ` -(``PyInit_modulename``) returns a :c:type:`PyModuleDef` instance with -non-``NULL`` :c:member:`~PyModuleDef.m_slots`. Before it is returned, the ``PyModuleDef`` instance must be initialized using the following function: - .. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *def) Ensure a module definition is a properly initialized Python object that @@ -167,7 +242,8 @@ using the following function: Return *def* cast to ``PyObject*``, or ``NULL`` if an error occurred. - Calling this function is required for :ref:`multi-phase-initialization`. + Calling this function is required before returning a :c:type:`PyModuleDef` + from a module initialization function. It should not be used in other contexts. Note that Python assumes that ``PyModuleDef`` structures are statically @@ -178,18 +254,37 @@ using the following function: .. versionadded:: 3.5 +For example, a module called ``spam`` would be defined like this:: + + static struct PyModuleDef spam_module = { + .m_base = PyModuleDef_HEAD_INIT, + .m_name = "spam", + ... + }; + + PyMODINIT_FUNC + PyInit_spam(void) + { + return PyModuleDef_Init(&spam_module); + } + + .. _single-phase-initialization: Legacy single-phase initialization -.................................. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. deprecated:: next -.. attention:: - Single-phase initialization is a legacy mechanism to initialize extension + Single-phase initialization is :term:`soft deprecated`. + It is a legacy mechanism to initialize extension modules, with known drawbacks and design flaws. Extension module authors are encouraged to use multi-phase initialization instead. -In single-phase initialization, the -:ref:`initialization function ` (``PyInit_modulename``) + However, there are no plans to remove support for it. + +In single-phase initialization, the old-style +:ref:`initializaton function ` (``PyInit_modulename``) should create, populate and return a module object. This is typically done using :c:func:`PyModule_Create` and functions like :c:func:`PyModule_AddObjectRef`. @@ -237,11 +332,18 @@ in the following ways: A single-phase ``PyInit_modulename`` function should create “its” module object as soon as possible, before any other module objects can be created. +* Attempts to import a single-phase initialization module reentrantly + from its own initialization function are likely to cause infinite recursion. + (The extension author may prevent this by manually inserting a partially + initialized module object in `sys.modules`.) + * Non-ASCII module names (``PyInitU_modulename``) are not supported. * Single-phase modules support module lookup functions like :c:func:`PyState_FindModule`. +* The module's :c:member:`PyModuleDef.m_slots` must be NULL. + .. [#testsinglephase] ``_testsinglephase`` is an internal module used in CPython's self-test suite; your installation may or may not include it. diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 49ffeab55850c0..8abf322d043bf0 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1680,7 +1680,8 @@ function. You can create and destroy them using the following functions: Only C-level static and global variables are shared between these module objects. - * For modules using single-phase initialization, + * For modules using legacy + :ref:`single-phase initialization `, e.g. :c:func:`PyModule_Create`, the first time a particular extension is imported, it is initialized normally, and a (shallow) copy of its module's dictionary is squirreled away. diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index 1994a3c7d01ca7..4b967cdb6f8a2a 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -97,13 +97,6 @@ Module Objects Note that Python code may rename a module by setting its :py:attr:`~module.__name__` attribute. -.. c:function:: void* PyModule_GetState(PyObject *module) - - Return the "state" of the module, that is, a pointer to the block of memory - allocated at module creation time, or ``NULL``. See - :c:member:`PyModuleDef.m_size`. - - .. c:function:: PyModuleDef* PyModule_GetDef(PyObject *module) Return a pointer to the :c:type:`PyModuleDef` struct from which the module was @@ -141,158 +134,165 @@ Module Objects unencodable filenames, use :c:func:`PyModule_GetFilenameObject` instead. -.. _pymoduledef: +.. _pymoduledef_slot: -Module definitions ------------------- +Module definition +----------------- -The functions in the previous section work on any module object, including -modules imported from Python code. +Modules defined using the C API are typically defined using an array of *slots* +The slots provide a “description" of how a module should be created. -Modules defined using the C API typically use a *module definition*, -:c:type:`PyModuleDef` -- a statically allocated, constant “description" of -how a module should be created. +.. versionchanged:: next -The definition is usually used to define an extension's “main” module object -(see :ref:`extension-modules` for details). -It is also used to -:ref:`create extension modules dynamically `. + Previously, a :c:type:`PyModuleDef` struct was necessary to define modules. + The older way of defining modules is still available: consult either the + :ref:`pymoduledef` section or earlier versions of this documentation + if you plan to support earlier Python versions. -Unlike :c:func:`PyModule_New`, the definition allows management of -*module state* -- a piece of memory that is allocated and cleared together -with the module object. -Unlike the module's Python attributes, Python code cannot replace or delete -data stored in module state. +The slots array is usually used to define an extension module's “main” +module object (see :ref:`extension-modules` for details). +It can also be used to +:ref:`create extension modules dynamically `. -.. c:type:: PyModuleDef +Unless specified otherwise, the same slot ID may not be repeated +in an array of slots. - The module definition struct, which holds all information needed to create - a module object. - This structure must be statically allocated (or be otherwise guaranteed - to be valid while any modules created from it exist). - Usually, there is only one variable of this type for each extension module. - .. c:member:: PyModuleDef_Base m_base +.. c:type:: PyModuleDef_Slot - Always initialize this member to :c:macro:`PyModuleDef_HEAD_INIT`. + .. c:member:: int slot - .. c:member:: const char *m_name + A slot ID, chosen from the available values explained below. - Name for the new module. + An ID of 0 marks the end of a :c:type:`!PyModuleDef_Slot` array. - .. c:member:: const char *m_doc + .. c:member:: void* value - Docstring for the module; usually a docstring variable created with - :c:macro:`PyDoc_STRVAR` is used. + Value of the slot, whose meaning depends on the slot ID. - .. c:member:: Py_ssize_t m_size + The value may not be NULL. + To leave a slot out, omit the :c:type:`PyModuleDef_Slot` entry entirely. - Module state may be kept in a per-module memory area that can be - retrieved with :c:func:`PyModule_GetState`, rather than in static globals. - This makes modules safe for use in multiple sub-interpreters. + .. versionadded:: 3.5 - This memory area is allocated based on *m_size* on module creation, - and freed when the module object is deallocated, after the - :c:member:`~PyModuleDef.m_free` function has been called, if present. +The available slot types are: - Setting it to a non-negative value means that the module can be - re-initialized and specifies the additional amount of memory it requires - for its state. - Setting ``m_size`` to ``-1`` means that the module does not support - sub-interpreters, because it has global state. - Negative ``m_size`` is only allowed when using - :ref:`legacy single-phase initialization ` - or when :ref:`creating modules dynamically `. +Description slots +................. - See :PEP:`3121` for more details. +.. c:macro:: Py_mod_name - .. c:member:: PyMethodDef* m_methods + Name for the new module, as a NUL-terminated UTF8-encoded ``const char *``. - A pointer to a table of module-level functions, described by - :c:type:`PyMethodDef` values. Can be ``NULL`` if no functions are present. + Note that modules are typically created using a + :py:class:`~importlib.machinery.ModuleSpec`, and when they are, the + name from the spec will be used instead of :c:data:`!Py_mod_name`. + However, it is still recommended to include this slot for introspection + and debugging purposes. - .. c:member:: PyModuleDef_Slot* m_slots + .. versionadded:: next - An array of slot definitions for multi-phase initialization, terminated by - a ``{0, NULL}`` entry. - When using legacy single-phase initialization, *m_slots* must be ``NULL``. + Use :c:member:`PyModuleDef.m_name` instead to support previous versions. - .. versionchanged:: 3.5 +.. c:macro:: Py_mod_doc - Prior to version 3.5, this member was always set to ``NULL``, - and was defined as: + Docstring for the module, as a NUL-terminated UTF8-encoded ``const char *``. + Usually set to a docstring variable created with :c:macro:`PyDoc_STRVAR`. - .. c:member:: inquiry m_reload + .. versionadded:: next - .. c:member:: traverseproc m_traverse + Use :c:member:`PyModuleDef.m_doc` instead to support previous versions. - A traversal function to call during GC traversal of the module object, or - ``NULL`` if not needed. - This function is not called if the module state was requested but is not - allocated yet. This is the case immediately after the module is created - and before the module is executed (:c:data:`Py_mod_exec` function). More - precisely, this function is not called if :c:member:`~PyModuleDef.m_size` is greater - than 0 and the module state (as returned by :c:func:`PyModule_GetState`) - is ``NULL``. +Feature slots +............. - .. versionchanged:: 3.9 - No longer called before the module state is allocated. +.. c:macro:: Py_mod_abi - .. c:member:: inquiry m_clear + A pointer to a :c:struct:`PyABIInfo` structure that describes the ABI that + the extension is using. - A clear function to call during GC clearing of the module object, or - ``NULL`` if not needed. + When the module is loaded, the :c:struct:`!PyABIInfo` in this slot is checked + using :c:func:`PyABIInfo_Check`. - This function is not called if the module state was requested but is not - allocated yet. This is the case immediately after the module is created - and before the module is executed (:c:data:`Py_mod_exec` function). More - precisely, this function is not called if :c:member:`~PyModuleDef.m_size` is greater - than 0 and the module state (as returned by :c:func:`PyModule_GetState`) - is ``NULL``. + A suitable :c:struct:`!PyABIInfo` variable can be defined using the + :c:macro:`PyABIInfo_VAR` macro, as in: - Like :c:member:`PyTypeObject.tp_clear`, this function is not *always* - called before a module is deallocated. For example, when reference - counting is enough to determine that an object is no longer used, - the cyclic garbage collector is not involved and - :c:member:`~PyModuleDef.m_free` is called directly. + .. code-block:: c - .. versionchanged:: 3.9 - No longer called before the module state is allocated. + PyABIInfo_VAR(abi_info); - .. c:member:: freefunc m_free + static PyModuleDef_Slot mymodule_slots[] = { + {Py_mod_abi, &abi_info}, + ... + }; - A function to call during deallocation of the module object, or ``NULL`` - if not needed. + .. versionadded:: 3.15 - This function is not called if the module state was requested but is not - allocated yet. This is the case immediately after the module is created - and before the module is executed (:c:data:`Py_mod_exec` function). More - precisely, this function is not called if :c:member:`~PyModuleDef.m_size` is greater - than 0 and the module state (as returned by :c:func:`PyModule_GetState`) - is ``NULL``. +.. c:macro:: Py_mod_multiple_interpreters - .. versionchanged:: 3.9 - No longer called before the module state is allocated. + Specifies one of the following values: + .. c:namespace:: NULL -Module slots -............ + .. c:macro:: Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED -.. c:type:: PyModuleDef_Slot + The module does not support being imported in subinterpreters. - .. c:member:: int slot + .. c:macro:: Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED - A slot ID, chosen from the available values explained below. + The module supports being imported in subinterpreters, + but only when they share the main interpreter's GIL. + (See :ref:`isolating-extensions-howto`.) - .. c:member:: void* value + .. c:macro:: Py_MOD_PER_INTERPRETER_GIL_SUPPORTED - Value of the slot, whose meaning depends on the slot ID. + The module supports being imported in subinterpreters, + even when they have their own GIL. + (See :ref:`isolating-extensions-howto`.) - .. versionadded:: 3.5 + This slot determines whether or not importing this module + in a subinterpreter will fail. -The available slot types are: + Multiple ``Py_mod_multiple_interpreters`` slots may not be specified + in one module definition. + + If ``Py_mod_multiple_interpreters`` is not specified, the import + machinery defaults to ``Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED``. + + .. versionadded:: 3.12 + +.. c:macro:: Py_mod_gil + + Specifies one of the following values: + + .. c:namespace:: NULL + + .. c:macro:: Py_MOD_GIL_USED + + The module depends on the presence of the global interpreter lock (GIL), + and may access global state without synchronization. + + .. c:macro:: Py_MOD_GIL_NOT_USED + + The module is safe to run without an active GIL. + + This slot is ignored by Python builds not configured with + :option:`--disable-gil`. Otherwise, it determines whether or not importing + this module will cause the GIL to be automatically enabled. See + :ref:`whatsnew313-free-threaded-cpython` for more detail. + + Multiple ``Py_mod_gil`` slots may not be specified in one module definition. + + If ``Py_mod_gil`` is not specified, the import machinery defaults to + ``Py_MOD_GIL_USED``. + + .. versionadded:: 3.13 + + +Creation and initialization slots +................................. .. c:macro:: Py_mod_create @@ -303,18 +303,20 @@ The available slot types are: :no-index-entry: :no-contents-entry: - The function receives a :py:class:`~importlib.machinery.ModuleSpec` - instance, as defined in :PEP:`451`, and the module definition. - It should return a new module object, or set an error + The function will be called with: + + - *spec*: a ``ModuleSpec``-like object, meaning that any attributes defined + for :py:class:`importlib.machinery.ModuleSpec` + have matching semantics. However, any of the attributes may be missing. + - *def*: ``NULL``, or the module definition if the module is created from one. + + The function should return a new module object, or set an error and return ``NULL``. This function should be kept minimal. In particular, it should not call arbitrary Python code, as trying to import the same module again may result in an infinite loop. - Multiple ``Py_mod_create`` slots may not be specified in one module - definition. - If ``Py_mod_create`` is not specified, the import machinery will create a normal module object using :c:func:`PyModule_New`. The name is taken from *spec*, not the definition, to allow extension modules to dynamically adjust @@ -328,6 +330,12 @@ The available slot types are: ``PyModuleDef`` has non-``NULL`` ``m_traverse``, ``m_clear``, ``m_free``; non-zero ``m_size``; or slots other than ``Py_mod_create``. + .. versionchanged:: next + + The *slots* argument may be ``ModuleSpec``-like object, rather than + a true :py:class:`~importlib.machinery.ModuleSpec` instance. + Note that previous versions of CPython did not enforce this. + .. c:macro:: Py_mod_exec Specifies a function that is called to *execute* the module. @@ -339,101 +347,397 @@ The available slot types are: :no-index-entry: :no-contents-entry: - If multiple ``Py_mod_exec`` slots are specified, they are processed in the - order they appear in the *m_slots* array. + See the :ref:`capi-module-support-functions` section for some useful + functions to call. -.. c:macro:: Py_mod_multiple_interpreters + For backwards compatibility, the :c:type:`PyModuleDef.m_slots` array may + contain multiple :c:macro:`!Py_mod_exec` slots; these are processed in the + order they appear in the array. + Elsewhere (that is, in arguments to :c:func:`PyModule_FromSlotsAndSpec` + and in return values of :samp:`PyModExport_{}`), repeating the slot + is not allowed. - Specifies one of the following values: + .. versionchanged:: next - .. c:namespace:: NULL + Repeated ``Py_mod_exec`` slots are disallowed, except in + :c:type:`PyModuleDef.m_slots`. - .. c:macro:: Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED +.. c:macro:: Py_mod_methods - The module does not support being imported in subinterpreters. + A pointer to a table of module-level functions, as an array of + :c:type:`PyMethodDef` values suitable as the *functions* argument to + :c:func:`PyModule_AddFunctions`. - .. c:macro:: Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED + Like other slot IDs, a slots array may only contain one + :c:macro:`!Py_mod_methods` entry. + To add functions from multiple :c:type:`PyMethodDef` arrays, call + :c:func:`PyModule_AddFunctions` in the :c:macro:`Py_mod_exec` function. - The module supports being imported in subinterpreters, - but only when they share the main interpreter's GIL. - (See :ref:`isolating-extensions-howto`.) + The table must be statically allocated (or otherwise guaranteed to outlive + the module object). - .. c:macro:: Py_MOD_PER_INTERPRETER_GIL_SUPPORTED + .. versionadded:: next - The module supports being imported in subinterpreters, - even when they have their own GIL. - (See :ref:`isolating-extensions-howto`.) + Use :c:member:`PyModuleDef.m_methods` instead to support previous versions. - This slot determines whether or not importing this module - in a subinterpreter will fail. - Multiple ``Py_mod_multiple_interpreters`` slots may not be specified - in one module definition. +.. _ext-module-state-slots: - If ``Py_mod_multiple_interpreters`` is not specified, the import - machinery defaults to ``Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED``. +Slots for defining module state +............................... - .. versionadded:: 3.12 +See :ref:`ext-module-state` for an explanation of module state. -.. c:macro:: Py_mod_gil +.. c:macro:: Py_mod_state_size - Specifies one of the following values: + Size of the module state, cast to a ``void*`` pointer. - .. c:namespace:: NULL + Setting this to a non-negative value means that the module can be + re-initialized and specifies the additional amount of memory it requires + for its state. - .. c:macro:: Py_MOD_GIL_USED + Setting the size to ``-1`` means that the + module does not support sub-interpreters, because it has global state. + Negative *size* is only allowed when using + :ref:`legacy single-phase initialization ` + or when :ref:`creating modules dynamically `. - The module depends on the presence of the global interpreter lock (GIL), - and may access global state without synchronization. + See :PEP:`3121` for more details. - .. c:macro:: Py_MOD_GIL_NOT_USED + Use :c:func:`PyModule_GetStateSize` to retrieve the size of a given module. - The module is safe to run without an active GIL. + .. versionadded:: next - This slot is ignored by Python builds not configured with - :option:`--disable-gil`. Otherwise, it determines whether or not importing - this module will cause the GIL to be automatically enabled. See - :ref:`whatsnew313-free-threaded-cpython` for more detail. + Use :c:member:`PyModuleDef.m_size` instead to support previous versions. - Multiple ``Py_mod_gil`` slots may not be specified in one module definition. +.. c:macro:: Py_mod_state_traverse - If ``Py_mod_gil`` is not specified, the import machinery defaults to - ``Py_MOD_GIL_USED``. + A traversal function to call during GC traversal of the module object. - .. versionadded:: 3.13 + The signature of the function, and meanings of the arguments, + is similar as for :c:member:`PyTypeObject.tp_traverse`: -.. c:macro:: Py_mod_abi + .. c:function:: int traverse_module_state(PyObject *module, visitproc visit, void *arg) + :no-index-entry: + :no-contents-entry: - A pointer to a :c:struct:`PyABIInfo` structure that describes the ABI that - the extension is using. + This function is not called if the module state was requested but is not + allocated yet. This is the case immediately after the module is created + and before the module is executed (:c:data:`Py_mod_exec` function). More + precisely, this function is not called if the state size + (:c:data:`Py_mod_state_size`) is greater than 0 and the module state + (as returned by :c:func:`PyModule_GetState`) is ``NULL``. - When the module is loaded, the :c:struct:`!PyABIInfo` in this slot is checked - using :c:func:`PyABIInfo_Check`. + .. versionadded:: next - A suitable :c:struct:`!PyABIInfo` variable can be defined using the - :c:macro:`PyABIInfo_VAR` macro, as in: + Use :c:member:`PyModuleDef.m_size` instead to support previous versions. - .. code-block:: c +.. c:macro:: Py_mod_state_clear - PyABIInfo_VAR(abi_info); + A clear function to call during GC clearing of the module object. - static PyModuleDef_Slot mymodule_slots[] = { - {Py_mod_abi, &abi_info}, - ... - }; + The signature of the function is: - .. versionadded:: 3.15 + .. c:function:: int clear_module_state(PyObject* module) + :no-index-entry: + :no-contents-entry: + This function is not called if the module state was requested but is not + allocated yet. This is the case immediately after the module is created + and before the module is executed (:c:data:`Py_mod_exec` function). More + precisely, this function is not called if the state size + (:c:data:`Py_mod_state_size`) is greater than 0 and the module state + (as returned by :c:func:`PyModule_GetState`) is ``NULL``. -.. _moduledef-dynamic: + Like :c:member:`PyTypeObject.tp_clear`, this function is not *always* + called before a module is deallocated. For example, when reference + counting is enough to determine that an object is no longer used, + the cyclic garbage collector is not involved and + the :c:macro:`Py_mod_state_free` function is called directly. + + .. versionadded:: next + + Use :c:member:`PyModuleDef.m_clear` instead to support previous versions. + +.. c:macro:: Py_mod_state_free + + A function to call during deallocation of the module object. + + The signature of the function is: + + .. c:function:: int free_module_state(PyObject* module) + :no-index-entry: + :no-contents-entry: + + This function is not called if the module state was requested but is not + allocated yet. This is the case immediately after the module is created + and before the module is executed (:c:data:`Py_mod_exec` function). More + precisely, this function is not called if the state size + (:c:data:`Py_mod_state_size`) is greater than 0 and the module state + (as returned by :c:func:`PyModule_GetState`) is ``NULL``. + + .. versionadded:: next + + Use :c:member:`PyModuleDef.m_free` instead to support previous versions. + +.. c:macro:: Py_mod_token + + A pointer intended to identify of the module state's memory layout. + See the :ref:`ext-module-token` section for details. + + This slot is not needed for typical extensions; it exists for specialized + use cases -- namely, dynamically created modules. + If you use the slot, you must ensure that: + + * The pointer outlives the class, so it's not reused for something else + while the class exists. + * It "belongs" to the extension module where the class lives, so it will not + clash with other extensions. + (Preventing clashes with other modules defined in the same extension is + the extension's responsibility; you don't need to use the token for this.) + * If the token points to a :c:type:`PyModuleDef` struct, the module should + behave as if it was created from that :c:type:`PyModuleDef`. + In particular, the module state must have matching layout and semantics. + + Modules created from :c:type:`PyModuleDef` allways use the address of + the :c:type:`PyModuleDef` as the token; :c:macro:`!Py_mod_token` cannot + be used in :c:member:`PyModuleDef.m_slots`. + + .. versionadded:: next + +.. _ext-module-state: + +Module state +------------ + +Extension modules can have *module state* -- a +piece of memory that is allocated on module creation, +and freed when the module object is deallocated. +The module state is specified using :ref:`dedicated slots `. + +Unlike the module's Python attributes, Python code cannot replace or delete +data stored in module state. + +Keeping per-module information in attributes and module state, rather than in +static globals, makes module objects *isolated* and safer for use in +multiple sub-interpreters. +It also helps Python do an orderly clean-up when it shuts down. + +Extensions that keep references to Python objects as part of module state must +implement :c:macro:`Py_mod_state_traverse` and :c:macro:`Py_mod_state_clear` +functions to avoid reference leaks. + +To retrieve the state from a given module, use the following functions: + +.. c:function:: void* PyModule_GetState(PyObject *module) + + Return the "state" of the module, that is, a pointer to the block of memory + allocated at module creation time, or ``NULL``. See + :c:macro:`Py_mod_size`. + + On error, return ``NULL`` with an exception set. + Use :c:func:`PyErr_Occurred` to tell this case apart from a missing + :c:type:`!PyModuleDef`. + + +.. c:function:: int PyModule_GetStateSize(PyObject *, Py_ssize_t *result) + + Set *\*result* to the size of the module's state, as specified using + :c:macro:`Py_mod_size` (or :c:member:`PyModuleDef.m_size`), + and return 0. + + On error, set *\*result* to -1, and return -1 with an exception set. + + .. versionadded:: next + + +.. _ext-module-token: + +Module token +............ + +Each module has an associated *token*, a pointer-sized value intended to +identify of the module state's memory layout. +This means that if you have a module object, but you are not sure if it +“belongs” to your extension, you can check using code like this: + +.. code-block:: c + + PyObject *module = + + void *module_token; + if (PyModule_GetToken(module, &module_token) < 0) { + return NULL; + } + if (module_token != your_token) { + if (!PyErr_Occurred()) { + PyErr_SetString(PyExc_ValueError, "unexpected module") + } + return NULL; + } + + // Now it's safe to cast: this module's state has the expected memory layout + struct my_state state = (struct my_state*)PyModule_GetState(module) + if (state == NULL) { + return NULL; + } + +A module's token -- and the *your_token* value to use in the above code -- is: + +- For modules defined with the :c:macro:`Py_mod_token` slot: the value + of that slot; +- For modules created from an ``PyModExport_*`` + :ref:`export hook `: the slots array that the export + hook returned (unless overriden with :c:macro:`Py_mod_token`). +- For modules created with :c:type:`PyModuleDef`: the address of that + :c:type:`PyModuleDef`. + +.. c:function:: int PyModule_GetToken(PyObject *module, void** result) + + Set *\*result* to the module's token and return 0. + See :c:macro:`Py_mod_token` for information about module tokens. + + On error, set *\*result* to NULL, and return -1 with an exception set. + + .. versionadded:: next + +See also :c:func:`PyType_GetModuleByToken`. + + +.. _module-from-slots: Creating extension modules dynamically -------------------------------------- -The following functions may be used to create a module outside of an -extension's :ref:`initialization function `. -They are also used in -:ref:`single-phase initialization `. +The following functions may be used to create an extension module dynamically, +rather than from an extension's :ref:`export hook `. + +.. c:function:: PyObject *PyModule_FromSlotsAndSpec(const PyModuleDef_Slot *slots, PyObject *spec) + + Create a new module object, given an array of :ref:`slots ` + and the :py:class:`~importlib.machinery.ModuleSpec` *spec*. + + The *slots* argument must point to an array of :c:type:`PyModuleDef_Slot` + structures, terminated by an entry slot with slot ID of 0 + (typically written as ``{0}`` or ``{0, NULL}`` in C). + The *slots* argument may not be ``NULL``. + + The *spec* argument may be any ``ModuleSpec``-like object, as described + in :c:macro:`Py_mod_create` documentation. + Currently, the *spec* must have a ``name`` attribute. + + On success, return the new module. + On error, return ``NULL`` with an exception set. + + Note that this does not process the module's execution slot + (:c:data:`Py_mod_exec`). + Both :c:func:`!PyModule_FromSlotsAndSpec` and :c:func:`PyModule_Exec` + must be called to fully initialize a module. + (Python's default import machinery takes additional steps between + these two calls, such as inserting the module object in + :py:attr:`sys.modules` and setting + :ref:`import-related attributes `.) + + The *slots* array only needs to be valid for the duration of the + :c:func:`!PyModule_FromSlotsAndSpec` call. + In particular, it may be heap-allocated. + + .. versionadded:: next + +.. c:function:: int PyModule_Exec(PyObject *) + + Execute the :c:data:`Py_mod_exec` slot(s) of the given module. + + On success, return 0. + On error, return -1 with an exception set. + + .. versionadded:: next + + + +.. _pymoduledef: + +Module definition struct +------------------------ + +Traditionally, extension modules were defined using a *module definition* +as the “description" of how a module should be created. +Rather than using an array of :ref:`slots ` directly, +the definition has dedicated members for most common functionality, +and allows additional slots as an extension mechanism. + +This way of defining modules is still available and there are no plans to +remove it. + +.. c:type:: PyModuleDef + + The module definition struct, which holds information needed to create + a module object. + + This structure must be statically allocated (or be otherwise guaranteed + to be valid while any modules created from it exist). + Usually, there is only one variable of this type for each extension module + defined this way. + + .. c:member:: PyModuleDef_Base m_base + + Always initialize this member to :c:macro:`PyModuleDef_HEAD_INIT`. + + .. c:member:: const char *m_name + const char *m_doc + Py_ssize_t m_size + PyMethodDef *m_methods + + These members correspond to the :c:macro:`Py_mod_name`, + :c:macro:`Py_mod_doc`, :c:macro:`Py_mod_state_size`, + and :c:macro:`Py_mod_methods` slots, respectively. + + Setting these members to NULL is equivalent to omitting the + corresponding slots. + + .. c:member:: PyModuleDef_Slot* m_slots + + An array of additional slots, terminated by a ``{0, NULL}`` entry. + + This array may not contain slots corresponding to :c:type:`PyModuleDef` + members. + For example, you cannot use a :c:macro:`Py_mod_name` slot; the module + name must be given as :c:member:`PyModuleDef.m_name`. + + .. versionchanged:: 3.5 + + Prior to version 3.5, this member was always set to ``NULL``, + and was defined as: + + .. c:member:: inquiry m_reload + + const char *m_doc + Py_ssize_t m_size + PyMethodDef* m_methods + traverseproc m_traverse + inquiry m_clear + freefunc m_free + + .. c:member:: traverseproc m_traverse + inquiry m_clear + freefunc m_free + + These members correspond to the :c:macro:`Py_mod_state_traverse`, + :c:macro:`Py_mod_state_clear`, and :c:macro:`Py_mod_state_free` slots, + respectively. + + Setting these members to NULL is equivalent to omitting the + corresponding slots. + + .. versionchanged:: 3.9 + + :c:member:`!m_traverse`, :c:member:`m_clear` and :c:member:`m_free` + functions are longer called before the module state is allocated. + +.. _moduledef-dynamic: + +The following API can be used to create modules from a :c:type:`!PyModuleDef` +struct: .. c:function:: PyObject* PyModule_Create(PyModuleDef *def) @@ -510,12 +814,13 @@ They are also used in useful for versioning. This may change in the future. +.. _capi-module-support-functions: + Support functions ----------------- -The following functions are provided to help initialize a module -state. -They are intended for a module's execution slots (:c:data:`Py_mod_exec`), +The following functions are provided to help initialize a module object. +They are intended for a module's execution slot (:c:data:`Py_mod_exec`), the initialization function for legacy :ref:`single-phase initialization `, or code that creates modules dynamically. diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index 58dd915e04f619..30f23625856a36 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -280,6 +280,8 @@ Implementing functions and methods Name of the method. + A ``NULL`` *ml_name* marks the end of a :c:type:`!PyMethodDef` array. + .. c:member:: PyCFunction ml_meth Pointer to the C implementation. diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst index 5bdbff4e0ad990..0fda20af0d2ac6 100644 --- a/Doc/c-api/type.rst +++ b/Doc/c-api/type.rst @@ -250,8 +250,8 @@ Type Objects ``Py_TYPE(self)`` may be a *subclass* of the intended class, and subclasses are not necessarily defined in the same module as their superclass. See :c:type:`PyCMethod` to get the class that defines the method. - See :c:func:`PyType_GetModuleByDef` for cases when :c:type:`!PyCMethod` cannot - be used. + See :c:func:`PyType_GetModuleByToken` for cases when :c:type:`!PyCMethod` + cannot be used. .. versionadded:: 3.9 @@ -269,10 +269,10 @@ Type Objects .. versionadded:: 3.9 -.. c:function:: PyObject* PyType_GetModuleByDef(PyTypeObject *type, struct PyModuleDef *def) +.. c:function:: PyObject* PyType_GetModuleByToken(PyTypeObject *type, const void *token) - Find the first superclass whose module was created from - the given :c:type:`PyModuleDef` *def*, and return that module. + Find the first superclass whose module has the given *token*, and + return that module. If no module is found, raises a :py:class:`TypeError` and returns ``NULL``. @@ -282,6 +282,22 @@ Type Objects and other places where a method's defining class cannot be passed using the :c:type:`PyCMethod` calling convention. + See :ref:`ext-module-token` for more information on module tokens. + + .. versionadded:: next + +.. c:function:: PyObject* PyType_GetModuleByDef(PyTypeObject *type, struct PyModuleDef *def) + + Find the first superclass whose module was created from the given + :c:type:`PyModuleDef` *def*, or whose *token* is equal to *def*, + and return that module. + + Note that modules created from a :c:type:`PyModuleDef` always have their + token set to the :c:type:`PyModuleDef`'s address. + + This function is equivalent to :c:func:`PyType_GetModuleByToken`, except + it returns a borrowed reference. + The returned reference is :term:`borrowed ` from *type*, and will be valid as long as you hold a reference to *type*. Do not release it with :c:func:`Py_DECREF` or similar. diff --git a/Doc/howto/free-threading-extensions.rst b/Doc/howto/free-threading-extensions.rst index 5647ab2d87c79c..668f1c06e8a36f 100644 --- a/Doc/howto/free-threading-extensions.rst +++ b/Doc/howto/free-threading-extensions.rst @@ -46,8 +46,9 @@ Multi-Phase Initialization .......................... Extensions that use multi-phase initialization (i.e., -:c:func:`PyModuleDef_Init`) should add a :c:data:`Py_mod_gil` slot in the -module definition. If your extension supports older versions of CPython, +:c:func:`PyModuleDef_Init` or :c:func:`PyModExport_modulename`) should add a +:c:data:`Py_mod_gil` slot in the module definition. +If your extension supports older versions of CPython, you should guard the slot with a :c:data:`PY_VERSION_HEX` check. :: @@ -60,12 +61,6 @@ you should guard the slot with a :c:data:`PY_VERSION_HEX` check. {0, NULL} }; - static struct PyModuleDef moduledef = { - PyModuleDef_HEAD_INIT, - .m_slots = module_slots, - ... - }; - Single-Phase Initialization ...........................