Description
Python v3.5 saw the implementation of multi-phase module initialisation described in PEP 489. This supports the reloading of extension modules and the release of all allocated memory when an interpreter is finalised (particularly useful when Python is embedded in a C/C++ application). As these are relatively minor use cases, and that the traditional initialisation methods are still supported, it hasn't been a priority to add the support to the sip
module or to SIP generated modules. However the recent changes to Python to support multiple sub-interpreters and the disabling of the GIL mean that the support should now be added.
Note that although extension modules will be able to be reloaded, the libraries they wrap cannot and they may contain their own mutable global data. Modules need to be written with this in mind, ie. that the state of a wrapped library may be influenced by the actions of a previously loaded copy of the module.
The main requirements for multi-phase module initialisation are:
- all global data must be immutable
- all mutable data must be stored in the module state
- all module-defined types must be created on the heap.
The support will be implemented as ABI v14. An extension module is written to target a particular ABI. For example, v12 (which uses a custom type for wrapping C/C++ enums) is targeted by PyQt5, and v13 (which uses the standard Python enum type for wrapping C/C++ enums) is targeted by PyQt6.
In more detail:
- At the moment ABIs use the same data structures for the definition and implementation of wrapped types and instances. This causes unnecessary problems in maintaining backwards compatibility. v14 will split these using immutable data structures, similar to
PyMethodDef
andPyType_Spec
, and mutable data structures that are completely opaque and specific to the ABI version. - PEP 573 added support for accessing the module state from most contexts. Note that access to the module state from slot methods (eg.
nb_add
) was only added (usingPyType_GetModuleByDef()
) in Python v3.11. It is assumed that this will not be a problem. - All wrapped methods will use
METH_FASTCALL
. - Contemporary techniques will be used to create wrapped types and instances. (The current implementation might still be using techniques that were required when supporting Python v1.5.)
- The functionality of v12 and v13 will be merged and specific functionality (eg. the nature of the support for C/C++ enums) will be configured at build time. ABIs v12 and v13 will be deprecated and eventually removed.