Skip to content

Add support for PEP 489 multi-phase module initialisation #34

Open
1 of 2 issues completed
Open
@philthompson10

Description

@philthompson10

Python v3.5 saw the implementation of multi-phase module initialisation described in PEP 489. This supports the reloading of extension modules and the release of all allocated memory when an interpreter is finalised (particularly useful when Python is embedded in a C/C++ application). As these are relatively minor use cases, and that the traditional initialisation methods are still supported, it hasn't been a priority to add the support to the sip module or to SIP generated modules. However the recent changes to Python to support multiple sub-interpreters and the disabling of the GIL mean that the support should now be added.

Note that although extension modules will be able to be reloaded, the libraries they wrap cannot and they may contain their own mutable global data. Modules need to be written with this in mind, ie. that the state of a wrapped library may be influenced by the actions of a previously loaded copy of the module.

The main requirements for multi-phase module initialisation are:

  • all global data must be immutable
  • all mutable data must be stored in the module state
  • all module-defined types must be created on the heap.

The support will be implemented as ABI v14. An extension module is written to target a particular ABI. For example, v12 (which uses a custom type for wrapping C/C++ enums) is targeted by PyQt5, and v13 (which uses the standard Python enum type for wrapping C/C++ enums) is targeted by PyQt6.

In more detail:

  • At the moment ABIs use the same data structures for the definition and implementation of wrapped types and instances. This causes unnecessary problems in maintaining backwards compatibility. v14 will split these using immutable data structures, similar to PyMethodDef and PyType_Spec, and mutable data structures that are completely opaque and specific to the ABI version.
  • PEP 573 added support for accessing the module state from most contexts. Note that access to the module state from slot methods (eg. nb_add) was only added (using PyType_GetModuleByDef()) in Python v3.11. It is assumed that this will not be a problem.
  • All wrapped methods will use METH_FASTCALL.
  • Contemporary techniques will be used to create wrapped types and instances. (The current implementation might still be using techniques that were required when supporting Python v1.5.)
  • The functionality of v12 and v13 will be merged and specific functionality (eg. the nature of the support for C/C++ enums) will be configured at build time. ABIs v12 and v13 will be deprecated and eventually removed.

Sub-issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions