[C API] PEP 741: Add PyInitConfig C API to customize the Python initialization

[vstinner]

Links to previous discussion of this feature:

https://discuss.python.org/t/fr-allow-private-runtime-config-to-enable-extending-without-breaking-the-pyconfig-abi/18004

Summary of proposal:

Add a text-based configuration to Python initialization

Pitch:

An API should be added to the limited C API customize the Python initialization (PyConfig API, PEP 587).

The problem is that the PyConfig C API is excluded from the limited C API, whereas the legacy initialization API is deprecated (ex: Py_VerboseFlag in Python 3.12) and being removed in Python 3.13 (ex: PySys_SetPath()).

Linked PRs

• [WIP] gh-107954: Add _PyConfig_Parse() #110145
• gh-107954: Refactor initconfig.c: add CONFIG_SPEC #110146
• [PEP 741] gh-107954: Add PyInitConfig C API #110176
• [PEP 741] gh-107954: Add PyConfig_Get() function #112609
• gh-107954: Add PyConfig_MEMBER_BOOL type to PyConfigSpec #116359
• gh-107954, PEP 741: Add PyConfig_Get() function #123472
• gh-107954, PEP 741: Add PyInitConfig C API #123502
• gh-107954, PEP 741: Adjust Python initialization config #123663
• gh-107954, PEP 741: Add PyInitConfig_AddModule() function #123668
• gh-107954: Fix configuration type for the perf profiler #124636
• gh-107954: Document PEP 741 in What's New 3.14 #127056
• gh-107954: Allow setting cpu_count in PyConfig_Set() #132954
• gh-107954: Add audit event to PyConfig_Set() #132958

[LostTime76]

As discussed in discourse, I do not think the text file is the best option for embedding. Depending on the embedding scenario, I think we will run into issues of finding the file due to how python is currently resolving paths. I have a very specific scenario that may be solved in easier ways. Even if you by chance get python to find the file in my particular scenario, I don't want the user of my embedded python exe to have to specify package paths that my exe should be able to set itself.

Its not as easy as putting the file in the same directory as the embedded exe or the main script. Take the scenario that the embedding exe is using the global python install or the embedding exe is launched as a child process from an external process where the process path is not the path of the embedded exe.

I think there may be much easier ways to solve my specific scenario. I would like to use as few APIs as possible and not duplicate how python is already parsing the options from the command line.

What about adding additional command line options that can be set from argv? If we could set additional package paths from argv, it would allow us to directly point to where we need to without hassle.

Environment variables do not work nicely in the embedded scenario, even though we have PYTHONPATH, because there is no way to cleanly set an environment variable from an already running process to be accessible from a loaded python dll.

[vstinner]

I'm thinking about a function taking a string, not a file.