[3.7] bpo-34247: Fix Python 3.7 initialization #8659

vstinner · 2018-08-03T20:25:07Z

-X dev: it is now possible to override the memory allocator using
PYTHONMALLOC even if the developer mode is enabled.
Add _Py_InitializeFromConfig()
Add _Py_Initialize_ReadEnvVars() to set global configuration
variables from environment variables
Fix the code to initialize Python: Py_Initialize() now also reads
environment variables
_Py_InitializeCore() can now be called repeatedly: the second
and subsequent calls only replace the configuration, they don't
recreate the main interpreter
Write unit tests on Py_Initialize() and the different ways to
configure Python
The isolated mode now always sets Py_IgnoreEnvironmentFlag and
Py_NoUserSiteDirectory to 1.
pymain_read_conf() now saves/restores the configuration
if the encoding changed

https://bugs.python.org/issue34247

* -X dev: it is now possible to override the memory allocator using PYTHONMALLOC even if the developer mode is enabled. * Add _Py_InitializeFromConfig() * Add _Py_Initialize_ReadEnvVars() to set global configuration variables from environment variables * Fix the code to initialize Python: Py_Initialize() now also reads environment variables * _Py_InitializeCore() can now be called twice: the second call only replaces the configuration. * Write unit tests on Py_Initialize() and the different ways to configure Python * The isolated mode now always sets Py_IgnoreEnvironmentFlag and Py_NoUserSiteDirectory to 1. * pymain_read_conf() now saves/restores the configuration if the encoding changed

Restore legacy_windows_fs_encoding change.

The symbol is not exported.

Replace _Py_InitializeEx_Private() with _Py_InitializeFromConfig(). The _Py_InitializeEx_Private() function has been removed.

vstinner · 2018-08-03T22:39:07Z

bedevere/backport-pr — Not a valid backport PR title.

Stupid bot. It doesn't like "[3.7] bpo-34247: Fix Python initialization" nor "bpo-34247: Fix Python 3.7 initialization". Hopefully, it doesn't prevent to merge the PR :-)

vstinner · 2018-08-03T23:06:22Z

@ronaldoussoren: I will merge this change in two weeks if I don't hear anything from you, since I know feel guilty of having introducing regressions in Python 3.7. This change adds new tests which is something very new for this part of the C API. The tests make me confident in my change :-) They helped me to fix bugs in this PR :-)

vstinner · 2018-08-03T23:07:50Z

This PR backports the best enhancements from the master branch without backporting everything, since I made too many changes in the master. And it seems that master is going to get much more changes, according to the PEP 432 discussion on python-dev. So I prefer to implement the minimum changes for 3.7 just to handle environment variables and to have working unit tests.

ncoghlan · 2018-08-04T05:40:56Z

Seeing if I can make the backport-pr bot happy by updating the PR title to refer to itself :)

(Update: I don't think the title edit triggered any of the bots to re-run, not even the backport-pr bot)

ncoghlan

This is looking pretty good to me - just a few readability/debuggability comments inline.

ncoghlan · 2018-08-04T06:10:39Z

Include/pylifecycle.h

 PyAPI_FUNC(int) _Py_IsCoreInitialized(void);
+#ifndef Py_LIMITED_API


We're already inside a Py_LIMITED_API guard here, so this isn't needed.

ncoghlan · 2018-08-04T06:13:46Z

Misc/NEWS.d/next/Core and Builtins/2018-08-03-21-59-06.bpo-34170.v1h_H2.rst

@@ -0,0 +1,2 @@
+-X dev: it is now possible to override the memory allocator using
+PYTHONMALLOC even if the developer mode is enabled.


We also need a NEWS entry for the bpo-34247 fix

ncoghlan · 2018-08-04T06:19:38Z

Modules/main.c

@@ -1714,6 +1731,27 @@ cmdline_get_env_flags(_Py_CommandLineDetails *cmdline)
 }


+/* Set global variable variables from environment variables */
+void
+_Py_Initialize_ReadEnvVars(void)


Something like _Py_Initialize_ReadEnvVarsNoAlloc might be a better name here, as this doesn't read all the env vars, only the ones that it can process without needing to allocate any memory to store the results.

With the current name, it gives the impression it reads all the environment variables, when it actually skips over the more complex ones like PYTHONPATH.

function renamed

ncoghlan · 2018-08-04T06:55:18Z

Python/pylifecycle.c

+
+        interp = tstate->interp;
+        if (interp == NULL) {
+            return _Py_INIT_ERR("can't make main interpreter");


This error message isn't right for the reconfiguration case. Perhaps something like "Core reconfiguration: no main interpreter found".

Then the thread state lookup above could also gain the "Core reconfiguration: " prefix.

I changed the error message to "no main interpreter found". The displayed error message contains the name of the current function _Py_InitializeCore_impl() thanks to _Py_INIT_ERR() magic. I don't want to add a prefix per error message. Such error should not occur in practice anyway.

ncoghlan · 2018-08-04T06:57:39Z

After reviewing the change, I slightly reworded the section in the commit message about _Py_InitializeCore

matrixise · 2018-08-04T07:46:10Z

Lib/test/test_embed.py

+
+    def check_config(self, testname, expected):
+        env = dict(os.environ)
+        for key in list(env):


I am curious, why do you need to convert the dict to a list? usually, you will iterate over the keys, is there a subtlety?

Because the loop body modifies the env dict.

Mariatta · 2018-08-04T14:44:55Z

bedevere/backport-pr status is not required and does not block merging. You'll notice the "Squash and merge" button is still enabled.

Mariatta · 2018-08-04T14:45:36Z

And the PR needs to start with [3.7].

ncoghlan · 2018-08-04T14:56:03Z

Ah, cool - that means the self-reference is a valid workaround, I just missed that the prefix was missing as well.

Mariatta · 2018-08-04T15:10:47Z

I've deployed the latest change in bedevere, so now it is only looking for the [X.Y] prefix, and it does not care about the (GH-NNNN) anymore.
But this status is now "Required".

Example:

* Remove a redundant #ifndef Py_LIMITED_API in pylifecycle.h. * Rename _Py_Initialize_ReadEnvVars() to _Py_Initialize_ReadEnvVarsNoAlloc(). * Change _Py_InitializeCore_impl() error message

vstinner

I addressed all @ncoghlan's comment in my new commit 6c3b5ec.

vstinner · 2018-08-04T22:20:18Z

Include/pylifecycle.h

 PyAPI_FUNC(int) _Py_IsCoreInitialized(void);
+#ifndef Py_LIMITED_API


vstinner · 2018-08-04T22:22:25Z

Misc/NEWS.d/next/Core and Builtins/2018-08-03-21-59-06.bpo-34170.v1h_H2.rst

@@ -0,0 +1,2 @@
+-X dev: it is now possible to override the memory allocator using
+PYTHONMALLOC even if the developer mode is enabled.


vstinner · 2018-08-04T22:23:50Z

Modules/main.c

@@ -1714,6 +1731,27 @@ cmdline_get_env_flags(_Py_CommandLineDetails *cmdline)
 }


+/* Set global variable variables from environment variables */
+void
+_Py_Initialize_ReadEnvVars(void)


function renamed

vstinner · 2018-08-04T22:26:47Z

Python/pylifecycle.c

+
+        interp = tstate->interp;
+        if (interp == NULL) {
+            return _Py_INIT_ERR("can't make main interpreter");


I changed the error message to "no main interpreter found". The displayed error message contains the name of the current function _Py_InitializeCore_impl() thanks to _Py_INIT_ERR() magic. I don't want to add a prefix per error message. Such error should not occur in practice anyway.

ncoghlan

+1 for going ahead with this version.

We may want to tweak the porting note in What's New, as I believe this may read more environment variables at init time than the interpreter did previously, so embedding applications that want more complete control may now need to set Py_IgnoreEnvironmentFlag when they could previously get by without it.

bedevere-bot · 2018-08-05T10:32:02Z

@vstinner: Please replace # with GH- in the commit message next time. Thanks!

the-knights-who-say-ni added the CLA signed label Aug 3, 2018

bedevere-bot added the awaiting merge label Aug 3, 2018

vstinner changed the title ~~bpo-34247: Fix Python initialization~~ [3.7] bpo-34247: Fix Python initialization Aug 3, 2018

vstinner mentioned this pull request Aug 3, 2018

bpo-34247: actually use environment variables in Py_Initialize #8521

Closed

vstinner requested a review from ronaldoussoren August 3, 2018 20:28

vstinner added 3 commits August 3, 2018 22:32

Fix compilation on Windows

249aba7

Restore legacy_windows_fs_encoding change.

Cannot test _Py_CheckHashBasedPycsMode

1533c6e

The symbol is not exported.

Fix _freeze_importlib.c

e219dca

Replace _Py_InitializeEx_Private() with _Py_InitializeFromConfig(). The _Py_InitializeEx_Private() function has been removed.

vstinner requested a review from a team August 3, 2018 22:02

vstinner changed the title ~~[3.7] bpo-34247: Fix Python initialization~~ bpo-34247: Fix Python 3.7 initialization Aug 3, 2018

ncoghlan changed the title ~~bpo-34247: Fix Python 3.7 initialization~~ bpo-34247: Fix Python 3.7 initialization (GH-8659) Aug 4, 2018

ncoghlan mentioned this pull request Aug 4, 2018

Bot to check if PR is not against the master branch, and it is not a backport PR python/core-workflow#211

Closed

ncoghlan reviewed Aug 4, 2018

View reviewed changes

matrixise reviewed Aug 4, 2018

View reviewed changes

ncoghlan changed the title ~~bpo-34247: Fix Python 3.7 initialization (GH-8659)~~ [3.7] bpo-34247: Fix Python 3.7 initialization (GH-8659) Aug 4, 2018

Mariatta changed the title ~~[3.7] bpo-34247: Fix Python 3.7 initialization (GH-8659)~~ [3.7] bpo-34247: Fix Python 3.7 initialization Aug 4, 2018

Add NEWS entry for bpo-34247 fix

6c3b5ec

* Remove a redundant #ifndef Py_LIMITED_API in pylifecycle.h. * Rename _Py_Initialize_ReadEnvVars() to _Py_Initialize_ReadEnvVarsNoAlloc(). * Change _Py_InitializeCore_impl() error message

vstinner commented Aug 4, 2018

View reviewed changes

ncoghlan approved these changes Aug 5, 2018

View reviewed changes

vstinner merged commit 0c90d6f into python:3.7 Aug 5, 2018

bedevere-bot removed the awaiting merge label Aug 5, 2018

vstinner deleted the fix_config branch August 5, 2018 10:32

		PyAPI_FUNC(int) _Py_IsCoreInitialized(void);
		#ifndef Py_LIMITED_API

		@@ -0,0 +1,2 @@
		-X dev: it is now possible to override the memory allocator using
		PYTHONMALLOC even if the developer mode is enabled.

Uh oh!

[3.7] bpo-34247: Fix Python 3.7 initialization #8659

[3.7] bpo-34247: Fix Python 3.7 initialization #8659

Uh oh!

Conversation

vstinner commented Aug 3, 2018 • edited by ncoghlan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Aug 3, 2018 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Aug 3, 2018

Uh oh!

vstinner commented Aug 3, 2018

Uh oh!

ncoghlan commented Aug 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ncoghlan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncoghlan commented Aug 4, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mariatta commented Aug 4, 2018

Uh oh!

Mariatta commented Aug 4, 2018

Uh oh!

ncoghlan commented Aug 4, 2018

Uh oh!

Mariatta commented Aug 4, 2018

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ncoghlan left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Aug 5, 2018

Uh oh!

Uh oh!

vstinner commented Aug 3, 2018 •

edited by ncoghlan

Loading

vstinner commented Aug 3, 2018 •

edited by bedevere-bot

Loading

ncoghlan commented Aug 4, 2018 •

edited

Loading