Allow customization of locations

[encukou]

From Setuptools 50.1.0:

Setuptools reverts using the included distutils by default. Platform maintainers and system integrators and others are strongly encouraged to set SETUPTOOLS_USE_DISTUTILS=local to help identify and work through the reported issues with distutils adoption, mainly to file issues and pull requests with pypa/distutils such that distutils performs as needed across every supported environment.

In Fedora we currently patch distutils.commands.install so that packages install by the user go to /usr/local/, but packages installed system-wide go to /usr/.
This avoids nearly all cases where users brick their system with sudo pip install. Details here.

With SETUPTOOLS_USE_DISTUTILS=local, the customization is ignored.

In hindsight, patching distutils is probably not the best way to customize install locations. But I'm not aware of a better way. It is currently done this way in at least Fedora & Debian; PyPI's distutils.commands.install also differs from CPython's.

[jaraco]

In #1, we addressed a similar issue on PyPy, where downstream packagers were patching the install command for PyPy. With the help of that team, we were able to distill changes that enabled those behaviors on PyPy using the INSTALL_SCHEMES mechanism. Perhaps that technique or a similar one for Debian (one attempt in #4) could be applied to enable the desired schemes in the appropriate context. In many ways, this issue is a duplicate of #2 except for the additional Fedora context.

[encukou]

I'd like to only have Fedora as an example. The fact that pypy and some distros need to modify this space is a hint that it is also needed by other systems – current and future; in distro defaults or in individual systems. I don't think we realistically can do a survey of all the modifications and integrate them into setuptools.
IMO, install schemes should be defined by the Python distributor, not Setuptools. And they should be used by Setuptools, Pip, and any other tool that needs to install packages.

Tzu-ping Chung suggested that the information can be moved to sysconfig, as part of the stdlib.
Another option* is to make this functionality customizable through site.py. I believe that's the perfect place for this.
But, even after support for for one of these is added, it won't be adopted instantly.

* I think I saw read this one of the Discourse threads, but I lost the link

[jaraco]

IMO, install schemes should be defined by the Python distributor, not Setuptools.

I fervently agree. And for a system like this to work, you probably need to have some body that establishes a standard for tools to follow, and then setuptools can honor that standard.

Given the challenges inherent in designing and gathering consensus on such a standard, my instinct is to start by taking the defacto standard of distutils and extend it to be more configurable. The suggestions of making these values available through sysconfig or site.py sound suitable.

[encukou]

IMO, install schemes should be defined by the Python distributor, not Setuptools.

I fervently agree. And for a system like this to work, you probably need to have some body that establishes a standard for tools to follow, and then setuptools can honor that standard.

Since this is something that needs to be done as distutils is deprecated/removed from the stdlib and moved to PyPA, it seems to me that the discussions should be either in a CPython space (where it's moving from) or in this repo (where it's moving to). I also think that some kind of shared design document would be helpful. From the discussions around PEP 632, it sounds like the PEP is not a good place for that. Is there a better place?

[doko42]

I share the intent to help users not to brick their system, having similar patches in Debian and Ubuntu.

The proposed solution however makes 'unix_local' use the same location as somebody doing

./configure --prefix=/usr/local.

Note that --prefix=/usr/local is implicit ( the default). You would now have a common modules dir for two different Python installations. I remember @warsaw suggested to use different locations for that.