Support for relative home path in pyvenv.cfg

[teruokun]

BPO	39469
Nosy	@brettcannon, @vsajip, @eryksun, @zooba, @teruokun
PRs	bpo-39469: Support relative home paths for pyvenv.cfg #18213

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-27.20:58:08.673>
labels = ['interpreter-core', 'type-feature', '3.9']
title = 'Support for relative home path in pyvenv.cfg'
updated_at = <Date 2020-02-05.00:08:02.180>
user = 'https://github.com/teruokun'

bugs.python.org fields:

activity = <Date 2020-02-05.00:08:02.180>
actor = 'steve.dower'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2020-01-27.20:58:08.673>
creator = 'Jeff.Edwards'
dependencies = []
files = []
hgrepos = []
issue_num = 39469
keywords = ['patch']
message_count = 6.0
messages = ['360800', '360886', '360887', '360893', '360908', '360909']
nosy_count = 5.0
nosy_names = ['brett.cannon', 'vinay.sajip', 'eryksun', 'steve.dower', 'Jeff.Edwards']
pr_nums = ['18213']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue39469'
versions = ['Python 3.9']

[teruokun]

Currently, the interpreter only supports absolute paths for the 'home' directory in the pyvenv.cfg file. While this works when the interpreter is always installed at a fixed location, it impacts the portability of virtual environments and can make it notably more-difficult if multiple virtual environments are shipped with a shared interpreter and are intended to be portable and working in any directory.

Many of these issues can be solved for if 'home' can use a directory relative to the directory of the pyvenv.cfg file. This is detected by the presence of a starting '.' in the value.

A common use-case for this is that a script-based tool (e.g. black or supervisor) may be shipped with a larger portable application where they are intended to share the same interpreter (to save on deployment size), but may have conflicting dependencies. Since the application only depends on the executable scripts, those packages could be packaged into their own virtual environments with their dependencies.

[brettcannon]

Do note that virtual environments are not designed to be portable in general, so this would be a fundamental change in the design and purpose of virtual environments.

[teruokun]

I would say they’re not designed to be, but the also aren’t designed to not
be portable. This is often useful where open network access isn’t
reasonable, so access to Pip/pipx/pipenv is limited at best.

Suffice to say, is there a significant reason to not allow it?

On Tue, Jan 28, 2020 at 10:28 AM Brett Cannon <[email protected]>
wrote:

Brett Cannon <[email protected]> added the comment:

Do note that virtual environments are not designed to be portable in
general, so this would be a fundamental change in the design and purpose of
virtual environments.

----------
nosy: +brett.cannon, vinay.sajip

---

Python tracker <[email protected]>
<https://bugs.python.org/issue39469\>

---

[brettcannon]

Suffice to say, is there a significant reason to not allow it?

We have to support that use-case forever. ;) In all seriousness, relative paths get tricky when you forget to resolve the path as appropriate (and in a way that people expect) and it requires making sure that this will work for the next 30 years, so it's not a small thing to take on.

[eryksun]

Suffice to say, is there a significant reason to not allow it?

It's poorly supported by packaging. In particular, relocating an environment isn't supported with entry-point scripts, which pip installs with a fully-qualified shebang. Moreover, entry-point scripts in Windows are created as exe files (e.g. "pip.exe") that embed the fully-qualified path of python[w].exe in the environment, plus a zipped __main__.py. For example, given an environment at "C:\Temp\env", running "C:\Temp\env\Scripts\pip.exe" in turn spawns a child process with the command line: "C:\Temp\env\Scripts\python.exe" "C:\Temp\env\Scripts\pip.exe". This breaks if the environment is renamed or relocated.

[teruokun]

Interesting, I hadn’t realized that it would embed the FQ Executable path,
but that does make sense overall. I guess I had always planned on fixing
the ‘bin’ directory anyway afterwards, it’s just that the lack of relative
home made it significantly harder to encapsulate multiple environments
running with the same interpreter without having to do a complete
reinstall, and venv did seem like the best and most-pythonic way to do it.

I’ll think about it a bit more

On Tue, Jan 28, 2020 at 2:33 PM Eryk Sun <[email protected]> wrote:

Eryk Sun <[email protected]> added the comment:

> Suffice to say, is there a significant reason to not allow it?

It's poorly supported by packaging. In particular, relocating an
environment isn't supported with entry-point scripts, which pip installs
with a fully-qualified shebang. Moreover, entry-point scripts in Windows
are created as exe files (e.g. "pip.exe") that embed the fully-qualified
path of python[w].exe in the environment, plus a zipped __main__.py. For
example, given an environment at "C:\Temp\env", running
"C:\Temp\env\Scripts\pip.exe" in turn spawns a child process with the
command line: "C:\Temp\env\Scripts\python.exe"
"C:\Temp\env\Scripts\pip.exe". This breaks if the environment is renamed or
relocated.

----------
nosy: +eryksun

---

Python tracker <[email protected]>
<https://bugs.python.org/issue39469\>

---

[ncoghlan]

While portable venvs aren't general purpose venvs, they do work if you purge the bin folder (Scripts on Windows) of everything other than python itself, and only support running commands that are accessible via the -m switch.

For that case (where the directly executable files are deleted outright rather than attempting to fix them up), pyvenv.cfg is the only file that requires modification if the relocation process also changes the path to the base runtime installation.

That said, it's useful to run a postinstall script after relocation in order to recompile the Python source files on the target system anyway, and it's reasonably straightforward to have that file generate a suitable pyvenv.cfg file by executing it with the desired base runtime environment:

from pathlib import Path
runtime_executable_path = Path(sys.executable).resolve()
runtime_version = ".".join(sys.version_info[:3]
venv_config = f"""\
home = {runtime_executable_path.parent}
include-system-site-packages = false
version = {runtime_version}
executable = {runtime_executable_path}
"""
Path(__file__).with_name("pyvenv.cfg").write_text(venv_config, , encoding="utf-8")

[thesayyn]

FWIW, this is a deal breaker for us when generating these venv using Bazel, as it prevents cache sharing of these files with other people. Relative paths are not new, and is well understood by most engineers already, and since venv is already an advanced concept, i don't see why this restriction can not be lifted.

[vsajip]

It's not clear what you mean when you say "only allowing is a deal breaker for us". Only allowing what? Sorry if I'm being dense.

[thesayyn]

"only allowing is a deal breaker for us".

sorry a typo. i meant this is a deal breaker for us.

[Vigilans]

Suffice to say, is there a significant reason to not allow it?

It's poorly supported by packaging. In particular, relocating an environment isn't supported with entry-point scripts, which pip installs with a fully-qualified shebang. Moreover, entry-point scripts in Windows are created as exe files (e.g. "pip.exe") that embed the fully-qualified path of python[w].exe in the environment, plus a zipped main.py. For example, given an environment at "C:\Temp\env", running "C:\Temp\env\Scripts\pip.exe" in turn spawns a child process with the command line: "C:\Temp\env\Scripts\python.exe" "C:\Temp\env\Scripts\pip.exe". This breaks if the environment is renamed or relocated.

uv venv now supports --relocatable flag to not relying on any absolute path to virtualenv in entrypoint scripts: astral-sh/uv#5515, though python executable's symlink and pyvenv.cfg's home field is still absolute path to base python's installation.

There is no issue in changing python symlink to relative path, so the support of relative path for home field in pyvenv.cfg is the last piece to a portable virtual environment.

For now, when home field is relative path, it is relative to the working directory of python process which easily varies. Making it relative to pyvenv.cfg's location would not be a breaking change.

[thesayyn]

For now, when home field is relative path, it is relative to the working directory of python process which easily varies.

I didn’t know this was even permitted. So relative path is permitted but resolved to the current pwd?

[Vigilans]

For now, when home field is relative path, it is relative to the working directory of python process which easily varies.

I didn’t know this was even permitted. So relative path is permitted but resolved to the current pwd?

Yes, and you will fail to start python if you are not running it in the same directory as pyvenv.cfg. There's some dicussions and workaround for it in Stack Overflow.

[ncoghlan]

i don't see why this restriction can not be lifted.

The project where I needed this has been published as open source since I commented on this issue, so the relevant post-install script is now available for direct linking: https://github.com/lmstudio-ai/venvstacks/blob/036d4e2629424fcd24e11cdec7d02110e39f6666/src/venvstacks/_injected/postinstall.py#L55

The interesting part of that function for purposes of this discussion is the deployed_path nested function, as that very intentionally does NOT resolve symlinks when determining the absolute path to put in the venv config file. That behaviour is most definitely NOT how venv creation normally handles symlinks (because it's not how Python startup handles them):

$ cat example-venv/pyvenv.cfg
home = /home/acoghlan/devel
include-system-site-packages = false
version = 3.12.6
executable = /usr/bin/python3.12
command = /home/acoghlan/devel/python-symlink -m venv /home/acoghlan/devel/example-venv

So even if relative path support is added (with the meaning of "relative to the venv config file"), there's an open question around whether it should resolve symlinks or not when making the relative path absolute. It possibly shouldn't (for the same reasons venvstacks doesn't: some platform environments may break with dynamic lib loading issues if you do that), but that would be an oddity relative to the way interpreter startup normally handles symlinks.

That said, the issue is still open because we think it's a reasonable thing to add, it's just that none of the current maintainers have a sufficient motivation to work on it (my looking into it for venvstacks is likely the closest any of us have come in years, and even that potential use case turned out to be better served by an external post-install script).