Build isolation can be a problem with the python-for-android cross compile environment, but disabling it also disables build-system.requires breaking some package installs

Environment

• pip version: latest on pypi right now - installed via -U pip - should be 19.1.1
• Python version: 3.7
• OS: ubuntu in docker, host is fedora 29 x64

Description
I have run into a case where when I specify --no-build-isolation, pip will install dependencies out of order and if any of these dependencies has a pyproject.toml, apparently pip may try to obtain wheel metadata of that dependency (by running its setup.py) without ensuring pyproject.toml's build-system.requires packages are around. This seems like a bug to me, because isn't the entire point of build-system.requires that these packages need to be around for the package to build or even run the setup.py correctly, no matter if the build is isolated or not?

I am also quite practically impacted by this bug or behavior because 1. I need to use pyproject.toml in dependencies due to setup_requires not liking Cython (the easy_install sandbox messes with the Cython compiler) and I use Cython .pxd deps which install_requires doesn't guarantee to be around at build, and 2. I am in a cross compilation environment (python-for-android) where --no-build-isolation is required for all builds because some packages cannot be installed without custom patching, so it's not safe for pip to temporarily reinstall some things as is inherently happening with the build isolation

Expected behavior
no setup.py is ever run even if just for obtaining wheel metadata without everything in build_requires being installed, even if --no-build-isolation is used

How to Reproduce

I don't have an example package right now, really sorry 😢 (because this all happened in python-for-android) but this should work:

1. Create new project using setup.py in a new folder /my/project/path
2. Specify a package as a dep that needs another one at build time, e.g. like pip install git+git://github.com/wobblui/wobblui@562daf6e796fdf8d8a50733b532f9cacae4d4df5 (later commits have a workaround!) - wobblui depends on https://github.com/JonasT/nettools for building the wheel and specifies this in its pyproject.toml/with build-system.requires
3. Install your new project using cd /my/project/path && pip install --user -U --no-build-isolation .
4. Keep fingers crossed that pip chooses an unlucky order for the deps, and indeed tries to e.g. generate wobblui wheel metadata before it tried to install nettools (which is what happened for me)

Output

(really sorry this looks so weird, it's in python-for-android)

[INFO]:    Launching package install...
[DEBUG]:   -> running bash -c '/home/userhome/.local/share/python-for-android/build/venv/bin/pip' install --no-build-isolation -c ._tmp_p4a_recipe_constraints.txt .
[DEBUG]:   	Processing /home/userhome/workspace
[DEBUG]:   	    Preparing wheel metadata ... done
[DEBUG]:   	Requirement already satisfied: Cython==0.29.12 in /home/userhome/.local/share/python-for-android/build/venv/lib/python3.6/site-packages (from -c ._tmp_p4a_recipe_constraints.txt (line 2)) (0.29.12)
[DEBUG]:   	Requirement already satisfied: Pillow==5.2.0 in /home/userhome/.local/share/python-for-android/build/python-installs/unnamed_dist_1 (from -c ._tmp_p4a_recipe_constraints.txt (line 3)) (5.2.0)
[DEBUG]:   	Collecting sfxscan@ git+https://github.com/JonasT/sfxscan.git# from git+https://github.com/JonasT/sfxscan.git (from myapp==0.1)
[DEBUG]:   	  Cloning https://github.com/JonasT/sfxscan.git to /tmp/pip-install-oyi75skp/sfxscan
[DEBUG]:   	  Running command git clone -q https://github.com/JonasT/sfxscan.git /tmp/pip-install-oyi75skp/sfxscan
[DEBUG]:   	  Running command git submodule update --init --recursive -q
[DEBUG]:   	    Preparing wheel metadata ... done
[DEBUG]:   	Collecting wobblui@ https://github.com/wobblui/wobblui/archive/master.zip# from https://github.com/wobblui/wobblui/archive/master.zip (from myapp==0.1)
[DEBUG]:   	  Downloading https://github.com/wobblui/wobblui/archive/master.zip (1.4MB)
     |################################| 1.4MB 390kB/s
[DEBUG]:   	    Preparing wheel metadata ... error
[DEBUG]:   	    ERROR: Complete output from command /home/userhome/.local/share/python-for-android/build/venv/bin/python3 /home/userhome/.local/share/python-for-android/build/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpz5xnpef1:
[DEBUG]:   	    ERROR: Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/image.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/image.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/filedialog.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/filedialog.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/button.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/button.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/woblog.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/woblog.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/label.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/label.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/perf.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/perf.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/texture.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/texture.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/timer.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/timer.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/modal.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/modal.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/textedit.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/textedit.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/widget.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/widget.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/osinfo.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/osinfo.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/color.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/color.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/box.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/box.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/list.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/list.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/dragselection.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/dragselection.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/cache.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/cache.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/__init__.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/__init__.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/widget_base_borderdraw.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/widget_base_borderdraw.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/gfx.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/gfx.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/scrollbarwidget.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/scrollbarwidget.pyx
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/window.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/window.pyx
[DEBUG]:   	
[DEBUG]:   	    Error compiling Cython file:
[DEBUG]:   	    ------------------------------------------------------------
[DEBUG]:   	    ...
[DEBUG]:   	    import sys
[DEBUG]:   	    import time
[DEBUG]:   	    import traceback
[DEBUG]:   	    import weakref
[DEBUG]:   	
[DEBUG]:   	    from nettools.cssparse cimport parse as cssparse
[DEBUG]:   	    ^
[DEBUG]:   	    ------------------------------------------------------------
[DEBUG]:   	
[DEBUG]:   	    src/wobblui/widget_base.pyx:33:0: 'nettools/cssparse.pxd' not found
[DEBUG]:   	
[DEBUG]:   	    Error compiling Cython file:
[DEBUG]:   	    ------------------------------------------------------------
[DEBUG]:   	    ...
[DEBUG]:   	    import sys
[DEBUG]:   	    import time
[DEBUG]:   	    import traceback
[DEBUG]:   	    import weakref
[DEBUG]:   	
[DEBUG]:   	    from nettools.cssparse cimport parse as cssparse
[DEBUG]:   	    ^
[DEBUG]:   	    ------------------------------------------------------------
[DEBUG]:   	
[DEBUG]:   	    src/wobblui/widget_base.pyx:33:0: 'nettools/cssparse/parse.pxd' not found
[DEBUG]:   	
[DEBUG]:   	    Error compiling Cython file:
[DEBUG]:   	    ------------------------------------------------------------
[DEBUG]:   	    ...
[DEBUG]:   	            if self._cached_custom_css_ruleset is not None:
[DEBUG]:   	                return self._cached_custom_css_ruleset
[DEBUG]:   	            if self._own_custom_css is None or \
[DEBUG]:   	                    len(self._own_custom_css.strip()) == 0:
[DEBUG]:   	                return None
[DEBUG]:   	            self._cached_custom_css_ruleset = cssparse(
[DEBUG]:   	                                             ^
[DEBUG]:   	    ------------------------------------------------------------
[DEBUG]:   	
[DEBUG]:   	    src/wobblui/widget_base.pyx:382:42: 'cssparse' is not a constant, variable or function identifier
[DEBUG]:   	
[DEBUG]:   	    Error compiling Cython file:
[DEBUG]:   	    ------------------------------------------------------------
[DEBUG]:   	    ...
[DEBUG]:   	            if self._cached_custom_css_ruleset_for_children is not None:
[DEBUG]:   	                return self._cached_custom_css_ruleset_for_children
[DEBUG]:   	            if self._children_custom_css is None or \
[DEBUG]:   	                    len(self._children_custom_css.strip()) == 0:
[DEBUG]:   	                return
[DEBUG]:   	            self._cached_custom_css_ruleset_for_children = cssparse(
[DEBUG]:   	                                                          ^
[DEBUG]:   	    ------------------------------------------------------------
[DEBUG]:   	
[DEBUG]:   	    src/wobblui/widget_base.pyx:394:55: 'cssparse' is not a constant, variable or function identifier
[DEBUG]:   	    Compiling /tmp/pip-install-oyi75skp/wobblui/src/wobblui/widget_base.pyx because it changed.
[DEBUG]:   	    [1/1] Cythonizing /tmp/pip-install-oyi75skp/wobblui/src/wobblui/widget_base.pyx
[DEBUG]:   	    Traceback (most recent call last):
[DEBUG]:   	      File "/home/userhome/.local/share/python-for-android/build/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
[DEBUG]:   	        main()
[DEBUG]:   	      File "/home/userhome/.local/share/python-for-android/build/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
[DEBUG]:   	        json_out['return_val'] = hook(**hook_input['kwargs'])
[DEBUG]:   	      File "/home/userhome/.local/share/python-for-android/build/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py", line 69, in prepare_metadata_for_build_wheel
[DEBUG]:   	        return hook(metadata_directory, config_settings)
[DEBUG]:   	      File "/home/userhome/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/build_meta.py", line 154, in prepare_metadata_for_build_wheel
[DEBUG]:   	        self.run_setup()
[DEBUG]:   	      File "/home/userhome/.local/share/python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/build_meta.py", line 140, in run_setup
[DEBUG]:   	        exec(compile(code, __file__, 'exec'), locals())
[DEBUG]:   	      File "setup.py", line 215, in <module>
[DEBUG]:   	        ext_modules = extensions(),
[DEBUG]:   	      File "setup.py", line 87, in extensions
[DEBUG]:   	        'linetrace': DEBUG_BUILD,
[DEBUG]:   	      File "/home/userhome/.local/share/python-for-android/build/venv/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1096, in cythonize
[DEBUG]:   	        cythonize_one(*args)
[DEBUG]:   	      File "/home/userhome/.local/share/python-for-android/build/venv/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1219, in cythonize_one
[DEBUG]:   	        raise CompileError(None, pyx_file)
[DEBUG]:   	    Cython.Compiler.Errors.CompileError: /tmp/pip-install-oyi75skp/wobblui/src/wobblui/widget_base.pyx
[DEBUG]:   	    ----------------------------------------
[DEBUG]:   	ERROR: Command "/home/userhome/.local/share/python-for-android/build/venv/bin/python3 /home/userhome/.local/share/python-for-android/build/venv/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpz5xnpef1" failed with error code 1 in /tmp/pip-install-oyi75skp/wobblui
[DEBUG]:

^ as you can see it runs wobblui's Cythonization, which happens because wobblui uses cythonize() to put together its extensions so this automatically runs when the setup.py is invoked (see here: https://github.com/wobblui/wobblui/blob/84bc49f34a329b7fccdd2593fba9d91d30379f20/setup.py#L69 ) even if just to get the metadata and not actually build the wheel. This wouldn't be an issue if nettools was installed first as specified here: https://github.com/wobblui/wobblui/blob/84bc49f34a329b7fccdd2593fba9d91d30379f20/pyproject.toml#L2 - which it is in build isolation, but not with --no-build-isolation, apparently

Sorry, I didn't specify something important clearly: obviously I could just make wobblui's setup.py run fine without the .pxd deps, as long as build_ext isn't actually invoked. (which should work fine for metadata thanks to get_requires_for_build_wheel, which can get metadata for modern setuptools without actually building the wheel)

So it's not like this is an unavoidable problem or anything. However, I think this behavior of pip is simply counter-intuitive given what the intended spirit of build-system.requires appears to be, since I don't think the average package creator will expect build-system.requires to ever not be around when pip runs the setup.py

Ok, now that I've put in the workaround pip also tries to actually fully build the wheel without build-system.requires being around in any way. (it's part of the Building wheels for collected packages: list, but pip or whatever does that actual wheel build just does them in arbitrary order and doesn't build those first that are in the build-system.requires list of other packages.) That now I can't work around or wouldn't know how, and seems like somewhat of a big problem... 🤔

[pfmoore]

no setup.py is ever run even if just for obtaining wheel metadata without everything in build_requires being installed, even if --no-build-isolation is used

I may be missing something here (I haven't fully understood the fairly complex example you gave) but surely using --no-build-isolation (which is documented as "Disable isolation when building a modern source distribution. Build dependencies specified by PEP 518 must be already installed if this option is used.") means that you've accepted the responsibility of installing any necessary build requirements yourself?

... means that you've accepted the responsibility of installing any necessary build requirements yourself?

@pfmoore ok that explains it. In this case, please treat this as a feature request!

Why disabled build isolation appears to be required for pip to work right in a cross-compile environment:

I tested in python-for-android and pip's build isolation will currently often break in its cross compile environment. The reason is python-for-android needs to preinstall some packages in a different way to make them work, and pip cannot just reinstall them for build isolation, it just doesn't work technically and the install fails if it attempts to do so. Summed up, I don't know what to do! I don't see how this could work

setup_requires would work fine because it doesn't use/enable build isolation, but it's deprecated, and breaks with Cython sometimes, and also python-for-android can't decide for any package not to use pyproject.toml - so not really a solution

Edit: sorry this context might be missing, I am one of the contributors of https://github.com/kivy/python-for-android making its setup.py integration work. So that is why I ended up here, trying to make things work reliably for build deps of packages. Basically all that is technically doable I can in theory make python-for-android do, it's just I don't even see any feasible way how this could work

Edit 2: trimmed this down a little to the core point

[pfmoore]

@pfmoore ok that explains it. In this case, please treat this as a feature request!

You'd need to explain the request you're making better. (And if it's an issue limited to cross-compiling environments, it would still probably not be high on anyone else's priority list, as I don't believe there are many people working on cross-compilation at the moment - so you may need to provide a PR yourself, or be prepared for a long wait).

Currently, pip has two "modes" of operation (described essentially in PEPs 517 and 518, although the implementation details aren't part of the standards). It either does nothing, expecting the user to set up the build environment, and then pip just runs the builds in the global environment, or it isolates individual builds in an environment with the build_dependencies specified in pyproject.toml preinstalled. There's no support (currently, or planned) for requesting build isolation on a per-build step basis, or for any other level of isolation. Your "feature request" is essentially about asking for another option, as I understand it - but in that case you'll need to specify what that option is, in a way that someone could code it.

I just don't know what to do! I don't see how this could work.

If you don't, then I doubt anyone else is likely to be able to 😢 Sorry, but this sounds like a pretty specialised problem, and it'll need domain knowledge to know what would solve it.

The problem is pretty simple actually: setup_requires is deprecated and broken (for Cython deps specifically, the install just plain crashes for some), and I need a replacement. And --build-isolation can per design not work in cross compile environments where package installs need to be customized and some packages cannot be "just reinstalled".

Is that really so complicated to understand? I think the base problem really is quite simple...

Basically I need something like setup_requires that actually works, without forcing me to use build isolation since that can't work. (again just per design, the whole idea of letting pip reinstall things on its own just doesn't work if some packages need to be patched in custom ways that right now cannot be feasibly taught to pip since it is way too specialized) Just to explain why no build isolation works, we basically install the affected to-be-patched packages first so pip already finds them present and doesn't attempt to reinstall them and it'll be fine - unless build isolation is active, which is the big problem.

I renamed the ticket, hopefully that should explain the problem better. It really all comes down again to setup_requires being deprecated, and build-system.requires supposedly to be used instead, but its core feature of "build isolation" just cannot work for us (unless build-system.requires is expanded to work differently and e.g. without build isolation, or we get another option like undeprecated setup_requires, or all of install_requires always installed before a wheel is built so it can be misused as wheel build time deps - not nice, but practical)

[pfmoore]

So is this just something that has never worked? As far as I am aware, Python has never formally supported cross compilation. If there's something that used to work, but has been broken by recent developments, that's one thing. But if you're trying something that has never worked, and no-one has ever really expected it to, then that's likely to be a much harder problem.

(I'm not trying to say that it shouldn't work, or that we don't want to support it, just that it's possible that no-one has ever even looked at this problem before now, and there may be more fundamental issues that you need to solve, rather than assuming that things are expected to work and it's just a matter of getting shallow issues fixed).

If there's something that used to work, but has been broken by recent developments, that's one thing.

Basically every setup_requires package moving to build-system.requires will break for us (=python-for-android) if it really depends on its build time deps. For these packages that would cause huge regressions for us, going from most of them working other than some Cython dep ones being broken, to none of them working without patching them all, which obviously we prefer not to do since it doesn't scale well

And the thing is, it makes sense to move away from setup_requires unless it is fixed. I get it, it does actually break for some things and if it is deprecated nobody wants to fix it. So I understand every package maintainer who does move away

But to me it looks like this will indirectly cause quite some regressions (for everyone heeding this setup_requires is deprecated call) which right now I see no way for us to avoid - unless the package maintainers are the ones taking on the burden of all this mess, and patching around with ugly hacks to make setup-requires work anyway and sticking to it even though it is deprecated. setup_requires can be made to always work when overriding build_ext and calling Cython manually in a separate process by the way, but this is horrible to do and no package maintainer will want to do this

Like, just to explain, my own package used setup_requires and worked fine - other than the Cython crash eventually showing up. At first I had an ugly workaround for that, then heard the "use build-system.requires" call and went with it - and now the package does always install right outside of python-for-android, but breaks in python-for-android. So basically I ran into the thing everyone with a package with build deps will run into when going this route, they will all break. (unless pip just happens to shuffle the install order right which is essentially just luck and it may just not do so, or none of the "wrong" deps are in neither install_requires nor build-system.requires so that build isolation can be left enabled and just happens to not reinstall something that can't just be reinstalled. But python-for-android can't know if build isolation is ever better left enabled or not for any package, and both variants may not ever reliably work)

Edit: and basically I can change python-for-android in whatever way to make this work, i just don't know how I could unless build isolation is not mandatory for using the new format. Hence this ticket

Did any of this make sense? I'd be up for a text chat or anything to explain all the details of this better if anything is still unclear. This is really a fundamental problem for us at python-for-android I would think (at least I can't see any obvious way to work around it given where pip is heading now) so I really want to help you understand what the issue is so hopefully we at p4a, or you at pip, can find a way to make it work, without all build deps using packages doing nasty hacks or ending up patched by us