pip install . --user -> [Errno 2] No such file or directory: 'meson'

[Nic30]

Hello,

I am trying to make package build by meson-python installable by pip using pip install --user . (a complete build script, CI config)

Recent changes in pip broke this and mesonpep517 has the same issue
jeandet/mesonpep517#1 (comment)

mesonpep517 has opened pull request which is fixing this, however merging takes some time and I simply need this so I tried meson-python.

The result is the same:

...
        File "/tmp/pip-build-env-j8gehd_x/overlay/local/lib/python3.10/dist-packages/mesonpy/__init__.py", line 477, in _meson
          return self._proc('meson', *args)
        File "/tmp/pip-build-env-j8gehd_x/overlay/local/lib/python3.10/dist-packages/mesonpy/__init__.py", line 472, in _proc
          subprocess.check_call(list(args))
        File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
          retcode = call(*popenargs, **kwargs)
        File "/usr/lib/python3.10/subprocess.py", line 345, in call
          with Popen(*popenargs, **kwargs) as p:
        File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'meson'

There is a full log from build

The problem is that pip does modify the PATH and meson executable will be missing. Note that this happens only if meson is not installed in the advance.
You can fix it by executing meson as a module function and not as an executable, as you are currently doing https://github.com/FFY00/meson-python/blob/main/mesonpy/__init__.py#L477 . (see changes in mesonpep517 pull request). (Note that the user typically does not have meson installed in advance.)

Ubuntu 22.04 LTS

• python3.10
• pip-22.0.2
• meson-0.63.0
• ninja-1.10.1
• meson_python-0.7.0

[rgommers]

(see changes in mesonpep517 pull request).

Did you link the wrong pull request? All that does is add ninja to the dependencies.

using pip install --user .

It'd be good to have this work. It should not be necessary though, more recent pip versions automatically add --user for system Python versions when run as user (and running as root is a bad idea).

You can fix it by executing meson as a module function and not as an executable

This seems fine to me as a fix for meson-python, since it has a hard dependency on meson so it should always be installed in the Python environment in use (this is not necessarily true when using meson directly, but that's not relevant here).

[eli-schwartz]

It's not necessarily true if you use for example python -m build --no-isolation --skip-dependency-check because you have seeded the environment yourself, but have Meson as a system tool in another version of python.

This could be relevant for e.g. Gentoo, where multiple versions of python are supported, but on the other hand they do have meson-python depend on >=dev-util/meson-0.60.0[${PYTHON_USEDEP}] so that may not matter.

Perhaps @thesamesam could clarify whether this matters.

[FFY00]

I am not sure what to say, the meson script should be there, as meson is a dependency. pip install --user . not working on certain scenarios is a major issue. I will have to debug things to find the issue, I have no good guesses right now.

[FFY00]

This seems fine to me as a fix for meson-python, since it has a hard dependency on meson so it should always be installed in the Python environment in use (this is not necessarily true when using meson directly, but that's not relevant here).

Not necessarily, we will use whatever Meson to build, but we do tell Meson which Python interpreter to use. Ideally, Meson would always be provided by the environment we are running on, and that is true in isolated build, but there might be situations when running without isolation where that might not be the case.

[rgommers]

I cannot reproduce this with SciPy, pip install --user . works just fine.

mesonpep517 has opened pull request which is fixing this, however merging takes some time and I simply need this so I tried meson-python.

That pull request does not seem related. I also searched the mesonpep517 repo, and did not find a single mention of --user.

I had a look in detail at the build log. It reads:

# Install pre-build deps
pip install -r requirements.txt;

# Install actual version from git

Installing build dependencies ... -� �\� �|� �/� �-� �done
Getting requirements to build wheel ... -� �error
....
FileNotFoundError: [Errno 2] No such file or directory: 'meson'

This looks like user error. The requirements.txt dependencies are installed into the active environment, and then the build command triggers an isolated build with zero build dependencies installed. The retrieved source seems to be missing a pyproject.toml file.

@Nic30 am I missing something, or can be close this as invalid?

[FFY00]

I think it would be beneficial for you to describe your dev environment setup.

Other thought, have you updated your pip? This can happen when building without isolation if the dependency is missing, though I assumed pip would be able to notice the missing dependency.

[Nic30]

Hello,

Did you link the wrong pull request? All that does is add ninja to the dependencies.

• It seems like I did, but I can not find relevant one now.

I am using:

https://github.com/Nic30/hwtHls/blob/llvm_mir_integration/pyproject.toml
https://github.com/Nic30/hwtHls/blob/llvm_mir_integration/Dockerfile

git clone -b llvm_mir_integration https://github.com/Nic30/hwtHls.git
cd hwtHls
docker build -t nic30/hwthls .

There is the log from build: docker-build.log

Step 21/22 : RUN pip3 install .
 ---> Running in c0e161803852
Defaulting to user installation because normal site-packages is not writeable
Processing /home/jovyan
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [35 lines of output]
      + meson setup --native-file=/home/jovyan/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 --prefix=/usr /home/jovyan /home/jovyan/.mesonpy-ntmsqwor/build
      Traceback (most recent call last):
        File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
....
        File "/tmp/pip-build-env-ah44erln/overlay/local/lib/python3.10/dist-packages/mesonpy/__init__.py", line 494, in _configure
        File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'meson'
      [end of output]

(I also used docker rmi $(docker images -q -f dangling=true) to remove all docker images to avoid any interference)

• python3-pip (22.0.2+dfsg-1)
• Installation takes ~3 min, if you need I can make some testing repo for this issue, but any project with meson-python should do.
• doc/requirements.txt does contain my other python packages, list can be seen in log, meson and meson-python is not there

[rgommers]

Defaulting to user installation because normal site-packages is not writeable

Hmm, maybe there's a difference between explicitly using --user and pip doing it implicitly for you. If on line 48 of your Dockerfile you change to RUN pip3 install --user ., does that fix it?

[Nic30]

Dockerfile you change to RUN pip3 install --user ., does that fix it?

No it is the same docker-build.log

@rgommers (or anyone else with working meson-python) What system, python, pip do you use?

[rgommers]

@rgommers (or anyone else with working meson-python) What system, python, pip do you use?

I'm using both Arch Linux and macOS, conda-forge environments, Python 3.8-3.10, and usually latest pip version.

[smcv]

I recently switched dbus-python to use Meson and meson-python for build, and I'm now hitting this same problem on a Debian test machine.

If I run pip with more debugging enabled, it produces some information that could be useful:

user@host:~$ pip install -vvv dbus-python
Using pip 22.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Defaulting to user installation because normal site-packages is not writeable
...
  Installing collected packages: patchelf, ninja, wheel, tomli, setuptools, pyparsing, meson, packaging, pyproject-metadata, meson-python
    changing mode of /tmp/pip-build-env-jfbiloif/overlay/local/bin/ninja to 755
    changing mode of /tmp/pip-build-env-jfbiloif/overlay/local/bin/wheel to 755
    changing mode of /tmp/pip-build-env-jfbiloif/overlay/local/bin/meson to 755
  Successfully installed meson-0.63.2 meson-python-0.8.1 ninja-1.10.2.3 packaging-21.3 patchelf-0.15.0.0 pyparsing-3.0.9 pyproject-metadata-0.6.1 setuptools-65.3.0 tomli-2.0.1 wheel-0.37.1
...
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  + meson setup --native-file=/tmp/pip-install-yodcermv/dbus-python_0cf4004be758468b825b7d76aba873f9/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 --prefix=/usr /tmp/pip-install-yodcermv/dbus-python_0cf4004be758468b825b7d76aba873f9 /tmp/pip-install-yodcermv/dbus-python_0cf4004be758468b825b7d76aba873f9/.mesonpy-nv5qulns/build
  Traceback (most recent call last):
...
  FileNotFoundError: [Errno 2] No such file or directory: 'meson'

I hacked in some debugging, and it looks as though pip is installing meson, ninja, wheel into /tmp/pip-build-env-xxx/overlay/local/bin/, but then setting PATH to include /tmp/pip-build-env-xxx/overlay/bin/, which makes me suspect a problem with Debian's patches to the packaged interpreter and libraries that are intended to get things installed into /usr/local/bin.

The workaround is to install meson (and patchelf and ninja) system-wide. dbus-python has other build-dependencies that need installing system-wide anyway: GLib, libdbus, pkg-config and a C compiler.

[smcv]

I see @Nic30 is using Ubuntu. Ubuntu is derived from Debian, so if this is a Debian-specific problem, that would explain why @Nic30 and I are seeing it but an Arch user is not.

[FFY00]

Oh, that makes sense, thank you for posting that output! This is another issue cause by Debian's patching of Python.

As a workaround, we can try to add that missing directory to PATH when invoking Meson.

[smcv]

I've reported this as https://bugs.debian.org/1019293. I think it's a bug in one of two Debian packages: either python3.x because the patches to Python that change the installation scheme are doing too much, or python3-pip because it should compensate for those patches when it sets up the PATH.