Missing library files at repair step & wheel configuration through pyproject

[marchelbling-aqemia]

Description

First of all, thank you for providing cibw to the community!

Missing library files at repair step

I'm trying to provide a pre-built linux only wheel for OpenBabel. I'm taking inspiration from https://github.com/njzjz/openbabel-wheel which is unfortunately not maintained anymore.

The basic setup is that the repository I build only has a CMakeLists.txt that will be used by cibw/scikit-build-core to build the wheel:

CMakeLists.txt

cmake_minimum_required(VERSION 3.11)
project(ObabelPythonDistributions CXX)

# download OpenBabel to ${OBABEL_SOURCE_ROOT}
include(FetchContent)
FetchContent_Declare(openbabel
  GIT_REPOSITORY    https://github.com/openbabel/openbabel
  GIT_TAG           ${OBABEL_VERSION}
)
FetchContent_GetProperties(openbabel)
if(NOT openbabel_POPULATED)
  FetchContent_Populate(openbabel)
  add_subdirectory(${openbabel_SOURCE_DIR} ${openbabel_BINARY_DIR})
  install(
    DIRECTORY ${openbabel_SOURCE_DIR}/scripts/python/openbabel
    DESTINATION ${CMAKE_INSTALL_PREFIX}/openbabel
  )
endif()

My pyproject defines the version and build flags I want to use:

pyproject.toml

[tool.scikit-build]
minimum-version = "0.3.0"

[tool.scikit-build.cmake.define]
OBABEL_VERSION = "f495cc6"
CMAKE_BUILD_TYPE = "Release"
WITH_INCHI = "ON"
PYTHON_BINDINGS = "ON"
RUN_SWIG = "ON"

[tool.cibuildwheel]
archs = ["x86_64"]
build = ["cp311-manylinux2014-x86_64"]
test-command = [
    'python -c "from openbabel import openbabel"',
    "obabel -:C -oxyz --gen3d",
]

[tool.cibuildwheel.linux]
before-all = [
    "yum install -y zlib-devel-1.2.7 eigen3-devel-3.3.7 libxml2-devel-2.9.1",
]

and I use Github actions to build everything.

The issue I have is that the wheel builds fine:

truncated install logs

2024-08-23T08:19:42.0620747Z   *** Installing project into wheel...
2024-08-23T08:19:42.1426718Z   -- Install configuration: "Release"
2024-08-23T08:19:42.1427901Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/openbabel/openbabel
2024-08-23T08:19:42.1437400Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/openbabel/openbabel/pybel.py
...
2024-08-23T08:19:42.2599648Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7.0.0
2024-08-23T08:19:42.2607617Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7
2024-08-23T08:19:42.2614782Z   -- Set non-toolchain portion of runtime path of "/tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7.0.0" to "/tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/lib"
2024-08-23T08:19:42.2621652Z   -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so
...
2024-08-23T08:19:42.4828912Z   *** Making wheel...
2024-08-23T08:19:45.5498324Z   *** Created aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl
2024-08-23T08:19:45.5688505Z   Building wheel for aqemia-openbabel (pyproject.toml): finished with status 'done'
2024-08-23T08:19:45.5786590Z   Created wheel for aqemia-openbabel: filename=aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl size=11102455 sha256=363d1d05dad22183022cd9e4f7c9f8a62d912bc44394f8835c03ef171b461551
2024-08-23T08:19:45.5788988Z   Stored in directory: /root/.cache/pip/wheels/21/14/79/baf708a1463c156ec37e5504ed91f66de9ca8b1bc844f71781
2024-08-23T08:19:45.5824955Z Successfully built aqemia-openbabel
2024-08-23T08:19:45.5940993Z Removed build tracker: '/tmp/pip-build-tracker-q2u7ftms'
2024-08-23T08:19:45.6790642Z     + /opt/python/cp38-cp38/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)'
2024-08-23T08:19:45.7065003Z     + rm -rf /tmp/cibuildwheel/repaired_wheel
2024-08-23T08:19:45.7091788Z     + mkdir -p /tmp/cibuildwheel/repaired_wheel
2024-08-23T08:19:45.7124575Z ##[endgroup]

however the auditwheel repair fails as it cannot find libraries since they only live in the wheel file

logs

2024-08-23T08:19:45.7126351Z ##[group]Repairing wheel...
2024-08-23T08:19:45.7126696Z 
2024-08-23T08:19:45.7130107Z     + sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl'
2024-08-23T08:19:46.0501385Z INFO:auditwheel.main_repair:Repairing aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl
2024-08-23T08:20:06.1838904Z Traceback (most recent call last):
2024-08-23T08:20:06.1870673Z   File "/usr/local/bin/auditwheel", line 8, in <module>
2024-08-23T08:20:06.1871685Z     sys.exit(main())
2024-08-23T08:20:06.1873139Z   File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/main.py", line 54, in main
2024-08-23T08:20:06.1874430Z     rval = args.func(args, p)
2024-08-23T08:20:06.1875837Z   File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/main_repair.py", line 173, in execute
2024-08-23T08:20:06.1877130Z     out_wheel = repair_wheel(
2024-08-23T08:20:06.1878534Z   File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/repair.py", line 78, in repair_wheel
2024-08-23T08:20:06.1879819Z     raise ValueError(
2024-08-23T08:20:06.1880749Z ValueError: Cannot repair wheel, because required library "libopenbabel.so.7" could not be located
2024-08-23T08:20:06.5140592Z ##[endgroup]
2024-08-23T08:20:06.5147019Z                                                              �[31m✕ �[0m20.80s
2024-08-23T08:20:06.5184462Z ##[error]Command ['sh', '-c', 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl'] failed with code 1. 

I believe I must be missing something obvious here but I don't see how to fix the problem except by using a before-all action that basically builds the project. This doubles the build time which is not great. Would there be other options that I missed?

Wheel configuration through pyproject

Also I'm struggling defining the wheels I want to build. I'm only interested in building cp311-manylinux2014-x86_64.
When I set my pyproject.toml (which is what I understand as the preferred way) to

 [tool.cibuildwheel]
 archs = ["x86_64"]
 build = ["cp311-manylinux2014-x86_64"]
 platform = ["linux"]

it seems that cibuildwheel does't pick the configuration and the action run fails with

cibuildwheel: No build identifiers selected: BuildSelector(build_config='cp311-manylinux2014-x86_64', skip_config='', requires_python=<SpecifierSet('<3.12,>=3.11')>, prerelease_pythons=False, free_threaded_support=False)

So I end up defining CIBW_ARCHS, CIWB_BUILD and CIWB_PLATFORM which I would rather not. Is there some precedence rule that I misunderstood?

Thanks a lot for any help!

Build log

No response

CI config

No response

[Czaki]

It looks like you should use build = ["cp311-manylinux2014_x86_64"]

[henryiii]

The build identifiers don’t include the manylinux docker image variant (like 2014). build = ["cp311-manylinux_x86_64"]

[Czaki]

I miss this update. How to specify now the manylinux image to use?

[joerick]

See the table in the docs for a full list of build identifiers, they don't include the manylinux variant as Henry says.

I miss this update. How to specify now the manylinux image to use?

You can choose the image you want to use with the manylinux-ARCH-image options. It's always been this way 🙃

[henryiii]

Do you have a link to the repo? I'd start by running pipx run build --wheel and seeing what is actually in the wheel.

[marchelbling-aqemia]

Thanks a lot for your answers!
Sorry, I did not pay attention to the fact that the variant was not part of the build parameter. This matter is all resolved. I assumed that part of the log (build_config='cp311-manylinux2014-x86_64') was actually confirming that this wasn't an issue in providing the right values

For the "double build", here is a repo reproducing my setup: https://github.com/marchelbling/wheels/.
Note that I actually realized that the Python dev dependencies were not picked by cmake and the python extension was not built. I updated the CMakeLists.txt but it seems that I still have issues at link time.

After re-reading the scikit-build-core documentation:

• https://scikit-build-core.readthedocs.io/en/latest/cmakelists.html#finding-python
• https://scikit-build-core.readthedocs.io/en/latest/faqs.html#finding-python

I'm starting to think that I will not manage to build OpenBabel using the manylinux image as I understand it does not include Python libraries. Am I correct?

[henryiii]

Wheels are not supposed to "embed" Python. Python is already present when are you loading a wheel, so you don't need to link to it, and it could even lock you to the patch release you linked against. So yes, the Python libs are removed when building the manylinux image. See pypa/manylinux#1185 for some context.

[marchelbling-aqemia]

Thank you @henryiii that really helped me! It would have been awesome that the PR was merged but it was not too difficult to apply the PR diff on manylinux main branch.

Now my wheel finally builds cleanly except for the "double" build though.
Note that I found another project that seems to have the same issue: https://github.com/spglib/spglib/blob/d9c572d1986b0eccc4ed5f18c8d827729a320313/pyproject.toml#L111-L112.
Please let me know if you would have a chance to look at what I could be doing wrong.

[marchelbling-aqemia]

I actually came across pypa/auditwheel#285.
I'm not sure I fully understand the conclusion from the thread. I'm really not sure what the "proper" way to fix this. I stopped building twice and I'm now doing a manual repair where I unzip the wheel and set LD_LIBRARY_PATH before running auditwheel repair but that doesn't seem "right".