[BUG] [REGRESSION] Fails on cross-compiling when host = target x84_64

[th0ma7]

setuptools version

60.5.0

Python version

3.10.2

OS

Linux - Synology DSM toolchain

Additional environment information

Part of the SynoCommunity ecosystem providing python package(s).
https://synocommunity.com/
Python 3.10 packages for all Synology NAS archs:
https://synocommunity.com/package/python310

Build using Synology toolchain.
Stages:

1. build host native
2. build cross-compile
3. create crossenv
4. install necessary dependencies into crossenv
5. build wheels to be packages

When updating to latest version (60.5.0), python 3.10 fails to build on cross-compiling ONLY when target=host x86_64.
It successfully build all other cross-compiled environment.

Related PR: SynoCommunity/spksrc#5101

Description

While completing the python310 package update it ended-up failing specifically when updating setuptools from 60.2.0 to 60.5.0.

Specific commit:
SynoCommunity/spksrc@ca2698a

Expected behavior

Successful build using previous setuptools release (60.2.0):
https://github.com/SynoCommunity/spksrc/pull/5101/checks?sha=1448434cb826ec5ac25b6cd66bc836b1b9818e19

How to Reproduce

See PR for additional details.

Output

All build outputs available thought checks on associated PR.
https://github.com/SynoCommunity/spksrc/pull/5101/checks

Failure seems to be happening post-build time, probably at linking time and makes me think it may have "lost" its LDFLAG somehow. Actual failure output, only happens on x86_64:

make[4]: Entering directory '/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/Python-3.10.2'
if test "no-framework" = "no-framework" ; then \
	/usr/bin/install -c python /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin/python3.10; \
else \
	/usr/bin/install -c -s Mac/pythonw /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin/python3.10; \
fi
if test "3.10" != "3.10"; then \
	if test -f /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin/python3.10 -o -h /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin/python3.10; \
	then rm -f /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin/python3.10; \
	fi; \
	(cd /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin; ln python3.10 python3.10); \
fi
if test "x" != "x" ; then \
	rm -f /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/binpython3.10-32; \
	lipo  \
		-output /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin/python3.10-32 \
		/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin/python3.10; \
fi
if test "x" != "x" ; then \
	rm -f /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/binpython3.10-intel64; \
	lipo  \
		-output /home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin/python3.10-intel64 \
		/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install/var/packages/python310/target/bin/python3.10; \
fi
 CC='/home/spksrc/python310.2/spksrc/toolchain/syno-x64-7.0/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc' LDSHARED='/home/spksrc/python310.2/spksrc/toolchain/syno-x64-7.0/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc -shared -L/home/spksrc/python310.2/spksrc/toolchain/syno-x64-7.0/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/lib -L/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib  -L/home/spksrc/python310.2/spksrc/toolchain/syno-x64-7.0/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/lib -L/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install//var/packages/python310/target/lib -Wl,--rpath-link,/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/install//var/packages/python310/target/lib -Wl,--rpath,/var/packages/python310/target/lib  -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' 	_TCLTK_INCLUDES='' _TCLTK_LIBS='' 	_PYTHON_PROJECT_BASE=/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/Python-3.10.2 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/Python-3.10.2/build/lib.linux-x86_64-3.10:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-gnu python3.10 ./setup.py  build
Traceback (most recent call last):
  File "/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/Python-3.10.2/./setup.py", line 50, in <module>
    from distutils.command.install import install
  File "/home/spksrc/python310.2/spksrc/native/python310/work-native/install/usr/local/lib/python3.10/site-packages/setuptools/_distutils/command/install.py", line 20, in <module>
    from .. import _collections
  File "/home/spksrc/python310.2/spksrc/native/python310/work-native/install/usr/local/lib/python3.10/site-packages/setuptools/__init__.py", line 18, in <module>
    from setuptools.dist import Distribution
  File "/home/spksrc/python310.2/spksrc/native/python310/work-native/install/usr/local/lib/python3.10/site-packages/setuptools/dist.py", line 37, in <module>
    from setuptools import windows_support
  File "/home/spksrc/python310.2/spksrc/native/python310/work-native/install/usr/local/lib/python3.10/site-packages/setuptools/windows_support.py", line 2, in <module>
    import ctypes
  File "/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/Python-3.10.2/Lib/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: libffi.so.8: cannot open shared object file: No such file or directory
make[4]: *** [Makefile:637: sharedmods] Error 1
make[4]: Leaving directory '/home/spksrc/python310.2/spksrc/spk/python310/work-x64-7.0/Python-3.10.2'
make[3]: *** [Makefile:126: python310_install] Error 2
make[3]: Leaving directory '/home/spksrc/python310.2/spksrc/cross/python310'
make[2]: *** [../../mk/spksrc.depend.mk:51: depend_target] Error 2
make[2]: Leaving directory '/home/spksrc/python310.2/spksrc/spk/python310'
make[1]: Leaving directory '/home/spksrc/python310.2/spksrc/spk/python310'

Error relates to libffi while it does exists and is part of default CFLAGS and LDFLAGS. And such error does not happens when cross-compiling for all other archs.

$ find . -name libffi.so.8
./work-x64-7.0/install/var/packages/python310/target/lib/libffi.so.8
./work-x64-7.0/libffi-3.4.2/x86_64-pc-linux-gnu/.libs/libffi.so.8

[th0ma7]

After quick testing I can confirm that issue first appeared with version 60.3.0.
I'll try to see if I can bisect to the actual commit in the coming days (not sure exactly how, but will investigate)

[th0ma7]

While doing more testing it looks like something fixed this issue between 60.5.3 and 60.5.4.
Using this successfully until next official release:

@$(PIP) install git+https://github.com/pypa/[email protected]#egg=setuptools

[jaraco]

After quick testing I can confirm that issue first appeared with version 60.3.0.

Looking at the history for 60.3.0, there were basically two changes. One was about the _distutils hack relating to get-pip, so seems unlikely. The other was a distutils change where config vars are now honored from sysconfig, which seems a likely culprit. Here's the code changes.

While doing more testing it looks like something fixed this issue between 60.5.3 and 60.5.4.

Oh, that's good news. And we can see that the 60.5.4 release did fix a known defect around building CPython's own stdlib, which it looks like you're trying to do above.

Therefore, I believe this issue is a duplicate of #3007. Let me know if you think that's not the case.

@$(PIP) install git+https://github.com/pypa/[email protected]#egg=setuptools

I'd recommend this standards-aligned syntax instead:

$ @$(PIP) install setuptools@git+https://github.com/pypa/[email protected]

[th0ma7]

I can't say wetter it's related or not, ain't sure.
I'll confirm if officially fixed with next stable release.

EDIT: @jaraco just mentioning but 60.5.x sub-releases are not available on pypi.