Add support for PyPy

[jaraco]

Thanks to the work of @mattip in pypa/setuptools#2221, support for PyPy in distutils is coming along nicely. This branch cherry-picks the distutils-related commits from that PR and addresses other test-related failures. Now the tests are passing (or skipped) except for one on PyPy:

$ tox -e pypy3
pypy3 installed: attrs==19.3.0,cffi==1.14.0,greenlet==0.4.13,importlib-metadata==1.7.0,more-itertools==8.4.0,packaging==20.4,pluggy==0.13.1,py==1.9.0,pyparsing==2.4.7,pytest==5.4.3,readline==6.2.4.1,six==1.15.0,wcwidth==0.2.5,zipp==3.1.0
pypy3 run-test-pre: PYTHONHASHSEED='3646014116'
pypy3 run-test: commands[0] | pytest
============================= test session starts ==============================
platform darwin -- Python 3.6.9[pypy-7.3.1-final], pytest-5.4.3, py-1.9.0, pluggy-0.13.1
cachedir: .tox/pypy3/.pytest_cache
rootdir: /Users/jaraco/code/public/pypa/distutils, inifile: pytest.ini
collected 286 items

distutils/tests/test_archive_util.py ....................                [  6%]
distutils/tests/test_bdist.py ...                                        [  8%]
distutils/tests/test_bdist_dumb.py ..                                    [  8%]
distutils/tests/test_bdist_msi.py s.                                     [  9%]
distutils/tests/test_bdist_rpm.py ss.                                    [ 10%]
distutils/tests/test_bdist_wininst.py s.                                 [ 11%]
distutils/tests/test_build.py ..                                         [ 11%]
distutils/tests/test_build_clib.py ......                                [ 13%]
distutils/tests/test_build_ext.py s.............s..............          [ 24%]
distutils/tests/test_build_py.py .......                                 [ 26%]
distutils/tests/test_build_scripts.py ....                               [ 27%]
distutils/tests/test_check.py .s.ss.                                     [ 30%]
distutils/tests/test_clean.py ..                                         [ 30%]
distutils/tests/test_cmd.py ........                                     [ 33%]
distutils/tests/test_config.py ....                                      [ 34%]
distutils/tests/test_config_cmd.py .....                                 [ 36%]
distutils/tests/test_core.py ......                                      [ 38%]
distutils/tests/test_cygwinccompiler.py ....                             [ 40%]
distutils/tests/test_dep_util.py ....                                    [ 41%]
distutils/tests/test_dir_util.py ........                                [ 44%]
distutils/tests/test_dist.py ........s......................             [ 55%]
distutils/tests/test_extension.py ...                                    [ 56%]
distutils/tests/test_file_util.py ......                                 [ 58%]
distutils/tests/test_filelist.py .............                           [ 62%]
distutils/tests/test_install.py .....s..                                 [ 65%]
distutils/tests/test_install_data.py ..                                  [ 66%]
distutils/tests/test_install_headers.py ..                               [ 67%]
distutils/tests/test_install_lib.py ......                               [ 69%]
distutils/tests/test_install_scripts.py ...                              [ 70%]
distutils/tests/test_log.py ..                                           [ 70%]
distutils/tests/test_msvc9compiler.py ssss.                              [ 72%]
distutils/tests/test_msvccompiler.py ssss.                               [ 74%]
distutils/tests/test_register.py .....s..s.                              [ 77%]
distutils/tests/test_sdist.py ......s.........                           [ 83%]
distutils/tests/test_spawn.py ...                                        [ 84%]
distutils/tests/test_sysconfig.py ..........s.s..                        [ 89%]
distutils/tests/test_text_file.py ..                                     [ 90%]
distutils/tests/test_unixccompiler.py F...                               [ 91%]
distutils/tests/test_upload.py .......                                   [ 94%]
distutils/tests/test_util.py ...........                                 [ 98%]
distutils/tests/test_version.py ....                                     [ 99%]
distutils/tests/test_versionpredicate.py .                               [100%]

=================================== FAILURES ===================================
_____________ UnixCCompilerTestCase.test_osx_cc_overrides_ldshared _____________

self = <distutils.tests.test_unixccompiler.UnixCCompilerTestCase testMethod=test_osx_cc_overrides_ldshared>

    @unittest.skipUnless(sys.platform == 'darwin', 'test only relevant for OS X')
    def test_osx_cc_overrides_ldshared(self):
        # Issue #18080:
        # ensure that setting CC env variable also changes default linker
        def gcv(v):
            if v == 'LDSHARED':
                return 'gcc-4.2 -bundle -undefined dynamic_lookup '
            return 'gcc-4.2'
        sysconfig.get_config_var = gcv
        with EnvironmentVarGuard() as env:
            env['CC'] = 'my_cc'
            del env['LDSHARED']
            sysconfig.customize_compiler(self.cc)
>       self.assertEqual(self.cc.linker_so[0], 'my_cc')
E       AssertionError: 'gcc' != 'my_cc'
E       - gcc
E       + my_cc

distutils/tests/test_unixccompiler.py:118: AssertionError
=========================== short test summary info ============================
FAILED distutils/tests/test_unixccompiler.py::UnixCCompilerTestCase::test_osx_cc_overrides_ldshared
================== 1 failed, 261 passed, 24 skipped in 4.12s ===================
ERROR: InvocationError for command /Users/jaraco/code/public/pypa/distutils/.tox/pypy3/bin/pytest (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   pypy3: commands failed

Matti, can you suggest the right fix for this issue? Once I have the tests passing on PyPy3 here, I'll fold that into pypa/setuptools@distutils.

[jaraco]

When I look at sysconfig_pypy, I find it has this diff for customize_compiler:

diff --git a/distutils/sysconfig.py b/distutils/sysconfig.py
index 879b6981ed..39e4a848a2 100644
--- a/distutils/sysconfig.py
+++ b/distutils/sysconfig.py
@@ -205,8 +205,8 @@ def customize_compiler(compiler):
                 _osx_support.customize_compiler(_config_vars)
                 _config_vars['CUSTOMIZED_OSX_COMPILER'] = 'True'
 
-        (cc, cxx, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
-            get_config_vars('CC', 'CXX', 'CFLAGS',
+        (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
+            get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
                             'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
 
         if 'CC' in os.environ:
@@ -229,7 +229,7 @@ def customize_compiler(compiler):
         if 'LDFLAGS' in os.environ:
             ldshared = ldshared + ' ' + os.environ['LDFLAGS']
         if 'CFLAGS' in os.environ:
-            cflags = cflags + ' ' + os.environ['CFLAGS']
+            cflags = opt + ' ' + os.environ['CFLAGS']
             ldshared = ldshared + ' ' + os.environ['CFLAGS']
         if 'CPPFLAGS' in os.environ:
             cpp = cpp + ' ' + os.environ['CPPFLAGS']

But if I apply that diff to this branch, not only does it not address the failure, it creates a new one.

[jaraco]

Aha, so looking at the history of sysconfig_pypy, the lingering presence of the use of opt there seems to be an artifact of the original copy (50b42f56b6c5). So the form as found in sysconfig_cpython is probably more correct, but that still doesn't explain why the test is failing.

[jaraco]

It seems the issue happens here when:

(Pdb) cc
'gcc -pthread -arch x86_64'
(Pdb) ldshared 
'gcc -pthread -shared -undefined dynamic_lookup'

Because ldshared doesn't start with 'gcc -pthread -arch x86_64', the override doesn't happen.

On CPython, cc = 'gcc', so the override does happen.

[jaraco]

I'm now all but convinced that test is a poorly-written test. It monkey-patches get_config_var, even though the underlying implementation relies on get_config_vars.

[jaraco]

If I update sysconfig with this diff:

diff --git a/distutils/sysconfig.py b/distutils/sysconfig.py
index 879b6981ed..110ff3a8b4 100644
--- a/distutils/sysconfig.py
+++ b/distutils/sysconfig.py
@@ -205,9 +205,12 @@ def customize_compiler(compiler):
                 _osx_support.customize_compiler(_config_vars)
                 _config_vars['CUSTOMIZED_OSX_COMPILER'] = 'True'
 
-        (cc, cxx, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
-            get_config_vars('CC', 'CXX', 'CFLAGS',
-                            'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
+        (cc, cxx, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = (
+            get_config_var(var)
+            for var in (
+                'CC', 'CXX', 'CFLAGS',
+                'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
+        )
 
         if 'CC' in os.environ:
             newcc = os.environ['CC']

The test passes.

[jaraco]

So the issue isn't pypy-specific, but the flaw is triggered by the conditions that happen to be present in my environment (darwin, Homebrew PyPy build).