Editable installs are not recognized as having typing when using py.typed (setuptools v64 issue)

[cmeyer]

Bug Report

mypy does not recognize that libraries installed as editable using pip are fully typed even though the py.typed file is correctly included.

To Reproduce

Working script which installs a NON-editable version of library:

# create python environment with pip
rm -rf nionui
git clone https://github.com/nion-software/nionui.git
pushd nionui
git checkout 0.6.4
python -m pip install mypy numpy imageio types-pytz types-tzlocal types-setuptools
# the next line is the only change between these two scripts
python -m pip install nionutils==0.4.4
mypy --namespace-packages --ignore-missing-imports --follow-imports=silent --install-types --non-interactive --strict --no-warn-redundant-casts --no-warn-unused-ignores -p nion.ui -p nionui_app.nionui_examples
popd
# success

Failing version which installs an editable version of a library:

# create python environment with pip
rm -rf nionui
git clone https://github.com/nion-software/nionui.git
pushd nionui
git checkout 0.6.4
python -m pip install mypy numpy imageio types-pytz types-tzlocal types-setuptools nionutils
# the next line is the only change between these two scripts
python -m pip install -e git+https://github.com/nion-software/nionutils.git@0.4.4#egg=nionutils
mypy --namespace-packages --ignore-missing-imports --follow-imports=silent --install-types --non-interactive --strict --no-warn-redundant-casts --no-warn-unused-ignores -p nion.ui -p nionui_app.nionui_examples
popd
# failure

Expected Behavior

These should work the same. The source client and underlying library are the same in both cases except one is installed as an editable install.

Actual Behavior

The failing version gives type errors:

nion/ui/DrawingContext.py:790: error: Returning Any from function declared to return "Optional[str]"
nion/ui/DrawingContext.py:794: error: Returning Any from function declared to return "Optional[str]"
nion/ui/DrawingContext.py:798: error: Returning Any from function declared to return "Optional[str]"
nion/ui/CanvasItem.py:2780: error: Class cannot subclass "Observable" (has type "Any")
nion/ui/UserInterface.py:3217: error: Returning Any from function declared to return "int"
nion/ui/UserInterface.py:3224: error: Returning Any from function declared to return "int"
nion/ui/UserInterface.py:3620: error: Class cannot subclass value of type "Any"
nion/ui/UserInterface.py:3634: error: Class cannot subclass value of type "Any"
nion/ui/Widgets.py:382: error: Returning Any from function declared to return "AbstractSet[int]"
nion/ui/Declarative.py:949: error: Class cannot subclass "Observable" (has type "Any")
nion/ui/Declarative.py:1112: error: Class cannot subclass "Observable" (has type "Any")
nion/ui/GridCanvasItem.py:256: error: Returning Any from function declared to return "int"
Found 12 errors in 6 files (checked 51 source files)

Your Environment

• Mypy version used: 0.971
• Mypy command-line flags: see script
• Mypy configuration options from mypy.ini (and other config files): None
• Python version used: Python 3.10.5
• Operating system and version: macOS, Ubuntu 20, didn't try Windows yet

Also note: this behavior seems to have changed around 2022-08-10 to 2022-08-11. Something in the Python ecosystem seems to have changed. Nevertheless, this seems to be a bug in mypy since it is all about whether py.typed is found or not.

[lawrence-law]

I also experienced this problem and traced it down to setuptools v64 causing the issue. This version of setuptools seems to cause editable packages to no longer be installed with an egg-link file and now uses another mechanism. See release notes here: https://setuptools.pypa.io/en/latest/history.html#v64-0-0

If the editable package belongs to you, you could modify your pyproject.toml like this:

requires = [
  "setuptools==63.4.3",
  "wheel"
]

Previously I had setuptools without any version locking so I presume it was using setuptools 64.0.1.

This seems to workaround the issue (albeit withholding setuptools versions) although perhaps a fix does sit with Mypy.

[JacobHayes]

Setting export SETUPTOOLS_ENABLE_FEATURES="legacy-editable" (eg: with direnv or in Dockerfiles) and reinstalling things also worked for me (and should work if you can't update the project's pyproject.toml).

[hauntsaninja]

Is there documentation on what exactly setuptools is doing now? mypy 0.971 should just be using sys.path, which is, y'know, the standard way of making packages findable

[JacobHayes]

I think the relevant PEP is PEP 660, but I'm not sure if it has all the details. With the new wheels, the site-packages directory ends up with files like this:

__editable__.org_pkg-2.1.0.pth # imports __editable___org_pkg_2_1_0_finder and calls install()
__editable___org_pkg_2_1_0_finder.py # some Finder magic w/ refs to the source path
org_pkg-2.1.0.dist-info/ # dist info like "normal" wheels

instead of an org-pkg.egg-link file (I don't know why this would break mypy though).

I can post back with a minimum reproducing example shortly.

[hauntsaninja]

Oh, so setuptools has switched to using some import hook based approach instead of static entries in pth files? Note that the super standard pth file static directory install that everyone has been using for decades is perfectly valid under PEP 660.

I don't think there's any way for IDEs and static analysis tools to support import hooks. See
https://mail.python.org/archives/list/typing-sig@python.org/thread/IIVBPYDZR5T5BGPAWFVYS5ZPYDXGVHQN/#OSWHT5VSRGKPSPYD7PQWR2M4OCSL5WO3 where maintainers of PyCharm, VSCode, Pyright, Pyre, etc are all on the same page about this.

This feels like a very bold move from setuptools. I'd recommend contacting them about this.

[JacobHayes]

I'd recommend contacting them about this.

Done: pypa/setuptools#3518, though I'm prepared for them to say similar. 😉

[hauntsaninja]

Yeah, well, so it goes. It's one thing if it was just mypy, but this will break all the tools 🤷

If a setuptools maintainer sees this, I recommend reading and then replying to that thread on typing-sig rather than discussing here.

[JacobHayes]

For posterity, I made a repro repo here which shows the issues w/ mypy and pylint for these installs.