Issue with having .pyi files in a separate folder (again)

[MarkCBell]

This is more of a question than an issue.

I'm starting on using mypy to typecheck a package in which I have stub files in a dedicated ./stubs folder. However even when I pass this path into the mypy_path option in mypy.ini these don't seem to be detected and I

The same issue appears to be reported here #4933 which includes a minimal example. Tweaking the mypy.ini file and making the ./stubs/mypy folder into a package by adding __init__.pyi files was suggested as a solution. I forked the package and implemented this suggestion (see https://github.com/MarkCBell/mypyx) however running mypy tests/test-main.py I still see the same error messages:

mypy/main.py:1: error: Function is missing a type annotation
mypy/main.py:5: error: Function is missing a type annotation
mypy/main.py:9: error: Function is missing a type annotation

As before, if I move ./stubs/mypy/main.pyi to ./mypy/ then these errors go away.

I am using mypy 0.620 installed under Python 3.6.5 by pip 18.0

Can someone please help me in knowing where am I exactly going wrong?

[ilevkivskyi]

Could you please give a bit more context? What exactly do you want to do?

Note: if you want to type-check a body of a function against its signature in a stub, then it is currently impossible, see however PR #5139

[MarkCBell]

I have a package with ./mypy/main.py and a test in ./tests/test-main.py and I want to create a stub file in ./stubs/mypy/main.pyi to keep the stub files separate so I also added mypy_path=./stubs/ to mypy.ini. However when I run mypy ./tests/test-main.py it doesn't appear to find any stub files and so raises errors that the functions are missing type annotations.

[ilevkivskyi]

Sorry I can't reproduce this. I just installed mypy from master, cloned your repo, and I see correct behaviour:

 $ mypy ./tests/test-main.py 
tests/test-main.py:3: error: "greeting" does not return a value
tests/test-main.py:5: error: Argument 1 to "hello_world" has incompatible type "str"; expected "int"

[MarkCBell]

Thanks for testing it out. That is a little strange, so I uninstalled mypy and reinstalled the mypy master directly from github but still see the same errors. These are the commands I ran:

$ pip install -U git+git://github.com/python/mypy.git
$ mypy --version
mypy 0.630+dev-2b246ddcab890fe6dc119fd18adbf752fa233d4c
$ git clone https://github.com/MarkCBell/mypyx
$ cd mypy
$ mypy tests/test-main.py
mypy/main.py:1: error: Function is missing a type annotation
mypy/main.py:5: error: Function is missing a type annotation
mypy/main.py:9: error: Function is missing a type annotation

Which version of Python, pip and os are you using? I'm on Python 3.6.5, pip 18.0 and macOS High Sierra 10.13.3.

[ilevkivskyi]

I did this on Python 3.4.3, and 3.6.1. I don't think Python version matters here. The only possible difference is that I never install from remote git repo using pip (because it doesn't work well with git submodules). I normally use git clone <mypy-repo> && cd mypy && git submodule update typeshed && pip install -U .

Another question, are you using a virtual environment?

[martinstein]

I'm having the same issue as @MarkCBell , both in my local project and when trying out the test case from this thread from https://github.com/MarkCBell/mypy

I'm on Windows 10, Python 3.7 in a virtual environment from venv, if that matters. For the test case, I've also tried slashes, backslashes and double backslashes for specifying the stubs-path, but the result is still:

mypy\main.py:1: error: Function is missing a type annotation
mypy\main.py:5: error: Function is missing a type annotation
mypy\main.py:9: error: Function is missing a type annotation

Does the virtual environment affect this somehow? @ilevkivskyi

[ilevkivskyi]

Hm, I just tried this on Mac and it doesn't work at all, both with and without a virtualenv. I does work however if I first cd to tests. So it looks like mypy prefers original .py files if it can find them in current directory, otherwise it uses the stubs.

[martinstein]

I'm able to make MyPy see the .pyi stub file if it sits in the exact same directory as the original .py module. However, according to the docs I was expecting that the same logic (higher priority for the .pyi file if found) would apply to the stub directory as well.

Can anybody else get a test-case to work where a module from your own code has a stub file in a stubs-directory (i.e. not in the same folder)? I'm still expecting that it's just me who's using the mypy_path option in mypy.ini in a wrong way, but if not, this seems like a bug/wrong behavior to me.

[gvanrossum]

If you use mypy -v you should see copious logging showing which files it is analyzing.

One rule to be aware of: mypy's search patch is something like this:

1. current directory
2. directories where sources given on command line live
3. $MYPYPATH
4. mypy_path from mypy.ini
5. site-packages etc.
6. typeshed

The other rule to know is that mypy searches each directory on the path in turn, looking for x.py and x.pyi, preferring x.pyi over x.py in the same directory. But if it finds only one or the other it prefers the earlier search path directory over later ones.

[ilevkivskyi]

@gvanrossum This is a nice short summary. However I noticed the order in your list is a bit different from the one documented in https://mypy.readthedocs.io/en/latest/running_mypy.html#how-imports-are-found, namely MYPYPATH stays first. So what should we fix, the docs or the implementation?

[martinstein]

Okay, thanks for the -v suggestion. I got @MarkCBell 's test-case working... it turns out naming the package in the test-case mypy was not ideal (since it conflicts with mypy itself, apparently). As soon as you rename that package to something else, e.g. mypyx then it works as expected.

Now I'll try to figure out why things are not working in my more advanced setup.

[martinstein]

Actually, sorry, please scratch my last message. I'm still unable to get that test-case working...

[martinstein]

The verbose log output (after renaming the test-package to mypyx to avoid conflicts) doesn't give any indication about the stubs directory:

LOG:  Mypy version 0.630
LOG:  Could not load cache for test-main: .mypy_cache\3.6\test-main.meta.json
LOG:  Metadata not found for test-main
LOG:  Parsing ./tests/test-main.py (test-main)
LOG:  Metadata fresh for mypyx: file C:\Users\Martin\Downloads\mypy-test-master\mypy-master\mypyx\__init__.py
LOG:  Could not load cache for mypyx.main: .mypy_cache\3.6\mypyx\main.meta.json
LOG:  Metadata not found for mypyx.main
LOG:  Parsing C:\Users\Martin\Downloads\mypy-test-master\mypy-master\mypyx\main.py (mypyx.main)
LOG:  Metadata fresh for builtins: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\builtins.pyi
LOG:  Metadata fresh for typing: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\typing.pyi
LOG:  Metadata fresh for abc: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\abc.pyi
LOG:  Metadata fresh for types: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\types.pyi
LOG:  Metadata fresh for sys: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\sys.pyi
LOG:  Metadata fresh for collections: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\collections\__init__.pyi
LOG:  Metadata fresh for _importlib_modulespec: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\_importlib_modulespec.pyi
LOG:  Metadata fresh for importlib.abc: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\importlib\abc.pyi
LOG:  Metadata fresh for collections.abc: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\collections\abc.pyi
LOG:  Metadata fresh for importlib: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\importlib\__init__.pyi
LOG:  Metadata fresh for os: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\os\__init__.pyi
LOG:  Metadata fresh for importlib.util: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\importlib\util.pyi
LOG:  Metadata fresh for io: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\io.pyi
LOG:  Metadata fresh for posix: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\posix.pyi
LOG:  Metadata fresh for os.path: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\os\path.pyi
LOG:  Metadata fresh for importlib.machinery: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\3\importlib\machinery.pyi
LOG:  Metadata fresh for codecs: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\2and3\codecs.pyi
LOG:  Metadata fresh for mmap: file c:\tools\py37-test\lib\site-packages\mypy\typeshed\stdlib\2and3\mmap.pyi
LOG:  Loaded graph with 21 nodes (0.027 sec)
LOG:  Found 4 SCCs; largest has 18 nodes
LOG:  Processing 1 queued fresh SCCs
LOG:  Processing SCC singleton (mypyx.main) as inherently stale
mypyx\main.py:1: error: Function is missing a type annotation
mypyx\main.py:5: error: Function is missing a type annotation
mypyx\main.py:9: error: Function is missing a type annotation
LOG:  Deleting mypyx.main C:\Users\Martin\Downloads\mypy-test-master\mypy-master\mypyx\main.py .mypy_cache\3.6\mypyx\main.meta.json .mypy_cache\3.6\mypyx\main.data.json
LOG:  Processing 1 queued fresh SCCs
LOG:  Processing SCC singleton (test-main) as inherently stale with stale deps (mypyx.main)
LOG:  Deleting test-main C:\Users\Martin\Downloads\mypy-test-master\mypy-master\tests\test-main.py .mypy_cache\3.6\test-main.meta.json .mypy_cache\3.6\test-main.data.json
LOG:  No fresh SCCs left in queue
LOG:  Build finished in 0.168 seconds with 21 modules, and 3 errors

[MarkCBell]

@martinstein Thank you for your comments, particularly about the name conflict. To ease debugging I have pushed a new version to the repository, following your suggestion, in which the folders have been renamed to mypyx. However I am still seeing the same incorrect output, namely:

mypyx\main.py:1: error: Function is missing a type annotation
mypyx\main.py:5: error: Function is missing a type annotation
mypyx\main.py:9: error: Function is missing a type annotation

[MarkCBell]

@gvanrossum Thank you for letting me know about the order in which locations are checked. After renaming my example to mypyx, I have verified that if the ./stubs/mypyx folder is moved to the current folder that I am in when I start mypy (location 1) or the folder where sources given on command line live (location 2) then mypy produces the expected output.

However, if I try to achieve this by setting MYPYPATH or mypy_path within mypy.ini to point to ./stubs/ then the mypy gives an incorrect output and the log produced by mypy -v ./tests/test-main.py contains LOG: Metadata not found for mypyx.main.