INTERNAL ERROR in mypy 0.610

[TheKevJames]

This issue appears when running python -m mypy --show-traceback --ignore-missing-imports PROJ on CircleCI in the python:3.6.6 image. Here's the CircleCI job definition:

  typecheck:
    docker:
      - image: python:3.6.6
    steps:
      - checkout
      - run: python -m pip install -r requirements.test.txt
      - run: python -m mypy --show-traceback --ignore-missing-imports PROJ

Oddly, running the above locally does not cause any issues:

$ docker run --rm -it -v /path/to/repo:/src python:3.6.6 bash
$ cd /src
$ python -m pip install -r requirements.test.txt
$ python -m mypy --show-traceback --ignore-missing-imports PROJ
$

Traceback:

PROJ/FILE.py: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.610
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/site-packages/mypy/__main__.py", line 11, in <module>
    main(None)
  File "/usr/local/lib/python3.6/site-packages/mypy/main.py", line 91, in main
    res = type_check_only(sources, bin_dir, options, flush_errors, fscache)  # noqa
  File "/usr/local/lib/python3.6/site-packages/mypy/main.py", line 148, in type_check_only
    fscache=fscache)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 185, in build
    flush_errors, fscache)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 285, in _build
    graph = dispatch(sources, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 2398, in dispatch
    process_graph(graph, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 2691, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 2801, in process_stale_scc
    graph[id].semantic_analysis()
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 2009, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath, self.options, patches)
  File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 1818, in wrap_context
    yield
  File "/usr/local/lib/python3.6/site-packages/mypy/build.py", line 2009, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath, self.options, patches)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 291, in visit_file
    v.type = self.anal_type(v.type)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 1578, in anal_type
    typ = t.accept(a)
  File "/usr/local/lib/python3.6/site-packages/mypy/types.py", line 203, in accept
    return visitor.visit_unbound_type(self)
  File "/usr/local/lib/python3.6/site-packages/mypy/typeanal.py", line 178, in visit_unbound_type
    typ = self.visit_unbound_type_nonoptional(t)
  File "/usr/local/lib/python3.6/site-packages/mypy/typeanal.py", line 186, in visit_unbound_type_nonoptional
    sym = self.lookup(t.name, t, suppress_errors=self.third_pass)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 3058, in lookup_qualified
    n = self.lookup(parts[0], ctx, suppress_errors=suppress_errors)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 3038, in lookup
    self.name_not_defined(name, ctx)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 3246, in name_not_defined
    self.fail(message, ctx)
  File "/usr/local/lib/python3.6/site-packages/mypy/semanal.py", line 3281, in fail
    self.errors.report(ctx.get_line(), ctx.get_column(), msg, blocker=blocker)
  File "/usr/local/lib/python3.6/site-packages/mypy/errors.py", line 260, in report
    target=self.current_target())
  File "/usr/local/lib/python3.6/site-packages/mypy/errors.py", line 208, in current_target
    return self.scope.current_target()
  File "/usr/local/lib/python3.6/site-packages/mypy/scope.py", line 30, in current_target
    assert self.module
AssertionError: 
PROJ/FILE.py: : note: use --pdb to drop into pdb
Exited with code 1

[emmatyping]

Hi, I think this is a duplicate of #5121. Can you run without --ignore-missing-imports to confirm? (It should say it cannot find the builtins module). I'm not yet sure why.

[emmatyping]

Also if you add git+https://github.com/python/mypy.git in place of mypy in your requirements.txt, I recently added some better debugging information around this.

[TheKevJames]

@ethanhs
Great guess, that definitely was the case. As an aside, I wonder if we can avoid hiding the builtins error when --ignore-missing-imports is passed. You'll notice that even the exit code changes in this case!

The linked ticket recommends "installing as recommended" rather than from a .zip; I've now installed both from PyPI and from git directly with the same result. Is there a different recommended installation method?

Happy to have this closed as a duplicate. Find attached the result of running all four combinations of args against my project below (the below was run against an install from GitHub at c77e27a).

$ python -m mypy PROJ
PROJ/__init__.py:1: error: Cannot find 'builtins' module. Typeshed appears broken!
Exited with code 2

$ python -m mypy --ignore-missing-imports PROJ
PROJ/FILE.py: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.620+dev-c77e27ad586338d68e7c003d976315d92d599f62
PROJ/FILE.py: : note: please use --show-traceback to print a traceback when reporting a bug
Exited with code 1

$ python -m mypy --show-traceback PROJ
PROJ/__init__.py:1: error: Cannot find 'builtins' module. Typeshed appears broken!
Exited with code 2

$ python -m mypy --show-traceback --ignore-missing-imports PROJ
# see traceback in inital report

[emmatyping]

@TheKevJames I'm glad you confirmed my suspicions. I definitely want to get to the bottom of this, because installing from PyPi and git both are the main recommended ways of installation (other than cloning manually then running pip install .). Can you tell me if typeshed exists in the container? (eg find / | grep typeshed).

Since it looks like your using the default Python docker container, I believe it is using Debian Stretch, is that correct?

Thank you for working with me on this!

[TheKevJames]

@ethanhs and that led me to the correct answer! Thank you!

Root cause:
I had CircleCI configured to cache my Python packages with:

      - restore_cache:
          keys:
            - cache-pip-{{ checksum "requirements.txt" }}{{ checksum "requirements.test.txt" }}
      - run: python -m pip install -r requirements.test.txt
      - save_cache:
          key: cache-pip-{{ checksum "requirements.txt" }}{{ checksum "requirements.test.txt" }}
          paths:
            - /usr/local/bin/codecov
            - /usr/local/bin/coverage
            - /usr/local/lib/python3.6/site-packages
            - ~/.cache/pip

The mypy typeshed does not live within the above listed directories, so any future builds using that cache would not have the typeshed copied in.

Replacing the save_cache step with the following (note the addition of /usr/local/lib/mypy) ensured that the typeshed would be cached properly:

      - save_cache:
          key: cache-pip-{{ checksum "requirements.txt" }}{{ checksum "requirements.test.txt" }}
          paths:
            - /usr/local/bin/codecov
            - /usr/local/bin/coverage
            - /usr/local/lib/mypy
            - /usr/local/lib/python3.6/site-packages
            - ~/.cache/pip

[emmatyping]

Excellent, glad you figured it out!

I think this might motivate me to finally move our typeshed installs to package data vs data files, as this is a clear example of where package data is better. That being said, I don't think it is advisable to cache site-packages, as not all packages install everything there.

[gvanrossum]

Maybe it's time to revive #2025?

[emmatyping]

@gvanrossum I think you might have mistyped the issue number?

[TheKevJames]

@ethanhs is probably right re: avoiding caching the site-packages directory. Mostly this list of cached dirs has been something I've built up over time as "all the places I need to cache to make pip not have to reinstall packages between CircleCI builds".

Unfortunately, there's very little insight into this from the CircleCI side of things. Starting from this support/discussion page is about the closest I've found to advice on this, and as you can see I've needed to add a few things to that list.

The site-packages cache is at least a part of the installed packages which I want cached. It definitely causes cache invalidation issues as shown in this ticket, but it also cuts off a significant amount of time from each build. Maybe I should re-evaluate those trade-offs...

[gvanrossum]

Oh, I meant the typeshed issue: python/typeshed#2025

[emmatyping]

Ah, I wasn't actually going to suggest that, but it is definitely a solution. I'll open an issue to explore the options.