dmypy: ValueError: value is not in list when running a second time #12744

zzzeek · 2022-05-07T16:15:24Z

I've had problems with this situation on and off for many weeks now, with 0.941 and now 0.950. sometimes it isn't happening at all, and other times, like at the moment, I get this stack trace reliably on any second run of dympy. Typically, once I get the files I'm typing in better shape, the error goes away. But to get there my process is greatly slowed by dmypy crashing on every non-initial run making it mostly not useful compared to running plain mypy.

reproduction steps follow below. the crash is:

$ dmypy  check lib
Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 230, in serve
  File "mypy/dmypy_server.py", line 273, in run_command
  File "mypy/dmypy_server.py", line 341, in cmd_check
  File "mypy/dmypy_server.py", line 394, in check
  File "mypy/dmypy_server.py", line 569, in fine_grained_increment_follow_imports
  File "mypy/server/update.py", line 245, in update
  File "mypy/server/update.py", line 328, in update_one
  File "mypy/server/update.py", line 393, in update_module
  File "mypy/server/update.py", line 615, in update_module_isolated
  File "mypy/build.py", line 2192, in type_check_first_pass
  File "mypy/checker.py", line 319, in check_first_pass
  File "mypy/checker.py", line 427, in accept
  File "mypy/errors.py", line 902, in report_internal_error
  File "mypy/checker.py", line 425, in accept
  File "mypy/nodes.py", line 1021, in accept
  File "mypy/checker.py", line 1810, in visit_class_def
  File "mypy/checker.py", line 427, in accept
  File "mypy/errors.py", line 902, in report_internal_error
  File "mypy/checker.py", line 425, in accept
  File "mypy/nodes.py", line 1092, in accept
  File "mypy/checker.py", line 2174, in visit_block
  File "mypy/checker.py", line 427, in accept
  File "mypy/errors.py", line 902, in report_internal_error
  File "mypy/checker.py", line 425, in accept
  File "mypy/nodes.py", line 740, in accept
  File "mypy/checker.py", line 776, in visit_func_def
  File "mypy/checker.py", line 780, in _visit_func_def
  File "mypy/checker.py", line 842, in check_func_item
  File "mypy/checker.py", line 1027, in check_func_def
  File "mypy/checker.py", line 427, in accept
  File "mypy/errors.py", line 902, in report_internal_error
  File "mypy/checker.py", line 425, in accept
  File "mypy/nodes.py", line 1092, in accept
  File "mypy/checker.py", line 2174, in visit_block
  File "mypy/checker.py", line 427, in accept
  File "mypy/errors.py", line 902, in report_internal_error
  File "mypy/checker.py", line 425, in accept
  File "mypy/nodes.py", line 1316, in accept
  File "mypy/checker.py", line 3581, in visit_if_stmt
  File "mypy/checkexpr.py", line 3983, in accept
  File "mypy/errors.py", line 902, in report_internal_error
  File "mypy/checkexpr.py", line 3981, in accept
  File "mypy/nodes.py", line 1897, in accept
  File "mypy/checkexpr.py", line 2244, in visit_comparison_expr
  File "mypy/checkexpr.py", line 3983, in accept
  File "mypy/errors.py", line 902, in report_internal_error
  File "mypy/checkexpr.py", line 3981, in accept
  File "mypy/nodes.py", line 1653, in accept
  File "mypy/checkexpr.py", line 197, in visit_name_expr
  File "mypy/checkexpr.py", line 224, in analyze_ref_expr
  File "mypy/checkmember.py", line 969, in type_object_type
ValueError: value is not in list

steps to reproduce:

using Python 3.10.0 in case that's important
dmypy 0.950 - this is a straight install from pypi, though ive had the issue in the past running from a source install also
git clone SQLAlchemy from gerrit and cd into new directory:

$ git clone https://gerrit.sqlalchemy.org/sqlalchemy/sqlalchemy
$ cd sqlalchemy

download a specific patch that is reproducing the issue:

$ git fetch https://gerrit.sqlalchemy.org/sqlalchemy/sqlalchemy refs/changes/25/3825/2 && git checkout FETCH_HEAD

start dmypy and run an initial check against lib. there's about 50 typing errors generated as expected

$ dmypy start
$ dmypy check lib
... output follows ...
Found 50 errors in 1 file (checked 700 source files)

make any change, like adding a blank line, to lib/sqlalchemy/orm/decl_api.py:

$ printf "\n" >> lib/sqlalchemy/orm/decl_api.py

run "dmypy recheck". crash

$ dympy recheck

... stack trace...
  File "mypy/checkexpr.py", line 197, in visit_name_expr
  File "mypy/checkexpr.py", line 224, in analyze_ref_expr
  File "mypy/checkmember.py", line 969, in type_object_type
ValueError: value is not in list

I'm able to reproduce this from running the steps above in a brand new directory.

I fully expect there is something in my code making it do this which I am sure is some horrible mistake I'm making, however, dmypy should tell me what the mistake is rather than crashing. Thanks and sorry for the complex reproduction instructions

The text was updated successfully, but these errors were encountered:

zzzeek · 2022-05-07T20:46:18Z

Here's a better error message from 0.960 in current master, from the daemon log file

lib/sqlalchemy/orm/decl_api.py:183: error: Call to untyped function "Synonym" in typed context  [no-untyped-call]
lib/sqlalchemy/orm/decl_api.py:273: error: Function is missing a type annotation  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:276: error: Function is missing a return type annotation  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:279: error: Function is missing a type annotation for one or more arguments  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:315: error: Incompatible return value type (got "Union[Mapped[_T], SQLCoreOperations[_T]]", expected "Union[InstrumentedAttribute[_T], _T]")  [return-value]
lib/sqlalchemy/orm/decl_api.py:319: error: Incompatible return value type (got "Union[Mapped[_T], SQLCoreOperations[_T]]", expected "Union[InstrumentedAttribute[_T], _T]")  [return-value]
lib/sqlalchemy/orm/decl_api.py:328: error: Returning Any from function declared to return "Union[InstrumentedAttribute[_T], _T]"  [no-any-return]
lib/sqlalchemy/orm/decl_api.py:331: error: Incompatible return value type (got "Union[Mapped[_T], SQLCoreOperations[_T]]", expected "Union[InstrumentedAttribute[_T], _T]")  [return-value]
lib/sqlalchemy/orm/decl_api.py:334: error: Function is missing a type annotation  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:335: error: Call to untyped function "_stateful_declared_attr" in typed context  [no-untyped-call]
lib/sqlalchemy/orm/decl_api.py:338: error: Function is missing a return type annotation  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:400: error: Missing type parameters for generic type "declared_attr"  [type-arg]
lib/sqlalchemy/orm/decl_api.py:401: error: Function is missing a type annotation  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:404: error: Function is missing a type annotation  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:407: error: Call to untyped function "_stateful_declared_attr" in typed context  [no-untyped-call]
lib/sqlalchemy/orm/decl_api.py:409: error: Function is missing a type annotation  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:413: error: Function is missing a return type annotation  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:455: error: Function is missing a return type annotation  [no-untyped-def]
lib/sqlalchemy/orm/decl_api.py:455: note: Use "-> None" if function does not return a value
lib/sqlalchemy/orm/decl_api.py:493: error: Missing type parameters for generic type "Mapper"  [type-arg]
lib/sqlalchemy/orm/decl_api.py:506: error: Variable "sqlalchemy.orm.decl_api.DeclarativeBaseNoMeta.registry" is not valid as a type  [valid-type]
lib/sqlalchemy/orm/decl_api.py:506: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
lib/sqlalchemy/orm/decl_api.py:508: error: Missing type parameters for generic type "Mapper"  [type-arg]
lib/sqlalchemy/orm/decl_api.py:524: error: Name "InstanceState" is not defined  [name-defined]
lib/sqlalchemy/orm/decl_api.py:586: error: Variable "sqlalchemy.orm.decl_api.DeclarativeBase.registry" is not valid as a type  [valid-type]
lib/sqlalchemy/orm/decl_api.py:586: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
lib/sqlalchemy/orm/decl_api.py:588: error: Missing type parameters for generic type "Mapper"  [type-arg]
lib/sqlalchemy/orm/decl_api.py:831: error: Missing type parameters for generic type "Type"  [type-arg]
lib/sqlalchemy/orm/decl_api.py:831: error: Missing type parameters for generic type "TypeEngine"  [type-arg]
lib/sqlalchemy/orm/decl_api.py:517: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.269adee09330f6f1836b0ae13e20450c9f708c2a
lib/sqlalchemy/orm/decl_api.py:517: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.269adee09330f6f1836b0ae13e20450c9f708c2a
lib/sqlalchemy/orm/decl_api.py:517: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.269adee09330f6f1836b0ae13e20450c9f708c2a
lib/sqlalchemy/orm/decl_api.py:516: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.269adee09330f6f1836b0ae13e20450c9f708c2a
lib/sqlalchemy/orm/decl_api.py:516: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.269adee09330f6f1836b0ae13e20450c9f708c2a
lib/sqlalchemy/orm/decl_api.py:493: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.269adee09330f6f1836b0ae13e20450c9f708c2a
lib/sqlalchemy/orm/decl_api.py:493: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.269adee09330f6f1836b0ae13e20450c9f708c2a
lib/sqlalchemy/orm/decl_api.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.269adee09330f6f1836b0ae13e20450c9f708c2a
Traceback (most recent call last):
  File "/home/classic/dev/mypy/mypy/dmypy_server.py", line 230, in serve
    resp = self.run_command(command, data)
  File "/home/classic/dev/mypy/mypy/dmypy_server.py", line 273, in run_command
    return method(self, **data)
  File "/home/classic/dev/mypy/mypy/dmypy_server.py", line 341, in cmd_check
    return self.check(sources, is_tty, terminal_width)
  File "/home/classic/dev/mypy/mypy/dmypy_server.py", line 394, in check
    messages = self.fine_grained_increment_follow_imports(sources)
  File "/home/classic/dev/mypy/mypy/dmypy_server.py", line 569, in fine_grained_increment_follow_imports
    messages = fine_grained_manager.update(changed, [])
  File "/home/classic/dev/mypy/mypy/server/update.py", line 245, in update
    result = self.update_one(changed_modules, initial_set, removed_set, blocking_error)
  File "/home/classic/dev/mypy/mypy/server/update.py", line 328, in update_one
    result = self.update_module(next_id, next_path, next_id in removed_set)
  File "/home/classic/dev/mypy/mypy/server/update.py", line 392, in update_module
    result = update_module_isolated(module, path, manager, previous_modules, graph,
  File "/home/classic/dev/mypy/mypy/server/update.py", line 618, in update_module_isolated
    state.type_check_first_pass()
  File "/home/classic/dev/mypy/mypy/build.py", line 2206, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/home/classic/dev/mypy/mypy/checker.py", line 315, in check_first_pass
    self.accept(d)
  File "/home/classic/dev/mypy/mypy/checker.py", line 423, in accept
    report_internal_error(err, self.errors.file, stmt.line, self.errors, self.options)
  File "/home/classic/dev/mypy/mypy/errors.py", line 962, in report_internal_error
    raise err
  File "/home/classic/dev/mypy/mypy/checker.py", line 421, in accept
    stmt.accept(self)
  File "/home/classic/dev/mypy/mypy/nodes.py", line 1028, in accept
    return visitor.visit_class_def(self)
  File "/home/classic/dev/mypy/mypy/checker.py", line 1806, in visit_class_def
    self.accept(defn.defs)
  File "/home/classic/dev/mypy/mypy/checker.py", line 423, in accept
    report_internal_error(err, self.errors.file, stmt.line, self.errors, self.options)
  File "/home/classic/dev/mypy/mypy/errors.py", line 962, in report_internal_error
    raise err
  File "/home/classic/dev/mypy/mypy/checker.py", line 421, in accept
    stmt.accept(self)
  File "/home/classic/dev/mypy/mypy/nodes.py", line 1099, in accept
    return visitor.visit_block(self)
  File "/home/classic/dev/mypy/mypy/checker.py", line 2168, in visit_block
    self.accept(s)
  File "/home/classic/dev/mypy/mypy/checker.py", line 423, in accept
    report_internal_error(err, self.errors.file, stmt.line, self.errors, self.options)
  File "/home/classic/dev/mypy/mypy/errors.py", line 962, in report_internal_error
    raise err
  File "/home/classic/dev/mypy/mypy/checker.py", line 421, in accept
    stmt.accept(self)
  File "/home/classic/dev/mypy/mypy/nodes.py", line 747, in accept
    return visitor.visit_func_def(self)
  File "/home/classic/dev/mypy/mypy/checker.py", line 772, in visit_func_def
    self._visit_func_def(defn)
  File "/home/classic/dev/mypy/mypy/checker.py", line 776, in _visit_func_def
    self.check_func_item(defn, name=defn.name)
  File "/home/classic/dev/mypy/mypy/checker.py", line 838, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/home/classic/dev/mypy/mypy/checker.py", line 1023, in check_func_def
    self.accept(item.body)
  File "/home/classic/dev/mypy/mypy/checker.py", line 423, in accept
    report_internal_error(err, self.errors.file, stmt.line, self.errors, self.options)
  File "/home/classic/dev/mypy/mypy/errors.py", line 962, in report_internal_error
    raise err
  File "/home/classic/dev/mypy/mypy/checker.py", line 421, in accept
    stmt.accept(self)
  File "/home/classic/dev/mypy/mypy/nodes.py", line 1099, in accept
    return visitor.visit_block(self)
  File "/home/classic/dev/mypy/mypy/checker.py", line 2168, in visit_block
    self.accept(s)
  File "/home/classic/dev/mypy/mypy/checker.py", line 423, in accept
    report_internal_error(err, self.errors.file, stmt.line, self.errors, self.options)
  File "/home/classic/dev/mypy/mypy/errors.py", line 962, in report_internal_error
    raise err
  File "/home/classic/dev/mypy/mypy/checker.py", line 421, in accept
    stmt.accept(self)
  File "/home/classic/dev/mypy/mypy/nodes.py", line 1323, in accept
    return visitor.visit_if_stmt(self)
  File "/home/classic/dev/mypy/mypy/checker.py", line 3575, in visit_if_stmt
    t = get_proper_type(self.expr_checker.accept(e))
  File "/home/classic/dev/mypy/mypy/checkexpr.py", line 3943, in accept
    report_internal_error(err, self.chk.errors.file,
  File "/home/classic/dev/mypy/mypy/errors.py", line 962, in report_internal_error
    raise err
  File "/home/classic/dev/mypy/mypy/checkexpr.py", line 3941, in accept
    typ = node.accept(self)
  File "/home/classic/dev/mypy/mypy/nodes.py", line 1904, in accept
    return visitor.visit_comparison_expr(self)
  File "/home/classic/dev/mypy/mypy/checkexpr.py", line 2215, in visit_comparison_expr
    left_type = self.accept(left)
  File "/home/classic/dev/mypy/mypy/checkexpr.py", line 3943, in accept
    report_internal_error(err, self.chk.errors.file,
  File "/home/classic/dev/mypy/mypy/errors.py", line 962, in report_internal_error
    raise err
  File "/home/classic/dev/mypy/mypy/checkexpr.py", line 3941, in accept
    typ = node.accept(self)
  File "/home/classic/dev/mypy/mypy/nodes.py", line 1660, in accept
    return visitor.visit_name_expr(self)
  File "/home/classic/dev/mypy/mypy/checkexpr.py", line 206, in visit_name_expr
    result = self.analyze_ref_expr(e)
  File "/home/classic/dev/mypy/mypy/checkexpr.py", line 233, in analyze_ref_expr
    result = type_object_type(node, self.named_type)
  File "/home/classic/dev/mypy/mypy/checkmember.py", line 970, in type_object_type
    init_index = info.mro.index(init_method.node.info)
ValueError: <TypeInfo sqlalchemy.orm.decl_api.DeclarativeBaseNoMeta> is not in list

ilevkivskyi · 2022-11-15T18:56:08Z

A simple test case to repro

[case testSqlalchemyCrash]
import m
[file m.py]
import typing
from typing import Any

class DeclarativeBaseNoMeta:
    if True:
        def __init__(self, **kw: Any):
            ...

DeclarativeBaseNoMeta
[file m.py.2]
import typing
from typing import Any

class DeclarativeBaseNoMeta:
    if True:
        def __init__(self, **kw: Any):
            ...

DeclarativeBaseNoMeta
# change
[builtins fixtures/dict.pyi]
[out]
==

It looks like original_def in this case points to the function itself, and this interferes with replace_object_state(). Probably not hard to fix, I will give it a try later today.

ilevkivskyi · 2022-11-15T21:24:16Z

Oh wow, it is actually worse. Because of a bug in visit_block() in astmerge, function/method definitions that appear in all nested positions (if, try, with, etc), will never get processed correctly by the daemon, their stale versions will persist. I guess this may be a major source of daemon bugs/crashes, will submit a PR in few minutes.

cc @JukkaL

Fixes #12744 The fix is straightforward. Current logic can produce a random mix of old and new nodes if there are functions/methods nested in other statements.

As I need dmypy to work without facing [1], I am running the latest build of mypy which seems so far to finally not have that issue. update constructs that latest mypy is being more picky about, including better typing for the _NONE_NAME symbol used in constraints (porting those elements from the Enum patch at I15ac3daee770408b5795746f47c1bbd931b7d26d) [1] python/mypy#12744 Change-Id: Ib3f56787fa65ea9bb2e6a0bccc4d99f54c516dad

zzzeek · 2022-11-29T23:25:46Z

my dmypy use is totally different since this bug was fixed, I can now leave it running and "dmypy check lib" doesn't crash even when I check out different branches. Previously, any code changes beyond a dozen lines would guaranteed crash every time. looks like it was all the same thing, thanks for fixing!

ilevkivskyi · 2022-11-30T09:09:33Z

Glad to hear it helped!

zzzeek added the bug mypy got something wrong label May 7, 2022

AlexWaygood added crash topic-daemon dmypy and removed bug mypy got something wrong labels May 7, 2022

ilevkivskyi mentioned this issue Nov 15, 2022

Correctly process nested definitions in astmerge #14104

Merged

ilevkivskyi closed this as completed in #14104 Nov 15, 2022

ilevkivskyi added a commit that referenced this issue Nov 15, 2022

Correctly process nested definitions in astmerge (#14104)

0d2a954

Fixes #12744 The fix is straightforward. Current logic can produce a random mix of old and new nodes if there are functions/methods nested in other statements.

AlexWaygood mentioned this issue Dec 2, 2022

dmypy deameon crash after switching branches #14234

Closed

zzzeek mentioned this issue Feb 8, 2023

dmypy: AssertionError: builtins.int crash for running a second time w/ source changes #14645

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

dmypy: ValueError: value is not in list when running a second time #12744

dmypy: ValueError: value is not in list when running a second time #12744

zzzeek commented May 7, 2022

zzzeek commented May 7, 2022

Uh oh!

ilevkivskyi commented Nov 15, 2022

Uh oh!

ilevkivskyi commented Nov 15, 2022

Uh oh!

zzzeek commented Nov 29, 2022

Uh oh!

ilevkivskyi commented Nov 30, 2022

Uh oh!

Uh oh!

dmypy: ValueError: value is not in list when running a second time #12744

dmypy: ValueError: value is not in list when running a second time #12744

Comments

zzzeek commented May 7, 2022

zzzeek commented May 7, 2022

Uh oh!

ilevkivskyi commented Nov 15, 2022

Uh oh!

ilevkivskyi commented Nov 15, 2022

Uh oh!

zzzeek commented Nov 29, 2022

Uh oh!

ilevkivskyi commented Nov 30, 2022

Uh oh!