Add daemon command to get type of an expression (#13209)

Implementation is straightforward (I also tried to make it future-proof, so it is easy to add new inspections). Few things to point out:
* I added a more flexible traverser class, that supports shared visit logic and allows for `O(log n)` search by early return.
* I noticed a bunch of places where line/column/full name were not set. Did a quick pass and fixed as many as I can.
* I also implement basic support for expression attributes and go to definition (there are few tricky corner cases left, but they can be addressed later).

The current _default_ logic is optimized for speed, while it may give stale results (e.g if file was edited, but has full tree loaded in memory). My thinking here is that users typically want to get inspections really fast (like on mouse hover over an expression), so the editor integrations that would use this can call with default flag values, and only use `--force-reload` or a full `dmypy recheck` if they know some file(s) were edited.

Co-authored-by: Ivan Levkivskyi <[email protected]>

Author: ilevkivskyi

docs/source/mypy_daemon.rst

@@ -152,6 +152,12 @@ Additional daemon flags
    Write performance profiling information to ``FILE``. This is only available
    for the ``check``, ``recheck``, and ``run`` commands.
 
+.. option:: --export-types
+
+   Store all expression types in memory for future use. This is useful to speed
+   up future calls to ``dmypy inspect`` (but uses more memory). Only valid for
+   ``check``, ``recheck``, and ``run`` command.
+
 Static inference of annotations
 *******************************
 
@@ -243,8 +249,129 @@ command.
 
    Set the maximum number of types to try for a function (default: ``64``).
 
-.. TODO: Add similar sections about go to definition, find usages, and
-   reveal type when added, and then move this to a separate file.
+Statically inspect expressions
+******************************
+
+The daemon allows to get declared or inferred type of an expression (or other
+information about an expression, such as known attributes or definition location)
+using ``dmypy inspect LOCATION`` command. The location of the expression should be
+specified in the format ``path/to/file.py:line:column[:end_line:end_column]``.
+Both line and column are 1-based. Both start and end position are inclusive.
+These rules match how mypy prints the error location in error messages.
+
+If a span is given (i.e. all 4 numbers), then only an exactly matching expression
+is inspected. If only a position is given (i.e. 2 numbers, line and column), mypy
+will inspect all *expressions*, that include this position, starting from the
+innermost one.
+
+Consider this Python code snippet:
+
+.. code-block:: python
+
+   def foo(x: int, longer_name: str) -> None:
+       x
+       longer_name
+
+Here to find the type of ``x`` one needs to call ``dmypy inspect src.py:2:5:2:5``
+or ``dmypy inspect src.py:2:5``. While for ``longer_name`` one needs to call
+``dmypy inspect src.py:3:5:3:15`` or, for example, ``dmypy inspect src.py:3:10``.
+Please note that this command is only valid after daemon had a successful type
+check (without parse errors), so that types are populated, e.g. using
+``dmypy check``. In case where multiple expressions match the provided location,
+their types are returned separated by a newline.
+
+Important note: it is recommended to check files with :option:`--export-types`
+since otherwise most inspections will not work without :option:`--force-reload`.
+
+.. option:: --show INSPECTION
+
+   What kind of inspection to run for expression(s) found. Currently the supported
+   inspections are:
+
+   * ``type`` (default): Show the best known type of a given expression.
+   * ``attrs``: Show which attributes are valid for an expression (e.g. for
+     auto-completion). Format is ``{"Base1": ["name_1", "name_2", ...]; "Base2": ...}``.
+     Names are sorted by method resolution order. If expression refers to a module,
+     then module attributes will be under key like ``"<full.module.name>"``.
+   * ``definition`` (experimental): Show the definition location for a name
+     expression or member expression. Format is ``path/to/file.py:line:column:Symbol``.
+     If multiple definitions are found (e.g. for a Union attribute), they are
+     separated by comma.
+
+.. option:: --verbose
+
+   Increase verbosity of types string representation (can be repeated).
+   For example, this will print fully qualified names of instance types (like
+   ``"builtins.str"``), instead of just a short name (like ``"str"``).
+
+.. option:: --limit NUM
+
+   If the location is given as ``line:column``, this will cause daemon to
+   return only at most ``NUM`` inspections of innermost expressions.
+   Value of 0 means no limit (this is the default). For example, if one calls
+   ``dmypy inspect src.py:4:10 --limit=1`` with this code
+
+   .. code-block:: python
+
+      def foo(x: int) -> str: ..
+      def bar(x: str) -> None: ...
+      baz: int
+      bar(foo(baz))
+
+   This will output just one type ``"int"`` (for ``baz`` name expression).
+   While without the limit option, it would output all three types: ``"int"``,
+   ``"str"``, and ``"None"``.
+
+.. option:: --include-span
+
+   With this option on, the daemon will prepend each inspection result with
+   the full span of corresponding expression, formatted as ``1:2:1:4 -> "int"``.
+   This may be useful in case multiple expressions match a location.
+
+.. option:: --include-kind
+
+   With this option on, the daemon will prepend each inspection result with
+   the kind of corresponding expression, formatted as ``NameExpr -> "int"``.
+   If both this option and :option:`--include-span` are on, the kind will
+   appear first, for example ``NameExpr:1:2:1:4 -> "int"``.
+
+.. option:: --include-object-attrs
+
+   This will make the daemon include attributes of ``object`` (excluded by
+   default) in case of an ``atts`` inspection.
+
+.. option:: --union-attrs
+
+   Include attributes valid for some of possible expression types (by default
+   an intersection is returned). This is useful for union types of type variables
+   with values. For example, with this code:
+
+   .. code-block:: python
+
+      from typing import Union
+
+      class A:
+          x: int
+          z: int
+      class B:
+          y: int
+          z: int
+      var: Union[A, B]
+      var
+
+   The command ``dmypy inspect --show attrs src.py:10:1`` will return
+   ``{"A": ["z"], "B": ["z"]}``, while with ``--union-attrs`` it will return
+   ``{"A": ["x", "z"], "B": ["y", "z"]}``.
+
+.. option:: --force-reload
+
+   Force re-parsing and re-type-checking file before inspection. By default
+   this is done only when needed (for example file was not loaded from cache
+   or daemon was initially run without ``--export-types`` mypy option),
+   since reloading may be slow (up to few seconds for very large files).
+
+.. TODO: Add similar section about find usages when added, and then move
+   this to a separate file.
 
 
 .. _watchman: https://facebook.github.io/watchman/

mypy/checker.py

@@ -1727,7 +1727,7 @@ def expand_typevars(
         # Make a copy of the function to check for each combination of
         # value restricted type variables. (Except when running mypyc,
         # where we need one canonical version of the function.)
-        if subst and not self.options.mypyc:
+        if subst and not (self.options.mypyc or self.options.inspections):
             result: List[Tuple[FuncItem, CallableType]] = []
             for substitutions in itertools.product(*subst):
                 mapping = dict(substitutions)
@@ -3205,7 +3205,7 @@ def check_assignment_to_multiple_lvalues(
                 lr_pairs = list(zip(left_lvs, left_rvs))
                 if star_lv:
                     rv_list = ListExpr(star_rvs)
-                    rv_list.set_line(rvalue.get_line())
+                    rv_list.set_line(rvalue)
                     lr_pairs.append((star_lv.expr, rv_list))
                 lr_pairs.extend(zip(right_lvs, right_rvs))
 
@@ -3406,7 +3406,7 @@ def check_multi_assignment_from_tuple(
                 list_expr = ListExpr(
                     [self.temp_node(rv_type, context) for rv_type in star_rv_types]
                 )
-                list_expr.set_line(context.get_line())
+                list_expr.set_line(context)
                 self.check_assignment(star_lv.expr, list_expr, infer_lvalue_type)
             for lv, rv_type in zip(right_lvs, right_rv_types):
                 self.check_assignment(lv, self.temp_node(rv_type, context), infer_lvalue_type)
@@ -4065,7 +4065,7 @@ def visit_if_stmt(self, s: IfStmt) -> None:
     def visit_while_stmt(self, s: WhileStmt) -> None:
         """Type check a while statement."""
         if_stmt = IfStmt([s.expr], [s.body], None)
-        if_stmt.set_line(s.get_line(), s.get_column())
+        if_stmt.set_line(s)
         self.accept_loop(if_stmt, s.else_body, exit_condition=s.expr)
 
     def visit_operator_assignment_stmt(self, s: OperatorAssignmentStmt) -> None:
@@ -6540,7 +6540,7 @@ def flatten_types(t: Type) -> List[Type]:
 
 def expand_func(defn: FuncItem, map: Dict[TypeVarId, Type]) -> FuncItem:
     visitor = TypeTransformVisitor(map)
-    ret = defn.accept(visitor)
+    ret = visitor.node(defn)
     assert isinstance(ret, FuncItem)
     return ret
 

mypy/checkmember.py

@@ -224,6 +224,10 @@ def _analyze_member_access(
     elif isinstance(typ, NoneType):
         return analyze_none_member_access(name, typ, mx)
     elif isinstance(typ, TypeVarLikeType):
+        if isinstance(typ, TypeVarType) and typ.values:
+            return _analyze_member_access(
+                name, make_simplified_union(typ.values), mx, override_info
+            )
         return _analyze_member_access(name, typ.upper_bound, mx, override_info)
     elif isinstance(typ, DeletedType):
         mx.msg.deleted_as_rvalue(typ, mx.context)

mypy/dmypy/client.py

@@ -83,6 +83,11 @@ def __init__(self, prog: str) -> None:
 p.add_argument("--junit-xml", help="Write junit.xml to the given file")
 p.add_argument("--perf-stats-file", help="write performance information to the given file")
 p.add_argument("files", metavar="FILE", nargs="+", help="File (or directory) to check")
+p.add_argument(
+    "--export-types",
+    action="store_true",
+    help="Store types of all expressions in a shared location (useful for inspections)",
+)
 
 run_parser = p = subparsers.add_parser(
     "run",
@@ -96,6 +101,11 @@ def __init__(self, prog: str) -> None:
     "--timeout", metavar="TIMEOUT", type=int, help="Server shutdown timeout (in seconds)"
 )
 p.add_argument("--log-file", metavar="FILE", type=str, help="Direct daemon stdout/stderr to FILE")
+p.add_argument(
+    "--export-types",
+    action="store_true",
+    help="Store types of all expressions in a shared location (useful for inspections)",
+)
 p.add_argument(
     "flags",
     metavar="ARG",
@@ -113,6 +123,11 @@ def __init__(self, prog: str) -> None:
 p.add_argument("-q", "--quiet", action="store_true", help=argparse.SUPPRESS)  # Deprecated
 p.add_argument("--junit-xml", help="Write junit.xml to the given file")
 p.add_argument("--perf-stats-file", help="write performance information to the given file")
+p.add_argument(
+    "--export-types",
+    action="store_true",
+    help="Store types of all expressions in a shared location (useful for inspections)",
+)
 p.add_argument(
     "--update",
     metavar="FILE",
@@ -164,6 +179,68 @@ def __init__(self, prog: str) -> None:
     help="Set the maximum number of types to try for a function (default 64)",
 )
 
+inspect_parser = p = subparsers.add_parser(
+    "inspect", help="Locate and statically inspect expression(s)"
+)
+p.add_argument(
+    "location",
+    metavar="LOCATION",
+    type=str,
+    help="Location specified as path/to/file.py:line:column[:end_line:end_column]."
+    " If position is given (i.e. only line and column), this will return all"
+    " enclosing expressions",
+)
+p.add_argument(
+    "--show",
+    metavar="INSPECTION",
+    type=str,
+    default="type",
+    choices=["type", "attrs", "definition"],
+    help="What kind of inspection to run",
+)
+p.add_argument(
+    "--verbose",
+    "-v",
+    action="count",
+    default=0,
+    help="Increase verbosity of the type string representation (can be repeated)",
+)
+p.add_argument(
+    "--limit",
+    metavar="NUM",
+    type=int,
+    default=0,
+    help="Return at most NUM innermost expressions (if position is given); 0 means no limit",
+)
+p.add_argument(
+    "--include-span",
+    action="store_true",
+    help="Prepend each inspection result with the span of corresponding expression"
+    ' (e.g. 1:2:3:4:"int")',
+)
+p.add_argument(
+    "--include-kind",
+    action="store_true",
+    help="Prepend each inspection result with the kind of corresponding expression"
+    ' (e.g. NameExpr:"int")',
+)
+p.add_argument(
+    "--include-object-attrs",
+    action="store_true",
+    help='Include attributes of "object" in "attrs" inspection',
+)
+p.add_argument(
+    "--union-attrs",
+    action="store_true",
+    help="Include attributes valid for some of possible expression types"
+    " (by default an intersection is returned)",
+)
+p.add_argument(
+    "--force-reload",
+    action="store_true",
+    help="Re-parse and re-type-check file before inspection (may be slow)",
+)
+
 hang_parser = p = subparsers.add_parser("hang", help="Hang for 100 seconds")
 
 daemon_parser = p = subparsers.add_parser("daemon", help="Run daemon in foreground")
@@ -321,12 +398,24 @@ def do_run(args: argparse.Namespace) -> None:
         # Bad or missing status file or dead process; good to start.
         start_server(args, allow_sources=True)
     t0 = time.time()
-    response = request(args.status_file, "run", version=__version__, args=args.flags)
+    response = request(
+        args.status_file,
+        "run",
+        version=__version__,
+        args=args.flags,
+        export_types=args.export_types,
+    )
     # If the daemon signals that a restart is necessary, do it
     if "restart" in response:
         print(f"Restarting: {response['restart']}")
         restart_server(args, allow_sources=True)
-        response = request(args.status_file, "run", version=__version__, args=args.flags)
+        response = request(
+            args.status_file,
+            "run",
+            version=__version__,
+            args=args.flags,
+            export_types=args.export_types,
+        )
 
     t1 = time.time()
     response["roundtrip_time"] = t1 - t0
@@ -383,7 +472,7 @@ def do_kill(args: argparse.Namespace) -> None:
 def do_check(args: argparse.Namespace) -> None:
     """Ask the daemon to check a list of files."""
     t0 = time.time()
-    response = request(args.status_file, "check", files=args.files)
+    response = request(args.status_file, "check", files=args.files, export_types=args.export_types)
     t1 = time.time()
     response["roundtrip_time"] = t1 - t0
     check_output(response, args.verbose, args.junit_xml, args.perf_stats_file)
@@ -406,9 +495,15 @@ def do_recheck(args: argparse.Namespace) -> None:
     """
     t0 = time.time()
     if args.remove is not None or args.update is not None:
-        response = request(args.status_file, "recheck", remove=args.remove, update=args.update)
+        response = request(
+            args.status_file,
+            "recheck",
+            export_types=args.export_types,
+            remove=args.remove,
+            update=args.update,
+        )
     else:
-        response = request(args.status_file, "recheck")
+        response = request(args.status_file, "recheck", export_types=args.export_types)
     t1 = time.time()
     response["roundtrip_time"] = t1 - t0
     check_output(response, args.verbose, args.junit_xml, args.perf_stats_file)
@@ -437,6 +532,25 @@ def do_suggest(args: argparse.Namespace) -> None:
     check_output(response, verbose=False, junit_xml=None, perf_stats_file=None)
 
 
+@action(inspect_parser)
+def do_inspect(args: argparse.Namespace) -> None:
+    """Ask daemon to print the type of an expression."""
+    response = request(
+        args.status_file,
+        "inspect",
+        show=args.show,
+        location=args.location,
+        verbosity=args.verbose,
+        limit=args.limit,
+        include_span=args.include_span,
+        include_kind=args.include_kind,
+        include_object_attrs=args.include_object_attrs,
+        union_attrs=args.union_attrs,
+        force_reload=args.force_reload,
+    )
+    check_output(response, verbose=False, junit_xml=None, perf_stats_file=None)
+
+
 def check_output(
     response: Dict[str, Any],
     verbose: bool,