Cannot infer return type if using Union[Tuple[Literal[]]...]

[Naddiseo]

• Are you reporting a bug, or opening a feature request? Bug
• Please insert below the code you are checking with mypy,
  or a mock-up repro if the source is private. We would appreciate
  if you try to simplify your case to a minimal repro.

from __future__ import annotations

from typing import Union,Tuple
from typing_extensions import Literal

def does_not_work(arg: int) -> Union[Tuple[str], Tuple[Literal[1]]]:
    if arg > 5:
        return ("a",)
    else:
        return (1,) # Incompatible return value type (got "Tuple[int]", expected "Union[Tuple[str], Tuple[Literal[1]]]")

• What is the actual behavior/output?
  It fails to recognize that (1,) is a Tuple[Literal[1]]

• What is the behavior/output you expect?

• What are the versions of mypy and Python you are using?
  Do you see the same issue after installing mypy from Git master? yes

• What are the mypy flags you are using? (For example --strict-optional)

  • strict-optional, python-version = 3.7, warn_return_any, --strict

I'm not sure if this is expected behaviour, or a limitation, but I thought I should point it out since I couldn't find any existing bugs for this.

[JukkaL]

I can reproduce the issue. Apparently the literal type context is sometimes lost.

Some observations:

• If I replace all Tuple[...] types with List[...], the issue persists.
• If I drop Tuple[...] and use Union[str, Literal[1]], there is no issue.
• If I drop the union type context and use Tuple[Literal[1]], there is no issue.
• Strictness flags aren't needed to reproduce the issue.

cc @Michael0x2a

[cipherboy]

I think this is a deeper issue than just affecting Literal. (Feel free to tell me it is a different issue though and that I should file it as such).

Take for example:

from typing import Union, List, Tuple

a: List[bool] = [True, True, False]

b: Union[List[Union[bool, int]], Tuple[Union[bool, int], ...]] = a

print(b)

Which outputs:

$ python3 example.py  && mypy example.py
[True, True, False]
example.py:5: error: Incompatible types in assignment (expression has type "List[bool]", variable has type "Union[List[Union[bool, int]], Tuple[Union[bool, int]]]")

(They are compatible: Take the List branch from the outer Union and bool is of type Union[bool, int]).

Interesting this works when using Tuple:

from typing import Union, List, Tuple

a: Tuple[bool, ...] = (True, True, False)

b: Union[List[Union[bool, int]], Tuple[Union[bool, int], ...]] = a

print(b)

Which outputs:

$ python3 example.py  && mypy example.py 
(True, True, False)

Or, a simpler reproducer:

from typing import Union, List

a: List[bool] = [True, True, False]

b: List[Union[bool, int]] = a

print(b)

Which outputs:

$ python3 example.py  && mypy example.py 
[True, True, False]
example.py:5: error: Incompatible types in assignment (expression has type "List[bool]", variable has type "List[Union[bool, int]]")
example.py:5: note: "List" is invariant -- see http://mypy.readthedocs.io/en/latest/common_issues.html#variance
example.py:5: note: Consider using "Sequence" instead, which is covariant

I think invariance here is a red herring. Using Sequence or Iterable does indeed solve this particular problem though.

Note that the base types obviously work:

from typing import Union

a: bool = True

b: Union[bool, int] = a

print(b)

Which outputs:

$ python3 example.py  && mypy example.py 
True

---

My use case is padding of arrays. The padding might be of a fixed type (say, bool), whereas the rest of the elements in the list might be of other types composed in the Union inside the List. The result still needs to have type List[Union[bool, ...]] and can't take on List[bool].

from typing import Union, List

class MyClass:
    pass

def pad(data: List[Union[bool, int, MyClass]], width):
    amount = width - len(data)
    padding = [False] * amount
    return padding + data

print(pad([1, 2, 3], 5))

Which outputs:

$ python3 example.py  && mypy example.py 
[False, False, 1, 2, 3]
example.py:9: error: Unsupported operand types for + ("List[bool]" and "List[Union[bool, int, MyClass]]")

At any rate, that's my 2c. Hope it helps.

[JelleZijlstra]

What you describe is a consequence of List being invariant. See https://mypy.readthedocs.io/en/latest/common_issues.html#invariance-vs-covariance.

[cipherboy]

I guess I don't understand how or perhaps, that I feel it is too strict. :)

(Do tell me where I'm wrong in the below -- perhaps I just need to accept that assignment across explicit, compatible types can be called variance).

My understanding is that variance mostly relates to a class hierarchy (which I guess bool and int have, but let's ignore that for now -- the same example works on bool and str), and the inferred type of the object. Notably, the possible strategies section for solving variance-related issues include adding explicit type information: that's what I'm doing in the above examples. I'm surprised it isn't sufficient though.

In the Generic types section, we're given a Shape/Circle example:

class Shape:
    pass

class Circle(Shape):
    def rotate(self):
        ...

I do completely agree that, given a variable annotated as List[Circle], adding an item of Shape is unsafe. re-typing the list to List[Union[Circle, Shape]] should be fine, or even List[Shape]: your original type is a subset of the target type. Similarly, consider the following code:

class Shape:
    def area(self):
        ...

class Circle(Shape):
    def rotate(self):
        ...

def sum_area(shapes: List[Shape]):
    return sum(map(lambda x: x.area(), shapes))

l: List[Circle] = [Circle(), Circle(), Circle()]
print(sum_area(l))

We get the same (IMO red herring) message about covariance:

$ mypy example.py
example.py:15: error: Argument 1 to "sum_area" has incompatible type "List[Circle]"; expected "List[Shape]"
example.py:15: note: "List" is invariant -- see http://mypy.readthedocs.io/en/latest/common_issues.html#variance
example.py:15: note: Consider using "Sequence" instead, which is covariant
Found 1 error in 1 file (checked 1 source file)

But my understanding is, since we're going from a derived type to its parent type (Circle -> Shape -- and then only using type information from Shape to check sum_area) -- this isn't a matter of inferring new types, but instead, inferring that the two types are actually compatible.

This is demonstrated by changing .area() to .rotate() above:

$ mypy example.py
example.py:12: error: "Shape" has no attribute "rotate"
example.py:15: error: Argument 1 to "sum_area" has incompatible type "List[Circle]"; expected "List[Shape]"
example.py:15: note: "List" is invariant -- see http://mypy.readthedocs.io/en/latest/common_issues.html#variance
example.py:15: note: Consider using "Sequence" instead, which is covariant
Found 2 errors in 1 file (checked 1 source file)

I guess we could all call this covariance (though, assigning across explicit types seems like it should avoid covariance related discussions, unlike implicit typing where available information at the time of assignment makes type inference hard).

But to me it seems worthwhile to then notate that one type of covariance is easy to handle (removing features by going up the parent -> child class hierarchy) and the other is hard (adding new features by going down the parent -> child class hierarchy). The former definitely seems like something mypy could (with time and code...:) learn to handle correctly; the latter I agree should be a type error with invariant types (and more leniently handled with covariant types).

I don't see the value in an explicit call to list (which solves the first example's covariance warning) when both halves are explicitly annotated. It seems like an unannotated copy lets it act as an unnecessary gluing mechanism between two explicit, compatible types.

---

The following typechecks though, so I guess I'll live with lacking the explicit type annotation on to_list:

from typing import Union, List, Tuple, Sequence, Iterable, Optional

class Variable:
    pass

VariableLike = Union[bool, int]
VariableActually = Union[VariableLike, Variable]

class Vector:
    items: Tuple[VariableActually, ...]
    
    def __init__(self, items: Iterable[VariableActually]):
        self.items = tuple(items)

VectorLike = Union[Tuple[VariableActually, ...], List[VariableActually], int]
VectorActually = Union[VectorLike, Vector]

def to_list(obj: int): # NB: can't be annotated as List[bool] -- otherwise to_vector fails below
    # Lazy implementation. 
    return [False, False, False, obj % 2 == 0]

def to_vector(other: VectorActually, width: Optional[int] = None):
    if isinstance(other, Vector):
        return to_vector(other.items, width=width)
    
    # From this point on, mypy does detect that the remaining
    # types out of VectorActually for other are only those in
    # VectorLike.
    
    if isinstance(other, int):
        obj = to_list(other)
        return to_vector(obj, width=width)
    
    # From this point on, mypy does detect that the remaining
    # types out of VectorActually (now VectorLike) are the List/Tuple
    # of VariableActually's.
    
    if width:
        l_other = len(other)
        if l_other > width:
            raise ValueError("...")

        padding: List[VariableActually] = [False] * (width - l_other)
        return Vector(padding + list(other))
    return Vector(other)

[Michael0x2a]

I ran into this issue/issues similar to this while working on some literal types stuff, but don't quite have time to fix this now. So, here are some notes so I don't lose context once I circle back around to this...

---

I think for the original example, the root cause comes from this: https://github.com/python/mypy/blob/master/mypy/checkexpr.py#L3067

In short, if we have a union of tuples, we currently just completely give up on attempting to select a literal context at all.

We could maybe do together by "splicing" together a union of tuples -- e.g. if we have Union[Tuple[A, B], Tuple[C, D]], make the context basically be Tuple[Union[A, C], Union[B, D]].

I'm a little surprised we don't already have logic for doing this sort of thing, actually.

---

While doing some more testing, I also ran into the following similar example:

def does_not_work() -> Tuple[Literal[1]]:
    x: Final = (1,)
    return x

In this case, I'm pretty sure the issue stems from how we attempt to figure out if we should be using a literal context in https://github.com/python/mypy/blob/master/mypy/checkexpr.py#L242 -- we don't attempt to destructure the type and instead enable the literal context only if the variable corresponds to an instance.

We actually have a test case that I think fails for this reason, actually -- testLiteralFinalGoesOnlyOneLevelDown. Probably this test should be rewritten: I don't think it's actually testing what it thinks it is.

One quick-and-dirty (?) way of fixing this specific issue would be to modify is_subtype so we consider Literal[1] to be a subtype of any ints with a last-known-value of Literal[1]. After all, we already get "destructuring" checks for free when doing subtype tests.

This would also incidentally fix the first example, but only as a side-effect and for literal types only.

This tactic in general is unsafe if we've accidentally forgotten to erase the last known value field somewhere though, so I'm hesitant to use it.

---

I'm pretty sure the issue @cipherboy reported is unrelated though, and really is a case of mypy correctly reporting an issue with variance. For example, attempting to pass in a List[Circle] into sum_areas is unsafe because given its signature, you could do this:

class Shape:
    def area(self): ...

class Circle(Shape):
    def rotate(self): ...

class Triangle(Shape):
    def elongate(self): ...

def sum_area(shapes: List[Shape]):
    shapes.append(Triangle())  # Type-checks, since Triangles are a kind of Shape
    return sum(s.area() for s in shapes)

circles: List[Circle] = [Circle(), Circle(), Circle()]
sum_area(circles)
for c in circles:
    c.rotate()  # Eventually crashes at runtime since Triangles don't have a 'rotate' method

[ilevkivskyi]

@Michael0x2a

We could maybe do together by "splicing" together a union of tuples -- e.g. if we have Union[Tuple[A, B], Tuple[C, D]], make the context basically be Tuple[Union[A, C], Union[B, D]].
I'm a little surprised we don't already have logic for doing this sort of thing, actually.

Note that tuple of two unions is a union of four tuples in general. But yes, mypy is not very good at union math and it is indeed special-cased, so we have things like multiassig_from_union(). Also note there is #7187 that I want to fix in very near future.

[s-weigand]

I just stumbled across a similar issue, but w/o nested Unions (so maybe this should a different issue).
To me, it looks like Union and Tuple just won't play nicely together or maybe I just don't understand Union 🤔 .

The code below the minimal example of a plugin system where plugin classes are added to a register (dict).

from typing import MutableMapping, Tuple, Union


class A:
    pass


class B:
    pass


register_a: MutableMapping[str, A] = {}
register_b: MutableMapping[str, B] = {}


def add_to_register(
    key: str,
    item_register_tuple: Union[Tuple[A, MutableMapping[str, A]], Tuple[B, MutableMapping[str, B]]],
):
    item, register = item_register_tuple
    register[key] = item

$ mypy --version
mypy 0.812
$ mypy --show-error-codes --ignore-missing-imports dummy.py
dummy.py:21: error: Incompatible types in assignment (expression has type "Union[A, B]", target has type "A")  [assignment]
dummy.py:21: error: Incompatible types in assignment (expression has type "Union[A, B]", target has type "B")  [assignment]

The way I see it Union[Tuple[A, MutableMapping[str, A]], Tuple[B, MutableMapping[str, B]]] should mean Tuple[A, MutableMapping[str, A]] or Tuple[B, MutableMapping[str, B]]] and there is no Union[A, B] involved.

Please correct me if there is an error in my interpretation.

[huzecong]

I'm seeing a similar error here:

from typing import *

TElement = Union[Tuple[Literal["int"], int], Tuple[Literal["str"], str]]

xs: List[TElement] = []
xs.append(("int", 1))
xs.append(("str", "what"))

and mypy interprets the append arguments as Tuple[str, int] and Tuple[str, str] instead the literal types (mypy-play).