Function line number information is not always correct

[emmatyping]

Currently, the line number of the function definition is not correct leading to issues such as #3869.

In the Python ast (as well as type-ast) the FunctionDef node we use to, for example, report missing return annotations gives a line number of the beginning of the first decorator (if any). Therefore it can incorrectly report the line number information we need to report errors.

Our currently logic increments the line number of any decorators, however, we cannot depend on this to be consistent. Consider this problematic (but completely valid!) example:

@decorate(
        "multi",
        "line"
        )
def foo(
        a: int,
        b: str,
        ):

        pass

The FunctionDef.lineno would be 1, so mypy will report test.py:2: error: Function is missing a return type annotation. However, the return should be on line 8! Note that blanks space between the body and end of the function definition node means we cannot decrement based on the first item in the body of the function.

As a first possible solution, we could modify typed_ast to give the line number of the returns if the function is not typed (currently it just puts the attribute as None), so that the information is known, but it clearly indicates that the node is empty. This is certainly not an ideal solution.

Corresponding issue opened in python/typed_ast#50

[gvanrossum]

IMO asking for the line number where the return annotation should go (i.e. where the '):' is) is overkill, but asking for the line number for the 'def' is reasonable.

[emmatyping]

Hm, that is probably true now that I think about it more. That would also have the added bonus of better error messages if a function name is redefined. I was thinking that since on larger functions it is common to split arguments over multiple lines, it would be nice to give an exact line number of where to put the annotation, but now that I think about it, if we point to the name, it shouldn't be too bad for the user to modify the return of that function.

Perhaps we could add a def_lineno attribute to FunctionDef nodes in typed_ast? Then refactor mypy to use that of course.

[gvanrossum]

That sounds reasonable.

[ilevkivskyi]

I think we should do the same for classes (i.e. add class_lineno), mypy currently provides special support to only one class decorator @runtime, but IIRC @JelleZijlstra already got confused once when I put a # type: ignore on the class decorator line.

[JelleZijlstra]

I think I would have been confused too if that ignore comment had not been on the decorator line. :)

[emmatyping]

I agree with @ilevkivskyi adding a class_lineno is also probably a good idea.

If nothing else, it will correctly report name re-definitions better.

If there are no objections, I will start on a patch to typed_ast.

I got busy with a PEP, so this is up for grabs

[emmatyping]

I should mention there is an open bpo issue about this exact problem. https://bugs.python.org/issue33211