Rethink how we package typeshed

[emmatyping]

Currently, we have a rather complicated method of finding data files. The current method we use to distribute typeshed is ripe for issues with finding the files.

I think we should stop playing whack-a-mole with issues and ship typeshed differently. Here are two ideas:

1. Ship typeshed as a subpackage of mypy, while still keeping it a module. I originally proposed this in Move to using package_data to install typeshed and xml files. #4143 but was hung up on the symlinks in typeshed (which no longer exist). This would involve:
   • moving the submodule into the mypy package.
   • updating setup.py to use package_data
   • adding these files to MANIFEST.in to be picked up in sdists
2. Release typeshed as its own package and pin to a particular version. This was attempted in Add setup.py typeshed#2025 but it was abandoned.

I have no particular preference for one over the other, but I think both are better than the current method.

[emmatyping]

This came up again in #5319 so I am raising priority to high.

[emmatyping]

I think I will go ahead with option 1, unless anyone objects. It shouldn't be too hard to adopt option 2 later if typeshed becomes a package.

[JukkaL]

I also prefer option 1, at least for now (unless there are some major unforeseen issues with this approach, perhaps).

[ilevkivskyi]

OK, lets go with option 1. This way we can even be more flexible (no need to depend on typeshed releases).

[emmatyping]

Okay, I submitted #5517, which implements option 1. Hopefully this will mean fewer broken installs!

[TheKevJames]

I have no particular input on whether option 1 or 2 is nicer/simpler/better/whatever, but if we're picking anything other than option 2 its probably worth cleaning up the PyPI typeshed package to avoid cases where someone accidentally gets the wrong version of typeshed by mistakenly installing the pypi version.

Maybe release a typeshed==1.0.0 with just a README.md saying "don't install this"?

[emmatyping]

@TheKevJames that isn't a problem. In my implementation in #5517 typeshed is never imported, so installing the typeshed package will not interfere (the path to the typshed files are found relative to the mypy package).