Description
Since driver is now merged, mypy.myunit is a package instead of a module; however all the logic is still in __init__.py and has irritating limitations.
Note that in these plans I am not yet making myunit and the driver aware of each other, just laying out a framework.
- Document which parts of
myunitare actually used from elsewhere. - Remove the unnecessary use of
Any. - Split "collect tests" logic from "run tests" logic, add
-lto only collect and print. - Split "collect results" logic from "print results" logic.
- Allow fetching a single test by name (see below) instead of collecting everything and then filtering even when there are no wildcards.
- Use
fnmatchinstead of hand-written glob functions (likely via regex conversion, so we can add a command-line option to take a regex directly). - Add tests for
myunititself. - Move assertions so you can do
from mypy.myunit.assertions import * - Add decorators in
mypy.myunit.annotations, see https://gist.github.com/o11c/ef8f0886d5967dfebc3d - Also add an decorator for "this is an
assert_foofunction" to replace the current traceback-cleaning logic, which is wrong if something goes wrong in the assert function itself. - Add a check that some
assert_foofunction is actually run for every testcase. - Record results of a test run to file so that diffs can be displayed from previous runs.
- Move mutable globals to
mypy.myunit.ick - Move main function and related logic to
mypy.myunit.main - Allow free functions to be used as tests instead of requiring them to be wrapped in a class.
- Support opting out of the tempdir logic.
- Support fixtures by looking up
(name.strip('_'), annotated_type)in current module and inmypy.myunit.fixtures(use this for tempdir maybe?) - Change data-driven testcases from overriding
casesto overridinginit, likely in a base class that takes care of callingparse_tests. - Support a "keep going" mode of assertion failure.
Anyway, the big change is global test naming.
New test names have more dots, and will look like mypy.test.testlex.LexerSuite.test_empty. It is not always as simple as getattr - modules need to be imported, classes need to be instantiated, data-driven cases need to be parsed, etc.
The first component name will always be a module, and most likely always be just 'mypy' depending on what is done about #724
I introduce two new classes, TestFactory and Test, to avoid refactoring the responsibilities of Suite and TestCase. There will be a bridge of course, or maybe the old classes will just be subclasses of the new ones. Possibly the old classes will go away entirely.
API will look something like:
TestOrTestFactory = Union['Test', 'TestFactory']
class Test:
name = ''
# Basically just TestCase without all the cruft,
# but also stores the annotations.
# It is now forbidden to create your own subclasses, use fixtures instead.
class TestFactory:
name = ''
@abstractmethod
def list_immediate_children(self) -> Iterable[str]: ...
@abstractmethod
def get_immediate_child(self, name: str) -> TestOrTestFactory: ...
@staticmethod
def list_recursive_children(name: str) -> Iterable[str]: ...
@staticmethod
def get_recursive_child(name: str) -> TestOrTestFactory: ...
class ModuleTestFactory(TestFactory):
def __init__(self, module: types.ModuleType) -> None: ...
# implementation uses `getattr`, `__path__`, and `__import__`
# to find both contained modules (but not imported ones from elsewhere!)
# and test objects (functions, classes, or instances) from the current module.
class SuiteTestFactory(TestFactory):
def __init__(self, suite: Suite) -> None: ...
# implementation uses `suite.cases()`, now restricted to only return `TestCase`s.
# (The only remaining suite with suites as children is the temporary `ListSuite`).
Bikeshed: what do we call the module in which Suite, TestCase, TestFactory, and Test reside? Same module or separate ones?