filter/skipif combination of cases from multiple parametrize_with_cases decorators

[eavanvalkenburg]

Hi, loving the library! I have a instance where I create a large number of test cases based on a combination of multiple parametrize_with_cases, a simple example here:

in test_package_cases.py:

def encrypted_encrypted():
    """Return a key for a encrypted test."""
    return KEY


def encrypted_unencrypted():
    """Return None for a unencrypted test."""
    return None


def proto_tcp():
    """Return TCP protocol."""
    return Protocol.TCP


def proto_udp():
    """Return UDP protocol."""
    return Protocol.UDP

and in test_package.py

    @parametrize_with_cases("encrypted", prefix="encrypted_")
    @parametrize_with_cases("protocol", prefix="proto_")
    def test_encrypted_over_protocols(
        self,
        encrypted,
        protocol)
       
       <test code>

This is a simplified example of what I want to achieve (I have one test that has 5 parametrize_with_cases, with a total of 80 tests coming out of it, which I love!), but let's say I now want to exclude the test for TCP Encrypted, I would have expected either a Filter or using something like pytest.mark.skipif(protocol==TCP and encrypted is not None) to do this, but that doesn't seem to work, and in the filter doc, I seem to read that a filter only applies to a single parametrize call, so you couldn't use both variables there.

The workaround right now is to skip in the code, but I don't really like that approach!

[smarie]

Thanks @eavanvalkenburg for the kind words ! Glad you're finding this lib useful.

I understand your problem, but I do not see a proper way to solve it. Maybe @saroad2 will have an API idea here as he is fond of filtering ;)

For a quick workaround I would go for the "if" statement combined with get_current_case_id.

Otherwise maybe a more general get_current_cases(request) returning a dictionary of argname(s): case_func could do the trick. However I'm afraid that this is almost impossible to do with the current implementation, since currently some case functions are transformed into lazy_values and others into fixtures, and the original case function is lost in the process... Maybe something to add to the pile of requirements in #170

Other ideas welcome ! @plammens @shaunc if you have brain bandwidth available :)

[eavanvalkenburg]

yeah, the get_current_case* isn't much easier then just using a if condition block against the actual parameters that were passed in because of the cases, so this will work for now, but would be cool if you guys think of something here, because that would make it much more explicit, to see a skipped in the log instead of a passed for those instances!

[shaunc]

I had this problem too, and I think I was mostly able to overcome it using tags. You can pass a list of tags to both case decorator and parametrize_with_cases. Tags seem mostly for inclusion rather than exclusion, (@smarie - how about boolean tag expression support?) but since you can tag with a list, with a little work you can often come up with a way to exclude. (e.g., "not_encrypted" tag, together with some "neutral" tag in other cases -- e.g. "all_protos" ...?)

In fact maybe the "neutral" tags can be avoided if you use @parametrize_with_cases directly?

Another technique I use is to select subsets by creating a fixture that combines all the cases, then use unpack_fixtures to create individual fixtures again, except limited to the subsets. For instance:

@fixture
@parametrize_with_cases("encrypted", prefix="encrypted_", has_tag="not_encrypted")
@parametrize_with_cases("protocol", prefix="proto_")
def ne_cases(encrypted, protocol):
    return (encrypted, protocol)

ne_encrypted, ne_protocol = unpack_fixtures("ne_encrypted, ne_protocol", ne_cases)

[smarie]

that would make it much more explicit, to see a skipped in the log instead of a passed for those instances!

If this is the point that bothers you, you can write pytest.skip(msg) in your if statement !

@shaunc thanks for adding your thoughts here ! The filtering features, with the new helper functions from @saroad2 (filters module, described in the very last paragraph of this section) indeed does the trick for a each @parametrize_with_cases independently. However @eavanvalkenburg 's issue is that he would like to make a filter based on a condition depending on two cases (crossing the two @parametrize_with_cases decorators) - this can not be adressed for now, and I'm not sure that using fixtures can be of any help here ?

[eavanvalkenburg]

Indeed crossing the decorator boundary is the issue! and thanks for the tip about the pytest.skip, didn't know that one, will use that for now, you can close this if you want (or keep it open in case you want to refer to it later as a feature)!

[shaunc]

@smarie -- two cases -- ah, sorry.

One could make new cases individual items of cross product in a pair, leaving some out, then gather together and unpack again to get fixtures running over the desired subset. A lot of writing to do manually, but perhaps a helper could automate.

[smarie]

@eavanvalkenburg I found a way for you to solve this issue: with the new fixture current_cases you can access the exact case function currently in use for each parameter. You can therefore do

@parametrize_with_cases("a", ...)
@parametrize_with_cases("b", ...)
def test_foo(a, b, current_cases):
    if current_cases["a"] is case_a and current_cases["b"] is case_b:
        pytest.skip("This combination of cases does not make sense")

Note that cases used to parametrize a fixture appear in this dictionary too, but one level deeper (under the fixture name).

Let me know if this solves your problem, and I'll close the issue then.

[smarie]

@eavanvalkenburg I'll close this now, but feel free to reopen if this did not solve your issue !