Is there a way to generate tests from a fixture?

[amitwer]

Hi,
I have a situation I don't know how to tackle.
For simplicity, let's assume I'm testing a server. This server has a dynamic list of "supported client versions". I want to run a test that would look like this:

@pytest.mark.group_a
def test_something(client_version)   
    #do something

and to get as many test runs as there are versions.

Connection to the server is done in a session-scoped fixture, and try as I might, I couldn't use pytest_generate_tests since metafunc does not have access to fixture values (or at least, I couldn't find one, assuming fixtures are calculated after the collection phase).
I need the requests to the server run only if "group_a" is selected, so replicating the logic in the server connection fixture is not helpful.

Is there a way to do that in pytest? generating a test from a fixture or from another test?

[RonnyPfannschmidt]

if you create the client connection at pytest_configure time, you can use it in both generate_tests and the fixture later on

[amitwer]

First of all - thank you for the answer.

Though, I'm not sure I fully understood it.

1. Is there a place where I can see the pytest cycle? I understood that collection happens before fixture value calculation, but I'm unsure about what phases are there (I didn't find something in the documentation, but might have missed it)
2. I'm not certain that I can create the connection before knowing that the specific test should run - in some cases I don't even have a server. But, I might be able to get this to work, which leads me to the question - how do I do that? will it just be accessible through metafunc.config?

[RonnyPfannschmidt]

https://github.com/pytest-dev/pytest/blob/master/src/_pytest/main.py#L320-L324 contains the basic loop

depending on complexity one can either stick stuff onto config, or make a custom plugin to add the extra data

@nicoddemus i took a short look at the docs, and i have not found a example of a parameterize + resource managing plugin, i wonder if we should add a example of a small plugin that initializes a resource with pytest configure/unconfigure and manages parameterize/fixtures on a object

[amitwer]

Thanks.
I'll probably need to write a custom plugin for that. not sure when and whether I'll be able to do that (in the meanwhile we have a test that runs a for loop, which isn't great but does that job - as long as it's passing)

So, before I dive into that, is there anything in the way pytest is structured that would prevent a plugin from expanding tests just before they run, or is it something worth investigating?

[RonnyPfannschmidt]

its breaks a large amount of expectations

the best way to build it is a plugin object like

# never executed pseudo code

class MyFixtureManager:

   def pytest_configure(self):
     self.res = self.makeit()

   @fixture
   def mydata(self, ...):
     ...

   def pytest_generate_tests(self, metafunc):
       if self.affects(metafunc)
           metafunc.parametrize(...)

[amitwer]

I suspected this is the case.
So, just so to make sure I understand - the current architecture assumes that:

1. Once collection phase is done, the items list is unchanging
2. There is no entity such as "looping test" (something that can be expanded once run), so each item in the list maps to exactly one test case to be executed

Did I get that right?

[nicoddemus]

@nicoddemus i took a short look at the docs, and i have not found a example of a parameterize + resource managing plugin, i wonder if we should add a example of a small plugin that initializes a resource with pytest configure/unconfigure and manages parameterize/fixtures on a object

Yeah I think so, good idea. Created #8208 to track this. 👍