[Testing] docker-compose run --rm manage test users runs zero tests (PyTest not used as TestRunner)

[chris48s]

The users app does have some tests defined in it: https://github.com/codebuddies/backend/tree/master/project/users/tests but they're not actually being run in CI:

backend/.github/workflows/test.yml

Lines 24 to 26 in 1607b81

	- name: Test Users
	run: docker-compose run --rm manage test users
	if: always()

$ docker-compose run --rm manage test users
Starting db ... done
System check identified no issues (0 silenced).

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

The main problem here is that there is a mix of pytest tests and django tests. Django's unittest.discover (which is what is being used in CI) isn't picking up the pytest ones.

In general, the project should standardize on one test runner for discovery and that is best done sooner rather than later. I'm happy to help out solving this one way or the other, but is anyone able to give a quick summary of how you ended up here and where you're trying to get to so I know which direction to go in?

[BethanyG]

@chris48s - thank you for the through look over - it is much appreciated.

To be honest, we "ended up here" because we haven't really been focusing on fully detailed tests through PyTest, but rather MVP functionality, and very very basic unittests, using unittest. We're not yet using PyTest. (Personally) I'm just starting to think through non-brittle tests and the use of PyTest, test data, coverage, and other goodies. We've got issues like #98 , #72, and #48 logged, and definitely need to log more. 😄

We've been using Cookiecutters default setup - and the Pytest tests you see in users are the default "example" tests supplied by that framework (as is the custom users module, which we have yet to extend). We're tipping off unittest in CI, because those are the only tests we've actually written.

In the last big tags changes + file re-org, it appears our conftest.py went missing, so even if we were tipping off PyTest, it's throwing a cranky-ass fit over the missing config and test collection fails.

I've been taking a pause since then, so just now noticing.

I'd like to restore conftest.py, and then do some thinking about tests, runners, test data, and testing. But I'd like to do that either in collaboration or knowing parallel, since I want to think through more thoroughly testing my own code as well as how to show/document/make it easy it for others (especially those new to testing and to Django).

Thoughts? The first step is replacing conftest.py and any other files that got lost in the reshuffle. I think the next is having a discussion about how to generate meaningful test data that's not fixture dependent, although that might be several discussions, since we want to have data we can pre-populate in staging for client testing and end-to-end testing as well as data we can generate for functional and unit testing of just the backend.

We'll also want to discuss maybe testing through postman and postman collections as well.

[BethanyG]

Correction. The conftest.py file was moved to the core/ folder...which is causing issues.

Playing with it..but will probably give up in a bit, since its late for me.

[chris48s]

OK. I hadn't clocked that

• the tests that exist in /users fail if you try to run them with pytest
• the tests that exist in /users were auto-generated tests rather than ones you'd written yourselves

Thoughts? The first step is replacing conftest.py and any other files that got lost in the reshuffle.

Agreed. Lets get to a stage where the test suite you already have actually works.

I think the next is having a discussion about how to generate meaningful test data that's not fixture dependent

Personally I'd say the second step is to get all your tests running with one test runner. If you start getting into evaluating test data factories first, you'll want to write more test code or tweak your existing tests to make that decision and that will be more annoying with your test suite split across 2 runners. It also means if that takes a while, at least contributors can lean on the existing test suite relatively easily while you're making those decisions.

Given pytest can run all of your existing tests with relatively few changes, I reckon it probably makes most sense to standardize on that. I've got a local branch where I've broken the back of that. Leave it with me. I'll submit a PR later this week.