Is your feature request related to a problem? Please describe.
tests/runtests.py contains 1,279 lines across preconditions, dashboard configuration checks, local utility checks, OpenVPN coverage, and 18 functional Selenium tests. Its single TestServices class owns application setup, persistent browser sessions, test fixtures, and unrelated feature checks, which makes it difficult to identify and fix state leaks.
An analysis of the latest 100 CI Build (Docker-OpenWISP) workflow runs from 2026-07-23 through 2026-08-31 found that 89 runs reached the Test step and executed 115 complete Python test-suite attempts. Twenty-eight attempts failed. The CI retry wrapper retried 26 of them, and 10 workflow runs eventually reported success despite one or more failed suite attempts. The retry loop currently reuses the same Compose stack, database, Redis state, volumes, and browser-related test state, so a green workflow does not establish that its first test attempt was clean.
The recurring failures were:
| Test |
Failed results |
Affected CI runs |
Failure signature |
test_add_superuser |
12 |
8 |
StaleElementReferenceException |
test_forgot_password |
10 |
7 |
Password-reset confirmation was absent |
test_console_errors |
6 |
4 |
Five stale-element errors and one failed static asset |
test_redis_buckets_are_separated |
5 |
1 |
Deterministic Redis URL parsing assertion |
test_topology_graph |
3 |
1 |
Deterministic topology rendering assertion |
The last two rows were deterministic regressions that exhausted all five retries. The Redis parsing issue has since been corrected. The former topology implementation has been replaced by the deterministic database-backed fixture introduced in #682. The first three rows are recurring Selenium synchronization failures that retries conceal.
Describe the solution you'd like
The primary goal is to stabilize the recurring Selenium failures so CI retries no longer conceal them. Reorganize the suite as supporting cleanup: it makes the browser, local utility, and container checks easier to navigate while preserving the existing test lifecycle and targeted-test entry point.
Split the existing test classes into a few direct files while keeping their code and lifecycle intact. Keep tests/runtests.py as a small compatibility runner that imports and re-exports the classes, so make runtests, make develop-pythontests, and python3 tests/runtests.py TestServices.test_name keep working.
tests/
runtests.py # Existing entry point and imports
test_core.py # Preconditions and dashboard configuration checks
test_selenium.py # TestServices unchanged
test_local_utils.py # TestLocalUtils unchanged
test_openvpn.py # TestOpenVPN unchanged
utils.py # Existing shared helpers
The reorganization alone does not isolate state leaks: TestServices will retain its existing class-level setup, persistent browser sessions, and test order. Do not introduce new test framework layers, mixins, fixture abstractions, or a new discovery command. Move each existing class as a whole and retain its current imports, setup, cleanup, and test names.
Address the recurring failures in focused follow-up PRs:
- Make user creation and resource selection wait for a fresh clickable element, avoiding the recurring stale-element failures.
- Make
test_forgot_password wait for its explicit success state instead of inspecting the page source after generic readiness.
- Keep the CI retry wrapper until these failures are fixed, but ensure its logs continue to show every failed suite attempt.
Acceptance criteria:
python3 tests/runtests.py TestServices.test_add_superuser continues to resolve and run the named test.
make develop-pythontests continues to execute all 35 tests through tests/runtests.py.
- The reorganization preserves class import order, test names, setup, cleanup, and all
__file__-relative test paths.
- The focused Selenium fixes pass without a retry before the retry wrapper is removed in a later change.
Describe alternatives you've considered
Keeping all tests in runtests.py makes the file difficult to navigate and obscures the distinction between browser, local utility, and container checks.
Splitting the Selenium methods into several new classes, adding framework layers, or changing test discovery would increase scope without addressing the immediate organization problem. Those changes are not part of this proposal.
Removing CI retries immediately would expose the failures but would not fix them.
Additional context
Relevant workflow runs:
This should be implemented in follow-up PRs rather than expanding #682 beyond its restored Selenium coverage.
Is your feature request related to a problem? Please describe.
tests/runtests.pycontains 1,279 lines across preconditions, dashboard configuration checks, local utility checks, OpenVPN coverage, and 18 functional Selenium tests. Its singleTestServicesclass owns application setup, persistent browser sessions, test fixtures, and unrelated feature checks, which makes it difficult to identify and fix state leaks.An analysis of the latest 100
CI Build (Docker-OpenWISP)workflow runs from 2026-07-23 through 2026-08-31 found that 89 runs reached the Test step and executed 115 complete Python test-suite attempts. Twenty-eight attempts failed. The CI retry wrapper retried 26 of them, and 10 workflow runs eventually reported success despite one or more failed suite attempts. The retry loop currently reuses the same Compose stack, database, Redis state, volumes, and browser-related test state, so a green workflow does not establish that its first test attempt was clean.The recurring failures were:
test_add_superuserStaleElementReferenceExceptiontest_forgot_passwordtest_console_errorstest_redis_buckets_are_separatedtest_topology_graphThe last two rows were deterministic regressions that exhausted all five retries. The Redis parsing issue has since been corrected. The former topology implementation has been replaced by the deterministic database-backed fixture introduced in #682. The first three rows are recurring Selenium synchronization failures that retries conceal.
Describe the solution you'd like
The primary goal is to stabilize the recurring Selenium failures so CI retries no longer conceal them. Reorganize the suite as supporting cleanup: it makes the browser, local utility, and container checks easier to navigate while preserving the existing test lifecycle and targeted-test entry point.
Split the existing test classes into a few direct files while keeping their code and lifecycle intact. Keep
tests/runtests.pyas a small compatibility runner that imports and re-exports the classes, somake runtests,make develop-pythontests, andpython3 tests/runtests.py TestServices.test_namekeep working.The reorganization alone does not isolate state leaks:
TestServiceswill retain its existing class-level setup, persistent browser sessions, and test order. Do not introduce new test framework layers, mixins, fixture abstractions, or a new discovery command. Move each existing class as a whole and retain its current imports, setup, cleanup, and test names.Address the recurring failures in focused follow-up PRs:
test_forgot_passwordwait for its explicit success state instead of inspecting the page source after generic readiness.Acceptance criteria:
python3 tests/runtests.py TestServices.test_add_superusercontinues to resolve and run the named test.make develop-pythontestscontinues to execute all 35 tests throughtests/runtests.py.__file__-relative test paths.Describe alternatives you've considered
Keeping all tests in
runtests.pymakes the file difficult to navigate and obscures the distinction between browser, local utility, and container checks.Splitting the Selenium methods into several new classes, adding framework layers, or changing test discovery would increase scope without addressing the immediate organization problem. Those changes are not part of this proposal.
Removing CI retries immediately would expose the failures but would not fix them.
Additional context
Relevant workflow runs:
This should be implemented in follow-up PRs rather than expanding #682 beyond its restored Selenium coverage.