Skip to content

[tests] Stabilize flaky integration tests and reorganize the suite #683

Description

@nemesifier

Is your feature request related to a problem? Please describe.
tests/runtests.py contains 1,279 lines across preconditions, dashboard configuration checks, local utility checks, OpenVPN coverage, and 18 functional Selenium tests. Its single TestServices class owns application setup, persistent browser sessions, test fixtures, and unrelated feature checks, which makes it difficult to identify and fix state leaks.

An analysis of the latest 100 CI Build (Docker-OpenWISP) workflow runs from 2026-07-23 through 2026-08-31 found that 89 runs reached the Test step and executed 115 complete Python test-suite attempts. Twenty-eight attempts failed. The CI retry wrapper retried 26 of them, and 10 workflow runs eventually reported success despite one or more failed suite attempts. The retry loop currently reuses the same Compose stack, database, Redis state, volumes, and browser-related test state, so a green workflow does not establish that its first test attempt was clean.

The recurring failures were:

Test Failed results Affected CI runs Failure signature
test_add_superuser 12 8 StaleElementReferenceException
test_forgot_password 10 7 Password-reset confirmation was absent
test_console_errors 6 4 Five stale-element errors and one failed static asset
test_redis_buckets_are_separated 5 1 Deterministic Redis URL parsing assertion
test_topology_graph 3 1 Deterministic topology rendering assertion

The last two rows were deterministic regressions that exhausted all five retries. The Redis parsing issue has since been corrected. The former topology implementation has been replaced by the deterministic database-backed fixture introduced in #682. The first three rows are recurring Selenium synchronization failures that retries conceal.

Describe the solution you'd like
The primary goal is to stabilize the recurring Selenium failures so CI retries no longer conceal them. Reorganize the suite as supporting cleanup: it makes the browser, local utility, and container checks easier to navigate while preserving the existing test lifecycle and targeted-test entry point.

Split the existing test classes into a few direct files while keeping their code and lifecycle intact. Keep tests/runtests.py as a small compatibility runner that imports and re-exports the classes, so make runtests, make develop-pythontests, and python3 tests/runtests.py TestServices.test_name keep working.

tests/
  runtests.py           # Existing entry point and imports
  test_core.py          # Preconditions and dashboard configuration checks
  test_selenium.py      # TestServices unchanged
  test_local_utils.py   # TestLocalUtils unchanged
  test_openvpn.py       # TestOpenVPN unchanged
  utils.py              # Existing shared helpers

The reorganization alone does not isolate state leaks: TestServices will retain its existing class-level setup, persistent browser sessions, and test order. Do not introduce new test framework layers, mixins, fixture abstractions, or a new discovery command. Move each existing class as a whole and retain its current imports, setup, cleanup, and test names.

Address the recurring failures in focused follow-up PRs:

  1. Make user creation and resource selection wait for a fresh clickable element, avoiding the recurring stale-element failures.
  2. Make test_forgot_password wait for its explicit success state instead of inspecting the page source after generic readiness.
  3. Keep the CI retry wrapper until these failures are fixed, but ensure its logs continue to show every failed suite attempt.

Acceptance criteria:

  1. python3 tests/runtests.py TestServices.test_add_superuser continues to resolve and run the named test.
  2. make develop-pythontests continues to execute all 35 tests through tests/runtests.py.
  3. The reorganization preserves class import order, test names, setup, cleanup, and all __file__-relative test paths.
  4. The focused Selenium fixes pass without a retry before the retry wrapper is removed in a later change.

Describe alternatives you've considered
Keeping all tests in runtests.py makes the file difficult to navigate and obscures the distinction between browser, local utility, and container checks.

Splitting the Selenium methods into several new classes, adding framework layers, or changing test discovery would increase scope without addressing the immediate organization problem. Those changes are not part of this proposal.

Removing CI retries immediately would expose the failures but would not fix them.

Additional context
Relevant workflow runs:

This should be implemented in follow-up PRs rather than expanding #682 beyond its restored Selenium coverage.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions