Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop end-to-end testing suite for CI #520

Open
aaronleopold opened this issue Dec 8, 2024 · 0 comments
Open

Develop end-to-end testing suite for CI #520

aaronleopold opened this issue Dec 8, 2024 · 0 comments
Labels
chore enhancement but more tedious help wanted Extra attention is needed testing

Comments

@aaronleopold
Copy link
Collaborator

aaronleopold commented Dec 8, 2024

It's becoming increasingly more difficult to properly test Stump in a way that ensures each iteration or new feature doesn't introduce regressions. Besides the (ever improving) unit test suite, full end-to-end tests have been done by hand, which is both time-consuming and error-prone. The latest 0.0.8 release is a prime example of this, as there were bad OPDS-related regressions that went unnoticed for multiple weeks.

This issue is to propose and discuss some ideas for automating the testing of base, core features to ensure that regressions are caught early and often. The workflow I have in mind is as follows:

  1. We already build an amd64 docker image as part of the CI pipeline. We should extend this with an additional step that deploys the image locally and properly tears it down after the tests are done
  2. An end-to-end test suite is created that runs against the local deployment and focuses on essential UI->backend interactions, e.g. using playwright
  3. An end-to-end test suite is created that runs against the local deployment and focuses on essential backend assertions and/or backend-specific features (e.g., OPDS feeds)

The first two steps are the most important, as they will catch the most common regressions. The third step is more of a nice-to-have, since if designed well the first two steps should catch most backend regressions as well.

Edit to add that testing through docker might add a layer of complexity that perhaps is not necessary, and it might be better for velocity to just kickoff a local instance of the server through cargo

@aaronleopold aaronleopold added help wanted Extra attention is needed chore enhancement but more tedious testing labels Dec 8, 2024
@github-project-automation github-project-automation bot moved this to Backlog in v0.1.x Dec 8, 2024
@aaronleopold aaronleopold moved this from Backlog to In Progress in v0.1.x Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chore enhancement but more tedious help wanted Extra attention is needed testing
Projects
Status: In Progress
Development

No branches or pull requests

1 participant