Migrate tests to GitHub Actions #3430
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This was (finally) passing at https://github.com/metabrainz/musicbrainz-server/actions/runs/12457761240 but there are probably (definitely) some issues with running them as part of a pull request that I'll need to address. 😛 |
mwiencek
commented
Dec 22, 2024
HACKING-PROD.md
Outdated
| `/home/musicbrainz/musicbrainz-server/`, and then you can run any test you want to check like this: | ||
|
|
||
| $ sudo -E -H -u musicbrainz carton exec -- prove -lv t/tests.t :: --tests Failing::Test | ||
| Finally, you will need to update the `musicbrainz-tests` image version in |
There was a problem hiding this comment.
(Reminder to myself that this documentation needs to be updated again.)
mwiencek
force-pushed
the
github-actions
branch
from
d037368 to
0620dee
Compare
mwiencek
force-pushed
the
github-actions
branch
14 times, most recently
from
96379b6 to
1cfc071
Compare
Recent Selenium builds have been failing with
WebDriverError: disconnected: not connected to DevTools
(failed to check if window was closed: disconnected: not connected to DevTools)
even though we haven't changed anything that I'm aware of. Trying to update our
Chrome dependencies here to see if that helps in any way.
Also bump selenium-webdriver to 4.27.0; I've combined it with this commit
because the changelog suggests it's somewhat tied to specific Chrome versions:
https://github.com/SeleniumHQ/selenium/blob/trunk/javascript/node/selenium-webdriver/CHANGES.md
This started failing during CI runs recently, and I have no clue why. (If you run the tests manually in your browser, it will also fail if you click on the page before it executes.) I'm just removing the test because the old autocomplete code is on the way out anyway.
We'd like to move Selenium tests away from Jenkins (which we moved to in machine with heavy MBS traffic, which can cause test slowdowns and flakiness. Maintaining Jenkins also isn't free and requires us to store some CI configuration outside of git. GitHub Actions looks like a suitable alternative for us, as its default runners provide more vCPUs and RAM (4 + 16GB) than CircleCI's (2 + 4GB), while also providing unlimited build minutes. Since we already use GitHub, that's one less service for us to rely on, and GHA integrates better with GitHub. The main disadvantage I could find is that there's no built-in way to SSH into a running tests container on GHA.
See previous commit, "Migrate from CircleCI to GitHub Actions."
It's now symlinked from the GitHub Actions workspace.
The metabrainz/musicbrainz-tests image will be built and cached prior to the two test jobs running. It no longer need to be built and pushed by hand in advance.
This silences progress bar log spam in the CI runs.
Copy the git checkout, create the test database, and compile static resources within the Dockerfile rather than as part of the workflow job steps. This will speed up each job by reducing the number of redundant steps.
This will be used in a subsequent commit to merge the coverage data with that from our Selenium tests
Up until this point, a downside of our JavaScript coverage reports was that they only included the Selenium tests. It'd be very helpful if we could also include coverage from our t/web.js test runs. This merges them together in a separate GitHub Actions job, using artifacts to pass the istanbul coverage data between jobs.
So it can be imported in files executed via bin/sucrase-node.
These take a long time to complete, and that time will only increase as we add more and more browser tests. Add a way to run partitions/subsets of the tests in separate jobs. We use greedy number partitioning [1] to partition the tests using their number of commands as their size. [1] https://en.wikipedia.org/wiki/Greedy_number_partitioning
Since there appears to be an actual *documented* way to cleanup images produced by our tests via the GHCR API, use that instead.
Many Perl tests make requests to pages rendered by the Node template-renderer. Prior to this commit, we weren't dumping istanbul coverage data from those tests. (The coverage is only dumped when the template-renderer process is cleanly shut down.)
There's no need to leave the Flow server running once we're done with it.
This comment wasn't entirely correct, as generating cpanfile.snapshot had nothing to do with Chrome updating. That was a side effect of rebuilding Dockerfile.tests, not cpanfile.snapshot. And it's no longer an issue since da2d499 anyway.
Pull requests don't have permission to push to ghcr.io, so we have to cache the image ourselves between jobs.
mwiencek
force-pushed
the
github-actions
branch
2 times, most recently
from
d282d6e to
8e37f1a
Compare
This may or may not help with debugging issues related to elements not being findable on the page. The screenshot is uploaded to the build artifacts.
The previous nyc_download step does not fail if no artifacts are found.
It seems that Chrome's `--headless=new` is now just `--headless`, with the old `--headless` becoming `--headless=old` [1]. But `--headless=new` still works, too. However, Firefox doesn't start in headless mode if you use `--headless=new`. Plain old `--headless` works for both browsers. [1] https://developer.chrome.com/docs/chromium/headless
We may want to push tests back onto `testsToRun`, which reduce doesn't allow as the array length is determined in advance. This also avoids a `require-atomic-updates` warning, which was a false-positive, but using a for-loop is simpler anyway.
These can take up quite a bit of space in the (limited) GitHub artifacts storage, and are only really useful if a particular test fails.
Chrome/chromedriver frequently and randomly fails on GitHub Actions, with messages including the joyful "tab crashed," and the highly transparent "unknown error." Let's see if the free and open source browser from Mozilla fares any better.
This seems to bypass some "element not clickable" issues in Firefox.
It made sense to use false prior to ce43326 when we had the old `handleAlert` commands verifying that these were shown, but now we must have the Firefox behavior align with Chrome.
mwiencek
force-pushed
the
github-actions
branch
from
8e37f1a to
cdb05db
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Solution
Other changes:
One downside I'll note is that we can no longer SSH into running jobs (like on CircleCI) by default. There might be a way to emulate this feature if we need it.
Testing
Just the new GitHub Actions workflow!