Skip to content

Commit

Permalink
E2E: Disable a flaky deployment test (#2016)
Browse files Browse the repository at this point in the history
Disables an E2E test that notoriously fails:

> When a new website version is deployed while the old version is still
loaded, creating a new site should still work.

I would love to improve this test, but this particular one is tedious
and the process could take a few days I am not willing to put in right
now. At the same time, keeping a failing test around lowers our
confidence in the E2E test overall. Skipping it and keeping the code
around for the next time we adjust the service worker seems like a good
compromise, especially since the deployments seem to be in a good place
now without many reports of the stale Playground version sticking around
for too long.
  • Loading branch information
adamziel authored Nov 22, 2024
1 parent 9f46e51 commit 782ffd3
Showing 1 changed file with 7 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,13 @@ for (const cachingEnabled of [true, false]) {
});
}

test(
/**
* This test is flaky and often fails on CI even after multiple retries. It
* lowers the confidence in the test suite so it's being skipped. It is still
* useful for manual testing when updating the service worker and may get
* improved the next time we change something in the service worker.
*/
test.skip(
'When a new website version is deployed while the old version is still loaded, ' +
'creating a new site should still work.',
async ({ website, page, wordpress }) => {
Expand Down

0 comments on commit 782ffd3

Please sign in to comment.