keep running tests even if earlier ones failed

[geekosaur]

Matt Pickering says we should always run all tests. This uses continue-on-error: true to run them as long as the build succeeded.

Template B: This PR does not modify behaviour or interface

E.g. the PR only touches documentation or tests, does refactorings, etc.

Include the following checklist in your PR:

• Patches conform to the coding conventions.
• Is this a PR that fixes CI? If so, it will need to be backported to older cabal release branches (ask maintainers for directions).

[ffaf1]

Aren't we going to saturate CI resources pretty fast this way?

[geekosaur]

Yes, as I observed with #10291. But see #10263; @mpickering is insisting it's necessary.

[Kleidukos]

Let's go then.

[fgaz]

https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstepscontinue-on-error

Prevents a job from failing when a step fails. Set to true to allow a job to pass when this step fails.

Won't all jobs always pass then?

[geekosaur]

I need to verify, but I believe they still get logged as "failure" and the post job catches them.

[mpickering]

I was suggesting that it would be a improvement to developer experience -- assuming there are a large amount of resources available.

If there are resource bottle-necks I would prefer to run a slimmed down pipeline at first (one ghc version on windows, linux, darwin) before engaging with the full pipeline.

[geekosaur]

We do have larger resources than most free users get. I don't think there's actually a problem here because it's not like it will take more resources than a successful run, unlike changes that require additional rebuilds solely because they can no longer use cached builds.

[geekosaur]

Okay, it looks like jobs succeed when they're not supposed to indeed. This will be a problem.

[geekosaur]

Will changing the workflow to loop within a single step (not job) rather than using separate steps for each test cause problems for anyone? If so, please speak up now.

[ulysses4ever]

It's not a hard preference but I like the UI of separate steps more, that's why I suggest to use the other solution in that SO thread, based on the if field of a step.

[geekosaur]

Also, I'm not sure this is accomplishing what it's supposed to, because of the way validate.sh runs tests: I think that is aborting on the deliberate lib-suite error and not running the cli-suite tests. That's not going to be fixed by hacking the workflow, and fixing it probably depends on #10320.

[geekosaur]

I (still) don't see a link to said SO, and I'm not sure an if: can solve either problem. You still need to arrange for the whole thing to fail at the end if any step did. Additionally, we already have if:s on the solver tests and would need to integrate those into it without causing them to look like failures if they aren't supposed to run.

[ulysses4ever]

The SO thread I linked in Matrix a couple weeks ago: https://stackoverflow.com/a/58859404/465100 I think it shouldn't suffer from the same false negatives issue (succeeding workflow having some steps failed) but I wouldn't bet on it: this needs experimenting.

For looping over steps of validate: I don't see a problem if that's a bash-loop and every call to validate.sh with a specific step is independent that way.

[geekosaur]

That does the wrong thing: it would, for example, merge with failed tests. if: success() || failure() must be used with great care, and if: always() with even greater care because, as several of the answers there state, you can leave yourself with a job that can't be canceled.

The SO question was to have a step run even if the job was failing. This is incorrect here, because we shouldn't run the tests if the build failed. Moreover, we don't want the job to be counted as a success if a test with that condition failed. There are some very complicated conditionals that could accomplish that in an if:, but the shell script is easier to read especially when the extra condition on the solver tests is included.