chore: disable flakey tests in the default runs

[dimaqq]

1. disable flakey tests that James found
2. a separate manual github action to run those

Ref: #1108 (comment)

[dimaqq]

Remaining issue: #1142 (integration tests don't terminate).

I wonder if this PR is still an improvement and should be merged...

[james-garner-canonical]

Remaining issue: #1142 (integration tests don't terminate).

Seems that this is also an inconsistent issue, as the second run of the integration tests on this PR did terminate.

However, it terminated with failures in two further tests! These tests didn't fail once in the 30 runs I did previously.

If it was just one additional test I'd assume you just got lucky and hit a 1/40 chance failure -- after all, it's pretty much even odds whether such an event would show up in a survey of just 30 runs. Getting two independent failures with such low probability is pretty unlikely though, so what's going on?

I wonder if this PR is still an improvement and should be merged...

For now I'm going to run the integration tests again here and also sample another 20 or so runs on main.

[benhoyt]

Seems reasonable to me as an interim solution, while we're working on improving the tests.

[james-garner-canonical]

Seems reasonable to me as an interim solution, while we're working on improving the tests.

I don't think that we should go forward with this as is, since it doesn't actually achieve the goal of making CI pass, due to:

1. exposing an issue with the integration test job failing due to timing out
2. integration tests still failing due to additional test failures not disabled by this PR (yet?)

But if we can get CI to consistently pass by this approach of disabling apparently flakey tests then I am in favour.

[james-garner-canonical]

20 more runs on main identifies 6 additional flakey tests -- table of just new failing tests below, and full table here. I think we should split the integration tests up into different tox environments (by file, I guess), so that they can be run serially in under the time limit for github actions. That way they should at least be (edit: more!) deterministic.

commit=50b42d013aee01536416e6334f99443f2b4f1e4c
n_jobs=20

path	test	# jobs
test_crossmodel.py	test_relate_with_offer	8
test_model.py	test_deploy_bundle_with_storage_constraint	8
test_secrets.py	test_list_secrets	3
test_model.py	test_storage_pools_on_lxd	2
test_controller.py	test_secrets_backend_lifecycle	1
test_model.py	test_add_and_list_storage	1

[james-garner-canonical]

Update: running the tests serially seems to eliminate a lot of flakiness! See PR and test output here.

====== 3 failed, 111 passed, 37 skipped, 1 warning in 4417.29s (1:13:37) =======

[dimaqq]

Going to close in favour of #1143 in a while.
Keeping the branch in case hard skips are warranted.

[dimaqq]

Closed in favour of #1149