24.3 Add config for marking broken integration tests

[strtgbb]

CI Fix or Improvement

Add tests/integration/broken_tests.json which is used to selectively replace FAILED and ERROR statuses with BROKEN

Supersedes #469

[altinity-robot]

This is an automated comment for commit a77f135 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	⏳ pending
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ failure
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	❌ failure
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	❌ failure

Successful checks

Check name	Description	Status
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker keeper image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docker server image	The check to build and optionally push the mentioned image to docker hub	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Mergeable Check	Checks if all other necessary checks are successful	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success

[Enmk]

Please fix mentioned issues.

Also, were you considering usage of @pytest.mark.xfail?
First of all: it is kind of standard to pytest, second it allows point-xfail and more....

https://docs.pytest.org/en/stable/how-to/skipping.html#xfail
https://docs.pytest.org/en/stable/reference/reference.html#pytest-mark-xfail-ref

[strtgbb]

Also, were you considering usage of @pytest.mark.xfail? First of all: it is kind of standard to pytest, second it allows point-xfail and more....

Yes, pytest marks were considered, however that requires modifying test code and makes it less convenient to see a list of all broken tests.

[strtgbb]

I haven't added a comment field yet because the error message seems clear enough. It will be trivial to add such when we require it.

[vzakaznikov]

I haven't added a comment field yet because the error message seems clear enough. It will be trivial to add such when we require it.

No, let's add support for a comment field now. We don't want to revisit this functionality for such a small thing. I would call that field reason instead of a comment.

[strtgbb]

let's add support for a comment field now.

Done. I don't know how to propagate the reason field upwards at this time, so it will just serve to help someone who is looking at the config file.

[Enmk]

LGTM