KAFKA-17672 Run quarantined tests separately

[mumrah]

Introduce new quarantinedTest that excludes tests tagged with "quarantined". Also introduce two new build parameters "maxQuarantineTestRetries" and "maxQuarantineTestRetryFailures".

[mumrah]

@chia7712 looks like #17336 fixed the issue

[mumrah]

In the build, we can see the main tests and the quarantined tests being run

https://github.com/apache/kafka/actions/runs/11130826247/job/30931421135?pr=17329#step:7:21

Parsing build/junit-xml/share-coordinator/test/TEST-org.apache.kafka.coordinator.share.ShareCoordinatorRecordHelpersTest.xml
  Parsing build/junit-xml/share-coordinator/test/TEST-org.apache.kafka.coordinator.share.ShareCoordinatorRecordSerdeTest.xml
  Parsing build/junit-xml/share-coordinator/test/TEST-org.apache.kafka.coordinator.share.ShareCoordinatorServiceTest.xml
  Parsing build/junit-xml/share-coordinator/test/TEST-org.apache.kafka.coordinator.share.ShareCoordinatorShardTest.xml
  
  // snip
  
  Parsing build/junit-xml/core/quarantinedTest/TEST-kafka.coordinator.transaction.ProducerIdManagerTest.xml

So the test collection and JUnit reporting are working.

[mumrah]

@chia7712 can you take a look?

[chia7712]

@mumrah thanks for this patch. that is cooooooooooooooool!

[chia7712]

Has it been included in the artifact? The artifact link (https://github.com/apache/kafka/actions/runs/11130826247/artifacts/2002753330) doesn't seem to include it.

[mumrah]

Fixed

[mumrah]

I realized we should not have quarantinedTest task explicitly fail the build like we do with test. This is because we are explicitly telling Gradle not to cache this task

    outputs.upToDateWhen { false }
    outputs.cacheIf { false }

This means we can also stop forcing ignoreFailures to true in the CI build. If we have more than 20 flaky failures in the quarantinedTest task, it will fail the overall Gradle run.

[chia7712]

If we have more than 20 flaky failures in the quarantinedTest task, it will fail the overall Gradle run.

pardon me, I assumed the quarantined tests should NOT impact the CI flow for PR

[chia7712]

How can we use deflake.yml to run flaky tests if they are excluded by the test task? Maybe we should enable deflake.yml to also run quarantinedTest?

[mumrah]

Good point, I will update that job.

[chia7712]

Could you please update KIP based on this new tag? the tag name in KIP is Flaky rather than Quarantined

[mumrah]

Actually, I wonder if @Flaky is better for the annotation.

The "quarantine" as a concept includes flaky tests as well as new integration tests, so it might make sense to keep the flaky annotation.

@Tag("quarantined")
public @interface Flaky 

or

@Tag("flaky")
public @interface Flaky 

WDYT?

[chia7712]

@tag("flaky")
public @interface Flaky

I prefer this one since it’s shorter and has been part of Kafka for a long time.

[mumrah]

I assumed the quarantined tests should NOT impact the CI flow for PR

Since we're defining this new task separately, we have some flexibility. We can have it fail after some high threshold (like 20 that I arbitrarily chose), or we can prevent it from ever failing the build.

In principle, I guess it's best to not fail the build and rely on the quarantined test reports to ensure we don't accumulate too many quarantined tests or have them for too long.