Specifying invocationCount, threadPoolSize and successPercentage causes tests to always pass

[Yharooer]

Thank you for looking into #3170 so promptly, however I think there is another related issue.

TestNG Version

Note: only the latest version is supported

7.11.0-SNAPSHOT at f55ca8e

Expected behavior

Expect the following test to fail:

import org.assertj.core.api.Assertions.assertThat
import org.testng.annotations.Test

class TestTest {
    @Test(invocationCount = 10, threadPoolSize = 10, successPercentage = 99)
    fun testTest() {
        assertThat(true).isFalse
    }
}

Actual behavior

The tests pass instead.

Correct behaviour is achieved if successPercentage is 100, or if threadPoolSize is omitted.

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

import org.assertj.core.api.Assertions.assertThat
import org.testng.annotations.Test

class TestTest {
    @Test(invocationCount = 10, threadPoolSize = 10, successPercentage = 99)
    fun testTest() {
        assertThat(true).isFalse
    }
}

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[krmahadevan]

@Yharooer - I dont think this scenario would work because TestNG internally does the following:

When TestNG is given a combination of invocation count "m" + thread pool size of "n" it internally converts them into "m" test workers each having an invocation count of "1".

So for this example, it considers that there are 10 tests (equivalent to invocationCount) all running at the same time (since we have a threadpool size of greater than 1). So all the tests are likely to finish at the same time.

Since the tests are all going to be running concurrently it becomes difficult to consider the successPercentage attribute

@juherr - WDYT?

[juherr]

@krmahadevan If you think it is a bad practice to mix these attributes, add a warning message they are used together. It will help users they are not using it in the right way.

[Yharooer]

I see.
Is it possible to check at the end of the tests that the successPercentage was achieved and override the test results? I think the same can be achieved by adding a custom listener to the test to check whether the failure rate was breached at the end of the tests. However it is inconvenient for users of TestNG to have to add this manually when from the user perspective it is not easy to understand why parallelisation should affect the result.