Specifying successPercentage and dataProvider (without invocationCount) causes tests to fail even if failure rate is not breached

[Yharooer]

Thank you for looking into #3170 so prompty, however I wanted to check whether this was an issue (or if it was intended).

TestNG Version

Note: only the latest version is supported

7.11.0-SNAPSHOT at f55ca8e

Expected behavior

The following test should pass:

class TestTests {
    @Test(successPercentage = 50, dataProvider = "test")
    fun testTest(number: Int) {
        if (number == 0) {
            println("FAIL")
            assertThat(true).isFalse
        } else {
            println("PASS")
            assertThat(true).isTrue
        }
    }

    @DataProvider(name = "test")
    fun dataProvider(): Array<Array<Int>> = arrayOf(arrayOf(0), arrayOf(1), arrayOf(2), arrayOf(3))
}

Expecting this to pass because the failure rate is 50%, but 75% of tests were successful.

Actual behavior

The test fails, even if the failure rate is not breached.

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

class TestTests {
    @Test(successPercentage = 50, dataProvider = "test")
    fun testTest(number: Int) {
        if (number == 0) {
            println("FAIL")
            assertThat(true).isFalse
        } else {
            println("PASS")
            assertThat(true).isTrue
        }
    }

    @DataProvider(name = "test")
    fun dataProvider(): Array<Array<Int>> = arrayOf(arrayOf(0), arrayOf(1), arrayOf(2), arrayOf(3))
}

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[krmahadevan]

@Yharooer -

Expecting this to pass because the failure rate is 50%, but 75% of tests were successful.

Please be informed that invocationCount and successPercentage are the right combos and successPercentage is NOT meant to be used with dataProvider. This is because only when a test method is invoked multiple times can we figure out success percentages. A data powered test method is not the same as invocation count, because the test data is different.

[krmahadevan]

I also am not able to reproduce the problem with the sample that you shared. Can you please check and tell me if the sample is correct and also share execution logs/screenshots etc of the failure that you are seeing at your end.

[Yharooer]

Yup that makes sense that this is expected behaviour. Just wanted to double check