TestNG 6.11 to 7.5.1, execution sort order is modified

[snagarjunas]

TestNG Version

Note: only the latest version is supported
7.5.1

Expected behavior

We were on TestNG 6.11 + Surefire 2.20 and maven 3.0.5. All tests in the package executed in alphabetical order

Actual behavior

We have now upgraded to TestNG 7.5.1 + Surefire 3.2.5 and maven 3.9.8 versions. Without any other changes in the pom file when we run the same command used previously, the execution order is changing.

We have tried setting runOrder (alphabetical / filesystem), but still execution order is not as before.

Is there any new setting introduced to execute in alphabetical order correctly as before?

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[github-actions]

Hi, @snagarjunas.
We need more information to reproduce the issue.

Please help share a Minimal, Reproducible Example that can be used to recreate the issue.

In addition to sharing a sample, please also add the following details:

• TestNG version being used.
• JDK version being used.
• How was the test run (IDE/Build Tools)?
• How does your build file (pom.xml | build.gradle | build.gradle.kts) look like ?

It would be better if you could share a sample project that can be directly used to reproduce the problem.
Reply to this issue when all information is provided, thank you.

[snagarjunas]

Thank you for your reply. Yeah, I understand that we need reproducible example to solve the problem. But there are many many tests in the package, many data provider tests, many test annotations. At this point even I do not have minimal reproducible code. But will try to get to it. Till then are there any newly added parameters to run the tests in packages in-order? Thanks.

[krmahadevan]

@snagarjunas - I dont think anything new was added around test execution.

You can try to experimenting with an implementation of IMethodInterceptor and see if that helps (7.5.1 was released over a year ago and so its very difficult to recollect what happened in which version. You should also upgrade to 7.10.2 while you are at this. Dont forget to bump up your JDK to JDK11 for 7.10.2)

Here's a sample that shows IMethodInterceptor in action

Sample test case

import org.testng.*;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

import java.util.Comparator;
import java.util.List;
import java.util.Random;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;

@Listeners(SampleTestCase.Statistics.class)
public class SampleTestCase {

    private static final Random RANDOM = new Random();

    @Test
    public void z() {
        sleepQuietly();
    }

    @Test
    public void y() {
        sleepQuietly();
    }

    @Test
    public void x() {
        sleepQuietly();
    }


    @Test(dependsOnMethods = "a")
    public void b() {
        sleepQuietly();
    }

    @Test
    public void a() {
        sleepQuietly();
    }

    private static void sleepQuietly() {
        //Forcing a random sleep so that we dont have methods which have the same start time and it gives us a
        //more realistic scenario.
        int time = RANDOM.nextInt(1000);
        if (time == 0) {
            time = 100;
        }
        try {
            TimeUnit.MILLISECONDS.sleep(time);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

    public static class Statistics implements ITestListener, IMethodInterceptor {
        @Override
        public void onFinish(ITestContext context) {
            //Lets print all the passed test methods sorted based on their start time
            //That will show us the order in which the methods were executed
            context.getPassedTests().getAllResults()
                    .stream().sorted(Comparator.comparing(ITestResult::getStartMillis))
                    .forEach(it -> System.err.println(pretty(it)));
        }

        @Override
        public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
            //Let's sort the methods based on their name
            return methods.stream()
                    .sorted(Comparator.comparing(o -> o.getMethod().getMethodName()))
                    .collect(Collectors.toList());
        }

        private static String pretty(ITestResult itr) {
            return itr.getMethod().getMethodName() + "() started at :" + itr.getStartMillis();
        }
    }
}

Execution output

a() started at :1722398209672
b() started at :1722398210672
x() started at :1722398210978
y() started at :1722398211940
z() started at :1722398212209

===============================================
Default Suite
Total tests run: 5, Passes: 5, Failures: 0, Skips: 0
===============================================

[snagarjunas]

@krmahadevan - In the above example can I add Priority to test y( ) to execute it firstly like below output -

y( )
a( )
b( )
x( )
z( )

[krmahadevan]

@snagarjunas - Please give it a try and then check what happens

[snagarjunas]

@krmahadevan - To expand before scenario, I have two Classes in a Package. And wanted one test to execute it at first and one test to execute at last. Sample below -

TestNG: 7.5.1
Java: 1.8.0_202
Run: Using whole package with 2 Classes ResultTest1 and ResultTest2

Class - ResultTest1

@Test
public void test1() throws InterruptedException {
    System.out.println("In ResultTest1 test1()");
}
@Test(priority = 100)
public void test2() throws InterruptedException {
    System.out.println("In ResultTest1 test2()");
}

Class - ResultTest2

@Test
public void test1() throws InterruptedException {
    System.out.println("In ResultTest2 test1()");
}
@Test(priority = -1)
public void test2() throws InterruptedException {
    System.out.println("In ResultTest2 test2()");
}

Output:
In ResultTest1 test1()
In ResultTest1 test2()
In ResultTest2 test2()
In ResultTest2 test1()

This is the output I get.

But I am looking for ResultTest2 > test2() to be executed first (priority is -1) considering both the classes in the package and ResultTest1 > test2() to be executed at the end of all the tests (priority is 100) in the package.

Expected Output:
In ResultTest2 test2()
In ResultTest1 test1()
In ResultTest2 test1()
In ResultTest1 test2()

Is there a possibility to achieve this without moving priority -1 test to base class or priority 100 to after suite using any configuration parameter to consider all tests before running in package?

Thank you.

[krmahadevan]

Couple of things.

You should upgrade to TestNG 7.10.2
Now for your use case I would suggest that you build a custom annotation (Say @RunFirst and @RunLast) and then read this annotation in your method interceptor and then sort the methods accordingly.