Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test run wrong order when use DependsOnGroups and Parallel by classes #2987

Open
2 of 7 tasks
hieuhth opened this issue Sep 25, 2023 · 6 comments
Open
2 of 7 tasks

Test run wrong order when use DependsOnGroups and Parallel by classes #2987

hieuhth opened this issue Sep 25, 2023 · 6 comments

Comments

@hieuhth
Copy link

hieuhth commented Sep 25, 2023

TestNG Version

7.8.x

Expected behavior

There are 3 classes are belong to group "First", and 3 classes are belong to group "Second". Group "Second" depends on group "First".

In test suite, Config to run parallel by classes, thread-count=6.
3 classes of group "First" will run parallel. After they run completely, 3 classes of group "Second" should be run parallel.

Actual behavior

3 classes of group "First" will run parallel. After they run completely, 3 classes of group "Second" run one by one, not parallel

Is the issue reproducible on runner?

There are 3 class of group "First"

public class FirstTest1 {
    @Test(groups = "First")
    public void FirstTest1() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run FirstTest1 " + new Date());
    }
}

public class FirstTest2 {
    @Test(groups = "First")
    public void FirstTest2() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run FirstTest2 " + new Date());
    }
}

public class FirstTest3 {
    @Test(groups = "First")
    public void FirstTest3() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run FirstTest3 " + new Date());
    }
}

And 3 classes of group "Second"

public class SecondTest1 {
    @Test(groups = "Second", dependsOnGroups = "First")
    public void SecondTest1() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run SecondTest1 " + new Date());
    }
}

public class SecondTest2 {
    @Test(groups = "Second", dependsOnGroups = "First")
    public void SecondTest2() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run SecondTest2 " + new Date());
    }
}

public class SecondTest3 {
    @Test(groups = "Second", dependsOnGroups = "First")
    public void SecondTest3() {
        try {
            Thread.sleep(5000);
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
        System.out.println("Run SecondTest3 " + new Date());
    }
}

Test suite:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Test Suite" parallel="classes" thread-count="10">
    <test name="Tests">
        <parameter name="browserValue" value="chrome"/>
        <classes>
            <class name="com.simplicia.test.FirstTest1"></class>
            <class name="com.simplicia.test.FirstTest2"></class>
            <class name="com.simplicia.test.FirstTest3"></class>
            <class name="com.simplicia.test.SecondTest1"></class>
            <class name="com.simplicia.test.SecondTest2"></class>
            <class name="com.simplicia.test.SecondTest3"></class>
        </classes>
    </test>
</suite>

Run test suite,
image

  • Shell
  • Maven
  • Gradle
  • Ant
  • Eclipse
  • IntelliJ
  • NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

@juherr
Copy link
Member

juherr commented Sep 25, 2023

Related to #2372

@krmahadevan
Copy link
Member

@hieuhth - What version of TestNG are you working with ?

I tried using 7.8.0 and I am not able to reproduce the issue. Please try using the latest version and post back your findings

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
...
... TestNG 7.8.0 by Cédric Beust ([email protected])
...

Run FirstTest3 Mon Sep 25 16:58:38 IST 2023
Run FirstTest2 Mon Sep 25 16:58:38 IST 2023
Run FirstTest1 Mon Sep 25 16:58:38 IST 2023
Run SecondTest2 Mon Sep 25 16:58:43 IST 2023
Run SecondTest1 Mon Sep 25 16:58:43 IST 2023
Run SecondTest3 Mon Sep 25 16:58:43 IST 2023

===============================================
Test Suite
Total tests run: 6, Passes: 6, Failures: 0, Skips: 0
===============================================
image

@hieuhth
Copy link
Author

hieuhth commented Sep 27, 2023

@krmahadevan I use testng 7.8.0.

I have just rechecked, I set JAVA_TOOL_OPTIONS -Dtestng.thread.affinity=true (to resolve a other issue in my project).
image

If I remove this config, issue will not occur.
How to use this config without issue which I posted?

@krmahadevan
Copy link
Member

Great. Then i guess we can close this issue @hieuhth ?

@hieuhth
Copy link
Author

hieuhth commented Sep 27, 2023

@krmahadevan I don't think so.

testng.thread.affinity : resort to running dependent methods on the same thread as the upstream method

-> I think we should only resort methods of same class when run parallel by classes, and depend is used.

@krmahadevan
Copy link
Member

@hieuhth

testng.thread.affinity : resort to running dependent methods on the same thread as the upstream method

A couple of things about this feature

  • This is an experimental feature.
  • This works ONLY when you have EXACTLY 1 upstream method.

-> I think we should only resort methods of same class when run parallel by classes, and depend is used.

I don't think that's possible. It can be done in a limited fashion when you use the above mentioned JVM argument, but this cannot be used as a generalised strategy. I say this because upstream can always be more than 1 method and so its not going to be possible to figure out on which thread to run a dependent method, when it has more than 1 upstream dependencies.

I would suggest that you relook at your tests. Having a thread affinity in tests kind of sounds like a code smell. I am guessing you are resorting to this because you are having some sort of a dependency on a thread local variable. I would suggest that you re-look at your approach and either resort to building customisations that would ensure that the same thread local variable gets copied over to the dependent thread (or) you remove that dependency completely from your test.

The plain vanilla execution of dependsOngroups and parallel running works fine without any issues. The experimental thread affinity feature was never guaranteed to work for all use cases. It was meant to work ONLY for a specific use case, wherein you have a method that depends on exactly 1 upstream. Also the side effects of this behaviour when coupled with other hard ordering features (such as dependsOnGroups) is NOT documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants