maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Subach (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SUREFIRE-1470) Unexpected report result when using rerunFailingTestsCount with groups filter in Surefire
Date Fri, 02 Feb 2018 19:16:00 GMT
Alexey Subach created SUREFIRE-1470:
---------------------------------------

             Summary: Unexpected report result when using rerunFailingTestsCount with groups
filter in Surefire
                 Key: SUREFIRE-1470
                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1470
             Project: Maven Surefire
          Issue Type: Bug
          Components: Maven Surefire Plugin
            Reporter: Alexey Subach


When the following plugin configuration is used:
{code:java}
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${surefire.version}</version>
    <configuration>
        <includes>
            <include>**/*.java</include>
        </includes>
        <groups>TestGroup</groups>
        <rerunFailingTestsCount>1</rerunFailingTestsCount>
    </configuration>
</plugin>{code}


And there is a test that fails first time it's run, the resultant *txt* report is unexpected
and only contains the result of last rerun.

Suppose we have the following test class:

{code:java}
@Category(TestGroup.class)
public class CustomTest {

    @Test
    public void alwaysPassingTest() {
        Assert.assertEquals(0, 0);
    }

    private static volatile int x = 0;

    @Test
    public synchronized void secondAttemptPassingTest() throws InterruptedException {
        Thread.sleep(5000);
        if (x++ == 0) {
            Assert.fail();
        }
    }

}
{code}

The resultant xml report is all right, but the txt one is

{code}
-------------------------------------------------------------------------------
Test set: CustomTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in CustomTest
{code}

Which is basically incorrect. The *txt* file is regenerated several times during the test
run.

The issue is present when groups filter (either exclusion or inclusion) is present. When there
is no group filter and only rerunFailingTestsCount is configured, then the resultant report
is the following one:

{code}
-------------------------------------------------------------------------------
Test set: CustomTest
-------------------------------------------------------------------------------
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.065 s <<< FAILURE!
- in CustomTest
secondAttemptPassingTest(CustomTest)  Time elapsed: 0.009 s  <<< FAILURE!
java.lang.AssertionError
	at CustomTest.secondAttemptPassingTest(CustomTest.java:18)
{code}

When no group filter is configured, Junit4Provider instead of JunitCore provider is picked
up and they seem to handle those cases differently.

The situation is worsened by the fact that TeamCity is not able to parse the results correctly
when the groups are configured along with rerunFailingTestsCount and it marks the tests that
passed after rerun as failed, although the overall Maven result is SUCCESS in both cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message