maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tomasz Halama (JIRA)" <j...@codehaus.org>
Subject [jira] (SUREFIRE-1043) Please add possibility to specify tests groups order for sequential run and tests groups content for parallel run.
Date Tue, 03 Dec 2013 11:37:45 GMT
Tomasz Halama created SUREFIRE-1043:
---------------------------------------

             Summary: Please add possibility to specify tests groups order for sequential
run and tests groups content for parallel run.
                 Key: SUREFIRE-1043
                 URL: https://jira.codehaus.org/browse/SUREFIRE-1043
             Project: Maven Surefire
          Issue Type: Improvement
            Reporter: Tomasz Halama


What we need is as following:

* For sequential run:
We would like to have some means to specify run order of whole groups of tests, during one
invocation of plugin. Content of the group should be determined by ant style expressions.
Inside each such group, tests should be run in order, specified by 'runOrder' parameter.

Example:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <configuration>
    <groupsOrder>
      <group>**/*ST</group>
      <group>com/something/SomeTest,com/something/SomeTest2</group>
      <group>**/*IT,**/IT*</group>
      <group>com/something/SomeTest3</group>
    </groupsOrder>
    <runOrder>alphabetical</runOrder>
  </configuration>
</plugin>

In above example run order of tests should be as following:
1. all tests, which have "ST" suffix, should be run in alphabetical order
2. com/something/SomeTest and com/something/SomeTest2 should be run in alphabetical order
3. all tests, which have "IT" suffix or prefix, should be run in alphabetical order
4. com/something/SomeTest3 should be run

Instead of adding some new parameter (like above 'groupsOrder' section), I think 'includes'
can be used for this purpose (each entry would determine a group), with some additional boolean
value, i.e. 'groupsByIncludes' (indicating, that each 'include' entry should be considered
as a separate group):

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <configuration>
    <includes>
      <include>**/*ST.java</include>
      <include>com/something/SomeTest.java,com/something/SomeTest2.java</include>
      <include>**/*IT.java,**/IT*.java</include>
      <include>com/something/SomeTest3.java</include>
    </includes>
    <runOrder>alphabetical</runOrder>
    <groupsByIncludes>true<groupsByIncludes>
  </configuration>
</plugin>

In our case we cannot simply divide one invocation of plugin into several ones (which would
solve this problem), because setuping tests context takes too much time.

This feature will also give possibility to run tests by some specific order (when each 'group'/'include'
would consist of only one entry). Possibility to set specific order could be very handy, when
i.e. you detected, that some of your tests are not independent and you want to reproduce the
problem.



* For parallel run:
We would like to have some means to specify tests groups for parallel run. By this I mean:
tests in each group will be run in parallel, but the groups itself will be run sequentially
(so only tests from the same group can be executed at the same time)
There should be also some annotation, which can be used to marked tests, which cannot be run
in parallel at all.
If 'includes' contains such tests - they should be run sequentially at the end.

To achieve all of this, new (i.e. groupsOrder) or 'includes' parameter can be used here, in
the same way as for sequential run, except 'parallel' attribute should be set to new value:
i.e. parallelGroup.

I think, that logging information, when each test starts and stops, would be also a good idea
(we would know exact tests, which are run at the same time).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message