maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tomasz Halama (JIRA)" <j...@codehaus.org>
Subject [jira] (SUREFIRE-1043) Please add possibility to specify tests groups order for sequential run and tests groups content for parallel run.
Date Tue, 03 Dec 2013 11:53:45 GMT

     [ https://jira.codehaus.org/browse/SUREFIRE-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tomasz Halama updated SUREFIRE-1043:
------------------------------------

    Description: 
What we need is as following:

For sequential run:
We would like to have some means to specify run order of whole groups of tests, during one
invocation of plugin. Content of the group should be determined by ant style expressions.
Inside each such group, tests should be run in order, specified by 'runOrder' parameter.

Example:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <configuration>
    <groupsOrder>
      <group>**/*ST</group>
      <group>com/something/SomeTest,com/something/SomeTest2</group>
      <group>**/*IT,**/IT*</group>
      <group>com/something/SomeTest3</group>
    </groupsOrder>
    <runOrder>alphabetical</runOrder>
  </configuration>
</plugin>

In above example run order of tests should be as following:
1. all tests, which have "ST" suffix, should be run in alphabetical order
2. com/something/SomeTest and com/something/SomeTest2 should be run in alphabetical order
3. all tests, which have "IT" suffix or prefix, should be run in alphabetical order
4. com/something/SomeTest3 should be run

Instead of adding some new parameter (like above 'groupsOrder' section), I think 'includes'
can be used for this purpose (each entry would determine a group), with some additional boolean
value, i.e. 'groupsByIncludes' (indicating, that each 'include' entry should be considered
as a separate group):

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <configuration>
    <includes>
      <include>**/*ST.java</include>
      <include>com/something/SomeTest.java,com/something/SomeTest2.java</include>
      <include>**/*IT.java,**/IT*.java</include>
      <include>com/something/SomeTest3.java</include>
    </includes>
    <runOrder>alphabetical</runOrder>
    <groupsByIncludes>true<groupsByIncludes>
  </configuration>
</plugin>

In our case we cannot simply divide one invocation of plugin into several ones (which would
solve this problem), because setuping tests context takes too much time.

This feature will also give possibility to run tests by some specific order (when each 'group'/'include'
would consist of only one entry). Possibility to set specific order could be very handy, when
i.e. you detected, that some of your tests are not independent and you want to reproduce the
problem.



For parallel run:
We would like to have some means to specify tests groups for parallel run. By this I mean:
tests in each group will be run in parallel, but the groups itself will be run sequentially
(so only tests from the same group can be executed at the same time)
There should be also some annotation, which can be used to marked tests, which cannot be run
in parallel at all.
If 'includes' contains such tests - they should be run sequentially at the end.

To achieve all of this, new (i.e. groupsOrder) or 'includes' parameter can be used here, in
the same way as for sequential run, except 'parallel' attribute should be set to new value:
i.e. parallelGroup.

I think, that logging information, when each test (each test method) starts and stops, would
be also a good idea (we would know exact tests, which are run at the same time).

  was:
What we need is as following:

For sequential run:
We would like to have some means to specify run order of whole groups of tests, during one
invocation of plugin. Content of the group should be determined by ant style expressions.
Inside each such group, tests should be run in order, specified by 'runOrder' parameter.

Example:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <configuration>
    <groupsOrder>
      <group>**/*ST</group>
      <group>com/something/SomeTest,com/something/SomeTest2</group>
      <group>**/*IT,**/IT*</group>
      <group>com/something/SomeTest3</group>
    </groupsOrder>
    <runOrder>alphabetical</runOrder>
  </configuration>
</plugin>

In above example run order of tests should be as following:
1. all tests, which have "ST" suffix, should be run in alphabetical order
2. com/something/SomeTest and com/something/SomeTest2 should be run in alphabetical order
3. all tests, which have "IT" suffix or prefix, should be run in alphabetical order
4. com/something/SomeTest3 should be run

Instead of adding some new parameter (like above 'groupsOrder' section), I think 'includes'
can be used for this purpose (each entry would determine a group), with some additional boolean
value, i.e. 'groupsByIncludes' (indicating, that each 'include' entry should be considered
as a separate group):

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <configuration>
    <includes>
      <include>**/*ST.java</include>
      <include>com/something/SomeTest.java,com/something/SomeTest2.java</include>
      <include>**/*IT.java,**/IT*.java</include>
      <include>com/something/SomeTest3.java</include>
    </includes>
    <runOrder>alphabetical</runOrder>
    <groupsByIncludes>true<groupsByIncludes>
  </configuration>
</plugin>

In our case we cannot simply divide one invocation of plugin into several ones (which would
solve this problem), because setuping tests context takes too much time.

This feature will also give possibility to run tests by some specific order (when each 'group'/'include'
would consist of only one entry). Possibility to set specific order could be very handy, when
i.e. you detected, that some of your tests are not independent and you want to reproduce the
problem.



For parallel run:
We would like to have some means to specify tests groups for parallel run. By this I mean:
tests in each group will be run in parallel, but the groups itself will be run sequentially
(so only tests from the same group can be executed at the same time)
There should be also some annotation, which can be used to marked tests, which cannot be run
in parallel at all.
If 'includes' contains such tests - they should be run sequentially at the end.

To achieve all of this, new (i.e. groupsOrder) or 'includes' parameter can be used here, in
the same way as for sequential run, except 'parallel' attribute should be set to new value:
i.e. parallelGroup.

I think, that logging information, when each test starts and stops, would be also a good idea
(we would know exact tests, which are run at the same time).

    
> Please add possibility to specify tests groups order for sequential run and tests groups
content for parallel run.
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1043
>                 URL: https://jira.codehaus.org/browse/SUREFIRE-1043
>             Project: Maven Surefire
>          Issue Type: Improvement
>    Affects Versions: 2.16
>            Reporter: Tomasz Halama
>
> What we need is as following:
> For sequential run:
> We would like to have some means to specify run order of whole groups of tests, during
one invocation of plugin. Content of the group should be determined by ant style expressions.
Inside each such group, tests should be run in order, specified by 'runOrder' parameter.
> Example:
> <plugin>
>   <groupId>org.apache.maven.plugins</groupId>
>   <artifactId>maven-failsafe-plugin</artifactId>
>   <configuration>
>     <groupsOrder>
>       <group>**/*ST</group>
>       <group>com/something/SomeTest,com/something/SomeTest2</group>
>       <group>**/*IT,**/IT*</group>
>       <group>com/something/SomeTest3</group>
>     </groupsOrder>
>     <runOrder>alphabetical</runOrder>
>   </configuration>
> </plugin>
> In above example run order of tests should be as following:
> 1. all tests, which have "ST" suffix, should be run in alphabetical order
> 2. com/something/SomeTest and com/something/SomeTest2 should be run in alphabetical order
> 3. all tests, which have "IT" suffix or prefix, should be run in alphabetical order
> 4. com/something/SomeTest3 should be run
> Instead of adding some new parameter (like above 'groupsOrder' section), I think 'includes'
can be used for this purpose (each entry would determine a group), with some additional boolean
value, i.e. 'groupsByIncludes' (indicating, that each 'include' entry should be considered
as a separate group):
> <plugin>
>   <groupId>org.apache.maven.plugins</groupId>
>   <artifactId>maven-failsafe-plugin</artifactId>
>   <configuration>
>     <includes>
>       <include>**/*ST.java</include>
>       <include>com/something/SomeTest.java,com/something/SomeTest2.java</include>
>       <include>**/*IT.java,**/IT*.java</include>
>       <include>com/something/SomeTest3.java</include>
>     </includes>
>     <runOrder>alphabetical</runOrder>
>     <groupsByIncludes>true<groupsByIncludes>
>   </configuration>
> </plugin>
> In our case we cannot simply divide one invocation of plugin into several ones (which
would solve this problem), because setuping tests context takes too much time.
> This feature will also give possibility to run tests by some specific order (when each
'group'/'include' would consist of only one entry). Possibility to set specific order could
be very handy, when i.e. you detected, that some of your tests are not independent and you
want to reproduce the problem.
> For parallel run:
> We would like to have some means to specify tests groups for parallel run. By this I
mean: tests in each group will be run in parallel, but the groups itself will be run sequentially
(so only tests from the same group can be executed at the same time)
> There should be also some annotation, which can be used to marked tests, which cannot
be run in parallel at all.
> If 'includes' contains such tests - they should be run sequentially at the end.
> To achieve all of this, new (i.e. groupsOrder) or 'includes' parameter can be used here,
in the same way as for sequential run, except 'parallel' attribute should be set to new value:
i.e. parallelGroup.
> I think, that logging information, when each test (each test method) starts and stops,
would be also a good idea (we would know exact tests, which are run at the same time).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message