lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dawid Weiss (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4463) add support for running the same test method many times
Date Sat, 06 Oct 2012 07:08:02 GMT


Dawid Weiss commented on LUCENE-4463:

Let me start from the beginning. I talked about it once but I can't find it now. 

1. testmethod vs. "tests.method"

The reason for the complication in "testmethod" vs. "tests.method" is stemming from how JUnit
works. A "test description" must (in practice) be unique, otherwise tools just go crazy. So
to make a test repeat, its name must be made unique. That's why if you do tests.iters=X every
repetition of a single test method will in fact be named uniquely as:

testMethod {#seq seed=[...]}

These are not things just added to the report, this is a "method name" as JUnit Description
object sees it. It's odd but it's a workaround that works and that is (as far as I'm concerned)
the only one possible.

So when you use "-Dtestname=X" this is an alias for "-Dtests.method=X" which will filter out
all these repeated tests (because effectively they don't match the mattern). In order to include
them, you need to add a glob like: "-Dtests.method=X*". Hoss and I added this to the test-help
to make it clear(er) a while back. 

2. Seeds and tests.dup

The "master" test seed is passed to <junit4> task once and it just stays there. Everything
else is derived from it. The "duplication" you see is a simple trick -- we just duplicate
the file name on input. Because every suite gets the same random seed (mixed with its class
name to make it more random across a single run), a duplicated identical suite will still
get the same master seed every time. 

This option was meant for accelerating a test scenario in which we want to repeat a single
suite/seed combination many times and want to do it using multiple parallel JVMs.

3. What Robert wants (across-jvm repetition of a single suite with a different seed each time).

Is effectively impossible right now without re-spinning <junit4> with a different seed
each time. 

I don't see I could marry all this into working with both the scenario above and with Robert
wants although I admit both are useful. A script (loop) doing an antcall would work but this
seems like an overkill. Fixing this at JUnit4 level isn't trivial either because the seed
is currently picked even before junit4 is started (to select the target charset).

> add support for running the same test method many times
> -------------------------------------------------------
>                 Key: LUCENE-4463
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Wish
>          Components: general/build
>            Reporter: Robert Muir
>         Attachments: LUCENE-4463.patch
> I have a shell script for this, mike has a python script, its annoying :)
> I want to do something like this:
> ant beast -Dtestcase=XXXX -Dtestmethod=YYYY -Diterations=100
> I would be happy with a simple loop that just invokes 'test' somehow: getting a fresh
new JVM to each iteration is desirable anyway (so you get fresh codecs, etc). 
> the -Dtests.iters is not really useful for this because it does not allow -Dtestmethod
and it does not give a fresh jvm.
> bonus points if it can use multiple jvms at the same time though :)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message