lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4463) add support for running the same test method many times
Date Sat, 06 Oct 2012 07:26:06 GMT


Robert Muir commented on LUCENE-4463:

I know its not easy: thats why its a "wish" task :)

I guess I'm just basically listing what could be seen as an expert feature here, but arguably
necessary if you are trying to do randomized tests.

The fact is that things dont always reproduce 100%, and you know this is definitely a failure
in our tests (e.g. the current situation motivated
me to open LUCENE-4460). But really part of random testing is you know, you dont have to try
to write targeted tests but just throw hardware
at the problem (which I'm doing... my office is really hot right now!).

The frustrating part is I think ideally you want to basically treat this whole randomized
test situation like a normal deterministic unit test, you know
like a normal developer would have, so you know you fixed the bug, even if the test isn't
great and doesnt reproduce 100%, you want to know its really
fixed rather than taking blind stabs, waiting to see if all the computers in your house running
full throttle will trip a bug in 24 hours to declare success :)

So I'm just basically opening this "wish" task to try to think of ways to make this easier
and more efficient.

I'd actually go so far to say the tests.iters is really outdated for lucene's tests these
days (since we have so much class-level "parameterization"
and we should be focusing on this tests.dups (and maybe removing the tests.iters totally).
Maybe thats just particular to us though, but as I mentioned
above I think we show some real use cases for parameterizing the entire test class with certain
things because it simplifies debugging.

> add support for running the same test method many times
> -------------------------------------------------------
>                 Key: LUCENE-4463
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Wish
>          Components: general/build
>            Reporter: Robert Muir
>         Attachments: LUCENE-4463.patch
> I have a shell script for this, mike has a python script, its annoying :)
> I want to do something like this:
> ant beast -Dtestcase=XXXX -Dtestmethod=YYYY -Diterations=100
> I would be happy with a simple loop that just invokes 'test' somehow: getting a fresh
new JVM to each iteration is desirable anyway (so you get fresh codecs, etc). 
> the -Dtests.iters is not really useful for this because it does not allow -Dtestmethod
and it does not give a fresh jvm.
> bonus points if it can use multiple jvms at the same time though :)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message