commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Herbert <alex.d.herb...@gmail.com>
Subject Re: [Rng] New XoShiRo generators
Date Wed, 20 Mar 2019 13:55:14 GMT

On 20/03/2019 13:08, Gilles Sadowski wrote:
>
>> - Perhaps there is another place for them?
>>
>> I am happy to not clutter the source repo and keep them somewhere for
>> devs. But where? You tell me. Maybe in the examples-stress module they
>> are OK as that one is not officially part of the library.
> Unless a script is robust and readily usable by a newbie (i.e. anyone
> but the author), there is the risk that it becomes cruft.
> What you did is great work but I doubt that many would be interested
> in detailed tables of the failures, once one can easily compare the number
> of failures.
> If more is needed, there is no shortcut to reading the doc of the test suites
> themselves and other resources on the web...
>
> So, to summarize, what I think is interesting is to make it easy to rerun
> the test suites and update the site (e.g. for the "current" JDK):
>   * a document that mentions the external tools requirement
>   * a standalone application for running "RandomStressTester"
>   * a script that collects results from the above run, formats them into the
>     "quality" table ready to be pasted into the "rng.apt" file, and copies the
>     output files to their appropriate place (*not* assuming a git repository)
>
> It would also be great to be able to easily start the many benchmarks
> and similarly collects the results into the user guide tables.

I will write a script that will collect the results from the benchmark, 
build the apt table and copy the output files to a directory.

Note that the benchmark files only contain the RNG class name:

# RNG: org.apache.commons.rng.core.source32.JDKRandom

This means the script has to be kept up to date with the corresponding 
enum name, and the desired output order for the apt table. I can add 
these to a simple array as the top which any newbie should be able to 
extend.

It would be desirable for the script to search for results files in a 
set of directories, identify them as dieharder (dh) or TestU01 (tu) 
results and then output the files for each run N of a unique generator M 
to the existing directory structure, e.g:

 > collate-benchmark-results.pl output dir1 dir2 ...

output/(dh|tu)/run_N/(dh|tu)_M

This should be done robustly such that the script can be pointed at the 
git source tree for the site and it will figure out that the files are 
already in the correct place. I think this would just involve sorting 
the paths to all the results numerically not alphabetically (e.g. dh_1 < 
dh_2 < dh_10) before processing them to output.


For the JMH benchmarks I tend to run e.g.

 > mvn test -P benchmark -Dbenchmark=NextDoublePerformance

I then have a script to parse the JSON in 
target/jmh-result.NextDoublePerformance.json to a Jira format or CSV 
format table. Usually I add some relative score columns in a spreadsheet 
then paste into Jira (which nicely accepts the tables).

It should not be too hard to generate the existing apt tables for 
performance. I can look into this. I am assuming that the tables are 
based on the highest 'number of samples' from each generator.


>
>>>> [...]
>>>> You then have a utility for dumping output of any random source to file
>>>> in a variety of formats.
>>>>
>>>> Although long output is not needed for the test suites it is useful for
>>>> native long generators.
>>>>
>>>> WDYT?
>>> Looks good!
>> OK. I will work on the raw data dumper as a Jira ticket. It is
>> encapsulated work that does not really effect anything else.
>>
>>
>> DieHarder has finished!
>>
>> I think my stupidity is what caused previous crashes. I was running the
>> stress test within the source tree and possibly git checkout onto
>> another branch makes some of the directory paths stale killing any
>> processes linked to those paths. I'll not do that again.
> Hence, the "standalone" application is the right choice it seems.
>
>> FYI: Here are the old results with incorrect byte order:
>>
>> XorShiftSerialComposite : 24, 25, 23 : 134.1 +/- 16.1
>> XorShiftXorComposite : 88, 105, 89 : 396.2 +/- 9.9
>> SplitXorComposite : 0, 0, 0 : 90.8 +/- 21.9
>>
>> Here are the new results with correct byte order:
>>
>> XorShiftSerialComposite : 13, 15, 10 : 105.5 +/- 1.8
>> XorShiftXorComposite : 57, 57, 57 : 102.9 +/- 1.5
>> SplitXorComposite : 0, 0, 0 : 99.9 +/- 3.2
>>
>> So interestingly passing the correct byte order lowers the number of
>> failures. There are still lots.
>>
>> And BigCrush (with the fix for passing the correct byte order):
>>
>> XorShiftSerialComposite : 40, 39, 39 : 608.2 +/- 3.9
>> XorShiftXorComposite : 54, 53, 53 : 646.8 +/- 10.9
>> SplitXorComposite : 0, 0, 0 : 625.8 +/- 0.2
> Curious to know whether it is also affected by the byte ordering.

I'll re-run BigCrush with the wrong byte ordering when I have updated 
the stress test code. I finished it yesterday but will look it over with 
fresh eyes before committing it.

Alex



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message