commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex D Herbert (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (RNG-88) Update the GenerationPerformance benchmark
Date Thu, 04 Apr 2019 11:15:00 GMT

    [ https://issues.apache.org/jira/browse/RNG-88?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809742#comment-16809742
] 

Alex D Herbert commented on RNG-88:
-----------------------------------

PS.

Note that the example for a benchmark class has two {{@Benchmark}} methods to test JMH timing
overhead. These should sum to approximately the same metric as the BASELINE implementation.
In the above case for {{nextDouble()}} here are the results:
||randomSourceName||Method||Score||Error||Median||Error/Score||
|BASELINE|nextDouble|2.57593197585943|0.003460918835402205|2.57636702779831|0.00134355987185862|
|ISAAC|nextDouble|11.492061181491472|0.03105190366683462|11.4930378375858|0.00270203083471616|
|JDK|nextDouble|19.07710097015598|0.13765564275214587|19.0508121852192|0.00721575269573155|
|KISS|nextDouble|9.205196709268291|0.006192317077079371|9.20614509815848|0.000672697963189055|
|MT|nextDouble|10.653521635100258|0.010420185228341606|10.6524402774538|0.00097809772066451|
|MT_64|nextDouble|6.8707343968410886|0.04118236194818618|6.8627989246393|0.00599388064936993|
|MWC_256|nextDouble|6.582187644902875|0.05365540468787355|6.57988293788998|0.00815160666673234|
|SPLIT_MIX_64|nextDouble|4.5286626926183935|0.008298205686561705|4.52667047056436|0.0018323744226055|
|TWO_CMRES|nextDouble|5.359469359437292|0.050123182243253586|5.34417676563963|0.00935226584605685|
|WELL_1024_A|nextDouble|13.632475941306263|2.3697998386644143|12.8807508310923|0.173834881416071|
|WELL_19937_A|nextDouble|18.45089096484812|0.03811822635555045|18.4441767343849|0.00206592876344951|
|WELL_19937_C|nextDouble|19.972261377408604|0.02239912024952394|19.9724925126603|0.0011215114716484|
|WELL_44497_A|nextDouble|19.858159613693175|0.0871231306453897|19.8435844247295|0.00438727114396412|
|WELL_44497_B|nextDouble|20.725319834240956|0.14352086213185533|20.7110885030745|0.00692490457468068|
|WELL_512_A|nextDouble|12.967886115136594|0.023823029376018285|12.9610798024776|0.00183707885498865|
|XOR_SHIFT_1024_S|nextDouble|5.3340350544353825|0.005344749168553254|5.33227602956032|0.00100200863211594|
|XOR_SHIFT_1024_S_PHI|nextDouble|5.5063780535656885|0.7355642816548434|5.34975699418824|0.133584050077805|
| |baselineDouble|2.3360076674923396|0.31171210270144445|2.26776285101707|0.133437962143361|
| |baselineVoid|0.2657420971315234|3.32183670742148E-4|0.265707936583524|0.00125002276390459|

So:
{noformat}
baselineDouble + baselineVoid ~ BASELINE
2.33 + 0.27 ~ 2.57
{noformat}
This shows the baseline implementation is being run.

Note that the Error output by JMH is is the 0.999 percentile of the T-distribution with n-degrees
of freedom multiplied by the standard error of the mean. This is a conservative +/- limit
on the true value of the mean. Dividing the Error by the Score gives a quick check to see
if there is a lot of variation in the mean. 

In this example WELL_1024_A has a single measurement that is 16.5, which is above the usual
12.9 time. Hence using the median for analysis to avoid skew from single outliers.

 

> Update the GenerationPerformance benchmark
> ------------------------------------------
>
>                 Key: RNG-88
>                 URL: https://issues.apache.org/jira/browse/RNG-88
>             Project: Commons RNG
>          Issue Type: Improvement
>          Components: examples
>    Affects Versions: 1.3
>            Reporter: Alex D Herbert
>            Assignee: Alex D Herbert
>            Priority: Minor
>         Attachments: baseline.jpg, next.png
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> The current GenerationPerformance benchmark runs all the generators to collect timing
data. However the act of running the test within JMH takes some time (test overhead). This
overhead should be measured and subtracted from the timing data to create a time attributed
only to the method exercised in the RNG.
> This can be done by creating a dummy RNG that returns a fixed value. The implementation
must be done in a manner where the JIT compiler is not able to remove the dummy method from
the execution path. This is achieved by returning state variables from the dummy RNG (not
final variables).
> Testing can be done using a variable number of iterations and the run-times assessed
for linearity. If the run time scale with the number of iterations then the JIT compiler has
not removed it from execution. The dummy RNG then serves as a baseline for comparison of true
implementations.
> This idea with examples is shown in [RNG-87|https://issues.apache.org/jira/browse/RNG-87]
which tested a variant of the MWC_256 algorithm.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message