commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Herbert <alex.d.herb...@gmail.com>
Subject Re: [Rng] New XoShiRo generators
Date Fri, 15 Mar 2019 11:25:53 GMT

> On 14 Mar 2019, at 14:10, Alex Herbert <alex.d.herbert@gmail.com> wrote:
> 
> Sorry, my earlier message was truncated.
> 
> On 13/03/2019 18:31, Gilles Sadowski wrote:
>> 
>>> I ran:
>>> 
>>> XorShiftXorComposite: XorComposite using XorShift1024Star +
>>> XorShift1024StarPhi with the same seed
>>> 
>>> XorShiftSerialComposite: SerialComposite using XorShift1024Star +
>>> XorShift1024StarPhi with the same seed
>>> 
>>> SplitXorComposite: XorComposite using XorShift1024Star + TwoCmres (this
>>> is a control)
>>> 
>>> 
>>> FAILURE counts for Dieharder:
>>> 
>>> XorShiftXorComposite : 89, 105, 104, 104, 105, 106, 105, 104
>>> XorShiftSerialComposite : 27, 23, 22
>>> SplitXorComposite : 0, 0, 0
> BigCrush has finally finished:
> 
> XorShiftXorComposite : 0, 3, 0
> XorShiftSerialComposite : 0, 0, 0
> SplitXorComposite : 0, 1, 0
> 
> These results are actually not bad.
> 
> I am wondering about the Dieharder results. I think that originally I did a xor operation
on nextLong for the XorShiftXorComposite. Although it should not matter (due to caching of
nextLong() to produce ints) I will redo this using nextInt for all composites as I just did
for BigCrush. Dieharder did take an extremely long time so to make sure I will repeat Dieharder
after a machine reboot. A linux update to c libraries may have affected it.
> 
> Alex
> 

I re-ran Dieharder. It still fails a lot.

XorShiftXorComposite : 88, 105, 89 : 396.2 +/- 9.9
XorShiftSerialComposite : 24, 25, 23 : 134.1 +/- 16.1
SplitXorComposite : 0, 0, 0 : 90.8 +/- 21.9

The numbers at the end are the average and SD of the run time in minutes.

So the good generator passes the test suite OK in 90 minutes. The bad generator takes over
6 hours and fails a lot. When I was periodically checking the results for completion I think
this time is mainly spent in the rgb_kstest_test. But why that takes so long I do not know.
It fails for the XorShiftXorComposite but passes for the XorShiftSerialComposite.

All results are now posted to the RNG-82 jira ticket.

Q. Whether to deprecate the XOR_SHIFT_1024_S?

- Dieharder says yes
- BigCrush says no (WHY?)

I have investigated whether the c code for calling TestU01 is not passing the values through.
A quick look at the stdin2testu01.c code here:

unsigned long nextInt(void *par,
                      void *sta) {
  StdinReader_state *state = (StdinReader_state *) sta;
  if (state->index >= BUFFER_LENGTH) {
    /* Refill. */
    fread(state->buffer, sizeof(unsigned long), BUFFER_LENGTH, stdin);
    state->index = 0;
  }

  uint32_t random = state->buffer[state->index];
  ++state->index; /* Next request. */

  return random;
}

state->buffer is a long and is read as a long. This is then used to return a uint32_t which
is a 32-bit unsigned. 

The c standard for long states that it can be 32-bits or more! So to check what happens on
my 64-bit test machine:

#include <stdio.h>
#include <limits.h>
#include <stdint.h>

int main(void){
    printf("sizeof(int) = %d\n", (int)sizeof(int));
    printf("sizeof(long) = %d\n", (int)sizeof(long));
    printf("sizeof(unsigned long) = %d\n", (int)sizeof(unsigned long));
    printf("sizeof(uint32_t) = %d\n", (int)sizeof(uint32_t));
    printf("sizeof(double) = %d\n", (int)sizeof(double));
    unsigned long value = 1;
    for (int i = 1; i <= 8 * (int)sizeof(unsigned long); i++) {
        printf("[%d] %lu = %u\n", i, value, (uint32_t) value);
	value = (value << 1);
    }
    return 0;
}

> gcc test.c && ./a.out

sizeof(int) = 4
sizeof(long) = 8
sizeof(unsigned long) = 8
sizeof(uint32_t) = 4
sizeof(double) = 8
…
[31] 1073741824 = 1073741824
[32] 2147483648 = 2147483648
[33] 4294967296 = 0
[34] 8589934592 = 0
…


ERROR: It seems my machine thinks a long is 8 bytes but the test code in stdin2testu01.c assumes
it is 4 bytes! So when the long is cast to uint32_t the upper 32-bits are lost.

This means that the test suite run on my machine for TestU01 is passing alternating ints from
UniformRandomProvider nextInt() through to the TestU01 test suite.

It may help explain why the test takes so long because it has to generate twice as many numbers.

Can you check your standard stress test environment for the size of (unsigned long) using
the above code?

Note: This does not make the BigCrush results invalid for a standard long provider. But it
does means that the BigCrush results correspond to the lower 32-bits from the long.

For an int provider it means that alternative ints are ignored. That is not good.

I am going to update stdin2testu01.c so that it passes all the input bits to TestU01 and try
again.

I’ve read some of the code and manual of TestU01 and the values that are returned from the
unif01_Gen->GetBits method (unsigned long) are assumed to be 32-bit unsigned. So I think
just changing stdin2testu01.c to read the buffer using uint32_t should work.

Alex




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message