spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: BytesToBytes and unaligned memory
Date Fri, 15 Apr 2016 15:43:05 GMT
Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with
the forced true value for unaligned ?

If the test failed, please pastebin the failure(s).

Thanks

On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <AROBERTS@uk.ibm.com> wrote:

> Ted, yep I'm working from the latest code which includes that unaligned
> check, for experimenting I've modified that code to ignore the unaligned
> check (just go ahead and say we support it anyway, even though our JDK
> returns false: the return value of java.nio.Bits.unaligned()).
>
> My Platform.java for testing contains:
>
> private static final boolean unaligned;
>
> static {
>   boolean _unaligned;
>   // use reflection to access unaligned field
>   try {
> *    System.out.println("Checking unaligned support");*
>     Class<?> bitsClass =
>       Class.forName("java.nio.Bits", false,
> ClassLoader.getSystemClassLoader());
>     Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned");
>     unalignedMethod.setAccessible(true);
>     _unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null));
>     *System.out.println("Used reflection and _unaligned is: " +
> _unaligned);*
> *    System.out.println("Setting to true anyway for experimenting");*
> *    _unaligned = true;*
>     } catch (Throwable t) {
>       // We at least know x86 and x64 support unaligned access.
>       String arch = System.getProperty("os.arch", "");
>       //noinspection DynamicRegexReplaceableByCompiledPattern
> *      // We don't actually get here since we find the unaligned method OK
> and it returns false (I override with true anyway)*
> *      // but add s390x incase we somehow fail anyway.*
> *      System.out.println("Checking for s390x, os.arch is: " + arch);*
> *      _unaligned =
> arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$");*
>     }
>     unaligned = _unaligned;
> *    System.out.println("returning: " + unaligned);*
>   }
> }
>
> Output is, as you'd expect, "used reflection and _unaligned is false,
> setting to true anyway for experimenting", and the tests pass.
>
> No other problems on the platform (pending a different pull request).
>
> Cheers,
>
>
>
>
>
>
>
> From:        Ted Yu <yuzhihong@gmail.com>
> To:        Adam Roberts/UK/IBM@IBMGB
> Cc:        "dev@spark.apache.org" <dev@spark.apache.org>
> Date:        15/04/2016 15:32
> Subject:        Re: BytesToBytes and unaligned memory
> ------------------------------
>
>
>
> I assume you tested 2.0 with SPARK-12181 .
>
> Related code from Platform.java if java.nio.Bits#unaligned() throws
> exception:
>
>       // We at least know x86 and x64 support unaligned access.
>       String arch = System.getProperty("os.arch", "");
>       //noinspection DynamicRegexReplaceableByCompiledPattern
>       _unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$");
>
> Can you give us some detail on how the code runs for JDKs on zSystems ?
>
> Thanks
>
> On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <*AROBERTS@uk.ibm.com*
> <AROBERTS@uk.ibm.com>> wrote:
> Hi, I'm testing Spark 2.0.0 on various architectures and have a question,
> are we sure if
> *core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java*
> <https://github.com/apache/spark/blob/96941b12f8b465df21423275f3cd3ade579b4fa1/core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java>
> really is attempting to use unaligned memory access (for the
> BytesToBytesMapOffHeapSuite tests specifically)?
>
> Our JDKs on zSystems for example return false for the
> java.nio.Bits.unaligned() method and yet if I skip this check and add s390x
> to the supported architectures (for zSystems), all thirteen tests here
> pass.
>
> The 13 tests here all fail as we do not pass the unaligned requirement
> (but perhaps incorrectly):
>
> *core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java*
> <https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java>
> and I know the unaligned checking is at
> *common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java*
> <https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java>
>
> Either our JDK's method is returning false incorrectly or this test isn't
> using unaligned memory access (so the requirement is invalid), there's no
> mention of alignment in the test itself.
>
> Any guidance would be very much appreciated, cheers
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

Mime
View raw message