spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Roberts <AROBE...@uk.ibm.com>
Subject Re: BytesToBytes and unaligned memory
Date Mon, 18 Apr 2016 13:49:50 GMT
Ted, yes with the forced true value all tests pass, we use the unaligned 
check in 15 other suites.

Our java.nio.Bits.unaligned() function checks that the detected os.arch 
value matches a list of known implementations (not including s390x).

We could add it to the known architectures in the catch block but this 
won't make a difference here as because we call unaligned() OK (no 
exception is thrown), we don't reach the architecture checking stage 
anyway.

I see in org.apache.spark.memory.MemoryManager that unaligned support is 
required for off-heap memory in Tungsten (perhaps incorrectly if no code 
ever exercises it in Spark?). Instead of having a requirement should we 
instead log a warning once that this is likely to lead to slow 
performance? What's the rationale for supporting unaligned memory access: 
it's my understanding that it's typically very slow, are there any design 
docs or perhaps a JIRA where I can learn more? 

Will run a simple test case exercising unaligned memory access for Linux 
on Z (without using Spark) and can also run the tests claiming to require 
unaligned memory access on a platform where unaligned memory access is 
definitely not supported for shorts/ints/longs. 

if these tests continue to pass then I think the Spark tests don't 
exercise unaligned memory access, cheers


 




From:   Ted Yu <yuzhihong@gmail.com>
To:     Adam Roberts/UK/IBM@IBMGB
Cc:     "dev@spark.apache.org" <dev@spark.apache.org>
Date:   15/04/2016 17:35
Subject:        Re: BytesToBytes and unaligned memory



I am curious if all Spark unit tests pass with the forced true value for 
unaligned.
If that is the case, it seems we can add s390x to the known architectures.

It would also give us some more background if you can describe 
how java.nio.Bits#unaligned() is implemented on s390x.

Josh / Andrew / Davies / Ryan are more familiar with related code. It 
would be good to hear what they think.

Thanks

On Fri, Apr 15, 2016 at 8:47 AM, Adam Roberts <AROBERTS@uk.ibm.com> wrote:
Ted, yeah with the forced true value the tests in that suite all pass and 
I know they're being executed thanks to prints I've added 

Cheers, 




From:        Ted Yu <yuzhihong@gmail.com> 
To:        Adam Roberts/UK/IBM@IBMGB 
Cc:        "dev@spark.apache.org" <dev@spark.apache.org> 
Date:        15/04/2016 16:43 
Subject:        Re: BytesToBytes and unaligned memory 



Can you clarify whether BytesToBytesMapOffHeapSuite passed or failed with 
the forced true value for unaligned ? 

If the test failed, please pastebin the failure(s). 

Thanks 

On Fri, Apr 15, 2016 at 8:32 AM, Adam Roberts <AROBERTS@uk.ibm.com> wrote: 

Ted, yep I'm working from the latest code which includes that unaligned 
check, for experimenting I've modified that code to ignore the unaligned 
check (just go ahead and say we support it anyway, even though our JDK 
returns false: the return value of java.nio.Bits.unaligned()). 

My Platform.java for testing contains: 

private static final boolean unaligned; 

static { 
  boolean _unaligned; 
  // use reflection to access unaligned field 
  try { 
    System.out.println("Checking unaligned support"); 
    Class<?> bitsClass = 
      Class.forName("java.nio.Bits", false, 
ClassLoader.getSystemClassLoader()); 
    Method unalignedMethod = bitsClass.getDeclaredMethod("unaligned"); 
    unalignedMethod.setAccessible(true); 
    _unaligned = Boolean.TRUE.equals(unalignedMethod.invoke(null)); 
    System.out.println("Used reflection and _unaligned is: " + 
_unaligned); 
    System.out.println("Setting to true anyway for experimenting"); 
    _unaligned = true; 
    } catch (Throwable t) { 
      // We at least know x86 and x64 support unaligned access. 
      String arch = System.getProperty("os.arch", ""); 
      //noinspection DynamicRegexReplaceableByCompiledPattern 
      // We don't actually get here since we find the unaligned method OK 
and it returns false (I override with true anyway) 
      // but add s390x incase we somehow fail anyway. 
      System.out.println("Checking for s390x, os.arch is: " + arch); 
      _unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|s390x|amd64)$"); 

    } 
    unaligned = _unaligned; 
    System.out.println("returning: " + unaligned); 
  } 
} 

Output is, as you'd expect, "used reflection and _unaligned is false, 
setting to true anyway for experimenting", and the tests pass. 

No other problems on the platform (pending a different pull request). 

Cheers, 







From:        Ted Yu <yuzhihong@gmail.com> 
To:        Adam Roberts/UK/IBM@IBMGB 
Cc:        "dev@spark.apache.org" <dev@spark.apache.org> 
Date:        15/04/2016 15:32 
Subject:        Re: BytesToBytes and unaligned memory 




I assume you tested 2.0 with SPARK-12181 . 

Related code from Platform.java if java.nio.Bits#unaligned() throws 
exception: 

      // We at least know x86 and x64 support unaligned access. 
      String arch = System.getProperty("os.arch", ""); 
      //noinspection DynamicRegexReplaceableByCompiledPattern 
      _unaligned = arch.matches("^(i[3-6]86|x86(_64)?|x64|amd64)$"); 

Can you give us some detail on how the code runs for JDKs on zSystems ? 

Thanks 

On Fri, Apr 15, 2016 at 7:01 AM, Adam Roberts <AROBERTS@uk.ibm.com> wrote: 

Hi, I'm testing Spark 2.0.0 on various architectures and have a question, 
are we sure if 
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java 
really is attempting to use unaligned memory access (for the 
BytesToBytesMapOffHeapSuite tests specifically)? 

Our JDKs on zSystems for example return false for the 
java.nio.Bits.unaligned() method and yet if I skip this check and add 
s390x to the supported architectures (for zSystems), all thirteen tests 
here pass. 

The 13 tests here all fail as we do not pass the unaligned requirement 
(but perhaps incorrectly): 
core/src/test/java/org/apache/spark/unsafe/map/BytesToBytesMapOffHeapSuite.java 
and I know the unaligned checking is at 
common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java 

Either our JDK's method is returning false incorrectly or this test isn't 
using unaligned memory access (so the requirement is invalid), there's no 
mention of alignment in the test itself. 

Any guidance would be very much appreciated, cheers 


Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU 



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU 



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Mime
View raw message