spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aniket Bhatnagar (JIRA)" <>
Subject [jira] [Created] (SPARK-18787) does not completely turn off direct buffer usage by Netty
Date Thu, 08 Dec 2016 16:46:58 GMT
Aniket Bhatnagar created SPARK-18787:

             Summary: does not completely turn off direct
buffer usage by Netty
                 Key: SPARK-18787
             Project: Spark
          Issue Type: Bug
    Affects Versions: 2.0.2
            Reporter: Aniket Bhatnagar

The documentation for the configuration states that it will
force all allocations from Netty to be on-heap but this currently does not happen. The reason
is that preferDirect of Netty's PooledByteBufAllocator doesn't completely eliminate use of
off heap by Netty. In order to completely stop netty from using off heap memory, we need to
set the following system properties:
- io.netty.noUnsafe=true
- io.netty.threadLocalDirectBufferSize=0

The proposal is to set properties (using System.setProperties) when the executor starts (before
any of the Netty classes load) or document these properties to hint users on how to completely
eliminate off Netty' heap footprint.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message