spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-735) memory leak in KryoSerializer
Date Tue, 21 Oct 2014 07:47:33 GMT

     [ https://issues.apache.org/jira/browse/SPARK-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Wendell resolved SPARK-735.
-----------------------------------
    Resolution: Fixed

I think this was fixed a long time ago.

> memory leak in KryoSerializer
> -----------------------------
>
>                 Key: SPARK-735
>                 URL: https://issues.apache.org/jira/browse/SPARK-735
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 0.7.0
>            Reporter: Reynold Xin
>            Assignee: Ram Sriharsha
>
> KryoSerializer uses a ThreadLocal object to store a kryo buffer, which is never cleaned
up.
> This becomes a serious problem in projects like Shark, where new threads are created
constantly by the thrift server. As long as the new thread references the kryo serializer,
it will create a new kryo buffer.
> A simple solution is to remove the ThreadLocal reference, and create a new buffer object
every time a new kryo serializer instance is created.
> This is not very expensive because large writes actually go through the serialization
stream interface, which reuse the buffer anyway. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message