spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Darabos <daniel.dara...@lynxanalytics.com>
Subject Re: Spark 1.6.1
Date Wed, 03 Feb 2016 13:16:08 GMT
On Tue, Feb 2, 2016 at 7:10 PM, Michael Armbrust <michael@databricks.com>
wrote:

> What about the memory leak bug?
>> https://issues.apache.org/jira/browse/SPARK-11293
>> Even after the memory rewrite in 1.6.0, it still happens in some cases.
>> Will it be fixed for 1.6.1?
>>
>
> I think we have enough issues queued up that I would not hold the release
> for that, but if there is a patch we should try and review it.  We can
> always do 1.6.2 when more issues have been resolved.  Is this an actual
> issue that is affecting a production workload or are we concerned about an
> edge case?
>

The way we (Lynx Analytics) use RDDs, this affects almost everything we do
in production. Thankfully it does not cause any issues, it just logs a lot
of errors. I think the adverse effect may be that the memory manager does
not have a fully correct picture. But as long as the leak fits in the
"other" (unmanaged) memory fraction this will not cause issues. We don't
see this as an urgent issue. Thanks!

Mime
View raw message