spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-1042) spark cleans all java broadcast variables when it hits the spark.cleaner.ttl
Date Tue, 21 Oct 2014 06:53:34 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Wendell resolved SPARK-1042.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 0.9.2

I think this was fixed back in 0.9.2

> spark cleans all java broadcast variables when it hits the spark.cleaner.ttl 
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-1042
>                 URL: https://issues.apache.org/jira/browse/SPARK-1042
>             Project: Spark
>          Issue Type: Bug
>          Components: Java API, Spark Core
>    Affects Versions: 0.8.0, 0.8.1, 0.9.0
>            Reporter: Tal Sliwowicz
>            Assignee: OuyangJin
>            Priority: Critical
>              Labels: memory_leak
>             Fix For: 0.9.2
>
>
> When setting spark.cleaner.ttl, spark performs the cleanup on time - but it cleans all
broadcast variables, not just the ones that are older than the ttl. This creates an exception
when the next mapPartitions runs because it cannot find the broadcast variable, even when
it was created immediately before running the task.
> Our temp workaround - not set the ttl and suffer from an ongoing memory leak (forces
a restart).
> We are using JavaSparkContext and our broadcast variables are Java HashMaps.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message