spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public
Date Tue, 21 Oct 2014 07:15:34 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178068#comment-14178068
] 

Patrick Wendell commented on SPARK-4030:
----------------------------------------

Hey Shivaram - IIRC we made this private to protect users from shooting themselves in the
foot and trying to use a broadcast variable that has been destroy()'d, potentially by another
thread or somewhere difficult to track down. If we expose this it would be good to test it
thoroughly (I think we put some logic in there around this already thinking that we might
open it up later). Also, it might be good to capture the callsite of the destroy call and
echo it back to users if they attempt to use an invalid broadcast variable.

> `destroy` method in Broadcast should be public
> ----------------------------------------------
>
>                 Key: SPARK-4030
>                 URL: https://issues.apache.org/jira/browse/SPARK-4030
>             Project: Spark
>          Issue Type: Bug
>          Components: Block Manager, Spark Core
>    Affects Versions: 1.1.0, 1.2.0
>            Reporter: Shivaram Venkataraman
>
> The destroy method in Broadcast.scala [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/broadcast/Broadcast.scala#L91]
is right now marked as private[spark]
> This prevents long-running applications from cleaning up memory used by broadcast variables
on the driver.  Also as broadcast variables are always created with persistence MEMORY_DISK,
this slows down jobs when old broadcast variables are flushed to disk. 
> Making `destroy` public can help applications control the lifetime.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message