spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shivaram Venkataraman (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-4031) Read broadcast variables on use
Date Tue, 21 Oct 2014 05:41:33 GMT
Shivaram Venkataraman created SPARK-4031:
--------------------------------------------

             Summary: Read broadcast variables on use
                 Key: SPARK-4031
                 URL: https://issues.apache.org/jira/browse/SPARK-4031
             Project: Spark
          Issue Type: Bug
          Components: Block Manager, Spark Core
            Reporter: Shivaram Venkataraman
            Assignee: Shivaram Venkataraman


This is a proposal to change the broadcast variable implementations in Spark to only read
values when they are used rather than on deserializing.

This change will be very helpful (and in our use cases required) for complex applications
which have a large number of broadcast variables. For example if broadcast variables are class
members, they are captured in closures even when they are not used.

We could also consider cleaning closures more aggressively, but that might be a more complex
change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message