Shivaram Venkataraman created SPARK-4031:
--------------------------------------------
Summary: Read broadcast variables on use
Key: SPARK-4031
URL: https://issues.apache.org/jira/browse/SPARK-4031
Project: Spark
Issue Type: Bug
Components: Block Manager, Spark Core
Reporter: Shivaram Venkataraman
Assignee: Shivaram Venkataraman
This is a proposal to change the broadcast variable implementations in Spark to only read
values when they are used rather than on deserializing.
This change will be very helpful (and in our use cases required) for complex applications
which have a large number of broadcast variables. For example if broadcast variables are class
members, they are captured in closures even when they are not used.
We could also consider cleaning closures more aggressively, but that might be a more complex
change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org
|