spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hao Ren <inv...@gmail.com>
Subject Broadcast variable questions
Date Wed, 21 Jan 2015 14:24:28 GMT
Hi,

Spark 1.2.0, standalone, local mode(for test)

Here are several questions on broadcast variable: 

1) Where is the broadcast variable cached on executors ? In memory or On
disk ?

I read somewhere, it was said these variables are stored in spark.local.dir.
But I can find any info in Spark 1.2 document. I encountered a problem with
broadcast variables. I have a loop in which a broadcast variable is created,
after 3 iteration, the used memory increased quickly until the full size,
and Spark is blocked, no error message, no exception, just blocked. I would
like to make sure whether it is caused by too many broadcast variables,
because I did not call unpersist() on each broadcast variable.

2) I find that broadcast variable has destroy() and unpersist() method,
what's the difference between them? If a broadcast variable is destroyed, is
it removed from where it is stored ?

Hao







--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Broadcast-variable-questions-tp21292.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message