spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hao Ren <>
Subject Broadcast variable questions
Date Wed, 21 Jan 2015 14:24:28 GMT

Spark 1.2.0, standalone, local mode(for test)

Here are several questions on broadcast variable: 

1) Where is the broadcast variable cached on executors ? In memory or On
disk ?

I read somewhere, it was said these variables are stored in spark.local.dir.
But I can find any info in Spark 1.2 document. I encountered a problem with
broadcast variables. I have a loop in which a broadcast variable is created,
after 3 iteration, the used memory increased quickly until the full size,
and Spark is blocked, no error message, no exception, just blocked. I would
like to make sure whether it is caused by too many broadcast variables,
because I did not call unpersist() on each broadcast variable.

2) I find that broadcast variable has destroy() and unpersist() method,
what's the difference between them? If a broadcast variable is destroyed, is
it removed from where it is stored ?


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message