spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: Communication between Driver and Executors
Date Tue, 18 Nov 2014 02:08:27 GMT
Hi,

so I didn't manage to get the Broadcast variable with a new value
distributed to my executors in YARN mode. In local mode it worked fine, but
when running on YARN either nothing happened (when unpersist() was called
on the driver) or I got a TimeoutException (when called on the executor).
I finally dropped the use of broadcast variables and added a HTTP polling
mechanism from the executors to the driver. I find that a bit suboptimal,
in particular since there is this whole Akka infrastructure already running
and I should be able to just send messages around. However, Spark does not
seem to encourage this. (In general I find that "private" is a bit overused
in the Spark codebase...)

Thanks
Tobias

Mime
View raw message