spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "deenar.toraskar" <deenar.toras...@db.com>
Subject Running a task once on each executor
Date Tue, 25 Mar 2014 17:03:07 GMT
Hi

Is there a way in Spark to run a function on each executor just once. I have
a couple of use cases. 

a) I use an external library that is a singleton. It keeps some global state
and provides some functions to manipulate it (e.g. reclaim memory. etc.) . I
want to check the global state of this library on each executor. 

b) To get jvm stats or instrumentation on each executor.

Currently I have a crude way of achieving something similar, I just run a
map on a large RDD that is hash partitioned, this does not guarantee that
the job would run just once.

Deenar



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-a-task-once-on-each-executor-tp3203.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message