spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From RK <prk...@yahoo.com.INVALID>
Subject How to propagate the value set in SparkContext.setCallSite() to all executors?
Date Thu, 01 Jan 2015 18:27:59 GMT
Here is a snippet of my spark streaming job code where I am setting call site after each of
the calls. I have intentionally left out the line # in the call site string that I am setting
to easily identify the values that I am setting from the ones Spark is looking up from stack
trace.

I am running this job in a single slave node as shown below.

During the execution of this job, when I look at the spark stages information, I see only
1 stage (Stage Id: 2) has the call site information that I have set. For all the other stages,
it looks like SparkContext is still going through stack trace.


How can I set the callSite information reliably when running a job on single/multiple slaves
& executors so that this information gets propagated across all JVMs?
Thanks,RK

Mime
View raw message