spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandip Mehta <sandip.mehta....@gmail.com>
Subject Re: Call Site - Spark Context
Date Thu, 01 Oct 2015 22:46:57 GMT
Thanks Robin. 

Regards
SM
> On 01-Oct-2015, at 3:15 pm, Robin East <robin.east@xense.co.uk> wrote:
> 
> From the comments in the code:
> 
> When called inside a class in the spark package, returns the name of the user code class
(outside the spark package) that called into Spark, as well as which Spark method they called.
This is used, for example, to tell users where in their code each RDD got created.
> Keep crawling up the stack trace until we find the first function not inside of the spark
package. We track the last (shallowest) contiguous Spark method. This might be an RDD transformation,
a SparkContext function (such as parallelize), or anything else that leads to instantiation
of an RDD. We also track the first (deepest) user method, file, and line.
> So basically it’s a mechanism to report where in the user’s code an RDD is created.
> -------------------------------------------------------------------------------
> Robin East
> Spark GraphX in Action Michael Malak and Robin East
> Manning Publications Co.
> http://www.manning.com/books/spark-graphx-in-action <http://www.manning.com/books/spark-graphx-in-action>
> 
> 
> 
> 
> 
>> On 1 Oct 2015, at 23:06, Sandip Mehta <sandip.mehta.sub@gmail.com <mailto:sandip.mehta.sub@gmail.com>>
wrote:
>> 
>> Hi,
>> 
>> I wanted to understand what is the purpose of Call Site in Spark Context?
>> 
>> Regards
>> SM
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <mailto:user-unsubscribe@spark.apache.org>
>> For additional commands, e-mail: user-help@spark.apache.org <mailto:user-help@spark.apache.org>
>> 
> 


Mime
View raw message