spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Hubregtsen <>
Subject What is the location in the source code of the computation of the elements in a map transformation?
Date Fri, 01 May 2015 21:06:53 GMT
I am trying to understand what the data and computation flow is in Spark, and
believe I fairly understand the Shuffle (both map and reduce side), but I do
not get what happens to the computation from the map stages. I know all maps
gets pipelined on the shuffle (when there is no other action in between),
but I can not find where the actual computation for the map happens (for
instance for => x+1), where does the +1 happen?). Any pointers to
files or functions are appreciated. 

I know compute of rdd/MapPartitionsRDD.scala gets called, but I loose track
of the lambda function after this. 



View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message