I am trying to understand what the data and computation flow is in Spark, and
believe I fairly understand the Shuffle (both map and reduce side), but I do
not get what happens to the computation from the map stages. I know all maps
gets pipelined on the shuffle (when there is no other action in between),
but I can not find where the actual computation for the map happens (for
instance for rdd.map(x => x+1), where does the +1 happen?). Any pointers to
files or functions are appreciated.
I know compute of rdd/MapPartitionsRDD.scala gets called, but I loose track
of the lambda function after this.
Thanks,
Tom
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/What-is-the-location-in-the-source-code-of-the-computation-of-the-elements-in-a-map-transformation-tp11971.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
|