spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zoltán Zvara <zoltan.zv...@gmail.com>
Subject Spark remote communication pattern
Date Thu, 09 Apr 2015 08:15:44 GMT
Dear Developers,

I'm trying to investigate the communication pattern regarding data-flow
during execution of a Spark program defined by an RDD chain. I'm
investigating from the Task point of view, and found out that the task type
ResultTask (as retrieving the iterator for its RDD for a given partition),
effectively asks the BlockManager to get the block from local or remote
location. What I do there is to include actual location data in BlockResult
so the task can tell where it retrieved the data from. I've found out that
ResultTask can issue a data-flow only in this case.

What's the case with the ShuffleMapTask? What happens there? I'm trying to
log locations which are included in the shuffle process. I would be happy
to receive a few hints regarding where remote communication is managed in
case of ShuffleMapTask.

Thanks!

Zoltán

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message