spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Umar Javed <umarj.ja...@gmail.com>
Subject difference between 'fetchWaitTime' and 'remoteFetchTime'
Date Wed, 20 Nov 2013 23:02:35 GMT
In the class ShuffleReadMetrics in executor/TaskMetrics.scala, there are
two variables:

1) fetchWaitTime: /**


   * Total time that is spent blocked waiting for shuffle to fetch data


   */

2) remoteFetchTime

/**


   * The total amount of time for all the shuffle fetches.  This adds up
time from overlapping

   *     shuffles, so can be longer than task time


   */

As I understand it, the difference between these two is that fetchWaitTime
is remoteFetchTime without the overlapped time counted exactly once. Is
that right? Can somebody explain the difference better?

thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message