spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Umar Javed <>
Subject difference between 'fetchWaitTime' and 'remoteFetchTime'
Date Wed, 20 Nov 2013 23:02:35 GMT
In the class ShuffleReadMetrics in executor/TaskMetrics.scala, there are
two variables:

1) fetchWaitTime: /**

   * Total time that is spent blocked waiting for shuffle to fetch data


2) remoteFetchTime


   * The total amount of time for all the shuffle fetches.  This adds up
time from overlapping

   *     shuffles, so can be longer than task time


As I understand it, the difference between these two is that fetchWaitTime
is remoteFetchTime without the overlapped time counted exactly once. Is
that right? Can somebody explain the difference better?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message