spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: difference between 'fetchWaitTime' and 'remoteFetchTime'
Date Tue, 26 Nov 2013 06:55:57 GMT
Hey Umar,

I dug into this a bit today out of curiosity since I also wasn't sure.

I updated the in-line documentation here:
https://github.com/apache/incubator-spark/pull/209/files

The more important metric is `fetchWaitTime` which indicates how much
of the task runtime was spent waiting for input data.

remoteFetchTime is an aggregation of all of the fetch delays for each
block... this second metric is a bit more convoluted because those
fetches can actually overlap, so if this is high it doesn't
necessarily indicate any latency hit.

- Patrick

On Mon, Nov 25, 2013 at 1:23 PM, Umar Javed <umarj.javed@gmail.com> wrote:
> Any clarification on this? thanks.
>
>
> On Wed, Nov 20, 2013 at 3:02 PM, Umar Javed <umarj.javed@gmail.com> wrote:
>
>> In the class ShuffleReadMetrics in executor/TaskMetrics.scala, there are
>> two variables:
>>
>> 1) fetchWaitTime: /**
>>
>>
>>    * Total time that is spent blocked waiting for shuffle to fetch data
>>
>>
>>    */
>>
>> 2) remoteFetchTime
>>
>> /**
>>
>>
>>    * The total amount of time for all the shuffle fetches.  This adds up
>> time from overlapping
>>
>>    *     shuffles, so can be longer than task time
>>
>>
>>    */
>>
>> As I understand it, the difference between these two is that fetchWaitTime
>> is remoteFetchTime without the overlapped time counted exactly once. Is
>> that right? Can somebody explain the difference better?
>>
>> thanks!
>>

Mime
View raw message