spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wxhsdp <>
Subject Re: details about event log
Date Tue, 13 May 2014 01:04:08 GMT
thank you very much, Andrew

by the difinition of "Fetch Wait Time", can i make a conclusion that task
pipelines block fetch and
job doing?

Andrew Or-2 wrote
> Hi wxhsdp,
> These times are computed from Java's System.currentTimeMillis(), which is
> "the
> difference, measured in milliseconds, between the current time and
> midnight, January 1, 1970 UTC." Thus, this quantity doesn't mean much by
> itself, but is only meaningful when you subtract it from another
> System.currentTimeMillis() to find the time elapsed. For instance, in your
> case (Finish Time - Launch Time) = 1862, which means it took 1862 ms for
> the task to complete (but the actual execution only took 1781 ms, the rest
> being overhead).
> Correct, (Shuffle Finish Time - Launch Time) is the total time it took for
> this task to fetch blocks, and (Finish Time - Shuffle Finish Time) is the
> actual execution after fetching all blocks.
> "Fetch Wait Time" on the other hand is the time spent blocking on the
> thread to wait for shuffle blocks while not doing anything else. For
> instance, the example given in the code comments is: "if block B is being
> fetched while the task is not finished with processing block A, it is not
> considered to be blocking on block B."
> "Shuffle Write Time" is the time written to write the shuffle files (only
> for ShuffleMapTask). It is in nanoseconds, which is slightly inconsistent
> with other values in these metrics.
> By the way, all of this information is available in the code comments:
> On Tue, May 6, 2014 at 11:10 PM, wxhsdp &lt;

> wxhsdp@

> &gt; wrote:
>> any ideas?  thanks!
>> --
>> View this message in context:
>> Sent from the Apache Spark User List mailing list archive at

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message