spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wxhsdp <wxh...@gmail.com>
Subject Re: details about event log
Date Tue, 13 May 2014 01:04:08 GMT
thank you very much, Andrew

by the difinition of "Fetch Wait Time", can i make a conclusion that task
pipelines block fetch and
job doing?


Andrew Or-2 wrote
> Hi wxhsdp,
> 
> These times are computed from Java's System.currentTimeMillis(), which is
> "the
> difference, measured in milliseconds, between the current time and
> midnight, January 1, 1970 UTC." Thus, this quantity doesn't mean much by
> itself, but is only meaningful when you subtract it from another
> System.currentTimeMillis() to find the time elapsed. For instance, in your
> case (Finish Time - Launch Time) = 1862, which means it took 1862 ms for
> the task to complete (but the actual execution only took 1781 ms, the rest
> being overhead).
> 
> Correct, (Shuffle Finish Time - Launch Time) is the total time it took for
> this task to fetch blocks, and (Finish Time - Shuffle Finish Time) is the
> actual execution after fetching all blocks.
> 
> "Fetch Wait Time" on the other hand is the time spent blocking on the
> thread to wait for shuffle blocks while not doing anything else. For
> instance, the example given in the code comments is: "if block B is being
> fetched while the task is not finished with processing block A, it is not
> considered to be blocking on block B."
> 
> "Shuffle Write Time" is the time written to write the shuffle files (only
> for ShuffleMapTask). It is in nanoseconds, which is slightly inconsistent
> with other values in these metrics.
> 
> By the way, all of this information is available in the code comments:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
> 
> 
> 
> 
> On Tue, May 6, 2014 at 11:10 PM, wxhsdp &lt;

> wxhsdp@

> &gt; wrote:
> 
>> any ideas?  thanks!
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/details-about-event-log-tp5411p5476.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/details-about-event-log-tp5411p5624.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message