spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <and...@databricks.com>
Subject Re: details about event log
Date Thu, 08 May 2014 23:40:47 GMT
Hi wxhsdp,

These times are computed from Java's System.currentTimeMillis(), which is "the
difference, measured in milliseconds, between the current time and
midnight, January 1, 1970 UTC." Thus, this quantity doesn't mean much by
itself, but is only meaningful when you subtract it from another
System.currentTimeMillis() to find the time elapsed. For instance, in your
case (Finish Time - Launch Time) = 1862, which means it took 1862 ms for
the task to complete (but the actual execution only took 1781 ms, the rest
being overhead).

Correct, (Shuffle Finish Time - Launch Time) is the total time it took for
this task to fetch blocks, and (Finish Time - Shuffle Finish Time) is the
actual execution after fetching all blocks.

"Fetch Wait Time" on the other hand is the time spent blocking on the
thread to wait for shuffle blocks while not doing anything else. For
instance, the example given in the code comments is: "if block B is being
fetched while the task is not finished with processing block A, it is not
considered to be blocking on block B."

"Shuffle Write Time" is the time written to write the shuffle files (only
for ShuffleMapTask). It is in nanoseconds, which is slightly inconsistent
with other values in these metrics.

By the way, all of this information is available in the code comments:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala




On Tue, May 6, 2014 at 11:10 PM, wxhsdp <wxhsdp@gmail.com> wrote:

> any ideas?  thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/details-about-event-log-tp5411p5476.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Mime
View raw message