spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fei Hu <>
Subject Spark application Runtime Measurement
Date Sun, 10 Jul 2016 04:03:05 GMT
Dear all,

I have a question about how to measure the runtime for a Spak application.
Here is an example:

   - On the Spark UI: the total duration time is 2.0 minutes = 120 seconds
   as following

[image: Screen Shot 2016-07-09 at 11.45.44 PM.png]

   - However, when I check the jobs launched by the application, the time
   is 13s + 0.8s + 4s = 17.8 seconds, which is much less than 120 seconds. I
   am not sure which time I should choose to measure the performance of the
   Spark application.

[image: Screen Shot 2016-07-09 at 11.48.26 PM.png]

   - I also check the event timeline as following. There is a big gap
   between the second job and the third job. I do not know what happened
   during that gap.

[image: Screen Shot 2016-07-09 at 11.53.29 PM.png]

Is there anyone who can help explain which time is the exact time to
measure the performance of a Spark application.

Thanks in advance,

View raw message