spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lantao Jin <jinlan...@gmail.com>
Subject Re: Spark application shows success but lots of tasks are skipped in UI
Date Wed, 23 Nov 2016 14:32:17 GMT
Look like this is the answer:
https://github.com/apache/spark/pull/3009
So that the answer is SUCCEED, right?

2016-11-23 21:45 GMT+08:00 Lantao Jin <jinlantao@gmail.com>:

> Hi all,
> I am confusing which is the final status of the Spark job, SUCCESS or
> FAILED.
> We can see there are 4 completed jobs in "Completed jobs" but in Stage
> tab, it still contains 4 active stages. The application is finished because
> below graphs were displayed in History Server UI.
>
>
> ​
> ​And the status of application in Yarn is also succeed. But in UI, lots of
> tasks were skipped.
>
> And in the driver log, below sentences can be found:
> 2016-11-23 05:03:54 INFO  TaskSetManager:54 - Finished task 1013.0 in
> stage 9.0 (TID 21796) in 201883 ms on lvsaishdc3dn1589.stratus.lvs.e
> bay.com (3600/3600)
> 2016-11-23 05:03:54 INFO  YarnScheduler:54 - Removed TaskSet 9.0, whose
> tasks have all completed, from pool
> 2016-11-23 05:03:54 INFO  DAGScheduler:54 - ResultStage 9 (sql at
> Transform.scala:90) finished in 202.679 s
> 2016-11-23 05:03:54 INFO  DAGScheduler:54 - Job 3 finished: sql at
> Transform.scala:90, took 777.949213 s
> 2016-11-23 05:04:09 INFO  DefaultWriterContainer:54 - Job
> job_201611230448_0000 committed.
>
> But also at the end of log, we can find some ERROR logs:
> 2016-11-23 05:04:12 ERROR LiveListenerBus:70 - SparkListenerBus has
> already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdated
> Info(BlockManagerId(753, lvsaishc3dn1235.stratus.lvs.ebay.com,
> 43988),broadcast_1_piece0,StorageLevel(memory, 1 replicas),43198,0))
> 2016-11-23 05:04:12 INFO  MemoryStore:54 - MemoryStore cleared
> 2016-11-23 05:04:12 INFO  BlockManager:54 - BlockManager stopped
>
> SO, HOW to determine the final status of a Spark job?
>
> ==
> Best regards,
> Alan
>
>

Mime
View raw message