spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lantao Jin <jinlan...@gmail.com>
Subject Spark application shows success but lots of tasks are skipped in UI
Date Wed, 23 Nov 2016 13:45:48 GMT
Hi all,
I am confusing which is the final status of the Spark job, SUCCESS or
FAILED.
We can see there are 4 completed jobs in "Completed jobs" but in Stage tab,
it still contains 4 active stages. The application is finished because
below graphs were displayed in History Server UI.


​
​And the status of application in Yarn is also succeed. But in UI, lots of
tasks were skipped.

And in the driver log, below sentences can be found:
2016-11-23 05:03:54 INFO  TaskSetManager:54 - Finished task 1013.0 in stage
9.0 (TID 21796) in 201883 ms on lvsaishdc3dn1589.stratus.lvs.e
bay.com (3600/3600)
2016-11-23 05:03:54 INFO  YarnScheduler:54 - Removed TaskSet 9.0, whose
tasks have all completed, from pool
2016-11-23 05:03:54 INFO  DAGScheduler:54 - ResultStage 9 (sql at
Transform.scala:90) finished in 202.679 s
2016-11-23 05:03:54 INFO  DAGScheduler:54 - Job 3 finished: sql at
Transform.scala:90, took 777.949213 s
2016-11-23 05:04:09 INFO  DefaultWriterContainer:54 - Job
job_201611230448_0000 committed.

But also at the end of log, we can find some ERROR logs:
2016-11-23 05:04:12 ERROR LiveListenerBus:70 - SparkListenerBus has already
stopped! Dropping event SparkListenerBlockUpdated(BlockUpdated
Info(BlockManagerId(753, lvsaishc3dn1235.stratus.lvs.ebay.com,
43988),broadcast_1_piece0,StorageLevel(memory, 1 replicas),43198,0))
2016-11-23 05:04:12 INFO  MemoryStore:54 - MemoryStore cleared
2016-11-23 05:04:12 INFO  BlockManager:54 - BlockManager stopped

SO, HOW to determine the final status of a Spark job?

==
Best regards,
Alan

Mime
View raw message