spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lantao Jin <jinlan...@gmail.com>
Subject Re: Spark application shows success but lots of tasks are skipped in UI
Date Thu, 24 Nov 2016 03:56:30 GMT
But for some cases like below, the AM tell us the application is succeed,
but Spark UI (history) shows there still contains active jobs.


​
And in driver log, we can still find the job finish logs:
2016-11-23 07:46:05 INFO  DAGScheduler:54 - ResultStage 9 (sql at
Transform.scala:90) finished in 103.496 s
2016-11-23 07:46:05 INFO  TaskSetManager:54 - Finished task 3495.0 in stage
9.0 (TID 24294) in 101996 ms on xxxhost (3600/3600)
2016-11-23 07:46:05 INFO  YarnScheduler:54 - Removed TaskSet 9.0, whose
tasks have all completed, from pool
2016-11-23 07:46:05 INFO  DAGScheduler:54 - Job 3 finished: sql at
Transform.scala:90, took 1079.033903 s
2016-11-23 07:46:19 INFO  DefaultWriterContainer:54 - Job
job_201611230726_0000 committed.
​

It's very strange.

==
Alan

2016-11-23 22:32 GMT+08:00 Lantao Jin <jinlantao@gmail.com>:

> Look like this is the answer:
> https://github.com/apache/spark/pull/3009
> So that the answer is SUCCEED, right?
>
> 2016-11-23 21:45 GMT+08:00 Lantao Jin <jinlantao@gmail.com>:
>
>> Hi all,
>> I am confusing which is the final status of the Spark job, SUCCESS or
>> FAILED.
>> We can see there are 4 completed jobs in "Completed jobs" but in Stage
>> tab, it still contains 4 active stages. The application is finished because
>> below graphs were displayed in History Server UI.
>>
>>
>> ​
>> ​And the status of application in Yarn is also succeed. But in UI, lots
>> of tasks were skipped.
>>
>> And in the driver log, below sentences can be found:
>> 2016-11-23 05:03:54 INFO  TaskSetManager:54 - Finished task 1013.0 in
>> stage 9.0 (TID 21796) in 201883 ms on lvsaishdc3dn1589.stratus.lvs.e
>> bay.com (3600/3600)
>> 2016-11-23 05:03:54 INFO  YarnScheduler:54 - Removed TaskSet 9.0, whose
>> tasks have all completed, from pool
>> 2016-11-23 05:03:54 INFO  DAGScheduler:54 - ResultStage 9 (sql at
>> Transform.scala:90) finished in 202.679 s
>> 2016-11-23 05:03:54 INFO  DAGScheduler:54 - Job 3 finished: sql at
>> Transform.scala:90, took 777.949213 s
>> 2016-11-23 05:04:09 INFO  DefaultWriterContainer:54 - Job
>> job_201611230448_0000 committed.
>>
>> But also at the end of log, we can find some ERROR logs:
>> 2016-11-23 05:04:12 ERROR LiveListenerBus:70 - SparkListenerBus has
>> already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdated
>> Info(BlockManagerId(753, lvsaishc3dn1235.stratus.lvs.ebay.com,
>> 43988),broadcast_1_piece0,StorageLevel(memory, 1 replicas),43198,0))
>> 2016-11-23 05:04:12 INFO  MemoryStore:54 - MemoryStore cleared
>> 2016-11-23 05:04:12 INFO  BlockManager:54 - BlockManager stopped
>>
>> SO, HOW to determine the final status of a Spark job?
>>
>> ==
>> Best regards,
>> Alan
>>
>>
>

Mime
View raw message