spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From amine_901 <>
Subject Job spark blocked and runs indefinitely
Date Wed, 11 Oct 2017 12:37:18 GMT
We encounter a problem on a Spark job 1.6(on yarn) that never ends, whene
several jobs launched simultaneously.
We found that by launching the job spark in yarn-client mode we do not have
this problem, unlike launching it in yarn-cluster mode.
it could be a trail to find the cause.

we changed the code to add a sparkContext.stop ()
Indeed, the SparkContext was created (val sparkContext = createSparkContext)
but not stopped. this solution has allowed us to decrease the number of jobs
that remains blocked but nevertheless we still have some jobs blocked.

by analyzing the logs we have found this log that repeats without stopping:
/17/09/29 11:04:37 DEBUG SparkEventPublisher: Enqueue
17/09/29 11:04:41 DEBUG ApplicationMaster: Sending progress
17/09/29 11:04:41 DEBUG ApplicationMaster: Number of pending allocations is
0. Sleeping for 5000. /

do someone has any idea about this issue ? 
Thank you in advance

Sent from:

To unsubscribe e-mail:

View raw message