spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Spark 1.3 Dynamic Allocation - Requesting 0 new executor(s) because tasks are backlogged
Date Sat, 21 Mar 2015 13:51:48 GMT
bq. Requesting 1 new executor(s) because tasks are backlogged

1 executor was requested.

Which hadoop release are you using ?

Can you check resource manager log to see if there is some clue ?

Thanks

On Fri, Mar 20, 2015 at 4:17 PM, Manoj Samel <manojsameltech@gmail.com>
wrote:

> Forgot to add - the cluster is idle otherwise so there should be no
> resource issues. Also the configuration works when not using Dynamic
> allocation.
>
> On Fri, Mar 20, 2015 at 4:15 PM, Manoj Samel <manojsameltech@gmail.com>
> wrote:
>
>> Hi,
>>
>> Running Spark 1.3 with secured Hadoop.
>>
>> Spark-shell with Yarn client mode runs without issue when not using
>> Dynamic Allocation.
>>
>> When Dynamic allocation is turned on, the shell comes up but same SQL
>> etc. causes it to loop.
>>
>> spark.dynamicAllocation.enabled=true
>> spark.dynamicAllocation.initialExecutors=1
>> spark.dynamicAllocation.maxExecutors=10
>> # Set IdleTime low for testing
>> spark.dynamicAllocation.executorIdleTimeout=60
>> spark.shuffle.service.enabled=true
>>
>> Following is the start of the messages and then it keeps looping with
>> "Requesting 0 new executors"
>>
>> 15/03/20 22:52:42 INFO storage.BlockManagerMaster: Updated info of block
>> broadcast_1_piece0
>> 15/03/20 22:52:42 INFO spark.SparkContext: Created broadcast 1 from
>> broadcast at DAGScheduler.scala:839
>> 15/03/20 22:52:42 INFO scheduler.DAGScheduler: Submitting 1 missing tasks
>> from Stage 0 (MapPartitionsRDD[3] at mapPartitions at Exchange.scala:100)
>> 15/03/20 22:52:42 INFO cluster.YarnScheduler: Adding task set 0.0 with 1
>> tasks
>> 15/03/20 22:52:47 INFO spark.ExecutorAllocationManager: Requesting 1 new
>> executor(s) because tasks are backlogged (new desired total will be 1)
>> 15/03/20 22:52:52 INFO spark.ExecutorAllocationManager: Requesting 0 new
>> executor(s) because tasks are backlogged (new desired total will be 1)
>> 15/03/20 22:52:57 WARN cluster.YarnScheduler: Initial job has not
>> accepted any resources; check your cluster UI to ensure that workers are
>> registered and have sufficient resources
>> 15/03/20 22:52:57 INFO spark.ExecutorAllocationManager: Requesting 0 new
>> executor(s) because tasks are backlogged (new desired total will be 1)
>> 15/03/20 22:53:02 INFO spark.ExecutorAllocationManager: Requesting 0 new
>> executor(s) because tasks are backlogged (new desired total will be 1)
>> 15/03/20 22:53:07 INFO spark.ExecutorAllocationManager: Requesting 0 new
>> executor(s) because tasks are backlogged (new desired total will be 1)
>> 15/03/20 22:53:12 INFO spark.ExecutorAllocationManager: Requesting 0 new
>> executor(s) because tasks are backlogged (new desired total will be 1)
>> 15/03/20 22:53:12 WARN cluster.YarnScheduler: Initial job has not
>> accepted any resources; check your cluster UI to ensure that workers are
>> registered and have sufficient resources
>> 15/03/20 22:53:17 INFO spark.ExecutorAllocationManager: Requesting 0 new
>> executor(s) because tasks are backlogged (new desired total will be 1)
>> 15/03/20 22:53:22 INFO spark.ExecutorAllocationManager: Requesting 0 new
>> executor(s) because tasks are backlogged (new desired total will be 1)
>> 15/03/20 22:53:27 INFO spark.ExecutorAllocationManager: Requesting 0 new
>> executor(s) because tasks are backlogged (new desired total will be 1)
>>
>
>

Mime
View raw message