spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
Date Sun, 08 Jul 2018 15:09:45 GMT
yarn.scheduler.capacity.maximum-am-resource-percent by default is set to
0.1 and I tried changing it to 1.0 and still no luck. same problem
persists. The master here is yarn and I just trying to spawn spark-shell
--master yarn --deploy-mode client and run a simple world count so I am not
sure why it would request for more resources?

On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin <yohannjardin@hotmail.com>
wrote:

> Following the logs from the resource manager:
>
> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent
> is insufficient to start a single application in queue, it is likely set
> too low. skipping enforcement to allow at least one application to start
>
> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent
> is insufficient to start a single application in queue for user, it is
> likely set too low. skipping enforcement to allow at least one
> application to start
>
> I’d say it has nothing to do with spark. Your master is just asking more
> resources than the default Yarn queue is allowed to provide.
> You might take a look at https://hadoop.apache.org/
> docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and
> search for maximum-am-resource-percent.
>
> Regards,
>
> *Yohann Jardin*
> Le 7/8/2018 à 4:40 PM, kant kodali a écrit :
>
> Hi,
>
> It's on local mac book pro machine that has 16GB RAM 512GB disk and 8
> vCpu! I am not running any code since I can't even spawn spark-shell with
> yarn as master as described in my previous email. I just want to run simple
> word count using yarn as master.
>
> Thanks!
>
> Below is the resource manager log once again if that helps
>
>
> 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.ParentQueue: Application added -
> appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root #applications:
> 1
>
> 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.CapacityScheduler: Accepted
> application application_1531059242261_0001 from user: xxx, in queue:
> default
>
> 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server.
> resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State
> change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED
>
> 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.
> resourcemanager.ApplicationMasterService: Registering app attempt :
> appattempt_1531059242261_0001_000001
>
> 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.
> resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_1531059242261_0001_000001 State change from NEW to SUBMITTED
>
> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent
> is insufficient to start a single application in queue, it is likely set
> too low. skipping enforcement to allow at least one application to start
>
> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.LeafQueue: maximum-am-resource-percent
> is insufficient to start a single application in queue for user, it is
> likely set too low. skipping enforcement to allow at least one
> application to start
>
> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.LeafQueue: Application
> application_1531059242261_0001 from user: xxx activated in queue: default
>
> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.LeafQueue: Application added - appId:
> application_1531059242261_0001 user: org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue:
> default #user-pending-applications: 0 #user-active-applications: 1
> #queue-pending-applications: 0 #queue-active-applications: 1
>
> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.
> resourcemanager.scheduler.capacity.CapacityScheduler: Added Application
> Attempt appattempt_1531059242261_0001_000001 to scheduler from user xxx in
> queue default
>
> 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server.
> resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_1531059242261_0001_000001 State change from SUBMITTED to
> SCHEDULED
>
>
>
>

Mime
View raw message