spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yohann jardin <yohannjar...@hotmail.com>
Subject Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
Date Sun, 08 Jul 2018 15:02:34 GMT
Following the logs from the resource manager:

2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue, it is
likely set too low. skipping enforcement to allow at least one application to start

2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue for user,
it is likely set too low. skipping enforcement to allow at least one application to start

I’d say it has nothing to do with spark. Your master is just asking more resources than
the default Yarn queue is allowed to provide.
You might take a look at https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
and search for maximum-am-resource-percent.

Regards,

Yohann Jardin

Le 7/8/2018 à 4:40 PM, kant kodali a écrit :
Hi,

It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 vCpu! I am not running
any code since I can't even spawn spark-shell with yarn as master as described in my previous
email. I just want to run simple word count using yarn as master.

Thanks!

Below is the resource manager log once again if that helps


2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Application added - appId: application_1531059242261_0001 user: xxx leaf-queue of parent:
root #applications: 1

2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Accepted application application_1531059242261_0001 from user: xxx, in queue: default

2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1531059242261_0001 State change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED

2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Registering app attempt : appattempt_1531059242261_0001_000001

2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1531059242261_0001_000001 State change from NEW to SUBMITTED

2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue, it is
likely set too low. skipping enforcement to allow at least one application to start

2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue for user,
it is likely set too low. skipping enforcement to allow at least one application to start

2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application application_1531059242261_0001 from user: xxx activated in queue: default

2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application added - appId: application_1531059242261_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@476750cd,
leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications:
0 #queue-active-applications: 1

2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Added Application Attempt appattempt_1531059242261_0001_000001 to scheduler from user xxx
in queue default

2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1531059242261_0001_000001 State change from SUBMITTED to SCHEDULED



Mime
View raw message