spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yohann jardin <yohannjar...@hotmail.com>
Subject Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
Date Sun, 08 Jul 2018 17:27:16 GMT
When you run on Yarn, you don’t even need to start a spark cluster (spark master and slaves).
Yarn receives a job and then allocate resources for the application master and then its workers.

Check the resources available in the node section of the resource manager UI (and is your
node actually detected as alive?), as well as the scheduler section to check the default queue
resources.
If you seem to lack resources for your driver, you can try to reduce the driver memory by
specifying “--driver-memory 512” for example, but I’d expect the default of 1g to be
low enough based on what you showed us.

Yohann Jardin

Le 7/8/2018 à 6:11 PM, kant kodali a écrit :
@yohann sorry I am assuming you meant application master if so I believe spark is the one
that provides application master. Is there anyway to look for how much resources are being
requested and how much yarn is allowed to provide? I would assume this is a common case if
so I am not sure why these numbers are not part of resource manager logs?

On Sun, Jul 8, 2018 at 8:09 AM, kant kodali <kanth909@gmail.com<mailto:kanth909@gmail.com>>
wrote:
yarn.scheduler.capacity.maximum-am-resource-percent by default is set to 0.1 and I tried changing
it to 1.0 and still no luck. same problem persists. The master here is yarn and I just trying
to spawn spark-shell --master yarn --deploy-mode client and run a simple world count so I
am not sure why it would request for more resources?

On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin <yohannjardin@hotmail.com<mailto:yohannjardin@hotmail.com>>
wrote:

Following the logs from the resource manager:

2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue, it is
likely set too low. skipping enforcement to allow at least one application to start

2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue for user,
it is likely set too low. skipping enforcement to allow at least one application to start

I’d say it has nothing to do with spark. Your master is just asking more resources than
the default Yarn queue is allowed to provide.
You might take a look at https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
and search for maximum-am-resource-percent.

Regards,

Yohann Jardin

Le 7/8/2018 à 4:40 PM, kant kodali a écrit :
Hi,

It's on local mac book pro machine that has 16GB RAM 512GB disk and 8 vCpu! I am not running
any code since I can't even spawn spark-shell with yarn as master as described in my previous
email. I just want to run simple word count using yarn as master.

Thanks!

Below is the resource manager log once again if that helps


2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
Application added - appId: application_1531059242261_0001 user: xxx leaf-queue of parent:
root #applications: 1

2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Accepted application application_1531059242261_0001 from user: xxx, in queue: default

2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1531059242261_0001 State change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED

2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
Registering app attempt : appattempt_1531059242261_0001_000001

2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1531059242261_0001_000001 State change from NEW to SUBMITTED

2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue, it is
likely set too low. skipping enforcement to allow at least one application to start

2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
maximum-am-resource-percent is insufficient to start a single application in queue for user,
it is likely set too low. skipping enforcement to allow at least one application to start

2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application application_1531059242261_0001 from user: xxx activated in queue: default

2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
Application added - appId: application_1531059242261_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@476750cd,
leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications:
0 #queue-active-applications: 1

2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Added Application Attempt appattempt_1531059242261_0001_000001 to scheduler from user xxx
in queue default

2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1531059242261_0001_000001 State change from SUBMITTED to SCHEDULED






Mime
View raw message