spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: spark-shell gets stuck in ACCEPTED state forever when ran in YARN client mode.
Date Mon, 09 Jul 2018 02:36:06 GMT
@yohann Thanks for shining some light! It is making more sense now.

I think you are correct when you stated: "Your application master is just
asking for more resources than the default Yarn queue is allowed to provide
".

Attached are the screenshots of the UI pages you mentioned. The thing that
catches my eye is the default queue resources under scheduler section. It
has the following

The default queue as listed in the screenshot has the following


Max Application Master Resources: <memory:0, vCores:0>
Used Application Master Resources: <memory:1024, vCores:1>

is this why my spark-shell gets stuck in ACCEPTED stated forever? I am
pretty much using the default config so is there a config I should add to
set the Max Application Master Resources?

Thanks!





On Sun, Jul 8, 2018 at 10:27 AM, yohann jardin <yohannjardin@hotmail.com>
wrote:

> When you run on Yarn, you don’t even need to start a spark cluster (spark
> master and slaves). Yarn receives a job and then allocate resources for the
> application master and then its workers.
>
> Check the resources available in the node section of the resource manager
> UI (and is your node actually detected as alive?), as well as the scheduler
> section to check the default queue resources.
> If you seem to lack resources for your driver, you can try to reduce the
> driver memory by specifying “--driver-memory 512” for example, but I’d
> expect the default of 1g to be low enough based on what you showed us.
>
> *Yohann Jardin*
> Le 7/8/2018 à 6:11 PM, kant kodali a écrit :
>
> @yohann sorry I am assuming you meant application master if so I believe
> spark is the one that provides application master. Is there anyway to look
> for how much resources are being requested and how much yarn is allowed to
> provide? I would assume this is a common case if so I am not sure why these
> numbers are not part of resource manager logs?
>
> On Sun, Jul 8, 2018 at 8:09 AM, kant kodali <kanth909@gmail.com> wrote:
>
>> yarn.scheduler.capacity.maximum-am-resource-percent by default is set to
>> 0.1 and I tried changing it to 1.0 and still no luck. same problem
>> persists. The master here is yarn and I just trying to spawn spark-shell
>> --master yarn --deploy-mode client and run a simple world count so I am not
>> sure why it would request for more resources?
>>
>> On Sun, Jul 8, 2018 at 8:02 AM, yohann jardin <yohannjardin@hotmail.com>
>> wrote:
>>
>>> Following the logs from the resource manager:
>>>
>>> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.LeafQueue:
>>> maximum-am-resource-percent is insufficient to start a single
>>> application in queue, it is likely set too low. skipping enforcement to
>>> allow at least one application to start
>>>
>>> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.LeafQueue:
>>> maximum-am-resource-percent is insufficient to start a single
>>> application in queue for user, it is likely set too low. skipping
>>> enforcement to allow at least one application to start
>>>
>>> I’d say it has nothing to do with spark. Your master is just asking more
>>> resources than the default Yarn queue is allowed to provide.
>>> You might take a look at https://hadoop.apache.org/docs
>>> /r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html and search
>>> for maximum-am-resource-percent.
>>>
>>> Regards,
>>>
>>> *Yohann Jardin*
>>> Le 7/8/2018 à 4:40 PM, kant kodali a écrit :
>>>
>>> Hi,
>>>
>>> It's on local mac book pro machine that has 16GB RAM 512GB disk and 8
>>> vCpu! I am not running any code since I can't even spawn spark-shell with
>>> yarn as master as described in my previous email. I just want to run simple
>>> word count using yarn as master.
>>>
>>> Thanks!
>>>
>>> Below is the resource manager log once again if that helps
>>>
>>>
>>> 2018-07-08 07:23:23,343 INFO org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.ParentQueue: Application added -
>>> appId: application_1531059242261_0001 user: xxx leaf-queue of parent: root #applications:
>>> 1
>>>
>>> 2018-07-08 07:23:23,344 INFO org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.CapacityScheduler: Accepted
>>> application application_1531059242261_0001 from user: xxx, in queue:
>>> default
>>>
>>> 2018-07-08 07:23:23,350 INFO org.apache.hadoop.yarn.server.
>>> resourcemanager.rmapp.RMAppImpl: application_1531059242261_0001 State
>>> change from SUBMITTED to ACCEPTED on event=APP_ACCEPTED
>>>
>>> 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.
>>> resourcemanager.ApplicationMasterService: Registering app attempt :
>>> appattempt_1531059242261_0001_000001
>>>
>>> 2018-07-08 07:23:23,370 INFO org.apache.hadoop.yarn.server.
>>> resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>>> appattempt_1531059242261_0001_000001 State change from NEW to SUBMITTED
>>>
>>> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.LeafQueue:
>>> maximum-am-resource-percent is insufficient to start a single
>>> application in queue, it is likely set too low. skipping enforcement to
>>> allow at least one application to start
>>>
>>> 2018-07-08 07:23:23,382 WARN org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.LeafQueue:
>>> maximum-am-resource-percent is insufficient to start a single
>>> application in queue for user, it is likely set too low. skipping
>>> enforcement to allow at least one application to start
>>>
>>> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.LeafQueue: Application
>>> application_1531059242261_0001 from user: xxx activated in queue:
>>> default
>>>
>>> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.LeafQueue: Application added -
>>> appId: application_1531059242261_0001 user: org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.LeafQueue$User@476750cd, leaf-queue:
>>> default #user-pending-applications: 0 #user-active-applications: 1
>>> #queue-pending-applications: 0 #queue-active-applications: 1
>>>
>>> 2018-07-08 07:23:23,382 INFO org.apache.hadoop.yarn.server.
>>> resourcemanager.scheduler.capacity.CapacityScheduler: Added Application
>>> Attempt appattempt_1531059242261_0001_000001 to scheduler from user xxx
>>> in queue default
>>>
>>> 2018-07-08 07:23:23,386 INFO org.apache.hadoop.yarn.server.
>>> resourcemanager.rmapp.attempt.RMAppAttemptImpl:
>>> appattempt_1531059242261_0001_000001 State change from SUBMITTED to
>>> SCHEDULED
>>>
>>>
>>>
>>>
>>
>
>

Mime
View raw message