spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jem Tucker <jem.tuc...@gmail.com>
Subject Re: Spark on YARN
Date Mon, 10 Aug 2015 07:24:49 GMT
Hi,

I have looked at the UI scheduler tab and it appears my new user was
allocated less cores than my other user, is there any way i can avoid this
happening?

Thanks,

Jem

On Sat, Aug 8, 2015 at 8:32 PM Shushant Arora <shushantarora09@gmail.com>
wrote:

> which is the scheduler on your cluster. Just check on RM UI scheduler tab
> and see your user and max limit of vcores for that user , is currently
> other applications of that user have occupies till max vcores of this user
> then that could be the reason of not allocating vcores to this user but for
> some other user  same applicatin is getting run since another user's max
> vcore limit is not reached.
>
> On Sat, Aug 8, 2015 at 10:07 PM, Jem Tucker <jem.tucker@gmail.com> wrote:
>
>> Hi dustin,
>>
>> Yes there are enough resources available, the same application run with a
>> different user works fine so I think it is something to do with permissions
>> but I can't work out where.
>>
>> Thanks ,
>>
>> Jem
>>
>> On Sat, 8 Aug 2015 at 17:35, Dustin Cote <dcote@cloudera.com> wrote:
>>
>>> Hi Jem,
>>>
>>> In the top of the RM web UI, do you see any available resources to spawn
>>> the application master container?
>>>
>>>
>>> On Sat, Aug 8, 2015 at 4:37 AM, Jem Tucker <jem.tucker@gmail.com> wrote:
>>>
>>>> Hi Sandy,
>>>>
>>>> The application doesn't fail, it gets accepted by yarn but the
>>>> application master never starts and the application state never changes to
>>>> running. I have checked in the resource manager and node manager logs and
>>>> nothing jumps out.
>>>>
>>>> Thanks
>>>>
>>>> Jem
>>>
>>>
>>>> On Sat, 8 Aug 2015 at 09:20, Sandy Ryza <sandy.ryza@cloudera.com>
>>>> wrote:
>>>>
>>>>> Hi Jem,
>>>>>
>>>>> Do they fail with any particular exception?  Does YARN just never end
>>>>> up giving them resources?  Does an application master start?  If so,
what
>>>>> are in its logs?  If not, anything suspicious in the YARN ResourceManager
>>>>> logs?
>>>>>
>>>>> -Sandy
>>>>>
>>>>> On Fri, Aug 7, 2015 at 1:48 AM, Jem Tucker <jem.tucker@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am running spark on YARN on the CDH5.3.2 stack. I have created
a
>>>>>> new user to own and run a testing environment, however when using
this user
>>>>>> applications I submit to yarn never begin to run, even if they are
the
>>>>>> exact same application that is successful with another user?
>>>>>>
>>>>>> Has anyone seen anything like this before?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Jem
>>>>>>
>>>>>
>>>>> --
>>>>
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "CDH Users" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to cdh-user+unsubscribe@cloudera.org.
>>>> For more options, visit
>>>> https://groups.google.com/a/cloudera.org/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> Dustin Cote
>>> Customer Operations Engineer
>>> <http://www.cloudera.com>
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to cdh-user+unsubscribe@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout.
>>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to cdh-user+unsubscribe@cloudera.org.
>> For more options, visit https://groups.google.com/a/cloudera.org/d/optout
>> .
>>
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscribe@cloudera.org.
> For more options, visit https://groups.google.com/a/cloudera.org/d/optout.
>

Mime
View raw message