spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: Spark on YARN memory utilization
Date Tue, 09 Dec 2014 07:15:45 GMT
Another thing to be aware of is that YARN will round up containers to the
nearest increment of yarn.scheduler.minimum-allocation-mb, which defaults
to 1024.

-Sandy

On Sat, Dec 6, 2014 at 3:48 PM, Denny Lee <denny.g.lee@gmail.com> wrote:

> Got it - thanks!
>
> On Sat, Dec 6, 2014 at 14:56 Arun Ahuja <aahuja11@gmail.com> wrote:
>
>> Hi Denny,
>>
>> This is due the spark.yarn.memoryOverhead parameter, depending on what
>> version of Spark you are on the default of this may differ, but it should
>> be the larger of 1024mb per executor or .07 * executorMemory.
>>
>> When you set executor memory, the yarn resource request is executorMemory
>> + yarnOverhead.
>>
>> - Arun
>>
>> On Sat, Dec 6, 2014 at 4:27 PM, Denny Lee <denny.g.lee@gmail.com> wrote:
>>
>>> This is perhaps more of a YARN question than a Spark question but i was
>>> just curious to how is memory allocated in YARN via the various
>>> configurations.  For example, if I spin up my cluster with 4GB with a
>>> different number of executors as noted below
>>>
>>>  4GB executor-memory x 10 executors = 46GB  (4GB x 10 = 40 + 6)
>>>  4GB executor-memory x 4 executors = 19GB (4GB x 4 = 16 + 3)
>>>  4GB executor-memory x 2 executors = 10GB (4GB x 2 = 8 + 2)
>>>
>>> The pattern when observing the RM is that there is a container for each
>>> executor and one additional container.  From the basis of memory, it looks
>>> like there is an additional (1GB + (0.5GB x # executors)) that is allocated
>>> in YARN.
>>>
>>> Just wondering why is this  - or is this just an artifact of YARN itself?
>>>
>>> Thanks!
>>>
>>>
>>

Mime
View raw message