spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denny Lee <denny.g....@gmail.com>
Subject Re: Spark on YARN memory utilization
Date Sat, 06 Dec 2014 23:48:23 GMT
Got it - thanks!
On Sat, Dec 6, 2014 at 14:56 Arun Ahuja <aahuja11@gmail.com> wrote:

> Hi Denny,
>
> This is due the spark.yarn.memoryOverhead parameter, depending on what
> version of Spark you are on the default of this may differ, but it should
> be the larger of 1024mb per executor or .07 * executorMemory.
>
> When you set executor memory, the yarn resource request is executorMemory
> + yarnOverhead.
>
> - Arun
>
> On Sat, Dec 6, 2014 at 4:27 PM, Denny Lee <denny.g.lee@gmail.com> wrote:
>
>> This is perhaps more of a YARN question than a Spark question but i was
>> just curious to how is memory allocated in YARN via the various
>> configurations.  For example, if I spin up my cluster with 4GB with a
>> different number of executors as noted below
>>
>>  4GB executor-memory x 10 executors = 46GB  (4GB x 10 = 40 + 6)
>>  4GB executor-memory x 4 executors = 19GB (4GB x 4 = 16 + 3)
>>  4GB executor-memory x 2 executors = 10GB (4GB x 2 = 8 + 2)
>>
>> The pattern when observing the RM is that there is a container for each
>> executor and one additional container.  From the basis of memory, it looks
>> like there is an additional (1GB + (0.5GB x # executors)) that is allocated
>> in YARN.
>>
>> Just wondering why is this  - or is this just an artifact of YARN itself?
>>
>> Thanks!
>>
>>
>

Mime
View raw message