spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Marscher <rmarsc...@localytics.com>
Subject Re: Executor memory allocations
Date Thu, 18 Jun 2015 14:07:35 GMT
It would be the "40%", although it's probably better to think of it as
shuffle vs. data cache and the remainder goes to tasks. As the comments for
the shuffle memory fraction configuration clarify that it will be taking
memory at the expense of the storage/data cache fraction:

spark.shuffle.memoryFraction0.2Fraction of Java heap to use for aggregation
and cogroups during shuffles, ifspark.shuffle.spill is true. At any given
time, the collective size of all in-memory maps used for shuffles is
bounded by this limit, beyond which the contents will begin to spill to
disk. If spills are often, consider increasing this value at the expense of
spark.storage.memoryFraction.

On Wed, Jun 17, 2015 at 6:02 PM, Corey Nolet <cjnolet@gmail.com> wrote:

> So I've seen in the documentation that (after the overhead memory is
> subtracted), the memory allocations of each executor are as follows (assume
> default settings):
>
> 60% for cache
> 40% for tasks to process data
>
>
> Reading about how Spark implements shuffling, I've also seen it say "20%
> of executor memory is utilized for shuffles" Does this 20% cut into the 40%
> for tasks to process data or the 60% for the data cache?
>

Mime
View raw message