spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Takeshi Yamamuro <linguin....@gmail.com>
Subject Re: Tuning Spark memory
Date Sat, 24 Sep 2016 03:55:41 GMT
Hi,

Currently, the memory fraction of shuffle and storage is automatically
tuned by a memory manager.
So, you do not need to care the parameter in most cases.
See
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L24

// maropu


On Fri, Sep 23, 2016 at 9:06 PM, tan shai <tan.shai284@gmail.com> wrote:

> Hi,
>
> I am working with Spark 2.0, the job starts by sorting the input data and
> storing the output on HDFS.
>
> I am getting Out of memory errors, the solution was to increase the value
> of spark.shuffle.memoryFraction from 0.2 to 0.8 and this solves the
> problem. But in the documentation I have found that this is a deprecated
> parameter.
>
> As I have understand, It was replaced by spark.memory.fraction. How to
> modify this parameter while taking into account the sort and storage on
> HDFS?
>
> Thanks.
>



-- 
---
Takeshi Yamamuro

Mime
View raw message