spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianneng Li <jiannen...@berkeley.edu>
Subject Questions about Spark Shuffle and Heap
Date Fri, 04 Dec 2015 12:01:33 GMT
Hi,

On the Spark Configuration page (
http://spark.apache.org/docs/1.5.2/configuration.html), the documentation
for spark.shuffle.memoryFraction mentions that the fraction is taken from
the Java heap. However, the documentation
for spark.shuffle.io.preferDirectBufs implies that off-heap memory might be
used instead during shuffles. Do these two explanation conflict each other?

In a related question, with Tungsten enabled, when does Spark use off-heap
memory?

Thanks,

Jianneng




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Questions-about-Spark-Shuffle-and-Heap-tp25564.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message