spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mrm <>
Subject Re: driver memory
Date Wed, 23 Jul 2014 11:18:34 GMT

I figured out my problem so I wanted to share my findings. I was basically
trying to broadcast an array with 4 million elements, and a size of
approximatively 150 MB. Every time I was trying to broadcast, I got an
OutOfMemory error. I fixed my problem by increasing the driver memory using:
export SPARK_MEM="2g"

Using SPARK_DAEMON_MEM or spark.executor.memory did not help in this case! I
don't have a good understanding of all these settings and I have the feeling
many people are in the same situation. 

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message