spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <>
Subject Spark driver memory breakdown
Date Fri, 26 Aug 2016 16:48:35 GMT

I alwayd underestimated the significant of setting spark.driver.memory

According to documents

It is the amount of memory to use for the driver process, i.e. where
SparkContext is initialized. (e.g. 1g, 2g).

I was running my application using Spark Standalone so the argument about
Local mode and one JVM do not come into it.

As I know:

* The driver program is the main program, which coordinates the executors
to run the Spark application.*

It is not clear to me whether the driver program also allocates the memory
to executors that run on workers.

I noticed that if you leave this driver memory low you end up with heap
space issue and the job crashes. So I had to increase the driver memory
from 1G to 8G to make the job run.

So in a nutshell how this driver memory is allocated in Standalone mode
given that we also have executer memory --executor-memory that I set


Dr Mich Talebzadeh

LinkedIn *

*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

View raw message