spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Muler <mulugeta.abe...@gmail.com>
Subject Standalone mode: is SPARK_WORKER_MEMORY per SPARK_WORKER_INSTANCE?
Date Mon, 31 Aug 2015 08:06:06 GMT
Hi,

Is SPARK_WORKER_MEMORY defined per SPARK_WORKER_INSTANCE (just like
SPARK_WORKER_CORES, the documentation is clear for cores)? Or is it a per
node?

For example, if I have 80g of available memory and I start my Spark app
with SPARK_WORKER_INSTANCES=8 (i.e. 8 worker jvms), then does
SPARK_WORKER_MEMORY represent the memory that each of these 8 worker JVMs
can give out to their executor JVMs? If so, then the appropriate value
would be SPARK_WORKER_MEMORY=80g/8=10g?

Thanks,

Mime
View raw message