spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Memory configuration of standalone clusters
Date Wed, 04 Dec 2013 20:42:30 GMT
Hello,

I have a few questions about configuring memory usage on standalone
clusters.  Can someone help me out?

1) The terms "slave" in ./bin/start-slaves.sh and "worker" in the docs seem
to be used interchangeably.  Are they the same?

2) On a worker/slave, is there only one JVM running that has all the data
in it, or is there a separate JVM spun up for each application (Hadoop
style)?  Ignoring the SPARK_WORKER_INSTANCES setting.

3) There are lots of configuration options for defining memory usage.  What
do they all mean?  Is my below summary correct?
a) SPARK_WORKER_MEMORY -- maximum amount of memory that the Spark worker
will ever use, regardless of applications started.  This sets the maximum
heap size of the Spark worker JVM
b) ./bin/start-worker.sh --memory -- same as SPARK_WORKER_MEMORY?  If both
are set, which takes priority?
c) SPARK_JAVA_OPTS="-Xmx512m -Xms512m" -- same as (a) and (b)?
d) -Dspark.executor.memory -- maximum amount of memory this particular
application will use on a single worker

If that's correct I'll send a PR to the docs that would have clarified
these for me.

http://spark.incubator.apache.org/docs/latest/configuration.html#system-properties
http://spark.incubator.apache.org/docs/latest/spark-standalone.html#cluster-launch-scripts


4) If I set SPARK_WORKER_MEMORY really high, I think I then also have to
set spark.executor.memory really high to take advantage of it (since
spark.executor.memory is 512m by default).  Is there a way to optimize a
cluster for the "one big application" scenario better than manually keeping
these in sync?  I'd like spark.executor.memory to match SPARK_WORKER_MEMORY
if it's not set, I think.


Thanks!
Andrew

Mime
View raw message