spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject little confused about SPARK_JAVA_OPTS alternatives
Date Wed, 14 May 2014 16:09:46 GMT
i have some settings that i think are relevant for my application. they are
spark.akka settings so i assume they are relevant for both executors and my
driver program.

i used to do:
SPARK_JAVA_OPTS="-Dspark.akka.frameSize=10000"

now this is deprecated. the alternatives mentioned are:
* some spark-submit settings which are not relevant to me since i do not
use spark-submit (i launch spark jobs from an existing application)
* spark.executor.extraJavaOptions to set -X options. i am not sure what -X
options are, but it doesnt sound like what i need, since its only for
executors
* SPARK_DAEMON_OPTS to set java options for standalone daemons (i.e.
master, worker), that sounds like i should not use it since i am trying to
change settings for an app, not a daemon.

am i missing the correct setting to use?
should i do -Dspark.akka.frameSize=10000 on my application launch directly,
and then also set spark.executor.extraJavaOptions? so basically repeat it?

Mime
View raw message