spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From MEETHU MATHEW <meethu2...@yahoo.co.in>
Subject options set in spark-env.sh is not reflecting on actual execution
Date Thu, 19 Jun 2014 05:21:13 GMT
Hi all,

I have a doubt regarding the options in spark-env.sh. I set the following values in the file
in master and 2 workers

SPARK_WORKER_MEMORY=7g
SPARK_EXECUTOR_MEMORY=6g
SPARK_DAEMON_JAVA_OPTS+="- Dspark.akka.timeout=300000 -Dspark.akka.frameSize=10000 -Dspark.blockManagerHeartBeatMs=800000
-Dspark.shuffle.spill=false

But SPARK_EXECUTOR_MEMORY is showing 4g in web UI.Do I need to change it anywhere else to
make it 4g and to reflect it in web UI.

A warning is coming that blockManagerHeartBeatMs is exceeding 450000 while executing a process
even though I set it to 800000.

So I doubt whether it should be set  as SPARK_MASTER_OPTS or SPARK_WORKER_OPTS..
 
Thanks & Regards, 
Meethu M
Mime
View raw message