spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grandl Robert <rgra...@yahoo.com.INVALID>
Subject run spark standalone mode
Date Thu, 12 Mar 2015 15:33:16 GMT
Hi guys,
I have a stupid question, but I am not sure how to get out of it. 
I deployed spark 1.2.1 on a cluster of 30 nodes. Looking at master:8088 I can see all the
workers I have created so far. (I start the cluster with sbin/start-all.sh)
However, when running a Spark SQL query or even spark-shell, I cannot see any job executing
at master webUI, but the jobs are able to finish. I suspect they are executing locally on
the master, but I don't understand why/how and why not on slave machines. 

My conf/spark-env.sh is as following:export SPARK_MASTER_IP="ms0220"
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/users/rgrandl/software/spark-1.2.1-bin-hadoop2.4/lib/snappy-java-1.0.4.1.jar

export SPARK_LOCAL_DIRS="/users/rgrandl/software/data/spark/local"

export SPARK_WORKER_MEMORY="52000M"
export SPARK_WORKER_INSTANCES="2"
export SPARK_WORKER_CORES="2"

export SPARK_WORKER_DIR="/users/rgrandl/software/data/spark/worker"
export SPARK_DAEMON_MEMORY="5200M"
#export SPARK_DAEMON_JAVA_OPTS="4800M"


While conf/slaves is populated with the list of machines used for workers. I have to mention
that spark-env.sh and slaves files are deployed on all machines. 

Thank you,Robert


Mime
View raw message