spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grandl Robert <rgra...@yahoo.com.INVALID>
Subject Re: run spark standalone mode
Date Thu, 12 Mar 2015 15:57:35 GMT
I figured out for spark-shell by passing the --master option. However, still troubleshooting
for launching sql queries. My current command is like that:
./bin/beeline -u jdbc:hive2://ms0220:10000 -n `whoami` -p ignored -f tpch_query10.sql
 

     On Thursday, March 12, 2015 10:37 AM, Grandl Robert <rgrandl@yahoo.com.INVALID>
wrote:
   

 Hi guys,
I have a stupid question, but I am not sure how to get out of it. 
I deployed spark 1.2.1 on a cluster of 30 nodes. Looking at master:8088 I can see all the
workers I have created so far. (I start the cluster with sbin/start-all.sh)
However, when running a Spark SQL query or even spark-shell, I cannot see any job executing
at master webUI, but the jobs are able to finish. I suspect they are executing locally on
the master, but I don't understand why/how and why not on slave machines. 

My conf/spark-env.sh is as following:export SPARK_MASTER_IP="ms0220"
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/users/rgrandl/software/spark-1.2.1-bin-hadoop2.4/lib/snappy-java-1.0.4.1.jar

export SPARK_LOCAL_DIRS="/users/rgrandl/software/data/spark/local"

export SPARK_WORKER_MEMORY="52000M"
export SPARK_WORKER_INSTANCES="2"
export SPARK_WORKER_CORES="2"

export SPARK_WORKER_DIR="/users/rgrandl/software/data/spark/worker"
export SPARK_DAEMON_MEMORY="5200M"
#export SPARK_DAEMON_JAVA_OPTS="4800M"


While conf/slaves is populated with the list of machines used for workers. I have to mention
that spark-env.sh and slaves files are deployed on all machines. 

Thank you,Robert



   
Mime
View raw message