spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yangcheng Huang <>
Subject RE: Spark on Mesos Issue - Do I need to install Spark on Mesos slaves
Date Fri, 10 Oct 2014 14:51:30 GMT
spark.executor.uri        hdfs://????:9000/user/????/spark-1.1.0-bin-hadoop2.4.tgz

From: Bijoy Deb []
Sent: den 10 oktober 2014 11:59
Subject: Spark on Mesos Issue - Do I need to install Spark on Mesos slaves

I am trying to submit a Spark job on Mesos using spark-submit from my Mesos-Master machine.
My SPARK_HOME = /vol1/spark/spark-1.0.2-bin-hadoop2

I have uploaded the spark-1.0.2-bin-hadoop2.tgz to hdfs so that the mesos slaves can download
it to invoke the Mesos Spark backend executor.

But on submitting the job, I can see the below error in 'stderr' logs on the Mesos slave machine:

sh: /vol1/spark/spark-1.0.2-bin-hadoop2/sbin/spark-executor: No such file or directory
Based on documentation,I understand that if I keep the spark-mesos binary file in hdfs,I dont
need to install Spark separately on the slave nodes.So, the SPARK_HOME or /vol1/spark/spark-1.0.2-bin-hadoop2/
path is non-existent on any of my slave machines and hence the error.
Now, my questions is:
Shouldn't the mesos-slave be looking for the spark-executor command in the temporary directory
where it is supposed to extract the spark-1.0.2-bin-hadoop2.tgz from hdfs,instead of the SPARK_HOME
directory?What am I doing wrong here?
Any help would be really appreciated.

View raw message