You can avoid install Spark on each node by uploading Spark distribution tarball file to HDFS setting spark.executor.uri to the HDFS location. In this way, Mesos will download and the tarball file before launching containers. Please refer to this Spark documentation page for details.

However, using spark.executor.uri together with fine-grained mode (which is the default mode) really kills performance, because Mesos downloads and extracts the tarball every time a Spark task (not application) is launched.

On 9/21/14 1:16 AM, John Omernik wrote:

I am running the Thrift server in SparkSQL, and running it on the node I compiled spark on.  When I run it, tasks only work if they landed on that node, other executors started on nodes I didn't compile spark on (and thus don't have the compile directory) fail.  Should spark be distributed properly with the executor uri in my spark-defaults for mesos?  

Here is the error on nodes with Lost executors 

sh: 1: /opt/mapr/spark/spark-1.1.0-SNAPSHOT/sbin/spark-executor: not found