spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <lian.cs....@gmail.com>
Subject Re: SparkSQL Thriftserver in Mesos
Date Fri, 26 Sep 2014 12:18:14 GMT
You can avoid install Spark on each node by uploading Spark distribution 
tarball file to HDFS setting |spark.executor.uri| to the HDFS location. 
In this way, Mesos will download and the tarball file before launching 
containers. Please refer to this Spark documentation page 
<http://spark.apache.org/docs/latest/running-on-mesos.html> for details.

However, using |spark.executor.uri| together with fine-grained mode 
(which is the default mode) really kills performance, because Mesos 
downloads and extracts the tarball every time a Spark /task/ (not 
application) is launched.

On 9/21/14 1:16 AM, John Omernik wrote:

> I am running the Thrift server in SparkSQL, and running it on the node 
> I compiled spark on.  When I run it, tasks only work if they landed on 
> that node, other executors started on nodes I didn't compile spark on 
> (and thus don't have the compile directory) fail.  Should spark be 
> distributed properly with the executor uri in my spark-defaults for 
> mesos?
>
> Here is the error on nodes with Lost executors
>
> sh: 1: /opt/mapr/spark/spark-1.1.0-SNAPSHOT/sbin/spark-executor: not found

‚Äč

Mime
View raw message