spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From juanpedromoreno <juanpmor...@gmail.com>
Subject Problem with Spark on Mesos
Date Tue, 04 Mar 2014 12:20:12 GMT
Hi, I'm using vagrant. I've built a cluster with MESOS and SPARK with the
following structure:

- 2 zookeeper nodes
- 2 master nodes  
- 3 slave nodes

In each master node, I've installed MESOS and Spark. 
My $SPARK_HOME/conf/spark_env.sh contains:

export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI=http://dominio.com/spark/spark-0.9.0-bin-cdh4.tgz
export MASTER=zk://192.168.30.101:2181,192.168.30.102:2181

The issue I'm having is that executors, launched by the slaves nodes, always
download the spark distribution from
http://dominio.com/spark/spark-0.9.0-bin-cdh4.tgz. 
Disk usage of slave nodes is continuously increasing, in addition to the
introduced latency which is due to downloading and decompressing the tgz
file. Is there any way to avoid it?

Thanks in advance.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-with-Spark-on-Mesos-tp2283.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message