spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Dzhagriev <dzh...@gmail.com>
Subject --jars for mesos cluster
Date Tue, 03 May 2016 16:11:18 GMT
Hello all,

In the Mesos related spark docs (
http://spark.apache.org/docs/1.6.0/running-on-mesos.html#cluster-mode) I
found this statement:

Note that jars or python files that are passed to spark-submit should be
> URIs reachable by Mesos slaves, as the Spark driver doesn’t automatically
> upload local jars.



However, when I'm trying to submit the job using:

./spark-submit --class MyDriver --master mesos://my-mesos:7077
--deploy-mode cluster --supervise --executor-memory 2G --jars
http://3rd-party.jar http://my.jar

I don't see the --jars on the slave machine:

sh -c './bin/spark-submit --name MyDriver --master mesos://my-mesos:7077
--driver-cores 1.0 --driver-memory 1024M --class MyDriver --executor-memory
2G ./my.jar '

And of course I'm running into the ClassNotFoundException as the 3rd party
library is not there.

Can someone, please, help how to specify the --jars correctly?

Thanks, Alex.

Mime
View raw message