spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lbustelo <>
Subject spark-submit fails to get jar from http source
Date Fri, 13 Jun 2014 19:41:35 GMT
I'm running a 1.0.0 standalone cluster based on amplab/dockerscripts with 3
workers. I'm testing out spark-submit and I'm getting errors using
*--deploy-mode cluster* and using an http:// url to my JAR. I'm getting the
following error back.

Sending launch command to spark://master:7077
Driver successfully submitted as driver-20140613191831-0009
... waiting before polling master for driver state
... polling master for driver state
State of driver-20140613191831-0009 is ERROR
Exception from cluster was: No FileSystem for scheme:

I verified that my jar URL is accessible from within the spark nodes
(workers and master). I also ran with the same URL and *--deploy-mode
client*, and things worked. 

Documentation at in section
/Advanced Dependency Management/ suggest that this should work.

Is this a known issue, or are my expectations wrong?

PS - I find it very limiting that spark-submit would not take care of
uploading my jar to the cluster. This is a fundamental requirement that most
frameworks support (i.e Storm, Hadoop, etc.). I do not consider this to be a
requirement specific to the JobServer work, rather part of the master's api.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message