spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Spark-submit not working when application jar is in hdfs
Date Sat, 28 Mar 2015 17:04:31 GMT
Looking at SparkSubmit#addJarToClasspath():

    uri.getScheme match {
      case "file" | "local" =>
...
      case _ =>
        printWarning(s"Skip remote jar $uri.")

It seems hdfs scheme is not recognized.

FYI

On Thu, Feb 26, 2015 at 6:09 PM, dilm <dmendoza@exist.com> wrote:

> I'm trying to run a spark application using bin/spark-submit. When I
> reference my application jar inside my local filesystem, it works. However,
> when I copied my application jar to a directory in hdfs, i get the
> following
> exception:
>
> Warning: Skip remote jar
> hdfs://localhost:9000/user/hdfs/jars/simple-project-1.0-SNAPSHOT.jar.
> java.lang.ClassNotFoundException: com.example.SimpleApp
>
> Here's the comand:
>
> $ ./bin/spark-submit --class com.example.SimpleApp --master local
> hdfs://localhost:9000/user/hdfs/jars/simple-project-1.0-SNAPSHOT.jar
>
> I'm using hadoop version 2.6.0, spark version 1.2.1
>
> In the official documentation‌​, it stated there that: "application-jar:
> Path to a bundled jar including your application and all dependencies. The
> URL must be globally visible inside of your cluster, for instance, an
> *hdfs:// path* or a file:// path that is present on all nodes." I'm
> thinking
> maybe this is a valid bug?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-not-working-when-application-jar-is-in-hdfs-tp21840.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message