spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Problem with version compatibility
Date Thu, 25 Jun 2015 13:50:24 GMT
Yes spark-submit adds all this for you. You don't bring Spark classes in
your app

On Thu, Jun 25, 2015, 4:01 PM jimfcarroll <jimfcarroll@gmail.com> wrote:

> Hi Sean,
>
> I'm packaging spark with my (standalone) driver app using maven. Any
> assemblies that are used on the mesos workers through extending the
> classpath or providing the jars in the driver (via the SparkConf) isn't
> packaged with spark (it seems obvious that would be a mistake).
>
> I need, for example, "RDD" on my classpath in order for my driver app to
> run. Are you saying I need to mark spark as provided in maven and include
> an
> installed distribution's lib directory jars on my classpath?
>
> I'm not using anything but the jar files from a Spark install in my driver
> so that seemed superfluous (and slightly more difficult to manage the
> deployment). Also, even if that's the case, I don't understand why the
> maven
> dependency of the same version of a deployable distribution would have
> different versions of classes in it than the deployable version itself.
>
> Thanks for your patience.
> Jim
>
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Problem-with-version-compatibility-tp12861p12889.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>

Mime
View raw message