spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jimfcarroll <>
Subject Re: Problem with version compatibility
Date Thu, 25 Jun 2015 13:01:14 GMT
Hi Sean,

I'm packaging spark with my (standalone) driver app using maven. Any
assemblies that are used on the mesos workers through extending the
classpath or providing the jars in the driver (via the SparkConf) isn't
packaged with spark (it seems obvious that would be a mistake).

I need, for example, "RDD" on my classpath in order for my driver app to
run. Are you saying I need to mark spark as provided in maven and include an
installed distribution's lib directory jars on my classpath?

I'm not using anything but the jar files from a Spark install in my driver
so that seemed superfluous (and slightly more difficult to manage the
deployment). Also, even if that's the case, I don't understand why the maven
dependency of the same version of a deployable distribution would have
different versions of classes in it than the deployable version itself.

Thanks for your patience.

View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message